Building/Customizing Containers

Introduction:

Containers offer a way to encapsulate a whole, sometimes complex, environment into a single file. It also allows software that was built in one environment to be able to be run in another environment. Probably the most widely known container system is Docker. Unfortunately, Docker poses security problems in HPC environments and Singularity (now called Apptainer) was created to address those issues. The ACG clusters use Singularity/Apptainer and thankfully, it was designed to be able to run Docker containers. So, there is a large set of containers that already exist that can be run on the ACG systems. However, often a container has most, but not all, of what you need. For instance, a TensorFlow container might not have all of the Python packages installed that you need. This page will show you how to create a container from an existing container and add packages into it.

The Definition file:

A text file, called a Definition file, is used to describe what needs to be done in order to build the Apptainer container. Here is an example:

Bootstrap: docker                              # Start from a Docker container

From: rocker/geospatial                        # Specify the starting container from Docker Hub

%files                                         # Copy files from the host system into the container

                                               # In this case,                                                                     

/opt/ohpc/pub/JAGS/gnu8-4.3.0 /opt/ohpc/pub/JAGS/gnu8-4.3.0

%environment

export PATH=/opt/ohpc/pub/JAGS/gnu8-4.3.0/bin:$PATH

export LD_LIBRARY_PATH=/opt/ohpc/pub/JAGS/gnu8-4.3.0/lib

export INCLUDE=/opt/ohpc/pub/JAGS/gnu8-4.3.0/include

export MANPATH=/opt/ohpc/pub/JAGS/gnu8-4.3.0/man

%post

export PATH=/opt/ohpc/pub/JAGS/gnu8-4.3.0/bin:$PATH

export LD_LIBRARY_PATH=/opt/ohpc/pub/JAGS/gnu8-4.3.0/lib

export INCLUDE=/opt/ohpc/pub/JAGS/gnu8-4.3.0/include

export MANPATH=/opt/ohpc/pub/JAGS/gnu8-4.3.0/man

R --vanilla -e 'install.packages("rjags", repos="http://cran.us.r-project.org")'

R --vanilla -e 'install.packages("landscapemetrics", repos="http://cran.us.r-project.org")'

R --vanilla -e 'install.packages("dismo", repos="http://cran.us.r-project.org")'

R --vanilla -e 'install.packages("randomForest", repos="http://cran.us.r-project.org")'

R --vanilla -e 'install.packages("kernlab", repos="http://cran.us.r-project.org")'

R --vanilla -e 'install.packages("tidyverse", repos="http://cran.us.r-project.org")'

R --vanilla -e 'install.packages("jagsUI", repos="http://cran.us.r-project.org")'

# %runscript

# R $@

%help

This container is the R rocker/geospatial container with the addition of the 

landscapemetrics R package. 

Clicking on the TensorFlow container will give a bunch of information on the container, including how to use it:

The path to the container can be found by clicking on the "Copy Image Path" button and choosing one of the versions of the container. This will be used when we need to create a Definition file. As an example, one for TensorFlow 2 points to: nvcr.io/nvidia/tensorflow:22.07-tf2-py3 and this will be put into the new Definition file on the second line that starts with "From: ". It's usually best to use the newest version unless you know of an incompatibility issue.

Creating the Definition file with the Nvidia Container path:

On Katahdin, open a text editor and create a new file called "new_container.def with the following contents, where the second line includes the path that you got from the NGC site in the previous step:

Bootstrap: docker

From: nvcr.io/nvidia/tensorflow:22.07-tf2-py3

%post

    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \

        coreutils  

    pip install matplotlib

    echo "Done"

The red line is what is being added to the container. You can add other things to the container using the "apt" command, other pip commands, and other ways too.

Save the file and then run the following commands to actually create the new container. The end result will be a file with a ".sif" extension that is placed in your home directory. The process will start by sshing to a system that has Nvidia GPUs. This might not always be necessary but it also takes the load off of the login node.

ssh node-g103

module load apptainer

export TMPDIR=$XDG_RUNTIME_DIR 

apptainer build --fakeroot $HOME/new_container.sif new_container.def

The third line sets up the TMPDIR variable for the apptainer command to use. This is done for a couple of reasons but the biggest benefit is that the XDG_RUNTIME_DIR variable points to a tmpfs volume that gets created when you ssh to the GPU system. This tmpfs volume is located in RAM so it is very fast. Setting TMPDIR to this directory in RAM can speed up the process of creating the container tremendously. 

The singularity command runs the "build" subcommand to build the .sif file and it uses the .def file to know how to build it. The "--fakeroot" parameter is needed in order for regular, not-root, accounts to be able to build the container.

Once the container has been created, you can use the container in a Slurm job with the following in your job submission script:

module load apptainer

apptainer run --nv new_container.sif python my_python_script.py

where "my_python_script.py" is the name of your python script.