Singularity on Quest

Singularity on Quest#

This page describes how to leverage containers on Quest through the tool Singularity.

Prefer Videos?

Try the Singularity video series.

Overview#

Container technology offers a way to package software along with its dependencies in a format that can be run by any computer. The advantages of containers include portability, reproducibility, and freedom for users to run their software of choice on any system, without having to rely on system administrators to install, configure, and maintain software. Singularity is a software container technology designed for scientific workloads on traditional HPC environments such as Quest. In addition to the benefits offered by other container technologies, Singularity has security features that limit the privileges of a container to be the same as the user running the container. This prevents containers from affecting shared resources such as hardware or networking that could impact other users. Singularity also uses a simple image format that makes moving images as easy as copying a single file.

Load the Singularity Module#

Singularity is installed as a module on Quest. To use it, either interactively or in a submission script, you must first load a Singularity module:

$ module load singularityce/4.3.1-gcc-8.5.0

As with any other software modules, you can see all available versions with:

$ module spider singularity

Getting Singularity Images On Quest#

In many cases, the application that you are interested in using on Quest may already have been containerized and made available and downloadable from public repositories such as those listed below. In these cases, the container can be downloaded using singularity pull.

Where Can I Find Container Images?#

Pulling Images#

When pulling containers Singularity needs to know two pieces of information, where the container is hosted and the name of the container. In general the syntax looks like <host>://<name>. For example,

$ singularity pull docker://ubuntu

will pull the ubuntu docker image into a Singularity Image File (SIF) called ubuntu_latest.sif.

By default, Singularity will pull the image with the tag latest but it is possible to pull a specific tag:

$ singularity pull docker://ubuntu:groovy

This will create ubuntu_groovy.sif.

Building Custom Singularity Images#

If the application you are interested in using is not publicly available as a container or the configuration of the container that is available does not suite your needs, then it is also possible to build your own container image on a Linux computer where you have root access or by using the Singularity remote builder . You can then copy or pull that container image to Quest to run it. To build a container using singularity, you must first define a recipe file which is a set of instructions that tells singularity all the commands that need to be run in order to install your application and in what environment those commands need to be run. Afterwards, you can use the singularity build command to use the recipe file to build a container.

In general, singularity build requires sudo privileges and thus cannot be run on Quest. You will need to run it on a Linux computer with sudo privileges. A virtual machine created with the free VirtualBox software or a cloud computing instance will work. You can also use Singularity’s remote builder via singularity build --remote.

Below, we describe how you go about making a Singularity Recipe file and what you should be thinking about as you construct it. Below we provide a small example of a Singularity Recipe called myrecipe.def but encourage you to look at the Singularity documentation for detailed description of the Recipe file format.

In order to build this container using the Singularity cloud builder, you would run the following commands

# Authenticate against the remote builder. Will request that you create a token and copy the token in order to authenticate.
$ singularity remote login
# Build the container in the remote builder
$ singularity build --remote myname.sif myrecipe.def

Running Singularity Containers#

There are multiple ways to run a Singularity container.

Interactive shell#

To run an interactive shell in a container (in order to explore the applications installed inside of it), use the singularity shell command. When you use the singularity shell command, it is best to imagine that you have logged into a completely new computer. In many instances, you will find that programs are installed inside of the container in the same location as they are on Quest and even have the same name. Despite this, these applications are completely unique and different than those installed on Quest. For example,

# The following is run on the Quest login node. The application "grep" exists on the Quest login nodes and is version 2.20
$ which grep
/usr/bin/grep
$ grep --version
grep (GNU grep) 2.20

# We start an interactive shell inside of the container. Even though "grep" is installed in the container in the same location as on the Quest login node, it is a completely different version of the application.
$ singularity shell ubuntu_hirsute.sif
Singularity> which grep
/usr/bin/grep
Singularity> grep --version
grep (GNU grep) 3.6

By default all environment variables (such as $HOME) from the host shell session where you execute singularity shell will be passed into the container shell environment unless they have been defined in the container already (e.g. by a Singularity recipe file or Dockerfile). The -e flag to singularity shell will run a shell in the container with a “clean” environment, that is without passing in environment variables from the host. For more details, see the Singularity documentation .

Run The Default Command With run#

Singularity recipes have a %runscript section which defines exactly which command will run when the container is run with the singularity run command or directly as an executable file. When an image is imported from Docker, the Dockerfile’s ENTRYPOINT instruction is used as the runscript. For example, the nuitrcs/hello-world container’s runscript is a shell script that prints “Hello world!”:

$ singularity run shub://nuitrcs/hello-world
Hello world!

Specify The Command To Run With exec#

Some containers do not have a runscript or may have multiple executables that you need to choose from. The singularity exec command is used to specify the exact command to run inside the container and is the preferred way to run singularity containers:

$ grep --version
grep (GNU grep) 2.20
$ singularity exec ubuntu_hirsute.sif grep --version
grep (GNU grep) 3.6

Binding Directories#

Singularity allows you to map directories on the host system to directories within your container using bind mounts. This allows you to read and write data on the host system with ease. On Quest, $HOME, /tmp, and /proc are mapped by default from the host into the container.

The --bind or -B option to the Singularity shell, run, and exec commands allows the user to specify additional directories from the host file system to map into the container. If the absolute path to a directory on the host is provided, than it will have the same folder structure/location inside of the container. For example, providing the following --bind=/projects/pXXXXX would make the contents of /projects/pXXXXX available in the location /projects/pXXXXX inside of the container. Optionally, users can specify a different name/location for the directory inside of the container using the format <absolute-path-to-host-directory>:<name-of-directory-in-container>. For example, providing the following --bind=/projects/pXXXXX:/my-folder would make the contents of /projects/pXXXXX available in the location /my-folder inside of the container. The --bind or -B option can be specified multiple times, or a comma-delimited string of bind path specifications can be used.

Note

All files and folders created in a bound folder will persists after exiting container and be available on the host file system in the corresponding directory.

# A folder which exists on Quest but is not one of /home, /tmp or /proc
$ ls /projects/intro
test.sh whereami.sh

# Without specifying --bind/-B, no folders in /projects are mapped onto the container by default and therefore cannot be found.
$ singularity shell ubuntu_hirsute.sif
Singularity> ls /projects/intro
ls: cannot access '/projects/intro': No such file or directory

# If you map the Quest directory /projects into the container then you can list the contents of the folder just like you can on the Quest login node.
$ singularity shell -B /projects:/projects ubuntu_hirsute.sif
Singularity> ls /projects/intro
test.sh whereami.sh

In this example, any files and folders you can normally access in /projects will also be accessible inside of the container.

Singularity Containers In Batch Jobs#

Singularity commands can be run in a batch submission script just like any other command. One approach to Singularity is to pull images at runtime and run them from a temporary directory, so they do not take up space on the system when not being used. Here is one such example job submission script:

This script assumes the seqs.fasta file is in the /project/pXXXXX/data directory, which is mapped into the container as /data using the -B option. The output will be written to /project/pXXXXX/data/output.blast.txt.

GPU Support#

Singularity supports GPU hardware via the --nv option, which uses the Nvidia drivers installed on the host system inside the container. If you have access to a GPU allocation on Quest, you can utilize it within a Singularity container. For more information, please see the Singularity Container section of Using GPUs on QUEST for an example of running a GPU enabled container on Quest and Singularity’s GPU Support page for more information.

MPI Support#

Singularity does integrate with the Message Passing Interface (MPI) and OpenMPI version 4.0.5 is supported out of the box. Quest’s mpi/openmpi-4.0.5-gcc-10.2.0 module should be used for best results, and OpenMPI will need to be installed inside the container. Please see the Singularity documentation on Open MPI Hybrid Container for more information. Assuming you have installed OpenMPI 4.0.5 inside of your container following the instructions from Singularity, then you should be able to run the following:

$ mpirun -np 4 singularity exec my_mpi_container.simg /script/to/run

Creating Modules to Wrap Calls to Containers#

It can be very tedious to call containers by the full command singularity exec -B /projects:/projects PATH/TO/CONTAINER/FILE/anaconda3_latest.sif. To alleviate this burden, you can create your own module which will create a convenience shell/bash function such that calling programs that actually live inside of the container will look and feel like calling programs that exist on Quest itself. Below we demonstrate how to create a module that will create a bash function python3 which actually runs the Python3 installed inside of an anaconda3 container.

$ mkdir -p modules/python3
$ touch modules/grep/3.8.8.lua
$ cat modules/python3/3.8.8.lua
help([[
Python 3.8.8 from anaconda
]])

local pkgName = "python3"
local version = "3.8.8"

whatis("Name: " .. pkgName)
whatis("Version: " .. version)

depends_on("singularity")

local bashStr = 'singularity exec -B /projects:/projects /projects/w10001/tempuser03/Singularity/part4/anaconda3_latest.sif python3 $@'
set_shell_function("python3",bashStr)

Once you have created the .lua module file, you can then add this module to your $PATH and can see how loading it will create a shell function called python3 which really wraps a singularity exec call.

$ module use modules/
$ module load python3/3.8.8
$ type python3
python3 is a function
python3 ()
{
  singularity exec -B /projects:/projects /projects/w10001/tempuser03/Singularity/part4/anaconda3_latest.sif python3 $@
}
$ python3 --version
Python 3.8.8

Best Practices#

Custom container images should contain a single application. Do not try to bundle many different executables in your image.
Use singularity exec to specify which command to run in your container rather than relying on singularity run.
When creating custom Singularity images, don’t put anything in $TMP or $HOME as those directories will be bind mounted at runtime.
Read the Singularity user documentation for more detail on all Singularity functionality.