Singularity on Quest

Singularity on Quest#

Singularity is a container tool available on Quest. Containers offer a way to package software along with its dependencies in a format that can be run by any computer. The advantages of containers include portability, reproducibility, and freedom for users to run their software of choice on any system, without having to rely on system administrators to install, configure, and maintain software. Singularity is a software container technology designed for scientific workloads on traditional HPC environments such as Quest. In addition to the benefits offered by other container technologies, Singularity has security features that limit the privileges of a container to be the same as the user running the container. This prevents containers from affecting shared resources such as hardware or networking that could impact other users. A Singularity container is also saved in a single file, called an image, making containers portable and easy to manage.

Prefer Videos?

Try the Singularity video series.

Load the Singularity Module#

Singularity is installed as a module on Quest. To use it, either interactively or in a submission script, you must first load a Singularity module:

$ module load singularityce/4.3.1-gcc-8.5.0

As with any other software modules, you can see all available versions with:

$ module spider singularity

Getting Singularity Images On Quest#

In many cases, the application that you are interested in using on Quest may already have been containerized and made available and downloadable from public repositories such as those listed below. In these cases, the container image can be downloaded using singularity pull. Singularity can pull Docker and NVIDIA NGC images directly.

Where Can I Find Container Images?#

Pulling Images#

When pulling an image, Singularity needs to know where the image is hosted and its name. In general, the syntax looks like <host>://<name>. For example,

$ singularity pull docker://ubuntu

will pull the Ubuntu Docker image into a Singularity Image File (SIF) called ubuntu_latest.sif.

By default, Singularity will pull the image with the tag latest but it is possible to pull a specific tag:

$ singularity pull docker://ubuntu:groovy

This will create ubuntu_groovy.sif, which is used in the examples below.

To download and run a small example container:

$ singularity pull hello-world.sif docker://hello-world
$ singularity run hello-world.sif
Hello from Docker!

Building Custom Singularity Images#

If the application you are interested in using is not publicly available as a container or the configuration of the container that is available does not suit your needs, then it is also possible to build your own container image on a Linux computer where you have root access or by using the Singularity remote builder . You can then copy or pull that container image to Quest to run it. To build a container using Singularity, you must first define a recipe file which is a set of instructions that tells Singularity all the commands that need to be run in order to install your application and in what environment those commands need to be run. Afterwards, you can use the singularity build command to use the recipe file to build a container.

In general, singularity build requires sudo privileges and cannot be run on Quest. Build the image on a Linux computer with sudo privileges, such as a virtual machine or cloud instance, or use Singularity’s remote builder with singularity build --remote.

Below, we describe how you go about making a Singularity Recipe file and what you should be thinking about as you construct it. A small example Singularity Recipe called myrecipe.def is provided below. The Singularity documentation has a detailed description of the Recipe file format.

myrecipe.def
    Bootstrap: docker   ### Where does the container live that you would like to build on top of.
    From: ubuntu:groovy    ### What is the name of the container that you would like to build on top of.

    ###%post: Imagine you had sudo privileges and a terminal window open, what commands would you run to install your program. Remember that you have access to all the functions and applications that exist in the container that you are building on top of. In our case, we are running on an Ubuntu OS container so we have access/the ability to install programs via apt-get.

    %post
      # Avoid prompts while the image is built remotely.
      export DEBIAN_FRONTEND=noninteractive

      # Upgrade system libraries for ubuntu 21.04 and install libraries we need for our program
      apt-get -y update && apt-get -y install python3 python3-pandas

      # Anything you could do on the command line, you can do here
      # lets make a directory
      mkdir -p /opt/bin

      # Make Python code
      echo "import pandas" > /opt/bin/example.py
      echo "print(\"I imported pandas...\")" >> /opt/bin/example.py

    #### %runscript Imagine you had just finished installing your program using the instructions in %post, what command(s) would you use to run it?

    %runscript 
      python3 /opt/bin/example.py

In order to build this container using the Singularity cloud builder, you would run the following commands

# Authenticate against the remote builder. Will request that you create a token and copy the token in order to authenticate.
$ singularity remote login
# Build the container in the remote builder
$ singularity build --remote myname.sif myrecipe.def

Running Singularity Containers#

There are multiple ways to run a Singularity container.

Interactive shell#

To run an interactive shell in a container (in order to explore the applications installed inside of it), use the singularity shell command. When you use the singularity shell command, it is best to imagine that you have logged into a completely new computer. In many instances, you will find that programs are installed inside of the container in the same location as they are on Quest and even have the same name. Despite this, these applications are completely unique and different than those installed on Quest. For example,

# The following is run on the Quest login node. The application "grep" exists on
# the Quest login nodes and is version 3.1.
$ which grep
/usr/bin/grep
$ grep --version
grep (GNU grep) 3.1

# We start an interactive shell inside of the container. Even though "grep" is installed in the container in the same location as on the Quest login node, it is a completely different version of the application.
$ singularity shell ubuntu_groovy.sif
Singularity> which grep
/usr/bin/grep
Singularity> grep --version
grep (GNU grep) 3.6

By default all environment variables (such as $HOME) from the host shell session where you execute singularity shell will be passed into the container shell environment unless they have been defined in the container already (e.g. by a Singularity recipe file or Dockerfile). The -e flag to singularity shell will run a shell in the container with a “clean” environment, that is without passing in environment variables from the host. For more details, see the Singularity documentation .

Run The Default Command With run#

Singularity recipes have a %runscript section that defines the command run by singularity run or when the image is executed directly. When an image is imported from Docker, the Dockerfile’s ENTRYPOINT instruction is used as the runscript. For example, the hello-world.sif image downloaded above prints a message:

$ singularity run hello-world.sif
Hello from Docker!

Specify The Command To Run With exec#

Some containers do not have a runscript or may have multiple executables that you need to choose from. The singularity exec command specifies the exact command to run inside the container and is the preferred way to run Singularity containers:

$ grep --version
grep (GNU grep) 3.1
$ singularity exec ubuntu_groovy.sif grep --version
grep (GNU grep) 3.6

Binding Directories#

Singularity allows you to map directories on the host system to directories within your container using bind mounts. This allows you to read and write data on the host system with ease. On Quest, $HOME, /tmp, and /proc are mapped by default from the host into the container.

The --bind or -B option to the Singularity shell, run, and exec commands allows the user to specify additional directories from the host file system to map into the container. If the absolute path to a directory on the host is provided, it has the same folder structure and location inside the container. For example, --bind=/projects/pXXXXX makes the contents of /projects/pXXXXX available at /projects/pXXXXX inside the container. Optionally, specify a different location in the container with <absolute-path-to-host-directory>:<path-in-container>, such as --bind=/projects/pXXXXX:/my-folder. The --bind or -B option can be specified multiple times, or as a comma-delimited string of bind path specifications.

Note

All files and folders created in a bound folder will persist after exiting the container and be available on the host file system in the corresponding directory.

# A folder which exists on Quest but is not one of /home, /tmp or /proc
$ ls /projects/intro
alphafold  class_log  data-shell  data-shell.zip  envs  errlog  examplejobs  genomics.sh  helloworld.py  job-script.sh  python_output.txt  test.sh  whereami.sh

# Without specifying --bind/-B, no folders in /projects are mapped onto the container by default and therefore cannot be found.
$ singularity shell ubuntu_groovy.sif
Singularity> ls /projects/intro
ls: cannot access '/projects/intro': No such file or directory

# If you map the Quest directory /projects into the container then you can list the contents of the folder just like you can on the Quest login node.
$ singularity shell -B /projects:/projects ubuntu_groovy.sif
Singularity> ls /projects/intro
alphafold  class_log  data-shell  data-shell.zip  envs  errlog  examplejobs  genomics.sh  helloworld.py  job-script.sh  python_output.txt  test.sh  whereami.sh

In this example, any files and folders you can normally access in /projects will also be accessible inside of the container.

Singularity Containers In Batch Jobs#

Singularity commands can be run in a batch submission script just like any other command. One approach to Singularity is to pull images at runtime and run them from a temporary directory, so they do not take up space on the system when not being used. Here is one such example job submission script:

jobscript.sh
#!/bin/bash
#SBATCH -A <account name>
#SBATCH -p <partition name>
#SBATCH -t 02:00:00
#SBATCH -N 1
#SBATCH -n 1
#SBATCH --mem=4G
#SBATCH --job-name="<job name>"

module load singularityce/4.3.1-gcc-8.5.0

# Store the image cache in your Quest scratch space.
export SINGULARITY_CACHEDIR=/scratch/<netid>/singularity-cache
mkdir -p "$SINGULARITY_CACHEDIR"

# Run the container
singularity exec -B /projects/pXXXXX/data:/data docker://biocontainers/blast blastx -query /data/seqs.fasta -out /data/output.blast.txt

This script assumes the seqs.fasta file is in the /projects/pXXXXX/data directory, which is mapped into the container as /data using the -B option. The output will be written to /projects/pXXXXX/data/output.blast.txt.

GPU Support#

Singularity supports GPU hardware via the --nv option, which uses the NVIDIA drivers installed on the host system inside the container. If you have access to a GPU allocation on Quest, you can utilize it within a Singularity container. For more information, see the Singularity Container section of Using GPUs on Quest for an example of running a GPU-enabled container on Quest and the Singularity GPU Support page for more information.

MPI Support#

Singularity does integrate with the Message Passing Interface (MPI) and OpenMPI version 4.0.5 is supported out of the box. Quest’s mpi/openmpi-4.0.5-gcc-10.2.0 module should be used for best results, and OpenMPI must be installed inside the container. See the Singularity documentation on Open MPI Hybrid Container for more information. Assuming you have installed OpenMPI 4.0.5 inside of your container following the instructions from Singularity, then you should be able to run the following:

$ mpirun -np 4 singularity exec my_mpi_container.sif /script/to/run

Creating Modules to Wrap Calls to Containers#

It can be very tedious to call containers by the full command singularity exec -B /projects:/projects PATH/TO/CONTAINER/FILE/anaconda3_latest.sif. To alleviate this burden, you can create your own module which will create a convenience shell/bash function such that calling programs that actually live inside of the container will look and feel like calling programs that exist on Quest itself. Below we demonstrate how to create a module that will create a bash function python3 which actually runs the Python3 installed inside of an anaconda3 container.

$ mkdir -p modules/python3
$ touch modules/python3/3.8.8.lua
$ cat modules/python3/3.8.8.lua
help([[
Python 3.8.8 from anaconda
]])

local pkgName = "python3"
local version = "3.8.8"

whatis("Name: " .. pkgName)
whatis("Version: " .. version)

depends_on("singularity")

local bashStr = 'singularity exec -B /projects:/projects /projects/pXXXXX/Singularity/part4/anaconda3_latest.sif python3 $@'
set_shell_function("python3",bashStr)

Once you have created the .lua module file, you can then add this module to your $PATH and can see how loading it will create a shell function called python3 which really wraps a singularity exec call.

$ module use modules/
$ module load python3/3.8.8
$ type python3
python3 is a function
python3 ()
{
  singularity exec -B /projects:/projects /projects/pXXXXX/Singularity/part4/anaconda3_latest.sif python3 $@
}
$ python3 --version
Python 3.8.8

Best Practices#

  • Custom container images should contain a single application. Do not try to bundle many different executables in your image.

  • Use singularity exec to specify which command to run in your container rather than relying on singularity run.

  • When creating custom Singularity images, don’t put anything in $TMP or $HOME as those directories will be bind mounted at runtime.

  • Read the Singularity user documentation for more detail on all Singularity functionality.