Mamba Virtual Environments on Quest

Mamba Virtual Environments on Quest#

Mamba is a fast, drop-in replacement for the Conda package and environment manager. It is used primarily for open-source data science packages for the Python and R programming languages, and it also supports languages such as C, C++, Fortran, Java, Scala, Ruby, and Lua.

Mamba and Conda create isolated environments containing the package and software versions needed for a project. This keeps package versions and dependencies separate between projects. Mamba and Conda share the same commands, environments, and configuration. This page uses mamba because it resolves and installs packages faster, but you can substitute conda for any mamba command. For a general introduction to environments, see Conda’s Managing environments guide.

Prefer Videos?

Try watching the Anaconda Virtual Environments workshop recording.

Loading Mamba#

Mamba is installed as a software module on Quest. To make the mamba and conda commands available, first load the module:

$ module purge
$ module load mamba/24.3.0

As with any other software module, you can see all available versions with:

$ module spider mamba

One-Time Setup#

Run the following commands once to configure your shell so that mamba and conda are ready every time you log in, without loading the module:

$ module load mamba/24.3.0
$ mamba init bash

This modifies your ~/.bashrc file. Log out of Quest and log back in for the change to take effect. After this setup, use mamba activate (or conda activate) to activate environments. The older source activate syntax still works, so existing batch scripts do not need to change.

Newer Versions and Configuration#

To use a newer version of Mamba than the module provides, install Miniforge into your home directory:

$ curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
$ bash Miniforge3-$(uname)-$(uname -m).sh

After the installer completes, start a new shell session or run source ~/miniforge3/bin/activate to use the installed version.

Mamba and Conda support many configuration options. See Using the .condarc configuration file for details.

Creating and Managing Environments#

Mamba/Conda environments are isolated project environments that manage distinct package requirements and dependencies for different projects. Creating and using an environment has three basic steps:

Create an environment with mamba create.
Activate the environment with mamba activate.
Install packages into the environment with mamba install.

By default, an environment’s packages are stored in your home directory under .conda/, and only you can use it. To create an environment that other members of your Project can use, create it in your /projects directory as described in Shared Environments in Project Directories.

To create a new environment in your home directory, enter:

$ mamba create --name <env_name>

where <env_name> is the name you want for your environment. Then activate it:

$ mamba activate <env_name>

Once activated, install packages into the environment:

$ mamba install <pkg>

The following commands create and activate an environment named my-virtenv-py312 with only Python 3.12 installed, located at ~/.conda/envs/my-virtenv-py312:

$ mamba create --name my-virtenv-py312 python=3.12 --yes
$ mamba activate my-virtenv-py312

The --yes option accepts the package-installation confirmation automatically. Omit it if you prefer to review and confirm the proposed installation.

You have full read, write, and execute privileges inside the environment, so you can install any packages you need. Use either mamba/conda or Python’s native package manager, pip , to install Python packages into the activated environment.

For example, after activating my-virtenv-py312, install additional packages with Mamba:

$ mamba install -c conda-forge scipy scikit-learn

If a Python package is not available through a Conda channel, install it with pip in the activated environment:

$ python -m pip install <package-name>

The following command creates an environment named my-virtenv-R-4.4 with only R 4.4 installed, located at ~/.conda/envs/my-virtenv-R-4.4:

$ mamba create --name my-virtenv-R-4.4 -c conda-forge r-base=4.4 --yes

To view a list of all your environments, enter:

$ mamba env list

To deactivate the active environment, enter:

$ mamba deactivate

To remove an environment, enter:

$ mamba env remove --name <env_name>

Installing Additional Packages#

You can install additional packages in a virtual environment once it has been created. First, make sure the environment is active.

$ mamba activate my-virtenv-py312

It is recommended to use the mamba install command. Mamba will prompt for any changes to existing packages prior to the installation. The example below installs the pandas and matplotlib packages into the environment.

$ mamba install pandas matplotlib

Some packages require using pip for installation. The pip command will install the package and make changes to existing packages without prompting you. This can create package conflicts for more complicated virtual environments.

pip install pytorch

Conda Channels#

Most research software for R, Python, Julia, and other languages is available through Conda and Mamba from one of three channels. A channel is the remote repository in which they look for a package:

anaconda is the default channel.
conda-forge is a community-maintained collection of packages.
bioconda provides bioinformatics software.

Specify a channel with the -c option, for example -c conda-forge. To help ensure that packages are compatible with each other, install all packages in an environment from the same channel.

For example, if you need C or C++ compilers in an environment, install them from conda-forge:

$ mamba install -c conda-forge gxx gcc

Shared Environments in Project Directories#

To share an environment with other members of your Project, create it in a project directory with the --prefix option instead of --name. Activate it by its full path rather than its name.

For example, to create an environment at /projects/pXXXXX/pythonenvs/env1 with the sqlalchemy package and Python 3.12, where pXXXXX is your storage account:

$ module load mamba/24.3.0
$ cd /projects/pXXXXX
$ mkdir pythonenvs
$ mamba create --prefix /projects/pXXXXX/pythonenvs/env1 sqlalchemy python=3.12 --yes

To use this environment, specify its full path when activating it:

$ module load mamba/24.3.0
$ mamba activate /projects/pXXXXX/pythonenvs/env1

Using Mamba in a Slurm Job#

After creating an environment, activate it in the Slurm job script before running your application. A job script that uses a Mamba/Conda environment should look like the following:

#!/bin/bash
#SBATCH --account=<account-name>
#SBATCH --partition=<partition-name>
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --mem=16GB
#SBATCH --time=1:00:00

module purge

eval "$(conda shell.bash hook)"

mamba activate /projects/pXXXXX/pythonenvs/env1

python script.py

conda shell.bash hook initializes the shell functions required for mamba activate; this command intentionally uses conda because Mamba uses Conda’s shell-initialization mechanism.

Accessing Environments in Jupyter#

To use an environment in Jupyter, install it as a Jupyter kernel from the command line. This cannot be done from within a Jupyter notebook.

Add a Python Environment#

Connect to the Quest login nodes.
Create an environment containing the ipykernel package, or add ipykernel to an existing environment. For example, to create one:

$ module purge
$ module load mamba/24.3.0
$ mamba create --name jupyter-kernel-py312 -c conda-forge python=3.12 numpy pandas matplotlib ipykernel --yes

To add ipykernel to an existing environment instead:

$ mamba activate <env_name>
$ mamba install -c conda-forge ipykernel

Activate the environment and connect it to Jupyter:

$ mamba activate jupyter-kernel-py312
$ python -m ipykernel install --user --name jupyter-kernel-py312 --display-name "Python (jupyter-kernel-py312)"

In this command, python -m ipykernel install adds the active environment as a Jupyter kernel; --user installs it only for your account; --name sets the kernel identifier; and --display-name sets its label in Jupyter. The kernel is available under the “New” menu. Packages added to the environment later are automatically available when using this kernel.

Add an R Environment#

Connect to the Quest login nodes.
Create an environment containing the r-irkernel package, or add r-irkernel to an existing environment. For example, to create one:

$ module purge
$ module load mamba/24.3.0
$ mamba create --name r-kernel -c conda-forge r-irkernel r=4.4 jupyterlab --yes

To add r-irkernel to an existing environment instead:

$ mamba activate <env_name>
$ mamba install -c conda-forge r-irkernel

Activate the environment and connect it to Jupyter:

$ mamba activate r-kernel
$ R -e "IRkernel::installspec(name = 'ir44', displayname = 'R 4.4')"

The last command adds the active environment to your list of available environments in Jupyter, available under the “New” menu. Packages added to this environment later are automatically available in its Jupyter kernel.

Jupyter New menu showing the installed Conda environment kernels

Tip

To list all installed Jupyter kernels, run jupyter kernelspec list.