Mamba or Conda Virtual Environments

Mamba or Conda Virtual Environments#

Mamba is a package and environment manager primarily used for open-source data science packages for the Python and R programming languages. It also supports other programming languages like C, C++, FORTRAN, Java, Scala, Ruby, and Lua.

Prefer Videos?

Try watching the Anaconda Virtual Environments workshop recording.

Using Mamba on Quest#

To use Mamba, first load the corresponding module:

$ module purge
$ module load mamba/24.3.0

“conda activate” vs. “source activate”:#

Older conda/mamba versions used the source activate <environment name> syntax to activate an environment on Linux, which differed from how environments were activated on Windows. In order for conda to work consistently across platforms, its behavior changed slightly in version 4.6, and now conda/mamba activate <environment name> works on all platforms. To use the new conda, run these commands one time to set up your environment:

$ module load mamba/24.3.0
$ mamba init bash

This modifies your ~/.bashrc file so that mamba/conda is ready to use every time you log in (without needing to load the module).

Then log out of Quest and log back in. Once you have done this, you can use conda activate instead of source activate, although the latter will still work, so existing batch scripts for example can continue to use source activate.

If you want a newer version of Mamba than what is available in the module, you can also install them into your HOME directory as follows

$ curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
$ bash Miniforge3-$(uname)-$(uname -m).sh

Conda/Mamba can also be configured with various options. Read more about Conda/Mamba configuration .

Creating Mamba/Conda environments and installing packages#

You can create new Mamba/Conda environments in one of your available directories. Mamba/Conda environments are isolated project environments designed to manage distinct package requirements and dependencies for different projects. We recommend using the mamba command for faster package solving, downloading, and installing. However, you can use the conda command, with various options, to install and inspect Conda environments.

The process for creating and using environments has a few basic steps:

  • Create an environment with mamba create

  • Activate the environment with mamba activate

  • Install packages into the environment with mamba install

To create a new Conda environment in your home directory, enter:

$ mamba create --name <env_name>

where <env_name> is the name your want for your environment. Then activate the environment:

$ mamba activate <env_name>

Once activated, you can then install packages into that environment:

$ mamba install <pkg>

Below we demonstrate examples of using anaconda to install Python and R. The commands below will create an isolated environment in your HOME folder in the following location ~/.conda/envs/my-virtenv-py312 with only Python 3.12 installed.

$ mamba create --name my-virtenv-py312 python=3.12 --yes
$ mamba activate my-virtenv-py312

Once the environment is activated, you have full control over what Python packages are installed into the environment. In many ways, you are the system administrator of this Python installation as you have full read/write/execute privileges inside of that folder in your HOME directory. At this point, you can use either conda to install additional Python packages into the environment or Python’s native package manager, PyPi , to install whatever Python package(s).

The commands below will create an isolated environment in your HOME folder in the following location ~/.conda/envs/my-virtenv-R-4.4 with only R 4.4 installed.

$ mamba create --name my-virtenv-R-4.4 -c conda-forge r-base=4.4 --yes

Explaining Anaconda “channels”#

An overwhelming majority of research software (including R, Python, Julia and more) is available via anaconda in one of three locations. These locations are called “channels”, which can be thought of as the remote/cloud repository in which anaconda looks for the package. These three locations are

–channel=conda-forge

–channel=anaconda  (a.k.a the default location)

–channel=bioconda

If you do opt to install packages directly from anaconda (which can help insure that all the packages are compatible with each other), it is good practice to make sure they are all installed from the same remote repository/channel.

Finally, if you are need of C and/or C++ compilers you can install those directly into your environment via

$ mamba install -c conda-forge gxx gcc

To share an environment with other members of your allocation, you can create the environment in a specific directory.

With conda, use the --prefix option to create the environment in a specific location, such as in your /projects/<account-name> directory. To use the environment (either interactively from the command line or as part of your job submission script), you then need to point to this directory when activating it.

For example, to create a conda environment called env1 in a subdirectory of a projects directory (here p10000 as an example), with the package sqlalchemy included, run the following:

$ module load mamba/24.3.0
$ cd /projects/p10000
$ mkdir pythonenvs
$ mamba create --prefix /projects/p10000/pythonenvs/env1 sqlalchemy python=3.12 --yes

To use this environment, specify the full path to the environment when you activate it:

$ module load mamba/24.3.0
$ source activate /projects/p10000/pythonenvs/env1

Please note that a version of the main application you are using (e.g., Python or R) is installed in the Conda environment, so the module versions of these should not be loaded when the Conda environment is activated. Enter module purge to unload all loaded modules.

To deactivate an environment, enter:

$ mamba deactivate

You can also create a new environment in your projects directory instead using the --prefix option. For example:

$ mamba create --prefix /projects/<account-name>/<env_name>

where <account-name> is the name of your allocation. Then activate the environment:

$ mamba activate /project/<project_id>/<env_name>

To view a list of all your Conda environments, enter:

$ mamba env list

To remove a Conda environment, enter:

$ mamba env remove --name <env_name>

Using Anaconda in a Slurm Job#

In order to submit jobs to the Slurm job scheduler, you will need to use the main application you are using with your Conda environment in batch mode. There are a few steps to follow:

  1. Create an application script

  2. Create a Slurm job script that runs the application script

  3. Submit the job script to the job scheduler with sbatch

Your application script should consist of the sequence of commands needed for your analysis.

A Slurm job script is a special type of Bash shell script that the Slurm job scheduler recognizes as a job. For a job using Anaconda, a Slurm job script should look something like the following:

#!/bin/bash
#SBATCH --account=<account-name>
#SBATCH --partition=<partition_name>
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --mem=16GB
#SBATCH --time=1:00:00

module purge

eval "$(conda shell.bash hook)"

conda activate /projects/pXXXXX/env

python script.py

Accessing Mamba/Conda Environments in Jupyter#

By default, jupyter will use the “root” environment of the Anaconda module you have loaded. If you are using a conda environment, for example to be able to install specific versions of software or because you are sharing an environment with other people in your group, you will need to manually install the environment into your jupyter configuration. This can only be done from the command line, not within the jupyter notebook. First, load whichever Anaconda module you are using and activate your environment, for example:

Note

To list all available Jupyter Kernels, run

$ module load python-anaconda3/2019.10
$ jupyter kernelspec list

Add a Python Environment#

  1. Connect to the Quest login nodes.

  2. Create a new conda environment if you don’t already have one. The environment needs to contain the ipykernel package. For example:

$ module purge all
$ module load mamba/24.3.0
$ mamba create --name jupyter-kernel-py312 -c conda-forge python=3.12 numpy pandas matplotlib ipykernel --yes

  1. Activate the environment and connect the environment to Jupyter. For example, for an environment named jupyter-kernel-py312

$ source activate jupyter-kernel-py312
$ python -m ipykernel install --user --name jupyter-kernel-py312 --display-name "Python (jupyter-kernel-py312)"

The last command adds the environment to your list of available environments in Jupyter, and once you have done this it will be available for you under the “New” menu.

Add an R Environment#

  1. Connect to the Quest login nodes.

  2. Create a new conda environment if you don’t already have one. The environment needs to contain the r-irkernel package. For example:

$ module purge all
$ module load mamba/24.3.0
$ mamba create -c conda-forge --name r-kernel r-irkernel r=4.4 jupyterlab

  1. Activate the environment and connect the environment to Jupyter. For example, for an environment named r-kernel

$ source activate r-kernel
$ R -e "IRkernel::installspec(name = 'ir44', displayname = 'R4.4')"

The last command adds the environment to your list of available environments in Jupyter, and once you have done this it will be available for you under the “New” menu.

jupyter new notebook menu