Python on Quest

Python on Quest#

Using Python on Login Nodes#

Python scripts can be run on the login nodes with some limits. Users cannot use more than 4 cores or 4 GB of memory in total for all the applications they are running on a login node. These limits are implemented to ensure a smooth operation for all users connecting to Quest through login nodes.

All Quest nodes natively include Python (version 3.6.8). However, this Python version has no package support and users are strongly encouraged to use Python distributions provided by the module system. You can see the available versions of Python with the command

$ module spider python

The command output will include several python/<distribution> modules. Currently, only Mamba distributions are actively supported on Quest. You can make a particular version of Python available to use by loading the corresponding module.

To start the interpreter, type python on the command line. You can also run your Python script directly by typing the following line on the command line

$ python <your_script_name.py>

Note

While module load python will load the current default version of Python, to ensure the future compatibility of your scripts with the Quest system, we highly recommend that you always load a specific python module or use python virtual environments.

Scheduled Jobs: Basic Python Use#

Python can be used interactively on log in without scheduling jobs as discussed above. Note that, however, the login node use is for development and non-computationally intensive tests runs. Compute nodes can be accessed via scheduling jobs on Quest. There are two classes of jobs: interactive and batch.

Interactive Use on the Compute Nodes#

To use Python interactively, please first submit an interactive job to obtain the cpu and memory resources you need from a compute node. Once your job starts, your command line will move to a compute node where you can proceed as if you are on the login node. If you are planning to do computationally intensive work while maintaining interactivity with the script or the interpreter, this type of use recommended. This is also the way to request resources to use an IDE such as Spyder (which is available as part of the Anaconda distributions of Python).

Batch Jobs#

In submission scripts for batch jobs, you must include the command to load the version of Python that you want to run, then add the line where you can call Python:

python <your_script_name.py>

which causes the output and errors to be written to the files for stdout, respectively, which are determined by your job submission parameters (see Everything You Need to Know about Using Slurm on Quest).

See Example of Python Batch Jobs for Quest for template files you can adapt for your work.

Python Package Management#

  • Anaconda

  • Using the default packages

  • User-Site Packages

Anaconda Virtual Environments#

You can use conda/mamba environments with the Anaconda/Mamba Python modules on Quest, or virtual environments with any Python module.

Tip

Please review this detailed guide on creating and using Mamba/Conda Virtual Environments on Quest.

Checking Available Packages Provided by Module#

It is important to make sure that the required packages for a Python script are already installed in the Python module you have loaded. If the packages are not available your job will fail. The most reliable way to test the availability is to import the package within the Python interpreter. If you don’t receive any errors after the import, that means the package is available. You can also inquire the version of the package with the Python command <package-name>.__version__ as most packages include this information. For instance if you would like to test Numpy’s existence and query its version you could do the following:

$ module load python-anaconda3/2019.10
$ python
>>> import numpy as np
>>> np.__version__

pip freeze will list packages in your current environment, but the list may be incomplete. It is best to test package availability by importing the package, as above.

Installing and using Python packages when module packages insufficient#

Although you may find references to using pip install --user <mypackage> as a method to install external Python packages on shared computing systems like Quest, we strongly recommend that you utilize virtual environments instead. Using --user to install Python packages locally can often lead to having multiple, sometimes conflicting, versions of a package. For instance, you may want to use TensorFlow which relies on having one version of numpy installed, but also you want to use some other package which requires a version of numpy which is incompatible with that TensorFlow installation. The method --user cannot handle these conflicts the way that having separate isolated virtual environments can.

Jupyter Overview#

To run Jupyter on Quest CPU or GPU compute nodes, please use Quest OnDemand. A video tutorial is available that demonstrates running a Jupyter Lab or Notebook on Quest OnDemand. The Quest Analytics Nodes also provide a way to run JupyterHub.

Tip

To access a virtual environment from within Jupyter, please review Accessing Mamba/Conda Environments in Jupyter.