Using Nightingale

Using Nightingale#

NCSA maintains more comprehensive user documentation for Nightingale . This page is intended to point out usage differences between Nightingale and other Northwestern systems that our researchers might be more familiar with.

Logging in to Nightingale#

Use Approved, Managed Devices Only

Researchers can only connect to Nightingale from encrypted, Northwestern-owned and managed devices, not personal devices.

After you have an account on Nightingale, log in to the system using a terminal application on your encrypted and Northwestern-owned and managed desktop or laptop. Because of the added security for Nightingale, the log in nodes are behind a secure bastion node. You can either first log in to Nightingale’s secure bastion node and then log in to a general access login node or you can use one command to jump hosts through the bastion node to a log in node.

Option 1: Two Step Log In

  1. Log in to the secure bastion with the ssh command, replacing <username> with your NCSA username.

ssh <username>@ngale-bastion-1.ncsa.illinois.edu

  1. When prompted, enter your NCSA password and hit “Enter”. Note that the terminal will not show the password being typed, but it is registering your input.

  2. When prompted for Duo authentication, if you are using Duo Mobile, enter 1 to send a push to your phone. If you are using a Duo hardware token, you will be prompted for a passcode from the hardware token. You will know when you have made it to the secure bastion because there will be a welcome message ending in “NCSA - All Sessions Being Recorded” and your command prompt will change to be your NCSA username@ngale-bastion-1.

  3. From the secure bastion, ssh to a general login node.

ssh ng-login01.ngale.internal.ncsa.edu

Or,

ssh ng-login02.ngale.internal.ncsa.edu

You do not need to provide your username when ssh-ing from the bastion node to the log in node. A very similar welcome message will appear when you are logged into the log in node, ending in “This server reboots each Monday of each month at 01:00 CDT” and your command prompt will change to your username@ng-login01 or username@ng-login02.

Option 2: One Step Jump Hosts

Combine the two ssh commands by adding the -J flag, which indicates you are jumping hosts. In the command below, replace <username> with your NCSA username.

ssh -J <username>@ngale-bastion-1.ncsa.illinois.edu <username>@ng-login01.ngale.internal.ncsa.edu

This allows you to type only one ssh command instead of two. You will still be prompted for your NCSA password, Duo authentication, and then your NCSA password again before landing on the login node.

More information about logging onto Nightingale can be found in the NCSA User Guide .

Software on Nightingale#

Nightingale Module System#

Nightingale uses a module system similar to Quest. However, there are a few modules loaded by default that are essential to normal Nightingale functioning so the command module restore should be used instead of module purge to reset your environment and unload all user-loaded modules. The Nightingale software page provides more information on what modules are available and how to load and unload them.

R Package Installs#

To enable R package installs, users must put the following in their .bashrc file or execute it every time they log into Nightingale:

export HTTPS_PROXY=http://ache-proxy.ncsa.illinois.edu:3128

The R module recommended for use on Nightingale is R/4.5.3. Users may install R packages from the terminal as follows:

  1. Load the module

module load R/4.5.3

  1. Enter an R console

R

  1. Install desired packages, for example:

install.packages("tidyverse")

R package installs cannot be done from within RStudio on Nightingale; they must be done from the terminal.

Conda Virtual Environments#

To enable building conda environments, users must put the following in their .bashrc file or execute it every time they log into Nightingale:

export HTTPS_PROXY=http://ache-proxy.ncsa.illinois.edu:3128

Both Anaconda and Miniconda are available as modules on Nightingale and can be used to build conda virtual environments. Use module avail to list the available modules and load whichever you prefer to use to make conda available in your environment. The Using Python on Nightingale page provides more information on building conda environments on Nightingale. While virtual environments are popular for managing python packages, they can be used to install many other pieces of software as well. Search conda-forge or bioconda to see if what you want to use is available.

Submitting Jobs on Nightingale#

Nightingale uses Slurm, just like Quest. However, the partition and account names are different. Your account name is provided in your welcome email and is usually the name of your project. The partitions name is cpu for CPU jobs, or the name of the type of GPU card you’re reserving for GPU jobs. See the NCSA User Documentation for the full breakdown of partitions and their capacity.

The default memory per CPU on Nightingale is 10GB. Please use --mem to adjust that if you need more or less memory. Note that unlike on Quest, if you request an amount of memory higher that what is available on any of the nodes in the partition, your job will not fail but it will be queued indefinitely without starting.