Transferring Data To and From Quest#
Often researchers will need to move data onto or off of the high-performance computing cluster in order to perform their research. This page provides a high-level overview of the different tools that are available to you for data transfer.
Caution
Sensitive and restricted data cannot be stored on Quest. See Quest Data Security Guidance for more details.
Note
Quest does not permit the use of ftp for data transfers. If your vendor provides you with a ftp download, please request an sftp alternative.
Overview#
There are several options for transferring data to and from Quest:
Globus (recommended): transfer files through a web application or command line interface; works with all file sizes, and is the best option for reliably transferring large files
Command Line Tools: options from transferring files when working in a terminal
Quest OnDemand File Explorer: transfer moderately-sized (< 5 GB) files between your computer and Quest through a web browser interface
Quest Analytics Nodes: those working on the Analytics Nodes can transfer small (< 500 MB) files through the RStudio, Jupyter, and SAS Studio interfaces
FastX GNOME Desktop: useful for transferring files that are accessed through a web browser, such as files in OneDrive or other online storage platforms
Cloud Storage Tools: software modules are available for cloud storage platform transfer tools
How do I choose?
We recommend using Globus whenever possible. However, your use case and file size often dictates the tools that you use. It’s most important to pay attention to the size of data being transferred and if the method can handle the amount of data.
Globus#
Globus is the recommended tool for transferring data to and from Quest. Globus can be used to transfer data between your personal computer and Quest, Quest and RDSS/FSMResFiles, and Quest and Globus endpoints at other institutions or data storage platforms. It manages large file transfers, handling network interruptions and other issues.
See the Globus documentation to learn how to use Globus with Quest.
Command Line Tools#
Several standard programs are available for transferring files between Quest and other systems, including your personal computer, from the command line.
Firewall Considerations when using Command Line Tools#
When transferring data to and from Quest with command line tools, it is important to consider what node you are running the command on and whether the tool will work on that node. Below is a box that summarize common tools or transfer protocols and whether they are allowed on a given node type on Quest.
Protocol/Tool |
Login/KLC Nodes |
Data Transfer Node ( |
Compute Nodes |
|---|---|---|---|
|
❌ |
❌ |
❌ |
|
✅ |
✅ |
✅ |
|
✅ |
✅ |
✅ |
|
❌ |
✅ |
❌ |
|
✅ |
✅ |
✅ |
Note
rsync is available on Quest. However, you will almost always rsync files from your local computer to Quest. Because of this, it is not included in the table above.
scp (Secure Copy)#
Note
scp is good to use when you need one-off transfers of small files/folders (~0-1GB in size) from your laptop to Quest.
Secure Copy (scp ) can be used to transfer files between Quest and your personal computer. scp is available in the terminal applications on Linux and MacOS systems. To use scp on a Windows computer, utilize the built-in PowerShell functionality or install PuTTY . Windows users may need to substitute the command pscp for scp in the examples below.
scp commands are executed on your computer, NOT on Quest. The first argument is where the file is currently, and the second is where the file should be transferred to.
To transfer files from your computer to Quest:
$ scp /PathToSourceFile/file.txt <netid>@login.quest.northwestern.edu:/PathToTargetDir/file.txt
For example, to transfer a file called myfile.txt in the current working directory on your computer to your home directory on Quest, the command might look like:
$ scp myfile.txt abc123@login.quest.northwestern.edu:/home/abc123/myfile.txt
To transfer files from Quest to your computer:
$ scp <netid>@login.quest.northwestern.edu:/PathToSourceDir/file.txt /PathToTargetDir/file.txt
Example, transferring a file from Quest into your current working directory with the same name:
$ scp abc123@login.quest.northwestern.edu:/home/myfile.txt .
For more examples of Secure Copy, see examples from Indiana University or this Geeks for Geeks tutorial .
Warning
scp will overwrite existing files without prompting.
rsync#
Note
rsync is good to use if you have a folder which contains small files (~0-1GB in size) that you routinely want synced between Quest and your laptop/workstation.
rsync is a Linux
command that can be used to synchronize the contents of folders either on the same computer or on two different computers, such as your personal computer and Quest.
Linux and Mac users will have rsync available automatically,
whereas Windows users will need to install and use PowerShell
or PuTTY .
Example using rsync to sync the contents of a directory called “myfolder” in the home directory of a user’s personal computer to their home directory on Quest. The user’s NetID is abc123
Run rsync from your computer, NOT Quest. Push local files to Quest.
## List the contents of ~/myfolder
$ ls ~/myfolder
myfile1.txt myfile2.txt
## Push local directory and files to Quest. This will run in archive mode (a), with compression (z) and verbosely (v)
$ rsync -avz myfolder abc123@login.quest.northwestern.edu:~
building file list ... done
myfolder/
myfolder/myfile1.txt
myfolder/myfile2.txt
sent 215 bytes received 70 bytes 190.00 bytes/sec
total size is 0 speedup is 0.00
Create a new file in ~/myfolder on Quest:
$ echo "hello world" > ~/myfolder/myfile3.txt
Pull the changes from Quest to your computer. Run these commands from your computer:
$ rsync -avz abc123@login.quest.northwestern.edu:~/myfolder .
receiving file list ... done
myfolder/
myfolder/myfile3.txt
sent 44 bytes received 213 bytes 171.33 bytes/sec
total size is 12 speedup is 0.05
## Note that the only file that is transferred or modified in anyway is the newly create file.
$ ls ~/myfolder
myfile1.txt myfile2.txt myfile3.txt
Warning
rsync will overwrite files in the destination which are different than the source without prompting.
wget#
wget can be used to download data from external servers to Quest. It can be used with files available over HTTP/HTTPS.
Run wget from Quest to download files directly to Quest.
Examples:
$ wget http://www.nber.org/nberces/bbg96_87.dta
$ wget https://github.com/pytorch/pytorch/releases/download/v1.12.0/pytorch-v1.12.0.tar.gz
With additional options, wget can also download whole directories of files
recursively and download from password-protected servers.
FastX GNOME Desktop#
To transfer files between Quest and storage platforms that are accessed via a web browser, such as OneDrive or Google Drive, use FastX to launch the Firefox web browser from Quest.
Connect to Quest with the FastX Desktop Client or FastX HTTP Client, and then open a GNOME Terminal. Type firefox to launch the Firefox web browser.
For large data transfers between OneDrive or Sharepoint and Quest, use the Globus OneDrive Collection instead.
Cloud Storage#
AWS S3#
The Amazon Web Services Command Line Interface (AWS CLI) is installed on Quest as a module. It can be used for file transfer with S3 buckets as described in the AWS CLI user guide .
Load the AWS CLI on Quest by running:
$ module load awscli/2.15.22
Google Cloud SDK#
To transfer files between Quest and a Google Cloud Storage location, use the Google Cloud Software Development Kit Command Line Interface (SDK CLI), which is installed as a module on Quest. To load this package, run:
$ module load gcloud/522.0.0
Following the instructions in the Google Cloud SDK Documentation to configure the CLI for your Google Cloud account.