Transferring Data To and From Quest

Transferring Data To and From Quest#

Often researchers will need to move data onto or off of the high-performance computing cluster in order to perform their research. This page provides a high-level overview of the different tools that are available to you for data transfer.

Caution

Sensitive and restricted data cannot be stored on Quest. See Quest Data Security Guidance for more details.

Overview#

There are several options for transferring data to and from Quest:

  • Globus (recommended): transfer files through a web application or command line interface; works with all file sizes, and is the best option for reliably transferring large files

  • Command Line Tools: options from transferring files when working in a terminal

  • Quest OnDemand File Explorer: transfer moderately-sized (< 5 GB) files between your computer and Quest through a web browser interface

  • Quest Analytics Nodes: those working on the Analytics Nodes can transfer small (< 500 MB) files through the RStudio, Jupyter, and SAS Studio interfaces

  • FastX GNOME Desktop: useful for transferring files that are accessed through a web browser, such as files in OneDrive or other online storage platforms

  • Cloud Storage Tools: software modules are available for cloud storage platform transfer tools

How do I choose?

We recommend using Globus whenever possible. However, your use case and file size often dictates the tools that you use. It’s most important to pay attention to the size of data being transferred and if the method can handle the amount of data.

Globus#

Globus is the recommended tool for transferring data to and from Quest. Globus can be used to transfer data between your personal computer and Quest, Quest and RDSS/FSMResFiles, and Quest and Globus endpoints at other institutions or data storage platforms. It manages large file transfers, handling network interruptions and other issues.

See the Globus documentation to learn how to use Globus with Quest.

Command Line Tools#

Several standard programs are available for transferring files between Quest and other systems, including your personal computer, from the command line.

scp (Secure Copy)#

Note

scp is good to use when you need one-off transfers of small files/folders (~0-1GB in size) from your laptop to Quest.

Secure Copy (scp ) can be used to transfer files between Quest and your personal computer. scp is available in the terminal applications on Linux and MacOS systems. To use scp on a Windows computer, utilize the built-in PowerShell functionality or install PuTTY . Windows users may need to substitute the command pscp for scp in the examples below.

scp commands are executed on your computer, NOT on Quest. The first argument is where the file is currently, and the second is where the file should be transferred to.

To transfer files from your computer to Quest:

$ scp /PathToSourceFile/file.txt <netid>@login.quest.northwestern.edu:/PathToTargetDir/file.txt

For example, to transfer a file called myfile.txt in the current working directory on your computer to your home directory on Quest, the command might look like:

$ scp myfile.txt abc123@login.quest.northwestern.edu:/home/abc123/myfile.txt

To transfer files from Quest to your computer:

$ scp <netid>@login.quest.northwestern.edu:/PathToSourceDir/file.txt /PathToTargetDir/file.txt

Example, transferring a file from Quest into your current working directory with the same name:

$ scp abc123@login.quest.northwestern.edu:/home/myfile.txt .

For more examples of Secure Copy, see examples from Indiana University or this Geeks for Geeks tutorial .

Warning

scp will overwrite existing files without prompting.

rsync#

Note

rsync is good to use if you have a folder which contains small files (~0-1GB in size) that you routinely want synced between Quest and your laptop/workstation.

rsync is a Linux command that can be used to synchronize the contents of folders either on the same computer or on two different computers, such as your personal computer and Quest. Linux and Mac users will have rsync available automatically, whereas Windows users will need to install and use PowerShell or PuTTY .

Example using rsync to sync the contents of a directory called “myfolder” in the home directory of a user’s personal computer to their home directory on Quest. The user’s NetID is abc123

Run rsync from your computer, NOT Quest. Push local files to Quest.

## List the contents of ~/myfolder
$ ls ~/myfolder
myfile1.txt	myfile2.txt

## Push local directory and files to Quest. This will run in archive mode (a), with compression (z) and verbosely (v)
$ rsync -avz myfolder abc123@login.quest.northwestern.edu:~
building file list ... done
myfolder/
myfolder/myfile1.txt
myfolder/myfile2.txt

sent 215 bytes  received 70 bytes  190.00 bytes/sec
total size is 0  speedup is 0.00

Create a new file in ~/myfolder on Quest:

$ echo "hello world" > ~/myfolder/myfile3.txt

Pull the changes from Quest to your computer. Run these commands from your computer:

$ rsync -avz abc123@login.quest.northwestern.edu:~/myfolder .
receiving file list ... done
myfolder/
myfolder/myfile3.txt

sent 44 bytes  received 213 bytes  171.33 bytes/sec
total size is 12  speedup is 0.05 

## Note that the only file that is transferred or modified in anyway is the newly create file.

$ ls ~/myfolder
myfile1.txt    myfile2.txt    myfile3.txt

Warning

rsync will overwrite files in the destination which are different than the source without prompting.

wget#

wget can be used to download data from external servers to Quest. It can be used with files available over HTTP/HTTPS.

Run wget from Quest to download files directly to Quest.

Examples:

$ wget http://www.nber.org/nberces/bbg96_87.dta
$ wget https://github.com/pytorch/pytorch/releases/download/v1.12.0/pytorch-v1.12.0.tar.gz

With additional options, wget can also download whole directories of files recursively and download from password-protected servers.

FastX GNOME Desktop#

To transfer files between Quest and storage platforms that are accessed via a web browser, such as OneDrive or Google Drive, use FastX to launch the Firefox web browser from Quest.

Connect to Quest with the FastX Desktop Client or FastX HTTP Client, and then open a GNOME Terminal. Type firefox to launch the Firefox web browser.

For large data transfers between OneDrive or Sharepoint and Quest, use the Globus OneDrive Collection instead.

Cloud Storage#

AWS S3#

The Amazon Web Services Command Line Interface (AWS CLI) is installed on Quest as a module. It can be used for file transfer with S3 buckets as described in the AWS CLI user guide .

Load the AWS CLI on Quest by running:

$ module load awscli/2.15.22

Google Cloud SDK#

To transfer files between Quest and a Google Cloud Storage location, use the Google Cloud Software Development Kit Command Line Interface (SDK CLI), which is installed as a module on Quest. To load this package, run:

$ module load gcloud/522.0.0

Following the instructions in the Google Cloud SDK Documentation to configure the CLI for your Google Cloud account.