Moving Data to the Cloud#
If you are not starting from scratch, one of your first tasks after receiving access to your new Northwestern cloud account will be moving your existing data to the cloud. Whether your data is on a local machine, Quest, RDSS, or another cloud environment, the options below cover the most common approaches for getting your data into your Northwestern cloud account. The right transfer method depends on where your data is coming from, how much data you have, and your comfort with command-line tools.
Before You Begin
You must have an active Northwestern cloud account before transferring data. See Requesting a Cloud Account if you have not yet set one up. You will also need a bucket or container, the storage space where your files will land. If one was not created during your account onboarding, see Create an Object Store Bucket below.
Situation |
Recommended Method |
|---|---|
Moving data from Quest (Northwestern’s HPC cluster) or RDSS (Research Data Storage Service) to AWS |
Globus |
Transferring large datasets (several GB or more) to AWS |
Globus |
Uploading a small number of files quickly |
Web Console |
Automating or scripting data workflows |
Command-Line Tools or Python |
Moving data from a laptop or personal computer |
Web Console or Command-Line Tools |
Moving data to Azure or Google Cloud |
Command-Line Tools |
Globus is available for transfers to AWS only; it does not support Azure Blob Storage or Google Cloud Storage as a destination. If you are unsure which method fits your situation, request a Research Cloud Consult .
Working With Sensitive or Regulated Data?
Confirm that your destination is approved for your data classification before moving data. See Cloud Security Best Practices for guidance.
Data Transfer Costs#
Moving data into your Northwestern cloud account can incur costs beyond ongoing storage, including per-file upload charges and, for some workflows, temporary storage charges when data passes through a standard storage tier before transitioning to a lower-cost class. These costs vary by cloud provider, storage class, and the number and size of files you transfer.
For a detailed cost breakdown and examples, see Data Transfer Costs in the Archiving Data in Amazon S3 Glacier Deep Archive tutorial.
Create an Object Store Bucket#
Most research data transferred to the cloud lands in object storage first, such as Amazon S3 , Azure Blob Storage , or Google Cloud Storage . From there, it can be accessed by virtual machines, analysis pipelines, or shared with collaborators.
In cloud object storage, a bucket (AWS and GCP) or container (Azure) is the top-level storage space where your files are kept. If a bucket or container was not created during your account onboarding, create one before transferring data.
See Creating an Amazon S3 Storage Bucket for step-by-step instructions, including guidance on choosing a region, configuring bucket settings, and finding your bucket’s unique identifier.
Sign in to the Azure Portal .
Click +Create to create a storage account.
Under Subscription, choose your default subscription, tied to your Azure account name.
Under Resource group, create a new resource group and provide a name.
Under Storage account name, choose a globally unique name.
Under Region, select (US) Central US.
Under Performance, leave the default set to Standard.
Leave the remaining settings at their defaults.
Click Review + Create, then click Create.
Wait for deployment to finish, then click Go to Resource.
In your new storage account, go to Data storage > Containers.
Click + Container, provide a name, and click Create.
Sign in to the Google Cloud Storage Console .
Click Create.
Provide a globally unique name.
Under Choose where to store your data, select region us-central1 (Iowa).
Click Create.
Install Command Line Tools for Your Cloud Provider#
Install and authenticate your provider’s CLI before transferring data from the command line. If you plan to use Globus or the web console instead, you can skip this section.
The AWS CLI is the primary tool for transferring data to and from Amazon S3. It supports multipart upload for large files, syncing directories, and scripted or automated workflows.
Installation
Download and install the AWS CLI from the official AWS documentation:
After installation, open your terminal and run the following to confirm it installed correctly:
aws --version
Authentication
Before transferring files, you must authenticate your AWS CLI session. Northwestern supports two methods:
Option 1: NetID Single Sign-On (recommended)
Northwestern provides SSO authentication using your NetID, which does not require managing separate access keys. See Use NetID Authentication with AWS for setup instructions.
Option 2: Access key configuration
If you have been issued an AWS access key and secret key, run the following command and follow the prompts:
aws configure
If you are unsure which method to use or need help getting set up, contact the Cloud Enablement Team via a Research Cloud Consult .
For full usage instructions, see the AWS CLI documentation .
The Azure CLI supports data transfer to and from Azure Blob Storage, including uploading, downloading, and syncing across Storage accounts and containers.
Installation
Download and install the Azure CLI from the official Microsoft documentation:
After installation, open your terminal and run the following to confirm it installed correctly:
az --version
Authentication
Before transferring files, sign in to your Azure account by running the following command. A browser window will open prompting you to sign in with your Northwestern credentials:
az login
For full usage instructions, see the Azure CLI documentation .
The GCP CLI provides gcloud for general Google Cloud operations and gsutil for interacting
with Google Cloud Storage. gsutil supports parallel composite uploads, resumable transfers, and
syncing, making it well suited for automated data workflows.
Installation
Download and install the Google Cloud CLI from the official Google documentation:
After installation, open your terminal and run the following to confirm it installed correctly:
gcloud --version
Authentication
Before transferring files, sign in to your Google Cloud account by running the following command. A browser window will open prompting you to sign in with your Northwestern credentials:
gcloud auth login
For full usage instructions, see the Google Cloud CLI documentation .
Transfer Your Data#
Using Globus#
Globus is the recommended option for transferring large datasets from Northwestern storage systems such as Quest or RDSS to Amazon S3. It provides high-performance, secure transfers with features including parallel transfers, encryption in transit, automated retries, and checksum verification. Globus does not require comfort with the command line and is a good starting point for most researchers moving large volumes of data to AWS.
Globus Supports AWS Only
Northwestern does not provide Globus collections for Azure or Google Cloud. To transfer data to those platforms, use the Web Console or Command-Line Tools sections below.
Globus runs server-side, meaning transfers continue even if you close your browser, lose your VPN connection, or shut down your computer. This makes it reliable for large or long-running transfers.
Northwestern provides Globus collections for Quest, RDSS, and Amazon S3. See the Globus documentation for setup instructions.
Using the Web Console#
Each cloud provider offers a web-based console where you can upload files directly from your browser with no software installation required. This is the simplest option for uploading a small number of files or getting started quickly. It is not recommended for large datasets or automated workflows.
Sign in to the AWS Console .
Go to the S3 service.
Click the name of your bucket.
Click Upload.
Click Add files or Add folder to browse your computer.
Click Upload to start the transfer.
Sign in to the Azure Portal .
Navigate to your Storage Account.
Under Data storage, click Containers, then click the name of your container.
Click Upload.
Browse and select your files.
Click Upload to start the transfer.
Sign in to the Google Cloud Storage Console .
Click the name of your bucket.
Click Upload, then select Upload files to choose individual files, or Upload folder to upload a full directory.
Select your files and the upload will begin automatically.
To verify your upload completed successfully, open your bucket or container after uploading and confirm your files appear with the expected names and sizes.
Using the Command Line#
Command-line tools are well suited for users comfortable with scripting and automation. They support uploading, downloading, syncing, and managing files, and can handle large files through multipart or resumable transfers. Install and authenticate your provider’s CLI (see Install Command Line Tools for Your Cloud Provider) before running the commands below.
Command-line transfers require an active computer with an uninterrupted network connection for the duration of the transfer. If your connection drops or your computer goes to sleep, the transfer may fail and need to be restarted. For large or long-running transfers to AWS, Globus is the more reliable option. If you are not comfortable with the command line, use the web console, or Globus for AWS transfers.
Use the AWS CLI to copy a file to your bucket:
aws s3 cp /path/to/local/file s3://your-bucket-name/
To upload an entire folder, use sync instead of cp:
aws s3 sync /path/to/local/folder s3://your-bucket-name/
For full command reference, see the AWS CLI S3 documentation .
Use the Azure CLI to upload a single file to your container:
az storage blob upload --account-name your-account-name --container-name your-container-name --name your-file-name --file /path/to/local/file
To upload an entire folder, use upload-batch instead:
az storage blob upload-batch --account-name your-account-name --destination your-container-name --source /path/to/local/folder
For full command reference, see the Azure CLI storage blob documentation .
Use gcloud storage to copy a file to your bucket:
gcloud storage cp /path/to/local/file gs://your-bucket-name/
To upload an entire folder, add the --recursive flag:
gcloud storage cp --recursive /path/to/local/folder gs://your-bucket-name/
For full command reference, see the gcloud storage documentation .
Using Python#
Each provider also offers a Python library for programmatic access to cloud storage, such as the AWS SDK for Python (Boto3), the Azure SDK for Python, and the Google Cloud client libraries for Python. These are well suited for building custom or automated data transfer workflows. Request a Research Cloud Consult to discuss building a Python-based workflow for your project.
Getting Help#
For large-scale or complex transfers, request a Research Cloud Consult and the RCDS Team will help you determine the most efficient and secure approach for your workload.