Using Storage Buckets

Google Cloud Storage Buckets are the primary way to store files long-term and share data between users in the SDE environment. This page covers how to move data between buckets and VMs, manage who has access to a bucket, and create or delete buckets.

See the Projects page for details on which specific buckets and VMs each role can access within your project.

Never transfer files to or from your laptop or workstation

Downloading files from any SDE storage bucket to your local computer is a violation of the Acceptable Use Policy. The only exception is an approved Egress bucket.

Uploading files from your local computer to any SDE bucket is also a violation. All data imports must go through the Data Ingress procedures.

When transferring data to or from a VM, run every command from a terminal on the VM, opened through either the VM desktop or an SSH-in-browser session.

---

Before you begin

Read this section once before running any command on this page. Everything here applies to every terminal procedure above.

Know which terminal you are in

You will encounter two different terminals in the SDE. They are not interchangeable, and running a command in the wrong one is the most common source of confusion.

Terminal	Where it runs	What it is for
SSH-in-browser (or a terminal on the VM desktop)	On the VM	All data work: file transfers, scripts, and analysis
Terminal (Mac) or Google Cloud SDK Shell (Windows)	On your own laptop or workstation	Setting up the secure tunnel to reach the VM desktop, nothing else

Every command on this page runs in SSH-in-browser. None of them belong in the Terminal or Google Cloud SDK Shell on your own machine.

Find out where you are with pwd

The terminal always has a current location on the VM filesystem, called the working directory. Think of it as the folder you currently have open.

To see your working directory at any time, run:

pwd

Use the dot shortcut to copy files here

Once you are in the folder where you want files to land, use . (a single dot) as the destination. The . means “the folder I am currently in.”

Instead of typing the full path:

gcloud storage cp gs://my-bucket-name/results.csv /home/john_doe_northwestern/data/

Navigate there first and use .:

gcloud storage cp gs://my-bucket-name/results.csv .

Both commands do exactly the same thing. The cd and . approach is easier to get right, especially with long paths.

Replace every placeholder before you run a command

Commands on this page use placeholder values written in lowercase-with-hyphens. Replace each one with your actual value.

Placeholder	Replace with
source-bucket-name	The bucket you are copying from, e.g. gcs-us-central1-workspace
destination-bucket-name	The bucket you are copying to, e.g. gcs-us-central1-project-data
my-bucket-name	The bucket you are working with, e.g. gcs-us-central1-workspace
file.txt	The actual filename, e.g. sample_data.csv
folder-name	The actual folder name, e.g. results
john_doe_northwestern	Your own username, as shown by pwd

Avoid spaces in file and folder names

Avoid spaces in bucket names, folder names, and filenames in the SDE. Use underscores (_) or hyphens (-) instead, for example my_results rather than my results. Spaces are technically valid but require extra quoting and are a frequent cause of failed commands. If you inherit files that already contain spaces, see Troubleshoot a failed transfer.

---

Find your bucket name

Almost every procedure on this page requires the full bucket name. You can look it up in the Google Cloud Console or list your buckets from the VM.

Bucket names appear in commands in the format gs://your-bucket-name. The gs:// prefix tells gcloud that the path refers to a Cloud Storage bucket rather than a folder on your VM.

Google Cloud Console

1. Log in to the Google Cloud Console.

2. Navigate to Cloud Storage > Buckets .

3. Confirm the correct project is selected in the project dropdown at the top of the page. Buckets from other projects will not appear in this list.

   

4. Find your bucket in the Name column and copy the name exactly as it appears.

SSH-in-browser

Run this command in your SSH-in-browser session or a terminal on the VM desktop to list every bucket your account can reach:

gcloud storage ls

Expected output:

gs://gcs-us-central1-workspace/
gs://gcs-us-central1-project-data/
gs://gcs-us-central1-egress/

To see what is inside one of them, add the bucket path:

gcloud storage ls gs://gcs-us-central1-workspace/

An empty result does not always mean an empty bucket

If the command returns nothing, you may not have permission to list that bucket even if you can read specific files inside it. Contact your Data Engineer to confirm your access.

---

Transfer files

Choose the transfer type that applies to your task:

• Copy or move files from a bucket to a VM

• Copy or move files from a VM to a bucket

• Copy or move files between two buckets

The source and destination can be in different projects. Both Researchers/Data Analysts and Data Engineers can perform transfers on the buckets and VMs they have access to.

Throughout this section, copy (cp) keeps the original file in place, and move (mv) deletes the original once the transfer completes.

Copy or move files from a bucket to a VM

Use this when you need to work with data on a VM, for example to run analysis scripts against a dataset stored in a bucket. This is the most common transfer task for Researchers and Data Analysts.

New to the terminal?

This procedure assumes you know your working directory and how to replace placeholders in a command. If you have not used a terminal before, read Before you begin first.

SSH-in-browser

Values you must substitute

Placeholder	Replace with
destination	The folder on the VM to copy into, e.g. ~/data, or . if you have already navigated into that folder
my-bucket-name	The bucket holding the files you want
file.txt	The name of the file in the bucket
folder-name	The name of the folder in the bucket

Copying files

gcloud storage cp gs://my-bucket-name/file.txt destination

Copying folders

gcloud storage cp -r gs://my-bucket-name/folder-name .

The -r flag means recursive. It copies the folder and everything inside it. Without -r, a folder path will not copy.

This example uses . as the destination, meaning the folder is copied into wherever you are currently working.

If a path or bucket name contains spaces, wrap it in quotes, e.g. "gs://my folder/file.txt"

Moving files

gcloud storage mv gs://my-bucket-name/file.txt destination

Moving folders

gcloud storage mv -r gs://my-bucket-name/folder-name destination

Move is permanent

mv deletes the file or folder from the bucket once it has been copied to the VM. Use cp instead, then delete the bucket copy manually after you confirm the transfer arrived.

Expected output

Copying gs://my-bucket-name/file.txt to file://destination/file.txt
  Completed files 1/1 | 240.5kiB/240.5kiB

To verify a transfer, list the destination folder:

ls -lh destination

Confirm your file or folder appears with a plausible size and current timestamp. If it does not, see Troubleshoot a failed transfer.

Save your results back to a bucket when you finish

Files on a VM are not backed up and can be lost if the VM is rebuilt. When your work is done, follow Copy or move files from a VM to a bucket to store the output somewhere durable.

Copy or move files from a VM to a bucket

Use this when you want to save output, results, or processed data from a VM back into a bucket.

No Console option for VM-to-bucket transfers

The Google Cloud Console cannot read files from a VM’s filesystem, so this procedure has a single SSH-in-browser tab. Do not attempt to work around this by uploading files from your own computer, which is a violation of the Acceptable Use Policy.

SSH-in-browser

Values you must substitute

Placeholder	Replace with
my-bucket-name	The bucket you are saving files into
source	The file or folder on the VM to upload, e.g. file.txt, or . if you have already navigated into that folder

Copying files

gcloud storage cp source gs://my-bucket-name/

You can also give the full path from anywhere on the VM:

gcloud storage cp /home/john_doe_northwestern/results/file.txt gs://my-bucket-name/

Copying folders

gcloud storage cp -r source gs://my-bucket-name/

The -r flag means recursive. It copies the folder and everything inside it. Without -r, a folder path will not copy.

If a path or bucket name contains spaces, wrap it in quotes, e.g. "gs://my folder/file.txt"

Moving files or folders

gcloud storage mv source gs://my-bucket-name/

Move is permanent

mv deletes the file from the VM once it has been copied to the bucket. If the upload is interrupted, you can lose the file entirely. Use cp and confirm the upload succeeded before deleting anything from the VM.

Expected output

Copying file:///home/john_doe_northwestern/results/file.txt to gs://my-bucket-name/file.txt
  Completed files 1/1 | 240.5kiB/240.5kiB

To verify a transfer, list the destination bucket:

gcloud storage ls -l gs://my-bucket-name/

Expected output:

246784  2026-08-21T14:32:10Z  gs://my-bucket-name/file.txt
TOTAL: 1 objects, 246784 bytes (241.0 KiB)

You can also open the bucket in the Google Cloud Console to confirm the files visually. If the file is missing, see Troubleshoot a failed transfer.

Copy or move files between two buckets

Use this when you need to share data between projects, duplicate a dataset, or reorganize files across buckets. The source and destination buckets can belong to different projects.

Google Cloud Console

Because both buckets live in the cloud, this is the one transfer on this page you can do entirely in the Console.

1. Log in to the Google Cloud Console and navigate to Cloud Storage > Buckets .

2. Open the source bucket, the one containing the files you want to transfer.

3. Select the file or files you want to copy or move.

4. Click Copy or Move in the toolbar.

5. In the destination dialog, select or enter the name of the destination bucket.

6. Confirm the operation.

7. Open the destination bucket and confirm the files appear there.

SSH-in-browser

Run these commands in your SSH-in-browser session or a terminal on the VM desktop. The files move directly between buckets in the cloud and never touch the VM.

Values you must substitute

Placeholder	Replace with
source-bucket-name	The bucket you are copying from
destination-bucket-name	The bucket you are copying to
file.txt	The name of the file you are transferring
folder-name	The name of the folder you are transferring

Confirming both bucket names

List each bucket to confirm the names are correct and that you can reach both:

gcloud storage ls gs://source-bucket-name/
gcloud storage ls gs://destination-bucket-name/

Copying files

gcloud storage cp gs://source-bucket-name/file.txt gs://destination-bucket-name/

Copying folders

gcloud storage cp -r gs://source-bucket-name/folder-name gs://destination-bucket-name/

The -r flag means recursive. It copies the folder and everything inside it. Without -r, a folder path will not copy.

If a path or bucket name contains spaces, wrap it in quotes, e.g. "gs://my folder/file.txt"

Moving files or folders

gcloud storage mv gs://source-bucket-name/file.txt gs://destination-bucket-name/

Move is permanent

mv deletes the file from the source bucket once it has been copied to the destination. If the transfer is interrupted partway through, the file can be removed from the source before it fully arrives. Use cp first if you are unsure, then delete the original once you have confirmed the copy.

Expected output

Copying gs://source-bucket-name/file.txt to gs://destination-bucket-name/file.txt
  Completed files 1/1 | 240.5kiB/240.5kiB

To verify a transfer, list the destination bucket:

gcloud storage ls gs://destination-bucket-name/

To include file sizes and timestamps:

gcloud storage ls -l gs://destination-bucket-name/

Confirm your file appears with the size you expect and a current timestamp. If it does not, see Troubleshoot a failed transfer.

Troubleshoot a failed transfer

Common errors

Error message	What it means	What to do
403 or does not have storage.objects.list access	You do not have permission on this bucket	Contact your Data Engineer to confirm your access, and see Access levels
404 or The specified bucket does not exist	The bucket name is wrong, or you are in the wrong project	Recheck the name using Find your bucket name
The following URLs matched no objects or files	The file or folder path inside the bucket is wrong	List the bucket contents with gcloud storage ls gs://my-bucket-name/ and copy the exact path
No such file or directory	The destination folder on the VM does not exist	Run pwd to see where you are, then create the folder with mkdir -p ~/data
Omitting directory or nothing is copied	You tried to copy a folder without the -r flag	Rerun the command with -r, for example gcloud storage cp -r ...

Files or folders with spaces in their names

If a path contains spaces, the terminal reads each space as a separator between arguments and the command fails or behaves unexpectedly. Wrap the entire path in double quotes:

gcloud storage cp "gs://my-bucket-name/my folder/my file.csv" ~/data/

The same rule applies to paths on the VM:

gcloud storage cp "/home/john_doe_northwestern/my results/my output.csv" gs://my-bucket-name/

And to recursive moves:

gcloud storage mv -r "gs://my-bucket-name/folder with spaces" gs://destination-bucket-name/

---

Creating a bucket

Create buckets only to support compliant workflows. Before you start, review the Projects page, because your project may already have preconfigured buckets that meet your needs.

Console only

Bucket creation is done in the Google Cloud Console so that the required region, storage class, access control, and data protection settings are applied and recorded consistently.

A new bucket has no user access until you assign permissions

This procedure creates the bucket, but no one can use it until you follow Add a user to a bucket. Plan to complete both in the same session.

Google Cloud Console

1. Log in to the Google Cloud Console, confirm the correct project is selected, and go to Cloud Storage > Buckets .

2. Click + Create.

   

3. Name your bucket. The name must:

   • Contain only lowercase letters, numbers, hyphens (-), and underscores (_)

   • Start and end with a letter or number

   • Be globally unique across all of Google Cloud, not just within your project

   Choose a name that makes the bucket’s purpose and project clear, for example gc-sde-project-name-datatype. See Google Cloud bucket naming requirements for the full set of rules.

4. Choose a region. Select us-central1 (Iowa). Keep this default, because SDE data residency requirements restrict where regulated data may be stored, and a bucket in another region falls outside the approved environment.

   

5. Choose a storage class. Keep the default Standard class. Other classes change how quickly data can be retrieved and add retrieval charges, which can conflict with the availability terms in your data agreement. Consult your SDE administrator before changing it.

   

6. Choose access control. The SDE requires Uniform access control, and this cannot be changed. Uniform access keeps all permissions at the bucket level, which is what makes access reviewable during a compliance audit.

   

7. Choose data protection settings. Keep both defaults:

   • Soft delete policy, enabled

   • Retention duration, default

   Soft delete lets you recover individual files deleted by accident within the retention window. Do not disable it.

   

8. Click Create.

   

9. Confirm you land on the Bucket details page and that the bucket name, region, and storage class match what you selected.

Grant access before you announce the bucket

Go directly to Add a user to a bucket and assign permissions to the appropriate users. Only then notify them that the bucket is ready.

---

Deleting a bucket

Delete buckets that are no longer needed to control storage costs and keep the project organized. Before you delete, confirm with your team that no active workflows depend on the bucket.

Deletion is permanent and immediate

Deleting a bucket permanently removes every file, folder, and piece of metadata it contains, and it cannot be undone. Soft delete does not protect you here: it only allows recovery of individual objects deleted from a bucket that still exists. Once the bucket itself is gone, there is nothing left to recover from.

Back up or transfer any data you need to keep before you proceed.

Console only

Bucket deletion is done in the Google Cloud Console so the action is confirmed explicitly and captured in the audit trail.

Google Cloud Console

1. Log in to the Google Cloud Console, confirm the correct project is selected, and go to Cloud Storage > Buckets .

2. Click the bucket name to open it and review its contents. Confirm nothing important remains, or that everything needed has been moved to another bucket using Copy or move files between two buckets.

3. Return to the Buckets list using the breadcrumb navigation.

4. Select the checkbox next to the bucket name, then click Delete in the toolbar.

   

5. In the confirmation dialog, type DELETE in capital letters in the text field and click Delete.

   

6. Confirm the bucket no longer appears in the Buckets list. The bucket and all of its contents are permanently gone.

Notify affected users

Tell anyone who previously had access that the bucket has been deleted, so they can update any scripts or workflows that referenced it. Scripts pointing at a deleted bucket will fail with a 404 error.

---

Manage bucket access

Bucket permissions control who can read, write, and manage data in Cloud Storage. Apply the principle of least privilege: grant users only the level of access they need for their work, and no more.

Review the access levels

Permission	What the user can do	When to use
Storage Object Viewer	Read and download files, list objects, view metadata. Cannot modify or delete anything.	Users who only need to read existing files.
Storage Object Creator	Upload new files only. Cannot read, modify, or delete existing files.	Users who need to deposit files without seeing existing content, for example automated data delivery.
Storage Object User	Upload, read, download, modify, and delete files. Cannot manage bucket permissions.	Users who need to actively read and write files as part of their work.
Storage Object Admin	Full control over objects, including creating, reading, updating, deleting, and managing object-level access. Cannot manage bucket-level permissions.	Users responsible for managing the contents of a bucket.

If you are unsure which level to assign, start with Storage Object Viewer and escalate only if the user’s workflow requires it.

Add a user to a bucket

You can only assign permissions to users who have already been onboarded to the SDE environment. Contact your SDE administrator if the user is not yet onboarded.

Console only

Bucket permissions are managed through the Google Cloud Console in the SDE. Granting IAM roles from the terminal is not permitted, because permission changes must be made through the audited Console workflow.

Google Cloud Console

1. Log in to the Google Cloud Console and confirm the correct project is selected in the project dropdown at the top of the page.

2. Go to Cloud Storage > Buckets and click the name of the bucket you want to manage.

3. Click the Permissions tab.

   

4. Turn off Show inherited roles in table. This filters the list to permissions set directly on this bucket, making it easier to see who you have explicitly granted access to.

   

5. Click Add Principal.

   

6. In the New principals field, enter the user’s Northwestern email address.

   

7. In the Assign roles dropdown, navigate to Cloud Storage and select the appropriate access level.

   

8. Click Save.

9. Confirm the user now appears in the permissions list with the role you assigned. Access takes effect immediately, and the user can reach the bucket from any VM they are authorized to use.

Tell the user their access is ready

After granting access, notify the user which bucket they can now reach and which VMs they can use it from. Users have no way to discover new access on their own.

Remove a user from a bucket

Remove access as soon as it is no longer needed, for example when a team member leaves a project or their role changes.

Console only

As with granting access, permission changes are made through the Google Cloud Console so they are captured in the audit trail.

Google Cloud Console

1. Log in to the Google Cloud Console and confirm the correct project is selected.

2. Go to Cloud Storage > Buckets and click the name of the bucket.

3. Click the Permissions tab.

   

4. Locate the user you want to remove. If you do not see them, turn on Show inherited roles in table, because they may have access through a project-level role rather than a bucket-specific one.

   

5. Click the trash can icon next to their name.

6. In the confirmation dialog, select Remove user@northwestern.edu from all roles on this resource. They may still have access via inherited roles.

   

7. Click Remove.

8. Confirm the user no longer appears in the permissions list. Access is revoked immediately, and they can no longer read or write any files in this bucket.

Inherited access

If the user still has access after removal, they may hold a project-level role that grants access to every bucket in the project. Project-level roles are managed separately and require a Data Engineer or project administrator to change.