Globus Collections

Globus Collections#

To use Globus to transfer your data, the storage location must have an associated Globus collection.

There are three main types of Globus collections:

  • Personal collections, for files on your computer. Install Globus Connect Personal on your computer to set up a collection.

  • Guest Collections , containing data others have shared with you, or files you have shared with others: Use Guest Collections to access files shared with you by other Globus users.

  • Institutional collections that provide access to Northwestern data storage systems: Use Northwestern managed collections to use Northwestern approved research data storage systems. Managed collection names are:

    • Northwestern Quest is for the Quest High Performance Computing Cluster.

    • Northwestern Quest RDSS to access files from RDSS and FSMResFiles ; note that a Quest account is also required to use this collection.

    • Northwestern OneDrive/SharePoint - High Assurance allows users to access files from OneDrive , SharePoint , and Teams . This collection has additional security features that accommodate the transfer of restricted data.

    • Northwestern Amazon S3 (Standard) allows users to access files from Amazon S3 .

Note

Accessing Research data storage outside Northwestern: Many data providers and supercomputing centers have their own Globus collections. Contact them to see if they use Globus. If Globus isn’t an option, email globus-help@northwestern.edu and we can help you move your data by an alternate route.

Using Collections#

All collections have a similar workflow:

  1. Prepare: Set up access to the underlying system and specific files

  2. Find a collection: Search for the collection in Globus File Manager

  3. Authenticate into the collection: Set up your user credentials to access the collection

  4. Find files: Navigate the file system within a collection to identify files to transfer

These steps are covered below. Once you have a collection and files set up in the Globus web application , you can then transfer files.

Prepare#

Each collection has a specific set of prerequisites for use that vary in complexity. Select the tab for the storage system you want to use to see what you need before you get started.

To use Globus to transfer data to/from your computer:

  1. Install Globus Connect Personal .

  2. Choose which files and directories you want to be accessible to Globus. By default, Globus will allow access to the home directory on your computer. See the following instructions for details on how to change this default.

  3. (Optional) To transfer data between two Globus Connect Personal collections, request to join Northwestern’s Globus subscription . Otherwise you will be restricted to transferring from your computer to collections installed on storage servers.

After installation and configuration, data in the directories you specified during setup will be visible through your collection in the Globus web interface under the Your Collections tab to find your collection

Your collections tab highlighted with a red box

See Transferring Files for details.

To transfer data to or from Quest, you must have an active Quest account. Quest storage is only appropriate for data that will be processed on Quest.

To use the RDSS collection you must:

  1. Have a Quest account: See information about Quest to apply for access.

  2. Have access to an RDSS share or FSMResFiles folder: See Managing access to RDSS Shares to find out how to request access to RDSS or contact fsmhelp@northwestern.edu to request access to FSMResFiles.

  3. Request access to, and log in to, the data transfer node: By default, the data transfer node that connects RDSS/FSMResFiles to Globus does not have access to your data. Follow the steps in Preparing qglobus12 to configure the data transfer node.

Once these steps are complete, you can use move on to authentication.

No setup is necessary to access your OneDrive files and files that are shared with you.

To access the contents of SharePoint document libraries, you have to follow the site in SharePoint by:

Click star to follow a SharePoint site

Once you have access and follow the SharePoint site, you will be able to access the files using Globus.

To use Globus with Amazon S3, you need to:

  1. Create an Amazon S3 bucket if you don’t have one already.

  2. Create an AWS IAM user

  3. Create AWS IAM permissions

  4. Apply these permissions to the user

  5. Download an AWS IAM access key that has permission to read from and write to S3.

Use the following instructions in Preparing an Amazon S3 bucket for Globus transfers.

To access data shared with you through a Guest collection, the collection owner must grant you permissions.

See How To Share Data Using Globus for instructions.

Find a collection#

To find a collection, log in to the Globus web app with your Northwestern NetID (more detailed login instructions) and then search for the name of the collection in the File Manager pane.

Click on the tabs for collection specific instructions

For collections created using Globus Connect Personal, look under the Your Collections tab to find data stored on local computers.

Find Globus Connect Personal Connections by clicking on the "Your Collections" tab.

Search for “Northwestern Quest” or use this direct link to the Northwestern Quest collection . Searching for “Northwestern Quest” may bring up multiple results; the Northwestern Quest collection is a “Subscribed Mapped Collection”.

Search for “Northwestern Quest RDSS” or use this direct link to the Northwestern Quest RDSS collection . If searching, make sure that the collection you choose is a “Subscribed Mapped Collection”.

Search for “Northwestern OneDrive/SharePoint - High Assurance” or use this direct link to the Northwestern OneDrive/SharePoint - High Assurance collection . If searching, make sure that the collection you choose is a “Subscribed Mapped Collection”.

Search for “Northwestern Amazon S3 (Standard)” or use this direct link to the Northwestern Amazon S3 (Standard) collection . If searching, make sure that the collection you choose is a “Subscribed Mapped Collection”.

For data shared with you via Guest collections, look under the Shared with you tab.

Find shared Guest collections by clicking on the "Shared with Me" tab.

Authenticate into the collection#

Now that you’ve found your collection, Globus needs to get your consent to use your credentials and verify your access to the underlying storage system.

Note

These authentication instructions apply to the Northwestern Quest, Northwestern Quest RDSS, and Northwestern OneDrive/SharePoint - High Assurance collections.

To authenticate into the collection:

  1. Once you’ve selected a collection, in the File Manager window, Globus will notify you that it requires your permission to authenticate into the storage system using your NetID and consent to access your data as part of the transfer process. Click Continue.

Authenticate into Globus Quest collection screen

  1. Then, Globus will ask you to select a Northwestern identity to use for authentication. Click netid@northwestern.edu where netid is your Northwestern NetID. If you don’t see this option, make sure that you logged in to the Globus web application using your Northwestern credentials, or have your Northwestern identity linked to your Globus account .

Screen where user selects a Northwestern identity

  1. After selecting Northwestern, you will see the Northwestern NetID login screen. Use your NetID and password to log in. If you have already logged in with your NetID in your browser recently, you might skip this step.

  2. Globus will then prompt you to allow the service access necessary to perform transfer, sharing and task management functions. Click Allow.

Consent to give Globus enough access to perform basic transfer and sharing functions

You should now see the contents of the collection in the File Manager window. The frequency with which you have to re-authenticate varies by collection.

  • Northwestern Quest - Every 10 days

  • Northwestern Quest RDSS - Every 11 days

  • Northwestern OneDrive/SharePoint - High Assurance - every 4 hours

  • Northwestern Amazon S3 (Standard) - does not expire

Find your files#

Once you have authenticated into the collection, you will be able to see your files in File Manager.

Click on tabs below for each collection for details on what you will see in file manager by default, how your files are organized within each type of collection, and tips on how to efficiently navigate the file structure.

The directory structure will match the computer Globus Connect Personal is installed on and will be limited to the directories that you granted access to during setup.

By default, Globus will navigate to the “home” directory for the user that installed the collection.

The directory structure is the same as the Quest file system and you will have access to anything in Globus that you have access to on Quest.

  • By default, Globus navigates to your Quest /home directory.

  • To access projects or scratch directories, you must navigate to the file path (e.g. /scratch/netid or /projects/allocationID). The easiest way is to type the path in the Path field of the File Manager under the Collection. Using the “up one folder” button in the GUI interface of the file list will work, but it will take a long time to load, as the next level up will list all home directories on Quest. Note that while you can see all directories, you can only access directories.

Tip: You can add a symbolic link (symlink) to your project or scratch directory into your home directory on Quest to make it easy to navigate to those folders. From a Quest Login Node, type a command like:

ln -s /projects/allocationID ./my_projects_folder

To create a symlink to /projects/allocationID called my_projects_folder. my_projects_folder will appear when you list files in the directory, and you can navigate to it in file explorer windows and on the command line.

If you cannot access Quest files/folders that you think you should have access to in Globus, email globus-help@northwestern.edu for further assistance.

The file system on the RDSS Globus collection will look different than it does when you mount RDSS:

  • By default, Globus navigates to a folder called /rdss/netid where netid is your Northwestern NetID. This directory is where your shares are mounted.

  • If you only have one path configured, the folder containing your files will be named resfiles or fsmresfiles depending on the server the path you configured included.

  • If you have multiple paths configured for the same server, the folder will have a number appended to the end of the name (eg: resfiles_01).

When you click on the folder containing one of your mounts, the contents of the folder will have the same structure as inside your RDSS share or FSMResFiles folder.

By default, Globus navigates to the root directory (/) of the collection, which contains 3 folders

  • /My Files contains all files stored in OneDrive . The structure will look the same as it does in OneDrive.

  • /Shared Libraries will have a subfolder for all SharePoint sites you follow. That subfolder will further have subfolders for all of the Document Libraries contained in that SharePoint site.

  • /Shared contains specific files/folders shared with you . You will see an alphabetical list of the individual files/folders that have been shared with you.

Globus file manager showing onedrive connector root directory

You can navigate to the files you would like to transfer from here by double clicking on the folder.

By default, Globus navigates to the root directory (/) of the collection. You will see a folder for each S3 bucket that the AWS IAM identity associated with the key/secret you used to authenticate has list permissions for.

To see the contents of the bucket, double click on the folder name. You will only be able to see the contents of the buckets you have read access to. The contents of the bucket will be the same as what you see in the S3 management console.

By default, Globus will navigate to the root directory set by the Guest collection owner. You will only be able to see files and folders that they have granted you access to.

Bookmarks#

We recommend using bookmarks for commonly used folders and collections to streamline your workflow.

Create a bookmark#

To create a bookmark,

  1. Navigate to the collection and path you commonly use

  2. Select the bookmark icon to the right of the path you would like to bookmark.

The icon will appear yellow if the path is bookmarked.

Bookmark a Globus collection

Access your bookmarks#

To access your bookmarks

  1. Click on the Collection search bar in File Manager window.

  2. Click the Bookmarks tab below the search bar

  3. Click on the bookmark to open it in File Manager

You can also edit or delete bookmarks from this window.

Note

Bookmarks in the Globus web application are not allowed on High Assurance collections like Northwestern OneDrive/SharePoint - High Assurance because the metadata for these bookmarks is stored on Globus’s AWS servers that are not approved for sensitive data.

We recommend bookmarking the direct links to High Assurance collections in your web browser for quick access.

Now that you have access to your files, see the transfer section to move your files between storage systems.