Video Tutorials#
The video tutorials and recorded workshops listed below can help you use Quest effectively.
Using Quest#
Introduction to Quest
An introductory video for users who are new to Quest. Topics include the services of the Research Computing and Data Services team, Quest system architecture, research allocations, parallel computing, file sharing, connecting to the system, and job submissions.
Logging into Quest
This video is a resource for users connecting to Quest for the first time. It provides an overview of different methods to connect to the Quest high-performance computing (HPC) cluster, including techniques using both the terminal on a local computer and applications such as FastX.
Introduction to the Text Editor: Vim
Text editors are important for batch scripts in high-performance computing environments and will be used in many of the following tutorial videos. This introductory video discusses how to use the command line text editor “vim.” Users can choose between Vim and Nano as their text editor on Quest.
Introduction to the Text Editor: Nano
Text editors are important for batch scripts in high-performance computing environments and will be used in many of the following tutorial videos. This introductory video discusses how to use the command line text editor “nano.” Users can choose between Vim and Nano as their text editor on Quest.
Navigating Quest via Shell
This is an important tutorial for users running batch scripts on Quest. It provides an overview of how to use bash (Bourne Again SHell) to navigate the Quest system. You will learn how to create and manage files and directories. The video provides a brief introduction to permissions, environment variables, and the differences between project directories and home directories to help you use Quest more efficiently.
Bash Scripting#
Bash Scripting Practice Part 1: General Considerations
This is the first video in the bash scripting series. It is a resource for users looking to generate scripts for batch computing or workflow pipelines. We recommend working through the full series to understand bash scripts. The video provides free resources for hands-on bash experience, covers best practices for creating bash scripts, and introduces key components of bash scripting on a high-performance computing system.
Bash Scripting Practice Part 2: Variables and Arguments
You will learn how to properly define and use variables and arguments in bash. The video expands on the topic of system variables as well as defining and using input arguments within a bash script.
Bash Scripting Practice Part 3: Loops
You will learn about loops and how to implement them in a bash script. The video analyzes the main structure of a loop and shows different ways of creating them in bash.
Bash Scripting Practice Part 4: Arrays
This video covers the concept of bash arrays and their ideal uses. It reviews the main elements that make up a bash array. You will learn how to create a bash array and reference it correctly within a bash script.
Bash Scripting Practice Part 5: Functions
This is the final video in the Bash Scripting Practice series. You will learn how to write a function in bash. The video discusses the integral parts that make up a function and explains how an alias is used.
Containerization Using Singularity#
Singularity Part 1: Why Containers?
This is the first video in the Singularity series. You will learn what a container is and how to use containers. The video explores the different kinds of problems containers solve and discusses the different programs available for installing containers. The tutorial covers the necessary steps to successfully download a container and use it on Quest. It shows how to install a container with a different operating system than the host system’s through a step-by-step example in the closing section.
Singularity Part 2: Using a Container
This video covers what singularity shell and singularity exec commands are and how they differ. You will learn the basics of using a container, including how to bind directories on the host system to the container.
Singularity Part 3: Build Your Own Container - The Container Definition File
You will learn the structure and recipe for making your own container. The video covers the sections that make up a container recipe: the environment section, the post section, and the run script section, and how they all come together when creating a container. It provides information on how to create containers in the cloud and download them locally to avoid using sudo commands. The video ends with an exercise to build on an Ubuntu container from Docker, install GCC-10, and compile a hello_world.c program.
Singularity Part 4: Advanced Container Usage – Use Modules to Obscure Container Commands; Use MPI in Containers
This is the final video in the Singularity series. It discusses how to use the module system to make containers easier to use. You will learn the benefits of using modules when creating containers compared to using bash functions. The video covers what is needed to run an MPI-enabled program using a container. In the final section, you will follow along with the exercise to create a singularity exec call that has Python 3 installed and uses a locally defined module.
Anaconda Virtual Environments#
Introduction to Anaconda Virtual Environments
Virtual environments are helpful when you need to install packages for coding languages such as R or Python. They provide a way to easily switch between different requirements without breaking an existing workflow. This video provides an overview of how to build virtual environments on Quest from the command-line interface.
Anaconda Virtual Environments - Mamba iPython Kernels and Managing Environments
This builds on the first Anaconda Virtual Environments workshop above. Virtual environments can also be added to kernels to use them on platforms such as Jupyter Notebook. The video shows how to modify an existing virtual environment to be used in a Jupyter Notebook kernel.
GitHub#
GitHub Actions
GitHub is a system that allows you to implement version control on Quest. This can be beneficial when multiple collaborators are working in the same codebase. Version control can also help restore code when lost. The video reviews how to automatically build and publish documentation, publish your code to the package manager of your choice (PyPI, Anaconda, Docker, etc.), and run unit tests.
MATLAB#
Using MATLAB on Quest
This video demonstrates how to effectively use MATLAB on Quest. It discusses various methods for submitting jobs, including directly from the MATLAB interface on Quest or on a local computer. The video shows how to write submission scripts that can submit and run MATLAB jobs on Quest. It explains how parallelization works in MATLAB and how to effectively request computing resources based on how your MATLAB program is parallelized. Finally, it demonstrates how to run a GPU-accelerated MATLAB program on Quest’s GPU resources.
Nextflow#
Nextflow on Quest
Nextflow is a programming framework that enables researchers to build complex, parallelized, and multi-language computational pipelines from existing scripts. This workshop introduces Quest users to Nextflow, including how to write a basic script and develop best practices on Quest. Users will receive the most benefit from this workshop if they are already familiar with shell scripting concepts and how to submit jobs on Quest.