Using Ollama on Quest

Using Ollama on Quest#

Ollama is a framework for building and running large language models (LLMs) on local machines. Running an LLM on Quest with Ollama requires a virtual environment with the Ollama Python library, a location to store downloaded models, and a batch job that starts the Ollama server before running your script.

Warning

Ollama provides access to open source models that anyone can contribute to. Only use models from trusted or verified sources, as some models contain malicious code. The Ollama infrastructure can also create security vulnerabilities if you run it on your own computer.

What You Will Learn#

After completing this tutorial, you will be able to:

Run an Ollama model within a Slurm batch job on Quest
Store and reuse models across jobs using shared storage
Configure a Python script to communicate with an Ollama server
Generate text from a single prompt, from a set of prompts, and in a structured format
Submit and manage batch jobs that use LLMs through Ollama

Note

While you can use Ollama to run an interactive chatbot on Quest, the chatbot will not be accessible outside of your Quest account. For this reason, we do not cover Ollama’s chatbot functionalities in this tutorial.

Prerequisites#

This tutorial assumes familiarity with Python, Quest, Slurm job submission, virtual environments, and basic large language model (LLM) usage.

Before starting, ensure you have:

An active Quest compute resource with a Slurm account ID (e.g., pXXXX)
Access to GPU partitions (if using GPU models); General Access Quest compute resources have access to GPU partitions

Replace placeholders in this tutorial as follows:

<netID>: your NetID
<account_id>: your Quest Slurm account ID (e.g., pXXXX)

Create a Virtual Environment Including Ollama#

To use Ollama models in batch jobs on Quest, create a mamba or conda virtual environment with:

The Ollama Python library
Any additional packages required by your script

Running a model with Ollama involves two separate components:

Ollama server: Loads models and generates responses. The server software is provided through a Quest module and runs as its own process during your job.
Ollama Python library: Sends requests from your script to the server. The library is installed in your virtual environment.

During job execution, the library sends requests to the server’s API over a local network port.

To create a virtual environment:

Connect to the Quest login nodes and load the mamba module:

$ module load mamba/24.3.0

Create and activate a virtual environment with mamba. The --prefix argument sets a custom environment location instead of the default (/home/<net_id>/.conda/envs/). Update the command with the path to your /projects directory. This example creates an environment called ollama-env.

$ mamba create --prefix=/projects/<account_id>/envs/ollama-env -c conda-forge python=3.12

$ mamba activate /projects/<account_id>/envs/ollama-env

Install the Ollama Python library and Pydantic. On conda-forge, the Python library is named ollama-python; the conda-forge package named ollama is the Ollama server software, which Quest provides through a module instead. Pydantic validates the structured output shown later in this tutorial.

$ mamba install -c conda-forge ollama-python pydantic

Install any additional Python packages required by your workflow. Replace or extend this list as needed for your script; pandas and matplotlib are provided in the command below as an example.

$ mamba install -c conda-forge pandas matplotlib

Store Ollama Models#

When you pull a model, Ollama downloads and stores it on disk. The underlying model data can be several gigabytes in size. By default, Ollama stores models in your home directory (/home/<netid>/). Because home directories on Quest have a storage limit of 80 GB, your home directory is not suitable for larger models or workflows that require multiple models.

To store models in a different location, set the OLLAMA_MODELS environment variable to a directory in /scratch or /projects. Scratch (/scratch) is best for very large models or temporary workflows as it has a 5 TB quota, but files are automatically deleted after 30 days of inactivity. Projects (/projects) is best for long-term storage and workflows that are shared with others with access to your Quest resources.

To store models in scratch space, open a terminal, log in to Quest, and run the following commands:

mkdir -p /scratch/<netid>/ollama-models
echo "export OLLAMA_MODELS=/scratch/<netid>/ollama-models" >> $HOME/.bashrc

After updating your ~/.bashrc, log out and back in again before submitting jobs to ensure the variable is applied in Slurm jobs. Alternatively, you can run:

source ~/.bashrc

You can also use your /projects/<account> directory instead of /scratch.

Example Script#

The example script basic_usage.py demonstrates three patterns that cover most Ollama workflows:

Generate a response: Send a single prompt to a model and read the text it returns.
Loop over prompts: Build prompts from a template and send each one in turn.
Request structured output: Constrain the model’s reply to a defined set of fields and validate it.

The full files are available in the GitHub example scripts repository . There are two files:

basic_usage.py: The Python script that sends prompts to the model.
submit_ollama_job.sh: The Slurm job submission script, which starts the Ollama server and then runs the Python script.

Sections of basic_usage.py appear below to discuss specific components.

Run this script with a batch job or in an interactive session, not on a Quest login node, as running resource-intensive workloads on login nodes can negatively impact other users and may result in your processes being terminated.

Connect to the Ollama Server#

The script imports the Ollama library, selects a model, and connects to the server that the batch job starts.

File: basic_usage.py

from ollama import Client
from pydantic import BaseModel, ValidationError
import os

llm_name = "llama3.1"

client = Client(
   host="http://localhost:" + os.environ.get("OLLAMA_PORT")
)

# Download model if not available already
client.pull(llm_name)

The Ollama server listens on a port chosen by the submission script, which exports it as the environment variable OLLAMA_PORT. Reading that variable rather than hard-coding a port number keeps the script working across jobs, because the port changes from job to job.

The call to client.pull() ensures the model is available. If the model is not already in your OLLAMA_MODELS directory, it downloads at this point during job execution. Model downloads take significant time and disk space (several gigabytes), so allow for the download time when setting your job time limit.

Every request in the rest of the script goes through this client object, so the server address is defined in one place.

Generate a Response#

A single call to client.generate() requires only a model and a prompt.

File: basic_usage.py

gen_answer = client.generate(
    model=llm_name,
    prompt='Tell me the story of Koko the gorilla in two limericks'
)

print(gen_answer)

print(gen_answer.response)

The returned object includes the generated text along with metadata such as timing and token counts. The response attribute holds the text alone.

Loop Over Multiple Prompts#

Most research workflows send many prompts rather than one. Build a template with a placeholder, then substitute a different value for each item you want to process, such as different subjects or individual responses to survey questions.

File: basic_usage.py

base_prompt = "Write a 10-line absurdist play about <<SUBJECT>>."

subjects = [
    'pretentious and scholarly platypuses',
    'the greatest marbles tournament in the world',
    'a heated discussion about tea: should milk be poured first or last?'
]

gen_answers = []
for subj in subjects:
    answer = client.generate(model=llm_name, prompt=base_prompt.replace('<<SUBJECT>>', subj))
    gen_answers.append(answer)

for answer in gen_answers:
    print(answer.response)

Each call is independent, so the model retains no memory of earlier prompts in the loop. To include earlier exchanges as context, put them in the prompt itself.

The example holds all results in a list and prints them at the end. For long-running jobs, write each response to a file as it is generated so that partial results survive a job that runs out of time.

Request Structured Output#

Passing a schema in the format argument constrains the reply to a fixed set of fields, which is useful for extracting data from text. Define the fields and their data types as a Pydantic model, then pass the model’s JSON schema.

Describe the fields in the prompt as well as in the schema. The schema constrains the shape of the reply; the prompt tells the model what each field means.

File: basic_usage.py

class PoemEvaluation(BaseModel):
    rating: float
    haiku: int
    rationale: str

gen_answer = client.generate(
    model=llm_name,
    prompt='''
        Please evaluate the poem below, you must reply in JSON format. The fields of your reply are:
        - rating (a rating from 0-10 of how good the poem is)
        - haiku (a binary 0/1, indicating if it is a haiku or not)
        - rationale (an explanation of why you have provided such rating)

        Poem:
        Scattered light's gentle dance
        Dancing waves of azure hue
        Blue heaven's wondrous peace
        ''',
    format=PoemEvaluation.model_json_schema()
)

print(gen_answer.response)

Validate the reply before using it. model_validate_json() confirms that the values match the declared types and returns an object whose fields are available as attributes.

File: basic_usage.py

try:
    valid_response = PoemEvaluation.model_validate_json(gen_answer.response)
    print(valid_response)
except ValidationError as e:
    print(f"Not a valid output format: {e}")

print(valid_response.rating)

Validation checks structure and data types, not accuracy of the content. Review generated values before using them in an analysis.

Run the Script on Quest#

To run basic_usage.py as a batch job on Quest, use the submission script submit_ollama_job.sh. This script handles starting the Ollama server, setting up networking, and running your Python script on a compute node.

To submit the job, use the command

sbatch submit_ollama_job.sh

Slurm Job Configuration#

Begin by specifying the appropriate SBATCH options for your job. In most cases, you only need to update your account ID and email address. If you change the model, the number of prompts, or other Ollama-specific parameters, adjust the job duration or GPU request as needed.

Some Ollama models require GPUs for efficient execution. For smaller models, CPU-only jobs may be sufficient.

File: submit_ollama_job.sh

#!/bin/bash
#SBATCH --account=<account_id>  
#SBATCH --partition=gengpu  
#SBATCH --nodes=1 
#SBATCH --ntasks-per-node=4 
#SBATCH --job-name=Ollama-batch-job 
#SBATCH --time=00:30:00 
#SBATCH --mem=40GB 
#SBATCH --gres=gpu:h100:1 
#SBATCH --output=output-%j.out 
#SBATCH --error=error-%j.err 
#SBATCH --mail-type=ALL 
#SBATCH --mail-user=<YOUR_EMAIL@northwestern.edu> 

For more information on job setup or SBATCH flags, see the Slurm job scheduler page, or specifically the Slurm configuration settings.

The example requests one H100 GPU card. To request any available GPU type instead, use:

#SBATCH --gres=gpu:1

For more details, see the GPUs on Quest page.

Find and Export a Free Port#

Ollama runs as a local server on the compute node and must listen on an available port. The submission script defines a set of helper functions, including find_port, which locates a port that is not already in use. These functions require no changes.

File: submit_ollama_job.sh

OLLAMA_PORT=$(find_port localhost 7000 11000)
export OLLAMA_PORT
echo $OLLAMA_PORT

Exporting OLLAMA_PORT makes the port number available to the Python script, which reads it to build the server address.

Load Required Modules and Set Environment Variables#

On Quest, loaded modules affect available libraries and compilers. Always purge previously loaded modules to avoid conflicts.

File: submit_ollama_job.sh

module purge
module load ollama/0.11.4
module load gcc/12.3.0-gcc
module load mamba/24.3.0

When running your own code, you can update the Ollama version if needed. To see all available Ollama module versions available on Quest, run:

$ module spider ollama

Next, export the host address so the Ollama server can listen for client requests. The second export makes the variable available inside the container used by the Ollama module.

File: submit_ollama_job.sh

export OLLAMA_HOST=0.0.0.0:${OLLAMA_PORT} 
export SINGULARITYENV_OLLAMA_HOST=0.0.0.0:${OLLAMA_PORT}

Start the Ollama Server#

With the module loaded and networking configured, start the Ollama server. All server activity is written to a log file for later inspection.

File: submit_ollama_job.sh

# Start Ollama service
ollama serve &> serve_ollama_${SLURM_JOBID}.log &
sleep 30

After starting the server, the sleep command builds in time to allow the server to initialize before the Python client connects. If your script fails to connect to the server, check the file serve_ollama_<jobid>.log for startup errors. Increase the sleep duration if the server is not ready in time.

The log file serve_ollama_<jobid>.log will contain details about server startup, model loading, and runtime behavior.

Activate the Virtual Environment#

Activate the virtual environment created earlier. These commands initialize the mamba environment inside the Slurm job. All three lines of code are required for proper activation within batch jobs. Update the environment path if you created it in a different location, and change the script name if needed.

File: submit_ollama_job.sh

# Activate virtual environment
eval "$('/hpc/software/mamba/24.3.0/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
source "/hpc/software/mamba/24.3.0/etc/profile.d/mamba.sh"
mamba activate /projects/<account_id>/envs/ollama-env 

Run the Script#

Finally, include a line to run the Python script.

File: submit_ollama_job.sh

# Run the Python script
python -u basic_usage.py

The Python script connects to the running Ollama server, pulls the specified model if needed, and generates the responses.

basic_usage.py prints its results rather than writing files, so the generated text appears in output-<jobid>.out after the job completes. The -u flag makes Python write output as it is produced, so you can follow progress in the log file while the job runs. When adapting the script for your own work, write results of the model calls to disk instead of just to the log files.

Troubleshooting#

Common issues:

Model fails to download: Check available storage in the OLLAMA_MODELS location and the time limit for the batch job
Connection errors: Verify OLLAMA_PORT and increase the sleep time after the server is started
Module errors: Ensure module purge and required modules are loaded
Missing packages: Confirm that ollama and pydantic are installed in the activated virtual environment
Validation errors on structured output: Check the ValidationError message, then clarify the field descriptions in the prompt

Check Slurm output (output-<jobid>.out) and error (error-<jobid>.err) files for details.

Using Ollama on Quest

Contents

Using Ollama on Quest#

What You Will Learn#

Prerequisites#

Create a Virtual Environment Including Ollama#

Store Ollama Models#

Example Script#

Connect to the Ollama Server#

Generate a Response#

Loop Over Multiple Prompts#

Request Structured Output#

Run the Script on Quest#

Slurm Job Configuration#

Find and Export a Free Port#

Load Required Modules and Set Environment Variables#

Start the Ollama Server#

Activate the Virtual Environment#

Run the Script#

Troubleshooting#