Running Jupyter Notebooks on Rosalind
Running Jupyter Notebooks on Rosalind
This page explains the resources Rosalind offers, how to choose between a whole GPU and a shared GPU, and how to run a Jupyter Lab session on the cluster.
Contents
- Partitions and Hardware
- Whole GPU or a Share of a GPU?
- GPU Usage Policy
- How Jupyter Runs on Rosalind
- Step 1: SSH Into Rosalind
- Step 2: Create the SBATCH Script
- Step 3: Create SSH Tunnel
- Running Jupyter on a GPU Node
Partitions and Hardware
Rosalind provides multiple Slurm partitions designed for different workloads. Please choose the appropriate partition based on your job size and runtime.
Available Partitions
| Partition | Max Time | Hardware | Intended Use |
|---|---|---|---|
debug |
2 hours | CPU nodes | Quick testing, debugging |
cpus |
12 hours | CPU nodes | Standard CPU workloads (default) |
batch |
7 days | CPU nodes | Long CPU runs (lowest priority) |
debug-gpu |
2 hours | GPU nodes | Short GPU testing |
gpus |
12 hours | GPU nodes | Standard GPU workloads |
CPU Nodes
Each CPU node provides:
- 384 CPU cores
- ~770 GB RAM
CPU nodes are shared resources. Jobs request specific cores and memory:
#SBATCH --cpus-per-task=32
#SBATCH --mem=64G
Please avoid requesting excessive cores or memory unless your workload requires it.
GPU Nodes
Each GPU node provides:
- 2 NVIDIA RTX PRO 6000 Blackwell GPUs (~96 GB GPU memory each)
- 384 CPU cores
- ~1 TB CPU RAM
To request a GPU, choose either a whole GPU (--gres=gpu:blackwell:1) or a share of a GPU (--gres=shard:N, a good fit for notebooks). See Whole GPU or a Share of a GPU? below.
Quick checks
You can quickly check if you can run on CPU or GPU nodes by running:
sbatch -p debug --wrap="hostname; sleep 120" --cpus-per-task=32 --mem=10G
For GPU nodes:
sbatch -p debug-gpu --gres=gpu:blackwell:1 --wrap="nvidia-smi -L; sleep 120"
Whole GPU or a Share of a GPU?
Rosalind has 4 GPUs (2 per GPU node), each with about 96 GB of GPU memory. There are two ways to request GPU time:
| You ask for | Meaning | Use it for |
|---|---|---|
--gres=gpu:blackwell:1 |
One whole GPU, exclusively yours | Large training runs that need the full card’s memory or speed |
--gres=shard:N |
N quarter-GPU shares, shared with other jobs |
Notebooks, development, inference, small to medium training |
How shards work. Each GPU is split into 4 shards, so a node has 8 shards. One shard is roughly a quarter of a GPU, about 24 GB of GPU memory. Up to 4 single-shard jobs can run on the same GPU at once, and up to 16 across the cluster.
| Request | About how much GPU memory |
|---|---|
--gres=shard:1 |
~24 GB |
--gres=shard:2 |
~48 GB |
--gres=shard:3 |
~72 GB |
--gres=shard:4 |
a full GPU’s worth (but it is still shared; use gpu:blackwell:1 if you need exclusive use) |
Shared vs full at a glance
Shared GPU (--gres=shard:N) |
Whole GPU (--gres=gpu:blackwell:1) |
|
|---|---|---|
| GPU memory | about 24 GB per shard (shard:2 is about 48 GB). Not enforced. |
The whole card, about 96 GB |
| Speed | Shared with up to 3 other jobs on the same GPU | Full speed, no neighbours |
| Neighbours | Yes. A neighbour using too much memory can make your job crash | None |
| Waiting | Can start whenever a share is free | Needs a GPU with no shard jobs on it, so it may wait longer |
| Best for | Notebooks, development, inference, small to medium training | Large training runs, benchmarks, anything that needs more than about 48 GB |
A job with shards sees exactly one GPU, as device 0 (CUDA_VISIBLE_DEVICES=0). Your job runs on one physical GPU, and its other shards are used by other people’s jobs.
What shards do not do. A shard is a scheduling quota, not a hardware limit:
- GPU memory is not enforced. If your job allocates more than its share, the other jobs on that card can run out of memory and crash with a CUDA out-of-memory error, or you can. Keep your usage under about 24 GB per shard you requested, and request more shards if you need more memory.
- Compute is shared. If other jobs are busy on the same GPU, yours will run slower than on a whole GPU.
- A whole-GPU request (
gpu:blackwell:1) can only start on a GPU that has no shard jobs on it, so it may wait while shard jobs are running.
Setting
CUDA_VISIBLE_DEVICESyourself is not needed and can make your job use the wrong GPU. Leave it as Slurm sets it.
Which one should I use for Jupyter?
- If your notebook needs a GPU, use a shared one. A notebook sits idle much of the time, so it does not need a whole card to itself. (If your code runs fine on CPU, use a CPU node instead; see the GPU Usage Policy.) Start with
shard:1(about 24 GB), orshard:2(about 48 GB) if your model needs more. - Use a whole GPU only if you need it: your job needs more than about 48 GB of GPU memory, or you are timing or benchmarking and need no neighbours.
- Whichever you choose, keep your GPU memory use within what you asked for, and shut your notebook down when you are done so the GPU is free for others.
Ready-to-paste headers for both are in Running Jupyter on a GPU Node. For GPU usage policy and more examples, see the HPC Server User Guide.
GPU Usage Policy (Important)
GPU nodes are limited and high-demand resources.
Please follow these guidelines:
- Do NOT submit to GPU partitions unless your code explicitly requires CUDA and requires very high memory
- If your workload runs on CPU, use:
--partition=cpus - If you do need a GPU for a notebook, prefer a shared GPU (
--gres=shard:N) over a whole one. See Whole GPU or a Share of a GPU?.
How Jupyter Runs on Rosalind
This section explains how to launch a Jupyter Lab session on the Rosalind cluster using Slurm and connect to it from your local machine.
The workflow is:
- SSH into
rosalind-login - Submit a Slurm job
- Create an SSH tunnel from your laptop
- Open Jupyter in your browser
Step 1: SSH Into Rosalind
From your local machine:
ssh <username>@rosalind-login
Replace <username> with your Vanderbilt username.
Step 2: Create the SBATCH Script
On rosalind-login, create a file (protip: name it run_jupyter.sbatch and keep in batch_scripts folder) and copy the following:
script you can copy and adapt to run
Jupyter Lab on the Rosalind cluster. The snippet is embedded below with
comments explaining what to edit. Copy the block into a file on the login
node (for example run_jupyter.sbatch), update the SBATCH headers, then
submit with sbatch run_jupyter.sbatch /path/to/project.
Paste the following:
#!/usr/bin/env bash
#SBATCH --job-name=jupyter_cpu
#SBATCH --partition=cpus
#SBATCH --time=04:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=32G
#SBATCH --output=logs/%x_%j.out
#SBATCH --mail-type=END,FAIL
set -euo pipefail
# -----------------------------
# Usage: sbatch jupyter_cpu.sbatch /path/to/workdir
# -----------------------------
TARGET_DIR="${1:-$PWD}"
mkdir -p logs
# Pick a random high port (avoid common collisions)
PORT="$(shuf -i 10000-20000 -n 1)"
NODE="$(hostname -s)"
USER="$(whoami)"
# -----------------------------
# Generate token ourselves so it ALWAYS appears in the SLURM log
# -----------------------------
export PYTHONUNBUFFERED=1
TOKEN="$(python - <<'PY'
import secrets
print(secrets.token_hex(24))
PY
)"
echo ""
echo "============================================================"
echo "Jupyter will run on compute node: ${NODE}"
echo "Port: ${PORT}"
echo ""
echo "From your LOCAL machine, run (in a new terminal):"
echo " ssh -N -L ${PORT}:127.0.0.1:${PORT} -J ${USER}@rosalind-login ${USER}@${NODE}"
echo ""
echo "Then open your browser at:"
echo " http://127.0.0.1:${PORT}/lab?token=${TOKEN}"
echo "Token:"
echo " ${TOKEN}"
echo "============================================================"
echo ""
CONDA="${HOME}/miniforge3/bin/conda"
# Start Jupyter; keep port fixed; send logs to the SLURM output
cd "${TARGET_DIR}"
echo -e "\n\nRunning JupyterLab...\n\n"
# Bind to localhost for safety on shared nodes; access via SSH tunnel above.
"${CONDA}" run -n spatial \
jupyter lab --no-browser --port="${PORT}" --ip=127.0.0.1 \
--ServerApp.token="${TOKEN}" \
--ServerApp.password="" \
--ServerApp.port_retries=0 \
--ServerApp.log_level=INFO \
2>&1 | tee -a "logs/jupyter_${USER}_${PORT}.log"
Make sure to update the SBATCH headers according to your needs (partition, time, resources). Then submit the job:
sbatch run_jupyter.sbatch /path/to/your/project
Check the job status with squeue -u <username> and wait for it to start running.
Step 3: Create SSH Tunnel
Open a new terminal on your local machine and run the command printed in the job output:
ssh -N -L <PORT>:<COMPUTE_NODE>:<PORT> <username>@rosalind-login
Replace <PORT>, <COMPUTE_NODE>, and <username> with the values from your job output.
Then open your browser:
http://localhost:<PORT>
Running Jupyter on a GPU Node
Use the same script as in Step 2 and the same steps to connect. The only change is the #SBATCH header at the top: request either a shared GPU or a whole GPU, as described in Whole GPU or a Share of a GPU?. The Jupyter Lab session then runs on the GPU node, and you can use it to run GPU-accelerated code.
Shared GPU (recommended for notebooks)
#!/usr/bin/env bash
#SBATCH --job-name=jupyter_gpu_shared
#SBATCH --partition=gpus # debug-gpu (2h), gpus (12h)
#SBATCH --gres=shard:2 # a share of a GPU: shard:1 ~ 24 GB, shard:2 ~ 48 GB
#SBATCH --time=04:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --output=logs/%x_%j.out
Whole GPU
#!/usr/bin/env bash
#SBATCH --job-name=jupyter_gpu
#SBATCH --partition=debug-gpu # debug-gpu (2h), gpus (12h)
#SBATCH --gres=gpu:blackwell:1 # 1 whole Blackwell GPU, exclusively yours
#SBATCH --time=02:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=32G
#SBATCH --output=logs/%x_%j.out
Everything below the header (the part that picks a port, creates the token and starts Jupyter Lab) is the same as in Step 2. Submit and connect the same way:
sbatch run_jupyter.sbatch /path/to/your/project