High Performance Computing at HZDR

High Performance Computing (HPC) is one of the core tasks of the Department of IT Infrastructure. In order to provide the users the opportunity to run large-scale computational calculations in the field of simulation and data processing at HZDR, the department of IT Infrastructure operates one central Linux cluster.

The Linux Clusters

RoSI

On Sept. 22nd 2025 the HPC cluster RoSI (Rossendorf Supercomputing Infrastructure) has officially started operation as part of an introductory workshop (videos of Day 1 and Day 2).

The cluster is based on the Linux derivative Ubuntu and the resource management is done with the help of the SLURM scheduler. For the network interconnect InfiniBand is used in the generations HDR (200 Gbit/s) and NDR (up to 800 Gbit/s). Also the connection to the parallel file system /bigdata is done with that. Access to the cluster is possible via the login nodes rosi4 and rosi5 as well as via the web access rosi.hzdr.de which is based on Open OnDemand.

CPU and GPU compute nodes can be accessed via the following partitions:

Partition	Nodes	CPU type	CPU cores per node	RAM per node	GPU	GPUs per node	user group
cpu-skylake	csk[001-056]	Intel Xeon Gold 6148	40	384 GB
cpu-rome	cro[001-028]	AMD Epyc 7702	128	512 GB
cpu-milan-reac	cmi[001-006]	AMD Epyc 7713	128	512 GB			FWOR
cpu-milan-reac	cmi[007-012]	AMD Epyc 7713	128	1024 GB			FWOR
cpu-milan	cmi[013-032]	AMD Epyc 7713	128	1024 GB
cpu-genoa	cge[001-002,027-054]	AMD Epyc 9654	192	1536 GB
cpu-genoa-casus	cge[003-026,055-070]	AMD Epyc 9654	192	1536 GB			FWU
cpu-turin	ctu[001-024]	AMD Epyc 9965	384	1536 GB
cpu-turin	ctu[025-037]	AMD Epyc 9965	384	2304 GB
gpu-v100	gv[001-032]	Intel Xeon Gold 6136	24	384 GB	Nvidia V100	4
gpu-a100	ga[006-009]	AMD Epyc 7302	32	1024 GB	Nvidia A100	8
gpu-a100	ga[010-015]	AMD Epyc 7763	128	4096 GB	Nvidia A100	4
gpu-h100	gh[001-002]	AMD Epyc 9654	192	2304 GB	Nvidia H100	8
gpu-h100-casus	gh003	AMD Epyc 9654	192	2304 GB	Nvidia H100	8	FWU
gpu-b200	gb[001-007]	AMD Epyc 9755	256	2304 GB	Nvidia B200	8	HAICORE
gpu-b200-casus	gb008	AMD Epyc 9755	256	2304 GB	Nvidia B200	8	FWU

RoSI Documentation

The guide below covers everything you need to log in, run jobs, and manage data on the rosi cluster. Read top to bottom if you're new, or jump to a topic using the links.

1. Getting access (SSH)

You connect to rosi over SSH from your own computer's terminal. Once connected, you land on a login node where you set things up and submit jobs.

You don't have to use the terminal for everything though. The web portal at rosi.hzdr.de (based on Open OnDemand) gives you a graphical interface to the cluster from your browser. It includes Jupyter Notebook, Code Server, a file browser, a web terminal, and other interactive applications — handy for editing files, running notebooks, or launching an interactive session without touching SSH.

Note: the web portal is only reachable from inside the HZDR intranet or when connected to the HZDR VPN.

Item	Value
Login hosts	rosi4 and rosi5
Jump hosts	uts.hzdr.de and mup.hzdr.de
Operating system	Ubuntu
Job scheduler	Slurm

Direct login — when you're in the office on the HZDR intranet:

ssh <USERNAME>@rosi4
# or
ssh <USERNAME>@rosi5

Via jump host — when you're at home / in the office but not connected to the HZDR intranet (home office, remote work, travelling):

The cluster login nodes (rosi4, rosi5) are only reachable from inside the HZDR network. When you're working from home or otherwise outside the intranet, you must connect through a jump host (bastion host) first. The jump host sits on the public internet and forwards your connection into the cluster. You use the same username for both the jump host and the cluster. The two available jump hosts are uts.hzdr.de and mup.hzdr.de.

# One-liner: connect through the jump host straight to rosi4
ssh -J <USERNAME>@uts.hzdr.de <USERNAME>@rosi4

# Or via mup:
ssh -J <USERNAME>@mup.hzdr.de <USERNAME>@rosi4

The -J flag tells SSH to use the first address as a "jump" (proxy) and connect to the second address through it. You'll be prompted for your password/key once for the jump host and once for the cluster (unless you've set up keys for both).

To make this easier, add a ProxyJump entry to your ~/.ssh/config (see below) so you can just type ssh rosi even from outside.

Set up an SSH key (one time only)

By default, logging in requires a password every single time. An SSH key pair solves this: you generate a matching pair of keys on your laptop, give the cluster your public key, and keep the private key safe on your machine. After that, the cluster recognizes your private key automatically and lets you in without a password. You only do this once.

Step 1 — Create the key pair on your local machine. Open your terminal and run:

ssh-keygen -t ed25519 -C "your_email@example.com"

You'll be asked where to save the file (just press Enter to accept the default) and whether to set a passphrase (you can leave it empty for convenience, or type one for extra security). This command creates two files in the ~/.ssh/ folder:

• ~/.ssh/id_ed25519 — your private key. This stays on your computer. Never send this to anyone or upload it anywhere.
• ~/.ssh/id_ed25519.pub — your public key. This is the one you'll give to the cluster. It's fine to share.

If your operating system is quite old and doesn't recognize ed25519, use the older RSA format instead (the result is the same, just a longer key):

ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

Step 2 — Copy your public key to the cluster (and the jump hosts). This tells the machine "this person is allowed to log in with their private key." The easiest way is one command per host:

# To the cluster (do this once; rosi5 shares the same account)
ssh-copy-id -i ~/.ssh/id_ed25519.pub <USERNAME>@rosi4

# To the jump hosts (only needed if you connect from outside the HZDR network)
ssh-copy-id -i ~/.ssh/id_ed25519.pub <USERNAME>@uts.hzdr.de
ssh-copy-id -i ~/.ssh/id_ed25519.pub <USERNAME>@mup.hzdr.de

Each command asks for your password one last time, then appends your public key to a file called authorized_keys on that host. After copying to rosi4, ssh rosi4 works without a password. After copying to a jump host, the first hop of ssh -J / ProxyJump also becomes passwordless — you'll only be prompted for the cluster itself if you haven't copied there yet.

Make login easy with an SSH config

Add this to ~/.ssh/config on your local machine so you can just type ssh rosi. If you're often outside the HZDR network, add a ProxyJump line so it automatically routes through the jump host:

# For direct access (on HZDR network)
Host rosi
    HostName rosi4
    User <USERNAME>
    IdentityFile ~/.ssh/id_ed25519

# For remote access (through jump host)
Host rosi-remote
    HostName rosi4
    User <USERNAME>
    IdentityFile ~/.ssh/id_ed25519
    ProxyJump <USERNAME>@uts.hzdr.de

Host rosi-alt
    HostName rosi5
    User <USERNAME>
    IdentityFile ~/.ssh/id_ed25519

• HostName — the real server address
• User — your cluster username (same for jump host and cluster)
• IdentityFile — path to your private key
• ProxyJump — routes your connection through the jump host automatically. Now ssh rosi-remote handles the two-hop connection for you.

Quick check after you log in:

hostname   # shows which login host you reached (rosi4 or rosi5)
pwd        # shows your current directory
ls         # lists your files

2. Running jobs (Slurm)

You never run heavy work directly on the login node — it would slow everything down for everyone. Instead, you submit jobs to Slurm, the cluster's scheduler. Slurm puts your job in a queue and starts it on a compute node as soon as enough CPUs/memory/GPUs are free. Think of it like ordering at a restaurant: you place your order (submit), wait in line (pending), then get your food (results).

Here are the three commands you'll use most often:

Command	What it does
squeue	Shows your jobs that are currently waiting or running. No extra flags needed — it automatically shows only your own jobs.
sbatch run.sh	Submits a job script for execution. This is how you "start" a job.
sacct -j <JOB_ID>	Looks up the final record of a finished job: did it succeed? How long did it take? How much memory did it use?

# Check your current jobs (only yours are shown)
squeue

# Submit a job
sbatch run.sh

# Look up a finished job by its ID
sacct -j <JOB_ID> --format=JobID,State,Elapsed,ExitCode

For all available options: squeue, sbatch, sacct.

Note: On rosi you usually do not need to pass -A/--account or --qos flags. Only add them if your project setup or an admin tells you to.

Choose a partition

A partition is a group of similar machines. Pick one based on whether you need CPUs or GPUs.

Type	Partitions
CPU	cpu-genoa, cpu-skylake, cpu-milan, cpu-rome, cpu-turin, short (quick tests), low (low priority)
GPU	gpu-a100, gpu-h100, gpu-v100

Rules of thumb: use cpu-* for CPU-only work, gpu-* when you need GPUs, short for quick smoke tests, and low only when low priority is fine.

# CPU job
sbatch --partition=cpu-genoa run.sh

# GPU job
sbatch --partition=gpu-a100 --gpus=1 run_gpu.sh

# Quick test
sbatch --partition=short --time=00:10:00 run.sh

# Inspect partitions
sinfo -s
scontrol show partition cpu-genoa

For all options: sinfo, scontrol.

Write a job script

A job script is a plain text file (usually ending in .sh) that has two parts:

1. Directives (lines starting with #SBATCH) — these tell Slurm what resources to give your job: which partition, how much time, how many CPUs, how much memory, and where to save output. Slurm reads these before the job starts.
2. Your actual commands — the shell commands that run on the compute node once Slurm allocates resources. This is where you load software modules and launch your program.

Here's a minimal example. Save it as run.sh in your home directory:

#!/bin/bash
#SBATCH --job-name=hello-rosi
#SBATCH --partition=cpu-genoa
#SBATCH --time=00:05:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --output=slurm-%j.out

set -euo pipefail

module purge
module load genoa
module load GCC/14.2.0

echo "Host: $(hostname)"
echo "Job ID: ${SLURM_JOB_ID}"
echo "CPUs: ${SLURM_CPUS_PER_TASK}"
sleep 30

Then submit it with sbatch run.sh. The #SBATCH lines are the important ones:

Directive	Meaning
--partition	Which pool of machines to use
--time	Maximum runtime (HH:MM:SS)
--cpus-per-task	Threads given to each task
--mem	Memory request (e.g. 8G)
--output	Where stdout/stderr go (%j = job ID)

Before submitting, you can sanity-check the script:

bash -n run.sh          # check for syntax errors
sbatch --test-only run.sh   # preview without actually running

Submit & manage jobs

Once you've submitted a job, you can check on it, pause it, or cancel it. Here's what each command does:

Command	What it does
squeue	List your waiting/running jobs (only yours are shown)
squeue -j <JOB_ID>	Show details for one specific job
scontrol hold <JOB_ID>	Pause a job so it doesn't start yet
scontrol release <JOB_ID>	Un-pause a held job
scancel <JOB_ID>	Cancel (kill) a job
scontrol show job <JOB_ID>	Show why a job is stuck waiting (e.g. not enough memory)

# Submit
sbatch run.sh

# Watch your queue
squeue
squeue -j <JOB_ID>

# Hold / release / cancel
scontrol hold <JOB_ID>
scontrol release <JOB_ID>
scancel <JOB_ID>

# Why is a job stuck waiting?
scontrol show job <JOB_ID>

For all options: squeue, scontrol, scancel.

If a job is stuck in PENDING state, run scontrol show job and look at the Reason field. Common reasons: Resources (not enough free CPUs/memory yet) or QOSJobLimit (you've hit a limit). Usually you just wait — the job will start when resources free up.

Interactive runs (for debugging)

Sometimes you don't want to write a full script, submit it, and wait for output files. You just want to try a few commands on a compute node and see the results immediately — like having a live terminal on a powerful machine. This is perfect for:

• Debugging: "Does this Python import work? Does this command give the error I expect?"
• Quick tests: "How long does this small calculation take?"
• Exploring: "What software is available? What does this dataset look like?"

The workflow has two steps: salloc reserves a compute node for you (like booking a meeting room), and srun launches your commands on that reserved node. Once you're done, type exit to release the node back to the cluster.

# Request resources, then launch commands with srun
salloc --partition=cpu-genoa --time=00:30:00 --cpus-per-task=8 --mem=16G
srun hostname
srun --ntasks=1 --cpus-per-task=8 python -c "import os; print(os.cpu_count())"

# Or get an interactive shell in one line
srun --partition=cpu-genoa --time=00:30:00 --cpus-per-task=8 --mem=16G --pty bash

# GPU interactive example
salloc --partition=gpu-a100 --gpus=1 --cpus-per-task=8 --mem=32G --time=01:00:00
srun --pty bash
nvidia-smi

# When done
exit

For all options: salloc, srun.

Job arrays (many tasks, same program)

Imagine you have 100 input files and need to run the same analysis on each one. Submitting 100 separate jobs would be messy — 100 job IDs to track, 100 log files, 100 entries in squeue. A job array solves this: you submit one job that Slurm automatically splits into 100 identical tasks, each handling one input file. You get one job ID, one set of logs, and can cancel individual tasks if needed.

The key addition to your script is the --array directive. In the example below, --array=0-9%3 means "create tasks numbered 0 through 9, running at most 3 at the same time." Save this as run_array.sh:

#!/bin/bash
#SBATCH --job-name=array-demo
#SBATCH --partition=cpu-milan
#SBATCH --time=00:10:00
#SBATCH --cpus-per-task=2
#SBATCH --mem=4G
#SBATCH --array=0-9%3
#SBATCH --output=slurm-%A_%a.out

set -euo pipefail

INPUT_FILE="inputs.txt"
ITEM=$(sed -n "$((SLURM_ARRAY_TASK_ID + 1))p" "$INPUT_FILE")

echo "Array job: ${SLURM_ARRAY_JOB_ID}"
echo "Task id : ${SLURM_ARRAY_TASK_ID}"
echo "Input   : ${ITEM}"
sleep 20

Key variables explained: SLURM_ARRAY_TASK_ID is the number of "which copy" is running (0, 1, 2…), SLURM_ARRAY_JOB_ID is the parent job ID, and %A/%a are placeholders that Slurm replaces with the array job/task IDs in filenames so each task gets its own log file.

To manage an array job:

# See all your array tasks
squeue

# Check status of all tasks in the array
sacct -j <ARRAY_JOB_ID> --format=JobID,JobName%20,State,Elapsed

# Cancel just one specific task (e.g. task 7)
scancel <ARRAY_JOB_ID>_7

For all options: squeue, sacct, scancel.

GPU jobs

If your workload uses a GPU (deep learning training, molecular dynamics, image processing, etc.), there are two extra things you must do compared to a normal CPU job:

1. Choose a GPU partition (e.g. gpu-a100, gpu-h100) instead of a cpu-* partition.
2. Request GPUs explicitly with --gpus=<number> so Slurm knows to allocate GPU hardware to your job.

Everything else (time, memory, CPUs per task) works the same way. Inside your script, you can check which GPU you got using nvidia-smi. Here's a minimal GPU job script — save it as run_gpu.sh:

#!/bin/bash
#SBATCH --job-name=gpu-smoke
#SBATCH --partition=gpu-h100
#SBATCH --time=00:20:00
#SBATCH --gpus=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --output=slurm-%j.out

set -euo pipefail

module purge
module load genoa

nvidia-smi
python -c "print('GPU job running on rosi')"

Common GPU flags: --gpus=<count>, --gpus-per-task=<count>, plus the usual --cpus-per-task and --mem. Verify your allocation with:

squeue -j <JOB_ID>
sacct -j <JOB_ID> --format=JobID,Partition,AllocTRES,State,Elapsed

For all options: sbatch, squeue, sacct.

Monitor & account

While your job runs and after it finishes, you can check on it using three tools. Think of them as different views of your job's life:

• squeue — "Is my job still waiting or running right now?"
• sstat — "How much CPU/memory is my job using right now?" (live view while running)
• sacct — "How did my job turn out?" (final report after it finishes)

# What's running or waiting right now?
squeue
squeue -j <JOB_ID>

# Live resource usage while a job is running
sstat -j <JOB_ID>.batch --format=JobID,MaxRSS,AveCPU,AvePages,AveDiskRead,AveDiskWrite

# Final report after the job finishes
sacct -j <JOB_ID> --format=JobID,JobName%20,Partition,State,ExitCode,Elapsed,AllocTRES

# All of today's jobs
sacct --starttime today --format=JobID,State,Elapsed,MaxRSS

For all options: squeue, sstat, sacct.

How to read the results:

• State — the outcome: COMPLETED (success), FAILED (error), TIMEOUT (ran too long), CANCELLED (you or admin stopped it)
• ExitCode — 0:0 means your program finished without errors. Anything else means something went wrong.
• Elapsed — how long the job actually ran
• MaxRSS — peak memory used. If this is close to your --mem request, consider requesting more.

Read your logs

When a job fails or behaves unexpectedly, the log files are almost always where you'll find the answer. Every job produces at least one log file containing everything your program printed (standard output) and any error messages (standard error). The filename is based on the job ID that Slurm assigns when you submit.

You can control where logs go by adding two directives to your script's #SBATCH lines. It's good practice to set both explicitly so you always know where to look:

#SBATCH --output=slurm-%j.out
#SBATCH --error=slurm-%j.err
# For arrays:
#SBATCH --output=slurm-%A_%a.out
#SBATCH --error=slurm-%A_%a.err

Inspect and triage:

ls -1 slurm-*.out slurm-*.err
less slurm-<JOB_ID>.out
grep -n "ERROR\|Traceback\|OOM\|Killed" slurm-<JOB_ID>.out slurm-<JOB_ID>.err

Connect the log to the scheduler state to diagnose:

sacct -j <JOB_ID> --format=JobID,State,ExitCode,Elapsed,MaxRSS

• State=FAILED + non-zero ExitCode → application or script error
• State=TIMEOUT → raise --time or speed up the run
• State=OUT_OF_MEMORY or high MaxRSS → raise the memory request

3. Working with data

Where you put files matters. Use the location that fits how long the data must live and who needs it.

Where to store files

Path	Best for	Avoid storing
/home/<USERNAME>	Scripts, configs, source code, small inputs	Large datasets, heavy job output
/bigdata/<TEAM_OR_PROJECT>	Active project data, large inputs/outputs, checkpoints (your team or project directory)	Tiny personal files, login configs
/bigdata/share	Team-readable shared results	Anything private

Recommended workflow: keep scripts in /home, put big run inputs/outputs in /bigdata, and publish shared artifacts under /bigdata/share.

Check usage & clean up

Watch your disk usage regularly so jobs don't fail because a filesystem fills up.

# How much space am I using?
du -sh /home/<USERNAME>
du -sh /bigdata/<TEAM_OR_PROJECT>

# What's taking space inside your project dir?
du -sh /bigdata/<TEAM_OR_PROJECT>/* 2>/dev/null

# How much free space is left on each filesystem?
df -h /home /bigdata /bigdata/share

Good hygiene: keep only active files in fast areas, archive or delete old outputs after each stage, and clean up temporary intermediates when runs finish. A simple cleanup pattern:

mkdir -p /bigdata/<TEAM_OR_PROJECT>/archive
mv /bigdata/<TEAM_OR_PROJECT>/old-results-* /bigdata/<TEAM_OR_PROJECT>/archive/

Move files in & out

You'll often need to upload input data to the cluster before a run, or download results back to your laptop after a run. scp (secure copy) does this over the same encrypted SSH connection you use to log in. The syntax is always: scp [source] [destination], where each side is either a local path or a user@host:path remote path.

A few tips:

• Add -r to copy entire directories (not just single files).
• Add -p to preserve file timestamps and permissions (useful for reproducibility).
• For very large transfers (many GB), consider splitting into chunks or using rsync if available, since it can resume interrupted transfers.

# Local → rosi
scp /local/path/input.dat <USERNAME>@rosi4:/bigdata/<TEAM_OR_PROJECT>/inputs/

# rosi → local
scp <USERNAME>@rosi4:/bigdata/<TEAM_OR_PROJECT>/results/output.csv /local/path/

# Copy whole directories (add -r)
scp -r /local/path/project/ <USERNAME>@rosi4:/bigdata/<TEAM_OR_PROJECT>/
scp -r <USERNAME>@rosi5:/bigdata/<TEAM_OR_PROJECT>/results/ /local/path/results/

# Keep timestamps/modes (optional, add -p)
scp -p /local/path/input.dat <USERNAME>@rosi4:/bigdata/<TEAM_OR_PROJECT>/inputs/

Share with your team

When multiple people need to read or write the same project data, you use Unix group permissions. The idea: assign the directory to a named group (e.g. your research group), then set permissions so members of that group can access it while everyone else cannot. This avoids the mess of copying files around or making everything world-readable.

Here's the standard three-step setup for a new shared project folder:

# Create a shared project directory
mkdir -p /bigdata/share/project-a

# Give it to your collaboration group
chgrp -R project-a /bigdata/share/project-a

# Owner+group read/write, nobody else
chmod -R 2770 /bigdata/share/project-a

Why 2770? The 770 gives full access to owner and group and none to others; the leading 2 sets the setgid bit so new files automatically inherit the directory's group.

Verify and set sensible defaults:

ls -ld /bigdata/share/project-a
umask 007   # new files stay private to owner + group by default

If ACLs are enabled, you can grant extra access without changing the group:

setfacl -m u:alice:rwx /bigdata/share/project-a
setfacl -m g:project-b:rx /bigdata/share/project-a

4. Good habits

These practices make your jobs restartable, observable, and reproducible — meaning you can rerun them later and get the same results.

Software environments (Lmod modules)

The cluster has hundreds of software packages installed (compilers, Python, R, MATLAB, specialized simulation tools, etc.). You don't install these yourself — they're managed by a system called Lmod. Think of modules like app stores for HPC: you load the ones you need into your current session, and they become available. When you log out, everything resets.

On rosi, modules are organized by architecture first. You load the architecture module (e.g. genoa for AMD Genoa CPUs) to unlock the tree of software built for that hardware, then load specific compilers and applications from within that tree. The typical sequence is:

module load genoa
module avail

module load GCC/14.2.0
module avail

# Find exact module names / versions
module spider <software>
module spider <software>/<version>

Cluster details: Ubuntu OS, Slurm scheduler, Lmod modules, example stack genoa → GCC/14.2.0. Run module avail again after each load to see what becomes available.

Reproducibility

Six months from now (or a collaborator next week), will you be able to rerun your analysis and get the same numbers? Reproducibility means keeping enough information about every run that anyone can recreate it exactly. This isn't just academic — it saves you hours of "wait, what version of Python was I using?" and makes your results trustworthy.

The four things to capture for every job:

1. Load an explicit module stack before submitting.
2. Save environment metadata at job start.
3. Keep inputs, scripts, and outputs in a stable layout.
4. Record the job ID, final state, elapsed time, and exit code.

Capture provenance early inside your script:

#!/bin/bash
#SBATCH --job-name=myrun
#SBATCH --output=logs/%x-%j.out

mkdir -p metadata
module list 2> metadata/modules-$SLURM_JOB_ID.txt
env | sort > metadata/env-$SLURM_JOB_ID.txt

Minimum things to keep: the submission script + arguments, the module list + environment, input identifiers/checksums, and the Slurm job ID + end state + exit code.

Slurm signalling, preemption & requeue (advanced)

Preemption means Slurm can kill a running job to free up resources for a higher-priority job. This is most likely to happen when using the low partition, which is designed for low-priority, preemptible workloads. If you want to use the low-priority partition, you must submit with both --partition=low (or -p low) and --account=low (or -A low):

# Submit a preemptible job on the low partition
sbatch -p low -A low --time=02:00:00 run.sh
# or equivalently:
sbatch --partition=low --account=low --time=02:00:00 run.sh

When your job gets preempted, by default Slurm marks it as PREEMPTED (failed) and you have to resubmit it manually. That's wasteful if your job had been running for hours.

Requeue solves this: you tell Slurm "if my job gets preempted, don't mark it failed — just put it back in the queue so it can run again later." Your job will restart from the beginning on a new node once resources are available.

Signalling is how your running program communicates with Slurm — for example, catching the preemption warning so it can save its state before being killed.

Here's how to set it up:

Step 1 — Enable requeue on your job:

# At submission:
sbatch --requeue run.sh

# Or as a directive inside your script:
#SBATCH --requeue

With --requeue, if Slurm preempts your job it goes back to PENDING instead of PREEMPTED. You can watch it come back to life in squeue.

Step 2 — Know what happens when your job is preempted:

When Slurm selects your job for preemption, it immediately sends it SIGTERM (the standard "please terminate" signal). After a short grace period set by the administrator (GraceTime, in seconds), Slurm follows up with SIGKILL if the job hasn't exited yet. Your program should catch SIGTERM to save any in-progress state (write a checkpoint, flush buffers) before terminating cleanly.

The amount of warning time is controlled on the cluster side via the partition's GraceTime setting — you don't set it per-job. So the practical rule is: always handle SIGTERM in long-running jobs so they can checkpoint gracefully.

Example: handling SIGTERM in Python:

import signal, sys, os

def handle_term(signum, frame):
    print(f"SIGTERM received (likely preemption or time limit). Saving state for job {os.environ.get('SLURM_JOB_ID', 'unknown')}...")
    # Save your checkpoint / intermediate results here
    # e.g. model.save('checkpoint.pt')
    print("State saved. Exiting cleanly.")
    sys.exit(0)

signal.signal(signal.SIGTERM, handle_term)

# ... your main computation loop ...
for epoch in range(1000):
    train()
    if epoch % 10 == 0:
        save_checkpoint(epoch)

A note on --signal (different use case):

You may also see --signal mentioned in other HPC guides. It does not relate to preemption — it tells Slurm to send a signal shortly before your job hits its --time limit, so your program can wrap up gracefully instead of being killed abruptly. For example, --signal=USR1@60 sends SIGUSR1 60 seconds before the time limit expires (the default lead time is 60 seconds if you omit the number).

# Warn my job 60s before its time limit so it can finish cleanly
sbatch --requeue --time=02:00:00 --signal=USR1@60 run.sh

# Or in your script:
#SBATCH --signal=USR1@60

Catch SIGUSR1 in your code to trigger a final checkpoint right before the time limit. This complements the SIGTERM handling above: SIGTERM covers preemption/failure, SIGUSR1 (via --signal) covers the scheduled time limit.

Inspecting jobs and nodes with scontrol show:

When something looks off — a job stuck in PENDING, a node marked DRAIN, or you want to confirm what a job was actually granted — scontrol show gives you the full detail that squeue/ sinfo summarize away.

# Full details of one job: state, reason, nodes, time limit, requeue flag, exit code
scontrol show job <JOB_ID>

# Just check whether requeue is enabled for a job
scontrol show job <JOB_ID> | grep Requeue

# Details of a specific compute node: state, flags, memory, GRES (GPUs), drain reason
scontrol show node <NODE_NAME>

# All nodes at once (useful to spot which are IDLE vs DRAINING/DOWN)
scontrol show node

In the show job output, pay attention to JobState and Reason (why it's pending, e.g. Resources or QOSJobLimit), Requeue (0/1), NodeList, and ExitCode. In show node, the State field (e.g. IDLE, ALLOCATED, DRAINED) plus any Reason= text tells you why a node isn't taking jobs.

You can also cross-check final outcomes in accounting:

# See how a job ended and why (including preemption)
sacct -j <JOB_ID> --format=JobID,State,ExitCode,Reason

In sacct, a job that was preempted without requeue shows State=PREEMPTED. A job that was requeued goes back to PENDING and runs again (its restart count increases); use -D/--duplicate to see every attempt.

For all options: scontrol, sbatch (--requeue, --signal), Preemption.

Get notified

Instead of constantly checking squeue, you can ask Slurm to email you when a job finishes or fails. Add --mail-type and --mail-user to your sbatch command:

# Email me when the job ends OR fails
sbatch --mail-type=END,FAIL --mail-user=<EMAIL> run.sh

You can still check status manually at any time:

# What's running right now?
squeue

# How did a specific job turn out?
sacct -j <JOB_ID> --format=JobID,State,Elapsed,ExitCode

• Use an actively monitored email for critical jobs.
• Prefer END,FAIL over noisy event sets unless you're debugging.
• Treat Slurm status (sacct) as authoritative — notifications supplement, not replace, it.

Quick reference

The handful of commands you'll reach for most often:

I want to…	Command
Log in	ssh <USERNAME>@rosi4
See my queued jobs	squeue
Submit a job	sbatch run.sh
Cancel a job	scancel <JOB_ID>
Check a finished job	sacct -j <JOB_ID> --format=JobID,State,Elapsed,ExitCode
See free space	df -h /home /bigdata
Copy a file up	scp file <USERNAME>@rosi4:/bigdata/<TEAM_OR_PROJECT>/
Copy a file down	scp <USERNAME>@rosi4:/bigdata/<TEAM_OR_PROJECT>/file .
Find software	module spider <software>

Don't run heavy jobs on the login node — always submit through Slurm. And keep large data in /bigdata, not /home.

Data Storage

Data can be stored on all HZDR fileservers and is available on all login nodes of the HPC clusters. Files which should be accessed or created by compute jobs on the cluster have to be stored in the cluster home directory of the user or in a project directory on /bigdata. Project directories are created at request.