Sharing the GPU servers

The ACSS GPU servers are shared by many researchers. A GPU held by a forgotten job, or a disk filled by an old cache, is unavailable for everyone else. These few rules keep the servers usable for all.

Important

Done with a GPU? Free it. A process that holds GPU memory blocks that GPU for everyone, even at 0 % utilisation. Check for leftovers at the end of every session — see Stop the processes you no longer use.

1. Check before you start#

Look at the GPUs before launching anything:

nvidia-smi

Read the Memory-Usage column, not only the utilisation. A free H100 NVL uses almost nothing (under 100 MiB); several GB used means someone is working on that GPU.

Note

0 % utilisation does not mean free. A model server waiting for requests keeps all its memory while showing 0 %. Pick another GPU.

Then pin your job to the free GPU you chose:

CUDA_VISIBLE_DEVICES=1 python train.py

• Use one GPU unless your model or batch does not fit — see How much VRAM do I need?

Announce heavy jobs#

Before launching a heavy job, post a message in the cluster's Discord channel. A job is heavy when it:

• uses several GPUs, or one GPU for more than a day;
• runs a model server meant to stay up (vLLM…);
• downloads or writes a lot of data (large models, datasets, checkpoints: more than ~100 GB);
• uses many CPU cores or much RAM for hours.

Say which machine and GPUs you will use, roughly how long, and how much disk space. Post again when you are done. Others can then plan their work, and nobody stops your job by mistake.

2. Stop the processes you no longer use#

Forgotten processes are the first reason why GPUs are unavailable: a finished experiment whose script still waits, a notebook kernel left open, a model server nobody uses, or a worker that survived after its parent was stopped. At the end of every session, run:

# 1. Which of my processes hold a GPU?
for pid in $(nvidia-smi --query-compute-apps=pid --format=csv,noheader); do
  [ "$(stat -c %U /proc/$pid 2>/dev/null)" = "$USER" ] && ps -o pid=,etime=,args= -p "$pid"
done

# 2. Stop the ones you no longer need (PID = first column above)
kill <PID>

# 3. vLLM: also stop the engines that may have survived their server
pkill -u "$USER" -f 'VLLM::[E]ngineCore'

The first command must then print nothing. nvidia-smi must no longer show your memory.

• kill asks the process to stop cleanly. Use kill -9 <PID> only if it is still there after 30 seconds: a brutal stop can leave orphaned workers behind.
• ps aux | grep vllm is not enough to find vLLM leftovers: the worker is named VLLM::EngineCore, in capitals.
• You can only stop your own processes. If someone else's process seems abandoned, say so in the cluster's Discord channel or tell Bruno.

3. Run long jobs in tmux — and close them#

A job started in a plain SSH session stops when the connection drops. Run long jobs in tmux, so that you can find them again and stop them properly:

tmux new -s myjob            # start a named session, then launch your job
# detach with Ctrl+B then D — the job keeps running
tmux ls                      # list your sessions
tmux attach -t myjob         # come back to it
tmux kill-session -t myjob   # close it when the job is over

When your work is done, close the session and check that nothing of yours still holds a GPU.

Notebooks#

A Jupyter kernel keeps its GPU memory until the kernel stops, even when no cell runs and the browser tab is closed. Shut down the kernel (Kernel → Shut Down Kernel) when you no longer need the GPU.

4. Model servers (vLLM)#

An inference server keeps its GPUs for as long as it runs, even when nobody sends requests. Forgotten servers are the first reason why GPUs are unavailable.

• Limit the memory it reserves: by default vLLM takes 90 % of each GPU. Use --gpu-memory-utilization for small models.
• Listen locally, with a key: --host 127.0.0.1 --api-key "$VLLM_API_KEY", then reach it through an SSH tunnel. A server listening on 0.0.0.0 without a key can be used by anyone on the network.
• Stop it as soon as you are done. Stopping the main process does not always stop its GPU worker: an orphaned VLLM::EngineCore process can keep the memory for days. Always check:

pgrep -u "$USER" -af 'VLLM::[E]ngineCore|vllm [s]erve'   # must print nothing

ps aux | grep vllm is not enough: the worker is named VLLM::EngineCore, in capitals. Full commands are in Good practices with vLLM.

5. Keep the disks clean#

Your home directory is on a disk shared by all users. Models and package caches grow silently: a single large model takes more than 100 GB.

du -sh ~/.cache/* 2>/dev/null | sort -rh   # what your caches weigh
hf cache ls                                # Hugging Face models you downloaded
hf cache rm model/org/modelname            # delete one you no longer use
uv cache clean                             # uv package cache
pip cache purge                            # pip package cache

Warning

Never put models, datasets or caches in /tmp, /var/tmp or /scratch. They are on the small system disk: when it is full, the whole server stops for everyone — sessions, running jobs and logs.

• Delete what you no longer use, especially downloaded models and intermediate results.
• Keep your own copy of important code, data and results: the servers are not backed up.
• Keep data out of Git: a repository should not contain data that can be downloaded or rebuilt — see the Project Guidelines.

6. AI coding agents#

If you let an AI coding agent run commands on the servers, you remain responsible for what it launches. Check the models it downloads and the servers it starts, stop them at the end of the session, and avoid modes that skip permission prompts.

7. In short#