AI agents on the servers

---
name: acss-cluster-good-practices
description: "Rules for any AI agent running commands on the ACSS GPU servers (the cluster nicknamed Marcelle: the 4×H100, 2×H100 and L40S machines, usually reached as `4h100`, `2h100`, `l40s`). Use it before launching any job, model server (vLLM, Ollama, TGI…), notebook, download or long-running process on these machines, and at the end of every session. The servers are shared by many researchers, with no scheduler and no quotas, so a forgotten process or a full disk blocks everyone."
---

# ACSS GPU cluster — good practices for AI agents

The ACSS GPU servers are shared by many researchers. There is no job scheduler and there are no quotas: every user, and every agent acting for a user, is responsible for not blocking the machines for others. You act on behalf of the user, so the user remains responsible for what you launch. When in doubt, stop and ask the user.

Full human documentation: the "Sharing the GPU servers", "AI Tools" and "Distributed & Multi-GPU" pages of the ACSS cluster documentation.

## Never

- **Never use `sudo`** or try to change system configuration, system packages, drivers or services. Ask the user to contact the ACSS team instead.
- **Never stop, renice or signal another user's processes.** You may only touch processes owned by the current user. If another user's process looks abandoned, tell the user and suggest they report it on the cluster's Discord channel.
- **Never expose a service to the network.** Model servers, notebooks, dashboards and APIs listen on `127.0.0.1` only, and model servers always require an API key. Never use `--host 0.0.0.0` (or an empty host). The user reaches services through an SSH tunnel.
- **Write only in the user's home directory.** Never put models, datasets, caches or large outputs in `/tmp`, `/var/tmp` or `/scratch`: they are on the small system disk, and if it fills up the whole server stops for everyone. Shared storage outside the home directory is reserved for system backups.
- **Never leave a process running that you started and no longer need**, and never end a session without the end-of-session check below.
- **Never start a heavy job without the user's explicit go-ahead** (see "Heavy jobs").

## Before launching anything on a GPU

1. Check which GPUs are free. Use a one-shot command, not `watch`:

   ```bash
   nvidia-smi --query-gpu=index,memory.used,memory.total,utilization.gpu --format=csv
   ```

   A GPU is free only if its **memory used** is near zero (under ~100 MiB on an H100). **0 % utilisation does not mean free:** an idle model server keeps all its memory.
2. Pick one free GPU explicitly and pin the job to it: `CUDA_VISIBLE_DEVICES=<index> …`. Never rely on the default (GPU 0) and never use all GPUs implicitly.
3. **Use one GPU** unless the model or batch genuinely does not fit. On the 4×H100, a two-GPU job must use an NVLink pair: GPUs `0,1` or `2,3`. The 2×H100 has no NVLink.
4. Check free disk space before downloading anything: `df -h ~`. Estimate the download size first (a model needs ~2 GB per billion parameters in bf16).
5. Prefer the smallest model that does the task. Test on a small sample (e.g. 100 documents) and measure time and quality before running on a full corpus or switching to a bigger model.

## Heavy jobs: ask the user first

A job is heavy when it:

- uses **several GPUs**, or one GPU for **more than a day**;
- runs a **model server** meant to stay up;
- downloads or writes **more than ~100 GB** (models, datasets, checkpoints);
- uses **many CPU cores or much RAM** for hours.

Before starting a heavy job, stop and tell the user: which machine and GPUs, roughly how long, how much disk. Remind them to announce it on the cluster's Discord channel. Start only after they confirm.

## Running processes

- **Run anything long in a named `tmux` session**, never with a bare `&` or `nohup` in a shell that will disappear. Use a recognisable name, for example `tmux new -d -s agent-<task> '<command>'`.
- **Keep track of everything you start:** tmux session names, PIDs, ports, GPUs. You must be able to stop all of it at the end.
- **Python environments:** install packages in a virtual environment in the user's home (`uv venv`, `python -m venv`), never system-wide.
- **Jupyter kernels** keep their GPU memory until the kernel is shut down, even when idle.

## Model servers (vLLM and similar)

```bash
export VLLM_API_KEY=$(openssl rand -hex 16)
echo "$VLLM_API_KEY" > ~/.vllm_key && chmod 600 ~/.vllm_key
tmux new -d -s agent-vllm "CUDA_VISIBLE_DEVICES=<free GPU> vllm serve <model> \
  --host 127.0.0.1 --port 8000 --api-key $VLLM_API_KEY \
  --gpu-memory-utilization 0.5"
```

- Always `--host 127.0.0.1` and `--api-key`. Never print the key in your output.
- vLLM reserves 90 % of each GPU's memory by default: lower `--gpu-memory-utilization` for small models.
- If the port is taken (`ss -ltn | grep 8000`), pick another one. Do not stop whatever is using it.
- **Stopping the main vLLM process does not always free the GPU:** its worker `VLLM::EngineCore` (in capitals) can survive as an orphan and hold the memory for days. `ps aux | grep vllm` does not find it. Always check with the command below.

## Stopping processes

- Stop processes with `kill <PID>` (SIGTERM) and wait up to 30 seconds. Use `kill -9` only if the process is still there: a brutal stop can leave orphaned GPU workers.
- Stop only what you started, or what the user explicitly asks you to stop. If you find other leftover processes of the user, list them and ask before stopping them.

## End of session — mandatory check

Run this before ending the session or handing back to the user:

```bash
# Processes of the current user holding a GPU
for pid in $(nvidia-smi --query-compute-apps=pid --format=csv,noheader); do
  [ "$(stat -c %U /proc/$pid 2>/dev/null)" = "$USER" ] && ps -o pid=,etime=,args= -p "$pid"
done
# vLLM servers and orphaned engines
pgrep -u "$USER" -af 'VLLM::[E]ngineCore|vllm [s]erve'
# tmux sessions
tmux ls 2>/dev/null
```

Then:

1. Stop everything you started that is no longer needed (servers, workers, notebook kernels, tmux sessions).
2. Re-run the check: it must show nothing of yours, unless the user explicitly asked to keep a job running.
3. Tell the user what is still running, on which machine and GPU, and how to stop it.
4. Point out large files you downloaded or created that may no longer be needed (`du -sh ~/.cache/* | sort -rh`, `hf cache ls`), and suggest deleting them. Delete only with the user's agreement.

## Disk hygiene

- Hugging Face models and package caches grow silently in `~/.cache`: a single large model can exceed 100 GB.
- Do not download a model or dataset again if it is already in the cache.
- Keep data, models and checkpoints out of Git repositories.
- The servers are not backed up: remind the user to keep their own copy of important results.