AI Tools

Overview of the AI tools available on the GPU servers and how to use them.

1. Overview#

None of these tools is installed system-wide: install what you need in your own environment (see Managing your Python environment).

Warning

Ollama is not supported on the servers. For inference, use vLLM or Hugging Face Transformers.

Hugging Face account and gated models#

Some models used in the examples (Llama, Gemma…) are gated: you must accept their licence on the model page on huggingface.co, then log in on the server with a personal access token:

hf auth login          # paste a token created at huggingface.co/settings/tokens

• Create a read-only token: it is enough to download models.
• The token is stored in ~/.cache/huggingface/token. Never put it in your code, a notebook or a Git repository.
• hf auth logout removes it, for example at the end of your project.

2. vLLM — high-performance inference#

vLLM is optimised for high-throughput inference. It uses PagedAttention for efficient GPU memory management and exposes an OpenAI-compatible API.

Install it in your environment:

uv pip install vllm

Good practices with vLLM#

The servers are shared and a vLLM server keeps its GPUs for as long as it runs — even at 0 % utilisation, even if nobody sends requests. Forgotten or orphaned vLLM processes are the most common reason why GPUs are unavailable.

• Check GPU availability first and pick a free GPU explicitly with CUDA_VISIBLE_DEVICES (nvidia-smi, see Quick Start).
• Use one GPU unless the model does not fit; on the 4×H100, use an NVLink pair for --tensor-parallel-size 2.
• Limit memory: by default vLLM reserves 90 % of each GPU's memory. Lower it with --gpu-memory-utilization if your model is small.
• Listen on 127.0.0.1 only and set an API key — a server on 0.0.0.0 without a key can be used by anyone on the Dauphine network.
• Run it in tmux, so that you can find it and stop it properly later.
• Stop it as soon as you are done, then check that no VLLM::EngineCore process is left (see below).

Launch a vLLM server#

tmux new -s vllm                       # run the server inside tmux
export VLLM_API_KEY=$(openssl rand -hex 16)
echo "$VLLM_API_KEY" > ~/.vllm_key && chmod 600 ~/.vllm_key

# single GPU (here GPU 2 — choose a free one)
CUDA_VISIBLE_DEVICES=2 vllm serve mistralai/Mistral-7B-Instruct-v0.3 \
  --host 127.0.0.1 --port 8000 \
  --api-key "$VLLM_API_KEY" \
  --gpu-memory-utilization 0.5

# two GPUs (tensor parallelism) — on the 4×H100 use an NVLink pair: 0,1 or 2,3
CUDA_VISIBLE_DEVICES=0,1 vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --host 127.0.0.1 --port 8000 \
  --api-key "$VLLM_API_KEY" \
  --tensor-parallel-size 2

Detach with Ctrl+B then D. If port 8000 is already taken (ss -ltn | grep 8000), choose another one, e.g. 8001.

Call the API#

From the server itself:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $(cat ~/.vllm_key)" \
  -d '{"model": "mistralai/Mistral-7B-Instruct-v0.3",
       "messages": [{"role": "user", "content": "Hello"}]}'

From your laptop, open an SSH tunnel and use http://localhost:8000 locally:

ssh -N -L 8000:127.0.0.1:8000 4h100

Use from Python#

from pathlib import Path
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key=Path("~/.vllm_key").expanduser().read_text().strip()
)
response = client.chat.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.3",
    messages=[{"role": "user", "content": "Hello"}]
)
print(response.choices[0].message.content)

Stop the server when done#

tmux attach -t vllm          # then Ctrl+C to stop the server
tmux kill-session -t vllm    # close the session

Stopping the main process does not always stop vLLM's GPU worker: an orphaned VLLM::EngineCore process can keep the GPU memory for days. Always check afterwards:

pgrep -u "$USER" -af 'VLLM::[E]ngineCore|vllm [s]erve'   # must print nothing
nvidia-smi                                          # your memory must be released

If something is left, stop it:

pkill -u "$USER" -f 'VLLM::[E]ngineCore'
pkill -u "$USER" -f 'vllm [s]erve'

3. How much VRAM do I need?#

Rule of thumb#

Common models#

Weights only, rounded. Count 20–30 % more for activations and, for a model server, the KV cache (the memory that holds the conversations in progress).

Note

Mixture-of-experts (MoE) models such as Qwen3-30B-A3B or Qwen3-235B-A22B must hold all their parameters in memory, but only a small part of them works for each token (3 B and 22 B here). They need the memory of a large model and run at the speed of a small one.

Recommended GPU setup by model#

Tip

H100 cards (compute capability 9.0) support FP8 natively — half the memory of bf16 and roughly double the throughput, with no significant quality loss for inference — and FlashAttention-3. See the NVLink rule before using several GPUs on the 4×H100.

dtype — choosing the right precision#

import torch
from transformers import AutoModelForCausalLM

# bf16 — preferred on H100 (more stable than fp16)
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.3",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# fp16 — standard, good balance
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.3",
    torch_dtype=torch.float16,
    device_map="auto"
)

Quantization — load large models with less VRAM#

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

# int8
bnb_config = BitsAndBytesConfig(load_in_8bit=True)

# int4
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.3",
    quantization_config=bnb_config,
    device_map="auto"
)

device_map="auto" spreads the model over every GPU it can see — on a shared server, all four GPUs of the 4×H100 if you do not restrict it. Always choose your GPUs first: CUDA_VISIBLE_DEVICES=2 python script.py.

4. Fine-tuning#

Full fine-tuning vs PEFT#

LoRA fine-tuning with PEFT#

import torch
from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model, TaskType

model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.3",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"]
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 6,815,744 || all params: 7,254,839,296 || trainable%: 0.0939

QLoRA — fine-tune large models with int4#

from transformers import BitsAndBytesConfig
from peft import prepare_model_for_kbit_training

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-70B",
    quantization_config=bnb_config,
    device_map="auto"
)
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, lora_config)

QLoRA brings the fine-tuning of a 70B model within reach of a single H100 (94 GB): the 4-bit weights take about 35–40 GB, the rest goes to activations and adapters. Full fine-tuning of the same model would need far more than the 376 GB of the 4×H100.

Unsloth — faster fine-tuning#

Unsloth is a free open-source library that speeds up LoRA and QLoRA fine-tuning and reduces its memory use compared with standard HuggingFace + PEFT, with an almost identical API. Gains depend on the model; the project reports up to 2× faster training with much less VRAM. It is best suited to single-GPU fine-tuning.

Install:

pip install unsloth

Usage:

from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="mistralai/Mistral-7B-Instruct-v0.3",
    max_seq_length=2048,
    dtype=torch.bfloat16,
    load_in_4bit=True
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"]
)

5. Other useful tools#

Flash Attention 2 — faster attention#

Speeds up training and inference on H100s by rewriting the attention computation to be more memory-efficient.

Tip

Try PyTorch's built-in attention first. Recent versions of Transformers use attn_implementation="sdpa" by default, which already relies on fast fused kernels and needs no extra install. Install flash-attn only if you measure a real gain.

Install:

pip install flash-attn --no-build-isolation

If no prebuilt wheel matches your versions of Python, PyTorch and CUDA, pip compiles it from source: this needs nvcc (on the 2×H100: export PATH=/usr/local/cuda-13.0/bin:$PATH) and can take a long time. Limit the parallel jobs with MAX_JOBS=8 so that the compilation does not take all the CPUs of the shared server.

Usage:

model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.3",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2"
)

Accelerate — simple multi-GPU#

HuggingFace Accelerate lets you run your existing training script on multiple GPUs with minimal changes to your code.

Install:

pip install accelerate

Usage:

accelerate config
accelerate launch train.py

Whisper — speech-to-text#

OpenAI Whisper is a speech-to-text model that supports 99 languages and works well on audio recordings, interviews, and podcasts — particularly useful for qualitative research in social sciences.

Install:

pip install openai-whisper

Whisper calls the ffmpeg program to read audio files. It is installed on the 4×H100, not on the 2×H100. Check with:

ffmpeg -version

Without ffmpeg, use faster-whisper instead (pip install faster-whisper): it decodes audio by itself, runs faster and uses less VRAM, with the same models.

Basic usage:

import whisper

model = whisper.load_model("large-v3")
result = model.transcribe("audio.mp3")
print(result["text"])

Available model sizes:

large-v3 is recommended for research — use it on an H100 for fast transcription.

Transcribe with language hint (faster):

result = model.transcribe("interview.mp3", language="fr")

Get word-level timestamps:

result = model.transcribe("audio.mp3", word_timestamps=True)
for segment in result["segments"]:
    print(segment["start"], segment["text"])

Label Studio — data annotation#

Label Studio is a general-purpose annotation tool for text, image, audio, and video.

Install:

pip install label-studio

Start the server:

label-studio start

Since you are on a remote server, you cannot open http://localhost:8080 directly in your browser. You need to forward the port via SSH — run this on your laptop:

ssh -L 8080:localhost:8080 4h100

Then open http://localhost:8080 in your browser.

ActiveTigger — active learning for NLP#

ActiveTigger is an active learning annotation platform, designed for text classification in social science research. It integrates active learning to reduce annotation effort: the model suggests the most informative samples to annotate, reducing the total number of labels needed.

See the ActiveTigger documentation for setup and usage.

AI coding assistants — Claude Code skill#

The team maintains a Claude Code skill, marcelle-remote, that lets Claude Code run commands, transfer files and manage long GPU jobs on the 4×H100 over SSH (tmux, NVLink-aware GPU selection, storage rules). Installation instructions are in the guide_connexion_marcelle repository (team members only). It requires the SSH config and key to work without a password.

What's next?#