magnus919_agent-skills/vllm/references/06-gpu-ops-and-lifecycle.md

vLLM GPU Operation, Upgrade, and Rollback

Last Updated: 2026-08-03 Sources: https://docs.vllm.ai/en/latest/configuration/engine_args/, https://docs.vllm.ai/en/latest/deployment/docker/, https://github.com/vllm-project/vllm/releases

GPU operation

• Inventory first: nvidia-smi (NVIDIA) or rocm-smi (AMD) shows device list, per-GPU memory, utilization, temperature, power, and ECC/error state. Record it before and after changes; a thermal or ECC change invalidates benchmark comparisons.
• Device selection: CUDA_VISIBLE_DEVICES="0,1" vllm serve ... restricts which GPUs the process sees; --tensor-parallel-size counts the visible devices in order. Device names are the container's view, not the host's — verify with nvidia-smi -L inside the container.
• Per-GPU memory matters: tensor parallelism assumes equal per-GPU memory. Mixed-capacity GPUs or other workloads sharing a GPU break the memory profile; pin the serving process to dedicated devices.
• Observability: /metrics exposes vllm:gpu_cache_usage_perc (KV cache pressure), vllm:num_requests_running, vllm:num_requests_waiting, and token counters. Prometheus scraping of /metrics is the standard wiring; alert on cache usage near 1.0 with requests waiting (capacity), repeated OOM, and probe failures.
• OOM at startup usually means weights + KV cache do not fit: reduce --max-model-len, switch to a quantized checkpoint, add GPUs, or (as a stopgap) --cpu-offload-gb. Lowering --gpu-memory-utilization does not help if the weights alone exceed memory.
• OOM mid-run means KV cache pressure at peak concurrency: shorten context, lower --max-num-seqs, or scale out replicas.

Upgrade

1. Pin before you start: image tag (or pip install vllm==<version>), model revision, and the full serving command from the serving config record. Without pins, upgrades are unreproducible and rollback is guesswork.
2. Read the release notes for the full span (for example v0.25.1 → v0.26.0): vLLM deprecates and renames flags, changes defaults (memory allocation, max-model-len derivation, CUDA graph capture), and adjusts kernel/hardware support between releases. A config that ran on one minor may not behave the same on the next.
3. Rehearse on a scratch instance with the real model and workload: install the new version, apply the config, confirm the model loads, and run the frozen benchmark.
4. Swap with a rollback plan: deploy the new pinned image, verify at the delivery boundary (/health, /v1/models, a representative request, re-benchmark), and keep the previous image + config ready to reapply.

Rollback

• Rollback is a redeploy of the previous pinned image + serving config. Because the config record is versioned and the image tag is pinned, the old state is reproducible by construction.
• Verify the rollback the same way as an upgrade: /health, /v1/models, one representative request, and the frozen benchmark. Do not assume the old behavior returned because the old image is back.
• Watch for config drift across versions: KV cache layout, defaults, and flag names change between releases. A rollback must restore the recorded config, not the current one.

Troubleshooting table

Symptom	First evidence to collect	Likely causes and next step
Server crashes at startup (OOM)	Side-by-side startup logs, nvidia-smi, image digests	Weights+KV don't fit; changed defaults after upgrade. Reduce max-model-len/memory utilization on scratch, or roll back the pinned image
/health 200 but requests fail	/v1/models, one bounded request	Model not loaded, served-name mismatch, missing chat template; check /load and startup log
404 on /v1/chat/completions	/v1/models, model type	Pooling/embedding model or missing chat template; see references/03-openai-api.md
High p99 latency	TTFT vs TPOT split, cache-usage metrics	Prefill blocking decode → chunked prefill; KV pressure → bound concurrency
Requests waiting, cache ~100%	vllm:gpu_cache_usage_perc, waiting gauge	At capacity: scale out or reduce context/concurrency; do not raise batch limits
Container killed at startup	kubectl get events, log KeyboardInterrupt: terminated	Probe failureThreshold too low for model load time; raise it
Slow restart every time	Startup duration, VLLM_CACHE_ROOT	Compile cache not persisted; mount ~/.cache/vllm volume

Hard boundaries

• No restart, redeploy, scale, or upgrade without an explicit human directive naming the target and a stated rollback path. Read-only discovery may proceed without confirmation.
• Never compare performance across versions, models, quants, parallelism, contexts, batches, or workloads as if one variable changed.
• Never expose development-only endpoints, request logs with prompt content, or credentials to chat or logs.