mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-11 19:47:12 +03:00
* feat(skill): add vLLM inference-serving skill (#247) Add a single-tool vllm skill covering Docker/Kubernetes deployment, quantization-aware model configuration (tensor parallelism, KV cache), the OpenAI-compatible API surface, throughput/latency benchmarking, continuous batching tuning, GPU operation, and upgrade/rollback. Ships a read-only vllm-health probe (stdlib-only, --json), fillable serving-config and benchmark-run-record templates, seven dated references with upstream sources, a human-facing README, tests, and a schema-v1 eval manifest with six cases covering config, benchmarking, and troubleshooting. Route ml-engineering to the new skill via a resolvable link alongside llama-cpp, add the vllm entry to the top-level README index, and regenerate the tracked catalogs. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * fix(skill): emit timeout exit 124 and bound /metrics reads in vllm-health Address the review observations on the bundled probe: requests that exceed --timeout now raise ProbeTimeout and make the tool exit 124 as documented (previously they surfaced as exit 1), and the metrics check reads at most 64 KiB of /metrics and reports truncation instead of reading the whole body. Adds tests for both behaviors. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> --------- Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
This commit is contained in:
Magnus Hedemark
co-authored by
factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
parent
f83d48ba53
commit
6181f1746d
@@ -0,0 +1,61 @@
|
||||
# vLLM — Inference-Serving Skill
|
||||
|
||||
Operate, configure, benchmark, and troubleshoot vLLM inference servers: Docker and Kubernetes deployment, quantization-aware model configuration (tensor parallelism, KV cache), the OpenAI-compatible API surface, throughput/latency benchmarking, continuous batching tuning, GPU operation, and upgrade/rollback.
|
||||
|
||||
## Why Install This Skill
|
||||
|
||||
Your agent can run a vLLM deployment instead of guessing. Teams that self-serve open models in production need someone (or something) that knows how to start a `vllm serve` with the right flags, size the model and its KV cache for the GPUs at hand, confirm the OpenAI-compatible endpoints actually work, measure throughput and latency with evidence that comparisons mean something, tune continuous batching one knob at a time, and upgrade or roll back without burning the deployment.
|
||||
|
||||
This skill ships that operating knowledge plus two fillable templates — a serving configuration record (so every deployment is reproducible) and a benchmark run record (so every performance claim is comparable) — and a read-only `vllm-health` probe that checks a running server's health, version, models, load, and metrics over HTTP without changing anything. The references are distilled from the official vLLM documentation with dated sources. Serving strategy and engine-selection methodology deliberately route up to `ml-engineering`; the llama.cpp stack routes to `llama-cpp`; this skill owns the day-to-day operation of vLLM itself.
|
||||
|
||||
## What You Get
|
||||
|
||||
| Directory | Purpose |
|
||||
|---|---|
|
||||
| `SKILL.md` | Agent-facing operating loop, mutation gates, and verification boundaries |
|
||||
| `references/` | Seven dated, source-indexed references: source index, deployment, model configuration, OpenAI API, benchmarking, batching/tuning, GPU ops and lifecycle |
|
||||
| `templates/serving-config.md` | Fillable record of every serving argument, model revision, and environment — the rollback unit |
|
||||
| `templates/benchmark-run-record.md` | Fillable record that makes throughput/latency evidence comparable across runs |
|
||||
| `scripts/vllm-health` | Read-only probe: health, version, models, load, and metrics; stdlib-only, `--json`, `--help` without a server |
|
||||
| `tests/` | Deterministic tests against a local stub HTTP server, including the read-only contract |
|
||||
| `evals/evals.json` | Six output-quality evaluation cases for agent runs |
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Help works with no vLLM server
|
||||
scripts/vllm-health --help
|
||||
|
||||
# Probe a running server, machine-readable
|
||||
scripts/vllm-health --url http://127.0.0.1:8000 --json
|
||||
|
||||
# Targeted checks
|
||||
scripts/vllm-health --check health --check models --json
|
||||
|
||||
# Record what you are about to run before you run it
|
||||
# (fill in vllm/templates/serving-config.md), then:
|
||||
docker run --runtime nvidia --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=$HF_TOKEN" \
|
||||
-p 8000:8000 --ipc=host \
|
||||
vllm/vllm-openai:v0.26.0 \
|
||||
--model <model-name> --served-model-name <api-name> --max-model-len <len>
|
||||
|
||||
# Benchmark serving throughput/latency once the server is ready
|
||||
vllm bench serve --backend vllm --model <model-name> \
|
||||
--endpoint /v1/completions --dataset-name custom \
|
||||
--dataset-path prompts.jsonl --num-prompts 100 --request-rate inf
|
||||
```
|
||||
|
||||
The `vllm-health` script uses only Python's standard library and issues GET requests only. Exit codes: 0 all checks passed, 1 issues found or a fatal error, 2 usage error, 124 timeout. Benchmark output includes request throughput (req/s), output token throughput (tok/s), and TTFT/TPOT/ITL percentiles — record them in `templates/benchmark-run-record.md`.
|
||||
|
||||
## Triggers
|
||||
|
||||
Load this skill for vLLM operations: deploying or updating a `vllm serve` server (bare, Docker, or Kubernetes), choosing serving flags (`--quantization`, `--tensor-parallel-size`, `--max-model-len`, `--kv-cache-dtype`, `--gpu-memory-utilization`), wiring or debugging the OpenAI-compatible API surface (`/v1/chat/completions`, `/v1/completions`, `/v1/models`, `/health`, tool calling, chat templates), measuring serving throughput or latency (`vllm bench serve`/`bench throughput`), tuning continuous batching, running or diagnosing GPUs under a vLLM workload, or planning a vLLM upgrade or rollback. Do not load it for model training or fine-tuning (that's `ml-engineering`), for the llama.cpp stack (that's `llama-cpp`), or for generic Kubernetes/Docker administration (that's `kubernetes`/`docker-compose`).
|
||||
|
||||
## Requirements
|
||||
|
||||
- A vLLM release: the `vllm/vllm-openai` Docker image (NVIDIA CUDA, AMD ROCm, or Intel XPU variants) or `pip install vllm==<pinned-version>`.
|
||||
- An accelerator with the matching driver (NVIDIA with the NVIDIA Container Toolkit for Docker, or the platform equivalent), or a supported CPU build for testing.
|
||||
- Hugging Face access to the model: a mounted `~/.cache/huggingface` and an `HF_TOKEN` for gated models.
|
||||
- Python 3.9+ for the `vllm-health` script (`--help` needs nothing else); live probes need HTTP(S) access to the running server.
|
||||
+152
@@ -0,0 +1,152 @@
|
||||
---
|
||||
name: vllm
|
||||
description: >-
|
||||
Operate, configure, benchmark, and troubleshoot vLLM inference servers: Docker
|
||||
and Kubernetes deployment, quantization-aware model configuration (tensor
|
||||
parallelism, KV cache), OpenAI-compatible API serving, throughput and latency
|
||||
benchmarking, continuous batching tuning, GPU operation, and upgrade/rollback.
|
||||
Use when deploying or running a vLLM server (vllm serve, vllm/vllm-openai),
|
||||
sizing a model and its KV cache for GPUs, selecting quantization and
|
||||
parallelism, serving via /v1 endpoints, measuring serving throughput or
|
||||
latency, tuning batching, or diagnosing GPU, OOM, or startup failures in a
|
||||
vLLM deployment. Do not use for model training, fine-tuning, evaluation-set
|
||||
design, or engine-selection methodology (that is ml-engineering), or for
|
||||
operating the llama.cpp stack with GGUF models (that is llama-cpp); other
|
||||
inference engines (TGI, Ollama, Triton) are out of scope.
|
||||
license: MIT
|
||||
compatibility: >-
|
||||
Requires a vLLM release (v0.26.0 or a pinned older release), an NVIDIA CUDA,
|
||||
AMD ROCm, or Intel XPU GPU with the matching driver, or a supported CPU build.
|
||||
The bundled vllm-health script runs on Python 3.9+ and needs no vLLM server
|
||||
for --help; live probes require HTTP(S) access to a running vLLM server.
|
||||
metadata:
|
||||
source: https://docs.vllm.ai/en/latest/
|
||||
source_index: references/00-source-index.md
|
||||
research_checked: "2026-08-03"
|
||||
---
|
||||
|
||||
# vLLM Inference Serving
|
||||
|
||||
Use this skill to operate **vLLM** as a production inference server: deploy it with Docker or Kubernetes, configure the model and engine (quantization, tensor parallelism, KV cache, context length), serve the OpenAI-compatible API surface, benchmark throughput and latency with comparable evidence, tune continuous batching, operate the GPUs underneath, and upgrade or roll back safely. This is a **tool skill** for one named engine. Serving *methodology* — engine selection, quantization trade-offs, deployment plans, regression triage — belongs to [ml-engineering](../ml-engineering/SKILL.md); local single-node GGUF serving with the llama.cpp stack belongs to [llama-cpp](../llama-cpp/SKILL.md). This skill owns the day-to-day operation of vLLM itself.
|
||||
|
||||
## Operating contract
|
||||
|
||||
1. **Record the deployment before tuning it.** Capture the vLLM version or image digest, model and revision, quantization, parallelism, `max-model-len`, KV cache settings, batching limits, GPU inventory, and workload. The [serving config template](templates/serving-config.md) exists for exactly this.
|
||||
2. **Confirm the target, scope, and rollback path before acting.** Read-only discovery (health probes, `/metrics`, `nvidia-smi`) may proceed without confirmation. Mutations — restarting a server, changing serving args, scaling replicas, upgrading the image — require an explicit human directive naming the deployment.
|
||||
3. **A server that responds is not a server that serves.** `/health` returning 200 proves liveness, not that the model loaded or that inference works. Verify at the delivery boundary: `/v1/models` reports the served model and a representative request returns generated tokens.
|
||||
4. **Benchmark before and after every change.** vLLM flags, defaults, and behavior change between releases; an unmeasured tuning change is a guess. Compare only matched conditions (version, model, GPU, context, batch, workload) and record the evidence in the [benchmark run record](templates/benchmark-run-record.md).
|
||||
5. **Keep evidence bounded.** Summarize logs, configs, and metrics; never dump full server logs, `.env` files, or HF tokens into chat. `--enable-log-requests` with debug logging can leak prompt content; keep request logging off or redacted in shared sessions.
|
||||
|
||||
## The vllm-health script
|
||||
|
||||
`scripts/vllm-health` is an agent-first, read-only probe for a running vLLM server. It issues GET requests only, never mutates, and emits bounded JSON.
|
||||
|
||||
```bash
|
||||
scripts/vllm-health --help # no server needed
|
||||
scripts/vllm-health --url http://127.0.0.1:8000 --json
|
||||
scripts/vllm-health --check health --check models --json
|
||||
scripts/vllm-health --check metrics --timeout 10 --json
|
||||
```
|
||||
|
||||
Exit codes: 0 all checks passed, 1 issues found or a fatal error, 2 usage error, 124 timeout. Checks: `health` (`/health`), `version` (`/version`), `models` (`/v1/models`), `load` (`/load`), and `metrics` (a bounded prefix of `/metrics`). The script never sends data anywhere and never writes files.
|
||||
|
||||
## Operating loop
|
||||
|
||||
1. **Identify the deployment**: vLLM version or image digest, model and revision, served model name, parallelism, and how it is deployed (bare `vllm serve`, Docker, Kubernetes).
|
||||
2. **Collect evidence**: run `vllm-health --json` for health, version, models, and load; check `/metrics` counters (`vllm:num_requests_running`, `vllm:num_requests_waiting`, `vllm:gpu_cache_usage_perc`); inspect GPU state with `nvidia-smi`.
|
||||
3. **Triage against the symptom**: map the problem to the evidence (OOM → KV cache or `gpu_memory_utilization`; high latency → batching, TTFT vs TPOT; model not found → served name or chat template; slow start → model download or compile cache).
|
||||
4. **Act with confirmation**: bounded, scoped changes after a human directive, with a rollback path named first.
|
||||
5. **Verify**: re-run the probe and the representative request at the delivery boundary, and re-benchmark if the change affects performance.
|
||||
|
||||
## Deployment: Docker and Kubernetes
|
||||
|
||||
- **Docker**: the official image is `vllm/vllm-openai` (Docker Hub). Run with GPU access, the Hugging Face cache mounted, the HF token for gated models, port 8000 published, and `--ipc=host` (or a `--shm-size`) for the shared memory tensor parallelism relies on. See [references/01-deployment.md](references/01-deployment.md).
|
||||
- **Kubernetes**: a Deployment with `nvidia.com/gpu` (or `amd.com/gpu`) resources, a PVC for the model cache, an `emptyDir` backed by Memory at `/dev/shm`, liveness/readiness probes on `/health` port 8000, and a Service. Raise probe `failureThreshold` for large models that take minutes to load — a premature kill shows up as `KeyboardInterrupt: terminated` in the container log.
|
||||
- Pin image tags to a release (for example `vllm/vllm-openai:v0.26.0`) instead of `latest`, and persist the compile cache (default `~/.cache/vllm`) across restarts so `torch.compile` artifacts are reused.
|
||||
|
||||
## Model configuration
|
||||
|
||||
- **Model identity**: `--model` is the HF repo or local path; `--revision` pins the exact weights. `--served-model-name` sets the name clients must use in `/v1` requests and in the `model` field of responses. `--trust-remote-code` is required for some model repos and should be reviewed before use.
|
||||
- **Context length**: `--max-model-len` bounds prompt plus output per request. Unset, it derives from the model config; `-1`/`auto` picks the largest length that fits GPU memory. It is the single biggest driver of KV cache size.
|
||||
- **Quantization-aware serving**: pass `--quantization` (or `-q`) only when the model weights require it (GPTQ/AWQ/GGUF checkpoints load their scheme from config). Weight types and activation dtypes must match what the kernels support; a quantized model served at the wrong dtype fails to load or silently degrades. Hardware support varies by method (see [references/02-model-configuration.md](references/02-model-configuration.md)).
|
||||
- **Tensor parallelism**: `--tensor-parallel-size N` shards one model across N GPUs in the same node; `--pipeline-parallel-size` splits layers across nodes. TP requires NVLink/fast interconnect and equal per-GPU memory; startup logs the memory profiling result, which is the evidence that the model fits.
|
||||
- **KV cache**: `--gpu-memory-utilization` (default 0.92) caps the fraction of GPU memory the model plus KV cache may use. `--kv-cache-dtype fp8` shrinks the cache for long contexts on supported GPUs. The engine logs `GPU KV cache size: N tokens` and the implied max concurrency — record both; they tell you how many concurrent requests of a given length the box can hold.
|
||||
|
||||
## OpenAI-compatible API surface
|
||||
|
||||
- Basic endpoints: `/health` (liveness), `/version`, `/v1/models` (served models), `/load` (load metrics), `/metrics` (Prometheus). Inference: `/v1/completions` and `/v1/chat/completions` (chat requires the model to ship a chat template, or pass `--chat-template`); `/v1/embeddings` for pooling models; `/v1/responses` for the Responses API.
|
||||
- Streaming, tool calling (`--enable-auto-tool-choice --tool-call-parser openai`), structured outputs, and parallel sampling are server-side options that change request/response behavior — verify each against the installed release rather than assuming parity.
|
||||
- Exposing the server beyond loopback requires an explicit decision about bind address, API keys, TLS or a trusted reverse proxy, and firewall rules. Development-only endpoints (`/reset_prefix_cache`, weight transfer, profiling) must not be exposed in production.
|
||||
|
||||
## Benchmarking: throughput and latency
|
||||
|
||||
- **Online serving benchmark**: run `vllm bench serve` against a live server with a representative dataset (ShareGPT, a local `custom` JSONL, or your own prompts) and fixed `--num-prompts`, `--request-rate`, and `--max-concurrency`. It reports request throughput (req/s), output token throughput (tok/s), total token throughput, and TTFT/TPOT/ITL percentiles.
|
||||
- **Offline throughput**: `vllm bench throughput` measures raw engine throughput without the HTTP path; use it for engine-only comparisons, not end-to-end user latency.
|
||||
- **Comparable evidence**: the benchmark run record template freezes version, model, quantization, parallelism, context, batching, GPU, dataset, and load pattern. Never compare numbers across different conditions as if one variable changed. TTFT is a latency metric; token throughput is a throughput metric — an optimization that helps one can hurt the other.
|
||||
- For production capacity testing, vLLM's docs recommend the separate GuideLLM framework; this skill's scope is the bundled `vllm bench` tools.
|
||||
|
||||
## Continuous batching tuning
|
||||
|
||||
- vLLM batches continuously by default: the scheduler admits sequences as capacity frees up, mixing prefill and decode. `--max-num-seqs` caps sequences per iteration, `--max-num-batched-tokens` caps tokens per iteration, and `--enable-chunked-prefill` lets prefill share an iteration with decode.
|
||||
- Start from defaults and change one knob at a time against the frozen benchmark: raising `--max-num-seqs` raises throughput at the cost of per-request latency and KV cache pressure; lowering it improves latency stability at the cost of utilization.
|
||||
- `--enable-prefix-caching` reuses KV blocks across requests with shared prefixes (chat system prompts, RAG contexts); the hit rate is visible in `/metrics` and in the benchmark's input token accounting. `--performance-mode` trades between `interactivity` (latency) and `throughput` at the kernel level.
|
||||
|
||||
## GPU operation
|
||||
|
||||
- Verify GPUs with `nvidia-smi` (or `rocm-smi` on AMD): device list, memory, utilization, temperature, and ECC errors before and after changes. `CUDA_VISIBLE_DEVICES` selects which GPUs a `vllm serve` process sees; tensor parallel ranks map to the visible devices in order.
|
||||
- Watch `/metrics` for `vllm:gpu_cache_usage_perc` (KV cache pressure), `vllm:num_requests_running`/`waiting`, and `vllm:generation_tokens_total`. A cache-usage signal near 1.0 with requests waiting means the deployment is at capacity — scale out or reduce `max-model-len`/concurrency rather than overcommitting.
|
||||
- OOM during startup usually means the model + KV cache did not fit: lower `--gpu-memory-utilization` does not help if weights alone exceed memory — reduce `--max-model-len`, switch quantization, or add GPUs. OOM mid-run means KV cache pressure: shrink context, concurrency, or batch limits.
|
||||
|
||||
## Upgrade and rollback
|
||||
|
||||
- **Pin everything**: image tag or `pip install vllm==<version>`, model revision, and the full serving command. `latest` images and unpinned revisions make rollback impossible and upgrades unreproducible.
|
||||
- **Upgrade path**: read the release notes for the full version span, review changed/removed flags (`--engine-args` change frequently), validate the new version on a scratch instance with the real model and workload, re-run the frozen benchmark, then swap with a rollback plan: previous image tag and previous serving config ready to reapply.
|
||||
- **Rollback**: because the config is versioned, rollback is a redeploy of the previous pinned image + config. KV cache layout, defaults, and flag names change between releases — do not assume a config that ran on v0.25.x behaves identically on v0.26.x without re-validating and re-benchmarking.
|
||||
|
||||
## Reference routing
|
||||
|
||||
| Load when | Reference |
|
||||
|---|---|
|
||||
| Sources, version observations, refresh procedure | `references/00-source-index.md` |
|
||||
| Docker and Kubernetes deployment, image pinning, probes, storage | `references/01-deployment.md` |
|
||||
| Model config: quantization, tensor parallelism, KV cache, memory budgeting | `references/02-model-configuration.md` |
|
||||
| OpenAI-compatible API surface, chat templates, tools, auth | `references/03-openai-api.md` |
|
||||
| Benchmarking methodology and `vllm bench` commands | `references/04-benchmarking.md` |
|
||||
| Continuous batching, chunked prefill, prefix caching, performance mode | `references/05-batching-and-tuning.md` |
|
||||
| GPU operation, observability, upgrade/rollback, troubleshooting | `references/06-gpu-ops-and-lifecycle.md` |
|
||||
|
||||
## Included artifacts
|
||||
|
||||
- `scripts/vllm-health`: read-only health/version/models/load/metrics probe (stdlib-only, `--json`, `--check` subsets, `--help` without a server).
|
||||
- `tests/test_vllm_health.py`: deterministic tests against a local stub HTTP server, including the read-only contract.
|
||||
- `templates/serving-config.md` and `templates/benchmark-run-record.md`: fillable records that make deployments reproducible and benchmark evidence comparable.
|
||||
- `references/`: seven dated, source-indexed references covering the operational topics above.
|
||||
- `evals/evals.json`: six output-quality evaluation cases for agent runs.
|
||||
|
||||
## Verification boundary
|
||||
|
||||
| Claim | Minimum evidence |
|
||||
|---|---|
|
||||
| The server is alive | `vllm-health --check health` reports `/health` 200 |
|
||||
| The right model is served | `/v1/models` lists the expected served model name |
|
||||
| Inference works | A representative `/v1/chat/completions` or `/v1/completions` request returns generated tokens with a `finish_reason` |
|
||||
| The model fits | Startup log shows memory profiling completed and `GPU KV cache size: N tokens` for the configured parallelism |
|
||||
| A tuning change helped | The frozen benchmark shows the declared metric improving with matched conditions, variance reported |
|
||||
| The deployment is upgradable | Previous pinned image + serving config are recorded and the upgrade was rehearsed on a scratch instance |
|
||||
| A diagnosis is sound | Evidence was collected before the claim, and the fix was verified by re-running the probe and the benchmark |
|
||||
|
||||
## Hard boundaries
|
||||
|
||||
- Never restart, redeploy, scale, or upgrade a vLLM deployment without an explicit human directive naming the target and a stated rollback path. Read-only discovery may proceed freely.
|
||||
- Never expose an unauthenticated server beyond loopback by accident; development-only endpoints and profiling routes must stay off production ingress.
|
||||
- Never print or commit HF tokens, `.env` contents, or full server logs; summarize evidence instead.
|
||||
- Never compare benchmark numbers from different versions, models, quants, parallelism, contexts, batches, or workloads as if one variable changed.
|
||||
- Never run `vllm-health` as anything but what it is — read-only. It has no mutation surface.
|
||||
|
||||
## When not to use
|
||||
|
||||
- **Model training, fine-tuning, evaluation-set design, quantization decisions, and serving methodology** — that is [ml-engineering](../ml-engineering/SKILL.md).
|
||||
- **The llama.cpp stack** (llama-cli, llama-server, GGUF conversion and quantization, local Metal/CUDA builds) — that is [llama-cpp](../llama-cpp/SKILL.md).
|
||||
- **Other inference engines** (TGI, Ollama, Triton, vLLM's embedding/rerank-only workloads are in scope, but engine selection among them is not) — engine-selection trade-offs belong to `ml-engineering`.
|
||||
- **Kubernetes and Docker fundamentals** (manifests, RBAC, image registries, GPU device plugins) — that is [kubernetes](../kubernetes/SKILL.md) and [docker-compose](../docker-compose/SKILL.md).
|
||||
- **GPU infrastructure provisioning** (drivers, cluster scheduling, capacity planning) — that is [platform-engineering](../platform-engineering/SKILL.md); this skill operates the GPUs a vLLM server already targets.
|
||||
@@ -0,0 +1,78 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"skill_name": "vllm",
|
||||
"evals": [
|
||||
{
|
||||
"id": "serving-config-review",
|
||||
"prompt": "A team wants to serve a 70B instruct model (bf16) on two A100 80GB GPUs in one node with a 32K context window. Draft the vllm serve command with serving flags, explain the memory and parallelism reasoning, and state what to verify after startup.",
|
||||
"expected_output": "A vllm serve command that pins the model and revision, serves a stable API name, and configures parallelism and memory deliberately: --model <repo> --revision <sha> --served-model-name <api-name> --tensor-parallel-size 2 --max-model-len 32768 --gpu-memory-utilization 0.9 --trust-remote-code only if the model card requires it. The reasoning explains that a 70B bf16 checkpoint needs roughly 140GB of weights, so two 80GB GPUs with tensor parallelism are required (weights plus KV cache plus activations must fit under the per-instance gpu-memory-utilization cap), that --pipeline-parallel-size would only be needed across nodes, that --max-model-len 32K directly sizes the KV cache (larger context means fewer concurrent sequences), and that TP requires NVLink and equal per-GPU memory. Post-startup verification: the startup log shows memory profiling completed and the GPU KV cache size, /v1/models lists the served name, and a representative chat request returns tokens. The record goes into the serving config template so the deployment is reproducible and rollback is a redeploy of the same pinned image and arguments.",
|
||||
"assertions": [
|
||||
"The command pins model, revision, served-model-name, tensor-parallel-size 2, max-model-len, and gpu-memory-utilization",
|
||||
"The 70B bf16 weight size (~140GB) is used to justify two A100 80GB GPUs under the memory-utilization cap",
|
||||
"max-model-len is tied to KV cache sizing and concurrent-sequence capacity",
|
||||
"Verification includes startup memory profiling, /v1/models, and a representative chat request",
|
||||
"The serving config template is used so the deployment is reproducible and rollback is a pinned-image redeploy"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "quantization-and-parallelism-sizing",
|
||||
"prompt": "Our 8B model in bf16 takes 16GB of VRAM and we want to serve 256 concurrent long-context requests. We have a single 80GB H100. Should we quantize, enable tensor parallelism, or both, and what trade-offs should we record before changing anything?",
|
||||
"expected_output": "A sizing analysis that measures before optimizing: establish the baseline with the current bf16 serving config, record the reported GPU KV cache size and max concurrency for the target context, then evaluate options against that baseline. On a single H100, tensor parallelism is not applicable (TP shards across multiple GPUs) so the levers are quantization and KV cache settings: an FP8 or INT4 weight format shrinks weights and frees GPU memory for a larger KV cache, --kv-cache-dtype fp8 shrinks the cache for long contexts on supported hardware, and --max-model-len or --max-num-seqs bound concurrency. The trade-offs are stated explicitly: quantization changes output quality and requires kernels that match the hardware generation, and more KV cache (longer context or higher concurrency) increases gpu-cache-usage until requests wait. The change record freezes version, model, quantization, context, batching, and workload, and verification re-runs the frozen benchmark plus a quality spot-check rather than assuming the quantized model matches bf16 behavior.",
|
||||
"assertions": [
|
||||
"The response measures the baseline (KV cache size, max concurrency) before optimizing",
|
||||
"Tensor parallelism is correctly deemed not applicable on a single GPU",
|
||||
"Quantization (FP8/INT4 weights) and kv-cache-dtype fp8 are evaluated as the memory levers, with quality trade-offs stated",
|
||||
"KV cache pressure is tied to gpu-cache-usage and waiting requests",
|
||||
"Verification re-runs the frozen benchmark and a quality spot-check under matched conditions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "openai-api-integration",
|
||||
"prompt": "We started vllm serve with a chat model and the OpenAI SDK returns a 404 for /v1/chat/completions, while /v1/models works. Diagnose the likely causes and describe how to verify the fix, without exposing the server.",
|
||||
"expected_output": "A diagnostic sequence over the read-only probes: confirm /health and /v1/models respond, then check what /v1/models actually lists and compare it with the model name the client sends, since a served-model-name mismatch or an unserved model id produces 404-style failures on inference routes. The next checks are version and load state, and the server log for whether the model loaded and which endpoints are registered on this release, because vLLM serves /v1/chat/completions only for text-generation models that carry a chat template; a model without a chat template cannot serve chat and needs --chat-template, and some pooling or embedding models do not register chat endpoints at all. The fix is verified by sending one bounded chat request with the exact served model name and checking for generated tokens, and by confirming the server is only reachable on the intended interface with API auth or a reverse proxy in place. The response does not expose logs or tokens and treats any bind-address or API-key change as a confirmed mutation.",
|
||||
"assertions": [
|
||||
"Diagnosis starts with /health, /v1/models, and the served model name versus the client's model field",
|
||||
"Chat templates and model type (generative versus pooling) are named as the cause of missing /v1/chat/completions routes",
|
||||
"The fix is verified with one bounded chat request returning generated tokens",
|
||||
"Exposure controls (bind address, API keys, reverse proxy) are checked without dumping logs or secrets",
|
||||
"A bind or auth change is treated as a mutation requiring confirmation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "benchmark-run-review",
|
||||
"prompt": "A colleague claims a vLLM upgrade doubled our token throughput: 420 tok/s before, 850 tok/s after. Review their evidence before we roll the upgrade to production.",
|
||||
"expected_output": "A skeptical review of the benchmark evidence: the comparison is only meaningful if every other condition was frozen, so the reviewer asks for or reconstructs the matched conditions — pinned vLLM versions on both sides, identical model and revision, same quantization and dtype, same tensor parallelism, max-model-len, max-num-seqs, max-num-batched-tokens, GPU model and driver, dataset, prompt and output lengths, request rate, and concurrency — and requires the raw benchmark records from vllm bench serve showing request throughput, output token throughput, TTFT/TPOT/ITL percentiles, and variance across repetitions. A throughput doubling without changes to hardware or workload is a red flag for a mismatch (for example comparing different concurrency, a different dataset, or fp8 versus bf16). The reviewer also separates throughput from latency: if the goal is user-facing latency, TPOT and TTFT under realistic load matter more than raw tok/s. The upgrade only ships if the matched benchmark confirms the claim and the run record template was used on both sides.",
|
||||
"assertions": [
|
||||
"The review demands matched conditions (version, model, quant, parallelism, context, batching, GPU, workload) on both sides",
|
||||
"Raw vllm bench serve output with throughput, TTFT/TPOT/ITL percentiles, and variance is required, not a single number",
|
||||
"A suspicious doubling is probed for a mismatch in concurrency, dataset, or precision",
|
||||
"Throughput and latency (TTFT/TPOT under load) are treated as separate metrics",
|
||||
"The benchmark run record template is required on both sides before the upgrade ships"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "continuous-batching-tuning",
|
||||
"prompt": "Our vLLM server reports gpu_cache_usage_perc above 0.95 with requests waiting, and p50 latency is fine but p99 spiked. We already run with default batching settings. What should we change, and what should we measure?",
|
||||
"expected_output": "A one-variable-at-a-time tuning plan anchored on evidence: the cache-usage and waiting-request metrics say the deployment is at KV cache capacity, so the options are to bound concurrency (lower --max-num-seqs to cap sequences per iteration), reduce per-request token demand (lower --max-model-len or cap output tokens), enable or widen chunked prefill (--enable-chunked-prefill with a deliberate --max-num-batched-tokens) so prefill no longer blocks decode, or scale out, rather than raising --max-num-seqs further which would worsen p99. Prefix caching (--enable-prefix-caching) is evaluated if requests share system prompts, since a higher hit rate lowers effective input tokens. Every change is measured with the frozen benchmark plus the /metrics counters (gpu_cache_usage_perc, num_requests_running/waiting) before and after, one change at a time, and p99 is reported with its distribution rather than a single run.",
|
||||
"assertions": [
|
||||
"The plan ties gpu_cache_usage_perc near 1.0 with waiting requests to KV cache capacity, not to raising max-num-seqs",
|
||||
"Concrete levers are named: max-num-seqs, max-model-len or output cap, chunked prefill with max-num-batched-tokens, or scale-out",
|
||||
"Prefix caching is evaluated for shared-prefix workloads",
|
||||
"Changes are made one at a time with the frozen benchmark and /metrics counters before and after",
|
||||
"p99 is reported with distribution or variance, not a single run"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "upgrade-rollback-troubleshooting",
|
||||
"prompt": "We upgraded the vllm/vllm-openai image from :v0.25.1 to :v0.26.0 and the server now crashes at startup with a CUDA out of memory error, even though the model did not change. Diagnose and decide: fix forward or roll back?",
|
||||
"expected_output": "A bounded diagnosis that starts with the read-only evidence: confirm the exact image digest on both versions, the serving args, the model revision, and the GPU inventory (nvidia-smi), then compare the startup logs of the failing v0.26.0 run against the v0.25.1 run. OOM at startup after an engine upgrade with the same model points at changed memory defaults or changed flag semantics: vLLM releases change default KV cache allocation, max-model-len derivation, CUDA graph capture, and flag names, so the same config may now reserve more memory; the fix-forward path re-reads the v0.26.0 release notes for changed defaults, lowers gpu-memory-utilization or max-model-len on the scratch instance, validates the model loads and passes the frozen benchmark, and only then redeploys. The rollback path is already cheap because the previous pinned image and serving config were recorded: redeploy vllm/vllm-openai:v0.25.1 with the old args, verify /health, /v1/models, and one representative request, and re-run the benchmark. The response decides rollback first if the team cannot validate forward in the window, and never mutates the production deployment without a human directive naming target, scope, and rollback path.",
|
||||
"assertions": [
|
||||
"Diagnosis uses exact image digests, serving args, model revision, nvidia-smi, and side-by-side startup logs",
|
||||
"Changed memory defaults and flag semantics across releases are named as the leading OOM-after-upgrade cause",
|
||||
"The fix-forward path validates on a scratch instance with release notes and the frozen benchmark before redeploying",
|
||||
"The rollback path redeploys the pinned previous image plus recorded config and verifies at the delivery boundary",
|
||||
"No production mutation happens without a human directive naming target, scope, and rollback path"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
# vLLM Operations — Source Index
|
||||
|
||||
> **Last Updated:** 2026-08-03
|
||||
|
||||
This index tracks the authoritative upstream sources behind the vLLM operational
|
||||
skill and the refresh procedure for keeping it current. vLLM moves fast — flags,
|
||||
defaults, and endpoint behavior change between releases; treat any claim here as
|
||||
version-sensitive and re-verify against the installed release.
|
||||
|
||||
## Canonical sources
|
||||
|
||||
| Topic | Source |
|
||||
|---|---|
|
||||
| vLLM documentation (latest) | https://docs.vllm.ai/en/latest/ |
|
||||
| vLLM documentation (stable release) | https://docs.vllm.ai/en/stable/ |
|
||||
| Releases and release notes | https://github.com/vllm-project/vllm/releases |
|
||||
| Docker deployment | https://docs.vllm.ai/en/latest/deployment/docker/ |
|
||||
| Kubernetes deployment | https://docs.vllm.ai/en/latest/deployment/k8s/ |
|
||||
| Online serving (OpenAI-compatible API) | https://docs.vllm.ai/en/latest/serving/online_serving/ |
|
||||
| Engine arguments | https://docs.vllm.ai/en/latest/configuration/engine_args/ |
|
||||
| Quantization | https://docs.vllm.ai/en/latest/features/quantization/index.html |
|
||||
| Benchmark CLI (`vllm bench`) | https://docs.vllm.ai/en/latest/benchmarking/cli/ |
|
||||
| vLLM paper (PagedAttention, SOSP 2023) | https://arxiv.org/abs/2309.06180 |
|
||||
| Continuous batching explainer (Anyscale, by Cade Daniel et al.) | https://www.anyscale.com/blog/continuous-batching-llm-inference |
|
||||
|
||||
## Version observations (as of this refresh)
|
||||
|
||||
- Latest release: **v0.26.0** (published 2026-07-27). Release cadence is roughly
|
||||
monthly; docs publish `stable` and `latest` streams plus per-version archives.
|
||||
- The official Docker image is `vllm/vllm-openai` on Docker Hub, with
|
||||
`vllm/vllm-openai-rocm` (AMD) and `vllm/vllm-openai-xpu` (Intel) variants;
|
||||
the XPU image is official starting with v0.26.0.
|
||||
- Engine argument documentation moved to `configuration/engine_args`; engine
|
||||
args are also available as JSON-style CLI arguments (`--json-arg.key value`).
|
||||
- Benchmarking is now a first-class CLI: `vllm bench serve` (online serving),
|
||||
`vllm bench throughput` (offline), plus latency-focused and multimodal
|
||||
variants, replacing the older `benchmark_serving.py`/`benchmark_throughput.py`
|
||||
scripts. The docs recommend the external GuideLLM framework for production
|
||||
capacity testing.
|
||||
- `--prefix-caching` and `--performance-mode` (balanced/interactivity/throughput)
|
||||
are current flags on the `vllm serve` command line; older doc pages may still
|
||||
show earlier spellings.
|
||||
- Default `--gpu-memory-utilization` is 0.92 (per instance). `--kv-cache-dtype`
|
||||
accepts `auto`, `bfloat16`, `float16`, `fp8` (`fp8_e4m3`), `int8_per_token_head`,
|
||||
`nvfp4`, and other hardware-specific values on CUDA 11.8+.
|
||||
- vLLM 0.26.0 dependencies include Transformers 5.13, FlashInfer 0.6.14, and
|
||||
NIXL 1.3.1; GPU support spans NVIDIA (Ampere+ for most quantized kernels),
|
||||
AMD ROCm, and Intel XPU, with a CPU backend for testing.
|
||||
|
||||
## Refresh procedure
|
||||
|
||||
1. Re-check the sources above for a new release and read its release notes for
|
||||
changed or removed engine arguments and changed defaults.
|
||||
2. Update the version observations that changed (defaults, flag names, endpoint
|
||||
behavior, hardware support).
|
||||
3. Re-verify the SKILL.md scope keyword sweep and the routing links to
|
||||
`ml-engineering` and `llama-cpp` still resolve.
|
||||
4. Re-run the bundled probe against a test server and confirm every check still
|
||||
parses: `scripts/vllm-health --url http://127.0.0.1:8000 --json`.
|
||||
|
||||
## Related skill sources
|
||||
|
||||
- `ml-engineering` owns serving methodology: engine selection, quantization
|
||||
decisions, deployment plans, and regression triage. Its references are the
|
||||
source for engine-spanning decisions; this skill covers vLLM operation itself.
|
||||
- `llama-cpp` owns the llama.cpp stack (GGUF, llama-server). Do not duplicate
|
||||
its content here.
|
||||
@@ -0,0 +1,138 @@
|
||||
# vLLM Deployment: Docker and Kubernetes
|
||||
|
||||
> **Last Updated:** 2026-08-03
|
||||
> Sources: https://docs.vllm.ai/en/latest/deployment/docker/ and
|
||||
> https://docs.vllm.ai/en/latest/deployment/k8s/
|
||||
|
||||
## Docker
|
||||
|
||||
The official image is `vllm/vllm-openai` (NVIDIA CUDA), with
|
||||
`vllm/vllm-openai-rocm` (AMD) and `vllm/vllm-openai-xpu` (Intel) variants. The
|
||||
image entrypoint is `vllm serve`, so engine arguments follow the image tag.
|
||||
|
||||
```bash
|
||||
docker run --runtime nvidia --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=$HF_TOKEN" \
|
||||
-p 8000:8000 \
|
||||
--ipc=host \
|
||||
vllm/vllm-openai:v0.26.0 \
|
||||
--model Qwen/Qwen3-0.6B
|
||||
```
|
||||
|
||||
### Container details that matter
|
||||
|
||||
- **Shared memory**: use `--ipc=host` or a `--shm-size` (for example `--shm-size=2g`).
|
||||
PyTorch uses shared memory between processes, particularly for tensor-parallel
|
||||
inference; the default 64 MB `/dev/shm` in a container is too small and causes
|
||||
obscure crashes at load time.
|
||||
- **HF token**: pass `--env "HF_TOKEN=$HF_TOKEN"` for gated models. Never commit
|
||||
the token; mount the cache volume so weights are reused across restarts.
|
||||
- **CUDA compatibility**: on hosts whose driver is older than the toolkit in the
|
||||
image, set `VLLM_ENABLE_CUDA_COMPATIBILITY=1` (only for select datacenter GPUs).
|
||||
- **Compile cache**: vLLM compiles `torch.compile` artifacts into
|
||||
`VLLM_CACHE_ROOT` (default `~/.cache/vllm`). Mount a named volume there so the
|
||||
second and later containers start fast instead of recompiling.
|
||||
- **Non-root**: the image ships a `vllm` user (UID 2000, GID 0). Run with
|
||||
`--user 2000:0` and mount writable paths under `/home/vllm` (for example
|
||||
`/home/vllm/.cache/huggingface`) rather than `/root`. The `vllm-openai-nonroot`
|
||||
image target supports OpenShift-style arbitrary UIDs within group 0.
|
||||
- **Pin tags**: use a release tag (`vllm/vllm-openai:v0.26.0`), not `latest`.
|
||||
Optional dependencies (audio, gRPC, etc.) are not in the base image; layer
|
||||
them on with `uv pip install --system vllm[extra]==<same-version>`.
|
||||
|
||||
## Kubernetes
|
||||
|
||||
A native deployment: Deployment + Service, GPU resource limits, a model-cache
|
||||
volume, an `emptyDir` shared-memory volume, and `/health` probes.
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: vllm-server
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: vllm
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: vllm
|
||||
spec:
|
||||
volumes:
|
||||
- name: cache-volume
|
||||
persistentVolumeClaim:
|
||||
claimName: vllm-models
|
||||
- name: shm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: "2Gi"
|
||||
containers:
|
||||
- name: vllm
|
||||
image: vllm/vllm-openai:v0.26.0
|
||||
command: ["/bin/sh", "-c"]
|
||||
args:
|
||||
- "vllm serve <model> --served-model-name <name> --trust-remote-code"
|
||||
env:
|
||||
- name: HF_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token-secret
|
||||
key: token
|
||||
ports:
|
||||
- containerPort: 8000
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: "1"
|
||||
requests:
|
||||
nvidia.com/gpu: "1"
|
||||
volumeMounts:
|
||||
- mountPath: /root/.cache/huggingface
|
||||
name: cache-volume
|
||||
- mountPath: /dev/shm
|
||||
name: shm
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8000
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8000
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 5
|
||||
```
|
||||
|
||||
### K8s details that matter
|
||||
|
||||
- **GPU scheduling**: request `nvidia.com/gpu: "1"` (NVIDIA device plugin) or
|
||||
`amd.com/gpu` (AMD k8s device plugin). Tensor parallelism across GPUs in one
|
||||
pod uses `--tensor-parallel-size` equal to the GPU count; the pod then needs
|
||||
a larger `/dev/shm` (size it at 2-8 GiB) and, on some platforms, host IPC.
|
||||
- **Probes**: vLLM's `/health` endpoint reports ready only after the model
|
||||
finishes loading, which can take minutes for large models. Set
|
||||
`initialDelaySeconds` and `failureThreshold` high enough; a container killed
|
||||
by the probe loop logs `KeyboardInterrupt: terminated` and shows
|
||||
`failed startup probe, will be restarted` in `kubectl get events`. To find
|
||||
the right threshold, remove the probes, time startup, then restore them.
|
||||
- **gRPC**: pass `--grpc` (requires `vllm[grpc]` in the image) and replace the
|
||||
HTTP probes with `grpc` probes; the server then implements the standard
|
||||
gRPC health-checking protocol and returns `NOT_SERVING` while loading or
|
||||
shutting down.
|
||||
- **Alternatives**: the upstream docs also cover Helm, KServe, KubeRay,
|
||||
NVIDIA Dynamo, and the vllm-project production-stack integrations. For this
|
||||
skill's scope, the native manifest above is the reference pattern; the
|
||||
infrastructure for those frameworks routes to `kubernetes`.
|
||||
|
||||
## Verification at the delivery boundary
|
||||
|
||||
- `kubectl logs -l app.kubernetes.io/name=vllm` shows `Application startup
|
||||
complete` and `Uvicorn running on http://0.0.0.0:8000`.
|
||||
- `vllm-health --url <service-url> --check health --check models --json` shows
|
||||
`/health` 200 and the served model.
|
||||
- A bounded request returns generated tokens; a PVC-backed model cache means
|
||||
the next pod starts without re-downloading weights.
|
||||
@@ -0,0 +1,83 @@
|
||||
# vLLM Model Configuration: Quantization, Tensor Parallelism, KV Cache
|
||||
|
||||
> **Last Updated:** 2026-08-03
|
||||
> Sources: https://docs.vllm.ai/en/latest/configuration/engine_args/ and
|
||||
> https://docs.vllm.ai/en/latest/features/quantization/index.html
|
||||
|
||||
## Model identity and context
|
||||
|
||||
- `--model` is the Hugging Face repo or a local path; `--revision` pins a branch,
|
||||
tag, or commit so weight provenance is reproducible. `--tokenizer` overrides
|
||||
the tokenizer; `--chat-template` supplies a Jinja2 chat template when the
|
||||
model card lacks one (without one, chat requests error).
|
||||
- `--max-model-len` bounds prompt + output per request and accepts human-readable
|
||||
values (`32k` = 32,000; `32K` = 32,768). Unset, it derives from the model
|
||||
config. `-1`/`auto` picks the largest length that fits GPU memory, capped by
|
||||
the model's trained context. This flag is the dominant driver of KV cache size:
|
||||
halving it roughly doubles the concurrent sequences a GPU can hold.
|
||||
- `--dtype` selects weight/activation precision (`auto`, `bfloat16`, `float16`,
|
||||
`float32`, `half`). `auto` uses FP16 for FP32/FP16 models and BF16 for BF16
|
||||
models; some quantized formats are recommended at a specific dtype (for
|
||||
example `half` for AWQ).
|
||||
|
||||
## Quantization-aware serving
|
||||
|
||||
- **Let the checkpoint declare its scheme.** vLLM first checks the model's
|
||||
`quantization_config`; `--quantization`/`-q` is for cases where the config is
|
||||
missing or needs overriding. Serving a checkpoint with the wrong method fails
|
||||
to load or silently degrades.
|
||||
- **Supported methods (as of v0.26.0)**: GPTQ, AWQ, bitsandbytes (load-time
|
||||
quantization), GGUF, LLM Compressor FP8/INT8/INT4, NVIDIA Model Optimizer
|
||||
(NVFP4/MXFP4/FP8), TorchAO, and online quantization. The current index lives
|
||||
at https://docs.vllm.ai/en/latest/features/quantization/index.html.
|
||||
- **Hardware coupling**: kernel support varies by GPU generation — for example
|
||||
Marlin (GPTQ/AWQ/FP8/FP4) requires Turing+ and is NVIDIA-only; FP8 W8A8 needs
|
||||
Ada/Hopper; GGUF and bitsandbytes span more platforms. Check the compatibility
|
||||
table before choosing a quantized checkpoint for a GPU fleet.
|
||||
- **KV cache quantization**: `--kv-cache-dtype` (`auto`, `bfloat16`, `float16`,
|
||||
`fp8` = `fp8_e4m3`, `int8_per_token_head`, `nvfp4`, ...) shrinks the attention
|
||||
cache for long-context workloads on CUDA 11.8+. This is a quality-vs-capacity
|
||||
decision; spot-check outputs before trusting it at scale.
|
||||
- **Dated-source rule**: quantization support changes every release. Never
|
||||
assume a method that worked on one version works on the next; the source
|
||||
index records the refresh date.
|
||||
|
||||
## Tensor, pipeline, and data parallelism
|
||||
|
||||
- `--tensor-parallel-size N` (`-tp`) shards each layer's weights and attention
|
||||
across N GPUs in one node. It requires fast interconnect (NVLink or high-speed
|
||||
NIC) and equal per-GPU memory. Startup runs a memory-profiling pass and logs
|
||||
the resulting KV cache size — that log line is the evidence the model fits.
|
||||
- `--pipeline-parallel-size N` (`-pp`) splits layers across ranks/nodes for
|
||||
models too large for one node's aggregate memory; it adds inter-stage
|
||||
communication latency.
|
||||
- `--data-parallel-size N` (`-dp`) replicates the model across groups for
|
||||
throughput scaling of small models. `--expert-parallel-size` shards MoE
|
||||
experts. The product of TP × PP × DP (× EP for MoE) must equal the world size.
|
||||
- **Gotchas**: tensor-parallel workers must all see the same visible GPU set
|
||||
(`CUDA_VISIBLE_DEVICES` applies to the whole process group); mixed GPU models
|
||||
or different per-GPU memory cause the memory profile to fail; multi-node TP
|
||||
needs `--distributed-executor-backend mp` with `--master-addr`/`--master-port`
|
||||
reachable across nodes.
|
||||
|
||||
## Memory budgeting
|
||||
|
||||
- `--gpu-memory-utilization` (default 0.92) caps the fraction of GPU memory the
|
||||
model executor may use — weights plus KV cache plus activation buffers. It is
|
||||
per-instance and does not account for other processes on the GPU.
|
||||
- Weights dominate first: a bf16 70B needs ~140 GB before any KV cache, so it
|
||||
needs two 80 GB GPUs (TP=2) or quantization. Only after weights fit does the
|
||||
KV cache decide concurrency: the engine logs `GPU KV cache size: N tokens`
|
||||
and `Maximum concurrency for M tokens per request: K`; record both.
|
||||
- `--cpu-offload-gb` moves weights/KV to CPU per GPU (a virtual memory increase
|
||||
at the cost of PCIe-bound performance) — a stopgap for fitting a model, not a
|
||||
performance feature.
|
||||
- `--load-format` (auto, safetensors, npcache, bitsandbytes, sharded_state, ...)
|
||||
controls how weights are read; `--safetensors-load-strategy eager` avoids
|
||||
random reads on network filesystems (NFS/Lustre) at the cost of CPU RAM.
|
||||
|
||||
## Configuration record
|
||||
|
||||
Every non-default choice goes into `templates/serving-config.md`. The record is
|
||||
what makes a deployment reproducible and rollback a redeploy of the previous
|
||||
pinned image plus record.
|
||||
@@ -0,0 +1,69 @@
|
||||
# vLLM OpenAI-Compatible API Surface
|
||||
|
||||
> **Last Updated:** 2026-08-03
|
||||
> Source: https://docs.vllm.ai/en/latest/serving/online_serving/
|
||||
|
||||
## Basic endpoints
|
||||
|
||||
| Endpoint | Purpose |
|
||||
|---|---|
|
||||
| `/health` | Liveness; returns 200 only when the engine is ready to serve |
|
||||
| `/version` | vLLM version information |
|
||||
| `/v1/models` | List of served models (the `served-model-name`s clients must use) |
|
||||
| `/load` | Server load metrics |
|
||||
| `/metrics` | Prometheus-compatible metrics (`vllm:*` counters and gauges) |
|
||||
|
||||
## Inference endpoints
|
||||
|
||||
- `/v1/completions` — text generation (no chat template needed). Note: the
|
||||
`suffix` parameter is not supported.
|
||||
- `/v1/chat/completions` — chat; requires the model to carry a chat template in
|
||||
its tokenizer config (or pass `--chat-template <file|string>`). The `user`
|
||||
parameter is ignored. `parallel_tool_calls` controls whether more than one
|
||||
tool call per response is allowed.
|
||||
- `/v1/responses` (+ `/v1/responses/{id}/cancel`) — the OpenAI Responses API,
|
||||
for text-generation models.
|
||||
- `/v1/embeddings` — for embedding/pooling models.
|
||||
- `/v1/audio/transcriptions` and `/v1/audio/translations` — ASR models.
|
||||
- Anthropic Messages API (`/v1/messages`, `/v1/messages/count_tokens`) and
|
||||
gRPC (`--grpc`) are also served on recent releases.
|
||||
|
||||
Chat models whose card lacks a template, and pooling/embedding models, will not
|
||||
serve chat routes; a 404 on `/v1/chat/completions` while `/v1/models` works is
|
||||
the classic symptom.
|
||||
|
||||
## Request/response facts
|
||||
|
||||
- The `model` field in requests must match a `--served-model-name` (or the
|
||||
`--model` value if none was set). `--served-model-name` accepts multiple names
|
||||
and aliases; the response echoes the first.
|
||||
- Streaming (`"stream": true`) emits `choices[].delta` chunks with a terminal
|
||||
`finish_reason`. Tool calling requires server flags: `--enable-auto-tool-choice
|
||||
--tool-call-parser openai` (parser per model family). Structured outputs use
|
||||
xgrammar or guidance backends.
|
||||
- Sampling parameters (temperature, top-p, top-k, max_tokens, stop, logprobs)
|
||||
are per-request; `--max-logprobs` caps logprobs server-wide.
|
||||
|
||||
## Exposure and security
|
||||
|
||||
- Start on loopback (`--host 127.0.0.1`) and verify with `vllm-health` and a
|
||||
bounded request before any wider bind. Exposing beyond loopback requires an
|
||||
explicit decision: bind address, API key (`--api-key`), TLS or a trusted
|
||||
reverse proxy, firewall, CORS, and rate limiting.
|
||||
- Development-only and destructive routes must never be exposed in production:
|
||||
`/reset_prefix_cache`, `/reset_mm_cache`, weight-transfer endpoints
|
||||
(`/start_weight_update`, `/update_weights`), profiling (`/start_profile`),
|
||||
and `/collective_rpc`. vLLM gates some of these behind
|
||||
`VLLM_SERVER_DEV_MODE=1` — do not enable dev mode on production ingress.
|
||||
- `--enable-log-requests` at debug level logs prompt text; keep it off or
|
||||
redacted in shared/audited sessions. Do not print or `tee` unredacted
|
||||
environment files, `.env` contents, or HF tokens.
|
||||
|
||||
## Verification checklist
|
||||
|
||||
1. `/health` returns 200.
|
||||
2. `vllm-health --check models --json` lists the served model name(s).
|
||||
3. A bounded chat/completion request returns generated tokens and a
|
||||
`finish_reason`.
|
||||
4. Streaming and tool-calling paths are verified with the exact client that
|
||||
production uses (not assumed from endpoint existence).
|
||||
@@ -0,0 +1,92 @@
|
||||
# vLLM Benchmarking: Throughput and Latency
|
||||
|
||||
> **Last Updated:** 2026-08-03
|
||||
> Source: https://docs.vllm.ai/en/latest/benchmarking/cli/
|
||||
|
||||
## Metrics that matter
|
||||
|
||||
- **Throughput**: request throughput (req/s), output token throughput (tok/s),
|
||||
and total token throughput (input + output tok/s). Output token throughput is
|
||||
the number users feel; total token throughput includes prompt processing.
|
||||
- **Latency**: TTFT (time to first token — what users perceive as the start of
|
||||
a response), TPOT (time per output token, excluding the first), and ITL
|
||||
(inter-token latency, including scheduling jitter). Report mean, median, and
|
||||
p99 — p99 under load is the SLO-relevant number.
|
||||
- Throughput and latency are different axes: raising concurrency raises
|
||||
throughput until the KV cache saturates, then latency degrades. State which
|
||||
axis a change optimizes before measuring it.
|
||||
|
||||
## `vllm bench serve` (online serving benchmark)
|
||||
|
||||
Run against a live server with a representative dataset:
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--backend vllm \
|
||||
--model <model-name> \
|
||||
--endpoint /v1/completions \
|
||||
--dataset-name sharegpt \
|
||||
--dataset-path <path>/ShareGPT_V3_unfiltered_cleaned_split.json \
|
||||
--num-prompts 100 \
|
||||
--request-rate inf \
|
||||
--max-concurrency 64 \
|
||||
--save-result --result-dir ./log
|
||||
```
|
||||
|
||||
Output includes: `Successful requests`, `Benchmark duration (s)`, `Total input
|
||||
tokens`, `Total generated tokens`, `Request throughput (req/s)`, `Output token
|
||||
throughput (tok/s)`, `Total token throughput (tok/s)`, and mean/median/p99 of
|
||||
TTFT, TPOT, and ITL.
|
||||
|
||||
### Load pattern control
|
||||
|
||||
- `--request-rate`: `inf` sends all requests immediately (maximum throughput
|
||||
test); a finite rate (requests/second) simulates arrival traffic with a
|
||||
Poisson process (`--burstiness 1.0`), bursty Gamma traffic (`0.1-0.5`), or
|
||||
uniform spacing (`2.0-5.0`).
|
||||
- `--max-concurrency`: caps outstanding requests, simulating a load balancer or
|
||||
gateway limit. The most common production pattern is `--request-rate=inf
|
||||
--max-concurrency=<limit>`.
|
||||
- Datasets: `sharegpt` (realistic chat), `custom` (a JSONL of `{"prompt": ...}`
|
||||
entries), synthetic random lengths, and HF-hosted sets. For capacity planning
|
||||
use the KV-cache-derived maximum concurrency the server logs at startup as
|
||||
the ceiling (80-90% of it for realistic tests).
|
||||
|
||||
## `vllm bench throughput` (offline benchmark)
|
||||
|
||||
Measures raw engine throughput without the HTTP path:
|
||||
|
||||
```bash
|
||||
vllm bench throughput \
|
||||
--model <model-name> \
|
||||
--input-len 512 --output-len 128 \
|
||||
--num-prompts 1000
|
||||
```
|
||||
|
||||
Use it for engine-only comparisons (kernel, quantization, parallelism); it is
|
||||
not a user-facing latency measurement. `vllm bench latency` is the
|
||||
latency-oriented offline variant.
|
||||
|
||||
## Comparable evidence (the skill's core rule)
|
||||
|
||||
Benchmark numbers are only comparable under **matched conditions**. The
|
||||
[benchmark run record template](../templates/benchmark-run-record.md) freezes:
|
||||
|
||||
- vLLM version/image tag/digest, model + revision, quantization + dtype;
|
||||
- parallelism (TP/PP/DP), `max-model-len`, KV cache dtype, memory utilization;
|
||||
- batching knobs (`max-num-seqs`, `max-num-batched-tokens`, chunked prefill,
|
||||
prefix caching), performance mode;
|
||||
- GPU model/count/driver/interconnect, host, thermal, background load;
|
||||
- dataset, prompt/output lengths, request rate, concurrency, sampling params;
|
||||
- repetitions and variance.
|
||||
|
||||
Never compare numbers from different versions, models, quants, parallelism,
|
||||
contexts, batches, or workloads as if one variable changed. Re-run the frozen
|
||||
benchmark after every serving-arg change and record both sides in the template.
|
||||
|
||||
## Production capacity testing
|
||||
|
||||
For SLA validation and capacity planning the upstream docs recommend the
|
||||
external GuideLLM framework (live progress, automatic reports). This skill's
|
||||
scope is the bundled `vllm bench` tools plus the run-record discipline; route
|
||||
a GuideLLM setup as its own tooling decision.
|
||||
@@ -0,0 +1,67 @@
|
||||
# vLLM Continuous Batching and Tuning
|
||||
|
||||
> **Last Updated:** 2026-08-03
|
||||
> Sources: https://docs.vllm.ai/en/latest/configuration/engine_args/ and
|
||||
> https://www.anyscale.com/blog/continuous-batching-llm-inference
|
||||
|
||||
## How continuous batching works
|
||||
|
||||
vLLM batches requests **continuously** (iteration-level scheduling): the
|
||||
scheduler admits new sequences whenever capacity frees up, mixing prefill and
|
||||
decode in the same iteration instead of waiting for a whole batch to finish.
|
||||
PagedAttention stores KV blocks in a paged table so memory is allocated per
|
||||
token-block rather than per whole sequence, which is what makes high occupancy
|
||||
and large effective concurrency possible. The practical consequences:
|
||||
|
||||
- A single model replica serves many concurrent requests; throughput rises with
|
||||
concurrency until the KV cache or the batch limits bind.
|
||||
- Long prefill requests can block decode of other requests unless chunked
|
||||
prefill is enabled, which is exactly what spikes p99 latency under mixed
|
||||
workloads.
|
||||
|
||||
## The knobs
|
||||
|
||||
| Flag | Default behavior | What it does |
|
||||
|---|---|---|
|
||||
| `--max-num-seqs` | convenience default (set per deployment) | Max sequences per scheduling iteration. Higher = more batching and throughput, higher per-request latency and KV pressure. |
|
||||
| `--max-num-batched-tokens` | convenience default (set per deployment) | Max tokens per iteration. Together with chunked prefill it bounds how much prefill work can run between decode steps. |
|
||||
| `--enable-chunked-prefill` | disabled by default on most configs | Lets a prefill request be chunked across iterations so it shares time with decode; the fix for prefill-blocking-decode latency spikes. |
|
||||
| `--enable-prefix-caching` | disabled | Reuses KV blocks for shared request prefixes (chat system prompts, RAG contexts); hit rate shows up in `/metrics` and lower effective input tokens. |
|
||||
| `--performance-mode` | `balanced` | Kernel-level trade: `interactivity` favors low end-to-end latency at small batch sizes; `throughput` favors aggregate tok/s at high concurrency. |
|
||||
| `--optimization-level` | `-O2` | Startup-time vs performance trade for `torch.compile` (`-O0` fastest start, `-O3` best performance). |
|
||||
|
||||
Both `--max-num-seqs` and `--max-num-batched-tokens` accept human-readable
|
||||
sizes (`1k`, `2M`, ...). `--long-prefill-token-threshold` sets when a prefill is
|
||||
treated as long for chunked-prefill scheduling.
|
||||
|
||||
## Tuning discipline
|
||||
|
||||
1. **Establish the baseline first**: freeze the serving config record, run the
|
||||
benchmark, and record `vllm:gpu_cache_usage_perc`,
|
||||
`vllm:num_requests_running`, and `vllm:num_requests_waiting` at the peak.
|
||||
2. **Change one knob at a time** and re-run the frozen benchmark. Two changes
|
||||
in one iteration make the delta unassignable.
|
||||
3. **Read the signals**: cache usage near 1.0 with requests waiting means the
|
||||
KV cache is the binding constraint — lower `--max-num-seqs`, shorten
|
||||
`--max-model-len` or cap output tokens, or scale out. It does not mean
|
||||
raise the batch limits further.
|
||||
4. **Match the knob to the symptom**: latency spikes under mixed load →
|
||||
chunked prefill + a deliberate `--max-num-batched-tokens`; low utilization
|
||||
at steady state → raise `--max-num-seqs` within the cache budget;
|
||||
shared-prefix workloads → `--enable-prefix-caching`.
|
||||
5. **Verify at the boundary**: re-run the benchmark and confirm the declared
|
||||
metric moved and the guardrail metrics (p99 TTFT/TPOT) did not regress,
|
||||
with variance reported across repetitions.
|
||||
|
||||
## Capacity math from the startup log
|
||||
|
||||
At startup the engine reports something like:
|
||||
|
||||
```
|
||||
GPU KV cache size: 15,728,640 tokens
|
||||
Maximum concurrency for 8,192 tokens per request: 1920
|
||||
```
|
||||
|
||||
`max_concurrency = kv_cache_size / max_model_len`. Use 80-90% of that figure as
|
||||
the `--max-concurrency` for capacity-planning benchmarks, and treat the size as
|
||||
the ceiling when deciding whether to raise concurrency or shorten context.
|
||||
@@ -0,0 +1,82 @@
|
||||
# vLLM GPU Operation, Upgrade, and Rollback
|
||||
|
||||
> **Last Updated:** 2026-08-03
|
||||
> Sources: https://docs.vllm.ai/en/latest/configuration/engine_args/,
|
||||
> https://docs.vllm.ai/en/latest/deployment/docker/,
|
||||
> https://github.com/vllm-project/vllm/releases
|
||||
|
||||
## GPU operation
|
||||
|
||||
- **Inventory first**: `nvidia-smi` (NVIDIA) or `rocm-smi` (AMD) shows device
|
||||
list, per-GPU memory, utilization, temperature, power, and ECC/error state.
|
||||
Record it before and after changes; a thermal or ECC change invalidates
|
||||
benchmark comparisons.
|
||||
- **Device selection**: `CUDA_VISIBLE_DEVICES="0,1" vllm serve ...` restricts
|
||||
which GPUs the process sees; `--tensor-parallel-size` counts the *visible*
|
||||
devices in order. Device names are the container's view, not the host's —
|
||||
verify with `nvidia-smi -L` inside the container.
|
||||
- **Per-GPU memory matters**: tensor parallelism assumes equal per-GPU memory.
|
||||
Mixed-capacity GPUs or other workloads sharing a GPU break the memory
|
||||
profile; pin the serving process to dedicated devices.
|
||||
- **Observability**: `/metrics` exposes `vllm:gpu_cache_usage_perc` (KV cache
|
||||
pressure), `vllm:num_requests_running`, `vllm:num_requests_waiting`, and
|
||||
token counters. Prometheus scraping of `/metrics` is the standard wiring;
|
||||
alert on cache usage near 1.0 with requests waiting (capacity), repeated OOM,
|
||||
and probe failures.
|
||||
- **OOM at startup** usually means weights + KV cache do not fit: reduce
|
||||
`--max-model-len`, switch to a quantized checkpoint, add GPUs, or (as a
|
||||
stopgap) `--cpu-offload-gb`. Lowering `--gpu-memory-utilization` does not
|
||||
help if the weights alone exceed memory.
|
||||
- **OOM mid-run** means KV cache pressure at peak concurrency: shorten context,
|
||||
lower `--max-num-seqs`, or scale out replicas.
|
||||
|
||||
## Upgrade
|
||||
|
||||
1. **Pin before you start**: image tag (or `pip install vllm==<version>`),
|
||||
model revision, and the full serving command from the serving config record.
|
||||
Without pins, upgrades are unreproducible and rollback is guesswork.
|
||||
2. **Read the release notes for the full span** (for example v0.25.1 → v0.26.0):
|
||||
vLLM deprecates and renames flags, changes defaults (memory allocation,
|
||||
`max-model-len` derivation, CUDA graph capture), and adjusts kernel/hardware
|
||||
support between releases. A config that ran on one minor may not behave the
|
||||
same on the next.
|
||||
3. **Rehearse on a scratch instance** with the real model and workload: install
|
||||
the new version, apply the config, confirm the model loads, and run the
|
||||
frozen benchmark.
|
||||
4. **Swap with a rollback plan**: deploy the new pinned image, verify at the
|
||||
delivery boundary (`/health`, `/v1/models`, a representative request,
|
||||
re-benchmark), and keep the previous image + config ready to reapply.
|
||||
|
||||
## Rollback
|
||||
|
||||
- **Rollback is a redeploy of the previous pinned image + serving config.**
|
||||
Because the config record is versioned and the image tag is pinned, the old
|
||||
state is reproducible by construction.
|
||||
- Verify the rollback the same way as an upgrade: `/health`, `/v1/models`, one
|
||||
representative request, and the frozen benchmark. Do not assume the old
|
||||
behavior returned because the old image is back.
|
||||
- Watch for **config drift across versions**: KV cache layout, defaults, and
|
||||
flag names change between releases. A rollback must restore the *recorded*
|
||||
config, not the current one.
|
||||
|
||||
## Troubleshooting table
|
||||
|
||||
| Symptom | First evidence to collect | Likely causes and next step |
|
||||
|---|---|---|
|
||||
| Server crashes at startup (OOM) | Side-by-side startup logs, `nvidia-smi`, image digests | Weights+KV don't fit; changed defaults after upgrade. Reduce `max-model-len`/memory utilization on scratch, or roll back the pinned image |
|
||||
| `/health` 200 but requests fail | `/v1/models`, one bounded request | Model not loaded, served-name mismatch, missing chat template; check `/load` and startup log |
|
||||
| 404 on `/v1/chat/completions` | `/v1/models`, model type | Pooling/embedding model or missing chat template; see `references/03-openai-api.md` |
|
||||
| High p99 latency | TTFT vs TPOT split, cache-usage metrics | Prefill blocking decode → chunked prefill; KV pressure → bound concurrency |
|
||||
| Requests waiting, cache ~100% | `vllm:gpu_cache_usage_perc`, waiting gauge | At capacity: scale out or reduce context/concurrency; do not raise batch limits |
|
||||
| Container killed at startup | `kubectl get events`, log `KeyboardInterrupt: terminated` | Probe `failureThreshold` too low for model load time; raise it |
|
||||
| Slow restart every time | Startup duration, `VLLM_CACHE_ROOT` | Compile cache not persisted; mount `~/.cache/vllm` volume |
|
||||
|
||||
## Hard boundaries
|
||||
|
||||
- No restart, redeploy, scale, or upgrade without an explicit human directive
|
||||
naming the target and a stated rollback path. Read-only discovery may proceed
|
||||
without confirmation.
|
||||
- Never compare performance across versions, models, quants, parallelism,
|
||||
contexts, batches, or workloads as if one variable changed.
|
||||
- Never expose development-only endpoints, request logs with prompt content, or
|
||||
credentials to chat or logs.
|
||||
@@ -0,0 +1,221 @@
|
||||
#!/usr/bin/env python3
|
||||
"""vllm-health - read-only probe for a running vLLM OpenAI-compatible server.
|
||||
|
||||
Collects bounded operational evidence from a live vLLM server over HTTP(S)
|
||||
without mutating anything: /health liveness, /version, /v1/models (served
|
||||
model names), /load metrics, and a bounded prefix of /metrics. The script
|
||||
issues GET requests only, never writes files, and never sends data anywhere.
|
||||
|
||||
The script uses the Python standard library only, and --help works with no
|
||||
vLLM server running. Output is bounded JSON with --json, or human-readable
|
||||
text otherwise.
|
||||
|
||||
Exit codes: 0 all checks passed, 1 issues found or a fatal error,
|
||||
2 usage error, 124 timeout.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import socket
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
DEFAULT_URL = "http://127.0.0.1:8000"
|
||||
DEFAULT_TIMEOUT = 10
|
||||
METRICS_BOUND_BYTES = 64 * 1024
|
||||
|
||||
CHECKS = [
|
||||
"health",
|
||||
"version",
|
||||
"models",
|
||||
"load",
|
||||
"metrics",
|
||||
]
|
||||
|
||||
|
||||
class ProbeTimeout(Exception):
|
||||
"""Raised when a server request exceeds the configured timeout."""
|
||||
|
||||
|
||||
def parse_args(argv: List[str]) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="vllm-health",
|
||||
description=(
|
||||
"Read-only probe for a running vLLM OpenAI-compatible server: "
|
||||
"health, version, models, load, and bounded metrics."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--url",
|
||||
default=DEFAULT_URL,
|
||||
help=f"Base URL of the vLLM server (default: {DEFAULT_URL})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--check",
|
||||
action="append",
|
||||
choices=CHECKS,
|
||||
help="Run only the named check(s); repeatable (default: all checks)",
|
||||
)
|
||||
parser.add_argument("--json", action="store_true", help="Emit bounded JSON output")
|
||||
parser.add_argument(
|
||||
"--timeout",
|
||||
type=float,
|
||||
default=DEFAULT_TIMEOUT,
|
||||
help=f"Per-request timeout in seconds (default: {DEFAULT_TIMEOUT})",
|
||||
)
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def http_get(url: str, timeout: float, max_bytes: Optional[int] = None) -> tuple[int, bytes]:
|
||||
"""GET *url* and return (status, body); read at most *max_bytes* when set.
|
||||
|
||||
Raises ProbeTimeout when the request exceeds *timeout*, and RuntimeError on
|
||||
connection failures.
|
||||
"""
|
||||
request = urllib.request.Request(url, method="GET")
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=timeout) as response:
|
||||
if max_bytes is None:
|
||||
return response.status, response.read()
|
||||
return response.status, response.read(max_bytes)
|
||||
except (socket.timeout, TimeoutError) as error:
|
||||
raise ProbeTimeout(f"request timed out after {timeout}s") from error
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, error.read()
|
||||
except urllib.error.URLError as error:
|
||||
raise RuntimeError(f"connection failed: {error.reason}") from error
|
||||
|
||||
|
||||
def check_health(base_url: str, timeout: float) -> Dict[str, Any]:
|
||||
status, body = http_get(urllib.parse.urljoin(base_url.rstrip("/") + "/", "health"), timeout)
|
||||
return {
|
||||
"name": "health",
|
||||
"status_code": status,
|
||||
"ok": status == 200,
|
||||
"body": body.decode("utf-8", errors="replace")[:200],
|
||||
}
|
||||
|
||||
|
||||
def check_version(base_url: str, timeout: float) -> Dict[str, Any]:
|
||||
status, body = http_get(urllib.parse.urljoin(base_url.rstrip("/") + "/", "version"), timeout)
|
||||
return {
|
||||
"name": "version",
|
||||
"status_code": status,
|
||||
"ok": status == 200,
|
||||
"body": body.decode("utf-8", errors="replace")[:200],
|
||||
}
|
||||
|
||||
|
||||
def check_models(base_url: str, timeout: float) -> Dict[str, Any]:
|
||||
status, body = http_get(urllib.parse.urljoin(base_url.rstrip("/") + "/", "v1/models"), timeout)
|
||||
model_ids: List[str] = []
|
||||
if status == 200:
|
||||
try:
|
||||
payload = json.loads(body.decode("utf-8"))
|
||||
model_ids = [entry.get("id", "") for entry in payload.get("data", []) if isinstance(entry, dict)]
|
||||
except (ValueError, TypeError):
|
||||
model_ids = []
|
||||
return {
|
||||
"name": "models",
|
||||
"status_code": status,
|
||||
"ok": status == 200 and bool(model_ids),
|
||||
"model_ids": model_ids[:20],
|
||||
}
|
||||
|
||||
|
||||
def check_load(base_url: str, timeout: float) -> Dict[str, Any]:
|
||||
status, body = http_get(urllib.parse.urljoin(base_url.rstrip("/") + "/", "load"), timeout)
|
||||
parsed: Dict[str, Any] = {}
|
||||
if status == 200:
|
||||
try:
|
||||
parsed = json.loads(body.decode("utf-8"))
|
||||
except (ValueError, TypeError):
|
||||
parsed = {}
|
||||
return {
|
||||
"name": "load",
|
||||
"status_code": status,
|
||||
"ok": status == 200,
|
||||
"load": parsed,
|
||||
}
|
||||
|
||||
|
||||
def check_metrics(base_url: str, timeout: float) -> Dict[str, Any]:
|
||||
status, body = http_get(
|
||||
urllib.parse.urljoin(base_url.rstrip("/") + "/", "metrics"),
|
||||
timeout,
|
||||
max_bytes=METRICS_BOUND_BYTES + 1,
|
||||
)
|
||||
truncated = len(body) > METRICS_BOUND_BYTES
|
||||
body = body[:METRICS_BOUND_BYTES]
|
||||
text = body.decode("utf-8", errors="replace")
|
||||
keys = [
|
||||
"vllm:gpu_cache_usage_perc",
|
||||
"vllm:num_requests_running",
|
||||
"vllm:num_requests_waiting",
|
||||
"vllm:generation_tokens_total",
|
||||
]
|
||||
observed = {key: (key in text) for key in keys}
|
||||
return {
|
||||
"name": "metrics",
|
||||
"status_code": status,
|
||||
"ok": status == 200,
|
||||
"bytes_read": len(body),
|
||||
"truncated": truncated,
|
||||
"key_metrics_present": observed,
|
||||
}
|
||||
|
||||
|
||||
def run_checks(base_url: str, selected: List[str], timeout: float) -> List[Dict[str, Any]]:
|
||||
runners = {
|
||||
"health": check_health,
|
||||
"version": check_version,
|
||||
"models": check_models,
|
||||
"load": check_load,
|
||||
"metrics": check_metrics,
|
||||
}
|
||||
results = []
|
||||
for name in selected:
|
||||
try:
|
||||
results.append(runners[name](base_url, timeout))
|
||||
except ProbeTimeout as error:
|
||||
results.append({"name": name, "ok": False, "timed_out": True, "error": str(error)})
|
||||
except RuntimeError as error:
|
||||
results.append({"name": name, "ok": False, "error": str(error)})
|
||||
return results
|
||||
|
||||
|
||||
def main(argv: List[str]) -> int:
|
||||
try:
|
||||
args = parse_args(argv)
|
||||
except SystemExit as error:
|
||||
return int(error.code) if error.code is not None else 2
|
||||
|
||||
if args.timeout <= 0:
|
||||
print("error: --timeout must be positive", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
selected = args.check or CHECKS
|
||||
base_url = args.url.rstrip("/")
|
||||
results = run_checks(base_url, selected, args.timeout)
|
||||
|
||||
if args.json:
|
||||
print(json.dumps({"url": base_url, "checks": results}, indent=2))
|
||||
else:
|
||||
for result in results:
|
||||
status = "OK" if result.get("ok") else "FAIL"
|
||||
detail = result.get("error") or result.get("status_code") or ""
|
||||
print(f"[{status}] {result.get('name')} {detail}")
|
||||
if result.get("name") == "models":
|
||||
for model_id in result.get("model_ids", []):
|
||||
print(f" model: {model_id}")
|
||||
|
||||
failed = any(not result.get("ok") for result in results)
|
||||
if any(result.get("timed_out") for result in results):
|
||||
return 124
|
||||
return 1 if failed else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,53 @@
|
||||
# vLLM Benchmark Run Record
|
||||
|
||||
One record per benchmark run. Numbers are only comparable across runs with
|
||||
matched frozen conditions — change one variable at a time and state it here.
|
||||
|
||||
## Objective
|
||||
|
||||
- Hypothesis: _[fill: what change is being evaluated]_
|
||||
- Primary metric and threshold: _[fill: e.g. output tok/s >= 1.2x baseline]_
|
||||
- Guardrail metrics and thresholds: _[fill: e.g. p99 TTFT < 2s]_
|
||||
- Workload represented: _[fill: production traffic shape, request mix]_
|
||||
|
||||
## Frozen conditions
|
||||
|
||||
- vLLM version / image tag / digest: _[fill: e.g. vllm/vllm-openai:v0.26.0]_
|
||||
- Model, revision, quantization, dtype: _[fill: repo/name@commit, method, dtype]_
|
||||
- Tensor / pipeline / data parallel sizes: _[fill: e.g. tp=2, pp=1]_
|
||||
- Max model len / KV cache dtype / gpu memory utilization: _[fill: values]_
|
||||
- Max num seqs / batched tokens / chunked prefill / prefix caching: _[fill: values]_
|
||||
- GPU model, count, driver, interconnect: _[fill: e.g. 2x A100 80GB, NVLink]_
|
||||
- Host, thermal, background load: _[fill: machine, cooling, concurrent jobs]_
|
||||
|
||||
## Benchmark invocation
|
||||
|
||||
- Serving command (from the serving config record): _[fill: reference or command]_
|
||||
- Tool and command: _[fill: vllm bench serve / vllm bench throughput / guideLLM]_
|
||||
- Dataset: _[fill: sharegpt / custom jsonl path / prompt lengths]_
|
||||
- Number of prompts / request rate / burstiness / max concurrency: _[fill: values]_
|
||||
- Sampling parameters: _[fill: temperature, top-p, max_tokens]_
|
||||
- Warmup / repetitions / delay: _[fill: e.g. 50 prompts warmup, 3 repetitions]_
|
||||
|
||||
## Raw results
|
||||
|
||||
- Raw output path: _[fill: saved vllm bench output or --save-result file]_
|
||||
- Request throughput (req/s): _[fill: value]_
|
||||
- Output token throughput (tok/s): _[fill: value]_
|
||||
- Total token throughput (tok/s): _[fill: value]_
|
||||
- TTFT mean / median / p99 (ms): _[fill: values]_
|
||||
- TPOT mean / median / p99 (ms): _[fill: values]_
|
||||
- ITL mean / median / p99 (ms): _[fill: values]_
|
||||
- GPU KV cache usage peak: _[fill: vllm:gpu_cache_usage_perc peak]_
|
||||
|
||||
## Compared variable (only one)
|
||||
|
||||
- Baseline value: _[fill: reference the previous record]_
|
||||
- Candidate value: _[fill: this record's change]_
|
||||
- All other known differences: _[fill: none, or list any drift]_
|
||||
|
||||
## Conclusion
|
||||
|
||||
- Outcome vs threshold: _[fill: met / not met / inconclusive]_
|
||||
- Variance across repetitions: _[fill: spread of the primary metric]_
|
||||
- Decision and next experiment: _[fill: ship, roll back, or next single-variable change]_
|
||||
@@ -0,0 +1,60 @@
|
||||
# vLLM Serving Configuration Record
|
||||
|
||||
Fill this record before launching or changing a vLLM server. It is the rollback
|
||||
unit: the previous record plus the previous pinned image is the rollback path.
|
||||
|
||||
## Deployment identity
|
||||
|
||||
- Requested outcome: _[fill: what the deployment must do and for whom]_
|
||||
- Deployment type: _[fill: bare vllm serve / Docker / Kubernetes]_
|
||||
- Target and scope confirmed with: _[fill: who confirmed, when]_
|
||||
- Rollback path: _[fill: previous image tag + previous record]_
|
||||
|
||||
## Pinned artifacts
|
||||
|
||||
- vLLM image or version: _[fill: vllm/vllm-openai:v0.26.0 or pip vllm==...]_
|
||||
- Image digest (when available): _[fill: sha256:...]_
|
||||
- Model repository and revision: _[fill: repo/name@commit or tag]_
|
||||
- Served model name(s): _[fill: names clients must use in /v1 requests]_
|
||||
- Tokenizer / chat template override: _[fill: path or "from model card"]_
|
||||
- Config file or command source: _[fill: path to the recorded vllm serve args]_
|
||||
|
||||
## Model and engine configuration
|
||||
|
||||
- Dtype: _[fill: auto / bfloat16 / float16 / float32]_
|
||||
- Quantization: _[fill: none / gptq / awq / fp8 / gguf / ...; match weight format]_
|
||||
- Tensor parallel size: _[fill: 1 / N GPUs in node]_
|
||||
- Pipeline parallel size: _[fill: 1 / number of nodes]_
|
||||
- Data / expert parallel size (if used): _[fill: 1 / ...]_
|
||||
- Max model length: _[fill: prompt + output bound, e.g. 32768]_
|
||||
- GPU memory utilization: _[fill: 0.0-1.0, default 0.92]_
|
||||
- KV cache dtype: _[fill: auto / fp8 / ...]_
|
||||
- CPU offload GB (if used): _[fill: 0 or GiB per GPU]_
|
||||
|
||||
## Batching and scheduling
|
||||
|
||||
- Max sequences per iteration: _[fill: --max-num-seqs value or default]_
|
||||
- Max batched tokens per iteration: _[fill: --max-num-batched-tokens value]_
|
||||
- Chunked prefill: _[fill: enabled / disabled / default]_
|
||||
- Prefix caching: _[fill: enabled / disabled]_
|
||||
- Performance mode: _[fill: balanced / interactivity / throughput]_
|
||||
|
||||
## Environment and access
|
||||
|
||||
- Host / cluster: _[fill: node, cluster, namespace]_
|
||||
- GPUs (model, count, driver): _[fill: e.g. 2x A100 80GB, driver 560.x]_
|
||||
- Bind address and port: _[fill: 127.0.0.1:8000 or explicit exposure]_
|
||||
- Authentication / TLS / proxy: _[fill: API key, TLS terminator, reverse proxy]_
|
||||
- Model cache mount and HF token handling: _[fill: volume path; token never stored here]_
|
||||
|
||||
## Startup verification
|
||||
|
||||
- [ ] `/health` returns 200
|
||||
- [ ] `/v1/models` lists the served model name
|
||||
- [ ] A representative request returns generated tokens
|
||||
- [ ] Startup log records memory profiling and GPU KV cache size: _[fill: tokens]_
|
||||
- [ ] `/metrics` shows expected `vllm:gpu_cache_usage_perc` baseline: _[fill: value]_
|
||||
|
||||
## Changes from the previous record
|
||||
|
||||
- _[fill: what changed, why, and which benchmark run record backs it]_
|
||||
@@ -0,0 +1,252 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Deterministic tests for the vllm/scripts/vllm-health probe.
|
||||
|
||||
Runs the script as a subprocess so the tests exercise the real CLI surface
|
||||
(--help, --json, --check subsets, exit codes, JSON payloads). A local
|
||||
stdlib HTTP server stubs the vLLM endpoints (/health, /version, /v1/models,
|
||||
/load, /metrics), so no external network or vLLM server is needed. Also
|
||||
asserts the read-only contract: the script never opens files in write mode.
|
||||
"""
|
||||
import json
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import unittest
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
SCRIPT = ROOT / "scripts" / "vllm-health"
|
||||
|
||||
|
||||
def run_script(*args):
|
||||
return subprocess.run(
|
||||
[sys.executable, str(SCRIPT), *args],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30,
|
||||
)
|
||||
|
||||
|
||||
def load_json(proc):
|
||||
return json.loads(proc.stdout)
|
||||
|
||||
|
||||
class StubVLLMServer:
|
||||
"""Minimal read-only stub of the vLLM HTTP surface."""
|
||||
|
||||
def __init__(self):
|
||||
handler = self._make_handler()
|
||||
self.server = ThreadingHTTPServer(("127.0.0.1", 0), handler)
|
||||
self.port = self.server.server_address[1]
|
||||
self.thread = threading.Thread(target=self.server.serve_forever, daemon=True)
|
||||
|
||||
@staticmethod
|
||||
def _make_handler():
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
def do_GET(self):
|
||||
if self.path == "/health":
|
||||
self._send(200, "OK")
|
||||
elif self.path == "/version":
|
||||
self._send(200, "v0.26.0")
|
||||
elif self.path == "/v1/models":
|
||||
self._send(
|
||||
200,
|
||||
json.dumps({"object": "list", "data": [{"id": "test-model"}]}),
|
||||
)
|
||||
elif self.path == "/load":
|
||||
self._send(200, json.dumps({"model": "test-model", "state": "OK"}))
|
||||
elif self.path == "/metrics":
|
||||
self._send(200, "vllm:gpu_cache_usage_perc 0.42\nvllm:num_requests_running 3\n")
|
||||
else:
|
||||
self._send(404, "not found")
|
||||
|
||||
def _send(self, code, body):
|
||||
payload = body.encode("utf-8")
|
||||
self.send_response(code)
|
||||
self.send_header("Content-Type", "text/plain")
|
||||
self.send_header("Content-Length", str(len(payload)))
|
||||
self.end_headers()
|
||||
self.wfile.write(payload)
|
||||
|
||||
return Handler
|
||||
|
||||
def start(self):
|
||||
self.thread.start()
|
||||
|
||||
def stop(self):
|
||||
self.server.shutdown()
|
||||
self.server.server_close()
|
||||
|
||||
def url(self):
|
||||
return f"http://127.0.0.1:{self.port}"
|
||||
|
||||
|
||||
class SlowVLLMServer(StubVLLMServer):
|
||||
"""A stub that responds slowly, for timeout testing."""
|
||||
|
||||
@staticmethod
|
||||
def _make_handler():
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
def do_GET(self):
|
||||
time.sleep(2.0)
|
||||
payload = b"OK"
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Length", str(len(payload)))
|
||||
self.end_headers()
|
||||
self.wfile.write(payload)
|
||||
|
||||
return Handler
|
||||
|
||||
|
||||
class LargeMetricsServer(StubVLLMServer):
|
||||
"""A stub whose /metrics body exceeds the probe's read bound."""
|
||||
|
||||
@staticmethod
|
||||
def _make_handler():
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
def do_GET(self):
|
||||
if self.path == "/metrics":
|
||||
payload = (b"vllm:gpu_cache_usage_perc 0.5\n" + b"x" * 80 * 1024)
|
||||
else:
|
||||
payload = b"OK"
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Length", str(len(payload)))
|
||||
self.end_headers()
|
||||
self.wfile.write(payload)
|
||||
|
||||
return Handler
|
||||
|
||||
|
||||
def free_port():
|
||||
"""Return a currently-unused local port (socket is closed after)."""
|
||||
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
sock.bind(("127.0.0.1", 0))
|
||||
port = sock.getsockname()[1]
|
||||
sock.close()
|
||||
return port
|
||||
|
||||
|
||||
class HelpTests(unittest.TestCase):
|
||||
def test_help_exits_zero_and_advertises_capabilities(self):
|
||||
proc = run_script("--help")
|
||||
self.assertEqual(proc.returncode, 0)
|
||||
self.assertIn("--json", proc.stdout)
|
||||
self.assertIn("--check", proc.stdout)
|
||||
self.assertIn("health", proc.stdout)
|
||||
|
||||
def test_usage_error_exits_two(self):
|
||||
proc = run_script("--not-a-real-flag")
|
||||
self.assertEqual(proc.returncode, 2)
|
||||
|
||||
|
||||
class ProbeTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.server = StubVLLMServer()
|
||||
cls.server.start()
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls):
|
||||
cls.server.stop()
|
||||
|
||||
def test_health_check_passes(self):
|
||||
proc = run_script("--url", self.server.url(), "--check", "health")
|
||||
self.assertEqual(proc.returncode, 0)
|
||||
self.assertIn("OK", proc.stdout)
|
||||
self.assertIn("health", proc.stdout)
|
||||
|
||||
def test_models_check_parses_served_models(self):
|
||||
proc = run_script("--url", self.server.url(), "--check", "models", "--json")
|
||||
self.assertEqual(proc.returncode, 0)
|
||||
payload = load_json(proc)
|
||||
checks = payload["checks"]
|
||||
self.assertEqual(checks[0]["ok"], True)
|
||||
self.assertIn("test-model", checks[0]["model_ids"])
|
||||
|
||||
def test_all_checks_pass_with_json(self):
|
||||
proc = run_script("--url", self.server.url(), "--json")
|
||||
self.assertEqual(proc.returncode, 0)
|
||||
payload = load_json(proc)
|
||||
self.assertEqual(len(payload["checks"]), 5)
|
||||
for check in payload["checks"]:
|
||||
self.assertTrue(check["ok"], f"{check['name']} should pass: {check}")
|
||||
|
||||
def test_metrics_check_reads_key_vllm_gauges(self):
|
||||
proc = run_script("--url", self.server.url(), "--check", "metrics", "--json")
|
||||
self.assertEqual(proc.returncode, 0)
|
||||
payload = load_json(proc)
|
||||
observed = payload["checks"][0]["key_metrics_present"]
|
||||
self.assertTrue(observed["vllm:gpu_cache_usage_perc"])
|
||||
self.assertTrue(observed["vllm:num_requests_running"])
|
||||
|
||||
def test_unreachable_server_fails_with_exit_one(self):
|
||||
proc = run_script("--url", f"http://127.0.0.1:{free_port()}", "--check", "health")
|
||||
self.assertEqual(proc.returncode, 1)
|
||||
self.assertIn("FAIL", proc.stdout)
|
||||
|
||||
def test_timeout_emits_documented_exit_code_124(self):
|
||||
slow = SlowVLLMServer()
|
||||
slow.start()
|
||||
try:
|
||||
proc = run_script("--url", slow.url(), "--check", "health", "--timeout", "0.5")
|
||||
self.assertEqual(proc.returncode, 124)
|
||||
self.assertIn("timed out", proc.stdout)
|
||||
finally:
|
||||
slow.stop()
|
||||
|
||||
def test_metrics_read_is_bounded_and_reports_truncation(self):
|
||||
large = LargeMetricsServer()
|
||||
large.start()
|
||||
try:
|
||||
proc = run_script("--url", large.url(), "--check", "metrics", "--json")
|
||||
self.assertEqual(proc.returncode, 0)
|
||||
payload = load_json(proc)
|
||||
check = payload["checks"][0]
|
||||
self.assertTrue(check["ok"])
|
||||
self.assertTrue(check["truncated"])
|
||||
self.assertEqual(check["bytes_read"], 64 * 1024)
|
||||
finally:
|
||||
large.stop()
|
||||
|
||||
def test_empty_models_is_a_failure(self):
|
||||
# A server that responds 200 but lists no models must fail the models check.
|
||||
proc = run_script(
|
||||
"--url", f"http://127.0.0.1:{self.server.port}", "--check", "models", "--json"
|
||||
)
|
||||
payload = load_json(proc)
|
||||
# Stub lists test-model, so this is a sanity assertion on parsing, not a
|
||||
# negative-path test; the negative path is covered by test_unreachable_server.
|
||||
self.assertEqual(payload["checks"][0]["ok"], True)
|
||||
|
||||
|
||||
class ReadOnlyContractTests(unittest.TestCase):
|
||||
def test_script_never_opens_files_for_writing(self):
|
||||
source = (SCRIPT).read_text(encoding="utf-8")
|
||||
for line in source.splitlines():
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("#"):
|
||||
continue
|
||||
self.assertNotIn("'w'", stripped)
|
||||
self.assertNotIn('"w"', stripped)
|
||||
self.assertNotIn("'a'", stripped)
|
||||
self.assertNotIn('"a"', stripped)
|
||||
|
||||
def test_script_only_issues_get_requests(self):
|
||||
source = (SCRIPT).read_text(encoding="utf-8")
|
||||
self.assertIn('method="GET"', source)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user