feat(skill): add vLLM inference-serving skill (#247) (#267)

* feat(skill): add vLLM inference-serving skill (#247) Add a single-tool vllm skill covering Docker/Kubernetes deployment, quantization-aware model configuration (tensor parallelism, KV cache), the OpenAI-compatible API surface, throughput/latency benchmarking, continuous batching tuning, GPU operation, and upgrade/rollback. Ships a read-only vllm-health probe (stdlib-only, --json), fillable serving-config and benchmark-run-record templates, seven dated references with upstream sources, a human-facing README, tests, and a schema-v1 eval manifest with six cases covering config, benchmarking, and troubleshooting. Route ml-engineering to the new skill via a resolvable link alongside llama-cpp, add the vllm entry to the top-level README index, and regenerate the tracked catalogs. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * fix(skill): emit timeout exit 124 and bound /metrics reads in vllm-health Address the review observations on the bundled probe: requests that exceed --timeout now raise ProbeTimeout and make the tool exit 124 as documented (previously they surfaced as exit 1), and the metrics check reads at most 64 KiB of /metrics and reports truncation instead of reading the whole body. Adds tests for both behaviors. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> --------- Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-09-11 19:47:12 +03:00 · 2026-08-03 19:31:36 -04:00
parent f83d48ba53
commit 6181f1746d
19 changed files with 1495 additions and 5 deletions
@@ -579,7 +579,7 @@
        "./ml-engineering"
      ],
      "strict": false,
-      "description": "Plan and execute production ML engineering work — model training and fine-tuning (LoRA/QLoRA), evaluation and eval-set design, quantization decisions, inference deployment, and regression triage, grounded in practical engineering patterns for production ML systems. Do not use for statistical modeling and experimental design (that's the data scientist) or for operating a specific inference engine (that's a tool skill such as llama-cpp)."
+      "description": "Plan and execute production ML engineering work — model training and fine-tuning (LoRA/QLoRA), evaluation and eval-set design, quantization decisions, inference deployment, and regression triage, grounded in practical engineering patterns for production ML systems. Do not use for statistical modeling and experimental design (that's the data scientist) or for operating a specific inference engine (that's a tool skill such as llama-cpp or vllm)."
    },
    {
      "name": "neckbeard",
@@ -1103,6 +1103,15 @@
      "strict": false,
      "description": "Verify work against explicit criteria using direct, source-faithful evidence, reproducible checks, and clear verdicts. Use before declaring an artifact, implementation, or claim complete; do not use for exploratory research without pass/fail criteria."
    },
    {
      "name": "vllm",
      "source": "./",
      "skills": [
        "./vllm"
      ],
      "strict": false,
      "description": "Operate, configure, benchmark, and troubleshoot vLLM inference servers: Docker and Kubernetes deployment, quantization-aware model configuration (tensor parallelism, KV cache), OpenAI-compatible API serving, throughput and latency benchmarking, continuous batching tuning, GPU operation, and upgrade/rollback. Use when deploying or running a vLLM server (vllm serve, vllm/vllm-openai), sizing a model and its KV cache for GPUs, selecting quantization and parallelism, serving via /v1 endpoints, measuring serving throughput or latency, tuning batching, or diagnosing GPU, OOM, or startup failures in a vLLM deployment. Do not use for model training, fine-tuning, evaluation-set design, or engine-selection methodology (that is ml-engineering), or for operating the llama.cpp stack with GGUF models (that is llama-cpp); other inference engines (TGI, Ollama, Triton) are out of scope."
    },
    {
      "name": "web-accessibility",
      "source": "./",
@@ -141,6 +141,7 @@
    "./transistor",
    "./vercel-eve",
    "./verification-methodology",
    "./vllm",
    "./web-accessibility",
    "./woodpecker-ci",
    "./yc-default-alive-calculator",
@@ -505,6 +505,10 @@ Build, deploy, self-host, secure, and troubleshoot durable backend AI agents wit
 Replace completion claims with a disciplined evidence trail that shows what was checked, what passed, and what remains uncertain.
 ### [vllm](vllm/SKILL.md)
 Operate, configure, benchmark, and troubleshoot vLLM inference servers: Docker and Kubernetes deployment, quantization-aware model configuration (tensor parallelism, KV cache), the OpenAI-compatible API surface, throughput/latency benchmarking, continuous batching tuning, GPU operation, and upgrade/rollback. Ships a read-only `vllm-health` probe (`--json`), serving-config and benchmark-run-record templates, 7 dated references, tests, and 6 evals. Routes serving methodology to ml-engineering and the llama.cpp stack to llama-cpp.
 ### [web-accessibility](web-accessibility/SKILL.md)
 Design, build, and review accessible web interfaces through native semantics, keyboard and focus contracts, error recovery, adaptive UI, assistive-technology testing, and WCAG 2.2-informed evidence without mistaking automated scans for conformance.
@@ -65,7 +65,7 @@
 - [mermaid-diagrams](mermaid-diagrams/SKILL.md): Author, render, and troubleshoot Mermaid diagrams for documentation, architecture, processes, and technical communication. Use when a text-based diagram needs to stay versionable.
 - [meshcore-packet-capture](meshcore-packet-capture/SKILL.md): Capture MeshCore Companion packets via BLE, serial, or TCP.
 - [migration-engineering](migration-engineering/SKILL.md): Plan and execute safe cross-system migrations — schema, data, API, infrastructure, and service — with compatibility windows, dual-running, backfills, reconciliation, cutover, deprecation, and cleanup. Covers expand/contract, reversible and irreversible recovery paths, migration observability, correctness evidence, ownership, and customer impact. Do not use for single-technology quick fixes, tool-specific how-to guides, or migrations whose scope does not cross a system boundary; do not prescribe one migration technology or claim rollback is always possible.
- [ml-engineering](ml-engineering/SKILL.md): Plan and execute production ML engineering work — model training and fine-tuning (LoRA/QLoRA), evaluation and eval-set design, quantization decisions, inference deployment, and regression triage, grounded in practical engineering patterns for production ML systems. Do not use for statistical modeling and experimental design (that's the data scientist) or for operating a specific inference engine (that's a tool skill such as llama-cpp).
+- [ml-engineering](ml-engineering/SKILL.md): Plan and execute production ML engineering work — model training and fine-tuning (LoRA/QLoRA), evaluation and eval-set design, quantization decisions, inference deployment, and regression triage, grounded in practical engineering patterns for production ML systems. Do not use for statistical modeling and experimental design (that's the data scientist) or for operating a specific inference engine (that's a tool skill such as llama-cpp or vllm).
 - [neckbeard](bundles/neckbeard/SKILL.md): Use when asked to fix, build, refactor, review, verify, or release software and the work is non-trivial — including delivering a change request (issue, ticket, or request) from intake through planning, gates, implementation, review, verified PR, and authorized post-merge release. neckbeard routes the change through framing, discovery, design, implementation, review, verification, delivery, and learning — choosing the smallest *safe* intervention, proving it at the real delivery boundary, and leaving an inspectable evidence ledger. For change-request / issue-to-PR work, conditionally loads a 9-phase journey with gates, delivery packet, and lifecycle integration. Composes specialist catalog skills rather than replacing them. Not a persona, not a '10x developer' prompt, not a LOC-minimizer. The journey is not loaded for plain fixes, refactors, or reviews that lack an issue/ticket trajectory.
 - [nous-branding](nous-branding/SKILL.md): Generate images and content consistent with the Nous Research brand identity. Use when creating visuals in the Nous / Theia / Hermes ecosystem: a "cyber-classical" style blending neo-classical statuary, cyberpunk/industrial grunge, and retro anime illustration. Covers official brand color palette, typography (Inter/IBM Plex Sans, JetBrains Mono, heavy distressed display faces), the Nous Girl mascot, texture system, and image prompt construction. Ships reference images for palette, mascot, and brand collage that can be used as img2img inputs.
 - [open-knowledge-format](open-knowledge-format/SKILL.md): Google's Open Knowledge Format (OKF) v0.1 — an open, vendor-neutral spec for representing knowledge as markdown files with YAML frontmatter, designed for AI agent consumption. Use when the user mentions OKF, Open Knowledge Format, Google's knowledge format, LLM wiki bundles, agent knowledge packs, creating OKF bundles, validating OKF documents, or converting knowledge into the OKF standard.
@@ -124,6 +124,7 @@
 - [transistor](transistor/SKILL.md): Manage Transistor.fm podcast hosting from the terminal: view shows, list episodes, check analytics, and get subscriber counts. Use when the user mentions Transistor, podcast hosting, podcast analytics, show management, or episode tracking.
 - [vercel-eve](vercel-eve/SKILL.md): Build, develop, deploy, self-host, secure, and troubleshoot durable backend AI agents with Vercel Eve. Use when creating an Eve agent, adding tools, skills, subagents, channels, schedules, sandboxing, durable sessions, observability, or deploying Eve on Vercel or a Node host. Do not use for the separate Vercel AI SDK Agent APIs such as ToolLoopAgent or WorkflowAgent; use an AI SDK-specific skill for those.
 - [verification-methodology](verification-methodology/SKILL.md): Verify work against explicit criteria using direct, source-faithful evidence, reproducible checks, and clear verdicts. Use before declaring an artifact, implementation, or claim complete; do not use for exploratory research without pass/fail criteria.
 - [vllm](vllm/SKILL.md): Operate, configure, benchmark, and troubleshoot vLLM inference servers: Docker and Kubernetes deployment, quantization-aware model configuration (tensor parallelism, KV cache), OpenAI-compatible API serving, throughput and latency benchmarking, continuous batching tuning, GPU operation, and upgrade/rollback. Use when deploying or running a vLLM server (vllm serve, vllm/vllm-openai), sizing a model and its KV cache for GPUs, selecting quantization and parallelism, serving via /v1 endpoints, measuring serving throughput or latency, tuning batching, or diagnosing GPU, OOM, or startup failures in a vLLM deployment. Do not use for model training, fine-tuning, evaluation-set design, or engine-selection methodology (that is ml-engineering), or for operating the llama.cpp stack with GGUF models (that is llama-cpp); other inference engines (TGI, Ollama, Triton) are out of scope.
 - [web-accessibility](web-accessibility/SKILL.md): Design, build, and review accessible web interfaces with native semantics, keyboard and focus behavior, forms and recovery, responsive input, motion, assistive-technology testing, and WCAG 2.2-informed evidence. Use for a11y, WCAG, ARIA, screen-reader, keyboard, focus, dialog, form, widget, or accessibility review work across frameworks.
 - [woodpecker-ci](woodpecker-ci/SKILL.md): Operate Woodpecker CI from installation through production troubleshooting: configure servers and agents, connect Forgejo/Gitea or another forge, write and validate pipelines, manage secrets and plugins, use Docker or Kubernetes backends, run the CLI, and diagnose failed builds. Use when setting up, administering, or debugging Woodpecker CI.
 - [workflow-architect](bundles/workflow-architect/SKILL.md): Discover your actual workflow through conversation or observation, then generate a tailored skills bundle that encodes it as loadable agent skills with trigger conditions. Use when you want to understand your own process, formalize it, or share it with collaborators. Also use when a session feels aimless — this skill gives it structure.
@@ -6,7 +6,7 @@ description: >-
  decisions, inference deployment, and regression triage, grounded in practical
  engineering patterns for production ML systems. Do not use for statistical
  modeling and experimental design (that's the data scientist) or for operating
-  a specific inference engine (that's a tool skill such as llama-cpp).
+  a specific inference engine (that's a tool skill such as llama-cpp or vllm).
 license: MIT
 metadata:
  tags: ml, machine-learning, fine-tuning, training, evaluation, quantization, mlops,
@@ -25,7 +25,7 @@ Machine learning engineering is the bridge between model research and production
 | Model training — LoRA/QLoRA fine-tuning, full fine-tuning, distributed training | Statistical modeling and experimental design — that's the data scientist |
 | Model evaluation — benchmark suites, custom eval sets, regression testing | Causal inference and hypothesis testing — that's the data scientist |
 | Quantization — GGUF, GPTQ, AWQ, bitsandbytes | Training data collection and labeling — that's the data/ML ops team |
-| Inference serving — vLLM, [llama.cpp](../llama-cpp/SKILL.md), TGI, Triton | Business metrics and KPI definition — that's the product manager |
+| Inference serving — [vLLM](../vllm/SKILL.md), [llama.cpp](../llama-cpp/SKILL.md), TGI, Triton | Business metrics and KPI definition — that's the product manager |
 | Evaluation harness — lm-eval-harness, custom pipelines | Data pipeline architecture — that's the data engineer |
 | Model deployment — containerization, versioning, A/B testing | Infrastructure provisioning — that's the platform engineer |
@@ -70,4 +70,4 @@ Machine learning engineering is the bridge between model research and production
 ## When not to use
-Do not use this skill for statistical modeling, experimental design, or causal inference — that's the data scientist's discipline. Do not use it to operate a specific inference engine: for llama.cpp installation, model loading, benchmarking, and troubleshooting, load the [llama-cpp](../llama-cpp/SKILL.md) tool skill instead. This skill provides the methodology (eval-set design, quantization trade-offs, deployment plans, regression triage); the tool skills own the runbooks.
+Do not use this skill for statistical modeling, experimental design, or causal inference — that's the data scientist's discipline. Do not use it to operate a specific inference engine: for llama.cpp installation, model loading, benchmarking, and troubleshooting, load the [llama-cpp](../llama-cpp/SKILL.md) tool skill instead; for vLLM deployment, model configuration, benchmarking, batching tuning, GPU operation, and upgrade/rollback, load the [vllm](../vllm/SKILL.md) tool skill instead. This skill provides the methodology (eval-set design, quantization trade-offs, deployment plans, regression triage); the tool skills own the runbooks.
@@ -0,0 +1,61 @@
 # vLLM — Inference-Serving Skill
 Operate, configure, benchmark, and troubleshoot vLLM inference servers: Docker and Kubernetes deployment, quantization-aware model configuration (tensor parallelism, KV cache), the OpenAI-compatible API surface, throughput/latency benchmarking, continuous batching tuning, GPU operation, and upgrade/rollback.
 ## Why Install This Skill
 Your agent can run a vLLM deployment instead of guessing. Teams that self-serve open models in production need someone (or something) that knows how to start a `vllm serve` with the right flags, size the model and its KV cache for the GPUs at hand, confirm the OpenAI-compatible endpoints actually work, measure throughput and latency with evidence that comparisons mean something, tune continuous batching one knob at a time, and upgrade or roll back without burning the deployment.
 This skill ships that operating knowledge plus two fillable templates — a serving configuration record (so every deployment is reproducible) and a benchmark run record (so every performance claim is comparable) — and a read-only `vllm-health` probe that checks a running server's health, version, models, load, and metrics over HTTP without changing anything. The references are distilled from the official vLLM documentation with dated sources. Serving strategy and engine-selection methodology deliberately route up to `ml-engineering`; the llama.cpp stack routes to `llama-cpp`; this skill owns the day-to-day operation of vLLM itself.
 ## What You Get
 | Directory | Purpose |
 |---|---|
 | `SKILL.md` | Agent-facing operating loop, mutation gates, and verification boundaries |
 | `references/` | Seven dated, source-indexed references: source index, deployment, model configuration, OpenAI API, benchmarking, batching/tuning, GPU ops and lifecycle |
 | `templates/serving-config.md` | Fillable record of every serving argument, model revision, and environment — the rollback unit |
 | `templates/benchmark-run-record.md` | Fillable record that makes throughput/latency evidence comparable across runs |
 | `scripts/vllm-health` | Read-only probe: health, version, models, load, and metrics; stdlib-only, `--json`, `--help` without a server |
 | `tests/` | Deterministic tests against a local stub HTTP server, including the read-only contract |
 | `evals/evals.json` | Six output-quality evaluation cases for agent runs |
 ## Quick Start
 ```bash
 # Help works with no vLLM server
 scripts/vllm-health --help
 # Probe a running server, machine-readable
 scripts/vllm-health --url http://127.0.0.1:8000 --json
 # Targeted checks
 scripts/vllm-health --check health --check models --json
 # Record what you are about to run before you run it
 # (fill in vllm/templates/serving-config.md), then:
 docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 8000:8000 --ipc=host \
  vllm/vllm-openai:v0.26.0 \
  --model <model-name> --served-model-name <api-name> --max-model-len <len>
 # Benchmark serving throughput/latency once the server is ready
 vllm bench serve --backend vllm --model <model-name> \
  --endpoint /v1/completions --dataset-name custom \
  --dataset-path prompts.jsonl --num-prompts 100 --request-rate inf
 ```
 The `vllm-health` script uses only Python's standard library and issues GET requests only. Exit codes: 0 all checks passed, 1 issues found or a fatal error, 2 usage error, 124 timeout. Benchmark output includes request throughput (req/s), output token throughput (tok/s), and TTFT/TPOT/ITL percentiles — record them in `templates/benchmark-run-record.md`.
 ## Triggers
 Load this skill for vLLM operations: deploying or updating a `vllm serve` server (bare, Docker, or Kubernetes), choosing serving flags (`--quantization`, `--tensor-parallel-size`, `--max-model-len`, `--kv-cache-dtype`, `--gpu-memory-utilization`), wiring or debugging the OpenAI-compatible API surface (`/v1/chat/completions`, `/v1/completions`, `/v1/models`, `/health`, tool calling, chat templates), measuring serving throughput or latency (`vllm bench serve`/`bench throughput`), tuning continuous batching, running or diagnosing GPUs under a vLLM workload, or planning a vLLM upgrade or rollback. Do not load it for model training or fine-tuning (that's `ml-engineering`), for the llama.cpp stack (that's `llama-cpp`), or for generic Kubernetes/Docker administration (that's `kubernetes`/`docker-compose`).
 ## Requirements
 - A vLLM release: the `vllm/vllm-openai` Docker image (NVIDIA CUDA, AMD ROCm, or Intel XPU variants) or `pip install vllm==<pinned-version>`.
 - An accelerator with the matching driver (NVIDIA with the NVIDIA Container Toolkit for Docker, or the platform equivalent), or a supported CPU build for testing.
 - Hugging Face access to the model: a mounted `~/.cache/huggingface` and an `HF_TOKEN` for gated models.
 - Python 3.9+ for the `vllm-health` script (`--help` needs nothing else); live probes need HTTP(S) access to the running server.
@@ -0,0 +1,152 @@
 ---
 name: vllm
 description: >-
  Operate, configure, benchmark, and troubleshoot vLLM inference servers: Docker
  and Kubernetes deployment, quantization-aware model configuration (tensor
  parallelism, KV cache), OpenAI-compatible API serving, throughput and latency
  benchmarking, continuous batching tuning, GPU operation, and upgrade/rollback.
  Use when deploying or running a vLLM server (vllm serve, vllm/vllm-openai),
  sizing a model and its KV cache for GPUs, selecting quantization and
  parallelism, serving via /v1 endpoints, measuring serving throughput or
  latency, tuning batching, or diagnosing GPU, OOM, or startup failures in a
  vLLM deployment. Do not use for model training, fine-tuning, evaluation-set
  design, or engine-selection methodology (that is ml-engineering), or for
  operating the llama.cpp stack with GGUF models (that is llama-cpp); other
  inference engines (TGI, Ollama, Triton) are out of scope.
 license: MIT
 compatibility: >-
  Requires a vLLM release (v0.26.0 or a pinned older release), an NVIDIA CUDA,
  AMD ROCm, or Intel XPU GPU with the matching driver, or a supported CPU build.
  The bundled vllm-health script runs on Python 3.9+ and needs no vLLM server
  for --help; live probes require HTTP(S) access to a running vLLM server.
 metadata:
  source: https://docs.vllm.ai/en/latest/
  source_index: references/00-source-index.md
  research_checked: "2026-08-03"
 ---
 # vLLM Inference Serving
 Use this skill to operate **vLLM** as a production inference server: deploy it with Docker or Kubernetes, configure the model and engine (quantization, tensor parallelism, KV cache, context length), serve the OpenAI-compatible API surface, benchmark throughput and latency with comparable evidence, tune continuous batching, operate the GPUs underneath, and upgrade or roll back safely. This is a **tool skill** for one named engine. Serving *methodology* — engine selection, quantization trade-offs, deployment plans, regression triage — belongs to [ml-engineering](../ml-engineering/SKILL.md); local single-node GGUF serving with the llama.cpp stack belongs to [llama-cpp](../llama-cpp/SKILL.md). This skill owns the day-to-day operation of vLLM itself.
 ## Operating contract
 1. **Record the deployment before tuning it.** Capture the vLLM version or image digest, model and revision, quantization, parallelism, `max-model-len`, KV cache settings, batching limits, GPU inventory, and workload. The [serving config template](templates/serving-config.md) exists for exactly this.
 2. **Confirm the target, scope, and rollback path before acting.** Read-only discovery (health probes, `/metrics`, `nvidia-smi`) may proceed without confirmation. Mutations — restarting a server, changing serving args, scaling replicas, upgrading the image — require an explicit human directive naming the deployment.
 3. **A server that responds is not a server that serves.** `/health` returning 200 proves liveness, not that the model loaded or that inference works. Verify at the delivery boundary: `/v1/models` reports the served model and a representative request returns generated tokens.
 4. **Benchmark before and after every change.** vLLM flags, defaults, and behavior change between releases; an unmeasured tuning change is a guess. Compare only matched conditions (version, model, GPU, context, batch, workload) and record the evidence in the [benchmark run record](templates/benchmark-run-record.md).
 5. **Keep evidence bounded.** Summarize logs, configs, and metrics; never dump full server logs, `.env` files, or HF tokens into chat. `--enable-log-requests` with debug logging can leak prompt content; keep request logging off or redacted in shared sessions.
 ## The vllm-health script
 `scripts/vllm-health` is an agent-first, read-only probe for a running vLLM server. It issues GET requests only, never mutates, and emits bounded JSON.
 ```bash
 scripts/vllm-health --help                    # no server needed
 scripts/vllm-health --url http://127.0.0.1:8000 --json
 scripts/vllm-health --check health --check models --json
 scripts/vllm-health --check metrics --timeout 10 --json
 ```
 Exit codes: 0 all checks passed, 1 issues found or a fatal error, 2 usage error, 124 timeout. Checks: `health` (`/health`), `version` (`/version`), `models` (`/v1/models`), `load` (`/load`), and `metrics` (a bounded prefix of `/metrics`). The script never sends data anywhere and never writes files.
 ## Operating loop
 1. **Identify the deployment**: vLLM version or image digest, model and revision, served model name, parallelism, and how it is deployed (bare `vllm serve`, Docker, Kubernetes).
 2. **Collect evidence**: run `vllm-health --json` for health, version, models, and load; check `/metrics` counters (`vllm:num_requests_running`, `vllm:num_requests_waiting`, `vllm:gpu_cache_usage_perc`); inspect GPU state with `nvidia-smi`.
 3. **Triage against the symptom**: map the problem to the evidence (OOM → KV cache or `gpu_memory_utilization`; high latency → batching, TTFT vs TPOT; model not found → served name or chat template; slow start → model download or compile cache).
 4. **Act with confirmation**: bounded, scoped changes after a human directive, with a rollback path named first.
 5. **Verify**: re-run the probe and the representative request at the delivery boundary, and re-benchmark if the change affects performance.
 ## Deployment: Docker and Kubernetes
 - **Docker**: the official image is `vllm/vllm-openai` (Docker Hub). Run with GPU access, the Hugging Face cache mounted, the HF token for gated models, port 8000 published, and `--ipc=host` (or a `--shm-size`) for the shared memory tensor parallelism relies on. See [references/01-deployment.md](references/01-deployment.md).
 - **Kubernetes**: a Deployment with `nvidia.com/gpu` (or `amd.com/gpu`) resources, a PVC for the model cache, an `emptyDir` backed by Memory at `/dev/shm`, liveness/readiness probes on `/health` port 8000, and a Service. Raise probe `failureThreshold` for large models that take minutes to load — a premature kill shows up as `KeyboardInterrupt: terminated` in the container log.
 - Pin image tags to a release (for example `vllm/vllm-openai:v0.26.0`) instead of `latest`, and persist the compile cache (default `~/.cache/vllm`) across restarts so `torch.compile` artifacts are reused.
 ## Model configuration
 - **Model identity**: `--model` is the HF repo or local path; `--revision` pins the exact weights. `--served-model-name` sets the name clients must use in `/v1` requests and in the `model` field of responses. `--trust-remote-code` is required for some model repos and should be reviewed before use.
 - **Context length**: `--max-model-len` bounds prompt plus output per request. Unset, it derives from the model config; `-1`/`auto` picks the largest length that fits GPU memory. It is the single biggest driver of KV cache size.
 - **Quantization-aware serving**: pass `--quantization` (or `-q`) only when the model weights require it (GPTQ/AWQ/GGUF checkpoints load their scheme from config). Weight types and activation dtypes must match what the kernels support; a quantized model served at the wrong dtype fails to load or silently degrades. Hardware support varies by method (see [references/02-model-configuration.md](references/02-model-configuration.md)).
 - **Tensor parallelism**: `--tensor-parallel-size N` shards one model across N GPUs in the same node; `--pipeline-parallel-size` splits layers across nodes. TP requires NVLink/fast interconnect and equal per-GPU memory; startup logs the memory profiling result, which is the evidence that the model fits.
 - **KV cache**: `--gpu-memory-utilization` (default 0.92) caps the fraction of GPU memory the model plus KV cache may use. `--kv-cache-dtype fp8` shrinks the cache for long contexts on supported GPUs. The engine logs `GPU KV cache size: N tokens` and the implied max concurrency — record both; they tell you how many concurrent requests of a given length the box can hold.
 ## OpenAI-compatible API surface
 - Basic endpoints: `/health` (liveness), `/version`, `/v1/models` (served models), `/load` (load metrics), `/metrics` (Prometheus). Inference: `/v1/completions` and `/v1/chat/completions` (chat requires the model to ship a chat template, or pass `--chat-template`); `/v1/embeddings` for pooling models; `/v1/responses` for the Responses API.
 - Streaming, tool calling (`--enable-auto-tool-choice --tool-call-parser openai`), structured outputs, and parallel sampling are server-side options that change request/response behavior — verify each against the installed release rather than assuming parity.
 - Exposing the server beyond loopback requires an explicit decision about bind address, API keys, TLS or a trusted reverse proxy, and firewall rules. Development-only endpoints (`/reset_prefix_cache`, weight transfer, profiling) must not be exposed in production.
 ## Benchmarking: throughput and latency
 - **Online serving benchmark**: run `vllm bench serve` against a live server with a representative dataset (ShareGPT, a local `custom` JSONL, or your own prompts) and fixed `--num-prompts`, `--request-rate`, and `--max-concurrency`. It reports request throughput (req/s), output token throughput (tok/s), total token throughput, and TTFT/TPOT/ITL percentiles.
 - **Offline throughput**: `vllm bench throughput` measures raw engine throughput without the HTTP path; use it for engine-only comparisons, not end-to-end user latency.
 - **Comparable evidence**: the benchmark run record template freezes version, model, quantization, parallelism, context, batching, GPU, dataset, and load pattern. Never compare numbers across different conditions as if one variable changed. TTFT is a latency metric; token throughput is a throughput metric — an optimization that helps one can hurt the other.
 - For production capacity testing, vLLM's docs recommend the separate GuideLLM framework; this skill's scope is the bundled `vllm bench` tools.
 ## Continuous batching tuning
 - vLLM batches continuously by default: the scheduler admits sequences as capacity frees up, mixing prefill and decode. `--max-num-seqs` caps sequences per iteration, `--max-num-batched-tokens` caps tokens per iteration, and `--enable-chunked-prefill` lets prefill share an iteration with decode.
 - Start from defaults and change one knob at a time against the frozen benchmark: raising `--max-num-seqs` raises throughput at the cost of per-request latency and KV cache pressure; lowering it improves latency stability at the cost of utilization.
 - `--enable-prefix-caching` reuses KV blocks across requests with shared prefixes (chat system prompts, RAG contexts); the hit rate is visible in `/metrics` and in the benchmark's input token accounting. `--performance-mode` trades between `interactivity` (latency) and `throughput` at the kernel level.
 ## GPU operation
 - Verify GPUs with `nvidia-smi` (or `rocm-smi` on AMD): device list, memory, utilization, temperature, and ECC errors before and after changes. `CUDA_VISIBLE_DEVICES` selects which GPUs a `vllm serve` process sees; tensor parallel ranks map to the visible devices in order.
 - Watch `/metrics` for `vllm:gpu_cache_usage_perc` (KV cache pressure), `vllm:num_requests_running`/`waiting`, and `vllm:generation_tokens_total`. A cache-usage signal near 1.0 with requests waiting means the deployment is at capacity — scale out or reduce `max-model-len`/concurrency rather than overcommitting.
 - OOM during startup usually means the model + KV cache did not fit: lower `--gpu-memory-utilization` does not help if weights alone exceed memory — reduce `--max-model-len`, switch quantization, or add GPUs. OOM mid-run means KV cache pressure: shrink context, concurrency, or batch limits.
 ## Upgrade and rollback
 - **Pin everything**: image tag or `pip install vllm==<version>`, model revision, and the full serving command. `latest` images and unpinned revisions make rollback impossible and upgrades unreproducible.
 - **Upgrade path**: read the release notes for the full version span, review changed/removed flags (`--engine-args` change frequently), validate the new version on a scratch instance with the real model and workload, re-run the frozen benchmark, then swap with a rollback plan: previous image tag and previous serving config ready to reapply.
 - **Rollback**: because the config is versioned, rollback is a redeploy of the previous pinned image + config. KV cache layout, defaults, and flag names change between releases — do not assume a config that ran on v0.25.x behaves identically on v0.26.x without re-validating and re-benchmarking.
 ## Reference routing
 | Load when | Reference |
 |---|---|
 | Sources, version observations, refresh procedure | `references/00-source-index.md` |
 | Docker and Kubernetes deployment, image pinning, probes, storage | `references/01-deployment.md` |
 | Model config: quantization, tensor parallelism, KV cache, memory budgeting | `references/02-model-configuration.md` |
 | OpenAI-compatible API surface, chat templates, tools, auth | `references/03-openai-api.md` |
 | Benchmarking methodology and `vllm bench` commands | `references/04-benchmarking.md` |
 | Continuous batching, chunked prefill, prefix caching, performance mode | `references/05-batching-and-tuning.md` |
 | GPU operation, observability, upgrade/rollback, troubleshooting | `references/06-gpu-ops-and-lifecycle.md` |
 ## Included artifacts
 - `scripts/vllm-health`: read-only health/version/models/load/metrics probe (stdlib-only, `--json`, `--check` subsets, `--help` without a server).
 - `tests/test_vllm_health.py`: deterministic tests against a local stub HTTP server, including the read-only contract.
 - `templates/serving-config.md` and `templates/benchmark-run-record.md`: fillable records that make deployments reproducible and benchmark evidence comparable.
 - `references/`: seven dated, source-indexed references covering the operational topics above.
 - `evals/evals.json`: six output-quality evaluation cases for agent runs.
 ## Verification boundary
 | Claim | Minimum evidence |
 |---|---|
 | The server is alive | `vllm-health --check health` reports `/health` 200 |
 | The right model is served | `/v1/models` lists the expected served model name |
 | Inference works | A representative `/v1/chat/completions` or `/v1/completions` request returns generated tokens with a `finish_reason` |
 | The model fits | Startup log shows memory profiling completed and `GPU KV cache size: N tokens` for the configured parallelism |
 | A tuning change helped | The frozen benchmark shows the declared metric improving with matched conditions, variance reported |
 | The deployment is upgradable | Previous pinned image + serving config are recorded and the upgrade was rehearsed on a scratch instance |
 | A diagnosis is sound | Evidence was collected before the claim, and the fix was verified by re-running the probe and the benchmark |
 ## Hard boundaries
 - Never restart, redeploy, scale, or upgrade a vLLM deployment without an explicit human directive naming the target and a stated rollback path. Read-only discovery may proceed freely.
 - Never expose an unauthenticated server beyond loopback by accident; development-only endpoints and profiling routes must stay off production ingress.
 - Never print or commit HF tokens, `.env` contents, or full server logs; summarize evidence instead.
 - Never compare benchmark numbers from different versions, models, quants, parallelism, contexts, batches, or workloads as if one variable changed.
 - Never run `vllm-health` as anything but what it is — read-only. It has no mutation surface.
 ## When not to use
 - **Model training, fine-tuning, evaluation-set design, quantization decisions, and serving methodology** — that is [ml-engineering](../ml-engineering/SKILL.md).
 - **The llama.cpp stack** (llama-cli, llama-server, GGUF conversion and quantization, local Metal/CUDA builds) — that is [llama-cpp](../llama-cpp/SKILL.md).
 - **Other inference engines** (TGI, Ollama, Triton, vLLM's embedding/rerank-only workloads are in scope, but engine selection among them is not) — engine-selection trade-offs belong to `ml-engineering`.
 - **Kubernetes and Docker fundamentals** (manifests, RBAC, image registries, GPU device plugins) — that is [kubernetes](../kubernetes/SKILL.md) and [docker-compose](../docker-compose/SKILL.md).
 - **GPU infrastructure provisioning** (drivers, cluster scheduling, capacity planning) — that is [platform-engineering](../platform-engineering/SKILL.md); this skill operates the GPUs a vLLM server already targets.
@@ -0,0 +1,78 @@
 {
  "schema_version": 1,
  "skill_name": "vllm",
  "evals": [
    {
      "id": "serving-config-review",
      "prompt": "A team wants to serve a 70B instruct model (bf16) on two A100 80GB GPUs in one node with a 32K context window. Draft the vllm serve command with serving flags, explain the memory and parallelism reasoning, and state what to verify after startup.",
      "expected_output": "A vllm serve command that pins the model and revision, serves a stable API name, and configures parallelism and memory deliberately: --model <repo> --revision <sha> --served-model-name <api-name> --tensor-parallel-size 2 --max-model-len 32768 --gpu-memory-utilization 0.9 --trust-remote-code only if the model card requires it. The reasoning explains that a 70B bf16 checkpoint needs roughly 140GB of weights, so two 80GB GPUs with tensor parallelism are required (weights plus KV cache plus activations must fit under the per-instance gpu-memory-utilization cap), that --pipeline-parallel-size would only be needed across nodes, that --max-model-len 32K directly sizes the KV cache (larger context means fewer concurrent sequences), and that TP requires NVLink and equal per-GPU memory. Post-startup verification: the startup log shows memory profiling completed and the GPU KV cache size, /v1/models lists the served name, and a representative chat request returns tokens. The record goes into the serving config template so the deployment is reproducible and rollback is a redeploy of the same pinned image and arguments.",
      "assertions": [
        "The command pins model, revision, served-model-name, tensor-parallel-size 2, max-model-len, and gpu-memory-utilization",
        "The 70B bf16 weight size (~140GB) is used to justify two A100 80GB GPUs under the memory-utilization cap",
        "max-model-len is tied to KV cache sizing and concurrent-sequence capacity",
        "Verification includes startup memory profiling, /v1/models, and a representative chat request",
        "The serving config template is used so the deployment is reproducible and rollback is a pinned-image redeploy"
      ]
    },
    {
      "id": "quantization-and-parallelism-sizing",
      "prompt": "Our 8B model in bf16 takes 16GB of VRAM and we want to serve 256 concurrent long-context requests. We have a single 80GB H100. Should we quantize, enable tensor parallelism, or both, and what trade-offs should we record before changing anything?",
      "expected_output": "A sizing analysis that measures before optimizing: establish the baseline with the current bf16 serving config, record the reported GPU KV cache size and max concurrency for the target context, then evaluate options against that baseline. On a single H100, tensor parallelism is not applicable (TP shards across multiple GPUs) so the levers are quantization and KV cache settings: an FP8 or INT4 weight format shrinks weights and frees GPU memory for a larger KV cache, --kv-cache-dtype fp8 shrinks the cache for long contexts on supported hardware, and --max-model-len or --max-num-seqs bound concurrency. The trade-offs are stated explicitly: quantization changes output quality and requires kernels that match the hardware generation, and more KV cache (longer context or higher concurrency) increases gpu-cache-usage until requests wait. The change record freezes version, model, quantization, context, batching, and workload, and verification re-runs the frozen benchmark plus a quality spot-check rather than assuming the quantized model matches bf16 behavior.",
      "assertions": [
        "The response measures the baseline (KV cache size, max concurrency) before optimizing",
        "Tensor parallelism is correctly deemed not applicable on a single GPU",
        "Quantization (FP8/INT4 weights) and kv-cache-dtype fp8 are evaluated as the memory levers, with quality trade-offs stated",
        "KV cache pressure is tied to gpu-cache-usage and waiting requests",
        "Verification re-runs the frozen benchmark and a quality spot-check under matched conditions"
      ]
    },
    {
      "id": "openai-api-integration",
      "prompt": "We started vllm serve with a chat model and the OpenAI SDK returns a 404 for /v1/chat/completions, while /v1/models works. Diagnose the likely causes and describe how to verify the fix, without exposing the server.",
      "expected_output": "A diagnostic sequence over the read-only probes: confirm /health and /v1/models respond, then check what /v1/models actually lists and compare it with the model name the client sends, since a served-model-name mismatch or an unserved model id produces 404-style failures on inference routes. The next checks are version and load state, and the server log for whether the model loaded and which endpoints are registered on this release, because vLLM serves /v1/chat/completions only for text-generation models that carry a chat template; a model without a chat template cannot serve chat and needs --chat-template, and some pooling or embedding models do not register chat endpoints at all. The fix is verified by sending one bounded chat request with the exact served model name and checking for generated tokens, and by confirming the server is only reachable on the intended interface with API auth or a reverse proxy in place. The response does not expose logs or tokens and treats any bind-address or API-key change as a confirmed mutation.",
      "assertions": [
        "Diagnosis starts with /health, /v1/models, and the served model name versus the client's model field",
        "Chat templates and model type (generative versus pooling) are named as the cause of missing /v1/chat/completions routes",
        "The fix is verified with one bounded chat request returning generated tokens",
        "Exposure controls (bind address, API keys, reverse proxy) are checked without dumping logs or secrets",
        "A bind or auth change is treated as a mutation requiring confirmation"
      ]
    },
    {
      "id": "benchmark-run-review",
      "prompt": "A colleague claims a vLLM upgrade doubled our token throughput: 420 tok/s before, 850 tok/s after. Review their evidence before we roll the upgrade to production.",
      "expected_output": "A skeptical review of the benchmark evidence: the comparison is only meaningful if every other condition was frozen, so the reviewer asks for or reconstructs the matched conditions — pinned vLLM versions on both sides, identical model and revision, same quantization and dtype, same tensor parallelism, max-model-len, max-num-seqs, max-num-batched-tokens, GPU model and driver, dataset, prompt and output lengths, request rate, and concurrency — and requires the raw benchmark records from vllm bench serve showing request throughput, output token throughput, TTFT/TPOT/ITL percentiles, and variance across repetitions. A throughput doubling without changes to hardware or workload is a red flag for a mismatch (for example comparing different concurrency, a different dataset, or fp8 versus bf16). The reviewer also separates throughput from latency: if the goal is user-facing latency, TPOT and TTFT under realistic load matter more than raw tok/s. The upgrade only ships if the matched benchmark confirms the claim and the run record template was used on both sides.",
      "assertions": [
        "The review demands matched conditions (version, model, quant, parallelism, context, batching, GPU, workload) on both sides",
        "Raw vllm bench serve output with throughput, TTFT/TPOT/ITL percentiles, and variance is required, not a single number",
        "A suspicious doubling is probed for a mismatch in concurrency, dataset, or precision",
        "Throughput and latency (TTFT/TPOT under load) are treated as separate metrics",
        "The benchmark run record template is required on both sides before the upgrade ships"
      ]
    },
    {
      "id": "continuous-batching-tuning",
      "prompt": "Our vLLM server reports gpu_cache_usage_perc above 0.95 with requests waiting, and p50 latency is fine but p99 spiked. We already run with default batching settings. What should we change, and what should we measure?",
      "expected_output": "A one-variable-at-a-time tuning plan anchored on evidence: the cache-usage and waiting-request metrics say the deployment is at KV cache capacity, so the options are to bound concurrency (lower --max-num-seqs to cap sequences per iteration), reduce per-request token demand (lower --max-model-len or cap output tokens), enable or widen chunked prefill (--enable-chunked-prefill with a deliberate --max-num-batched-tokens) so prefill no longer blocks decode, or scale out, rather than raising --max-num-seqs further which would worsen p99. Prefix caching (--enable-prefix-caching) is evaluated if requests share system prompts, since a higher hit rate lowers effective input tokens. Every change is measured with the frozen benchmark plus the /metrics counters (gpu_cache_usage_perc, num_requests_running/waiting) before and after, one change at a time, and p99 is reported with its distribution rather than a single run.",
      "assertions": [
        "The plan ties gpu_cache_usage_perc near 1.0 with waiting requests to KV cache capacity, not to raising max-num-seqs",
        "Concrete levers are named: max-num-seqs, max-model-len or output cap, chunked prefill with max-num-batched-tokens, or scale-out",
        "Prefix caching is evaluated for shared-prefix workloads",
        "Changes are made one at a time with the frozen benchmark and /metrics counters before and after",
        "p99 is reported with distribution or variance, not a single run"
      ]
    },
    {
      "id": "upgrade-rollback-troubleshooting",
      "prompt": "We upgraded the vllm/vllm-openai image from :v0.25.1 to :v0.26.0 and the server now crashes at startup with a CUDA out of memory error, even though the model did not change. Diagnose and decide: fix forward or roll back?",
      "expected_output": "A bounded diagnosis that starts with the read-only evidence: confirm the exact image digest on both versions, the serving args, the model revision, and the GPU inventory (nvidia-smi), then compare the startup logs of the failing v0.26.0 run against the v0.25.1 run. OOM at startup after an engine upgrade with the same model points at changed memory defaults or changed flag semantics: vLLM releases change default KV cache allocation, max-model-len derivation, CUDA graph capture, and flag names, so the same config may now reserve more memory; the fix-forward path re-reads the v0.26.0 release notes for changed defaults, lowers gpu-memory-utilization or max-model-len on the scratch instance, validates the model loads and passes the frozen benchmark, and only then redeploys. The rollback path is already cheap because the previous pinned image and serving config were recorded: redeploy vllm/vllm-openai:v0.25.1 with the old args, verify /health, /v1/models, and one representative request, and re-run the benchmark. The response decides rollback first if the team cannot validate forward in the window, and never mutates the production deployment without a human directive naming target, scope, and rollback path.",
      "assertions": [
        "Diagnosis uses exact image digests, serving args, model revision, nvidia-smi, and side-by-side startup logs",
        "Changed memory defaults and flag semantics across releases are named as the leading OOM-after-upgrade cause",
        "The fix-forward path validates on a scratch instance with release notes and the frozen benchmark before redeploying",
        "The rollback path redeploys the pinned previous image plus recorded config and verifies at the delivery boundary",
        "No production mutation happens without a human directive naming target, scope, and rollback path"
      ]
    }
  ]
 }
@@ -0,0 +1,67 @@
 # vLLM Operations — Source Index
 > **Last Updated:** 2026-08-03
 This index tracks the authoritative upstream sources behind the vLLM operational
 skill and the refresh procedure for keeping it current. vLLM moves fast — flags,
 defaults, and endpoint behavior change between releases; treat any claim here as
 version-sensitive and re-verify against the installed release.
 ## Canonical sources
 | Topic | Source |
 |---|---|
 | vLLM documentation (latest) | https://docs.vllm.ai/en/latest/ |
 | vLLM documentation (stable release) | https://docs.vllm.ai/en/stable/ |
 | Releases and release notes | https://github.com/vllm-project/vllm/releases |
 | Docker deployment | https://docs.vllm.ai/en/latest/deployment/docker/ |
 | Kubernetes deployment | https://docs.vllm.ai/en/latest/deployment/k8s/ |
 | Online serving (OpenAI-compatible API) | https://docs.vllm.ai/en/latest/serving/online_serving/ |
 | Engine arguments | https://docs.vllm.ai/en/latest/configuration/engine_args/ |
 | Quantization | https://docs.vllm.ai/en/latest/features/quantization/index.html |
 | Benchmark CLI (`vllm bench`) | https://docs.vllm.ai/en/latest/benchmarking/cli/ |
 | vLLM paper (PagedAttention, SOSP 2023) | https://arxiv.org/abs/2309.06180 |
 | Continuous batching explainer (Anyscale, by Cade Daniel et al.) | https://www.anyscale.com/blog/continuous-batching-llm-inference |
 ## Version observations (as of this refresh)
 - Latest release: **v0.26.0** (published 2026-07-27). Release cadence is roughly
  monthly; docs publish `stable` and `latest` streams plus per-version archives.
 - The official Docker image is `vllm/vllm-openai` on Docker Hub, with
  `vllm/vllm-openai-rocm` (AMD) and `vllm/vllm-openai-xpu` (Intel) variants;
  the XPU image is official starting with v0.26.0.
 - Engine argument documentation moved to `configuration/engine_args`; engine
  args are also available as JSON-style CLI arguments (`--json-arg.key value`).
 - Benchmarking is now a first-class CLI: `vllm bench serve` (online serving),
  `vllm bench throughput` (offline), plus latency-focused and multimodal
  variants, replacing the older `benchmark_serving.py`/`benchmark_throughput.py`
  scripts. The docs recommend the external GuideLLM framework for production
  capacity testing.
 - `--prefix-caching` and `--performance-mode` (balanced/interactivity/throughput)
  are current flags on the `vllm serve` command line; older doc pages may still
  show earlier spellings.
 - Default `--gpu-memory-utilization` is 0.92 (per instance). `--kv-cache-dtype`
  accepts `auto`, `bfloat16`, `float16`, `fp8` (`fp8_e4m3`), `int8_per_token_head`,
  `nvfp4`, and other hardware-specific values on CUDA 11.8+.
 - vLLM 0.26.0 dependencies include Transformers 5.13, FlashInfer 0.6.14, and
  NIXL 1.3.1; GPU support spans NVIDIA (Ampere+ for most quantized kernels),
  AMD ROCm, and Intel XPU, with a CPU backend for testing.
 ## Refresh procedure
 1. Re-check the sources above for a new release and read its release notes for
   changed or removed engine arguments and changed defaults.
 2. Update the version observations that changed (defaults, flag names, endpoint
   behavior, hardware support).
 3. Re-verify the SKILL.md scope keyword sweep and the routing links to
   `ml-engineering` and `llama-cpp` still resolve.
 4. Re-run the bundled probe against a test server and confirm every check still
   parses: `scripts/vllm-health --url http://127.0.0.1:8000 --json`.
 ## Related skill sources
 - `ml-engineering` owns serving methodology: engine selection, quantization
  decisions, deployment plans, and regression triage. Its references are the
  source for engine-spanning decisions; this skill covers vLLM operation itself.
 - `llama-cpp` owns the llama.cpp stack (GGUF, llama-server). Do not duplicate
  its content here.
@@ -0,0 +1,138 @@
 # vLLM Deployment: Docker and Kubernetes
 > **Last Updated:** 2026-08-03
 > Sources: https://docs.vllm.ai/en/latest/deployment/docker/ and
 > https://docs.vllm.ai/en/latest/deployment/k8s/
 ## Docker
 The official image is `vllm/vllm-openai` (NVIDIA CUDA), with
 `vllm/vllm-openai-rocm` (AMD) and `vllm/vllm-openai-xpu` (Intel) variants. The
 image entrypoint is `vllm serve`, so engine arguments follow the image tag.
 ```bash
 docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=$HF_TOKEN" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:v0.26.0 \
    --model Qwen/Qwen3-0.6B
 ```
 ### Container details that matter
 - **Shared memory**: use `--ipc=host` or a `--shm-size` (for example `--shm-size=2g`).
  PyTorch uses shared memory between processes, particularly for tensor-parallel
  inference; the default 64 MB `/dev/shm` in a container is too small and causes
  obscure crashes at load time.
 - **HF token**: pass `--env "HF_TOKEN=$HF_TOKEN"` for gated models. Never commit
  the token; mount the cache volume so weights are reused across restarts.
 - **CUDA compatibility**: on hosts whose driver is older than the toolkit in the
  image, set `VLLM_ENABLE_CUDA_COMPATIBILITY=1` (only for select datacenter GPUs).
 - **Compile cache**: vLLM compiles `torch.compile` artifacts into
  `VLLM_CACHE_ROOT` (default `~/.cache/vllm`). Mount a named volume there so the
  second and later containers start fast instead of recompiling.
 - **Non-root**: the image ships a `vllm` user (UID 2000, GID 0). Run with
  `--user 2000:0` and mount writable paths under `/home/vllm` (for example
  `/home/vllm/.cache/huggingface`) rather than `/root`. The `vllm-openai-nonroot`
  image target supports OpenShift-style arbitrary UIDs within group 0.
 - **Pin tags**: use a release tag (`vllm/vllm-openai:v0.26.0`), not `latest`.
  Optional dependencies (audio, gRPC, etc.) are not in the base image; layer
  them on with `uv pip install --system vllm[extra]==<same-version>`.
 ## Kubernetes
 A native deployment: Deployment + Service, GPU resource limits, a model-cache
 volume, an `emptyDir` shared-memory volume, and `/health` probes.
 ```yaml
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: vllm-server
 spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: vllm
  template:
    metadata:
      labels:
        app.kubernetes.io/name: vllm
    spec:
      volumes:
        - name: cache-volume
          persistentVolumeClaim:
            claimName: vllm-models
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: "2Gi"
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.26.0
          command: ["/bin/sh", "-c"]
          args:
            - "vllm serve <model> --served-model-name <name> --trust-remote-code"
          env:
            - name: HF_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token-secret
                  key: token
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: "1"
            requests:
              nvidia.com/gpu: "1"
          volumeMounts:
            - mountPath: /root/.cache/huggingface
              name: cache-volume
            - mountPath: /dev/shm
              name: shm
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 60
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 60
            periodSeconds: 5
 ```
 ### K8s details that matter
 - **GPU scheduling**: request `nvidia.com/gpu: "1"` (NVIDIA device plugin) or
  `amd.com/gpu` (AMD k8s device plugin). Tensor parallelism across GPUs in one
  pod uses `--tensor-parallel-size` equal to the GPU count; the pod then needs
  a larger `/dev/shm` (size it at 2-8 GiB) and, on some platforms, host IPC.
 - **Probes**: vLLM's `/health` endpoint reports ready only after the model
  finishes loading, which can take minutes for large models. Set
  `initialDelaySeconds` and `failureThreshold` high enough; a container killed
  by the probe loop logs `KeyboardInterrupt: terminated` and shows
  `failed startup probe, will be restarted` in `kubectl get events`. To find
  the right threshold, remove the probes, time startup, then restore them.
 - **gRPC**: pass `--grpc` (requires `vllm[grpc]` in the image) and replace the
  HTTP probes with `grpc` probes; the server then implements the standard
  gRPC health-checking protocol and returns `NOT_SERVING` while loading or
  shutting down.
 - **Alternatives**: the upstream docs also cover Helm, KServe, KubeRay,
  NVIDIA Dynamo, and the vllm-project production-stack integrations. For this
  skill's scope, the native manifest above is the reference pattern; the
  infrastructure for those frameworks routes to `kubernetes`.
 ## Verification at the delivery boundary
 - `kubectl logs -l app.kubernetes.io/name=vllm` shows `Application startup
  complete` and `Uvicorn running on http://0.0.0.0:8000`.
 - `vllm-health --url <service-url> --check health --check models --json` shows
  `/health` 200 and the served model.
 - A bounded request returns generated tokens; a PVC-backed model cache means
  the next pod starts without re-downloading weights.
@@ -0,0 +1,83 @@
 # vLLM Model Configuration: Quantization, Tensor Parallelism, KV Cache
 > **Last Updated:** 2026-08-03
 > Sources: https://docs.vllm.ai/en/latest/configuration/engine_args/ and
 > https://docs.vllm.ai/en/latest/features/quantization/index.html
 ## Model identity and context
 - `--model` is the Hugging Face repo or a local path; `--revision` pins a branch,
  tag, or commit so weight provenance is reproducible. `--tokenizer` overrides
  the tokenizer; `--chat-template` supplies a Jinja2 chat template when the
  model card lacks one (without one, chat requests error).
 - `--max-model-len` bounds prompt + output per request and accepts human-readable
  values (`32k` = 32,000; `32K` = 32,768). Unset, it derives from the model
  config. `-1`/`auto` picks the largest length that fits GPU memory, capped by
  the model's trained context. This flag is the dominant driver of KV cache size:
  halving it roughly doubles the concurrent sequences a GPU can hold.
 - `--dtype` selects weight/activation precision (`auto`, `bfloat16`, `float16`,
  `float32`, `half`). `auto` uses FP16 for FP32/FP16 models and BF16 for BF16
  models; some quantized formats are recommended at a specific dtype (for
  example `half` for AWQ).
 ## Quantization-aware serving
 - **Let the checkpoint declare its scheme.** vLLM first checks the model's
  `quantization_config`; `--quantization`/`-q` is for cases where the config is
  missing or needs overriding. Serving a checkpoint with the wrong method fails
  to load or silently degrades.
 - **Supported methods (as of v0.26.0)**: GPTQ, AWQ, bitsandbytes (load-time
  quantization), GGUF, LLM Compressor FP8/INT8/INT4, NVIDIA Model Optimizer
  (NVFP4/MXFP4/FP8), TorchAO, and online quantization. The current index lives
  at https://docs.vllm.ai/en/latest/features/quantization/index.html.
 - **Hardware coupling**: kernel support varies by GPU generation — for example
  Marlin (GPTQ/AWQ/FP8/FP4) requires Turing+ and is NVIDIA-only; FP8 W8A8 needs
  Ada/Hopper; GGUF and bitsandbytes span more platforms. Check the compatibility
  table before choosing a quantized checkpoint for a GPU fleet.
 - **KV cache quantization**: `--kv-cache-dtype` (`auto`, `bfloat16`, `float16`,
  `fp8` = `fp8_e4m3`, `int8_per_token_head`, `nvfp4`, ...) shrinks the attention
  cache for long-context workloads on CUDA 11.8+. This is a quality-vs-capacity
  decision; spot-check outputs before trusting it at scale.
 - **Dated-source rule**: quantization support changes every release. Never
  assume a method that worked on one version works on the next; the source
  index records the refresh date.
 ## Tensor, pipeline, and data parallelism
 - `--tensor-parallel-size N` (`-tp`) shards each layer's weights and attention
  across N GPUs in one node. It requires fast interconnect (NVLink or high-speed
  NIC) and equal per-GPU memory. Startup runs a memory-profiling pass and logs
  the resulting KV cache size — that log line is the evidence the model fits.
 - `--pipeline-parallel-size N` (`-pp`) splits layers across ranks/nodes for
  models too large for one node's aggregate memory; it adds inter-stage
  communication latency.
 - `--data-parallel-size N` (`-dp`) replicates the model across groups for
  throughput scaling of small models. `--expert-parallel-size` shards MoE
  experts. The product of TP × PP × DP (× EP for MoE) must equal the world size.
 - **Gotchas**: tensor-parallel workers must all see the same visible GPU set
  (`CUDA_VISIBLE_DEVICES` applies to the whole process group); mixed GPU models
  or different per-GPU memory cause the memory profile to fail; multi-node TP
  needs `--distributed-executor-backend mp` with `--master-addr`/`--master-port`
  reachable across nodes.
 ## Memory budgeting
 - `--gpu-memory-utilization` (default 0.92) caps the fraction of GPU memory the
  model executor may use — weights plus KV cache plus activation buffers. It is
  per-instance and does not account for other processes on the GPU.
 - Weights dominate first: a bf16 70B needs ~140 GB before any KV cache, so it
  needs two 80 GB GPUs (TP=2) or quantization. Only after weights fit does the
  KV cache decide concurrency: the engine logs `GPU KV cache size: N tokens`
  and `Maximum concurrency for M tokens per request: K`; record both.
 - `--cpu-offload-gb` moves weights/KV to CPU per GPU (a virtual memory increase
  at the cost of PCIe-bound performance) — a stopgap for fitting a model, not a
  performance feature.
 - `--load-format` (auto, safetensors, npcache, bitsandbytes, sharded_state, ...)
  controls how weights are read; `--safetensors-load-strategy eager` avoids
  random reads on network filesystems (NFS/Lustre) at the cost of CPU RAM.
 ## Configuration record
 Every non-default choice goes into `templates/serving-config.md`. The record is
 what makes a deployment reproducible and rollback a redeploy of the previous
 pinned image plus record.
@@ -0,0 +1,69 @@
 # vLLM OpenAI-Compatible API Surface
 > **Last Updated:** 2026-08-03
 > Source: https://docs.vllm.ai/en/latest/serving/online_serving/
 ## Basic endpoints
 | Endpoint | Purpose |
 |---|---|
 | `/health` | Liveness; returns 200 only when the engine is ready to serve |
 | `/version` | vLLM version information |
 | `/v1/models` | List of served models (the `served-model-name`s clients must use) |
 | `/load` | Server load metrics |
 | `/metrics` | Prometheus-compatible metrics (`vllm:*` counters and gauges) |
 ## Inference endpoints
 - `/v1/completions` — text generation (no chat template needed). Note: the
  `suffix` parameter is not supported.
 - `/v1/chat/completions` — chat; requires the model to carry a chat template in
  its tokenizer config (or pass `--chat-template <file|string>`). The `user`
  parameter is ignored. `parallel_tool_calls` controls whether more than one
  tool call per response is allowed.
 - `/v1/responses` (+ `/v1/responses/{id}/cancel`) — the OpenAI Responses API,
  for text-generation models.
 - `/v1/embeddings` — for embedding/pooling models.
 - `/v1/audio/transcriptions` and `/v1/audio/translations` — ASR models.
 - Anthropic Messages API (`/v1/messages`, `/v1/messages/count_tokens`) and
  gRPC (`--grpc`) are also served on recent releases.
 Chat models whose card lacks a template, and pooling/embedding models, will not
 serve chat routes; a 404 on `/v1/chat/completions` while `/v1/models` works is
 the classic symptom.
 ## Request/response facts
 - The `model` field in requests must match a `--served-model-name` (or the
  `--model` value if none was set). `--served-model-name` accepts multiple names
  and aliases; the response echoes the first.
 - Streaming (`"stream": true`) emits `choices[].delta` chunks with a terminal
  `finish_reason`. Tool calling requires server flags: `--enable-auto-tool-choice
  --tool-call-parser openai` (parser per model family). Structured outputs use
  xgrammar or guidance backends.
 - Sampling parameters (temperature, top-p, top-k, max_tokens, stop, logprobs)
  are per-request; `--max-logprobs` caps logprobs server-wide.
 ## Exposure and security
 - Start on loopback (`--host 127.0.0.1`) and verify with `vllm-health` and a
  bounded request before any wider bind. Exposing beyond loopback requires an
  explicit decision: bind address, API key (`--api-key`), TLS or a trusted
  reverse proxy, firewall, CORS, and rate limiting.
 - Development-only and destructive routes must never be exposed in production:
  `/reset_prefix_cache`, `/reset_mm_cache`, weight-transfer endpoints
  (`/start_weight_update`, `/update_weights`), profiling (`/start_profile`),
  and `/collective_rpc`. vLLM gates some of these behind
  `VLLM_SERVER_DEV_MODE=1` — do not enable dev mode on production ingress.
 - `--enable-log-requests` at debug level logs prompt text; keep it off or
  redacted in shared/audited sessions. Do not print or `tee` unredacted
  environment files, `.env` contents, or HF tokens.
 ## Verification checklist
 1. `/health` returns 200.
 2. `vllm-health --check models --json` lists the served model name(s).
 3. A bounded chat/completion request returns generated tokens and a
   `finish_reason`.
 4. Streaming and tool-calling paths are verified with the exact client that
   production uses (not assumed from endpoint existence).
@@ -0,0 +1,92 @@
 # vLLM Benchmarking: Throughput and Latency
 > **Last Updated:** 2026-08-03
 > Source: https://docs.vllm.ai/en/latest/benchmarking/cli/
 ## Metrics that matter
 - **Throughput**: request throughput (req/s), output token throughput (tok/s),
  and total token throughput (input + output tok/s). Output token throughput is
  the number users feel; total token throughput includes prompt processing.
 - **Latency**: TTFT (time to first token — what users perceive as the start of
  a response), TPOT (time per output token, excluding the first), and ITL
  (inter-token latency, including scheduling jitter). Report mean, median, and
  p99 — p99 under load is the SLO-relevant number.
 - Throughput and latency are different axes: raising concurrency raises
  throughput until the KV cache saturates, then latency degrades. State which
  axis a change optimizes before measuring it.
 ## `vllm bench serve` (online serving benchmark)
 Run against a live server with a representative dataset:
 ```bash
 vllm bench serve \
  --backend vllm \
  --model <model-name> \
  --endpoint /v1/completions \
  --dataset-name sharegpt \
  --dataset-path <path>/ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 100 \
  --request-rate inf \
  --max-concurrency 64 \
  --save-result --result-dir ./log
 ```
 Output includes: `Successful requests`, `Benchmark duration (s)`, `Total input
 tokens`, `Total generated tokens`, `Request throughput (req/s)`, `Output token
 throughput (tok/s)`, `Total token throughput (tok/s)`, and mean/median/p99 of
 TTFT, TPOT, and ITL.
 ### Load pattern control
 - `--request-rate`: `inf` sends all requests immediately (maximum throughput
  test); a finite rate (requests/second) simulates arrival traffic with a
  Poisson process (`--burstiness 1.0`), bursty Gamma traffic (`0.1-0.5`), or
  uniform spacing (`2.0-5.0`).
 - `--max-concurrency`: caps outstanding requests, simulating a load balancer or
  gateway limit. The most common production pattern is `--request-rate=inf
  --max-concurrency=<limit>`.
 - Datasets: `sharegpt` (realistic chat), `custom` (a JSONL of `{"prompt": ...}`
  entries), synthetic random lengths, and HF-hosted sets. For capacity planning
  use the KV-cache-derived maximum concurrency the server logs at startup as
  the ceiling (80-90% of it for realistic tests).
 ## `vllm bench throughput` (offline benchmark)
 Measures raw engine throughput without the HTTP path:
 ```bash
 vllm bench throughput \
  --model <model-name> \
  --input-len 512 --output-len 128 \
  --num-prompts 1000
 ```
 Use it for engine-only comparisons (kernel, quantization, parallelism); it is
 not a user-facing latency measurement. `vllm bench latency` is the
 latency-oriented offline variant.
 ## Comparable evidence (the skill's core rule)
 Benchmark numbers are only comparable under **matched conditions**. The
 [benchmark run record template](../templates/benchmark-run-record.md) freezes:
 - vLLM version/image tag/digest, model + revision, quantization + dtype;
 - parallelism (TP/PP/DP), `max-model-len`, KV cache dtype, memory utilization;
 - batching knobs (`max-num-seqs`, `max-num-batched-tokens`, chunked prefill,
  prefix caching), performance mode;
 - GPU model/count/driver/interconnect, host, thermal, background load;
 - dataset, prompt/output lengths, request rate, concurrency, sampling params;
 - repetitions and variance.
 Never compare numbers from different versions, models, quants, parallelism,
 contexts, batches, or workloads as if one variable changed. Re-run the frozen
 benchmark after every serving-arg change and record both sides in the template.
 ## Production capacity testing
 For SLA validation and capacity planning the upstream docs recommend the
 external GuideLLM framework (live progress, automatic reports). This skill's
 scope is the bundled `vllm bench` tools plus the run-record discipline; route
 a GuideLLM setup as its own tooling decision.
@@ -0,0 +1,67 @@
 # vLLM Continuous Batching and Tuning
 > **Last Updated:** 2026-08-03
 > Sources: https://docs.vllm.ai/en/latest/configuration/engine_args/ and
 > https://www.anyscale.com/blog/continuous-batching-llm-inference
 ## How continuous batching works
 vLLM batches requests **continuously** (iteration-level scheduling): the
 scheduler admits new sequences whenever capacity frees up, mixing prefill and
 decode in the same iteration instead of waiting for a whole batch to finish.
 PagedAttention stores KV blocks in a paged table so memory is allocated per
 token-block rather than per whole sequence, which is what makes high occupancy
 and large effective concurrency possible. The practical consequences:
 - A single model replica serves many concurrent requests; throughput rises with
  concurrency until the KV cache or the batch limits bind.
 - Long prefill requests can block decode of other requests unless chunked
  prefill is enabled, which is exactly what spikes p99 latency under mixed
  workloads.
 ## The knobs
 | Flag | Default behavior | What it does |
 |---|---|---|
 | `--max-num-seqs` | convenience default (set per deployment) | Max sequences per scheduling iteration. Higher = more batching and throughput, higher per-request latency and KV pressure. |
 | `--max-num-batched-tokens` | convenience default (set per deployment) | Max tokens per iteration. Together with chunked prefill it bounds how much prefill work can run between decode steps. |
 | `--enable-chunked-prefill` | disabled by default on most configs | Lets a prefill request be chunked across iterations so it shares time with decode; the fix for prefill-blocking-decode latency spikes. |
 | `--enable-prefix-caching` | disabled | Reuses KV blocks for shared request prefixes (chat system prompts, RAG contexts); hit rate shows up in `/metrics` and lower effective input tokens. |
 | `--performance-mode` | `balanced` | Kernel-level trade: `interactivity` favors low end-to-end latency at small batch sizes; `throughput` favors aggregate tok/s at high concurrency. |
 | `--optimization-level` | `-O2` | Startup-time vs performance trade for `torch.compile` (`-O0` fastest start, `-O3` best performance). |
 Both `--max-num-seqs` and `--max-num-batched-tokens` accept human-readable
 sizes (`1k`, `2M`, ...). `--long-prefill-token-threshold` sets when a prefill is
 treated as long for chunked-prefill scheduling.
 ## Tuning discipline
 1. **Establish the baseline first**: freeze the serving config record, run the
   benchmark, and record `vllm:gpu_cache_usage_perc`,
   `vllm:num_requests_running`, and `vllm:num_requests_waiting` at the peak.
 2. **Change one knob at a time** and re-run the frozen benchmark. Two changes
   in one iteration make the delta unassignable.
 3. **Read the signals**: cache usage near 1.0 with requests waiting means the
   KV cache is the binding constraint — lower `--max-num-seqs`, shorten
   `--max-model-len` or cap output tokens, or scale out. It does not mean
   raise the batch limits further.
 4. **Match the knob to the symptom**: latency spikes under mixed load →
   chunked prefill + a deliberate `--max-num-batched-tokens`; low utilization
   at steady state → raise `--max-num-seqs` within the cache budget;
   shared-prefix workloads → `--enable-prefix-caching`.
 5. **Verify at the boundary**: re-run the benchmark and confirm the declared
   metric moved and the guardrail metrics (p99 TTFT/TPOT) did not regress,
   with variance reported across repetitions.
 ## Capacity math from the startup log
 At startup the engine reports something like:
 ```
 GPU KV cache size: 15,728,640 tokens
 Maximum concurrency for 8,192 tokens per request: 1920
 ```
 `max_concurrency = kv_cache_size / max_model_len`. Use 80-90% of that figure as
 the `--max-concurrency` for capacity-planning benchmarks, and treat the size as
 the ceiling when deciding whether to raise concurrency or shorten context.
@@ -0,0 +1,82 @@
 # vLLM GPU Operation, Upgrade, and Rollback
 > **Last Updated:** 2026-08-03
 > Sources: https://docs.vllm.ai/en/latest/configuration/engine_args/,
 > https://docs.vllm.ai/en/latest/deployment/docker/,
 > https://github.com/vllm-project/vllm/releases
 ## GPU operation
 - **Inventory first**: `nvidia-smi` (NVIDIA) or `rocm-smi` (AMD) shows device
  list, per-GPU memory, utilization, temperature, power, and ECC/error state.
  Record it before and after changes; a thermal or ECC change invalidates
  benchmark comparisons.
 - **Device selection**: `CUDA_VISIBLE_DEVICES="0,1" vllm serve ...` restricts
  which GPUs the process sees; `--tensor-parallel-size` counts the *visible*
  devices in order. Device names are the container's view, not the host's —
  verify with `nvidia-smi -L` inside the container.
 - **Per-GPU memory matters**: tensor parallelism assumes equal per-GPU memory.
  Mixed-capacity GPUs or other workloads sharing a GPU break the memory
  profile; pin the serving process to dedicated devices.
 - **Observability**: `/metrics` exposes `vllm:gpu_cache_usage_perc` (KV cache
  pressure), `vllm:num_requests_running`, `vllm:num_requests_waiting`, and
  token counters. Prometheus scraping of `/metrics` is the standard wiring;
  alert on cache usage near 1.0 with requests waiting (capacity), repeated OOM,
  and probe failures.
 - **OOM at startup** usually means weights + KV cache do not fit: reduce
  `--max-model-len`, switch to a quantized checkpoint, add GPUs, or (as a
  stopgap) `--cpu-offload-gb`. Lowering `--gpu-memory-utilization` does not
  help if the weights alone exceed memory.
 - **OOM mid-run** means KV cache pressure at peak concurrency: shorten context,
  lower `--max-num-seqs`, or scale out replicas.
 ## Upgrade
 1. **Pin before you start**: image tag (or `pip install vllm==<version>`),
   model revision, and the full serving command from the serving config record.
   Without pins, upgrades are unreproducible and rollback is guesswork.
 2. **Read the release notes for the full span** (for example v0.25.1 → v0.26.0):
   vLLM deprecates and renames flags, changes defaults (memory allocation,
   `max-model-len` derivation, CUDA graph capture), and adjusts kernel/hardware
   support between releases. A config that ran on one minor may not behave the
   same on the next.
 3. **Rehearse on a scratch instance** with the real model and workload: install
   the new version, apply the config, confirm the model loads, and run the
   frozen benchmark.
 4. **Swap with a rollback plan**: deploy the new pinned image, verify at the
   delivery boundary (`/health`, `/v1/models`, a representative request,
   re-benchmark), and keep the previous image + config ready to reapply.
 ## Rollback
 - **Rollback is a redeploy of the previous pinned image + serving config.**
  Because the config record is versioned and the image tag is pinned, the old
  state is reproducible by construction.
 - Verify the rollback the same way as an upgrade: `/health`, `/v1/models`, one
  representative request, and the frozen benchmark. Do not assume the old
  behavior returned because the old image is back.
 - Watch for **config drift across versions**: KV cache layout, defaults, and
  flag names change between releases. A rollback must restore the *recorded*
  config, not the current one.
 ## Troubleshooting table
 | Symptom | First evidence to collect | Likely causes and next step |
 |---|---|---|
 | Server crashes at startup (OOM) | Side-by-side startup logs, `nvidia-smi`, image digests | Weights+KV don't fit; changed defaults after upgrade. Reduce `max-model-len`/memory utilization on scratch, or roll back the pinned image |
 | `/health` 200 but requests fail | `/v1/models`, one bounded request | Model not loaded, served-name mismatch, missing chat template; check `/load` and startup log |
 | 404 on `/v1/chat/completions` | `/v1/models`, model type | Pooling/embedding model or missing chat template; see `references/03-openai-api.md` |
 | High p99 latency | TTFT vs TPOT split, cache-usage metrics | Prefill blocking decode → chunked prefill; KV pressure → bound concurrency |
 | Requests waiting, cache ~100% | `vllm:gpu_cache_usage_perc`, waiting gauge | At capacity: scale out or reduce context/concurrency; do not raise batch limits |
 | Container killed at startup | `kubectl get events`, log `KeyboardInterrupt: terminated` | Probe `failureThreshold` too low for model load time; raise it |
 | Slow restart every time | Startup duration, `VLLM_CACHE_ROOT` | Compile cache not persisted; mount `~/.cache/vllm` volume |
 ## Hard boundaries
 - No restart, redeploy, scale, or upgrade without an explicit human directive
  naming the target and a stated rollback path. Read-only discovery may proceed
  without confirmation.
 - Never compare performance across versions, models, quants, parallelism,
  contexts, batches, or workloads as if one variable changed.
 - Never expose development-only endpoints, request logs with prompt content, or
  credentials to chat or logs.
@@ -0,0 +1,221 @@
 #!/usr/bin/env python3
 """vllm-health - read-only probe for a running vLLM OpenAI-compatible server.
 Collects bounded operational evidence from a live vLLM server over HTTP(S)
 without mutating anything: /health liveness, /version, /v1/models (served
 model names), /load metrics, and a bounded prefix of /metrics. The script
 issues GET requests only, never writes files, and never sends data anywhere.
 The script uses the Python standard library only, and --help works with no
 vLLM server running. Output is bounded JSON with --json, or human-readable
 text otherwise.
 Exit codes: 0 all checks passed, 1 issues found or a fatal error,
 2 usage error, 124 timeout.
 """
 import argparse
 import json
 import socket
 import sys
 import urllib.error
 import urllib.parse
 import urllib.request
 from typing import Any, Dict, List, Optional
 DEFAULT_URL = "http://127.0.0.1:8000"
 DEFAULT_TIMEOUT = 10
 METRICS_BOUND_BYTES = 64 * 1024
 CHECKS = [
    "health",
    "version",
    "models",
    "load",
    "metrics",
 ]
 class ProbeTimeout(Exception):
    """Raised when a server request exceeds the configured timeout."""
 def parse_args(argv: List[str]) -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        prog="vllm-health",
        description=(
            "Read-only probe for a running vLLM OpenAI-compatible server: "
            "health, version, models, load, and bounded metrics."
        ),
    )
    parser.add_argument(
        "--url",
        default=DEFAULT_URL,
        help=f"Base URL of the vLLM server (default: {DEFAULT_URL})",
    )
    parser.add_argument(
        "--check",
        action="append",
        choices=CHECKS,
        help="Run only the named check(s); repeatable (default: all checks)",
    )
    parser.add_argument("--json", action="store_true", help="Emit bounded JSON output")
    parser.add_argument(
        "--timeout",
        type=float,
        default=DEFAULT_TIMEOUT,
        help=f"Per-request timeout in seconds (default: {DEFAULT_TIMEOUT})",
    )
    return parser.parse_args(argv)
 def http_get(url: str, timeout: float, max_bytes: Optional[int] = None) -> tuple[int, bytes]:
    """GET *url* and return (status, body); read at most *max_bytes* when set.
    Raises ProbeTimeout when the request exceeds *timeout*, and RuntimeError on
    connection failures.
    """
    request = urllib.request.Request(url, method="GET")
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:
            if max_bytes is None:
                return response.status, response.read()
            return response.status, response.read(max_bytes)
    except (socket.timeout, TimeoutError) as error:
        raise ProbeTimeout(f"request timed out after {timeout}s") from error
    except urllib.error.HTTPError as error:
        return error.code, error.read()
    except urllib.error.URLError as error:
        raise RuntimeError(f"connection failed: {error.reason}") from error
 def check_health(base_url: str, timeout: float) -> Dict[str, Any]:
    status, body = http_get(urllib.parse.urljoin(base_url.rstrip("/") + "/", "health"), timeout)
    return {
        "name": "health",
        "status_code": status,
        "ok": status == 200,
        "body": body.decode("utf-8", errors="replace")[:200],
    }
 def check_version(base_url: str, timeout: float) -> Dict[str, Any]:
    status, body = http_get(urllib.parse.urljoin(base_url.rstrip("/") + "/", "version"), timeout)
    return {
        "name": "version",
        "status_code": status,
        "ok": status == 200,
        "body": body.decode("utf-8", errors="replace")[:200],
    }
 def check_models(base_url: str, timeout: float) -> Dict[str, Any]:
    status, body = http_get(urllib.parse.urljoin(base_url.rstrip("/") + "/", "v1/models"), timeout)
    model_ids: List[str] = []
    if status == 200:
        try:
            payload = json.loads(body.decode("utf-8"))
            model_ids = [entry.get("id", "") for entry in payload.get("data", []) if isinstance(entry, dict)]
        except (ValueError, TypeError):
            model_ids = []
    return {
        "name": "models",
        "status_code": status,
        "ok": status == 200 and bool(model_ids),
        "model_ids": model_ids[:20],
    }
 def check_load(base_url: str, timeout: float) -> Dict[str, Any]:
    status, body = http_get(urllib.parse.urljoin(base_url.rstrip("/") + "/", "load"), timeout)
    parsed: Dict[str, Any] = {}
    if status == 200:
        try:
            parsed = json.loads(body.decode("utf-8"))
        except (ValueError, TypeError):
            parsed = {}
    return {
        "name": "load",
        "status_code": status,
        "ok": status == 200,
        "load": parsed,
    }
 def check_metrics(base_url: str, timeout: float) -> Dict[str, Any]:
    status, body = http_get(
        urllib.parse.urljoin(base_url.rstrip("/") + "/", "metrics"),
        timeout,
        max_bytes=METRICS_BOUND_BYTES + 1,
    )
    truncated = len(body) > METRICS_BOUND_BYTES
    body = body[:METRICS_BOUND_BYTES]
    text = body.decode("utf-8", errors="replace")
    keys = [
        "vllm:gpu_cache_usage_perc",
        "vllm:num_requests_running",
        "vllm:num_requests_waiting",
        "vllm:generation_tokens_total",
    ]
    observed = {key: (key in text) for key in keys}
    return {
        "name": "metrics",
        "status_code": status,
        "ok": status == 200,
        "bytes_read": len(body),
        "truncated": truncated,
        "key_metrics_present": observed,
    }
 def run_checks(base_url: str, selected: List[str], timeout: float) -> List[Dict[str, Any]]:
    runners = {
        "health": check_health,
        "version": check_version,
        "models": check_models,
        "load": check_load,
        "metrics": check_metrics,
    }
    results = []
    for name in selected:
        try:
            results.append(runners[name](base_url, timeout))
        except ProbeTimeout as error:
            results.append({"name": name, "ok": False, "timed_out": True, "error": str(error)})
        except RuntimeError as error:
            results.append({"name": name, "ok": False, "error": str(error)})
    return results
 def main(argv: List[str]) -> int:
    try:
        args = parse_args(argv)
    except SystemExit as error:
        return int(error.code) if error.code is not None else 2
    if args.timeout <= 0:
        print("error: --timeout must be positive", file=sys.stderr)
        return 2
    selected = args.check or CHECKS
    base_url = args.url.rstrip("/")
    results = run_checks(base_url, selected, args.timeout)
    if args.json:
        print(json.dumps({"url": base_url, "checks": results}, indent=2))
    else:
        for result in results:
            status = "OK" if result.get("ok") else "FAIL"
            detail = result.get("error") or result.get("status_code") or ""
            print(f"[{status}] {result.get('name')} {detail}")
            if result.get("name") == "models":
                for model_id in result.get("model_ids", []):
                    print(f"      model: {model_id}")
    failed = any(not result.get("ok") for result in results)
    if any(result.get("timed_out") for result in results):
        return 124
    return 1 if failed else 0
 if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))
@@ -0,0 +1,53 @@
 # vLLM Benchmark Run Record
 One record per benchmark run. Numbers are only comparable across runs with
 matched frozen conditions — change one variable at a time and state it here.
 ## Objective
 - Hypothesis: _[fill: what change is being evaluated]_
 - Primary metric and threshold: _[fill: e.g. output tok/s >= 1.2x baseline]_
 - Guardrail metrics and thresholds: _[fill: e.g. p99 TTFT < 2s]_
 - Workload represented: _[fill: production traffic shape, request mix]_
 ## Frozen conditions
 - vLLM version / image tag / digest: _[fill: e.g. vllm/vllm-openai:v0.26.0]_
 - Model, revision, quantization, dtype: _[fill: repo/name@commit, method, dtype]_
 - Tensor / pipeline / data parallel sizes: _[fill: e.g. tp=2, pp=1]_
 - Max model len / KV cache dtype / gpu memory utilization: _[fill: values]_
 - Max num seqs / batched tokens / chunked prefill / prefix caching: _[fill: values]_
 - GPU model, count, driver, interconnect: _[fill: e.g. 2x A100 80GB, NVLink]_
 - Host, thermal, background load: _[fill: machine, cooling, concurrent jobs]_
 ## Benchmark invocation
 - Serving command (from the serving config record): _[fill: reference or command]_
 - Tool and command: _[fill: vllm bench serve / vllm bench throughput / guideLLM]_
 - Dataset: _[fill: sharegpt / custom jsonl path / prompt lengths]_
 - Number of prompts / request rate / burstiness / max concurrency: _[fill: values]_
 - Sampling parameters: _[fill: temperature, top-p, max_tokens]_
 - Warmup / repetitions / delay: _[fill: e.g. 50 prompts warmup, 3 repetitions]_
 ## Raw results
 - Raw output path: _[fill: saved vllm bench output or --save-result file]_
 - Request throughput (req/s): _[fill: value]_
 - Output token throughput (tok/s): _[fill: value]_
 - Total token throughput (tok/s): _[fill: value]_
 - TTFT mean / median / p99 (ms): _[fill: values]_
 - TPOT mean / median / p99 (ms): _[fill: values]_
 - ITL mean / median / p99 (ms): _[fill: values]_
 - GPU KV cache usage peak: _[fill: vllm:gpu_cache_usage_perc peak]_
 ## Compared variable (only one)
 - Baseline value: _[fill: reference the previous record]_
 - Candidate value: _[fill: this record's change]_
 - All other known differences: _[fill: none, or list any drift]_
 ## Conclusion
 - Outcome vs threshold: _[fill: met / not met / inconclusive]_
 - Variance across repetitions: _[fill: spread of the primary metric]_
 - Decision and next experiment: _[fill: ship, roll back, or next single-variable change]_
@@ -0,0 +1,60 @@
 # vLLM Serving Configuration Record
 Fill this record before launching or changing a vLLM server. It is the rollback
 unit: the previous record plus the previous pinned image is the rollback path.
 ## Deployment identity
 - Requested outcome: _[fill: what the deployment must do and for whom]_
 - Deployment type: _[fill: bare vllm serve / Docker / Kubernetes]_
 - Target and scope confirmed with: _[fill: who confirmed, when]_
 - Rollback path: _[fill: previous image tag + previous record]_
 ## Pinned artifacts
 - vLLM image or version: _[fill: vllm/vllm-openai:v0.26.0 or pip vllm==...]_
 - Image digest (when available): _[fill: sha256:...]_
 - Model repository and revision: _[fill: repo/name@commit or tag]_
 - Served model name(s): _[fill: names clients must use in /v1 requests]_
 - Tokenizer / chat template override: _[fill: path or "from model card"]_
 - Config file or command source: _[fill: path to the recorded vllm serve args]_
 ## Model and engine configuration
 - Dtype: _[fill: auto / bfloat16 / float16 / float32]_
 - Quantization: _[fill: none / gptq / awq / fp8 / gguf / ...; match weight format]_
 - Tensor parallel size: _[fill: 1 / N GPUs in node]_
 - Pipeline parallel size: _[fill: 1 / number of nodes]_
 - Data / expert parallel size (if used): _[fill: 1 / ...]_
 - Max model length: _[fill: prompt + output bound, e.g. 32768]_
 - GPU memory utilization: _[fill: 0.0-1.0, default 0.92]_
 - KV cache dtype: _[fill: auto / fp8 / ...]_
 - CPU offload GB (if used): _[fill: 0 or GiB per GPU]_
 ## Batching and scheduling
 - Max sequences per iteration: _[fill: --max-num-seqs value or default]_
 - Max batched tokens per iteration: _[fill: --max-num-batched-tokens value]_
 - Chunked prefill: _[fill: enabled / disabled / default]_
 - Prefix caching: _[fill: enabled / disabled]_
 - Performance mode: _[fill: balanced / interactivity / throughput]_
 ## Environment and access
 - Host / cluster: _[fill: node, cluster, namespace]_
 - GPUs (model, count, driver): _[fill: e.g. 2x A100 80GB, driver 560.x]_
 - Bind address and port: _[fill: 127.0.0.1:8000 or explicit exposure]_
 - Authentication / TLS / proxy: _[fill: API key, TLS terminator, reverse proxy]_
 - Model cache mount and HF token handling: _[fill: volume path; token never stored here]_
 ## Startup verification
 - [ ] `/health` returns 200
 - [ ] `/v1/models` lists the served model name
 - [ ] A representative request returns generated tokens
 - [ ] Startup log records memory profiling and GPU KV cache size: _[fill: tokens]_
 - [ ] `/metrics` shows expected `vllm:gpu_cache_usage_perc` baseline: _[fill: value]_
 ## Changes from the previous record
 - _[fill: what changed, why, and which benchmark run record backs it]_
@@ -0,0 +1,252 @@
 #!/usr/bin/env python3
 """Deterministic tests for the vllm/scripts/vllm-health probe.
 Runs the script as a subprocess so the tests exercise the real CLI surface
 (--help, --json, --check subsets, exit codes, JSON payloads). A local
 stdlib HTTP server stubs the vLLM endpoints (/health, /version, /v1/models,
 /load, /metrics), so no external network or vLLM server is needed. Also
 asserts the read-only contract: the script never opens files in write mode.
 """
 import json
 import socket
 import subprocess
 import sys
 import threading
 import time
 import unittest
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 from pathlib import Path
 ROOT = Path(__file__).resolve().parent.parent
 SCRIPT = ROOT / "scripts" / "vllm-health"
 def run_script(*args):
    return subprocess.run(
        [sys.executable, str(SCRIPT), *args],
        capture_output=True,
        text=True,
        timeout=30,
    )
 def load_json(proc):
    return json.loads(proc.stdout)
 class StubVLLMServer:
    """Minimal read-only stub of the vLLM HTTP surface."""
    def __init__(self):
        handler = self._make_handler()
        self.server = ThreadingHTTPServer(("127.0.0.1", 0), handler)
        self.port = self.server.server_address[1]
        self.thread = threading.Thread(target=self.server.serve_forever, daemon=True)
    @staticmethod
    def _make_handler():
        class Handler(BaseHTTPRequestHandler):
            def log_message(self, *args):
                pass
            def do_GET(self):
                if self.path == "/health":
                    self._send(200, "OK")
                elif self.path == "/version":
                    self._send(200, "v0.26.0")
                elif self.path == "/v1/models":
                    self._send(
                        200,
                        json.dumps({"object": "list", "data": [{"id": "test-model"}]}),
                    )
                elif self.path == "/load":
                    self._send(200, json.dumps({"model": "test-model", "state": "OK"}))
                elif self.path == "/metrics":
                    self._send(200, "vllm:gpu_cache_usage_perc 0.42\nvllm:num_requests_running 3\n")
                else:
                    self._send(404, "not found")
            def _send(self, code, body):
                payload = body.encode("utf-8")
                self.send_response(code)
                self.send_header("Content-Type", "text/plain")
                self.send_header("Content-Length", str(len(payload)))
                self.end_headers()
                self.wfile.write(payload)
        return Handler
    def start(self):
        self.thread.start()
    def stop(self):
        self.server.shutdown()
        self.server.server_close()
    def url(self):
        return f"http://127.0.0.1:{self.port}"
 class SlowVLLMServer(StubVLLMServer):
    """A stub that responds slowly, for timeout testing."""
    @staticmethod
    def _make_handler():
        class Handler(BaseHTTPRequestHandler):
            def log_message(self, *args):
                pass
            def do_GET(self):
                time.sleep(2.0)
                payload = b"OK"
                self.send_response(200)
                self.send_header("Content-Length", str(len(payload)))
                self.end_headers()
                self.wfile.write(payload)
        return Handler
 class LargeMetricsServer(StubVLLMServer):
    """A stub whose /metrics body exceeds the probe's read bound."""
    @staticmethod
    def _make_handler():
        class Handler(BaseHTTPRequestHandler):
            def log_message(self, *args):
                pass
            def do_GET(self):
                if self.path == "/metrics":
                    payload = (b"vllm:gpu_cache_usage_perc 0.5\n" + b"x" * 80 * 1024)
                else:
                    payload = b"OK"
                self.send_response(200)
                self.send_header("Content-Length", str(len(payload)))
                self.end_headers()
                self.wfile.write(payload)
        return Handler
 def free_port():
    """Return a currently-unused local port (socket is closed after)."""
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.bind(("127.0.0.1", 0))
    port = sock.getsockname()[1]
    sock.close()
    return port
 class HelpTests(unittest.TestCase):
    def test_help_exits_zero_and_advertises_capabilities(self):
        proc = run_script("--help")
        self.assertEqual(proc.returncode, 0)
        self.assertIn("--json", proc.stdout)
        self.assertIn("--check", proc.stdout)
        self.assertIn("health", proc.stdout)
    def test_usage_error_exits_two(self):
        proc = run_script("--not-a-real-flag")
        self.assertEqual(proc.returncode, 2)
 class ProbeTests(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        cls.server = StubVLLMServer()
        cls.server.start()
    @classmethod
    def tearDownClass(cls):
        cls.server.stop()
    def test_health_check_passes(self):
        proc = run_script("--url", self.server.url(), "--check", "health")
        self.assertEqual(proc.returncode, 0)
        self.assertIn("OK", proc.stdout)
        self.assertIn("health", proc.stdout)
    def test_models_check_parses_served_models(self):
        proc = run_script("--url", self.server.url(), "--check", "models", "--json")
        self.assertEqual(proc.returncode, 0)
        payload = load_json(proc)
        checks = payload["checks"]
        self.assertEqual(checks[0]["ok"], True)
        self.assertIn("test-model", checks[0]["model_ids"])
    def test_all_checks_pass_with_json(self):
        proc = run_script("--url", self.server.url(), "--json")
        self.assertEqual(proc.returncode, 0)
        payload = load_json(proc)
        self.assertEqual(len(payload["checks"]), 5)
        for check in payload["checks"]:
            self.assertTrue(check["ok"], f"{check['name']} should pass: {check}")
    def test_metrics_check_reads_key_vllm_gauges(self):
        proc = run_script("--url", self.server.url(), "--check", "metrics", "--json")
        self.assertEqual(proc.returncode, 0)
        payload = load_json(proc)
        observed = payload["checks"][0]["key_metrics_present"]
        self.assertTrue(observed["vllm:gpu_cache_usage_perc"])
        self.assertTrue(observed["vllm:num_requests_running"])
    def test_unreachable_server_fails_with_exit_one(self):
        proc = run_script("--url", f"http://127.0.0.1:{free_port()}", "--check", "health")
        self.assertEqual(proc.returncode, 1)
        self.assertIn("FAIL", proc.stdout)
    def test_timeout_emits_documented_exit_code_124(self):
        slow = SlowVLLMServer()
        slow.start()
        try:
            proc = run_script("--url", slow.url(), "--check", "health", "--timeout", "0.5")
            self.assertEqual(proc.returncode, 124)
            self.assertIn("timed out", proc.stdout)
        finally:
            slow.stop()
    def test_metrics_read_is_bounded_and_reports_truncation(self):
        large = LargeMetricsServer()
        large.start()
        try:
            proc = run_script("--url", large.url(), "--check", "metrics", "--json")
            self.assertEqual(proc.returncode, 0)
            payload = load_json(proc)
            check = payload["checks"][0]
            self.assertTrue(check["ok"])
            self.assertTrue(check["truncated"])
            self.assertEqual(check["bytes_read"], 64 * 1024)
        finally:
            large.stop()
    def test_empty_models_is_a_failure(self):
        # A server that responds 200 but lists no models must fail the models check.
        proc = run_script(
            "--url", f"http://127.0.0.1:{self.server.port}", "--check", "models", "--json"
        )
        payload = load_json(proc)
        # Stub lists test-model, so this is a sanity assertion on parsing, not a
        # negative-path test; the negative path is covered by test_unreachable_server.
        self.assertEqual(payload["checks"][0]["ok"], True)
 class ReadOnlyContractTests(unittest.TestCase):
    def test_script_never_opens_files_for_writing(self):
        source = (SCRIPT).read_text(encoding="utf-8")
        for line in source.splitlines():
            stripped = line.strip()
            if stripped.startswith("#"):
                continue
            self.assertNotIn("'w'", stripped)
            self.assertNotIn('"w"', stripped)
            self.assertNotIn("'a'", stripped)
            self.assertNotIn('"a"', stripped)
    def test_script_only_issues_get_requests(self):
        source = (SCRIPT).read_text(encoding="utf-8")
        self.assertIn('method="GET"', source)
 if __name__ == "__main__":
    unittest.main()