magnus919_agent-skills/vllm

vLLM — Inference-Serving Skill

Operate, configure, benchmark, and troubleshoot vLLM inference servers: Docker and Kubernetes deployment, quantization-aware model configuration (tensor parallelism, KV cache), the OpenAI-compatible API surface, throughput/latency benchmarking, continuous batching tuning, GPU operation, and upgrade/rollback.

Why Install This Skill

Your agent can run a vLLM deployment instead of guessing. Teams that self-serve open models in production need someone (or something) that knows how to start a vllm serve with the right flags, size the model and its KV cache for the GPUs at hand, confirm the OpenAI-compatible endpoints actually work, measure throughput and latency with evidence that comparisons mean something, tune continuous batching one knob at a time, and upgrade or roll back without burning the deployment.

This skill ships that operating knowledge plus two fillable templates — a serving configuration record (so every deployment is reproducible) and a benchmark run record (so every performance claim is comparable) — and a read-only vllm-health probe that checks a running server's health, version, models, load, and metrics over HTTP without changing anything. The references are distilled from the official vLLM documentation with dated sources. Serving strategy and engine-selection methodology deliberately route up to ml-engineering; the llama.cpp stack routes to llama-cpp; this skill owns the day-to-day operation of vLLM itself.

What You Get

Directory	Purpose
SKILL.md	Agent-facing operating loop, mutation gates, and verification boundaries
references/	Seven dated, source-indexed references: source index, deployment, model configuration, OpenAI API, benchmarking, batching/tuning, GPU ops and lifecycle
templates/serving-config.md	Fillable record of every serving argument, model revision, and environment — the rollback unit
templates/benchmark-run-record.md	Fillable record that makes throughput/latency evidence comparable across runs
scripts/vllm-health	Read-only probe: health, version, models, load, and metrics; stdlib-only, --json, --help without a server
tests/	Deterministic tests against a local stub HTTP server, including the read-only contract
evals/evals.json	Six output-quality evaluation cases for agent runs

Quick Start

# Help works with no vLLM server
scripts/vllm-health --help

# Probe a running server, machine-readable
scripts/vllm-health --url http://127.0.0.1:8000 --json

# Targeted checks
scripts/vllm-health --check health --check models --json

# Record what you are about to run before you run it
# (fill in vllm/templates/serving-config.md), then:
docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 8000:8000 --ipc=host \
  vllm/vllm-openai:v0.26.0 \
  --model <model-name> --served-model-name <api-name> --max-model-len <len>

# Benchmark serving throughput/latency once the server is ready
vllm bench serve --backend vllm --model <model-name> \
  --endpoint /v1/completions --dataset-name custom \
  --dataset-path prompts.jsonl --num-prompts 100 --request-rate inf

The vllm-health script uses only Python's standard library and issues GET requests only. Exit codes: 0 all checks passed, 1 issues found or a fatal error, 2 usage error, 124 timeout. Benchmark output includes request throughput (req/s), output token throughput (tok/s), and TTFT/TPOT/ITL percentiles — record them in templates/benchmark-run-record.md.

Triggers

Load this skill for vLLM operations: deploying or updating a vllm serve server (bare, Docker, or Kubernetes), choosing serving flags (--quantization, --tensor-parallel-size, --max-model-len, --kv-cache-dtype, --gpu-memory-utilization), wiring or debugging the OpenAI-compatible API surface (/v1/chat/completions, /v1/completions, /v1/models, /health, tool calling, chat templates), measuring serving throughput or latency (vllm bench serve/bench throughput), tuning continuous batching, running or diagnosing GPUs under a vLLM workload, or planning a vLLM upgrade or rollback. Do not load it for model training or fine-tuning (that's ml-engineering), for the llama.cpp stack (that's llama-cpp), or for generic Kubernetes/Docker administration (that's kubernetes/docker-compose).

Requirements

• A vLLM release: the vllm/vllm-openai Docker image (NVIDIA CUDA, AMD ROCm, or Intel XPU variants) or pip install vllm==<pinned-version>.
• An accelerator with the matching driver (NVIDIA with the NVIDIA Container Toolkit for Docker, or the platform equivalent), or a supported CPU build for testing.
• Hugging Face access to the model: a mounted ~/.cache/huggingface and an HF_TOKEN for gated models.
• Python 3.9+ for the vllm-health script (--help needs nothing else); live probes need HTTP(S) access to the running server.