mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-11 19:47:12 +03:00
6181f1746d
* feat(skill): add vLLM inference-serving skill (#247) Add a single-tool vllm skill covering Docker/Kubernetes deployment, quantization-aware model configuration (tensor parallelism, KV cache), the OpenAI-compatible API surface, throughput/latency benchmarking, continuous batching tuning, GPU operation, and upgrade/rollback. Ships a read-only vllm-health probe (stdlib-only, --json), fillable serving-config and benchmark-run-record templates, seven dated references with upstream sources, a human-facing README, tests, and a schema-v1 eval manifest with six cases covering config, benchmarking, and troubleshooting. Route ml-engineering to the new skill via a resolvable link alongside llama-cpp, add the vllm entry to the top-level README index, and regenerate the tracked catalogs. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * fix(skill): emit timeout exit 124 and bound /metrics reads in vllm-health Address the review observations on the bundled probe: requests that exceed --timeout now raise ProbeTimeout and make the tool exit 124 as documented (previously they surfaced as exit 1), and the metrics check reads at most 64 KiB of /metrics and reports truncation instead of reading the whole body. Adds tests for both behaviors. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> --------- Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
3.7 KiB
3.7 KiB
vLLM Operations — Source Index
Last Updated: 2026-08-03
This index tracks the authoritative upstream sources behind the vLLM operational skill and the refresh procedure for keeping it current. vLLM moves fast — flags, defaults, and endpoint behavior change between releases; treat any claim here as version-sensitive and re-verify against the installed release.
Canonical sources
| Topic | Source |
|---|---|
| vLLM documentation (latest) | https://docs.vllm.ai/en/latest/ |
| vLLM documentation (stable release) | https://docs.vllm.ai/en/stable/ |
| Releases and release notes | https://github.com/vllm-project/vllm/releases |
| Docker deployment | https://docs.vllm.ai/en/latest/deployment/docker/ |
| Kubernetes deployment | https://docs.vllm.ai/en/latest/deployment/k8s/ |
| Online serving (OpenAI-compatible API) | https://docs.vllm.ai/en/latest/serving/online_serving/ |
| Engine arguments | https://docs.vllm.ai/en/latest/configuration/engine_args/ |
| Quantization | https://docs.vllm.ai/en/latest/features/quantization/index.html |
Benchmark CLI (vllm bench) |
https://docs.vllm.ai/en/latest/benchmarking/cli/ |
| vLLM paper (PagedAttention, SOSP 2023) | https://arxiv.org/abs/2309.06180 |
| Continuous batching explainer (Anyscale, by Cade Daniel et al.) | https://www.anyscale.com/blog/continuous-batching-llm-inference |
Version observations (as of this refresh)
- Latest release: v0.26.0 (published 2026-07-27). Release cadence is roughly
monthly; docs publish
stableandlateststreams plus per-version archives. - The official Docker image is
vllm/vllm-openaion Docker Hub, withvllm/vllm-openai-rocm(AMD) andvllm/vllm-openai-xpu(Intel) variants; the XPU image is official starting with v0.26.0. - Engine argument documentation moved to
configuration/engine_args; engine args are also available as JSON-style CLI arguments (--json-arg.key value). - Benchmarking is now a first-class CLI:
vllm bench serve(online serving),vllm bench throughput(offline), plus latency-focused and multimodal variants, replacing the olderbenchmark_serving.py/benchmark_throughput.pyscripts. The docs recommend the external GuideLLM framework for production capacity testing. --prefix-cachingand--performance-mode(balanced/interactivity/throughput) are current flags on thevllm servecommand line; older doc pages may still show earlier spellings.- Default
--gpu-memory-utilizationis 0.92 (per instance).--kv-cache-dtypeacceptsauto,bfloat16,float16,fp8(fp8_e4m3),int8_per_token_head,nvfp4, and other hardware-specific values on CUDA 11.8+. - vLLM 0.26.0 dependencies include Transformers 5.13, FlashInfer 0.6.14, and NIXL 1.3.1; GPU support spans NVIDIA (Ampere+ for most quantized kernels), AMD ROCm, and Intel XPU, with a CPU backend for testing.
Refresh procedure
- Re-check the sources above for a new release and read its release notes for changed or removed engine arguments and changed defaults.
- Update the version observations that changed (defaults, flag names, endpoint behavior, hardware support).
- Re-verify the SKILL.md scope keyword sweep and the routing links to
ml-engineeringandllama-cppstill resolve. - Re-run the bundled probe against a test server and confirm every check still
parses:
scripts/vllm-health --url http://127.0.0.1:8000 --json.
Related skill sources
ml-engineeringowns serving methodology: engine selection, quantization decisions, deployment plans, and regression triage. Its references are the source for engine-spanning decisions; this skill covers vLLM operation itself.llama-cppowns the llama.cpp stack (GGUF, llama-server). Do not duplicate its content here.