mirror of https://github.com/magnus919/agent-skills.git synced 2026-09-11 19:47:12 +03:00

Files

T

6181f1746d feat(skill): add vLLM inference-serving skill (#247 ) (#267 )

* feat(skill): add vLLM inference-serving skill (#247)

Add a single-tool vllm skill covering Docker/Kubernetes deployment,
quantization-aware model configuration (tensor parallelism, KV cache),
the OpenAI-compatible API surface, throughput/latency benchmarking,
continuous batching tuning, GPU operation, and upgrade/rollback.

Ships a read-only vllm-health probe (stdlib-only, --json), fillable
serving-config and benchmark-run-record templates, seven dated
references with upstream sources, a human-facing README, tests, and a
schema-v1 eval manifest with six cases covering config, benchmarking,
and troubleshooting.

Route ml-engineering to the new skill via a resolvable link alongside
llama-cpp, add the vllm entry to the top-level README index, and
regenerate the tracked catalogs.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

* fix(skill): emit timeout exit 124 and bound /metrics reads in vllm-health

Address the review observations on the bundled probe: requests that exceed
--timeout now raise ProbeTimeout and make the tool exit 124 as documented
(previously they surfaced as exit 1), and the metrics check reads at most
64 KiB of /metrics and reports truncation instead of reading the whole body.
Adds tests for both behaviors.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

---------

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

2026-08-03 19:31:36 -04:00

2.6 KiB

Raw Permalink Blame History

vLLM Serving Configuration Record

Fill this record before launching or changing a vLLM server. It is the rollback unit: the previous record plus the previous pinned image is the rollback path.

Deployment identity

Requested outcome: [fill: what the deployment must do and for whom]
Deployment type: [fill: bare vllm serve / Docker / Kubernetes]
Target and scope confirmed with: [fill: who confirmed, when]
Rollback path: [fill: previous image tag + previous record]

Pinned artifacts

vLLM image or version: [fill: vllm/vllm-openai:v0.26.0 or pip vllm==...]
Image digest (when available): [fill: sha256:...]
Model repository and revision: [fill: repo/name@commit or tag]
Served model name(s): [fill: names clients must use in /v1 requests]
Tokenizer / chat template override: [fill: path or "from model card"]
Config file or command source: [fill: path to the recorded vllm serve args]

Model and engine configuration

Dtype: [fill: auto / bfloat16 / float16 / float32]
Quantization: [fill: none / gptq / awq / fp8 / gguf / ...; match weight format]
Tensor parallel size: [fill: 1 / N GPUs in node]
Pipeline parallel size: [fill: 1 / number of nodes]
Data / expert parallel size (if used): [fill: 1 / ...]
Max model length: [fill: prompt + output bound, e.g. 32768]
GPU memory utilization: [fill: 0.0-1.0, default 0.92]
KV cache dtype: [fill: auto / fp8 / ...]
CPU offload GB (if used): [fill: 0 or GiB per GPU]

Batching and scheduling

Max sequences per iteration: [fill: --max-num-seqs value or default]
Max batched tokens per iteration: [fill: --max-num-batched-tokens value]
Chunked prefill: [fill: enabled / disabled / default]
Prefix caching: [fill: enabled / disabled]
Performance mode: [fill: balanced / interactivity / throughput]

Environment and access

Host / cluster: [fill: node, cluster, namespace]
GPUs (model, count, driver): [fill: e.g. 2x A100 80GB, driver 560.x]
Bind address and port: [fill: 127.0.0.1:8000 or explicit exposure]
Authentication / TLS / proxy: [fill: API key, TLS terminator, reverse proxy]
Model cache mount and HF token handling: [fill: volume path; token never stored here]

Startup verification

/health returns 200
/v1/models lists the served model name
A representative request returns generated tokens
Startup log records memory profiling and GPU KV cache size: [fill: tokens]
/metrics shows expected vllm:gpu_cache_usage_perc baseline: [fill: value]

Changes from the previous record

[fill: what changed, why, and which benchmark run record backs it]

2.6 KiB Raw Permalink Blame History