mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-11 19:47:12 +03:00
6181f1746d
* feat(skill): add vLLM inference-serving skill (#247) Add a single-tool vllm skill covering Docker/Kubernetes deployment, quantization-aware model configuration (tensor parallelism, KV cache), the OpenAI-compatible API surface, throughput/latency benchmarking, continuous batching tuning, GPU operation, and upgrade/rollback. Ships a read-only vllm-health probe (stdlib-only, --json), fillable serving-config and benchmark-run-record templates, seven dated references with upstream sources, a human-facing README, tests, and a schema-v1 eval manifest with six cases covering config, benchmarking, and troubleshooting. Route ml-engineering to the new skill via a resolvable link alongside llama-cpp, add the vllm entry to the top-level README index, and regenerate the tracked catalogs. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * fix(skill): emit timeout exit 124 and bound /metrics reads in vllm-health Address the review observations on the bundled probe: requests that exceed --timeout now raise ProbeTimeout and make the tool exit 124 as documented (previously they surfaced as exit 1), and the metrics check reads at most 64 KiB of /metrics and reports truncation instead of reading the whole body. Adds tests for both behaviors. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> --------- Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2.4 KiB
2.4 KiB
vLLM Benchmark Run Record
One record per benchmark run. Numbers are only comparable across runs with matched frozen conditions — change one variable at a time and state it here.
Objective
- Hypothesis: [fill: what change is being evaluated]
- Primary metric and threshold: [fill: e.g. output tok/s >= 1.2x baseline]
- Guardrail metrics and thresholds: [fill: e.g. p99 TTFT < 2s]
- Workload represented: [fill: production traffic shape, request mix]
Frozen conditions
- vLLM version / image tag / digest: [fill: e.g. vllm/vllm-openai:v0.26.0]
- Model, revision, quantization, dtype: [fill: repo/name@commit, method, dtype]
- Tensor / pipeline / data parallel sizes: [fill: e.g. tp=2, pp=1]
- Max model len / KV cache dtype / gpu memory utilization: [fill: values]
- Max num seqs / batched tokens / chunked prefill / prefix caching: [fill: values]
- GPU model, count, driver, interconnect: [fill: e.g. 2x A100 80GB, NVLink]
- Host, thermal, background load: [fill: machine, cooling, concurrent jobs]
Benchmark invocation
- Serving command (from the serving config record): [fill: reference or command]
- Tool and command: [fill: vllm bench serve / vllm bench throughput / guideLLM]
- Dataset: [fill: sharegpt / custom jsonl path / prompt lengths]
- Number of prompts / request rate / burstiness / max concurrency: [fill: values]
- Sampling parameters: [fill: temperature, top-p, max_tokens]
- Warmup / repetitions / delay: [fill: e.g. 50 prompts warmup, 3 repetitions]
Raw results
- Raw output path: [fill: saved vllm bench output or --save-result file]
- Request throughput (req/s): [fill: value]
- Output token throughput (tok/s): [fill: value]
- Total token throughput (tok/s): [fill: value]
- TTFT mean / median / p99 (ms): [fill: values]
- TPOT mean / median / p99 (ms): [fill: values]
- ITL mean / median / p99 (ms): [fill: values]
- GPU KV cache usage peak: [fill: vllm:gpu_cache_usage_perc peak]
Compared variable (only one)
- Baseline value: [fill: reference the previous record]
- Candidate value: [fill: this record's change]
- All other known differences: [fill: none, or list any drift]
Conclusion
- Outcome vs threshold: [fill: met / not met / inconclusive]
- Variance across repetitions: [fill: spread of the primary metric]
- Decision and next experiment: [fill: ship, roll back, or next single-variable change]