magnus919_agent-skills/llama-cpp/templates/benchmark-comparison.md

llama.cpp Benchmark Comparison

Objective

• Hypothesis:
• Primary metric and threshold:
• Guardrail metrics and thresholds:
• Workload represented:

Frozen conditions

• Host / power / thermal / background load:
• llama.cpp commit and build options:
• Driver / backend / devices / interconnect:
• Model file hash / quant / template / sidecars:
• Prompt and output lengths / context depth:
• Context / batch / ubatch / threads / slots:
• Cache / Flash Attention / offload / split:
• Warmup / repetitions / delay:

Compared variable

• Baseline value:
• Candidate value:
• All other known differences:

Commands and raw artifacts

• Baseline command/output path:
• Candidate command/output path:
• End-to-end or server command/output path:
• Quality fixture/results:

Results

Metric	Baseline	Candidate	Delta	Variance / confidence	Threshold
Prompt processing
Token generation
TTFT
End-to-end latency
Throughput
Peak RAM
Peak device memory
Task quality / success
Error rate

Verdict

• Accept / reject / inconclusive:
• Evidence:
• Confounders or unverified boundaries:
• Rollback or next experiment: