mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-12 12:06:29 +03:00
Magnus HedemarkandGitHub
c7c4d3b74f
Engineering: backend-engineering, frontend-engineering, data-engineering, ml-engineering, platform-engineering, qa-methodology Executive: go-to-market, legal-strategy, operational-design, org-design, product-strategy ml-engineering: added missing training-infrastructure.md reference qa-methodology: added test-data-management, performance-testing, security-testing references All frontmatter converted to agent-skills convention. Source: https://github.com/magnus919/hermes-profiles
1.3 KiB
1.3 KiB
Model Evaluation
Benchmark Selection
| What you want to measure | Recommended benchmarks |
|---|---|
| General reasoning | MMLU-Pro, GPQA, ARC-Challenge |
| Code generation | HumanEval+, SWE-Bench, BigCodeBench |
| Instruction following | MT-Bench, AlpacaEval, Arena-Hard |
| Tool calling | BFCL, ToolBench, jdhodges tool-call eval |
| Safety | TruthfulQA, BBQ, ToxiGen |
| Math | GSM8K, MATH, AIME |
Custom Eval Design
When off-the-shelf benchmarks don't capture your domain:
- Collect 50-200 representative examples from your actual use cases
- Define a scoring rubric — what constitutes correct, partially correct, and incorrect
- Include adversarial examples — edge cases, ambiguous inputs, known failure modes
- Run baseline (base model) first — establish the ceiling before fine-tuning
- Track per-example — aggregate scores hide regressions in specific capabilities
Regression Tracking
| Before change | After change | Interpretation |
|---|---|---|
| Score A | Score A ± noise | No detectable effect |
| Score A | Score A - delta | Regression — investigate |
| Score A | Score A + delta | Improvement — verify on held-out set |
| Score A, B regressed | Score A + delta, B regressed | Tradeoff — intentional? |