mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-16 22:16:52 +03:00
Magnus HedemarkandGitHub
c7c4d3b74f
Engineering: backend-engineering, frontend-engineering, data-engineering, ml-engineering, platform-engineering, qa-methodology Executive: go-to-market, legal-strategy, operational-design, org-design, product-strategy ml-engineering: added missing training-infrastructure.md reference qa-methodology: added test-data-management, performance-testing, security-testing references All frontmatter converted to agent-skills convention. Source: https://github.com/magnus919/hermes-profiles
32 lines
1.3 KiB
Markdown
32 lines
1.3 KiB
Markdown
# Model Evaluation
|
|
|
|
## Benchmark Selection
|
|
|
|
| What you want to measure | Recommended benchmarks |
|
|
|--------------------------|----------------------|
|
|
| General reasoning | MMLU-Pro, GPQA, ARC-Challenge |
|
|
| Code generation | HumanEval+, SWE-Bench, BigCodeBench |
|
|
| Instruction following | MT-Bench, AlpacaEval, Arena-Hard |
|
|
| Tool calling | BFCL, ToolBench, jdhodges tool-call eval |
|
|
| Safety | TruthfulQA, BBQ, ToxiGen |
|
|
| Math | GSM8K, MATH, AIME |
|
|
|
|
## Custom Eval Design
|
|
|
|
When off-the-shelf benchmarks don't capture your domain:
|
|
|
|
1. **Collect 50-200 representative examples** from your actual use cases
|
|
2. **Define a scoring rubric** — what constitutes correct, partially correct, and incorrect
|
|
3. **Include adversarial examples** — edge cases, ambiguous inputs, known failure modes
|
|
4. **Run baseline (base model) first** — establish the ceiling before fine-tuning
|
|
5. **Track per-example** — aggregate scores hide regressions in specific capabilities
|
|
|
|
## Regression Tracking
|
|
|
|
| Before change | After change | Interpretation |
|
|
|---|---|---|
|
|
| Score A | Score A ± noise | No detectable effect |
|
|
| Score A | Score A - delta | Regression — investigate |
|
|
| Score A | Score A + delta | Improvement — verify on held-out set |
|
|
| Score A, B regressed | Score A + delta, B regressed | Tradeoff — intentional? |
|