mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-16 14:06:27 +03:00
92299e1238
Add a schema-valid eval manifest (6 cases: fine-tuning plan review, eval-set design, quantization decision, deployment plan, regression triage, training-run reproducibility), three fillable templates (training-run record, eval regression table, quantization decision record), a stdlib eval-set overlap/leakage checker with a unittest suite, routing to the llama-cpp tool skill, and a README Quick Start documenting the script. Closes #240. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
38 lines
2.2 KiB
Markdown
38 lines
2.2 KiB
Markdown
# ML Engineering
|
|
|
|
Machine learning engineering methodology — model training, fine-tuning (LoRA/QLoRA), evaluation, quantization, deployment, and MLOps pipeline design. Grounded in practical engineering patterns for production ML systems.
|
|
|
|
## Why Install This Skill
|
|
|
|
Your agent makes informed decisions about fine-tuning approaches, quantization trade-offs, GPU selection, and serving architecture with real VRAM budgets and benchmarks. Fillable templates turn training runs, eval comparisons, and quantization decisions into reviewable records, and the bundled eval-overlap checker catches train/eval contamination before it invalidates a benchmark.
|
|
|
|
## What You Get
|
|
|
|
| Directory | Purpose |
|
|
|-----------|---------|
|
|
| `SKILL.md` | Core methodology, trigger conditions, reference index |
|
|
| `references/` | Deep-dive reference files loaded on demand |
|
|
| `templates/` | Fillable records: training-run record, eval regression table, quantization decision record |
|
|
| `scripts/` | `check-eval-overlap.py` — detects test-set leakage between train and eval corpora |
|
|
| `evals/` | Output-quality eval manifest for the skill's methodology cases |
|
|
|
|
## Triggers
|
|
|
|
Setting up fine-tuning runs, quantizing models, selecting training infrastructure, deploying inference servers, evaluating model quality, or triaging a model regression.
|
|
|
|
## Requirements
|
|
|
|
Assumes familiarity with PyTorch/HuggingFace ecosystem. References cover vLLM, llama.cpp, TGI, DeepSpeed, and accelerate. The bundled script needs only Python 3 (standard library).
|
|
|
|
## Quick Start
|
|
|
|
Check an eval corpus for leakage against your training data before trusting any eval score:
|
|
|
|
```bash
|
|
python3 ml-engineering/scripts/check-eval-overlap.py --train data/train/ --eval data/eval/
|
|
```
|
|
|
|
Each eval file is reported with its overlap fraction against the training corpus; an eval file that shares more than 10% of its text with training is flagged `LEAK` and the script exits 1, so it can gate a CI pipeline. Add `--json` for machine-readable output, `--token-ngram 5` to compare token sequences instead of character shingles, and `--max-overlap-fraction 0.05` to tighten the threshold.
|
|
|
|
Load SKILL.md for the methodology overview and reference table, then load specific references or templates as needed for the task at hand.
|