mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-14 21:16:36 +03:00
Magnus Hedemark
fe5b275d00
Greenfield SkillOpt: 3 epochs for deepset Haystack skill. Pipeline DAG model, document stores, retrievers, evaluation, deployment. Epoch 1 — Prominence: Hard-gate on Pipeline DAG vs LCEL pipe model Epoch 2 — Decision Guidance: Where to Start, Framework Routing Guide Epoch 3 — Pattern Expansion: Hybrid RAG pattern, evaluation pipeline, deployment 11 files: SKILL.md, 6 references, 3 templates, 1 script.
54 lines
1.7 KiB
Markdown
54 lines
1.7 KiB
Markdown
# Haystack Evaluation
|
|
|
|
## Evaluation Pipeline
|
|
|
|
Evaluation in Haystack is a pipeline itself — add evaluator components to measure your pipeline's outputs.
|
|
|
|
```python
|
|
from haystack import Pipeline
|
|
from haystack.components.evaluators import DeepEvalEvaluator, DeepEvalMetric, SASEvaluator
|
|
|
|
eval_pipeline = Pipeline()
|
|
eval_pipeline.add_component("faithfulness", DeepEvalEvaluator(
|
|
metric=DeepEvalMetric.FAITHFULNESS,
|
|
metric_params={"model": "gpt-4o-mini"}
|
|
))
|
|
```
|
|
|
|
## Available Evaluators
|
|
|
|
| Evaluator | What it measures | Type |
|
|
|-----------|-----------------|------|
|
|
| `DeepEvalEvaluator` | Faithfulness, relevancy, context recall | LLM-as-judge |
|
|
| `SASEvaluator` | Semantic answer similarity | Embedding-based |
|
|
| `LLMEvaluator` | Custom criteria via instruction + examples | LLM-as-judge |
|
|
| `DocumentMAPEvaluator` | Mean average precision for retrieval | Statistical |
|
|
|
|
## Evaluation Workflow
|
|
|
|
```python
|
|
from haystack import Pipeline
|
|
from haystack.components.evaluators import SASEvaluator
|
|
|
|
# Run your query pipeline
|
|
results = query_pipeline.run(...)
|
|
|
|
# Build evaluation pipeline
|
|
eval_pipeline = Pipeline()
|
|
eval_pipeline.add_component("sa_eval", SASEvaluator())
|
|
eval_result = eval_pipeline.run({
|
|
"sa_eval": {
|
|
"predicted_answers": [results["generator"]["replies"][0]],
|
|
"golden_answers": ["Expected answer text"]
|
|
}
|
|
})
|
|
print(eval_result["sa_eval"]["score"])
|
|
```
|
|
|
|
## Best Practices
|
|
|
|
- Evaluate on a held-out golden dataset (not your training queries)
|
|
- Use multiple metrics — faithfulness catches hallucinations, relevancy catches retrieval misses
|
|
- Build evaluation into CI/CD for regression detection
|
|
- For production, schedule periodic evaluation runs against new data
|