mirror of https://github.com/magnus919/agent-skills.git synced 2026-09-11 19:47:12 +03:00

Files

T

Magnus Hedemark fe5b275d00 feat: add haystack — expert skill for production search pipelines

Greenfield SkillOpt: 3 epochs for deepset Haystack skill.
Pipeline DAG model, document stores, retrievers, evaluation, deployment.

Epoch 1 — Prominence: Hard-gate on Pipeline DAG vs LCEL pipe model
Epoch 2 — Decision Guidance: Where to Start, Framework Routing Guide
Epoch 3 — Pattern Expansion: Hybrid RAG pattern, evaluation pipeline, deployment

11 files: SKILL.md, 6 references, 3 templates, 1 script.

2026-07-09 14:53:43 -04:00

1.7 KiB

Raw Blame History

Haystack Evaluation

Evaluation Pipeline

Evaluation in Haystack is a pipeline itself — add evaluator components to measure your pipeline's outputs.

from haystack import Pipeline
from haystack.components.evaluators import DeepEvalEvaluator, DeepEvalMetric, SASEvaluator

eval_pipeline = Pipeline()
eval_pipeline.add_component("faithfulness", DeepEvalEvaluator(
    metric=DeepEvalMetric.FAITHFULNESS,
    metric_params={"model": "gpt-4o-mini"}
))

Available Evaluators

Evaluator	What it measures	Type
`DeepEvalEvaluator`	Faithfulness, relevancy, context recall	LLM-as-judge
`SASEvaluator`	Semantic answer similarity	Embedding-based
`LLMEvaluator`	Custom criteria via instruction + examples	LLM-as-judge
`DocumentMAPEvaluator`	Mean average precision for retrieval	Statistical

Evaluation Workflow

from haystack import Pipeline
from haystack.components.evaluators import SASEvaluator

# Run your query pipeline
results = query_pipeline.run(...)

# Build evaluation pipeline
eval_pipeline = Pipeline()
eval_pipeline.add_component("sa_eval", SASEvaluator())
eval_result = eval_pipeline.run({
    "sa_eval": {
        "predicted_answers": [results["generator"]["replies"][0]],
        "golden_answers": ["Expected answer text"]
    }
})
print(eval_result["sa_eval"]["score"])

Best Practices

Evaluate on a held-out golden dataset (not your training queries)
Use multiple metrics — faithfulness catches hallucinations, relevancy catches retrieval misses
Build evaluation into CI/CD for regression detection
For production, schedule periodic evaluation runs against new data

1.7 KiB Raw Blame History