mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-17 14:36:29 +03:00
Magnus Hedemark
95046675bc
Greenfield SkillOpt: 3 epochs for a Stanford DSPy compiler skill. DSPy is a fundamentally different paradigm from chain/RAG frameworks. Epoch 1 — Prominence: - Hard-gate blockquote: 'DSPy is NOT a chain framework' - Core Paradigm section with runnable code example early Epoch 2 — Decision Guidance: - Framework Routing Guide (DSPy vs LlamaIndex vs LangChain vs LangGraph) - Where to Start table mapping entry points - Troubleshooting table with reference links Epoch 3 — Pattern Expansion: - Optimizer selection cheat sheet from official docs - Caching, compilation cost management, save/load - FAQ covering paradigm confusion, errors, deployment 12 files: SKILL.md, 7 references, 3 templates, 1 script. v1.0.0 -> v1.0.3 across 3 epochs. All API surfaces validated against dspy.ai official docs — optimizer selection guide, caching, core modules, FAQ. Signed-off-by: Jasper <jasper@montcastle.bitches>
1.8 KiB
1.8 KiB
DSPy Evaluation
Defining Metrics
Metrics are Python functions. They can return bool, int, or float.
def exact_match(example, pred, trace=None):
return example.answer == pred.answer
def f1_score(example, pred, trace=None):
pred_tokens = set(pred.answer.split())
gold_tokens = set(example.answer.split())
if not pred_tokens or not gold_tokens:
return 0.0
precision = len(pred_tokens & gold_tokens) / len(pred_tokens)
recall = len(pred_tokens & gold_tokens) / len(gold_tokens)
return 2 * precision * recall / (precision + recall) if (precision + recall) else 0.0
dspy.Evaluate
from dspy.evaluate import Evaluate
evaluator = Evaluate(devset=devset, metric=exact_match, num_threads=8)
score = evaluator(compiled_program)
print(f"Accuracy: {score}")
Metrics with Feedback (for GEPA)
GEPA is the only optimizer that reads Prediction(score, feedback):
def metric_with_feedback(example, pred, trace=None):
score = 1.0 if pred.answer == example.answer else 0.0
feedback = "Correct" if score else f"Expected: {example.answer}, Got: {pred.answer}"
return dspy.Prediction(score=score, feedback=feedback)
Creating Datasets
from dspy.datasets import DataLoader
# From list of dicts
trainset = [dspy.Example(question="Q1", answer="A1").with_inputs("question")]
devset = [dspy.Example(question="Q2", answer="A2").with_inputs("question")]
# Or use DataLoader for common formats
dl = DataLoader()
dataset = dl.from_json("data.json")
Best Practices
- Split trainset and devset/valet — never optimize on your evaluation set
- Use float metrics for fine-grained optimization signal
- For GEPA, always provide feedback in the metric
- Start with exact_match, graduate to semantic similarity for subjective tasks
- Parallelize evaluation with
num_threads