magnus919_agent-skills/agent-evals-and-observability/references/evaluation-design.md

Evaluation Design

Write a task contract: inputs and context, permitted tools and permissions, expected outcome, prohibited outcomes, environment assumptions, and observable completion evidence. Add a trajectory contract when the path matters: eligible tool selection and arguments, authorization checks, state transitions, grounding sources, recovery behavior, stopping/escalation, and permitted side effects.

Select evidence rather than requiring every evaluation type:

Evidence	Useful when	Boundary
Unit	A prompt formatter, tool adapter, parser, or guard can be isolated	Does not establish integration behavior
Component	Router, retriever, policy layer, or tool boundary interacts internally	May hide production environment behavior
Scenario	A complete task must run under controlled conditions	Results depend on fixture fidelity
Regression	A known behavior or escaped failure must remain protected	Cannot cover unknown failures
Adversarial	Misuse, injection, unsafe actions, or leakage are plausible	Challenge coverage is never exhaustive
Online	Distribution shift, service variability, or user outcomes matter	Confounding and consent constrain interpretation
Human review	Domain or subjective criteria cannot be directly automated	Review remains rubric- and context-sensitive

Use synthetic, curated, replayed-production, adversarial, and regression cases as distinct declared sources. Choose a mix based on task harm, reversibility, novelty, external side effects, and availability of valid evidence.