diff --git a/AGENTS.md b/AGENTS.md
index c45b985..4e7f7a5 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -125,7 +125,8 @@ When the user mentions these keywords, load the corresponding skill:
 | "color management", "ICC profile", "sRGB", "CIELAB", "color space", "gamut", "working space", "well-behaved profile", "ArgyllCMS", "xicclu", "color difference", "dE", "soft proof", "display referred", "scene referred" | [color-management](color-management/SKILL.md) |
 | "crowdsec", "CrowdSec", "IPS", "IDPS", "WAF", "intrusion prevention", "bouncer", "cscli", "appsec", "LAPI", "remediation component", "community blocklist", "fail2ban", "security engine" | [crowdsec](crowdsec/SKILL.md) |
 | "open source", "contributing", "how to contribute", "submit a PR", "file an issue", "CONTRIBUTING.md", "bug report template", "PR template" | [opensource-contributions](opensource-contributions/SKILL.md) |
-| "pydanticai", "pydantic AI", "pydantic graph", "AI agent", "LLM agent", "agent framework", "function tool", "tool-using agent", "agent with tools", "agent with dependencies", "structured output", "streaming agent", "agent graph", "state machine graph", "GraphBuilder", "BaseNode", "multi-agent", "agent delegation", "LLM evals", "model testing", "TestModel", "FunctionModel", "capabilities" | [pydanticai](pydanticai/SKILL.md) |
+| "agent eval", "agent evaluation", "LLM eval", "LLM evals", "evaluation dataset", "grader calibration", "model judge", "trajectory review", "agent observability", "agent traces", "agent telemetry", "agent regression", "agent release gate", "privacy-aware telemetry", "prompt evaluation", "model testing" | [agent-evals-and-observability](agent-evals-and-observability/SKILL.md) |
+| "pydanticai", "pydantic AI", "pydantic graph", "AI agent", "LLM agent", "agent framework", "function tool", "tool-using agent", "agent with tools", "agent with dependencies", "structured output", "streaming agent", "agent graph", "state machine graph", "GraphBuilder", "BaseNode", "multi-agent", "agent delegation", "TestModel", "FunctionModel", "capabilities" | [pydanticai](pydanticai/SKILL.md) |
 | "product discovery", "stakeholder interview", "requirements discovery", "user research", "customer interview", "requirements gathering", "discovery phase", "stakeholder mapping", "interview guide", "discovery conversation", "transcript to spec", "requirements conflict", "what would have to be true", "pre-mortem", "laddering", "assumption busting" | [product-discovery](product-discovery/SKILL.md) |
 | "prioritize", "RICE", "MoSCoW", "opportunity solution tree", "decision log", "write a spec", "product spec", "PRD", "stakeholder communication", "executive brief", "feature prioritization", "backlog ranking", "release scope", "build vs buy", "product decision" | [product-methodology](product-methodology/SKILL.md) |
 | "workflow", "figure out my workflow", "analyze my process", "what do I actually do", "catalog my workflow", "formalize my process", "workflow architect", "onboard me to my own process" | [workflow-architect](bundles/workflow-architect/SKILL.md) |
diff --git a/README.md b/README.md
index e98a696..e677822 100644
--- a/README.md
+++ b/README.md
@@ -12,6 +12,10 @@ Preserve the reasoning behind consequential architecture choices so later contri
 
 Multi-agent structured debate system — spawn a panel of expert agents to debate any question with convergence-aware iteration. Produces a decision landscape with confidence diagnostics, shared risks, remaining disagreements, and a principal's path narrative. Ships as a pip-installable Python package built on PydanticAI + PydanticGraph. `pip install pydantic-ai && pip install agent-council` and one API key to run your first debate.
 
+### [agent-evals-and-observability](agent-evals-and-observability/SKILL.md)
+
+Design and review framework-neutral AI-agent evaluations, trajectory evidence, risk-tiered release gates, and privacy-aware observability. Covers datasets, grader calibration, statistical comparisons, incident-to-case learning, optional OpenTelemetry interoperability, and templates for durable evidence.
+
 ### [agent-skills](agent-skills/SKILL.md)
 
 Reference for the Agent Skills open format itself — directory structure, frontmatter schema, naming conventions, and progressive disclosure model. Use this meta-skill when creating or reviewing any other skill in this repository.
diff --git a/agent-evals-and-observability/README.md b/agent-evals-and-observability/README.md
new file mode 100644
index 0000000..5fef645
--- /dev/null
+++ b/agent-evals-and-observability/README.md
@@ -0,0 +1,34 @@
+# Agent Evals and Observability
+
+Build evidence for AI-agent changes without confusing a dashboard with proof of quality.
+
+## Why Install This Skill
+
+Agent behavior can look good in a demo yet fail through an unsafe tool call, a bad recovery path, stale data, or a silent production regression. This skill helps your agent turn those risks into task and trajectory contracts, datasets, appropriate graders, and release evidence.
+
+It also keeps observability useful without turning it into a privacy liability. Your agent can design minimized traces and metrics, analyze a regression fairly, and make a release decision that keeps hard safety and privacy invariants separate from ordinary quality indicators.
+
+## What You Get
+
+| Contents | Provides |
+|---|---|
+| `SKILL.md` | Framework-neutral workflow and routing |
+| `references/` | Evaluation, statistics, trajectory, privacy, OTel, and source guidance |
+| `templates/` | Fillable plans, manifests, grader specs, reviews, reports, and gates |
+
+## Quick Start
+
+Ask: `Create an eval plan and release gate for this agent change.`
+
+Expected result: a risk-based plan that names the task contract, evidence, privacy limits, uncertainty, rollback path, and decision owner.
+
+## Triggers
+
+- Agent evaluation, LLM evals, evaluation dataset, grader, or model judge
+- Agent observability, traces, telemetry, trajectory review, or production monitoring
+- Regression analysis, prompt/model/tool release gate, or incident-to-eval learning
+- Privacy-aware logging, redaction, retention, or trace sampling for an agent
+
+## Requirements
+
+No package, vendor account, or API key is required. Use the agent framework and telemetry backend already selected by the project. OpenTelemetry GenAI is optional interoperability guidance only.
diff --git a/agent-evals-and-observability/SKILL.md b/agent-evals-and-observability/SKILL.md
new file mode 100644
index 0000000..aa633d5
--- /dev/null
+++ b/agent-evals-and-observability/SKILL.md
@@ -0,0 +1,61 @@
+---
+name: agent-evals-and-observability
+description: >-
+  Design, run, review, or release framework- and vendor-neutral evaluations and
+  observability for AI agents. Use when defining agent evals, datasets, graders,
+  trajectory review, regression analysis, release gates, production traces, or
+  privacy-aware telemetry. Covers task and trajectory contracts, statistical
+  comparisons, and incident-to-case learning; route framework implementation to
+  pydanticai or langgraph when needed.
+license: MIT
+compatibility: No runtime dependency. Host-, framework-, model-, and telemetry-backend-neutral methodology.
+metadata:
+  source: "Curated from primary and official sources listed in references/source-index.md; checked 2026-07-13"
+---
+
+# Agent Evals and Observability
+
+Evaluation asks whether behavior meets a defined criterion on a declared dataset or production sample. Observability supplies traces, logs, metrics, correlations, and diagnostic context. Use both; neither proves what the other does.
+
+## Workflow
+
+1. Define the decision, risk, task contract, trajectory contract, and unacceptable outcomes. Select evidence by harm, reversibility, and deployment stage, not a staged completeness scale.
+2. Create an immutable dataset version and manifest before comparing versions. Declare provenance, rights/consent, slices, fixtures, expected side effects, contamination risk, limitations, retention, and changelog.
+3. Select complementary graders that observe the claimed property. Use deterministic checks for observable mechanics; use execution/environment checks for state and side effects; use human, model-judge, pairwise/ranking, domain, and safety review where appropriate.
+4. Run the candidate and baseline under comparable conditions. Preserve run configuration, stochastic repeats where variability affects the decision, failures/timeouts, and trajectory evidence.
+5. Report a multidimensional profile and uncertainty. Use paired comparisons where possible; inspect slices, missingness, base rates, confounders, effect sizes, and multiple comparisons. Non-significance is not equivalence.
+6. Apply a risk-tiered release gate: hard safety, privacy, authorization, and side-effect invariants cannot be averaged away. Record authority, insufficient-evidence outcomes, rollback, and follow-up.
+7. Instrument production with minimized, redacted telemetry. Feed verified incidents and near misses into cases after consent, transformation, and contamination review.
+
+Stop when the supported decision, evidence gaps, residual risks, and responsible owner are recorded. Escalate rather than infer a pass when required evidence is unavailable or conflicts.
+
+## Load By Need
+
+| Need | Load |
+|---|---|
+| Choose evaluation evidence and contracts | [references/evaluation-design.md](references/evaluation-design.md) |
+| Build immutable cases, fixtures, and provenance | [references/datasets.md](references/datasets.md) |
+| Select or calibrate graders | [references/graders.md](references/graders.md) |
+| Define measures or compare runs | [references/metrics-and-statistics.md](references/metrics-and-statistics.md) |
+| Review tools, state, recovery, or side effects | [references/trajectory-review.md](references/trajectory-review.md) |
+| Triage a regression or decide release readiness | [references/regression-and-release.md](references/regression-and-release.md) |
+| Design traces, logs, metrics, or privacy controls | [references/production-observability.md](references/production-observability.md) |
+| Interoperate with OpenTelemetry | [references/opentelemetry-genai.md](references/opentelemetry-genai.md) |
+| Need framework-specific implementation | [references/framework-routing.md](references/framework-routing.md) |
+| Exercise the methodology with safe probes | [references/synthetic-scenarios.md](references/synthetic-scenarios.md) |
+| Verify a source claim or status | [references/source-index.md](references/source-index.md) |
+
+## Templates
+
+Use [templates/eval-plan.md](templates/eval-plan.md), [templates/dataset-manifest.md](templates/dataset-manifest.md), [templates/grader-specification.md](templates/grader-specification.md), [templates/trace-review.md](templates/trace-review.md), [templates/run-report.md](templates/run-report.md), and [templates/release-gate.md](templates/release-gate.md). They are decision records, not checklists that manufacture evidence.
+
+## Guardrails
+
+- Do not use a scalar score, keyword hit, schema validation, or repeated output as proof beyond the property it directly observes.
+- Do not capture prompts, outputs, tool arguments, intermediate reasoning, credentials, or personal data by default. Minimize before export, redact early, restrict access, set retention/deletion paths, and prepare incident response.
+- Do not treat human judgment as definitive truth. Human and model graders require explicit rubrics, calibration, disagreement analysis, and revalidation when conditions change.
+- Keep prevalence-oriented production samples separate from risk-enriched challenge cases. Do not silently reweight either into the other.
+
+## When Not To Use
+
+Use [systematic-debugging](../systematic-debugging/SKILL.md) to investigate one active defect before proposing fixes. Use [verification-methodology](../verification-methodology/SKILL.md) for general evidence-backed completion claims, [secure-software-engineering](../secure-software-engineering/SKILL.md) to design preventive controls, and framework skills for SDK-specific code.
diff --git a/agent-evals-and-observability/references/datasets.md b/agent-evals-and-observability/references/datasets.md
new file mode 100644
index 0000000..28655b6
--- /dev/null
+++ b/agent-evals-and-observability/references/datasets.md
@@ -0,0 +1,9 @@
+# Datasets and Fixtures
+
+Freeze each evaluated dataset version as an immutable artifact under the repository's chosen naming scheme. Record a changelog rather than prescribing a particular version-label convention. Preserve the exact case, fixture, configuration boundary, and manifest needed to reproduce a run.
+
+The manifest must state provenance; rights and consent; creation and review owners; task taxonomy; slices; source class; expected outcomes; fixtures and reset method; expected side effects; contamination and leakage risk; limitations; access; retention; deletion handling; and change history.
+
+Use isolated sandboxes or resettable fixtures for stateful tools. Inspect actual environment state to grade idempotency or side effects; identical text is not sufficient evidence. Treat production-derived data as a controlled source: verify rights, minimize and transform it, redact before storage, limit access, define deletion, and prevent it from entering prompts or training/evaluation targets where it would contaminate comparisons.
+
+Report prevalence-oriented samples separately from deliberately risk-enriched or adversarial cases. Slices should expose meaningful differences such as task class, tool availability, locale, policy path, or input condition; they are not a license to infer outcomes for unrepresented groups.
diff --git a/agent-evals-and-observability/references/evaluation-design.md b/agent-evals-and-observability/references/evaluation-design.md
new file mode 100644
index 0000000..1ae71a8
--- /dev/null
+++ b/agent-evals-and-observability/references/evaluation-design.md
@@ -0,0 +1,17 @@
+# Evaluation Design
+
+Write a task contract: inputs and context, permitted tools and permissions, expected outcome, prohibited outcomes, environment assumptions, and observable completion evidence. Add a trajectory contract when the path matters: eligible tool selection and arguments, authorization checks, state transitions, grounding sources, recovery behavior, stopping/escalation, and permitted side effects.
+
+Select evidence rather than requiring every evaluation type:
+
+| Evidence | Useful when | Boundary |
+|---|---|---|
+| Unit | A prompt formatter, tool adapter, parser, or guard can be isolated | Does not establish integration behavior |
+| Component | Router, retriever, policy layer, or tool boundary interacts internally | May hide production environment behavior |
+| Scenario | A complete task must run under controlled conditions | Results depend on fixture fidelity |
+| Regression | A known behavior or escaped failure must remain protected | Cannot cover unknown failures |
+| Adversarial | Misuse, injection, unsafe actions, or leakage are plausible | Challenge coverage is never exhaustive |
+| Online | Distribution shift, service variability, or user outcomes matter | Confounding and consent constrain interpretation |
+| Human review | Domain or subjective criteria cannot be directly automated | Review remains rubric- and context-sensitive |
+
+Use synthetic, curated, replayed-production, adversarial, and regression cases as distinct declared sources. Choose a mix based on task harm, reversibility, novelty, external side effects, and availability of valid evidence.
diff --git a/agent-evals-and-observability/references/framework-routing.md b/agent-evals-and-observability/references/framework-routing.md
new file mode 100644
index 0000000..f451660
--- /dev/null
+++ b/agent-evals-and-observability/references/framework-routing.md
@@ -0,0 +1,13 @@
+# Framework Routing
+
+Use this skill for methodology, evidence, data governance, comparison, and privacy regardless of framework.
+
+| Need | Route |
+|---|---|
+| PydanticAI test models, Pydantic Evals, or framework traces | [pydanticai](../../pydanticai/SKILL.md), especially `references/testing-evals.md` |
+| LangGraph state, routing, checkpointers, or LangGraph implementation | [langgraph](../../langgraph/SKILL.md), especially `references/evals.md` |
+| A general defect discovered by an eval | [systematic-debugging](../../systematic-debugging/SKILL.md) |
+| General completion evidence | [verification-methodology](../../verification-methodology/SKILL.md) |
+| Preventive AI, authorization, or data-boundary controls | [secure-software-engineering](../../secure-software-engineering/SKILL.md) |
+
+Do not require an SDK, hosted evaluator, or tracing backend. Translate framework events into the task and trajectory contracts before comparing implementations.
diff --git a/agent-evals-and-observability/references/graders.md b/agent-evals-and-observability/references/graders.md
new file mode 100644
index 0000000..c14ca51
--- /dev/null
+++ b/agent-evals-and-observability/references/graders.md
@@ -0,0 +1,33 @@
+# Grader Families and Calibration
+
+Compose graders by the property each can observe; no universal taxonomy or single grader is sufficient.
+
+| Family | Direct evidence | Common boundary |
+|---|---|---|
+| Deterministic/programmatic | Schema, arithmetic, exact state-independent properties | A regex or keyword does not prove semantics, safety, or grounding |
+| Execution/environment | Sandbox state, permissions, tool results, side effects | Requires faithful fixtures and reset evidence |
+| Human-rubric | Contextual/domain judgment against an anchored rubric | Training, fatigue, incentives, and context affect results |
+| Model judge | Structured assessment at scale | Prompt/model changes and bias require calibration |
+| Pairwise/ranking | Relative preference or ordering | Randomize position and blind identity; full round robin is only one costly design |
+| Domain/safety | Specialist or policy criteria | Must define scope, false positives, false negatives, and escalation |
+
+For model judges and subjective rubrics, hold out calibration material from tuning. Blind candidate identity, randomize order where relevant, inspect rationales, compare errors and disagreement by slice, and use an agreement/error analysis appropriate to label type and decision. Predefine acceptable error for the risk; do not impose universal rater counts, agreement thresholds, or interpretations of kappa/correlation. Revalidate after grader prompt, model, rubric, task distribution, or policy changes.
+
+Human review is valuable evidence, not definitive truth. Resolve disagreement through rubric refinement, adjudication rules, or escalation appropriate to the decision, and retain the disagreement rather than averaging it away.
+
+## Build A Grader Contract
+
+For every grader, record the property, inputs visible to it, output schema, direction/scale, abstention and error behavior, reference material, versioned implementation or prompt, cost/latency constraints, slices where it is valid, known failure modes, and false-positive/false-negative consequences. A reference answer is evidence, not automatically correct; version and review it like any other oracle.
+
+Deterministic checks should target properties such as parseability, exact identifiers, allowed tool names, numerical invariants, or observed sandbox state. They should not use keyword presence as a proxy for factual grounding or policy compliance. Execution graders should query the environment after the run, distinguish planned from committed effects, and verify cleanup or reconciliation.
+
+A model judge needs the task contract, only the evidence required to judge it, an anchored rubric with counterexamples, an abstain/insufficient-context path, and a structured rationale. Do not ask a judge to verify external facts it cannot access. Blind candidate identity and irrelevant metadata; randomize order in pairwise work; test order reversal and legitimate format/length variation. Sampled pairs, tournaments, active selection, and full round robins are different designs with different coverage and cost.
+
+## Calibrate And Revalidate
+
+1. Freeze rubric, calibration cases, human-review instructions, and candidate identities before the comparison.
+2. Collect independent labels where the decision requires them; preserve reviewer identity or role in restricted metadata so systematic disagreement can be investigated.
+3. Compare confusion/error patterns and disagreement by task and slice. Select agreement statistics for the label scale and sampling design rather than treating one coefficient as universal.
+4. Inspect rationales and adjudicate sampled disagreements. Distinguish rubric ambiguity, missing evidence, grader bias, reviewer error, and legitimately plural answers.
+5. Define where the grader can gate, where it can screen for review, and where it must abstain. Freeze the accepted grader version.
+6. Revalidate after changes to the grader model, prompt, rubric, reference answers, task distribution, policy, or telemetry available to the grader.
diff --git a/agent-evals-and-observability/references/metrics-and-statistics.md b/agent-evals-and-observability/references/metrics-and-statistics.md
new file mode 100644
index 0000000..7aeab70
--- /dev/null
+++ b/agent-evals-and-observability/references/metrics-and-statistics.md
@@ -0,0 +1,29 @@
+# Metrics and Statistical Reasoning
+
+Report a profile by task and slice: task outcome; grounding to supplied or retrieved evidence; tool selection, arguments, ordering, authorization, and side effects; safety; reliability and recovery; cost/resource use; and latency. Include failures, timeouts, abstentions, and missing data rather than conditioning only on completed runs.
+
+Compare candidate and baseline on the same cases when feasible. Preserve paired outcomes, effect sizes, uncertainty intervals suited to the metric, and run-level conditions. For stochastic systems, repeat runs when stochastic variation can change the decision; report variability rather than treating one draw as stable.
+
+Choose sample size through the decision risk, baseline rate, metric distribution, planned pairing, slice coverage, minimum detectable effect, and available resources. Do not use fixed sample quotas. Plan power or sensitivity before a consequential comparison. A non-significant result does not show equivalence.
+
+Predeclare primary comparisons where practical. Investigate multiple comparisons, changing composition, missingness, base rates, selection effects, temporal/environment confounders, and aggregate reversals. Use correction or hierarchical interpretation appropriate to the question, and show slice results instead of relying on a blended average.
+
+## Define Each Measure
+
+A metric contract names the property; unit or label set; population and denominator; included failures, retries, and abstentions; aggregation and tail/slice views; preferred direction; data source; measurement error; baseline; decision threshold or comparison rule; owner; and failure behavior when data is missing. Token counts are not automatically cost, latency averages can hide tail harm, and user feedback is a selected noisy signal rather than ground truth.
+
+Keep hard invariants categorical. Report ordinary indicators separately, for example:
+
+- task outcome and completion evidence;
+- grounded claims or evidence use, with the actual source context available to the grader;
+- tool selection, arguments, authorization checks, ordering, and committed side effects;
+- safety/privacy/policy outcomes and grader false-positive/false-negative behavior;
+- reliability, retries, recovery, loops, timeouts, and escalation;
+- latency distribution and time to useful outcome; and
+- resource use and monetary cost from the billing/usage source that actually defines them.
+
+## Choose Comparison Methods From The Data
+
+For paired binary outcomes, preserve the discordant case pairs and use an interval/test suitable for paired proportions. For continuous or heavy-tailed outcomes, consider paired bootstrap, permutation, or a justified model and report distributional views. For ordinal or rubric scores, preserve the scale and rater structure instead of pretending equal numeric distance. If a release needs to establish practical equivalence or non-inferiority, predefine the acceptable margin and design for that question; failure to detect a difference is not equivalence.
+
+Choose repetitions and sample size from estimated variance/base rate, minimum effect worth detecting, pairing, slice coverage, desired error risk, and decision consequence. Run sensitivity analysis when inputs are uncertain. If the design cannot resolve the decision, report insufficient evidence rather than converting a wide interval into a pass.
diff --git a/agent-evals-and-observability/references/opentelemetry-genai.md b/agent-evals-and-observability/references/opentelemetry-genai.md
new file mode 100644
index 0000000..8115e43
--- /dev/null
+++ b/agent-evals-and-observability/references/opentelemetry-genai.md
@@ -0,0 +1,7 @@
+# Optional OpenTelemetry GenAI Interoperability
+
+OpenTelemetry GenAI semantic conventions are an optional interoperability layer, not this skill's canonical data model or a prerequisite for evaluation. The `open-telemetry/semantic-conventions-genai` repository was checked at commit `63f8200eee093730ce845d26ce2aafb621b0807e` on 2026-07-13 and labeled Development. Verify current status and canonical names before adopting any convention.
+
+If useful, map local concepts such as agent invocation, model request, tool execution, workflow step, evaluator result, duration, token/resource use, and correlation to the then-current conventions. Keep an internal event contract that works without OTel and avoid copied example hostnames, paths, prompts, model names, or histogram bucket boundaries.
+
+Prompt and output content are opt-in and sensitive under this guidance. Do not enable content capture by default; apply the minimization, redaction, access, retention, deletion, and incident controls in [production observability](production-observability.md) first.
diff --git a/agent-evals-and-observability/references/production-observability.md b/agent-evals-and-observability/references/production-observability.md
new file mode 100644
index 0000000..e11aced
--- /dev/null
+++ b/agent-evals-and-observability/references/production-observability.md
@@ -0,0 +1,28 @@
+# Production Observability and Privacy
+
+Observability provides diagnostic context, not an automatic evaluation verdict. Correlate minimized traces, structured logs, metrics, deployment/configuration identity, and user feedback with a non-identifying invocation identifier. Record sampling rules and bias: errors, high-cost paths, and opted-out traffic may be under- or over-represented.
+
+Default to metadata such as operation class, outcome category, duration, resource counters, and redacted error class. Treat prompt/output text, tool arguments/results, retrieved content, intermediate state, credentials, and personal data as sensitive opt-in data. Minimize before export; use structured allowlists and redaction at the earliest boundary; test redaction; enforce least-privilege access; set purpose-bound retention; support deletion; and maintain incident response and access-audit procedures.
+
+Sampling, retention, access, and alert conditions are local risk decisions. Record their rationale, limitations, failure behavior, and how a privacy incident is contained without expanding collection indiscriminately.
+
+## Telemetry Contract
+
+Define the question before the signal. For each trace, log, metric, or feedback field, record purpose, producer, schema/version, sensitivity, cardinality, sampling, redaction/minimization, access, retention/deletion, correlation, owner, and what happens if collection or export fails. Keep high-cardinality identifiers out of metric labels; use restricted traces or logs for case-level diagnosis.
+
+Useful backend-neutral signals can include:
+
+- invocation/workflow/step outcome and bounded duration;
+- model/tool/policy/configuration version identifiers;
+- error and stop categories, retry/loop/escalation counts, and tool outcome class;
+- resource or billed-usage counters from their authoritative source;
+- committed/rolled-back side-effect category; and
+- evaluator name/version/result linked by an opaque non-identifying run ID.
+
+Do not synthesize user, conversation, or tenant identity from content. Do not expose chain-of-thought as telemetry. If content is genuinely required for a bounded investigation, use an approved opt-in path with purpose limitation, field allowlists, transformation/redaction before export, restricted access, deletion, and auditable shutdown.
+
+## Alerts, Feedback, And Incident Learning
+
+Alerts should point to an actionable operational condition and owner; they do not establish why quality changed. Define the population, sampling path, expected delay, missing-data behavior, and false-positive/false-negative costs. User feedback, support tickets, judge scores, and anomaly detectors are candidate signals with selection and measurement bias, not automatic labels.
+
+For an incident: contain harm; preserve the smallest authorized evidence; record deployment/configuration and sampling gaps; reproduce in a safe fixture; distinguish agent, tool, data, policy, infrastructure, grader, and telemetry failures; correct the system; then add a transformed regression case only after rights/privacy/contamination review. Verify the fix offline and at the deployed boundary without broadening production capture merely to make diagnosis easier.
diff --git a/agent-evals-and-observability/references/regression-and-release.md b/agent-evals-and-observability/references/regression-and-release.md
new file mode 100644
index 0000000..bfd7af1
--- /dev/null
+++ b/agent-evals-and-observability/references/regression-and-release.md
@@ -0,0 +1,27 @@
+# Regression, Triage, and Release Gates
+
+For each change, identify the baseline, frozen dataset, configuration differences, primary decision metrics, uncertainty method, slices, and hard invariants. Compare paired runs where possible. A regression investigation begins with reproduction, trace and fixture review, configuration and environment differences, then a narrow root-cause hypothesis. Do not tune against the same incident case without recording contamination.
+
+Turn confirmed incidents and near misses into candidates for new cases only after privacy, rights, redaction, and retention review. Classify whether the failure was task contract, trajectory, tool/environment, grader, data, deployment, or observability failure.
+
+Risk-tier gates should have an authority, evidence owner, and one of: approve, approve with recorded conditions, hold for insufficient evidence, or block. Define hard invariants separately from statistical indicators. Examples of hard invariants include unauthorized action, unapproved sensitive-data disclosure, or an unreconciled harmful side effect. They cannot be offset by quality, cost, or latency results. Every gate records rollback/containment, monitoring during rollout, and the next review trigger.
+
+## Compare Without Moving The Goalposts
+
+Freeze candidate and baseline identifiers, dataset version, grader versions, fixtures, environment, sampling/randomization plan, primary comparisons, slices, and exclusions before the decision run. Use the same cases and comparable external conditions where feasible. Record provider/model drift, cache state, concurrency, tool/data versions, and instrumentation changes that could explain a delta.
+
+Report case-level paired outcomes, slice deltas, uncertainty/effect size, hard-invariant results, failures/timeouts/missing data, cost/resource and latency distributions, grader disagreement, and observed confounders. Separate exploratory findings from preregistered release evidence. Do not tune on the held-out decision set and then report the same run as independent confirmation.
+
+## Build The Gate
+
+A release matrix should state for each dimension:
+
+| Dimension | Evidence | Decision rule | Hard or statistical | Owner | Failure/insufficient-data action |
+|---|---|---|---|---|---|
+| Task/trajectory outcome | Cases, state checks, trace review | Risk- and slice-specific | Statistical or categorical | | |
+| Safety/privacy/authorization/side effects | Adversarial cases and environment evidence | Explicit invariant | Hard | | |
+| Reliability/recovery | Failure injection and deployed evidence | Workload-specific | Statistical plus hard cases | | |
+| Cost/resource/latency | Metered distributions | Budget/SLO-specific | Statistical | | |
+| Operability | Correlation, alerts, rollback rehearsal | Evidence checklist | Categorical | | |
+
+The owner chooses a verdict from the documented outcomes, not from an unweighted average. Conditional approval names the condition, monitor, expiration/review trigger, and authority to halt. Rollout evidence must be linked to the exact released configuration, and rollback must account for state and side effects rather than only code version.
diff --git a/agent-evals-and-observability/references/source-index.md b/agent-evals-and-observability/references/source-index.md
new file mode 100644
index 0000000..92da491
--- /dev/null
+++ b/agent-evals-and-observability/references/source-index.md
@@ -0,0 +1,14 @@
+# Source and Status Index
+
+Sources inform decisions; they become binding only when adopted by applicable policy, contract, or regulation. Versions and statuses were checked 2026-07-13 unless otherwise stated. Recheck rolling documentation and Development-status conventions before consequential use.
+
+| Source | Exact status/version checked | Primary URL | Decision use |
+|---|---|---|---|
+| NIST Artificial Intelligence Risk Management Framework | NIST AI 100-1, AI RMF 1.0, January 2023; NIST reports revision work underway | https://doi.org/10.6028/NIST.AI.100-1 | Voluntary risk and governance framing, not an eval threshold catalog. |
+| NIST AI RMF Generative AI Profile | NIST AI 600-1, July 2024 | https://doi.org/10.6028/NIST.AI.600-1 | Voluntary generative-AI risk considerations, not a release formula. |
+| OpenTelemetry GenAI semantic conventions | Development status at repository commit `63f8200eee093730ce845d26ce2aafb621b0807e` dated 2026-07-08 | https://github.com/open-telemetry/semantic-conventions-genai/tree/63f8200eee093730ce845d26ce2aafb621b0807e/docs/gen-ai | Optional moving interoperability guidance; prompt/output content is opt-in and sensitive. |
+| PydanticAI testing documentation | Rolling official documentation checked 2026-07-13 | https://ai.pydantic.dev/testing/ | PydanticAI-specific unit and model-substitution implementation. |
+| Pydantic Evals documentation | Rolling official documentation checked 2026-07-13 | https://ai.pydantic.dev/evals/ | Pydantic-specific dataset, experiment, evaluator, and span-evaluation APIs. |
+| LangGraph testing documentation | Rolling official documentation checked 2026-07-13 | https://docs.langchain.com/oss/python/langgraph/test | LangGraph-specific node, graph, state, and partial-execution testing. |
+| LangSmith evaluation concepts | Rolling vendor documentation checked 2026-07-13 | https://docs.langchain.com/langsmith/evaluation-concepts | Vendor implementation example only; its fixed example counts and product workflow are not methodology requirements. |
+| “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” | Zheng et al., arXiv:2306.05685 v4, 2023-12-24 | https://arxiv.org/abs/2306.05685v4 | Evidence that judge behavior can include position, verbosity, and self-enhancement biases in the studied setting; not universal bias magnitudes. |
diff --git a/agent-evals-and-observability/references/synthetic-scenarios.md b/agent-evals-and-observability/references/synthetic-scenarios.md
new file mode 100644
index 0000000..98263c8
--- /dev/null
+++ b/agent-evals-and-observability/references/synthetic-scenarios.md
@@ -0,0 +1,14 @@
+# Synthetic Scenario Probes
+
+Use synthetic placeholders only. Do not put plausible credentials, identifiers, or personal data into fixtures.
+
+| Probe | Contract and evidence |
+|---|---|
+| Multi-domain routing | A request needs two declared domains. Verify complete routing/coverage, permitted handoffs, and no unsupported extra domain. |
+| Tool error and side effect | A sandboxed tool reports a transient error after a declared partial action. Verify state inspection, safe recovery or escalation, and no duplicate side effect. |
+| Injection and privacy | Untrusted retrieved text requests disclosure or instruction override. Verify policy path, no sensitive content in telemetry, and redaction test evidence. |
+| Grader calibration/disagreement | Blinded candidate outputs include legitimate variation and ambiguous cases. Compare rubric, judge, and reviewer disagreement; record resolution and limitations. |
+| Regression detection | Candidate and baseline run the same frozen cases with recorded configuration. Report paired deltas, intervals, slices, missingness, and an insufficient-evidence outcome when warranted. |
+| Privacy-safe incident diagnosis | A synthetic trace has only opaque correlation IDs, redacted fields, error category, and state summary. Verify diagnosis can identify the next evidence request without recovering raw content. |
+
+Run each probe through its selected graders and release-gate template. A probe passes only when its stated evidence exists; it does not certify unrelated properties.
diff --git a/agent-evals-and-observability/references/trajectory-review.md b/agent-evals-and-observability/references/trajectory-review.md
new file mode 100644
index 0000000..3f5ab95
--- /dev/null
+++ b/agent-evals-and-observability/references/trajectory-review.md
@@ -0,0 +1,35 @@
+# Trace and Trajectory Review
+
+Review trajectories when correctness, safety, cost, or recovery depends on how the agent acted. Use the trace-review template to inspect:
+
+- tool selection, argument validity, ordering, and permission checks;
+- state transitions, retries, recovery, stopping, escalation, and loop controls;
+- grounding: retrieval, tool-result use, citations, and unsupported claims;
+- planned and observed side effects, including compensating or rollback actions;
+- correlation between final outcome, latency, resources, errors, and environment state.
+
+Specify permitted alternatives rather than forcing a single path when more than one safe trajectory can satisfy the contract. Grade environment-visible effects for idempotency and recovery. Capture the smallest trace fields needed for the claim; redact before review and record inaccessible evidence as a gap.
+
+## Minimal Run And Step Contract
+
+Keep a backend-neutral internal record even when exporting to a tracing system:
+
+- run/case/experiment identifiers and parent-child correlation;
+- dataset, agent/workflow, prompt/policy, model/provider, tool, grader, code, fixture, and deployment versions;
+- environment and permission boundary;
+- step type, start/end or sequence, outcome/error category, retry/attempt relation, and stop reason;
+- selected tool and bounded/redacted arguments/result summary;
+- state transition and planned/committed/rolled-back side effects;
+- grounding/evidence identifiers that can be resolved under access control; and
+- final outcome plus linked grader results.
+
+Do not manufacture a conversation ID from content, expose chain-of-thought, or copy raw inputs merely to make a trace look complete. Record when sampling or access removed evidence.
+
+## Review Protocol
+
+1. Reconstruct the intended task and trajectory contracts without looking at the final score.
+2. Follow the state and permission boundary through each step; compare tool arguments and observed effects to the contract.
+3. Locate the first divergence, not only the last visible error. Distinguish model choice, orchestration, tool, data, environment, policy, and telemetry failures.
+4. Test whether recovery avoided duplicate or unreconciled effects and whether stopping/escalation happened at the right boundary.
+5. Compare final-answer quality with path quality. A correct answer from an unauthorized or fragile path still fails the trajectory contract.
+6. Record evidence gaps, alternative valid paths, root-cause hypotheses, and the case/fixture changes needed before rerun.
diff --git a/agent-evals-and-observability/templates/dataset-manifest.md b/agent-evals-and-observability/templates/dataset-manifest.md
new file mode 100644
index 0000000..6e26c98
--- /dev/null
+++ b/agent-evals-and-observability/templates/dataset-manifest.md
@@ -0,0 +1,10 @@
+# Dataset Manifest
+
+- Dataset name and immutable version identifier:
+- Purpose, task taxonomy, and source class: synthetic / curated / replayed-production / adversarial / regression
+- Provenance, rights/consent, creator, reviewer, and creation date:
+- Cases, slices, expected outcomes, and known limitations:
+- Fixture locations, environment version, reset method, and expected side effects:
+- Contamination/leakage risks and controls:
+- Access controls, retention, deletion path, and incident contact:
+- Change history and superseded versions:
diff --git a/agent-evals-and-observability/templates/eval-plan.md b/agent-evals-and-observability/templates/eval-plan.md
new file mode 100644
index 0000000..f6c514f
--- /dev/null
+++ b/agent-evals-and-observability/templates/eval-plan.md
@@ -0,0 +1,29 @@
+# Eval Plan
+
+## Decision and Risk
+
+- Change and decision:
+- Owner and authority:
+- Harm, reversibility, and deployment stage:
+- Task contract and prohibited outcomes:
+- Trajectory contract and permitted side effects:
+
+## Evidence Design
+
+| Evidence type | Risk addressed | Dataset/version and slices | Grader | Limitations |
+|---|---|---|---|---|
+| | | | | |
+
+## Comparison and Analysis
+
+- Baseline and candidate configuration:
+- Pairing and stochastic-repeat plan:
+- Primary metrics, effect sizes, uncertainty method, and minimum detectable effect rationale:
+- Missingness, base-rate, multiple-comparison, and confounder handling:
+- Privacy, rights, access, retention, deletion, and contamination controls:
+
+## Decision Rule
+
+- Hard invariants:
+- Statistical indicators and insufficient-evidence disposition:
+- Rollback/containment and monitoring:
diff --git a/agent-evals-and-observability/templates/grader-specification.md b/agent-evals-and-observability/templates/grader-specification.md
new file mode 100644
index 0000000..ee6c669
--- /dev/null
+++ b/agent-evals-and-observability/templates/grader-specification.md
@@ -0,0 +1,10 @@
+# Grader Specification
+
+- Property claimed and observable evidence:
+- Grader family and why it fits:
+- Inputs, output schema, rubric anchors, and abstain/escalate behavior:
+- Known blind spots, false-positive/false-negative harms, and affected slices:
+- Calibration set isolation and candidate blinding/order randomization:
+- Human/model disagreement analysis and adjudication rule:
+- Revalidation triggers: model, prompt, rubric, policy, task distribution, or environment change
+- Version, owner, access, retention, and audit evidence:
diff --git a/agent-evals-and-observability/templates/release-gate.md b/agent-evals-and-observability/templates/release-gate.md
new file mode 100644
index 0000000..27ce5ed
--- /dev/null
+++ b/agent-evals-and-observability/templates/release-gate.md
@@ -0,0 +1,15 @@
+# Risk-Tiered Release Gate
+
+- Change, risk tier, decision authority, and evidence owner:
+- Dataset/run/trace report references:
+
+| Category | Hard invariant or indicator | Evidence | Status | Owner/action |
+|---|---|---|---|---|
+| Safety/privacy/authorization | | | | |
+| Task and trajectory outcome | | | | |
+| Reliability/recovery | | | | |
+| Cost/resources and latency | | | | |
+
+- Decision: approve / approve with conditions / hold for insufficient evidence / block
+- Rollback or containment path:
+- Rollout monitoring, retention/deletion confirmation, and review trigger:
diff --git a/agent-evals-and-observability/templates/run-report.md b/agent-evals-and-observability/templates/run-report.md
new file mode 100644
index 0000000..be87e94
--- /dev/null
+++ b/agent-evals-and-observability/templates/run-report.md
@@ -0,0 +1,14 @@
+# Run Report
+
+- Candidate, baseline, dataset version, fixture/environment, and run date:
+- Pairing, stochastic repeats, failures/timeouts, and missingness:
+
+| Dimension and slice | Baseline | Candidate | Effect and uncertainty | Interpretation |
+|---|---|---|---|---|
+| | | | | |
+
+- Multiple-comparison and confounder handling:
+- Trajectory and side-effect findings:
+- Grader calibration/disagreement status:
+- Privacy and provenance exceptions:
+- Conclusion: improve / regress / inconclusive, with evidence gaps:
diff --git a/agent-evals-and-observability/templates/trace-review.md b/agent-evals-and-observability/templates/trace-review.md
new file mode 100644
index 0000000..1f4e9ed
--- /dev/null
+++ b/agent-evals-and-observability/templates/trace-review.md
@@ -0,0 +1,9 @@
+# Trace Review Record
+
+- Invocation/correlation identifier (non-identifying):
+- Task and trajectory contract version:
+- Dataset case or approved production-sample provenance:
+- Tool selection, arguments, ordering, permissions, and observed environment state:
+- State transitions, grounding evidence, recovery, stopping, escalation, and side effects:
+- Outcome, errors, latency/resources, and redaction/access limitations:
+- Findings, evidence links, root-cause hypothesis, and follow-up owner: