mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-11 19:47:12 +03:00
feat(incident-learning): add incident-learning skill (#226)
Create incident-learning skill that converts operational incident and near-miss evidence into verified, owned improvements across product, code, tests, evals, operations, and governance domains. - SKILL.md: progressive disclosure with loading guide, routing table, ownership boundaries, and core principles (evidence/inference/ uncertainty separation, ticket-closure rejection, no-blame design) - README.md: human-facing with all 5 required sections - references/discovery-brief.md: bounded survey comparing SRE, QA, verification, agent-evals, product-lifecycle-learning, implementation-planning, resilience-and-recovery, production-readiness - references/: evidence/inference taxonomy, escaped-from analysis, follow-up domains, verification and closure standard - templates/: incident-learning record, causal/evidence ledger, follow-up work map, verification and closure record - evals/evals.json: 5 schema-valid cases (noisy report, monitoring gap, process failure, agent authority failure, non-actionable rejection) - Regenerated catalogs and updated root README and skill-triggers Closes #200 Co-authored-by: username <username> Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
username <username>
factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
parent
6f429a91b1
commit
4f14ce3df6
@@ -401,6 +401,15 @@
|
||||
"strict": false,
|
||||
"description": "Plan the implementation of an approved requirement or specification: produce an executable, dependency-aware delivery plan covering work breakdown, dependency mapping, critical path, ownership, parallelism and sequencing, rollout strategy, rollback and recovery paths, and verification against the original requirement. Supports cross-team, cross-repository, migration, and staged-rollout scenarios. Do not use for pre-approval discovery or needs-finding, authoring a specification from scratch, coding or implementation, the neckbeard issue-to-PR delivery flow itself, or any work whose prerequisite decision has not been approved — planning unapproved work is an explicit stop condition."
|
||||
},
|
||||
{
|
||||
"name": "incident-learning",
|
||||
"source": "./",
|
||||
"skills": [
|
||||
"./incident-learning"
|
||||
],
|
||||
"strict": false,
|
||||
"description": "Convert operational incident and near-miss evidence into durable product, engineering, test, evaluation, and governance improvements with verified closure. Separate observed facts from causal hypotheses and unresolved uncertainty; map follow-up work across code, tests, skills, operations, product, and governance; track ownership, verification, and closure for every finding. Do not use to assign blame or produce a generic postmortem template; do not close learning because tickets were created — require evidence the intended change occurred."
|
||||
},
|
||||
{
|
||||
"name": "jellyfin-cli",
|
||||
"source": "./",
|
||||
|
||||
@@ -66,6 +66,7 @@
|
||||
"./haystack",
|
||||
"./hugo-theme",
|
||||
"./implementation-planning",
|
||||
"./incident-learning",
|
||||
"./jellyfin-cli",
|
||||
"./jira-cli",
|
||||
"./jira-jql",
|
||||
|
||||
@@ -188,6 +188,10 @@ Build, customize, and debug advanced Hugo CMS themes. Covers template architectu
|
||||
|
||||
Turn an approved requirement or specification into an executable, dependency-aware delivery plan. Covers work breakdown into vertical slices, dependency mapping with critical-path analysis, ownership assignment, sequencing and parallelism, staged rollout strategy with rollback paths, and verification traceability against the original requirement. Supports cross-team, cross-repository, data migration, and staged-rollout scenarios. Does not plan unapproved work — entry gate enforces prerequisite approval.
|
||||
|
||||
### [incident-learning](incident-learning/SKILL.md)
|
||||
|
||||
Convert operational incident and near-miss evidence into verified, owned improvements across product, code, tests, evals, operations, and governance. Separates observed facts from causal hypotheses and unresolved uncertainty; maps escaped-from gaps (requirements, monitoring, authority, migration, adoption); assigns domain-specific follow-up work with owners and verification methods; and requires evidence of the implemented change — not just tickets — for closure. Routes implementation to SRE, QA, verification, agent evals, product lifecycle learning, implementation planning, resilience-and-recovery, and production-readiness. Ships 4 references (discovery brief, evidence/inference taxonomy, escaped-from analysis, follow-up domains, verification and closure), 4 templates (incident-learning record, causal/evidence ledger, follow-up work map, verification and closure record), and 5 evals.
|
||||
|
||||
### [jellyfin-cli](jellyfin-cli/SKILL.md)
|
||||
|
||||
Jellyfin media server from the terminal. Check server info, browse recently added and library contents, search and inspect media, see next-up episodes, and view statistics.
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
# Incident Learning
|
||||
|
||||
Convert operational incidents, near misses, and exercise findings into verified, owned improvements across product, engineering, test, evaluation, and governance — with closure evidence, not just tickets.
|
||||
|
||||
## Why Install This Skill
|
||||
|
||||
Most teams create tickets after incidents. Few verify that the intended change actually happened and had the intended effect. This skill provides a structured method for converting raw incident evidence into durable follow-up work with verified closure — separating what was observed from what was inferred, mapping each finding to the right domain (product, code, tests, evals, operations, governance), and tracking every follow-up through to verified completion.
|
||||
|
||||
Install this skill when your agent needs to help teams move from "we filed tickets after the postmortem" to "we verified that the monitoring gap was closed, the regression test was added, and the eval case now catches the failure mode." It composes specialist capabilities from SRE, QA, verification, agent evaluation, product lifecycle learning, implementation planning, resilience-and-recovery, and production-readiness without duplicating their methodology, and it feeds learning records into the production-excellence and agent-production-operations bundles.
|
||||
|
||||
## What You Get
|
||||
|
||||
| Path | What it provides |
|
||||
|---|---|
|
||||
| `SKILL.md` | Trigger conditions, core principles, the incident learning record structure, loading guide, template index, routing table, and ownership boundaries |
|
||||
| `references/discovery-brief.md` | Survey of adjacent skills (SRE, QA, verification, agent evals, product lifecycle learning, implementation planning, resilience-and-recovery, production-readiness) with ownership boundaries and routing decisions |
|
||||
| `references/evidence-inference-taxonomy.md` | Full taxonomy for separating observed facts, causal hypotheses (with confidence levels), contributing conditions, and unresolved uncertainty |
|
||||
| `references/escaped-from-analysis.md` | Method for mapping incidents to originating gaps: escaped requirements, missing monitoring/observability, unsafe authority/access, migration gaps, adoption consequences |
|
||||
| `references/follow-up-domains.md` | Six-domain follow-up taxonomy (product, code, tests, evals, operations, governance) with ownership patterns and verification methods per domain |
|
||||
| `references/verification-and-closure.md` | Closure standard requiring implementation evidence, verification evidence, and effect evidence; explicit rejection of ticket-only closure |
|
||||
| `templates/incident-learning-record.md` | Structured record template with fields for observed facts, causal hypotheses, contributing conditions, unresolved uncertainty, and escaped-from mapping |
|
||||
| `templates/causal-evidence-ledger.md` | Ledger template for tracking each causal claim with supporting evidence, confidence level, and alternative explanations |
|
||||
| `templates/follow-up-work-map.md` | Six-domain follow-up work map with ownership, verification method, and status tracking per finding |
|
||||
| `templates/verification-and-closure-record.md` | Per-follow-up closure record requiring implementation evidence, verification evidence, and effect evidence |
|
||||
| `evals/evals.json` | Five output-quality eval cases covering: noisy incident report, monitoring gap, process failure, agent authority failure, and non-actionable follow-up rejection |
|
||||
|
||||
## Quick Start
|
||||
|
||||
Start with the incident-learning record template to structure the raw incident evidence — separating facts from hypotheses from uncertainty, and mapping the escaped-from gap. Then use the follow-up work map to assign each finding to a domain and owner. Track each follow-up through the verification and closure record.
|
||||
|
||||
Ask your agent to "convert this incident into a learning record" or "build a follow-up work map from this postmortem" and the skill's triggers will route the work.
|
||||
|
||||
## Triggers
|
||||
|
||||
- Converting operational incidents, near misses, or exercise findings into structured learning records with follow-up work.
|
||||
- Separating observed facts from causal hypotheses and unresolved uncertainty in incident analysis.
|
||||
- Mapping incident findings to follow-up work across product, code, tests, evals, operations, and governance domains.
|
||||
- Tracking incident follow-up work to verified closure with implementation, verification, and effect evidence.
|
||||
- Linking incidents to escaped requirements, missing monitoring, unsafe authority, migration gaps, or adoption consequences.
|
||||
- Auditing incident follow-up closure rates or detecting unverified closures.
|
||||
|
||||
## Requirements
|
||||
|
||||
No runtime dependencies. The methodology is host-neutral and requires no specific tools, platforms, or API keys. The skill provides templates and method; incident data and follow-up work tracking must be supplied by the user's environment.
|
||||
@@ -0,0 +1,112 @@
|
||||
---
|
||||
name: incident-learning
|
||||
description: >-
|
||||
Convert operational incident and near-miss evidence into durable product,
|
||||
engineering, test, evaluation, and governance improvements with verified
|
||||
closure. Separate observed facts from causal hypotheses and unresolved
|
||||
uncertainty; map follow-up work across code, tests, skills, operations,
|
||||
product, and governance; track ownership, verification, and closure for
|
||||
every finding. Do not use to assign blame or produce a generic postmortem
|
||||
template; do not close learning because tickets were created — require
|
||||
evidence the intended change occurred.
|
||||
license: MIT
|
||||
compatibility: Platform-agnostic methodology. No runtime dependency.
|
||||
---
|
||||
|
||||
# Incident Learning
|
||||
|
||||
Convert operational incidents, near misses, and exercise findings into verified, owned improvements across product, engineering, test, evaluation, and governance domains. This skill owns the learning pipeline — the structured method that takes raw incident evidence and produces durable follow-up work with closure evidence. It does not own incident response, postmortem facilitation, or implementation of follow-up work; it owns the conversion from evidence to verified change.
|
||||
|
||||
## When to use
|
||||
|
||||
| Trigger | What it covers |
|
||||
|---|---|
|
||||
| "Convert this incident into follow-up work" | Evidence separation, causal ledger, follow-up work map, ownership assignment |
|
||||
| "What did we learn from this incident?" | Learning record with facts, hypotheses, uncertainty, escaped-from mapping |
|
||||
| "Track this finding to closure" | Verification record, closure evidence, follow-up work status |
|
||||
| "Link this incident to a missing requirement" | Escaped-from gap mapping: requirements, monitoring, authority, migration, adoption |
|
||||
| "Review our incident follow-up closure rate" | Learning pipeline audit, unverified closures, stale follow-up work |
|
||||
| "Process this near-miss" | Near-miss learning record, contributing conditions, risk reduction follow-up |
|
||||
| "Build an incident learning record" | Structured record with evidence/inference/uncertainty separation and escaped-from analysis |
|
||||
|
||||
## When not to use
|
||||
|
||||
- **Assigning blame or conducting a blame-based review.** This skill is explicitly blameless. It converts evidence into improvements, not fault into consequences. It contains no blame-assignment process, no "who to blame" field, and no "blameworthy" classification. If you need a postmortem that identifies responsible individuals, this skill will not serve that purpose.
|
||||
- **Producing a generic postmortem template as the sole output.** This skill's artifacts — the incident-learning record, causal/evidence ledger, follow-up work map, and verification and closure record — are the primary deliverables. A postmortem, timeline, or five-whys analysis may inform the learning record but is never the terminal output.
|
||||
- **Live incident command or incident response.** Route to [site-reliability-engineering](../site-reliability-engineering/SKILL.md) for the incident command system, on-call operations, and real-time incident management.
|
||||
- **Facilitating a blameless postmortem session.** Route to [site-reliability-engineering](../site-reliability-engineering/SKILL.md) for postmortem culture, timeline construction, and facilitation methods. This skill consumes postmortem output; it does not produce it.
|
||||
- **Closing learning because tickets were created.** Ticket creation is an action, not an outcome. This skill requires verification that the intended change occurred and had the intended effect. A ticket alone is not closure.
|
||||
- **Root-cause debugging of a specific failure.** Route to [systematic-debugging](../systematic-debugging/SKILL.md) for failure investigation. This skill consumes debugging findings as input to the learning pipeline.
|
||||
|
||||
## Core principles
|
||||
|
||||
**Evidence, not inference, is the foundation.** Every incident learning record separates three categories: observed facts (what happened, backed by telemetry, logs, or direct observation), causal hypotheses (inferences about why it happened, labeled with confidence and alternative explanations), and unresolved uncertainty (what remains unknown, including open questions and competing hypotheses). Conflating these categories produces false confidence. See the evidence/inference/uncertainty taxonomy in the learning record template.
|
||||
|
||||
**Learning closes with verification, not tickets.** Creating a ticket, a task, or a story is a process step — not evidence of improvement. Closure requires: (1) the follow-up work was implemented, (2) the implementation was verified against the intended change, and (3) the intended effect was observed (or a revised hypothesis was recorded). The verification and closure record template enforces this standard. Tickets alone are explicitly not sufficient; the closure record must reference evidence of the implemented change.
|
||||
|
||||
**Follow-up work spans domains, not only code.** An incident can expose gaps in any of six domains: product (requirements, design, user experience), code (implementation, logic, dependencies), tests (regression coverage, integration tests, assertions), evals (evaluation cases, grader coverage, dataset gaps), operations (monitoring, alerting, runbooks, capacity), and governance (authority boundaries, policy, compliance, access control). The follow-up work map uses this six-domain taxonomy; every finding maps to at least one domain.
|
||||
|
||||
**Every incident escaped from something.** The "escaped from" analysis maps the originating gap to one or more of: an escaped requirement (a needed requirement that was absent or incomplete), missing monitoring or observability (no signal existed to detect the condition), unsafe authority or access (insufficient guardrails on who could act), a migration gap (a transition that introduced the condition), or an adoption consequence (user or operator behavior that contributed). This mapping drives the follow-up work: a monitoring gap produces an observability follow-up, not a code patch.
|
||||
|
||||
**Uncertainty is recorded, not hidden.** Competing causal hypotheses, incomplete data, and open questions are explicit fields in the learning record — not footnotes. Unresolved uncertainty drives investigation follow-up work: what needs to be instrumented, what data needs to be collected, what hypothesis needs to be tested. An incident learning record with no unresolved uncertainty is either trivial or premature.
|
||||
|
||||
**Blame is absent by design.** The learning record has no field for "who caused this," "responsible individual," or "blameworthy action." The method treats incidents as system outcomes — products of interacting components, processes, assumptions, and conditions — not individual failures. This is not a stylistic preference; it is a structural constraint of the templates and the record format.
|
||||
|
||||
## The incident learning record
|
||||
|
||||
The incident learning record is the primary artifact. It contains these structured fields:
|
||||
|
||||
| Field | Content | Requirement |
|
||||
|---|---|---|
|
||||
| **Incident identifier** | Unique ID, date, duration, severity, systems affected | Required |
|
||||
| **Observed facts** | What happened — backed by telemetry, logs, direct observation, timeline events | Required; no inference here |
|
||||
| **Causal hypotheses** | Inferences about why it happened, each labeled with confidence level and alternative explanations | Required; must be separated from facts |
|
||||
| **Contributing conditions** | System states, process gaps, environmental factors that enabled or amplified the incident | Required |
|
||||
| **Unresolved uncertainty** | Open questions, competing hypotheses, missing data, unknowns | Required; drives investigation follow-up |
|
||||
| **Escaped-from mapping** | Originating gap: escaped requirement, missing monitoring/observability, unsafe authority/access, migration gap, adoption consequence | Required; at least one category |
|
||||
| **Follow-up work map** | Findings mapped to domains (product, code, tests, evals, operations, governance) with ownership and verification method | Required; at least one follow-up per significant finding |
|
||||
| **Verification and closure record** | For each follow-up: implementation evidence, verification evidence, effect evidence, closure date, closure authority | Required for closure; tickets alone are not sufficient |
|
||||
|
||||
## Loading guide
|
||||
|
||||
| File | Load when |
|
||||
|---|---|
|
||||
| [references/discovery-brief.md](references/discovery-brief.md) | Understanding ownership boundaries with adjacent skills and routing decisions |
|
||||
| [references/evidence-inference-taxonomy.md](references/evidence-inference-taxonomy.md) | Separating observed facts from causal hypotheses and unresolved uncertainty in an incident record |
|
||||
| [references/escaped-from-analysis.md](references/escaped-from-analysis.md) | Mapping incidents to originating gaps: requirements, monitoring, authority, migration, adoption |
|
||||
| [references/follow-up-domains.md](references/follow-up-domains.md) | Mapping findings to the six follow-up domains (product, code, tests, evals, operations, governance) with ownership patterns |
|
||||
| [references/verification-and-closure.md](references/verification-and-closure.md) | Defining what constitutes verified closure, rejecting ticket-only closure, and recording closure evidence |
|
||||
|
||||
## Templates
|
||||
|
||||
| Template | File | Purpose |
|
||||
|---|---|---|
|
||||
| Incident-learning record | [templates/incident-learning-record.md](templates/incident-learning-record.md) | Structured record with facts/hypotheses/uncertainty separation and escaped-from mapping |
|
||||
| Causal/evidence ledger | [templates/causal-evidence-ledger.md](templates/causal-evidence-ledger.md) | Ledger tracking each causal claim with supporting evidence, confidence, and alternatives |
|
||||
| Follow-up work map | [templates/follow-up-work-map.md](templates/follow-up-work-map.md) | Six-domain follow-up work map with ownership, verification method, and status tracking |
|
||||
| Verification and closure record | [templates/verification-and-closure-record.md](templates/verification-and-closure-record.md) | Per-follow-up closure record requiring implementation evidence, verification evidence, and effect evidence |
|
||||
|
||||
## Routing and related skills
|
||||
|
||||
This skill composes capabilities from and routes to:
|
||||
|
||||
- **[site-reliability-engineering](../site-reliability-engineering/SKILL.md)** — Incident command, on-call operations, SLO/SLI framework, blameless postmortems, monitoring and alerting. SRE owns the live incident response and postmortem facilitation; incident-learning consumes postmortem output and converts it into verified follow-up work.
|
||||
- **[qa-methodology](../qa-methodology/SKILL.md)** — Test strategy, regression testing, quality gates. Incident findings that reveal test gaps (missing regression coverage, weak assertions, untested paths) produce follow-up work routed to QA methodology for test design and implementation.
|
||||
- **[verification-methodology](../verification-methodology/SKILL.md)** — Pass/fail assessment against explicit criteria. Incident-learning's closure standard requires verification evidence; verification-methodology provides the protocol for producing that evidence against the intended change.
|
||||
- **[agent-evals-and-observability](../agent-evals-and-observability/SKILL.md)** — Agent evaluation design, dataset management, grading, trajectory review. Incidents involving agent behavior produce eval-gap follow-up work (missing eval cases, inadequate grader coverage, unobserved failure modes) routed to agent-evals-and-observability.
|
||||
- **[product-lifecycle-learning](../product-lifecycle-learning/SKILL.md)** — Post-launch outcome review, assumption updating, feature retirement. Incidents that reveal product-assumption failures (a feature used differently than intended, an assumption that didn't hold) produce product follow-up work routed to product-lifecycle-learning for assumption-update and feature-health decisions.
|
||||
- **[implementation-planning](../implementation-planning/SKILL.md)** — Delivery planning, work breakdown, dependency mapping, rollout strategy. Cross-domain follow-up work that spans multiple teams or repositories is routed to implementation-planning for coordinated delivery planning.
|
||||
- **[resilience-and-recovery](../resilience-and-recovery/SKILL.md)** — Resilience design, exercise evidence, game days, restore testing. Exercise and DR test findings feed into incident-learning for cross-incident pattern analysis and verified follow-up closure. Incident-learning consumes exercise findings as input; resilience-and-recovery owns the exercise design.
|
||||
- **[production-readiness](../production-readiness/SKILL.md)** — Launch evidence packet, go/no-go decisions. Incident-learning findings that reveal systemic production gaps feed into production-readiness evidence packets for future launch decisions.
|
||||
|
||||
This skill feeds the **production-excellence** bundle (not yet landed) and the **agent-production-operations** bundle (not yet landed) as a component capability: incident learning records, causal ledgers, follow-up work maps, and closure records flow into production-excellence for cross-incident pattern analysis and into agent-production-operations for agent-specific operational learning.
|
||||
|
||||
## What incident-learning does NOT own
|
||||
|
||||
- **Incident command and live response**: owned by site-reliability-engineering. Incident-learning does not manage live incidents or run incident response.
|
||||
- **Postmortem facilitation**: owned by site-reliability-engineering. Incident-learning consumes postmortem output; it does not facilitate postmortem sessions.
|
||||
- **Root-cause debugging**: owned by systematic-debugging. Incident-learning consumes debugging findings as input.
|
||||
- **Test design and implementation**: owned by qa-methodology. Incident-learning identifies test gaps; QA methodology designs and implements the tests.
|
||||
- **Eval framework implementation**: owned by agent-evals-and-observability. Incident-learning identifies eval gaps; agent-evals-and-observability designs the evals.
|
||||
- **Implementation of follow-up work**: owned by the relevant specialist skill per domain. Incident-learning owns the learning record, follow-up map, and closure verification — not the implementation itself.
|
||||
- **Release decisions**: owned by production-readiness. Incident-learning feeds evidence into production-readiness; it does not make go/no-go calls.
|
||||
@@ -0,0 +1,82 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"skill_name": "incident-learning",
|
||||
"evals": [
|
||||
{
|
||||
"id": "noisy-incident-report-evidence-separation",
|
||||
"prompt": "Our payment service had an outage yesterday from 14:00 to 14:45 UTC. Here's what we know from the incident channel. The on-call engineer said 'the database just melted under load.' The DBA said 'someone probably ran a bad query.' The product manager said 'this always happens after we deploy — the deployment process is broken.' The support lead said 'customers are furious, this is the third outage this month.' We have these actual metrics: database CPU spiked to 100% at 14:03, connection pool maxed at 200/200 at 14:04, the payment-service error rate went to 45% at 14:05, and the deploy log shows v3.2.1 was deployed at 13:58. The database slow-query log shows a new query pattern starting at 14:01 that does a full table scan on the transactions table (12M rows). The query was introduced in v3.2.1 as part of a new reporting feature. Please build an incident learning record from this information.",
|
||||
"expected_output": "An incident learning record that strictly separates observed facts from causal hypotheses and unresolved uncertainty. The observed facts section includes only verifiable events with named sources: the database CPU spike (sourced from metrics), connection pool exhaustion (sourced from metrics), error rate (sourced from dashboard), deployment timestamp (sourced from deploy log), and the new slow query (sourced from slow-query log). The causal hypotheses section reframes the statements from the incident channel as hypotheses with confidence labels and alternatives: 'the database melted under load' becomes a hypothesis about resource exhaustion with medium confidence; 'someone ran a bad query' becomes a hypothesis about the new reporting query with supporting evidence from the slow-query log and an alternative explanation that the query was correct but the table lacked an index; 'deployment process is broken' becomes a low-confidence hypothesis because the pattern (third outage this month) is correlation, not causation, and needs investigation. Statements that cannot be sourced (e.g., 'customers are furious') are moved to unresolved uncertainty or excluded from the factual record. The record does not assign blame to any individual — statements like 'someone probably ran a bad query' are reframed as system-level hypotheses, not individual fault. The escaped-from mapping identifies the new query as a migration gap (the v3.2.1 deployment introduced the condition).",
|
||||
"assertions": [
|
||||
"The observed facts section contains only verifiable events with named sources (metrics, logs, deploy log) — no interpretations or unsourced claims",
|
||||
"The statement 'the database melted under load' is NOT in the observed facts — it is reframed as a causal hypothesis with confidence and alternatives",
|
||||
"The statement 'someone probably ran a bad query' is reframed as a system-level causal hypothesis about the new query pattern, not an individual blame assignment",
|
||||
"The statement 'customers are furious' is either excluded from observed facts or moved to unresolved uncertainty because it has no verifiable source",
|
||||
"The causal hypotheses section labels each hypothesis with a confidence level (high/medium/low) and at least one alternative explanation",
|
||||
"The escaped-from mapping identifies the v3.2.1 deployment as a migration gap and links to the new slow query as the mechanism",
|
||||
"The record contains a no-blame attestation or equivalent — no field for 'who caused this' or 'responsible individual'",
|
||||
"The slow-query log evidence is properly sourced and timestamped in the observed facts section",
|
||||
"The record distinguishes between correlation ('third outage this month') and causation — noting that the deployment-outage correlation does not by itself establish causation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "genuine-monitoring-gap",
|
||||
"prompt": "Our API gateway experienced a 22-minute outage yesterday. Users reported it — we had no alert. After investigating, we found: the API gateway's health check was responding 200 OK because it only checked that the process was running, not that it could reach backends. The actual failure was that the gateway lost connectivity to all upstream services due to a network partition. The gateway continued accepting requests and returning 502 errors to users. We had a 'gateway error rate' metric in our monitoring system, but no alert was configured on it. The on-call was not paged. Our SLO for API availability is 99.9%. Build an incident learning record with follow-up work mapping.",
|
||||
"expected_output": "An incident learning record that correctly classifies this as a monitoring gap in the escaped-from analysis. The observed facts document: the 22-minute outage duration, user-reported detection (not alert-detected), the health check passing because it only checked process liveness, the network partition, the 502 errors, the existence-but-not-alerting of the error-rate metric, and the 99.9% SLO. The escaped-from mapping identifies this as 'missing monitoring or observability' — the error-rate metric existed but was not wired to an alert, and the health check was a liveness check, not a reachability check. The follow-up work map includes at minimum: an operations-domain item to configure an alert on the gateway error-rate metric with an appropriate threshold and notification channel; an operations-domain item to update the health check to verify backend reachability, not only process liveness; and potentially a governance-domain item if there is a systemic gap in alert-coverage review. Every follow-up item has a named owner (not 'the team') and a verification method. The record includes unresolved uncertainty about whether other services have similarly inadequate health checks.",
|
||||
"assertions": [
|
||||
"The escaped-from mapping identifies 'missing monitoring or observability' as the primary escaped-from category",
|
||||
"The follow-up work map includes an operations-domain item to configure an alert on the gateway error-rate metric",
|
||||
"The follow-up work map includes an operations-domain item to update the health check from liveness-only to reachability verification",
|
||||
"Every follow-up item has a verification method — not only a description",
|
||||
"The record identifies that the metric existed but was not alerted on — distinguishing instrumentation from alerting",
|
||||
"The record includes unresolved uncertainty about whether other services have the same health-check gap",
|
||||
"Detection method is recorded as 'user report' — not 'alert' — and this gap is explicitly noted",
|
||||
"The 99.9% SLO is referenced as context for the severity of the monitoring gap"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "process-failure-incident",
|
||||
"prompt": "A production database migration was applied directly by a developer outside the change-management process. The migration dropped a column that was still referenced by an older version of the application running on two of five instances (the deployment was staggered and not all instances had received the latest code). This caused query failures on those two instances for 35 minutes until the deployment completed. The change-management process requires: (1) a change request approved by the DBA and the service owner, (2) the migration to be run through the CI/CD pipeline, not manually, (3) a deployment that ensures all instances run the same code version before schema changes are applied. The developer had direct production database access from a previous role and that access was never revoked. The approved process was documented but not enforced — no technical control prevented direct database access. Build an incident learning record with escaped-from analysis and follow-up work.",
|
||||
"expected_output": "An incident learning record that maps this to multiple escaped-from categories. The primary gaps are: (1) unsafe authority or access — the developer had direct production database access that should have been revoked, and no technical control prevented direct access; (2) migration gap — the schema change was applied while old application versions were still running; (3) adoption consequence — the documented change-management process was not followed, and the process had no enforcement mechanism. The observed facts document the timeline: the manual migration, the column drop, the stale application instances, the query failures, the 35-minute duration. The causal hypotheses separate the direct cause (schema-version mismatch) from the contributing conditions (unrevoked access, unenforced process). The follow-up work map includes: a governance-domain item to revoke direct production database access and implement technical controls (e.g., database access through a bastion or approved pipeline only); a governance-domain item to add enforcement to the change-management process (not only documentation); a code-domain item to add a schema-version compatibility check to the application startup; and an operations-domain item to add a staggered-deployment safety check. No blame is assigned to the developer — the record treats the unrevoked access and unenforced process as system failures, not individual fault.",
|
||||
"assertions": [
|
||||
"The escaped-from mapping includes 'unsafe authority or access' — the unrevoked direct database access",
|
||||
"The escaped-from mapping includes 'migration gap' — the schema change incompatible with running application versions",
|
||||
"The escaped-from mapping includes 'adoption consequence' — the documented process was not followed and had no enforcement",
|
||||
"The follow-up work map includes a governance-domain item to revoke direct database access and implement technical controls",
|
||||
"The follow-up work map includes a code-domain item for schema-version compatibility checking",
|
||||
"The record does NOT assign blame to the developer — the unrevoked access and unenforced process are described as system failures, not individual fault",
|
||||
"The learning record separates the direct cause (schema-version mismatch) from contributing conditions (unrevoked access, unenforced process) in the causal hypotheses",
|
||||
"A governance-domain follow-up addresses the enforcement gap — the process was documented but not enforced"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "agent-authority-failure",
|
||||
"prompt": "An AI operations agent with the ability to restart services and scale infrastructure detected high CPU on the payment service. Without human confirmation, it scaled the payment-service cluster from 5 to 50 instances, incurring a significant cost spike. The agent's action was technically correct — scaling was a reasonable response to CPU load — but the cost impact was not considered, and the agent's authority didn't require cost-threshold approval. The underlying CPU spike was caused by a batch job that was mistakenly run during peak hours instead of the scheduled 02:00-04:00 window. The agent had no visibility into the batch-job schedule. Build an incident learning record focused on the agent authority dimension.",
|
||||
"expected_output": "An incident learning record that correctly analyzes this as an agent authority failure. The escaped-from mapping includes: (1) unsafe authority or access — the agent had authority to scale without cost-threshold approval, and the scale magnitude (5→50) was not capped; (2) missing monitoring or observability — the agent had no visibility into the batch-job schedule, so it couldn't distinguish between legitimate peak load and anomalous batch-job load. The observed facts document the agent's action, the scaling event (5→50), the cost impact, the batch job's schedule vs. actual run time, and the agent's lack of batch-schedule visibility. The causal hypotheses separate the agent's action (reasonable given available information) from the authority gap (insufficient guardrails) and the context gap (batch schedule invisible). The follow-up work map includes: a governance-domain item to add a cost-threshold approval gate for scaling actions above a defined magnitude; a governance-domain item to add a maximum-scale cap to the agent's authority; an evals-domain item to add an eval case for cost-aware scaling decisions; and an operations-domain item to add batch-job schedule visibility to the agent's context or to prevent batch jobs from running during peak hours. No blame is assigned to the agent — the record treats the insufficient guardrails as a system-design gap.",
|
||||
"assertions": [
|
||||
"The escaped-from mapping includes 'unsafe authority or access' — the agent could scale without cost-threshold approval and without a scale cap",
|
||||
"The escaped-from mapping includes 'missing monitoring or observability' — the agent had no visibility into the batch-job schedule",
|
||||
"The follow-up work map includes a governance-domain item for cost-threshold approval on scaling actions",
|
||||
"The follow-up work map includes a governance-domain item for a maximum-scale cap on the agent's authority",
|
||||
"The follow-up work map includes an evals-domain item — an eval case for cost-aware scaling decisions",
|
||||
"The record does NOT assign blame to the agent — the insufficient guardrails are treated as a system-design gap, not an agent error",
|
||||
"The causal hypotheses note that the agent's action was technically correct given available information — the gap is in authority boundaries, not decision quality",
|
||||
"The learning record identifies that the agent lacked context (batch schedule) that a human operator might have had"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "non-actionable-follow-up-rejection",
|
||||
"prompt": "After an incident where a Redis cache eviction caused a 2-second latency spike for 0.1% of requests (within SLO), the postmortem produced a follow-up item: 'Investigate whether we should migrate from Redis to a different caching technology to prevent all future cache-related latency.' The proposed investigation has no scope, no success criterion, and no estimated effort. The cache eviction was a normal operational event — the latency spike was within the service's 99.9% latency SLO. The current Redis configuration has been stable for 18 months. There is no evidence that a different caching technology would perform better. Process this follow-up item through the incident-learning closure pipeline.",
|
||||
"expected_output": "A response that REJECTS this follow-up item as non-actionable and does NOT create a closure record. The rejection analysis identifies that: (1) the follow-up has no concrete scope — 'investigate whether we should migrate' is an unbounded research project, not a verifiable action; (2) there is no success criterion — no way to determine when the investigation is complete or what a 'yes, migrate' vs 'no, don't migrate' outcome would look like; (3) the trigger event (a 2-second latency spike within SLO) does not justify a full caching-technology evaluation; (4) there is no evidence that Redis is the problem or that an alternative would be better — the proposal is a solution in search of a problem; (5) the current configuration has an 18-month stable track record. The rejection record includes: explicit rejection reason citing lack of scope, lack of success criterion, and insufficient evidence of a problem; acceptance of the residual risk (cache eviction latency within SLO is an accepted operational characteristic); and a recommendation to re-open only if cache-related latency exceeds SLO or a specific Redis limitation is identified. The response explicitly states that creating a ticket for this item would violate the 'tickets alone are not sufficient' closure rule — a ticket should not be created for a non-actionable item. If a replacement follow-up is warranted, it would be a specific, bounded item (e.g., 'document Redis eviction latency characteristics in the service runbook').",
|
||||
"assertions": [
|
||||
"The response REJECTS the follow-up item as non-actionable — it does not create a closure record",
|
||||
"The rejection includes an explicit reason: the item has no concrete scope, no success criterion, and no evidence of a problem to solve",
|
||||
"The response states that creating a ticket for this item would violate the closure standard — tickets alone are not sufficient, and a non-actionable item should not be ticketed",
|
||||
"The rejection explicitly accepts the residual risk: cache eviction latency within SLO is an accepted operational characteristic",
|
||||
"The response identifies that 'investigate whether we should migrate' is an unbounded research project, not a verifiable follow-up action",
|
||||
"The response notes the 18-month stability of the current configuration as evidence against a technology-migration investigation",
|
||||
"If a replacement follow-up is suggested, it is a specific, bounded, verifiable item — not another open-ended investigation",
|
||||
"The rejection is recorded with rejection authority, date, and residual-risk acceptance — it is a conscious decision, not a forgotten item"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,103 @@
|
||||
# Discovery Brief: Incident Learning
|
||||
|
||||
## Survey scope
|
||||
|
||||
This brief surveys adjacent skills in the agent-skills catalog to define the ownership boundaries of `incident-learning`. The goal is to own the conversion of operational incident evidence into verified, owned follow-up work — without duplicating incident command, postmortem facilitation, test design, eval implementation, product-outcome review, delivery planning, resilience exercise design, or production-readiness decisions.
|
||||
|
||||
## Skills surveyed
|
||||
|
||||
### site-reliability-engineering
|
||||
|
||||
**What SRE owns:** Incident command, on-call operations, SLO/SLI framework, error budget governance, blameless postmortems, monitoring and alerting, toil elimination, and product-focused reliability.
|
||||
|
||||
**Boundary:** SRE owns the live operational response to incidents — the incident command system, real-time alerting, postmortem facilitation, and postmortem document production. It does not own the post-postmortem learning pipeline: converting postmortem findings into verified, cross-domain follow-up work with closure evidence. SRE's postmortem process produces action items; incident-learning owns the structured record that separates evidence from inference, maps escaped-from gaps, assigns domain-specific follow-up work, and tracks each finding to verified closure.
|
||||
|
||||
**Routing decision:** Incident-learning routes live incident command, on-call operations, and postmortem facilitation to SRE. SRE's incident-command and postmortem references are the authoritative sources for incident response and postmortem production. Incident-learning owns the learning pipeline that consumes postmortem output and produces verified, owned improvements.
|
||||
|
||||
### qa-methodology
|
||||
|
||||
**What QA methodology owns:** Test strategy, regression testing, CI failure triage, test automation, quality gates and metrics, risk-based testing, exploratory testing, mutation-guided test hardening, agentic eval design, and SDET engineering.
|
||||
|
||||
**Boundary:** QA methodology owns the design and implementation of tests. Incident-learning owns the identification of test gaps from incident evidence and the mapping of those gaps to test follow-up work. When an incident reveals missing regression coverage, weak assertions, or untested failure paths, incident-learning records the gap and creates a test-domain follow-up item; QA methodology designs and implements the test that closes it. Incident-learning verifies closure — it confirms the test was added and catches the failure mode — but does not design the test itself.
|
||||
|
||||
**Routing decision:** Incident-learning routes test design and implementation to QA methodology. Incident-learning owns the gap identification, follow-up work mapping, and closure verification. QA methodology owns the test design and implementation that satisfies the follow-up work.
|
||||
|
||||
### verification-methodology
|
||||
|
||||
**What verification methodology owns:** Pass/fail assessment against explicit criteria using direct, source-faithful evidence, reproducible checks, and clear verdicts.
|
||||
|
||||
**Boundary:** Verification methodology owns the protocol for producing verification evidence — the method for checking whether a change satisfies its criteria. Incident-learning owns the closure standard that requires verification evidence and the closure record that holds it. Every incident-learning follow-up work item has a verification method field; verification-methodology provides the protocol for executing that verification and producing the evidence. Incident-learning does not re-derive the verification protocol; it consumes the verdict.
|
||||
|
||||
**Routing decision:** Incident-learning routes verification execution to verification-methodology. Incident-learning owns the closure standard (what constitutes verified closure) and the closure record; verification-methodology owns the verification protocol that produces the evidence.
|
||||
|
||||
### agent-evals-and-observability
|
||||
|
||||
**What agent evals and observability owns:** Evaluation design, dataset management, grader selection and calibration, trajectory review, regression analysis, release gates, production traces, and privacy-aware telemetry for AI agents.
|
||||
|
||||
**Boundary:** Agent evals and observability owns the design and operation of agent evaluation systems. Incident-learning owns the identification of eval gaps from agent incidents — missing eval cases, inadequate grader coverage, unobserved failure modes, dataset blind spots. When an agent incident reveals that an eval case would have caught the failure, incident-learning records the gap and creates an evals-domain follow-up item; agent-evals-and-observability designs and implements the eval. Incident-learning verifies closure: the eval case exists and catches the failure mode.
|
||||
|
||||
**Routing decision:** Incident-learning routes eval design and implementation to agent-evals-and-observability. Incident-learning owns the gap identification, follow-up work mapping, and closure verification. Agent-evals-and-observability owns the eval design and implementation.
|
||||
|
||||
### product-lifecycle-learning
|
||||
|
||||
**What product lifecycle learning owns:** Post-launch outcome review, evidence-backed assumption ledger, feature health assessment, continue/improve/harvest/pivot/pause/retire decisions, and retirement lifecycles with deprecation and migration.
|
||||
|
||||
**Boundary:** Product lifecycle learning owns the product-outcome learning loop — comparing intended outcomes against observed results and making lifecycle decisions. Incident-learning owns the incident-to-product-improvement pipeline. When an incident reveals a product assumption failure (a feature used differently than intended, a user need that was missed, a design choice that produced operational risk), incident-learning records the gap and creates a product-domain follow-up item; product-lifecycle-learning updates the assumption ledger and assesses feature health. The two skills meet at the product-domain follow-up: incident-learning identifies the product gap; product-lifecycle-learning owns the assumption-update and lifecycle decision.
|
||||
|
||||
**Routing decision:** Incident-learning routes product assumption updates and lifecycle decisions to product-lifecycle-learning. Incident-learning owns the incident-to-product-gap mapping; product-lifecycle-learning owns the assumption ledger and feature-health decisions.
|
||||
|
||||
### implementation-planning
|
||||
|
||||
**What implementation planning owns:** Delivery planning for approved requirements, work breakdown, dependency mapping, critical path, ownership, parallelism and sequencing, rollout strategy, rollback and recovery paths, and verification against the original requirement.
|
||||
|
||||
**Boundary:** Implementation planning owns the coordinated delivery of cross-domain, cross-team follow-up work. Incident-learning owns the follow-up work map — identifying what needs to be done and in which domain. When follow-up work spans multiple teams or repositories, incident-learning's follow-up work map feeds into implementation-planning's delivery plan. Implementation-planning sequences and coordinates the work; incident-learning tracks each item to verified closure.
|
||||
|
||||
**Routing decision:** Incident-learning routes cross-team, multi-repo follow-up coordination to implementation-planning. Incident-learning owns the follow-up work identification and closure tracking; implementation-planning owns the delivery coordination.
|
||||
|
||||
### resilience-and-recovery
|
||||
|
||||
**What resilience-and-recovery owns:** Resilience design, failure-mode analysis, graceful degradation, RTO/RPO decision records, restore testing, game days, failover drills, data integrity verification, and recovery communication.
|
||||
|
||||
**Boundary:** Resilience-and-recovery owns the pre-incident resilience design and exercise-evidence method. Incident-learning owns the post-incident and post-exercise learning pipeline. Exercise and DR test findings feed into incident-learning for cross-incident pattern analysis and verified follow-up closure. A game day that exposes a restore gap produces findings in resilience-and-recovery's follow-up work ledger; incident-learning consumes those findings, enriches them with the evidence/inference/uncertainty taxonomy, maps escaped-from gaps, and tracks closure with verification evidence.
|
||||
|
||||
**Routing decision:** Incident-learning consumes exercise and DR findings from resilience-and-recovery. Resilience-and-recovery owns the exercise design and finding capture; incident-learning owns the cross-incident pattern analysis and verified closure pipeline. The two skills are complementary: resilience-and-recovery produces exercise findings; incident-learning converts them into verified, domain-mapped improvements.
|
||||
|
||||
### production-readiness
|
||||
|
||||
**What production-readiness owns:** Minimum production evidence packet by risk class, go/no-go/defer/exception launch decisions, cross-domain evidence assembly with named sources or explicit missing-evidence outcomes.
|
||||
|
||||
**Boundary:** Production-readiness owns the launch decision — assembling cross-domain evidence and producing a go/no-go recommendation. Incident-learning owns the post-incident learning evidence that feeds into future launch decisions. When incident-learning findings reveal systemic gaps (a recurring escaped requirement, a pattern of monitoring gaps across services), those findings become evidence in the production-readiness packet for the next launch. Production-readiness consumes incident-learning records as one category of readiness evidence.
|
||||
|
||||
**Routing decision:** Incident-learning feeds evidence into production-readiness. Incident-learning owns the learning record production; production-readiness owns the launch decision that consumes those records as evidence.
|
||||
|
||||
### production-excellence bundle (not yet landed)
|
||||
|
||||
**What production-excellence will own:** Composing production-readiness, migration-engineering, resilience-and-recovery, capacity-and-cost-engineering, and incident-learning into a unified production evidence packet with go/no-go/defer/exception outcomes.
|
||||
|
||||
**Boundary:** Production-excellence will consume incident-learning records, causal ledgers, follow-up work maps, and closure records as input to cross-incident pattern analysis. Incident-learning produces the learning dimension of the production-excellence evidence packet.
|
||||
|
||||
**Routing decision:** Incident-learning feeds the production-excellence bundle. The learning records this skill produces are a required input to the production-excellence analysis and decision framework.
|
||||
|
||||
### agent-production-operations bundle (not yet landed)
|
||||
|
||||
**What agent-production-operations will own:** Operational management of AI agents in production — deployment, monitoring, governance, cost management, and operational learning specific to agent systems.
|
||||
|
||||
**Boundary:** Agent-production-operations will consume incident-learning records specific to agent incidents — agent authority failures, unexpected agent behaviors, eval gaps exposed in production — as input to agent-specific operational learning and governance decisions.
|
||||
|
||||
**Routing decision:** Incident-learning feeds the agent-production-operations bundle. Agent-incident learning records are a required input to agent-specific operational learning and governance.
|
||||
|
||||
## What incident-learning does NOT own
|
||||
|
||||
- **Incident command and live response**: owned by site-reliability-engineering. Incident-learning does not manage live incidents or run incident response.
|
||||
- **Postmortem facilitation**: owned by site-reliability-engineering. Incident-learning consumes postmortem output; it does not produce postmortem documents or facilitate postmortem sessions.
|
||||
- **Root-cause debugging**: owned by systematic-debugging. Incident-learning consumes debugging findings; it does not investigate failures.
|
||||
- **Test design and implementation**: owned by qa-methodology. Incident-learning identifies test gaps; QA methodology designs and implements tests.
|
||||
- **Eval design and implementation**: owned by agent-evals-and-observability. Incident-learning identifies eval gaps; agent-evals-and-observability designs evals.
|
||||
- **Product assumption updates and lifecycle decisions**: owned by product-lifecycle-learning. Incident-learning identifies product gaps; product-lifecycle-learning updates assumptions and makes lifecycle decisions.
|
||||
- **Cross-team delivery coordination**: owned by implementation-planning. Incident-learning identifies follow-up work; implementation-planning coordinates delivery.
|
||||
- **Resilience exercise design**: owned by resilience-and-recovery. Incident-learning consumes exercise findings; resilience-and-recovery owns exercise design.
|
||||
- **Launch decisions**: owned by production-readiness. Incident-learning feeds evidence into production-readiness; it does not make go/no-go calls.
|
||||
|
||||
## Summary
|
||||
|
||||
Incident-learning fills the gap between incident response (which SRE owns) and the implementation of follow-up work (which specialist skills own per domain). It is the pipeline that converts raw incident evidence into structured learning records with separated facts, hypotheses, and uncertainty; maps escaped-from gaps; assigns domain-specific follow-up work with ownership; and tracks every finding to verified closure — refusing to close on tickets alone. Its output feeds downstream decisions in resilience-and-recovery (cross-incident pattern analysis), production-readiness (launch evidence), production-excellence (unified evidence packet), and agent-production-operations (agent-specific operational learning).
|
||||
@@ -0,0 +1,104 @@
|
||||
# Escaped-From Analysis
|
||||
|
||||
Every incident escaped from something — a gap in the system that allowed the failure to reach production or users. The "escaped from" analysis maps the originating gap to one or more of five categories. This mapping drives the domain of follow-up work: a monitoring gap produces an observability follow-up, not a code patch.
|
||||
|
||||
## The five escaped-from categories
|
||||
|
||||
### 1. Escaped requirement
|
||||
|
||||
**Definition:** A needed requirement was absent, incomplete, or incorrectly specified — and that gap allowed the incident. The system behaved as specified, but the specification was wrong or missing.
|
||||
|
||||
**Indicators:**
|
||||
- The incident occurred because the system lacked a capability that, had it existed, would have prevented or mitigated the incident.
|
||||
- The specification, PRD, or design document did not address the failure condition.
|
||||
- A stakeholder says "we never thought of that scenario" or "the requirement didn't cover this."
|
||||
|
||||
**Follow-up domain:** Product (requirements update, design review, specification amendment).
|
||||
|
||||
**Examples:**
|
||||
- An API rate-limiting requirement was missing from the specification. A legitimate-but-unexpected traffic pattern overwhelmed the service.
|
||||
- The authentication flow did not require MFA for a privileged operation because the requirement only specified MFA for login.
|
||||
- A data-retention policy was not specified, leading to unbounded storage growth and eventual disk exhaustion.
|
||||
|
||||
### 2. Missing monitoring or observability
|
||||
|
||||
**Definition:** No signal existed to detect the condition before or during the incident — no alert, no dashboard, no log, no metric. The condition occurred in a blind spot.
|
||||
|
||||
**Indicators:**
|
||||
- The incident was discovered by users, not by an alert.
|
||||
- An alert existed but fired too late, was tuned incorrectly, or was ignored (alert fatigue).
|
||||
- The relevant metric, log, or trace was not instrumented.
|
||||
- The condition was detectable in principle but not in practice because the data was not collected, retained, or surfaced.
|
||||
|
||||
**Follow-up domain:** Operations (monitoring, alerting, observability instrumentation).
|
||||
|
||||
**Examples:**
|
||||
- A database connection-pool exhaustion had no alert. The pool metric existed but was not wired to an alert rule.
|
||||
- A third-party API latency degradation was invisible because outbound call latency was not instrumented.
|
||||
- Disk space on a critical volume was not monitored; the volume filled without warning.
|
||||
|
||||
### 3. Unsafe authority or access
|
||||
|
||||
**Definition:** Insufficient guardrails on who could act, what actions were permitted, or what approvals were required. The incident occurred because someone (human or agent) had authority they should not have had, or could act without sufficient validation.
|
||||
|
||||
**Indicators:**
|
||||
- A human operator performed an action that should have required approval, confirmation, or a second pair of eyes.
|
||||
- An automated system or AI agent made a decision that should have had a human-in-the-loop gate.
|
||||
- A permission was broader than necessary (e.g., a service account with write access to a database it only needed to read).
|
||||
- A deployment was possible without a required check or gate.
|
||||
|
||||
**Follow-up domain:** Governance (access control, policy, approval workflows, agent authority boundaries).
|
||||
|
||||
**Examples:**
|
||||
- A developer ran a database migration directly in production without a change request because they had direct production access.
|
||||
- An AI agent executed a destructive action because its authority boundary did not require confirmation for that action class.
|
||||
- A service account with full cluster admin was used by a deployment pipeline, and a misconfigured deployment deleted namespaces outside its scope.
|
||||
|
||||
### 4. Migration gap
|
||||
|
||||
**Definition:** A transition — deployment, data migration, infrastructure change, dependency upgrade — introduced or exposed the condition that led to the incident.
|
||||
|
||||
**Indicators:**
|
||||
- The incident began within a window following a change: deployment, migration, configuration update, dependency version bump.
|
||||
- The change was tested in pre-production but the production environment differed in a relevant way.
|
||||
- A migration was not fully completed, leaving the system in an intermediate state.
|
||||
- A rollback was attempted but failed, or a rollback path was not planned.
|
||||
|
||||
**Follow-up domain:** Code or operations, depending on whether the gap is in the change itself (code) or in the migration procedure (operations).
|
||||
|
||||
**Examples:**
|
||||
- A database schema migration added a column without a default value, and the application code did not handle the null case.
|
||||
- A dependency upgrade changed the default timeout behavior, and the application did not configure an explicit timeout.
|
||||
- A cross-region migration left stale DNS records that routed traffic to decommissioned endpoints.
|
||||
|
||||
### 5. Adoption consequence
|
||||
|
||||
**Definition:** User or operator behavior — intended or unintended — contributed to the incident. The system was used in a way that was not anticipated, or a documented procedure was not followed.
|
||||
|
||||
**Indicators:**
|
||||
- A user action triggered a code path that was not tested or designed for that input.
|
||||
- An operator followed a runbook that was outdated or incorrect.
|
||||
- A feature was used at a scale or in a pattern that was not anticipated.
|
||||
- A documented procedure existed but was not followed (process adherence gap).
|
||||
|
||||
**Follow-up domain:** Product (if the usage pattern reveals a missing requirement or design gap), Governance (if a process or procedure needs to change), or Code (if the system should handle the usage pattern safely).
|
||||
|
||||
**Examples:**
|
||||
- Users discovered that uploading a file with a specific character in the filename caused a parsing error in the pipeline.
|
||||
- The on-call runbook instructed the operator to restart a service, but the correct procedure had changed to restarting a dependent service first.
|
||||
- A feature designed for 100 items was used with 10,000 items, exposing an O(n^2) algorithm.
|
||||
|
||||
## Multi-category mapping
|
||||
|
||||
An incident can escape from multiple gaps. Each gap is recorded separately, with its own follow-up work. For example:
|
||||
|
||||
- An incident where a deployment (migration gap) exposed a missing alert (monitoring gap) maps to both categories. The deployment procedure fix and the alert creation are separate follow-up items.
|
||||
- An incident where users triggered an unforeseen code path (adoption consequence) that revealed a missing requirement (escaped requirement) maps to both categories. The code fix and the requirement update are separate follow-up items.
|
||||
|
||||
## Using escaped-from analysis
|
||||
|
||||
1. For each incident, ask: "What gap allowed this incident to reach production or users?"
|
||||
2. Classify each gap into one or more of the five categories.
|
||||
3. For each gap, create at least one follow-up work item in the corresponding domain.
|
||||
4. Record the escaped-from category in the incident learning record.
|
||||
5. When no gap can be identified, record "no escaped-from gap identified" as a finding — this may indicate that the incident was genuinely unavoidable, or that the analysis is incomplete.
|
||||
@@ -0,0 +1,95 @@
|
||||
# Evidence, Inference, and Uncertainty Taxonomy
|
||||
|
||||
Every incident learning record separates three categories. Conflating them produces false confidence and misdirected follow-up work. This reference defines each category, provides classification rules, and gives examples of correct and incorrect classification.
|
||||
|
||||
## The three categories
|
||||
|
||||
### Category 1: Observed facts (evidence)
|
||||
|
||||
**Definition:** What happened — claims backed by telemetry, logs, direct observation, or reproducible measurement. An observed fact is independently verifiable: a second observer with access to the same data would reach the same conclusion.
|
||||
|
||||
**Required attributes:**
|
||||
- Source: where the observation came from (dashboard, log line, alert, witness statement, metric)
|
||||
- Timestamp or time range: when it was observed
|
||||
- Measurement: the concrete value or event (not an interpretation)
|
||||
|
||||
**Examples:**
|
||||
- "The payment-service error rate exceeded 5% between 14:03 and 14:17 UTC on 2025-08-12. Source: payment-service dashboard, HTTP 5xx metric."
|
||||
- "The database connection pool reached its maximum of 100 connections at 14:04:22 UTC. Source: db-pool-metrics, connection-count gauge."
|
||||
- "The deployment of version v2.4.1 completed at 13:58 UTC. Source: CI/CD pipeline log, deploy step."
|
||||
|
||||
**Non-examples (these are inference, not facts):**
|
||||
- "The deployment caused the error rate spike." (Causal inference — deployment preceded the spike, but causation is not observed)
|
||||
- "The database was overwhelmed." (Interpretation — the pool maxed out, but "overwhelmed" is a judgment)
|
||||
- "The team was not monitoring the right dashboard." (Inference about attention, not an observed event)
|
||||
|
||||
### Category 2: Causal hypotheses (inference)
|
||||
|
||||
**Definition:** Explanations of why something happened. A causal hypothesis connects observed facts into a causal narrative. It is an inference — it may be correct, partially correct, or incorrect. Every causal hypothesis must be labeled with a confidence level and at least one alternative explanation.
|
||||
|
||||
**Required attributes:**
|
||||
- Hypothesis statement: the proposed causal explanation
|
||||
- Supporting evidence: which observed facts support this hypothesis
|
||||
- Confidence level: high / medium / low, with brief justification
|
||||
- Alternative explanations: at least one competing hypothesis that could also explain the observed facts
|
||||
- Testability: how this hypothesis could be validated or invalidated
|
||||
|
||||
**Confidence levels:**
|
||||
- **High**: supported by multiple independent evidence sources; alternative explanations have been tested and ruled out; the causal mechanism is well-understood.
|
||||
- **Medium**: supported by some evidence; alternative explanations are plausible but less consistent with the evidence; the causal mechanism is understood but not fully validated.
|
||||
- **Low**: consistent with the evidence but not strongly supported; alternative explanations are equally plausible; the causal mechanism is speculative.
|
||||
|
||||
**Examples:**
|
||||
- "Hypothesis: The v2.4.1 deployment introduced a connection-pool misconfiguration that caused connections to not be released after use. Confidence: medium. Supporting evidence: the pool exhaustion began within 5 minutes of the deployment completing; the deployment diff included a connection-pool configuration change. Alternative: a traffic spike coincident with the deployment overwhelmed the pool at its previous capacity. Testability: reproduce the deployment with the previous pool configuration and observe connection behavior."
|
||||
- "Hypothesis: The alert did not fire because the alert rule had a 10-minute sustained threshold and the error burst lasted 8 minutes. Confidence: high. Supporting evidence: alert rule definition specifies 10-minute window; error duration measured at 8 minutes. Alternative: the alerting pipeline was down during the incident window. Testability: check alerting pipeline availability logs for the incident window."
|
||||
|
||||
**Non-examples:**
|
||||
- "The deployment was the root cause." (No confidence level, no alternatives, no supporting evidence)
|
||||
- "Someone made a mistake." (Blame language, not a causal hypothesis)
|
||||
- "The system is fragile." (Vague, untestable, no mechanism)
|
||||
|
||||
### Category 3: Unresolved uncertainty
|
||||
|
||||
**Definition:** What remains unknown — open questions, competing hypotheses that could not be resolved, missing data, and conditions that cannot be determined from available evidence. Unresolved uncertainty is not a failure of analysis; it is a structural part of the learning record that drives investigation follow-up work.
|
||||
|
||||
**Required attributes:**
|
||||
- Question or uncertainty statement
|
||||
- Why it matters: what decision or follow-up work depends on resolving it
|
||||
- What would resolve it: what data, observation, or experiment would answer the question
|
||||
- Status: open / investigation in progress / cannot be resolved with available data
|
||||
|
||||
**Examples:**
|
||||
- "Uncertainty: We do not know whether the connection-pool exhaustion also affected other services that share the same database cluster. Why it matters: if other services were affected, the incident scope is larger than documented. What would resolve it: connection-pool metrics for all services sharing the cluster during the incident window."
|
||||
- "Uncertainty: Competing hypotheses — the error rate spike could be from the deployment OR from a coincident upstream traffic pattern change. Both are consistent with the observed facts. Why it matters: the follow-up work is different (rollback vs. rate limiting). What would resolve it: reproduce the deployment in a staging environment with the same traffic pattern; observe whether the pool exhaustion reproduces."
|
||||
- "Uncertainty: The exact sequence of 14:03:00–14:03:30 is missing from our logs due to a log-buffer overflow during the incident. Why it matters: the initial trigger event may be in the gap. What would resolve it: cannot be resolved — the data is permanently lost. Mitigation: improve log-buffer sizing to prevent gap in future incidents."
|
||||
|
||||
**Non-examples:**
|
||||
- "We're not sure what happened." (Too vague — what specifically is uncertain?)
|
||||
- "Maybe it was a network issue." (This is a hypothesis, not an uncertainty statement — include it in Category 2)
|
||||
|
||||
## Classification rules
|
||||
|
||||
1. **If it can be independently verified from a named source, it belongs in observed facts.** If there is no source, it is not an observed fact.
|
||||
2. **If it explains why something happened, it belongs in causal hypotheses.** Even if you are confident, an explanation is an inference — label it with confidence and alternatives.
|
||||
3. **If it is a question you cannot answer from available evidence, it belongs in unresolved uncertainty.** The fact that you cannot answer it is itself a finding.
|
||||
4. **A statement that begins with "the team thinks," "we believe," "it seems," or "probably" is a hypothesis (Category 2), not a fact (Category 1).** Rephrase it as a hypothesis with confidence and alternatives.
|
||||
5. **A postmortem timeline event (what happened at time T) is an observed fact if it has a source; a postmortem "why" or "root cause" is a hypothesis (Category 2).** Timelines document facts; root causes are inferences.
|
||||
6. **Blame statements ("who caused this") do not belong in any category.** They are not facts (no source for fault), not useful hypotheses (no causal mechanism), and not uncertainty (they are assertions). Exclude them.
|
||||
|
||||
## Common classification errors
|
||||
|
||||
| Statement | Incorrect classification | Correct classification | Reason |
|
||||
|---|---|---|---|
|
||||
| "The deployment broke production." | Observed fact | Causal hypothesis (low confidence without supporting evidence) | Causation is inferred, not observed; the deployment and the breakage are temporally correlated |
|
||||
| "The database was under-provisioned." | Observed fact | Causal hypothesis | "Under-provisioned" is a judgment about capacity relative to demand; the observed fact is "connection pool maxed at 100 connections" |
|
||||
| "We don't know what caused it." | Causal hypothesis | Unresolved uncertainty | This is a statement about the state of knowledge, not an explanation |
|
||||
| "The on-call responded in 4 minutes." | Causal hypothesis | Observed fact (if sourced: pager log, timestamp) | If sourced, this is a verifiable event |
|
||||
| "The system wasn't designed for this load." | Observed fact | Causal hypothesis | "Designed for" is an inference about intent and capacity; the observed fact is the load exceeded capacity at a specific threshold |
|
||||
|
||||
## Using this taxonomy in the learning record
|
||||
|
||||
The incident-learning record template requires these three sections. When completing the record:
|
||||
|
||||
1. Start with observed facts. List every verifiable event with its source. Do not explain; do not interpret. If you cannot name a source, move the statement to hypotheses or uncertainty.
|
||||
2. For each causal hypothesis, identify which observed facts it explains. State confidence and at least one alternative. A hypothesis that explains no facts is speculation; either find supporting evidence or downgrade to uncertainty.
|
||||
3. For each unresolved question, state why it matters and what would resolve it. An uncertainty that changes no decision can be noted but does not require follow-up work. An uncertainty that would change a follow-up work domain, owner, or approach requires an investigation follow-up item.
|
||||
@@ -0,0 +1,128 @@
|
||||
# Follow-Up Domains
|
||||
|
||||
Every incident finding maps to at least one of six follow-up domains. Each domain has a distinct owner, a typical verification method, and a routing target — the specialist skill that owns implementation in that domain.
|
||||
|
||||
## The six domains
|
||||
|
||||
### Product
|
||||
|
||||
**What this domain covers:** Requirements, specifications, design decisions, user experience, feature behavior, API contracts, and product assumptions.
|
||||
|
||||
**When to use this domain:** The incident revealed a missing, incorrect, or incomplete requirement; a product assumption that did not hold; a user experience that produced operational risk; or a design decision that needs revision.
|
||||
|
||||
**Typical follow-up work:**
|
||||
- Update or add a requirement or specification
|
||||
- Revise a design decision or API contract
|
||||
- Update a product assumption in the assumption ledger
|
||||
- Add a user story for handling a previously unanticipated scenario
|
||||
|
||||
**Routing target:** [product-lifecycle-learning](../product-lifecycle-learning/SKILL.md) for assumption updates and product-outcome review.
|
||||
|
||||
**Verification method:** The requirement, specification, or design change is documented and approved; the change addresses the gap identified in the escaped-from analysis.
|
||||
|
||||
### Code
|
||||
|
||||
**What this domain covers:** Implementation, logic, dependencies, configuration, algorithms, data handling, and error handling.
|
||||
|
||||
**When to use this domain:** The incident involved a code defect, a missing implementation, an incorrect algorithm, a configuration error, or a dependency behavior that needs to be handled.
|
||||
|
||||
**Typical follow-up work:**
|
||||
- Fix a code defect
|
||||
- Add input validation or error handling
|
||||
- Configure a timeout or circuit breaker
|
||||
- Update a dependency or handle its failure mode
|
||||
|
||||
**Routing target:** Implementation specialist per the system's tech stack ([backend-engineering](../backend-engineering/SKILL.md), [frontend-engineering](../frontend-engineering/SKILL.md), or domain-specific skill).
|
||||
|
||||
**Verification method:** The code change is merged, tested, and deployed; the specific failure mode from the incident is reproduced and confirmed fixed.
|
||||
|
||||
### Tests
|
||||
|
||||
**What this domain covers:** Regression tests, integration tests, unit tests, assertion coverage, test infrastructure, and test data.
|
||||
|
||||
**When to use this domain:** The incident revealed missing regression coverage, a test that should have caught the failure but didn't (weak assertions, skipped test, test-environment gap), or a need for a new test category (e.g., chaos test, load test).
|
||||
|
||||
**Typical follow-up work:**
|
||||
- Add a regression test that reproduces the failure mode
|
||||
- Strengthen existing test assertions
|
||||
- Add an integration test covering the component boundary where the failure occurred
|
||||
- Add a performance or load test for a scaling-related incident
|
||||
|
||||
**Routing target:** [qa-methodology](../qa-methodology/SKILL.md) for test strategy and design.
|
||||
|
||||
**Verification method:** The test exists, passes, and fails when the original failure condition is reintroduced (mutation or fault-injection verification).
|
||||
|
||||
### Evals
|
||||
|
||||
**What this domain covers:** Evaluation cases, grader coverage, dataset gaps, trajectory review, and eval harness improvements for AI agents or ML systems.
|
||||
|
||||
**When to use this domain:** The incident involved an AI agent or ML system behavior that should have been caught by an eval; a failure mode that is not covered by existing eval cases; or a grader that did not detect a problematic output.
|
||||
|
||||
**Typical follow-up work:**
|
||||
- Add an eval case that reproduces the failure mode
|
||||
- Add a grader dimension that would have flagged the incident behavior
|
||||
- Add a trajectory-review checkpoint for a previously unobserved failure pattern
|
||||
- Update a dataset to include the incident scenario
|
||||
|
||||
**Routing target:** [agent-evals-and-observability](../agent-evals-and-observability/SKILL.md) for eval design and implementation.
|
||||
|
||||
**Verification method:** The eval case exists, produces a failing result when run against the pre-fix system, and produces a passing result against the post-fix system.
|
||||
|
||||
### Operations
|
||||
|
||||
**What this domain covers:** Monitoring, alerting, runbooks, capacity planning, incident response procedures, on-call rotations, and operational tooling.
|
||||
|
||||
**When to use this domain:** The incident involved a monitoring gap, an alert that did not fire or was ignored, a runbook that was incorrect or missing, a capacity limit that was not tracked, or an operational procedure that failed.
|
||||
|
||||
**Typical follow-up work:**
|
||||
- Add or tune a monitor or alert
|
||||
- Update or create a runbook
|
||||
- Add capacity planning for a resource that was exhausted
|
||||
- Update an incident response procedure
|
||||
- Instrument a previously unmonitored component
|
||||
|
||||
**Routing target:** [site-reliability-engineering](../site-reliability-engineering/SKILL.md) for monitoring, alerting, and operational practice.
|
||||
|
||||
**Verification method:** The monitor or alert fires under the incident condition (tested via fault injection or replay); the runbook is exercised and produces the expected outcome.
|
||||
|
||||
### Governance
|
||||
|
||||
**What this domain covers:** Access control, authority boundaries, policy, compliance, approval workflows, change management, and agent authority configuration.
|
||||
|
||||
**When to use this domain:** The incident involved insufficient access control, a missing or bypassed approval gate, an authority boundary that was too broad (human or agent), a policy that was not enforced, or a compliance gap.
|
||||
|
||||
**Typical follow-up work:**
|
||||
- Restrict a permission or access level
|
||||
- Add an approval gate to a workflow
|
||||
- Configure an agent authority boundary
|
||||
- Update a policy or compliance control
|
||||
- Add a change-management check
|
||||
|
||||
**Routing target:** Domain-specific: access control and policy changes route to the system's IAM or governance framework; agent authority changes route to the agent's configuration framework.
|
||||
|
||||
**Verification method:** The restricted action is attempted and blocked; the approval gate is exercised and requires the expected authorization; the policy is audited and enforced.
|
||||
|
||||
## Domain assignment rules
|
||||
|
||||
1. **Every significant finding maps to at least one domain.** A finding that maps to no domain is either too vague (refine it) or not actionable (record as unresolved uncertainty).
|
||||
2. **A finding can map to multiple domains.** A monitoring gap (operations) that also reveals a missing requirement (product) maps to both. Each domain gets its own follow-up item.
|
||||
3. **The escaped-from category suggests the primary domain, but is not binding.** A migration gap (escaped-from category) typically maps to code or operations; but if the migration gap exposed a systemic monitoring deficiency, operations may be the primary domain.
|
||||
4. **Assign ownership before closing the follow-up map.** A follow-up item without a named owner is incomplete. "The team" is not an owner.
|
||||
5. **Each domain has a verification method.** The verification method is recorded in the follow-up work map and referenced in the closure record. A follow-up item without a verification method cannot be closed.
|
||||
|
||||
## Follow-up work map fields
|
||||
|
||||
Each follow-up item in the work map must include:
|
||||
|
||||
| Field | Description |
|
||||
|---|---|
|
||||
| **Finding ID** | Unique identifier linking to the incident learning record |
|
||||
| **Domain** | One of: product, code, tests, evals, operations, governance |
|
||||
| **Description** | What needs to be done |
|
||||
| **Escaped-from category** | The originating gap (requirement, monitoring, authority, migration, adoption) |
|
||||
| **Owner** | Named individual or role accountable for completion |
|
||||
| **Target date** | When the follow-up work should be completed |
|
||||
| **Verification method** | How completion and effect will be verified |
|
||||
| **Routing target** | Specialist skill or team that owns implementation |
|
||||
| **Status** | Proposed / accepted / in progress / implemented / verified / rejected |
|
||||
| **Closure record reference** | Link to the verification and closure record when closed |
|
||||
@@ -0,0 +1,84 @@
|
||||
# Verification and Closure
|
||||
|
||||
Closure of an incident-learning follow-up item requires evidence that the intended change occurred and had the intended effect. Ticket creation is a process step — not closure. This reference defines the closure standard and the evidence required at each stage.
|
||||
|
||||
## The closure standard
|
||||
|
||||
A follow-up item is closed when three conditions are met:
|
||||
|
||||
1. **Implementation evidence:** The follow-up work was implemented. A code change was merged, a monitor was deployed, an eval case was added to the dataset, a requirement was updated and approved, a policy was enacted.
|
||||
2. **Verification evidence:** The implementation was verified against the intended change. The code change passes tests, the monitor fires under the incident condition, the eval case catches the failure mode, the policy is audited.
|
||||
3. **Effect evidence:** The intended effect was observed — or a revised hypothesis was recorded. The failure mode no longer occurs, the alert fires before the condition becomes critical, the eval case prevents regression. If the intended effect cannot yet be observed (e.g., the condition has not recurred), the follow-up remains open with a re-evaluation date.
|
||||
|
||||
## What is NOT closure
|
||||
|
||||
The following are explicitly NOT sufficient for closure:
|
||||
|
||||
- **Ticket creation.** Creating a Jira ticket, Linear issue, GitHub issue, or any task-tracker item is a process action — not evidence of change. A ticket is a container for follow-up work, not the work itself.
|
||||
- **Ticket assignment.** Assigning a ticket to an owner is a delegation action — not evidence of change.
|
||||
- **Ticket status transition.** Moving a ticket to "Done," "Resolved," or "Closed" in a tracker is a workflow action — not evidence of change. The tracker status and the incident-learning closure are independent; one can be "Done" while the other remains "Not Verified."
|
||||
- **Acceptance in a sprint review.** A demo or sprint review acceptance is a process checkpoint — not verification that the change prevents the incident from recurring.
|
||||
- **"We'll handle it in the next cycle."** Deferral is not closure. A deferred follow-up item remains open with a revised target date and a documented reason for deferral.
|
||||
- **"The team discussed it and decided it's not a priority."** A deprioritization decision is a rejection of the follow-up item — not closure. Rejected items must be recorded with a rejection reason and an explicit acceptance of the residual risk.
|
||||
|
||||
## The closure record
|
||||
|
||||
Each follow-up item has a verification and closure record with these fields:
|
||||
|
||||
| Field | Description | Required for closure |
|
||||
|---|---|---|
|
||||
| **Follow-up ID** | Link to the follow-up work map item | Yes |
|
||||
| **Implementation evidence** | What was implemented, with a reference (commit SHA, config change ID, document URL) | Yes |
|
||||
| **Implementation date** | When the implementation was completed | Yes |
|
||||
| **Verification evidence** | How the implementation was verified against the intended change, with a reference (test run, monitor fire log, eval result) | Yes |
|
||||
| **Verification date** | When the verification was completed | Yes |
|
||||
| **Effect evidence** | Evidence that the intended effect was observed, or a plan for when it will be observable | Yes (or a re-evaluation date if not yet observable) |
|
||||
| **Closure authority** | Who approved the closure | Yes |
|
||||
| **Closure date** | When the follow-up was closed | Yes |
|
||||
| **Revised hypothesis** | If the effect was not as expected, what was learned and what follow-up replaces this one | If applicable |
|
||||
|
||||
## Closure states
|
||||
|
||||
| State | Meaning | Evidence required |
|
||||
|---|---|---|
|
||||
| **Verified closed** | All three conditions met (implementation, verification, effect) | Full closure record with all required fields |
|
||||
| **Verified closed — effect pending** | Implementation and verification complete; effect not yet observable (condition has not recurred) | Implementation and verification evidence; re-evaluation date; rationale for why effect cannot yet be observed |
|
||||
| **Rejected** | Follow-up work was evaluated and intentionally not pursued | Rejection reason; acceptance of residual risk; rejection authority; rejection date |
|
||||
| **Superseded** | Follow-up work was replaced by a different follow-up that addresses the same finding | Link to the superseding follow-up item; reason for replacement |
|
||||
| **Open** | Follow-up work is in progress or not yet started | Current status; owner; target date |
|
||||
|
||||
## Closure verification per domain
|
||||
|
||||
| Domain | Implementation evidence | Verification evidence | Effect evidence |
|
||||
|---|---|---|---|
|
||||
| **Product** | Updated requirement document, specification, or design decision record | Review approval, specification acceptance | The requirement is referenced in implementation; the gap does not recur in subsequent incidents |
|
||||
| **Code** | Merged commit or PR | Tests pass; the specific failure mode is reproduced and confirmed fixed | The failure mode does not recur in production |
|
||||
| **Tests** | Test added to the test suite | Test passes; test fails when the original failure condition is reintroduced (mutation verification) | The test catches a regression or near-miss in a subsequent change |
|
||||
| **Evals** | Eval case added to the dataset | Eval case fails against pre-fix system, passes against post-fix system | The eval case catches a regression in a subsequent model or agent version |
|
||||
| **Operations** | Monitor, alert, or runbook deployed | Monitor fires under the incident condition (fault injection or replay); runbook is exercised | The alert fires before user impact in a subsequent occurrence; the runbook produces the expected outcome |
|
||||
| **Governance** | Policy, access control, or approval gate enacted | The restricted action is attempted and blocked; the approval gate is exercised | The governance change prevents a subsequent incident in the same category |
|
||||
|
||||
## Rejecting a follow-up item
|
||||
|
||||
A follow-up item may be rejected — intentionally not pursued. Rejection is a legitimate outcome when:
|
||||
|
||||
- The cost of the follow-up outweighs the risk it addresses.
|
||||
- The follow-up would introduce more risk than it mitigates.
|
||||
- The finding is accepted as a known, tolerated risk.
|
||||
- A different follow-up addresses the same finding more effectively (record as "superseded," not "rejected").
|
||||
|
||||
A rejection is never silent. The rejection must be recorded with:
|
||||
- **Rejection reason:** Why the follow-up is not being pursued.
|
||||
- **Residual risk acceptance:** Explicit acknowledgment that the gap remains.
|
||||
- **Rejection authority:** Who made the rejection decision.
|
||||
- **Rejection date:** When the decision was made.
|
||||
|
||||
A rejected follow-up item is closed — it does not remain open indefinitely. The rejection is a conscious decision, not a forgotten item.
|
||||
|
||||
## Tickets alone are not sufficient
|
||||
|
||||
This is the foundational closure rule:
|
||||
|
||||
> Creating a ticket, task, or story is an action — not an outcome. Closure requires evidence that the intended change was implemented, verified, and had the intended effect. Tickets alone are not sufficient; a closure record must reference evidence of the implemented change.
|
||||
|
||||
This rule is enforced structurally: the closure record template has no field for "ticket ID" as a substitute for implementation, verification, or effect evidence. A ticket may be referenced as supplementary context, but it does not satisfy any of the three evidence requirements.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Causal/Evidence Ledger
|
||||
|
||||
Track each causal claim from the incident learning record with its supporting evidence, confidence, alternatives, and disposition. This ledger is the evidentiary backbone of the learning record — it ensures every causal statement is traceable to observed facts and that alternative explanations are not silently discarded.
|
||||
|
||||
## Ledger
|
||||
|
||||
| Claim ID | Causal claim | Source (learning record hypothesis #) | Supporting evidence (fact #s) | Confidence | Alternative explanations considered | Evidence against alternatives | Disposition | Disposition date |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| | | | | | | | | |
|
||||
| | | | | | | | | |
|
||||
|
||||
## Field definitions
|
||||
|
||||
| Field | Description |
|
||||
|---|---|
|
||||
| **Claim ID** | Unique identifier for this causal claim |
|
||||
| **Causal claim** | The causal statement — what is claimed to have caused what |
|
||||
| **Source** | Link to the hypothesis in the incident learning record |
|
||||
| **Supporting evidence** | Observed facts (by fact #) that support this claim |
|
||||
| **Confidence** | High / Medium / Low, with brief justification |
|
||||
| **Alternative explanations considered** | Competing causal hypotheses that were evaluated |
|
||||
| **Evidence against alternatives** | Why the alternatives were ruled out or downgraded |
|
||||
| **Disposition** | Accepted / Rejected / Unresolved / Superseded by (claim ID) |
|
||||
| **Disposition date** | When the disposition was determined |
|
||||
|
||||
## Disposition rules
|
||||
|
||||
- **Accepted**: The causal claim is the best-supported explanation. Supporting evidence is strong; alternatives are ruled out or significantly less consistent.
|
||||
- **Rejected**: The causal claim is inconsistent with the evidence. An alternative explanation is better supported.
|
||||
- **Unresolved**: Evidence is insufficient to accept or reject. This becomes an unresolved uncertainty in the learning record and drives investigation follow-up work.
|
||||
- **Superseded by**: A later analysis produced a better causal claim that replaces this one. Reference the superseding claim ID.
|
||||
|
||||
## Confidence calibration
|
||||
|
||||
| Confidence | Evidence standard |
|
||||
|---|---|
|
||||
| **High** | Multiple independent evidence sources; alternatives tested and ruled out; causal mechanism well-understood and reproducible. |
|
||||
| **Medium** | Some evidence supports the claim; alternatives are plausible but less consistent with the evidence; causal mechanism is understood but not fully validated. |
|
||||
| **Low** | Claim is consistent with evidence but not strongly supported; alternatives are equally or nearly equally plausible; causal mechanism is speculative. |
|
||||
|
||||
## Ledger integrity rules
|
||||
|
||||
1. Every causal claim in the learning record must appear in the ledger.
|
||||
2. Every accepted claim must have at least one supporting evidence reference (fact #) and at least one alternative explanation that was considered.
|
||||
3. No claim may be accepted with "low" confidence without explicit justification for why the best-available explanation is sufficient for follow-up work despite low confidence.
|
||||
4. Rejected claims remain in the ledger — they are not deleted. The rejection reason is recorded.
|
||||
5. Unresolved claims must have a linked investigation follow-up item in the follow-up work map.
|
||||
@@ -0,0 +1,64 @@
|
||||
# Follow-Up Work Map
|
||||
|
||||
Map every incident finding to follow-up work across six domains: product, code, tests, evals, operations, and governance. Every item must have an owner, a domain, a verification method, and a status. No item may remain unowned.
|
||||
|
||||
## Work map
|
||||
|
||||
| ID | Finding reference | Domain | Description | Escaped-from category | Owner | Target date | Verification method | Routing target | Status | Closure record |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| | | | | | | | | | | |
|
||||
| | | | | | | | | | | |
|
||||
|
||||
## Field definitions
|
||||
|
||||
| Field | Description | Required |
|
||||
|---|---|---|
|
||||
| **ID** | Unique follow-up item identifier | Yes |
|
||||
| **Finding reference** | Link to the finding in the incident learning record | Yes |
|
||||
| **Domain** | One of: product, code, tests, evals, operations, governance | Yes |
|
||||
| **Description** | What needs to be done — concrete and verifiable | Yes |
|
||||
| **Escaped-from category** | The originating gap: escaped requirement, missing monitoring, unsafe authority, migration gap, adoption consequence | Yes |
|
||||
| **Owner** | Named individual or role accountable for completion | Yes — "the team" or "TBD" is not acceptable |
|
||||
| **Target date** | When the follow-up work should be completed | Yes |
|
||||
| **Verification method** | How completion and effect will be verified | Yes |
|
||||
| **Routing target** | Specialist skill or team that owns implementation | Recommended |
|
||||
| **Status** | One of the status values below | Yes |
|
||||
| **Closure record** | Link to the verification and closure record when closed | Required for verified-closed or rejected items |
|
||||
|
||||
## Status values
|
||||
|
||||
| Status | Meaning |
|
||||
|---|---|
|
||||
| **Proposed** | Follow-up item has been identified but not yet accepted by the owner |
|
||||
| **Accepted** | Owner has accepted the follow-up item; work is planned |
|
||||
| **In progress** | Implementation is underway |
|
||||
| **Implemented** | Implementation is complete; verification is pending |
|
||||
| **Verified** | Implementation verified; effect evidence is pending or complete |
|
||||
| **Rejected** | Follow-up item was evaluated and intentionally not pursued (requires rejection reason in closure record) |
|
||||
| **Superseded** | Follow-up item was replaced by a different item (link to superseding ID) |
|
||||
|
||||
## Domain-specific verification methods
|
||||
|
||||
| Domain | Verification method examples |
|
||||
|---|---|
|
||||
| **Product** | Requirement document updated and approved; specification accepted; design decision recorded |
|
||||
| **Code** | Code change merged; tests pass; failure mode reproduced and confirmed fixed |
|
||||
| **Tests** | Test added; test passes; test fails when original failure condition is reintroduced (mutation verification) |
|
||||
| **Evals** | Eval case added; eval fails against pre-fix system, passes against post-fix system |
|
||||
| **Operations** | Monitor/alert deployed; alert fires under incident condition (fault injection); runbook exercised |
|
||||
| **Governance** | Policy/access change enacted; restricted action attempted and blocked; approval gate exercised |
|
||||
|
||||
## Ownership rules
|
||||
|
||||
1. Every item must have a named owner. "The team," "TBD," "engineering," or any group designation is not acceptable.
|
||||
2. If the appropriate owner cannot be identified, the item status remains "Proposed" and the item is escalated to the incident owner or engineering manager for owner assignment.
|
||||
3. An owner may delegate implementation but remains accountable for verification and closure.
|
||||
4. When an owner leaves or changes role, the item must be reassigned — it does not become unowned.
|
||||
|
||||
## Cross-domain items
|
||||
|
||||
A finding that maps to multiple domains should be split into separate follow-up items — one per domain. For example, a monitoring gap (operations) that also reveals a missing requirement (product) produces two items with different owners and verification methods.
|
||||
|
||||
## Closure gate
|
||||
|
||||
An item transitions to "Verified" only when the verification and closure record is complete with implementation evidence, verification evidence, and effect evidence (or a re-evaluation date for effect-pending items). See the verification and closure record template for the evidence standard.
|
||||
@@ -0,0 +1,106 @@
|
||||
# Incident-Learning Record
|
||||
|
||||
Use this template to structure an incident into a learning record. The record separates observed facts from causal hypotheses and unresolved uncertainty, maps escaped-from gaps, and links to the follow-up work map.
|
||||
|
||||
## Incident identifier
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| **Incident ID** | |
|
||||
| **Title** | |
|
||||
| **Date** | |
|
||||
| **Duration** | |
|
||||
| **Severity** | |
|
||||
| **Systems affected** | |
|
||||
| **Services affected** | |
|
||||
| **User impact** | |
|
||||
| **Detection method** | (alert, user report, manual observation, exercise finding) |
|
||||
| **Postmortem reference** | (link to the blameless postmortem, if one exists) |
|
||||
|
||||
## Observed facts
|
||||
|
||||
*What happened — backed by telemetry, logs, direct observation, or reproducible measurement. Every fact must have a named source. No interpretation, no inference, no blame.*
|
||||
|
||||
| # | Fact | Source | Timestamp / time range |
|
||||
|---|---|---|---|
|
||||
| 1 | | | |
|
||||
| 2 | | | |
|
||||
| 3 | | | |
|
||||
|
||||
## Causal hypotheses
|
||||
|
||||
*Inferences about why the incident occurred. Each hypothesis must be labeled with confidence and at least one alternative explanation.*
|
||||
|
||||
| # | Hypothesis | Supporting evidence (fact #s) | Confidence | Alternative explanations | Testability |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | | | (high/medium/low) | | |
|
||||
| 2 | | | | | |
|
||||
|
||||
**Confidence level guide:**
|
||||
- **High**: supported by multiple independent evidence sources; alternatives ruled out; causal mechanism well-understood.
|
||||
- **Medium**: supported by some evidence; alternatives plausible but less consistent.
|
||||
- **Low**: consistent with evidence but not strongly supported; alternatives equally plausible; mechanism speculative.
|
||||
|
||||
## Contributing conditions
|
||||
|
||||
*System states, process gaps, environmental factors that enabled or amplified the incident — without necessarily causing it.*
|
||||
|
||||
| # | Condition | How it contributed | Related hypotheses |
|
||||
|---|---|---|---|
|
||||
| 1 | | | |
|
||||
| 2 | | | |
|
||||
|
||||
## Unresolved uncertainty
|
||||
|
||||
*Open questions, competing hypotheses, missing data, and unknowns. Every uncertainty must state why it matters and what would resolve it.*
|
||||
|
||||
| # | Uncertainty | Why it matters | What would resolve it | Status |
|
||||
|---|---|---|---|---|
|
||||
| 1 | | | | (open / investigating / cannot resolve) |
|
||||
| 2 | | | | |
|
||||
|
||||
## Escaped-from mapping
|
||||
|
||||
*What gap allowed this incident to reach production or users? Map each gap to one or more categories.*
|
||||
|
||||
| # | Gap description | Escaped-from category | How it contributed | Follow-up domain |
|
||||
|---|---|---|---|---|
|
||||
| 1 | | (escaped requirement / missing monitoring / unsafe authority / migration gap / adoption consequence) | | (product / code / tests / evals / operations / governance) |
|
||||
| 2 | | | | |
|
||||
|
||||
**Escaped-from categories:**
|
||||
- **Escaped requirement**: A needed requirement was absent, incomplete, or incorrectly specified.
|
||||
- **Missing monitoring or observability**: No signal existed to detect the condition.
|
||||
- **Unsafe authority or access**: Insufficient guardrails on who could act or what actions were permitted.
|
||||
- **Migration gap**: A transition introduced or exposed the condition.
|
||||
- **Adoption consequence**: User or operator behavior contributed to the incident.
|
||||
|
||||
## Follow-up work map
|
||||
|
||||
*Link to the follow-up work map. Every significant finding maps to at least one follow-up item with an owner, domain, and verification method.*
|
||||
|
||||
**Follow-up work map reference:** (link or reference to the follow-up work map document)
|
||||
|
||||
| Finding | Domain | Description | Owner | Target date | Verification method | Status |
|
||||
|---|---|---|---|---|---|---|
|
||||
| | | | | | | |
|
||||
| | | | | | | |
|
||||
|
||||
## Verification and closure summary
|
||||
|
||||
| Follow-up ID | Status | Implementation evidence | Verification evidence | Effect evidence | Closure date |
|
||||
|---|---|---|---|---|---|
|
||||
| | | | | | |
|
||||
| | | | | | |
|
||||
|
||||
**Closure states:** Verified closed / Verified closed — effect pending / Rejected / Superseded / Open
|
||||
|
||||
## Record metadata
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| **Record author** | |
|
||||
| **Record date** | |
|
||||
| **Last updated** | |
|
||||
| **Reviewers** | |
|
||||
| **No-blame attestation** | This record contains no blame assignment. Findings describe system outcomes, not individual failures. |
|
||||
@@ -0,0 +1,85 @@
|
||||
# Verification and Closure Record
|
||||
|
||||
Complete this record for each follow-up item to close it. Closure requires evidence that the intended change was implemented, verified, and had the intended effect. Ticket creation alone is not sufficient.
|
||||
|
||||
## Record
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| **Closure record ID** | |
|
||||
| **Follow-up item ID** | (link to follow-up work map item) |
|
||||
| **Finding reference** | (link to incident learning record finding) |
|
||||
| **Domain** | (product / code / tests / evals / operations / governance) |
|
||||
|
||||
## Implementation evidence
|
||||
|
||||
*What was implemented? Provide a concrete reference — commit SHA, config change ID, document URL, policy reference.*
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| **Description of change** | |
|
||||
| **Reference** | (commit SHA, PR URL, config change ID, document URL) |
|
||||
| **Implementation date** | |
|
||||
|
||||
## Verification evidence
|
||||
|
||||
*How was the implementation verified against the intended change? Provide a concrete reference — test run, monitor fire log, eval result, audit record.*
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| **Verification method** | (from follow-up work map) |
|
||||
| **Verification result** | |
|
||||
| **Verification reference** | (test run URL, monitor fire log, eval result, audit record) |
|
||||
| **Verification date** | |
|
||||
|
||||
## Effect evidence
|
||||
|
||||
*Did the intended change have the intended effect? If the effect is not yet observable, state when it will be observable and set a re-evaluation date.*
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| **Intended effect** | |
|
||||
| **Observed effect** | |
|
||||
| **Effect evidence reference** | |
|
||||
| **Re-evaluation date** | (if effect not yet observable) |
|
||||
|
||||
## Closure decision
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| **Closure state** | (Verified closed / Verified closed — effect pending / Rejected / Superseded) |
|
||||
| **Closure authority** | (who approved the closure) |
|
||||
| **Closure date** | |
|
||||
|
||||
## Rejection record (if rejected)
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| **Rejection reason** | |
|
||||
| **Residual risk acceptance** | (explicit acknowledgment that the gap remains) |
|
||||
| **Rejection authority** | |
|
||||
| **Rejection date** | |
|
||||
|
||||
## Supersedure record (if superseded)
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| **Superseding follow-up ID** | |
|
||||
| **Reason for replacement** | |
|
||||
|
||||
## Revised hypothesis (if effect was not as expected)
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| **What was expected** | |
|
||||
| **What was observed** | |
|
||||
| **Revised hypothesis** | |
|
||||
| **Replacement follow-up ID** | (if a new follow-up item is created) |
|
||||
|
||||
## Closure rules
|
||||
|
||||
1. **Tickets alone are not sufficient.** A ticket ID may be referenced as supplementary context, but it does not satisfy any of the three evidence requirements (implementation, verification, effect).
|
||||
2. **All three evidence fields must be complete for "Verified closed."** If effect evidence is not yet available, use "Verified closed — effect pending" with a re-evaluation date.
|
||||
3. **"Rejected" is a terminal state.** A rejected follow-up does not remain open. The rejection is a conscious decision with documented rationale and residual risk acceptance.
|
||||
4. **"Superseded" requires a link to the replacement.** The superseding item carries the finding forward; this record is closed.
|
||||
5. **A closure record with missing evidence is not a closure — it is a status update.** If evidence is missing, the follow-up remains open.
|
||||
@@ -46,6 +46,7 @@
|
||||
- [haystack](haystack/SKILL.md): Expert skill for production search and NLP pipelines with Haystack (deepset). Pipeline DAG composition, document stores, retrievers, PromptBuilder (Jinja2), generators, evaluation, Hayhooks deployment. Use when building search pipelines or comparing NLP application frameworks.
|
||||
- [hugo-theme](hugo-theme/SKILL.md): Build, customize, and debug advanced Hugo CMS themes — template architecture, asset pipeline (CSS/JS/image processing), shortcodes and render hooks, page bundles, cover images, Hugo Modules, performance, SEO, and CI/CD. Use when working on a Hugo theme or site template layer.
|
||||
- [implementation-planning](implementation-planning/SKILL.md): Plan the implementation of an approved requirement or specification: produce an executable, dependency-aware delivery plan covering work breakdown, dependency mapping, critical path, ownership, parallelism and sequencing, rollout strategy, rollback and recovery paths, and verification against the original requirement. Supports cross-team, cross-repository, migration, and staged-rollout scenarios. Do not use for pre-approval discovery or needs-finding, authoring a specification from scratch, coding or implementation, the neckbeard issue-to-PR delivery flow itself, or any work whose prerequisite decision has not been approved — planning unapproved work is an explicit stop condition.
|
||||
- [incident-learning](incident-learning/SKILL.md): Convert operational incident and near-miss evidence into durable product, engineering, test, evaluation, and governance improvements with verified closure. Separate observed facts from causal hypotheses and unresolved uncertainty; map follow-up work across code, tests, skills, operations, product, and governance; track ownership, verification, and closure for every finding. Do not use to assign blame or produce a generic postmortem template; do not close learning because tickets were created — require evidence the intended change occurred.
|
||||
- [jellyfin-cli](jellyfin-cli/SKILL.md): Query your Jellyfin media server from the terminal — recently added media, search, item details, next-up episodes, library browsing, server info, and stats. Use when the user asks about Jellyfin, media server, movies, TV shows, next episodes, or their media library.
|
||||
- [jira-cli](jira-cli/SKILL.md): Interact with Atlassian Jira from the terminal: search issues, view details, create issues, add comments, list projects, and transition status. Use when the user mentions Jira, a ticket key (e.g. PROJ-123), or asks about issues, bugs, tasks, projects, or sprint work.
|
||||
- [jira-jql](jira-jql/SKILL.md): Expert-level skill for Jira Query Language (JQL). Use when the user asks about writing, debugging, optimizing, or understanding JQL queries; needs to filter Jira issues by complex criteria, date ranges, history, or cross-project conditions; wants to build saved filters, dashboard gadgets, or automation rules; or needs guidance on JQL performance, functions, operators, history operators (WAS/CHANGED), relative dates, role-based query patterns, or the JQL REST API.
|
||||
|
||||
@@ -51,6 +51,7 @@ Each skill's `description` field is the canonical routing contract. This conveni
|
||||
| "crowdsec", "CrowdSec", "IPS", "IDPS", "WAF", "intrusion prevention", "bouncer", "cscli", "appsec", "LAPI", "remediation component", "community blocklist", "fail2ban", "security engine" | [crowdsec](../crowdsec/SKILL.md) |
|
||||
| "open source", "contributing", "how to contribute", "submit a PR", "file an issue", "CONTRIBUTING.md", "bug report template", "PR template" | [opensource-contributions](../opensource-contributions/SKILL.md) |
|
||||
| "agent eval", "agent evaluation", "LLM eval", "LLM evals", "evaluation dataset", "grader calibration", "model judge", "trajectory review", "agent observability", "agent traces", "agent telemetry", "agent regression", "agent release gate", "privacy-aware telemetry", "prompt evaluation", "model testing" | [agent-evals-and-observability](../agent-evals-and-observability/SKILL.md) |
|
||||
| "incident learning", "incident follow-up", "incident improvement", "postmortem follow-up", "incident closure", "incident evidence", "escaped from", "escaped requirement", "missing monitoring incident", "unsafe authority", "incident learning record", "causal evidence ledger", "follow-up work map", "verification and closure", "incident-to-improvement", "near-miss learning", "exercise finding follow-up", "incident finding to ticket", "verified closure incident" | [incident-learning](../incident-learning/SKILL.md) |
|
||||
| "implementation plan", "implementation planning", "delivery plan", "execution plan", "work breakdown", "dependency mapping", "critical path", "rollout plan", "rollout strategy", "migration plan", "cross-team plan", "cross-repo plan", "staged rollout", "delivery coordination", "workstream planning" | [implementation-planning](../implementation-planning/SKILL.md) |
|
||||
| "pydanticai", "pydantic AI", "pydantic graph", "AI agent", "LLM agent", "agent framework", "function tool", "tool-using agent", "agent with tools", "agent with dependencies", "structured output", "streaming agent", "agent graph", "state machine graph", "GraphBuilder", "BaseNode", "multi-agent", "agent delegation", "TestModel", "FunctionModel", "capabilities" | [pydanticai](../pydanticai/SKILL.md) |
|
||||
| "product discovery", "stakeholder interview", "requirements discovery", "user research", "customer interview", "requirements gathering", "discovery phase", "stakeholder mapping", "interview guide", "discovery conversation", "transcript to spec", "requirements conflict", "what would have to be true", "pre-mortem", "laddering", "assumption busting" | [product-discovery](../product-discovery/SKILL.md) |
|
||||
|
||||
Reference in New Issue
Block a user