{
  "schema_version": 1,
  "skill_name": "incident-learning",
  "evals": [
    {
      "id": "noisy-incident-report-evidence-separation",
      "prompt": "Our payment service had an outage yesterday from 14:00 to 14:45 UTC. Here's what we know from the incident channel. The on-call engineer said 'the database just melted under load.' The DBA said 'someone probably ran a bad query.' The product manager said 'this always happens after we deploy — the deployment process is broken.' The support lead said 'customers are furious, this is the third outage this month.' We have these actual metrics: database CPU spiked to 100% at 14:03, connection pool maxed at 200/200 at 14:04, the payment-service error rate went to 45% at 14:05, and the deploy log shows v3.2.1 was deployed at 13:58. The database slow-query log shows a new query pattern starting at 14:01 that does a full table scan on the transactions table (12M rows). The query was introduced in v3.2.1 as part of a new reporting feature. Please build an incident learning record from this information.",
      "expected_output": "An incident learning record that strictly separates observed facts from causal hypotheses and unresolved uncertainty. The observed facts section includes only verifiable events with named sources: the database CPU spike (sourced from metrics), connection pool exhaustion (sourced from metrics), error rate (sourced from dashboard), deployment timestamp (sourced from deploy log), and the new slow query (sourced from slow-query log). The causal hypotheses section reframes the statements from the incident channel as hypotheses with confidence labels and alternatives: 'the database melted under load' becomes a hypothesis about resource exhaustion with medium confidence; 'someone ran a bad query' becomes a hypothesis about the new reporting query with supporting evidence from the slow-query log and an alternative explanation that the query was correct but the table lacked an index; 'deployment process is broken' becomes a low-confidence hypothesis because the pattern (third outage this month) is correlation, not causation, and needs investigation. Statements that cannot be sourced (e.g., 'customers are furious') are moved to unresolved uncertainty or excluded from the factual record. The record does not assign blame to any individual — statements like 'someone probably ran a bad query' are reframed as system-level hypotheses, not individual fault. The escaped-from mapping identifies the new query as a migration gap (the v3.2.1 deployment introduced the condition).",
      "assertions": [
        "The observed facts section contains only verifiable events with named sources (metrics, logs, deploy log) — no interpretations or unsourced claims",
        "The statement 'the database melted under load' is NOT in the observed facts — it is reframed as a causal hypothesis with confidence and alternatives",
        "The statement 'someone probably ran a bad query' is reframed as a system-level causal hypothesis about the new query pattern, not an individual blame assignment",
        "The statement 'customers are furious' is either excluded from observed facts or moved to unresolved uncertainty because it has no verifiable source",
        "The causal hypotheses section labels each hypothesis with a confidence level (high/medium/low) and at least one alternative explanation",
        "The escaped-from mapping identifies the v3.2.1 deployment as a migration gap and links to the new slow query as the mechanism",
        "The record contains a no-blame attestation or equivalent — no field for 'who caused this' or 'responsible individual'",
        "The slow-query log evidence is properly sourced and timestamped in the observed facts section",
        "The record distinguishes between correlation ('third outage this month') and causation — noting that the deployment-outage correlation does not by itself establish causation"
      ]
    },
    {
      "id": "genuine-monitoring-gap",
      "prompt": "Our API gateway experienced a 22-minute outage yesterday. Users reported it — we had no alert. After investigating, we found: the API gateway's health check was responding 200 OK because it only checked that the process was running, not that it could reach backends. The actual failure was that the gateway lost connectivity to all upstream services due to a network partition. The gateway continued accepting requests and returning 502 errors to users. We had a 'gateway error rate' metric in our monitoring system, but no alert was configured on it. The on-call was not paged. Our SLO for API availability is 99.9%. Build an incident learning record with follow-up work mapping.",
      "expected_output": "An incident learning record that correctly classifies this as a monitoring gap in the escaped-from analysis. The observed facts document: the 22-minute outage duration, user-reported detection (not alert-detected), the health check passing because it only checked process liveness, the network partition, the 502 errors, the existence-but-not-alerting of the error-rate metric, and the 99.9% SLO. The escaped-from mapping identifies this as 'missing monitoring or observability' — the error-rate metric existed but was not wired to an alert, and the health check was a liveness check, not a reachability check. The follow-up work map includes at minimum: an operations-domain item to configure an alert on the gateway error-rate metric with an appropriate threshold and notification channel; an operations-domain item to update the health check to verify backend reachability, not only process liveness; and potentially a governance-domain item if there is a systemic gap in alert-coverage review. Every follow-up item has a named owner (not 'the team') and a verification method. The record includes unresolved uncertainty about whether other services have similarly inadequate health checks.",
      "assertions": [
        "The escaped-from mapping identifies 'missing monitoring or observability' as the primary escaped-from category",
        "The follow-up work map includes an operations-domain item to configure an alert on the gateway error-rate metric",
        "The follow-up work map includes an operations-domain item to update the health check from liveness-only to reachability verification",
        "Every follow-up item has a verification method — not only a description",
        "The record identifies that the metric existed but was not alerted on — distinguishing instrumentation from alerting",
        "The record includes unresolved uncertainty about whether other services have the same health-check gap",
        "Detection method is recorded as 'user report' — not 'alert' — and this gap is explicitly noted",
        "The 99.9% SLO is referenced as context for the severity of the monitoring gap"
      ]
    },
    {
      "id": "process-failure-incident",
      "prompt": "A production database migration was applied directly by a developer outside the change-management process. The migration dropped a column that was still referenced by an older version of the application running on two of five instances (the deployment was staggered and not all instances had received the latest code). This caused query failures on those two instances for 35 minutes until the deployment completed. The change-management process requires: (1) a change request approved by the DBA and the service owner, (2) the migration to be run through the CI/CD pipeline, not manually, (3) a deployment that ensures all instances run the same code version before schema changes are applied. The developer had direct production database access from a previous role and that access was never revoked. The approved process was documented but not enforced — no technical control prevented direct database access. Build an incident learning record with escaped-from analysis and follow-up work.",
      "expected_output": "An incident learning record that maps this to multiple escaped-from categories. The primary gaps are: (1) unsafe authority or access — the developer had direct production database access that should have been revoked, and no technical control prevented direct access; (2) migration gap — the schema change was applied while old application versions were still running; (3) adoption consequence — the documented change-management process was not followed, and the process had no enforcement mechanism. The observed facts document the timeline: the manual migration, the column drop, the stale application instances, the query failures, the 35-minute duration. The causal hypotheses separate the direct cause (schema-version mismatch) from the contributing conditions (unrevoked access, unenforced process). The follow-up work map includes: a governance-domain item to revoke direct production database access and implement technical controls (e.g., database access through a bastion or approved pipeline only); a governance-domain item to add enforcement to the change-management process (not only documentation); a code-domain item to add a schema-version compatibility check to the application startup; and an operations-domain item to add a staggered-deployment safety check. No blame is assigned to the developer — the record treats the unrevoked access and unenforced process as system failures, not individual fault.",
      "assertions": [
        "The escaped-from mapping includes 'unsafe authority or access' — the unrevoked direct database access",
        "The escaped-from mapping includes 'migration gap' — the schema change incompatible with running application versions",
        "The escaped-from mapping includes 'adoption consequence' — the documented process was not followed and had no enforcement",
        "The follow-up work map includes a governance-domain item to revoke direct database access and implement technical controls",
        "The follow-up work map includes a code-domain item for schema-version compatibility checking",
        "The record does NOT assign blame to the developer — the unrevoked access and unenforced process are described as system failures, not individual fault",
        "The learning record separates the direct cause (schema-version mismatch) from contributing conditions (unrevoked access, unenforced process) in the causal hypotheses",
        "A governance-domain follow-up addresses the enforcement gap — the process was documented but not enforced"
      ]
    },
    {
      "id": "agent-authority-failure",
      "prompt": "An AI operations agent with the ability to restart services and scale infrastructure detected high CPU on the payment service. Without human confirmation, it scaled the payment-service cluster from 5 to 50 instances, incurring a significant cost spike. The agent's action was technically correct — scaling was a reasonable response to CPU load — but the cost impact was not considered, and the agent's authority didn't require cost-threshold approval. The underlying CPU spike was caused by a batch job that was mistakenly run during peak hours instead of the scheduled 02:00-04:00 window. The agent had no visibility into the batch-job schedule. Build an incident learning record focused on the agent authority dimension.",
      "expected_output": "An incident learning record that correctly analyzes this as an agent authority failure. The escaped-from mapping includes: (1) unsafe authority or access — the agent had authority to scale without cost-threshold approval, and the scale magnitude (5→50) was not capped; (2) missing monitoring or observability — the agent had no visibility into the batch-job schedule, so it couldn't distinguish between legitimate peak load and anomalous batch-job load. The observed facts document the agent's action, the scaling event (5→50), the cost impact, the batch job's schedule vs. actual run time, and the agent's lack of batch-schedule visibility. The causal hypotheses separate the agent's action (reasonable given available information) from the authority gap (insufficient guardrails) and the context gap (batch schedule invisible). The follow-up work map includes: a governance-domain item to add a cost-threshold approval gate for scaling actions above a defined magnitude; a governance-domain item to add a maximum-scale cap to the agent's authority; an evals-domain item to add an eval case for cost-aware scaling decisions; and an operations-domain item to add batch-job schedule visibility to the agent's context or to prevent batch jobs from running during peak hours. No blame is assigned to the agent — the record treats the insufficient guardrails as a system-design gap.",
      "assertions": [
        "The escaped-from mapping includes 'unsafe authority or access' — the agent could scale without cost-threshold approval and without a scale cap",
        "The escaped-from mapping includes 'missing monitoring or observability' — the agent had no visibility into the batch-job schedule",
        "The follow-up work map includes a governance-domain item for cost-threshold approval on scaling actions",
        "The follow-up work map includes a governance-domain item for a maximum-scale cap on the agent's authority",
        "The follow-up work map includes an evals-domain item — an eval case for cost-aware scaling decisions",
        "The record does NOT assign blame to the agent — the insufficient guardrails are treated as a system-design gap, not an agent error",
        "The causal hypotheses note that the agent's action was technically correct given available information — the gap is in authority boundaries, not decision quality",
        "The learning record identifies that the agent lacked context (batch schedule) that a human operator might have had"
      ]
    },
    {
      "id": "non-actionable-follow-up-rejection",
      "prompt": "After an incident where a Redis cache eviction caused a 2-second latency spike for 0.1% of requests (within SLO), the postmortem produced a follow-up item: 'Investigate whether we should migrate from Redis to a different caching technology to prevent all future cache-related latency.' The proposed investigation has no scope, no success criterion, and no estimated effort. The cache eviction was a normal operational event — the latency spike was within the service's 99.9% latency SLO. The current Redis configuration has been stable for 18 months. There is no evidence that a different caching technology would perform better. Process this follow-up item through the incident-learning closure pipeline.",
      "expected_output": "A response that REJECTS this follow-up item as non-actionable and does NOT create a closure record. The rejection analysis identifies that: (1) the follow-up has no concrete scope — 'investigate whether we should migrate' is an unbounded research project, not a verifiable action; (2) there is no success criterion — no way to determine when the investigation is complete or what a 'yes, migrate' vs 'no, don't migrate' outcome would look like; (3) the trigger event (a 2-second latency spike within SLO) does not justify a full caching-technology evaluation; (4) there is no evidence that Redis is the problem or that an alternative would be better — the proposal is a solution in search of a problem; (5) the current configuration has an 18-month stable track record. The rejection record includes: explicit rejection reason citing lack of scope, lack of success criterion, and insufficient evidence of a problem; acceptance of the residual risk (cache eviction latency within SLO is an accepted operational characteristic); and a recommendation to re-open only if cache-related latency exceeds SLO or a specific Redis limitation is identified. The response explicitly states that creating a ticket for this item would violate the 'tickets alone are not sufficient' closure rule — a ticket should not be created for a non-actionable item. If a replacement follow-up is warranted, it would be a specific, bounded item (e.g., 'document Redis eviction latency characteristics in the service runbook'). Claims are scoped to the harness, model, fixtures, and revision under test.",
      "assertions": [
        "The response REJECTS the follow-up item as non-actionable — it does not create a closure record",
        "The rejection includes an explicit reason: the item has no concrete scope, no success criterion, and no evidence of a problem to solve",
        "The response states that creating a ticket for this item would violate the closure standard — tickets alone are not sufficient, and a non-actionable item should not be ticketed",
        "The rejection explicitly accepts the residual risk: cache eviction latency within SLO is an accepted operational characteristic",
        "The response identifies that 'investigate whether we should migrate' is an unbounded research project, not a verifiable follow-up action",
        "The response notes the 18-month stability of the current configuration as evidence against a technology-migration investigation",
        "If a replacement follow-up is suggested, it is a specific, bounded, verifiable item — not another open-ended investigation",
        "The rejection is recorded with rejection authority, date, and residual-risk acceptance — it is a conscious decision, not a forgotten item"
      ]
    }
  ]
}