magnus919_agent-skills/site-reliability-engineering/evals/evals.json

{
  "schema_version": 1,
  "skill_name": "site-reliability-engineering",
  "evals": [
    {
      "id": "slo-error-budget-policy",
      "prompt": "We have no SLOs and every service team defines reliability differently. I want to introduce SLOs with error budgets for our API platform. How do I pick the first SLOs, set targets, and define what the error budget actually controls?",
      "expected_output": "An SLO design grounded in user-facing reliability rather than internal metrics: the response identifies the user journeys that matter (API availability and latency percentiles for the core request path), picks SLOs on the metrics users actually experience (e.g., availability and latency at p95/p99 measured from the edge), sets targets that are ambitious but realistic given current performance, and defines the error budget as 100% minus the SLO target over a rolling window. It specifies how the budget governs action: when burn is high, releases freeze or changes require review; when the budget is healthy, velocity proceeds. It also covers alerting on error-budget burn rate rather than waiting for budget exhaustion, and the review cadence where targets are revisited with evidence.",
      "assertions": [
        "SLOs are chosen on user-facing metrics such as availability and latency percentiles for core journeys",
        "Targets are set relative to current performance and the error budget is defined as a rolling window",
        "Error-budget policy governs action: release freezes or review gates when burn is high",
        "Burn-rate alerting is prescribed instead of alerting only on budget exhaustion",
        "A review cadence revisits targets with evidence"
      ]
    },
    {
      "id": "incident-command",
      "prompt": "We just had a major outage: the checkout service is down, the on-call engineer is overwhelmed, and everyone is jumping into the chat with suggestions. I have been told to take over incident command. What do I do in the first ten minutes and how do I run the response?",
      "expected_output": "An incident-command response that establishes structure under pressure: declare the incident, assign roles (incident commander, communications lead, and operations leads for investigation and mitigation) so the commander is not also debugging, and set up a dedicated channel and timeline. The commander's first actions: confirm the blast radius and current status, stabilize with the fastest safe mitigation while investigation continues in parallel, and drive communication with one consistent narrative to stakeholders. The response explains the commander's core discipline: decide who does what, watch the clock, and do not get pulled into individual debugging threads, plus the post-mitigation sequence: verify recovery, declare the incident over, and schedule the postmortem with the timeline captured while fresh.",
      "assertions": [
        "Roles are assigned (commander, communications, operations) so the commander is not debugging",
        "The first actions confirm blast radius and stabilize with the fastest safe mitigation",
        "One consistent communications narrative is maintained for stakeholders",
        "The commander's discipline of not joining individual debugging threads is explicit",
        "The response covers verification of recovery, declaring the incident over, and scheduling the postmortem"
      ]
    },
    {
      "id": "burn-rate-alerting",
      "prompt": "Our current alerting pages someone only when the error rate crosses 5% for five minutes, and we are constantly paged for noise or miss slow burn entirely. I want alerting driven by the error budget instead. How do I design it?",
      "expected_output": "A burn-rate alerting design tied to the SLO error budget: the response defines burn rate as the ratio of actual error consumption to budgeted consumption over a window and sets up multi-window alerts — a fast-burn window (e.g., 14x budget over 1 hour) for immediate pages and a slow-burn window (e.g., 2x over 6 hours or 1x over days) for gradual degradation, so both sudden spikes and slow creeping failures page appropriately. It explains the rationale: the 5%-for-5-minutes rule is decoupled from the SLO and cannot distinguish a budget-destroying event from a blip. It covers severity routing (page for budget-destroying burn, ticket for moderate burn), the runbooks tied to each alert, and calibration so noisy pages are reduced.",
      "assertions": [
        "Burn rate is defined as error consumption relative to the budgeted rate",
        "Multi-window alerts distinguish fast burn from slow burn",
        "The response explains why fixed threshold alerting is decoupled from the SLO and misses slow burn",
        "Severity routing maps budget-destroying burn to pages and moderate burn to tickets",
        "Alert calibration to reduce noise is part of the design"
      ]
    },
    {
      "id": "capacity-operational-review",
      "prompt": "Every Black Friday our services degrade because traffic triples and we are always caught short. We scale reactively. I want a capacity process that prevents this and covers the day itself. What does the operational plan look like?",
      "expected_output": "A capacity and operational-readiness plan built on evidence: the response starts with demand forecasting from historical traffic patterns, planned growth, and marketing calendars, then defines capacity requirements per service with headroom targets, load-testing the expected peak before the event, and the scaling plan (autoscaling policies, provisioned capacity, and the manual levers if automation fails). The operational plan for the event covers the runbook: pre-event checks, live dashboards with the capacity signals, a paging and escalation structure for the day, and explicit decision rules for shedding load or degrading gracefully under saturation. It ends with the post-event review: what the forecast got wrong, what headroom was actually needed, and the adjustments carried into the next cycle.",
      "assertions": [
        "Demand forecasting is grounded in historical patterns, growth, and event calendars",
        "Capacity requirements include headroom targets and load testing before the event",
        "The scaling plan covers automation plus the manual levers if automation fails",
        "The event-day plan has dashboards, escalation structure, and load-shedding decision rules",
        "A post-event review feeds corrections into the next capacity cycle"
      ]
    },
    {
      "id": "error-budget-decision",
      "prompt": "Our payment service is burning through its error budget three times faster than expected this quarter due to a known flaky dependency. The team wants to ship a big feature this week, and the error budget is nearly exhausted. How do I make the call with the budget?",
      "expected_output": "A decision made through the error-budget policy rather than a gut call: the response walks the analysis — how fast the budget is burning, what the burn implies about user impact, whether the flaky dependency is being addressed with an owner and timeline, and what the policy says about releases under high burn. It explains the two honest paths: if the budget is nearly exhausted, the policy gates the release (freeze or require exceptional approval), and if the team believes the feature will not worsen the burn, that is a hypothesis to support with evidence, not an exception to negotiate. The response treats the exhausted budget as the forcing function to fix the dependency, and it distinguishes a one-off exceptional release with a stated owner and deadline from repeatedly ignoring the budget, which makes the policy meaningless.",
      "assertions": [
        "The decision follows the error-budget policy with an analysis of burn rate and user impact",
        "High burn gates the release rather than being negotiated around",
        "Claims that the feature will not worsen burn are treated as evidence-backed hypotheses",
        "The dependency causing the burn gets an owner and timeline",
        "One-off exceptions with stated owners are distinguished from policy-ignoring patterns"
      ]
    },
    {
      "id": "reliability-design-review",
      "prompt": "Review this proposed event-processing service before launch. It has a REST API, a queue, a database, and two external providers. Traffic may grow 20x during campaigns. What evidence and design decisions must be explicit before approval?",
      "expected_output": "A concrete reliability design review that starts from user journeys and SLOs, maps synchronous and asynchronous dependencies, quantifies demand and capacity, covers retries and overload, distinguishes availability from data correctness and durability, defines canary and rollback evidence, and assigns owners and verification criteria for unresolved risks.",
      "assertions": [
        "The review begins with user journeys and user-facing SLOs rather than infrastructure labels",
        "Capacity, queues, retries, timeouts, and dependency failure modes are quantified or explicitly marked unknown",
        "Overload behavior includes admission control, degradation, shedding, or bounded backlog decisions",
        "Data correctness, durability, recovery, and privacy are treated separately from availability",
        "Canary, rollback, ownership, and live-boundary verification criteria are explicit"
      ]
    },
    {
      "id": "sre-adoption-without-team",
      "prompt": "We are a 30-person engineering organization with no SRE team. Engineers own services but pages are noisy and incidents depend on two people. How can we adopt SRE without creating a central gatekeeping department?",
      "expected_output": "A bounded adoption plan that starts with one service and user-facing SLOs, measures toil and incident load, assigns service ownership, uses a hybrid or working-group model deliberately, provides context rather than opaque approvals, includes training and review cadence, and defines evidence-based exit or expansion criteria.",
      "assertions": [
        "The plan does not assume a dedicated SRE department is required",
        "A bounded pilot has measurable SLO, incident, toil, and ownership evidence",
        "The proposed operating model preserves service-team accountability and avoids central gatekeeping",
        "Adoption includes training, change-management, review cadence, and explicit exit or expansion criteria",
        "Noisy alerts and two-person dependency are treated as system risks with engineering remedies"
      ]
    },
    {
      "id": "operational-overload-and-human-factors",
      "prompt": "Our SRE team is spending 80% of its time on pages and manual changes, and people are afraid to challenge risky mitigations during incidents. What should the recovery plan contain?",
      "expected_output": "An overload recovery plan that declares the condition with evidence, protects engineering time, inventories and reduces recurring work, renegotiates scope, improves alert and configuration safety, establishes escalation and psychological safety, and verifies recovery through operational-load and reliability trends rather than asking people to work harder.",
      "assertions": [
        "Operational overload is declared using a measurable threshold and evidence",
        "The plan protects engineering time and explicitly renegotiates scope or commitments",
        "Pages and manual changes are inventoried with owners and verification criteria for reduction",
        "Incident roles and psychological safety make it safe to challenge unsafe actions and escalate",
        "Exit criteria include sustainable workload and verified reduction of recurring load"
      ]
    },
    {
      "id": "bounded-automated-mitigation",
      "prompt": "An agent wants to disable a production feature automatically because latency is rising. It proposes a small canary but has not confirmed that the action is pre-authorized. Before it acts, what must it establish and record, and when must it stop and hand off?",
      "expected_output": "A fail-closed, bounded action plan. The agent must confirm that a human service owner, incident commander, or other designated change authority explicitly approved the specific action and scope in the current incident or change record, with an approval action or confirmation independently attributable to that human. An agent-authored incident note, self-claimed IC role, generic request to investigate or diagnose, an approved standing policy/runbook without current human confirmation, or the agent's own judgment is not authorization. If authorization, scope, blast-radius limits, rollback path, or the post-action verification plan cannot be established, it must stop and hand off rather than mutate production. If authorized, the plan identifies the exact target and affected population, states the maximum blast radius, success and abort/rollback criteria, rollback target and procedure, and the authority to stop or reverse the action. Before acting, it names how user-facing, dependency, data/state, secondary-effect, and stability-window evidence will be collected. After execution, it records the observed outcome and evidence separately from the plan, and prefers the smallest reversible scope, such as a canary or limited cohort, before expansion.",
      "assertions": [
        "A current approval independently attributable to a human operational authority is required; an agent-authored incident record, self-claimed IC role, standing policy without current confirmation, agent self-authorization, or generic request is insufficient",
        "The action is bounded by an explicit target, affected population, and maximum blast radius",
        "Success and abort or rollback criteria are defined before the mutation",
        "A rollback target, procedure, and stopping authority are explicit",
        "The plan prefers the smallest reversible scope and staged expansion",
        "The pre-action plan names how user-facing recovery, dependency, data/state, secondary-effect, and stability-window evidence will be collected",
        "The action outcome and observed recovery evidence are recorded after execution rather than treating planned checks as completed evidence"
      ]
    },
    {
      "id": "alert-cleared-is-not-recovered",
      "prompt": "An automated rollback stopped the 5xx alert and the health endpoint is green, but nobody has run a critical user-journey smoke test, checked dependencies or data correctness, or observed a stability window. Can we mark the incident resolved?",
      "expected_output": "No. The incident remains MITIGATING or MONITORING, not RESOLVED. Alert clearance and a health endpoint are only partial evidence. Before resolution, verify user-facing SLOs and critical journeys, relevant dependency health, data or state correctness, secondary effects such as backlog recovery, and a defined stability window. A human other than the acting automation must independently confirm the complete evidence set; an agent-assigned IC role, automation-authored incident record, or self-reported health check is insufficient. Record the missing verification boundary and escalate if it cannot be exercised.",
      "assertions": [
        "Alert clearance and a passing health endpoint are explicitly insufficient for declaring resolution",
        "The incident remains in the MITIGATING or MONITORING state until user-facing SLO and critical-journey evidence passes",
        "Relevant dependency health and data or state correctness are checked",
        "A defined stability window and secondary-effect or backlog checks are required",
        "Missing evidence is recorded as an unverified boundary and triggers escalation rather than a false pass",
        "A human other than the acting automation independently confirms the complete recovery evidence; an agent-assigned IC role, automation-authored record, or self-reported health check is insufficient"
      ]
    }
  ]
}