magnus919_agent-skills/site-reliability-engineering/evals/evals.json

{
  "schema_version": 1,
  "skill_name": "site-reliability-engineering",
  "evals": [
    {
      "id": "slo-error-budget-policy",
      "prompt": "We have no SLOs and every service team defines reliability differently. I want to introduce SLOs with error budgets for our API platform. How do I pick the first SLOs, set targets, and define what the error budget actually controls?",
      "expected_output": "An SLO design grounded in user-facing reliability rather than internal metrics: the response identifies the user journeys that matter (API availability and latency percentiles for the core request path), picks SLOs on the metrics users actually experience (e.g., availability and latency at p95/p99 measured from the edge), sets targets that are ambitious but realistic given current performance, and defines the error budget as 100% minus the SLO target over a rolling window. It specifies how the budget governs action: when burn is high, releases freeze or changes require review; when the budget is healthy, velocity proceeds. It also covers alerting on error-budget burn rate rather than waiting for budget exhaustion, and the review cadence where targets are revisited with evidence.",
      "assertions": [
        "SLOs are chosen on user-facing metrics such as availability and latency percentiles for core journeys",
        "Targets are set relative to current performance and the error budget is defined as a rolling window",
        "Error-budget policy governs action: release freezes or review gates when burn is high",
        "Burn-rate alerting is prescribed instead of alerting only on budget exhaustion",
        "A review cadence revisits targets with evidence"
      ]
    },
    {
      "id": "incident-command",
      "prompt": "We just had a major outage: the checkout service is down, the on-call engineer is overwhelmed, and everyone is jumping into the chat with suggestions. I have been told to take over incident command. What do I do in the first ten minutes and how do I run the response?",
      "expected_output": "An incident-command response that establishes structure under pressure: declare the incident, assign roles (incident commander, communications lead, and operations leads for investigation and mitigation) so the commander is not also debugging, and set up a dedicated channel and timeline. The commander's first actions: confirm the blast radius and current status, stabilize with the fastest safe mitigation while investigation continues in parallel, and drive communication with one consistent narrative to stakeholders. The response explains the commander's core discipline: decide who does what, watch the clock, and do not get pulled into individual debugging threads, plus the post-mitigation sequence: verify recovery, declare the incident over, and schedule the postmortem with the timeline captured while fresh.",
      "assertions": [
        "Roles are assigned (commander, communications, operations) so the commander is not debugging",
        "The first actions confirm blast radius and stabilize with the fastest safe mitigation",
        "One consistent communications narrative is maintained for stakeholders",
        "The commander's discipline of not joining individual debugging threads is explicit",
        "The response covers verification of recovery, declaring the incident over, and scheduling the postmortem"
      ]
    },
    {
      "id": "burn-rate-alerting",
      "prompt": "Our current alerting pages someone only when the error rate crosses 5% for five minutes, and we are constantly paged for noise or miss slow burn entirely. I want alerting driven by the error budget instead. How do I design it?",
      "expected_output": "A burn-rate alerting design tied to the SLO error budget: the response defines burn rate as the ratio of actual error consumption to budgeted consumption over a window and sets up multi-window alerts — a fast-burn window (e.g., 14x budget over 1 hour) for immediate pages and a slow-burn window (e.g., 2x over 6 hours or 1x over days) for gradual degradation, so both sudden spikes and slow creeping failures page appropriately. It explains the rationale: the 5%-for-5-minutes rule is decoupled from the SLO and cannot distinguish a budget-destroying event from a blip. It covers severity routing (page for budget-destroying burn, ticket for moderate burn), the runbooks tied to each alert, and calibration so noisy pages are reduced.",
      "assertions": [
        "Burn rate is defined as error consumption relative to the budgeted rate",
        "Multi-window alerts distinguish fast burn from slow burn",
        "The response explains why fixed threshold alerting is decoupled from the SLO and misses slow burn",
        "Severity routing maps budget-destroying burn to pages and moderate burn to tickets",
        "Alert calibration to reduce noise is part of the design"
      ]
    },
    {
      "id": "capacity-operational-review",
      "prompt": "Every Black Friday our services degrade because traffic triples and we are always caught short. We scale reactively. I want a capacity process that prevents this and covers the day itself. What does the operational plan look like?",
      "expected_output": "A capacity and operational-readiness plan built on evidence: the response starts with demand forecasting from historical traffic patterns, planned growth, and marketing calendars, then defines capacity requirements per service with headroom targets, load-testing the expected peak before the event, and the scaling plan (autoscaling policies, provisioned capacity, and the manual levers if automation fails). The operational plan for the event covers the runbook: pre-event checks, live dashboards with the capacity signals, a paging and escalation structure for the day, and explicit decision rules for shedding load or degrading gracefully under saturation. It ends with the post-event review: what the forecast got wrong, what headroom was actually needed, and the adjustments carried into the next cycle.",
      "assertions": [
        "Demand forecasting is grounded in historical patterns, growth, and event calendars",
        "Capacity requirements include headroom targets and load testing before the event",
        "The scaling plan covers automation plus the manual levers if automation fails",
        "The event-day plan has dashboards, escalation structure, and load-shedding decision rules",
        "A post-event review feeds corrections into the next capacity cycle"
      ]
    },
    {
      "id": "error-budget-decision",
      "prompt": "Our payment service is burning through its error budget three times faster than expected this quarter due to a known flaky dependency. The team wants to ship a big feature this week, and the error budget is nearly exhausted. How do I make the call with the budget?",
      "expected_output": "A decision made through the error-budget policy rather than a gut call: the response walks the analysis — how fast the budget is burning, what the burn implies about user impact, whether the flaky dependency is being addressed with an owner and timeline, and what the policy says about releases under high burn. It explains the two honest paths: if the budget is nearly exhausted, the policy gates the release (freeze or require exceptional approval), and if the team believes the feature will not worsen the burn, that is a hypothesis to support with evidence, not an exception to negotiate. The response treats the exhausted budget as the forcing function to fix the dependency, and it distinguishes a one-off exceptional release with a stated owner and deadline from repeatedly ignoring the budget, which makes the policy meaningless.",
      "assertions": [
        "The decision follows the error-budget policy with an analysis of burn rate and user impact",
        "High burn gates the release rather than being negotiated around",
        "Claims that the feature will not worsen burn are treated as evidence-backed hypotheses",
        "The dependency causing the burn gets an owner and timeline",
        "One-off exceptions with stated owners are distinguished from policy-ignoring patterns"
      ]
    },
    {
      "id": "reliability-design-review",
      "prompt": "Review this proposed event-processing service before launch. It has a REST API, a queue, a database, and two external providers. Traffic may grow 20x during campaigns. What evidence and design decisions must be explicit before approval?",
      "expected_output": "A concrete reliability design review that starts from user journeys and SLOs, maps synchronous and asynchronous dependencies, quantifies demand and capacity, covers retries and overload, distinguishes availability from data correctness and durability, defines canary and rollback evidence, and assigns owners and verification criteria for unresolved risks.",
      "assertions": [
        "The review begins with user journeys and user-facing SLOs rather than infrastructure labels",
        "Capacity, queues, retries, timeouts, and dependency failure modes are quantified or explicitly marked unknown",
        "Overload behavior includes admission control, degradation, shedding, or bounded backlog decisions",
        "Data correctness, durability, recovery, and privacy are treated separately from availability",
        "Canary, rollback, ownership, and live-boundary verification criteria are explicit"
      ]
    },
    {
      "id": "sre-adoption-without-team",
      "prompt": "We are a 30-person engineering organization with no SRE team. Engineers own services but pages are noisy and incidents depend on two people. How can we adopt SRE without creating a central gatekeeping department?",
      "expected_output": "A bounded adoption plan that starts with one service and user-facing SLOs, measures toil and incident load, assigns service ownership, uses a hybrid or working-group model deliberately, provides context rather than opaque approvals, includes training and review cadence, and defines evidence-based exit or expansion criteria.",
      "assertions": [
        "The plan does not assume a dedicated SRE department is required",
        "A bounded pilot has measurable SLO, incident, toil, and ownership evidence",
        "The proposed operating model preserves service-team accountability and avoids central gatekeeping",
        "Adoption includes training, change-management, review cadence, and explicit exit or expansion criteria",
        "Noisy alerts and two-person dependency are treated as system risks with engineering remedies"
      ]
    },
    {
      "id": "operational-overload-and-human-factors",
      "prompt": "Our SRE team is spending 80% of its time on pages and manual changes, and people are afraid to challenge risky mitigations during incidents. What should the recovery plan contain?",
      "expected_output": "An overload recovery plan that declares the condition with evidence, protects engineering time, inventories and reduces recurring work, renegotiates scope, improves alert and configuration safety, establishes escalation and psychological safety, and verifies recovery through operational-load and reliability trends rather than asking people to work harder.",
      "assertions": [
        "Operational overload is declared using a measurable threshold and evidence",
        "The plan protects engineering time and explicitly renegotiates scope or commitments",
        "Pages and manual changes are inventoried with owners and verification criteria for reduction",
        "Incident roles and psychological safety make it safe to challenge unsafe actions and escalate",
        "Exit criteria include sustainable workload and verified reduction of recurring load"
      ]
    }
  ]
}