From c032580197f2f370d4f96895fafc09b36c21f4dd Mon Sep 17 00:00:00 2001 From: Magnus Hedemark Date: Sun, 2 Aug 2026 18:17:42 -0400 Subject: [PATCH] feat(resilience-and-recovery): add resilience-and-recovery skill (#223) Add the resilience-and-recovery skill for designing, exercising, and evidencing graceful degradation, disaster recovery, and restoration behavior across systems and dependencies. - SKILL.md with both resilience patterns (graceful degradation and restore-based recovery), exercise-evidence requirement, HA vs recoverability distinction, context-dependent RTO/RPO, and structured plan-template fields - README.md with all five required human-facing sections - 7 reference files covering failure modes, recovery plan template, exercise design, RTO/RPO decision record, data integrity verification, recovery communication, and follow-up work ledger - 1 bounded discovery brief surveying SRE, platform, data, security, release, and incident-learning - 5 output-quality eval cases covering dependency outage, restore test, regional failure, degraded-but-available path, and recovery exercise with unowned gap escalation - Routing to site-reliability-engineering, platform-engineering, data-engineering, secure-software-engineering, release-engineering, and incident-learning; feeds production-excellence bundle - Root README catalog entry, skill-triggers index, and regenerated marketplace/Codex/llms.txt catalogs Co-authored-by: username Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> --- .claude-plugin/marketplace.json | 9 ++ .codex-plugin/plugin.json | 1 + README.md | 4 + llms.txt | 1 + references/skill-triggers.md | 1 + resilience-and-recovery/README.md | 40 +++++ resilience-and-recovery/SKILL.md | 139 ++++++++++++++++++ resilience-and-recovery/evals/evals.json | 75 ++++++++++ .../references/data-integrity-verification.md | 58 ++++++++ .../references/discovery-brief.md | 77 ++++++++++ .../exercise-design-and-evidence.md | 70 +++++++++ .../failure-modes-and-dependencies.md | 74 ++++++++++ .../references/follow-up-work-ledger.md | 78 ++++++++++ .../references/recovery-communication.md | 66 +++++++++ .../references/recovery-plan-template.md | 69 +++++++++ .../references/rto-rpo-decision-record.md | 69 +++++++++ 16 files changed, 831 insertions(+) create mode 100644 resilience-and-recovery/README.md create mode 100644 resilience-and-recovery/SKILL.md create mode 100644 resilience-and-recovery/evals/evals.json create mode 100644 resilience-and-recovery/references/data-integrity-verification.md create mode 100644 resilience-and-recovery/references/discovery-brief.md create mode 100644 resilience-and-recovery/references/exercise-design-and-evidence.md create mode 100644 resilience-and-recovery/references/failure-modes-and-dependencies.md create mode 100644 resilience-and-recovery/references/follow-up-work-ledger.md create mode 100644 resilience-and-recovery/references/recovery-communication.md create mode 100644 resilience-and-recovery/references/recovery-plan-template.md create mode 100644 resilience-and-recovery/references/rto-rpo-decision-record.md diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index ca7e2df..92c931e 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -806,6 +806,15 @@ "strict": false, "description": "Plan, conduct, evaluate, and synthesize rigorous research. Use for journalistic, industry, or technical investigations that need credible evidence and a traceable method." }, + { + "name": "resilience-and-recovery", + "source": "./", + "skills": [ + "./resilience-and-recovery" + ], + "strict": false, + "description": "Design, exercise, and evidence graceful degradation, disaster recovery, and restoration behavior across systems and dependencies. Covers failure-mode analysis, RTO/RPO decision records, restore testing, game days, failover drills, data integrity verification, and recovery communication. Do not use for live incident command or incident response; route to site-reliability-engineering for those. Do not use for infrastructure implementation details; route to platform-engineering." + }, { "name": "restic", "source": "./", diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json index 7284932..ae0a20a 100644 --- a/.codex-plugin/plugin.json +++ b/.codex-plugin/plugin.json @@ -109,6 +109,7 @@ "./release-engineering", "./remote-systems-administration", "./research-methodology", + "./resilience-and-recovery", "./restic", "./secure-software-engineering", "./security-audit-methodology", diff --git a/README.md b/README.md index 67e7eed..a19c329 100644 --- a/README.md +++ b/README.md @@ -376,6 +376,10 @@ Run a repeatable research-to-notes sequence: gather sources, record URLs and dat Turn an open question into a bounded, evidence-led investigation rather than a plausible-sounding synthesis. +### [resilience-and-recovery](resilience-and-recovery/SKILL.md) + +Design, exercise, and evidence resilience and recovery behavior across systems and dependencies — graceful degradation, restore-based recovery, RTO/RPO decision records, game days, failover drills, data integrity verification, and recovery communication. Ships 7 references, 1 template, 1 discovery brief, and 5 evals. + ### [restic](restic/SKILL.md) Install, configure, operate, secure, automate, tune, troubleshoot, and recover restic backups across local, SFTP, S3-compatible, cloud, and REST backends. Ships deep references for repository selection, backup design, retention/prune, restore drills, threat modeling, performance, observability, migration, and recovery, plus safe preflight and verification scripts and scheduling templates. diff --git a/llms.txt b/llms.txt index b0a238a..b27ef67 100644 --- a/llms.txt +++ b/llms.txt @@ -91,6 +91,7 @@ - [remote-systems-administration](remote-systems-administration/SKILL.md): Administer and troubleshoot remote Linux, FreeBSD, NetBSD, OpenBSD, and macOS systems safely, one host or a fleet at a time. Use when a task requires SSH, Ansible, Paramiko, POSIX diagnostics, service management, software updates, system configuration, firewall changes, or evidence-led remote operations. - [research-and-vault](bundles/research-and-vault/SKILL.md): Chain web research, atomic extraction, and durable knowledge capture into a repeatable workflow when the same research-to-notes sequence is needed. - [research-methodology](research-methodology/SKILL.md): Plan, conduct, evaluate, and synthesize rigorous research. Use for journalistic, industry, or technical investigations that need credible evidence and a traceable method. +- [resilience-and-recovery](resilience-and-recovery/SKILL.md): Design, exercise, and evidence graceful degradation, disaster recovery, and restoration behavior across systems and dependencies. Covers failure-mode analysis, RTO/RPO decision records, restore testing, game days, failover drills, data integrity verification, and recovery communication. Do not use for live incident command or incident response; route to site-reliability-engineering for those. Do not use for infrastructure implementation details; route to platform-engineering. - [restic](restic/SKILL.md): Install, configure, operate, secure, automate, tune, troubleshoot, and recover restic backups across local, SFTP, S3-compatible, cloud, and REST backends. Use when creating or managing a restic repository, designing backup or retention policy, validating restores, handling repository health or locks, moving repositories, or building safe scheduled backup jobs. Do not use for a generic file-copy task that does not need encrypted, deduplicated snapshots. - [secure-software-engineering](secure-software-engineering/SKILL.md): Use when designing or implementing software securely: define security requirements, threat-model a feature, choose secure defaults, design authentication and authorization, handle untrusted data and secrets, evaluate dependencies, or review security-sensitive changes. Use for prevention during requirements, design, implementation, and review; not for post-build security assessments or scanning an existing codebase. - [security-audit-methodology](security-audit-methodology/SKILL.md): Plan authorized security reviews with threat modeling, architecture and dependency audits, and vulnerability classification. Use for scoped defensive security assessment. diff --git a/references/skill-triggers.md b/references/skill-triggers.md index f2df537..cd101f4 100644 --- a/references/skill-triggers.md +++ b/references/skill-triggers.md @@ -73,6 +73,7 @@ Each skill's `description` field is the canonical routing contract. This conveni | "research-methodology", "research methodology" | [research-methodology](../research-methodology/SKILL.md) | | "release plan", "release engineering", "release train", "rollback plan", "version bump", "changelog", "DORA metrics", "progressive delivery", "canary release", "feature flag cleanup", "release readiness", "SemVer", "emergency release", "hotfix" | [release-engineering](../release-engineering/SKILL.md) | | "remote system administration", "remote administration", "remote SSH", "SSH administration", "Ansible administration", "Paramiko automation", "remote Linux", "remote FreeBSD", "remote NetBSD", "remote OpenBSD", "remote macOS", "remote service", "remote firewall", "remote package update", "fleet administration", "fleet rollout", "remote launchctl", "remote rcctl", "remote systemctl" | [remote-systems-administration](../remote-systems-administration/SKILL.md) | +| "resilience", "recovery", "graceful degradation", "disaster recovery", "DR plan", "game day", "restore test", "failover drill", "RTO", "RPO", "recovery plan", "resilience plan", "data integrity after restore", "recovery exercise", "follow-up work ledger", "recovery communication", "dependency loss", "circuit breaker resilience", "degradation path", "feature shedding" | [resilience-and-recovery](../resilience-and-recovery/SKILL.md) | | "restic", "restic backup", "restic repository", "restic restore", "restic prune", "restic forget", "restic check", "restic snapshots", "encrypted backup", "backup retention", "restore drill", "backup integrity", "repository lock", "S3 backup", "SFTP backup" | [restic](../restic/SKILL.md) | | "technical-documentation", "technical documentation" | [technical-documentation](../technical-documentation/SKILL.md) | | "secure software engineering", "build securely", "secure by design", "security requirements", "security acceptance criteria", "threat model this design", "secure code review", "secure defaults", "authorization design", "multi-tenant isolation", "release evidence", "SBOM", "software supply chain", "AI security", "LLM security", "prompt injection", "tool authorization", "RAG security" | [secure-software-engineering](../secure-software-engineering/SKILL.md) | diff --git a/resilience-and-recovery/README.md b/resilience-and-recovery/README.md new file mode 100644 index 0000000..003f433 --- /dev/null +++ b/resilience-and-recovery/README.md @@ -0,0 +1,40 @@ +# Resilience and Recovery + +Design, exercise, and evidence resilience and recovery behavior across systems and dependencies — producing exercise-backed resilience plans, not only design documents. + +## Why Install This Skill + +Most teams have recovery plans. Few have tested them. This skill provides a structured method for designing resilience into systems — graceful degradation, restore-based recovery, dependency analysis, RTO/RPO decision records, game days, failover drills, data integrity verification, and recovery communication. It treats exercise evidence as the standard of proof: a recovery plan that has never been exercised is not a recovery capability. + +Install this skill when your agent needs to help teams move from "we have a DR plan somewhere" to "we exercised our restore last quarter and here is the evidence." It composes specialist capabilities from SRE, platform, data, security, and release engineering without duplicating their methodology, and it feeds resilience evidence into the production-excellence bundle for go/no-go readiness decisions. + +## What You Get + +| Path | What it provides | +|---|---| +| `SKILL.md` | Trigger conditions, resilience patterns (graceful degradation and restore-based recovery), resilience plan template fields, core principles, routing table, and progressive-disclosure loading guide | +| `references/discovery-brief.md` | Survey of adjacent skills (SRE, platform, data, security, release, incident-learning) with ownership boundaries and routing decisions | +| `references/failure-modes-and-dependencies.md` | Method for mapping failure modes, analyzing dependency behavior under failure, and designing degradation paths with tier-based feature shedding | +| `references/recovery-plan-template.md` | Structured resilience/recovery plan template with all required fields: system boundary, failure modes, dependency map, degradation choices, RTO/RPO, data integrity, recovery procedure, communication plan, exercise schedule, follow-up ledger | +| `references/exercise-design-and-evidence.md` | Game-day, restore-test, and failover-drill design; scenario definition; evidence recording; exercise-finding classification | +| `references/rto-rpo-decision-record.md` | Context-specific RTO/RPO decision-record template with tradeoff analysis and per-system examples | +| `references/data-integrity-verification.md` | Post-restore data validation: checksums, row counts, application-level consistency checks, reconciliation protocol | +| `references/recovery-communication.md` | Stakeholder notification planning: templates, channels, escalation paths, timing | +| `references/follow-up-work-ledger.md` | Converting exercise findings into owned implementation, test, and operational follow-up work with verification gates | +| `evals/evals.json` | Five output-quality eval cases covering dependency outage, restore test, regional failure, degraded-but-available path, and recovery exercise exposing an unowned gap | + +## Quick Start + +Start with the failure-modes reference to map your system's failure surface, then use the recovery-plan template to produce a structured plan. The RTO/RPO decision record and exercise-design reference turn the plan into testable evidence. + +Ask your agent to "design a resilience plan for " or "prepare a game day for " and the skill's triggers will route the work. + +## Triggers + +- Design, exercise, and evidence resilience and recovery behavior across systems: failure-mode analysis, graceful degradation, restore-based recovery, RTO/RPO decision records, game days, failover drills, restore testing, data integrity verification, and recovery communication. +- Requests to assess disaster recovery readiness, plan a failover drill, verify data integrity after a restore, or convert exercise findings into follow-up work. +- Work where a system's resilience must be proven through exercise evidence rather than design documentation alone. + +## Requirements + +No runtime dependencies. The methodology is host-neutral and requires no specific tools, platforms, or API keys. Exercise execution may require access to the target system and its recovery tooling; the skill provides the method, not the execution environment. diff --git a/resilience-and-recovery/SKILL.md b/resilience-and-recovery/SKILL.md new file mode 100644 index 0000000..3925be5 --- /dev/null +++ b/resilience-and-recovery/SKILL.md @@ -0,0 +1,139 @@ +--- +name: resilience-and-recovery +description: >- + Design, exercise, and evidence graceful degradation, disaster recovery, and + restoration behavior across systems and dependencies. Covers failure-mode + analysis, RTO/RPO decision records, restore testing, game days, failover + drills, data integrity verification, and recovery communication. Do not use + for live incident command or incident response; route to + site-reliability-engineering for those. Do not use for infrastructure + implementation details; route to platform-engineering. +license: MIT +compatibility: No runtime dependency. Host-neutral methodology. +--- + +# Resilience and Recovery + +Design, exercise, and evidence resilience and recovery behavior across systems and their dependencies. This skill joins failure modes, dependency behavior, degradation choices, restore testing, disaster recovery, game days, failover, data integrity, and recovery communication into a single method — producing an exercise-backed resilience plan, not only a design document. + +## When to use + +| Trigger | What it covers | +|---|---| +| "Design a resilience plan for this system" | Failure-mode mapping, dependency analysis, degradation choices, RTO/RPO decision record, recovery plan template | +| "Run a game day or restore test" | Exercise design, scenario definition, evidence recording, follow-up work ledger | +|"Assess our disaster recovery readiness" | DR plan review against exercise evidence, gap analysis, data integrity verification | +| "What happens if this dependency fails?" | Dependency-loss scenarios, degradation paths, circuit-breaker and fallback strategy | +| "Define our RTO and RPO" | Context-specific decision record with tradeoff analysis, not universal prescription | +| "Verify data integrity after a restore" | Post-restore validation procedures, checksum and consistency checks, reconciliation protocol | +| "Plan a failover drill" | Failover exercise design, pre-conditions, success criteria, rollback/failback plan, evidence recording | + +## When not to use + +- **Live incident command or incident response**: route to [site-reliability-engineering](../site-reliability-engineering/SKILL.md) for the incident command system, on-call operations, and real-time incident management. This skill owns the pre-incident resilience design and exercise-evidence method; SRE owns the live response. +- **Infrastructure implementation details**: route to [platform-engineering](../platform-engineering/SKILL.md) for infrastructure-as-code, CI/CD pipeline implementation, container orchestration, and service networking. This skill owns the resilience requirements and exercise evidence; platform engineering owns the implementation. +- **Release rollout and rollback mechanics**: route to [release-engineering](../release-engineering/SKILL.md) for progressive delivery, canary deployments, feature flags, and rollback runbooks. This skill owns resilience verification of those mechanics through exercises. +- **Backup implementation and pipeline operations**: route to [data-engineering](../data-engineering/SKILL.md) for backup strategy implementation, WAL archiving, and snapshot management. This skill owns the restore-testing evidence and data-integrity verification protocol. +- **Security incident containment and forensics**: route to [secure-software-engineering](../secure-software-engineering/SKILL.md) for security incident response and threat containment. This skill owns the resilience dimension of security incidents — ensuring recovery capability survives a security event. +- **Post-incident learning and verification**: route to incident-learning for converting incident findings into verified follow-up work with closure evidence. This skill feeds exercise and DR test findings into the incident-learning pipeline. + +## Loading guide + +| File | Load when | +|---|---| +| [references/failure-modes-and-dependencies.md](references/failure-modes-and-dependencies.md) | Mapping failure modes, analyzing dependency loss, or designing degradation paths | +| [references/recovery-plan-template.md](references/recovery-plan-template.md) | Building or reviewing a resilience/recovery plan with structured fields | +| [references/exercise-design-and-evidence.md](references/exercise-design-and-evidence.md) | Designing a game day, restore test, failover drill, or recording exercise evidence | +| [references/rto-rpo-decision-record.md](references/rto-rpo-decision-record.md) | Defining RTO/RPO with context-specific tradeoffs and a decision-record template | +| [references/data-integrity-verification.md](references/data-integrity-verification.md) | Verifying data correctness and consistency after restore or failover | +| [references/recovery-communication.md](references/recovery-communication.md) | Planning who to notify and when during recovery events | +| [references/follow-up-work-ledger.md](references/follow-up-work-ledger.md) | Converting exercise findings into owned implementation, test, and operational work | +| [references/discovery-brief.md](references/discovery-brief.md) | Understanding ownership boundaries with adjacent skills | + +## Core principles + +**Resilience is proven by exercise, not design.** A recovery plan that has never been tested is a hope, not a capability. Every resilience claim must be backed by exercise evidence: a game day, a restore test, a failover drill, or a chaos experiment. Design documentation alone is not sufficient. Recovery plans require evidence from exercises — not just design claims. Exercise evidence, not only design documentation, is the standard of proof. + +**High availability is not recoverability.** HA (redundancy, failover, clustering) keeps a system running through component failure. Recovery (backup, restore, disaster recovery) rebuilds a system after it has failed. HA is not a substitute for recovery: redundant systems can still experience data corruption, logical errors, or cascading failures that propagate across replicas. A system with 99.99% availability but no tested restore capability is not resilient — it is available but unrecoverable. Both HA and recovery are required; neither replaces the other. + +**RTO and RPO are context-dependent, not universal.** Recovery time and recovery point objectives depend on the system's role, data classification, user impact, regulatory requirements, and cost. There is no universal "RTO should be < 1 hour." A payment system, an internal wiki, and a batch analytics pipeline have fundamentally different RTO/RPO profiles. Define RTO/RPO per system through a decision record with explicit tradeoffs, not by copying a template value. See the RTO/RPO decision record reference for the structured template. + +**Dependencies define your blast radius.** Every upstream and downstream dependency is a failure mode. A resilience plan that does not account for dependency behavior under failure is incomplete. Map what happens when each dependency is unavailable, degraded, or slow — and what your system promises to its own consumers in each case. + +**Degrade gracefully, not completely.** When a dependency or internal component fails, the system should continue operating in a reduced-but-acceptable mode rather than failing completely. Define acceptable degradation paths: which features are shed, which remain, and what user experience results. Not every failure justifies a full outage. + +**Exercise failures are gifts.** When an exercise exposes a gap — a restore that took too long, a failover that lost data, a dependency loss that cascaded unexpectedly — that finding is an asset. Convert every exercise finding into owned follow-up work with a named owner, a target date, and a verification method. An exercise that reveals no gaps was not thorough enough or the system is untested. + +**Recovery communication is part of recovery.** During a recovery event, stakeholders need to know: what happened, what is affected, what is being done, when to expect resolution, and who to contact. Pre-plan communication templates, notification channels, and escalation paths. Communication failures during recovery compound technical failures. + +## Resilience patterns + +### Pattern A: Graceful degradation + +The system continues operating in a reduced-but-acceptable mode when a component or dependency fails. This is not a full outage — it is a deliberate choice to shed non-critical capability while preserving core function. + +**Decision criteria for degradation:** +- Is the failing component essential to the system's core function? If yes, degradation may not be acceptable — the system may need to fail closed or fail safe. +- Can the remaining capability serve users acceptably for the expected recovery window? If users cannot accomplish their primary task, degradation is not working. +- What is the blast radius of continuing in degraded mode? Does degraded operation risk data corruption, security exposure, or cascading failure? + +**Examples:** +- An e-commerce checkout loses the recommendation engine but still accepts orders. Recommendations are shed; ordering is preserved. +- A dashboard loses a real-time metrics feed but still displays cached data from the last refresh with a "data may be stale" indicator. Real-time is shed; observability is preserved. +- An API gateway loses a downstream microservice and returns a cached response or a graceful fallback payload instead of a 500 error. Freshness is shed; availability is preserved. + +**Feature shedding by tier:** +- **Tier 1 (preserve):** Core function — the system's reason to exist. Must remain available. +- **Tier 2 (shed if necessary):** Enhancing features — improve experience but are not essential. Shed first. +- **Tier 3 (shed early):** Nice-to-have — non-critical embellishments. Shed immediately under stress. + +### Pattern B: Restore-based recovery + +The system is recovered from a backup, snapshot, or replica after a failure that cannot be mitigated through redundancy or degradation. This covers disaster recovery, data corruption recovery, and full-system rebuild. + +**Decision criteria for restore:** +- Is the failure mode one that HA cannot handle (data corruption, logical error, region loss, ransomware)? +- What is the verified RTO — how long will the restore take, and has this been tested? +- What is the verified RPO — how much data will be lost, and has the backup been validated? +- Has the restore procedure been exercised end-to-end, including data integrity verification? + +**Examples:** +- A primary database region is lost. The system fails over to a DR region and restores from the most recent validated cross-region backup. RTO and RPO are measured against the decision record. +- A logical data corruption (bad deployment, bug) propagates to all replicas. HA does not help — every replica is corrupted. The system restores from a point-in-time backup before the corruption event, and data integrity is verified post-restore. +- A ransomware event encrypts primary and replica data stores. The system restores from an air-gapped, immutable backup. Restore testing confirmed this capability within the RTO defined in the decision record. + +**Restore testing requirements:** +- Full end-to-end restore must be exercised, not only backup verification. +- Data integrity must be verified after restore: checksums, row counts, application-level consistency checks. +- Restore must be measured against the RTO decision record. A restore that meets RTO on paper but not in practice is a gap. +- Restore procedures must be documented, versioned, and owned. The owner is accountable for exercise results. + +## Resilience plan template fields + +Every resilience plan must include these structured fields: + +| Field | Description | Required evidence | +|---|---|---| +| **System boundary** | What is in scope and out of scope for this plan | Architecture diagram or boundary document | +| **Failure modes** | How the system can fail: component, dependency, region, data, operator error | Failure-mode analysis with likelihood and impact | +| **Dependency map** | Upstream systems this system depends on; downstream systems that depend on this system | Named systems, failure behavior per dependency, consumer contracts | +| **Degradation choices** | What is shed and what is preserved under each failure scenario | Tier assignments with rationale, user-impact assessment | +| **RTO/RPO decision record** | Context-specific recovery objectives per system and scenario | Decision-record template with tradeoffs, not hardcoded numbers | +| **Data integrity** | How data correctness and consistency are verified after recovery | Post-restore validation procedure, checksums, reconciliation protocol | +| **Recovery procedure** | Step-by-step recovery process with pre-conditions and success criteria | Versioned procedure, owner assignment, last-exercise date | +| **Communication plan** | Who to notify, when, and through what channels during each recovery scenario | Notification templates, stakeholder list, escalation contacts | +| **Exercise schedule** | When each recovery scenario was last exercised and when it will be re-exercised | Exercise evidence: date, scenario, result, findings | +| **Follow-up work ledger** | Exercise findings converted to owned implementation/test/operational work | Owner, target date, verification method, status per finding | + +## Routing and related skills + +This skill composes capabilities from and routes to: + +- **[site-reliability-engineering](../site-reliability-engineering/SKILL.md)** — Incident command, on-call operations, SLO/SLI framework, error budgets. Resilience-and-recovery owns the pre-incident design and exercise evidence; SRE owns the live incident response. +- **[platform-engineering](../platform-engineering/SKILL.md)** — Infrastructure implementation, CI/CD, container orchestration, service networking. Resilience-and-recovery owns the resilience requirements; platform engineering owns the implementation that satisfies them. +- **[data-engineering](../data-engineering/SKILL.md)** — Backup strategy implementation, WAL archiving, snapshot management. Resilience-and-recovery owns the restore-testing evidence and data-integrity verification protocol. +- **[secure-software-engineering](../secure-software-engineering/SKILL.md)** — Security requirements, threat modeling, secure design. Resilience-and-recovery owns the resilience dimension of security incidents — ensuring recovery capability survives a security event. +- **[release-engineering](../release-engineering/SKILL.md)** — Progressive delivery, canary deployments, rollback runbooks. Resilience-and-recovery verifies those mechanics through exercises and feeds evidence into release readiness. +- **incident-learning** (wave 5) — Converts incident and exercise findings into verified follow-up work with closure evidence. Resilience-and-recovery feeds exercise and DR test findings into the incident-learning pipeline for cross-incident pattern analysis. + +This skill feeds the **production-excellence** bundle (wave 6) as a component capability: resilience evidence — exercise results, RTO/RPO decision records, follow-up work ledgers — flows into the production-excellence evidence packet for go/no-go/defer/exception readiness decisions. diff --git a/resilience-and-recovery/evals/evals.json b/resilience-and-recovery/evals/evals.json new file mode 100644 index 0000000..0c52689 --- /dev/null +++ b/resilience-and-recovery/evals/evals.json @@ -0,0 +1,75 @@ +{ + "schema_version": 1, + "skill_name": "resilience-and-recovery", + "evals": [ + { + "id": "dependency-outage-degradation-choice", + "prompt": "Our e-commerce platform depends on a third-party recommendation engine. The recommendation service has been experiencing intermittent outages (5-10 minute brownouts, 2-3 times per day). When it's down, product pages return 500 errors because our code does a synchronous call to the recommendation API without a timeout. We need a resilience plan for this dependency. The core e-commerce function (browse catalog, add to cart, checkout) must remain available. Recommendations are enhancing, not core.", + "expected_output": "A resilience plan addressing the recommendation-engine dependency. The plan identifies the dependency as a Tier 2 (enhancing) function that should be shed when unavailable. It defines: a timeout value for the recommendation API call, a circuit breaker configuration with threshold and reset behavior, a fallback that returns the product page without recommendations (graceful degradation — the core function of browsing and purchasing is preserved), and a consumer contract stating that product pages will serve without recommendations when the dependency is degraded. The plan includes a degradation choice record: recommendations are shed, catalog browsing and checkout are preserved. It specifies that the degradation path must be exercised in a game day before the plan is considered verified.", + "assertions": [ + "The plan identifies the recommendation engine as a Tier 2 enhancing dependency that should be shed on failure, not a core function", + "The plan defines a specific timeout value for the dependency call, not 'add a timeout'", + "The plan describes a circuit breaker with threshold, reset behavior, or equivalent failure-isolation mechanism", + "The plan defines a fallback that returns product pages without recommendations — graceful degradation, not a 500 error", + "The plan includes a consumer contract stating what the system promises its users when the dependency is down", + "The plan requires exercise evidence (a game day) before the degradation path is considered verified, not only design approval", + "Core e-commerce functions (catalog browsing, cart, checkout) are explicitly preserved in the degradation path" + ] + }, + { + "id": "restore-test-with-data-integrity", + "prompt": "Our PostgreSQL primary database stores customer orders and payment records. We take nightly backups, but we have never tested a full restore. The last backup verification was a pg_verifybackup check 6 months ago, which passed. We need to design and evidence a restore test that proves we can recover within our documented RTO of 4 hours and RPO of 1 hour, and that the restored data is correct and consistent. The order table has ~50M rows. Payment records must be transactionally consistent with orders.", + "expected_output": "A restore-test design that goes beyond backup verification. The test design includes: (1) a pre-restore baseline recording expected row counts and checksums for the order and payment tables; (2) a full end-to-end restore procedure with step-by-step instructions; (3) success criteria measuring actual restore time against the 4-hour RTO and actual data freshness against the 1-hour RPO; (4) a data integrity verification procedure including structural checks (row counts, schema match), checksum comparison, application-level consistency (every payment record has a corresponding order, no orphaned payments), and cross-table reconciliation; (5) an exercise evidence record capturing measured RTO, measured RPO, data integrity verdict, and any gaps; (6) a finding that backup verification alone (pg_verifybackup) is not sufficient evidence — only a full end-to-end restore test with data integrity verification constitutes valid recovery evidence. The response explicitly states that the existing pg_verifybackup check does not satisfy the exercise-evidence standard.", + "assertions": [ + "The test design includes an end-to-end restore procedure with step-by-step instructions, not only a backup verification check", + "The test design defines pre-restore baseline recording with expected row counts and/or checksums", + "Success criteria measure actual restore time against the 4-hour RTO and actual data freshness against the 1-hour RPO", + "The data integrity verification includes application-level consistency between orders and payments, not only structural checks", + "The response explicitly states that backup verification alone (pg_verifybackup) is not sufficient recovery evidence", + "The test design produces an exercise evidence record with measured RTO, measured RPO, and data integrity verdict", + "The response distinguishes between backup verification and restore testing — they are not the same thing" + ] + }, + { + "id": "regional-failure-dr-failover", + "prompt": "Our SaaS platform runs in a single AWS region (us-east-1). The board has asked for a disaster recovery plan after a competitor experienced a 14-hour region-wide outage. We need to decide our DR strategy, define RTO and RPO for a regional failure scenario, and produce a failover exercise plan. The platform serves 10,000 concurrent users with a 99.9% uptime SLA. Data volumes: 2TB relational database, 500GB object storage, 200GB search index. The platform has never been deployed outside us-east-1. Engineering estimates a multi-region deployment would cost ~40% more in infrastructure.", + "expected_output": "A DR plan with an RTO/RPO decision record that does NOT prescribe universal values. The decision record includes: a context-specific RTO (e.g., 2 hours, 4 hours, or another value with explicit rationale based on user impact, SLA, and cost tradeoffs); a context-specific RPO with rationale based on data classification, user tolerance for data loss, and cost of tighter RPO; explicit tradeoffs between recovery speed and infrastructure cost (the 40% cost increase is weighed against the RTO/RPO targets); a failover exercise plan with pre-conditions, success criteria, and evidence recording. The plan does NOT say 'RTO should be < 1 hour' or prescribe a universal value. It includes a decision-record template field for RTO and RPO (not hardcoded numbers) and explains how RTO/RPO differ between this SaaS platform and examples like a payment system or an internal wiki. The plan identifies dependencies on DNS failover, data replication, and application configuration that must be addressed before multi-region DR is viable.", + "assertions": [ + "The RTO/RPO decision record includes context-specific values with explicit rationale, not universal prescriptions", + "The plan does NOT contain 'RTO should be < 1 hour' or any universal RTO/RPO value without context", + "The plan explicitly weighs the 40% infrastructure cost increase against RTO/RPO targets as a tradeoff", + "The plan includes a decision-record template field for RTO and RPO, not hardcoded numbers", + "The plan explains how RTO/RPO differ between this system and at least one comparison system (payment system, internal wiki, etc.)", + "The failover exercise plan includes pre-conditions, success criteria, and evidence recording", + "The plan identifies dependencies (DNS, data replication, application config) that must be addressed for DR viability" + ] + }, + { + "id": "degraded-but-available-path", + "prompt": "Our customer-facing dashboard aggregates data from 5 microservices: user profiles, billing, usage metrics, support tickets, and feature flags. The usage-metrics service has been unreliable — it experiences latency spikes and occasional 503 errors under load. When it fails, the entire dashboard returns a 500 error page because the frontend calls all 5 services synchronously before rendering. We want the dashboard to remain available even when the metrics service is degraded. The core function is: users must be able to view their profile, billing, and support tickets. Metrics and feature flags are enhancing, not core.", + "expected_output": "A graceful degradation plan for the dashboard. The plan tiers the 5 microservices: Tier 1 (preserve) — user profiles, billing, support tickets (core function); Tier 2 (shed if necessary) — usage metrics, feature flags (enhancing). It defines: timeouts for each service call; a circuit breaker for the metrics service that opens after N failures and serves a cached or empty response; a fallback UI that renders the dashboard with a 'metrics temporarily unavailable' indicator instead of a 500 error; a consumer contract stating the dashboard remains available with partial data during dependency degradation. The plan specifies that the degraded state is acceptable for the duration of the metrics-service recovery window (up to the documented RTO for that dependency). It requires a game day to verify the degradation path — design documentation alone is not sufficient.", + "assertions": [ + "The plan tiers the 5 microservices into preserve (profiles, billing, tickets) and shed (metrics, flags) categories", + "The plan defines graceful degradation — dashboard remains available with partial data, not a 500 error page", + "The plan includes a fallback UI with a 'metrics temporarily unavailable' indicator or equivalent user-facing degraded-state message", + "The plan defines a maximum degraded-operation window after which escalation is required", + "The plan requires a game day to exercise the degradation path, not only design review or code change", + "The response explicitly distinguishes graceful degradation from full outage — the dashboard is degraded but available, not down" + ] + }, + { + "id": "recovery-exercise-unowned-gap", + "prompt": "We ran a game day simulating a complete loss of our primary Kafka cluster. The exercise revealed that while our application failed over to the secondary Kafka cluster within the RTO, the data replication lag between primary and secondary was 12 minutes — meaning we lost 12 minutes of events. Our RPO for the event stream is documented as 'near zero' but was never defined with a specific value. The secondary cluster is owned by the platform-infrastructure team; the application team that ran the game day does not own the replication configuration. The infra team was not part of the game day. The application team wants to close the exercise as 'pass with gaps' and move on. How should this exercise finding be handled?", + "expected_output": "A response that REFUSES to close the exercise without addressing the gap. The response: (1) classifies the finding as a blocker or gap — the documented RPO of 'near zero' is not met (12 minutes of data loss) and the RPO was never defined with a specific measurable value; (2) identifies that the gap is an unowned cross-team dependency — the application team cannot resolve it alone because the platform-infra team owns the replication configuration; (3) creates a follow-up work ledger entry for the finding with the gap documented, but the entry REQUIRES an owner — the gap must be assigned to a named individual on the platform-infra team before the exercise can close; (4) escalates the unowned gap: if no owner can be assigned, the exercise finding is escalated to the system owner or engineering manager rather than closed without ownership; (5) requires a re-exercise after the replication configuration is updated and the RPO is defined with a specific measurable value. The response explicitly states that closing the exercise with an unowned gap is not acceptable — the gap must have an owner and a verification path before the exercise is complete.", + "assertions": [ + "The response REFUSES to close the exercise with an unowned gap — it does not accept 'pass with gaps' without ownership assignment", + "The finding is classified as a blocker or gap, not an observation or improvement", + "The response creates a follow-up work ledger entry that requires a named owner, not 'the team' or 'TBD'", + "The unowned gap is escalated to a system owner or engineering manager if no owner can be assigned on the platform-infra team", + "The response requires a re-exercise after the replication gap is resolved, not only a documentation update", + "The response identifies that 'near zero' RPO is not a measurable value and requires a specific RPO definition", + "The response states that closing an exercise with an unowned cross-team gap is not acceptable — ownership must be assigned before closure" + ] + } + ] +} diff --git a/resilience-and-recovery/references/data-integrity-verification.md b/resilience-and-recovery/references/data-integrity-verification.md new file mode 100644 index 0000000..bf2ab4f --- /dev/null +++ b/resilience-and-recovery/references/data-integrity-verification.md @@ -0,0 +1,58 @@ +# Data Integrity Verification + +## Purpose + +Verify that data is correct and consistent after a restore, failover, or recovery event. A successful restore that produces corrupted or inconsistent data is a failed recovery. Data integrity verification is a required step in every restore-based recovery exercise. + +## Verification levels + +| Level | What it checks | When to use | +|---|---|---| +| **Structural** | Files exist, schemas match, row counts are expected | Every restore — fast, automated, catches gross failures | +| **Checksum** | Cryptographic hash of data blocks or files matches pre-backup hash | Every restore for systems where data corruption is a risk | +| **Application-level** | Business rules hold — foreign keys resolve, aggregates compute, workflows execute | Restores of critical systems; DR failovers | +| **Reconciliation** | Restored data matches an independent source of truth (e.g., event log replay, replica comparison) | High-assurance restores; financial/healthcare data | + +## Post-restore validation procedure + +### 1. Pre-restore baseline +- Record expected row counts per table or collection. +- Record expected checksums for critical data sets. +- Record schema version and migration state. + +### 2. Structural validation +- [ ] Restored files or database exist and are accessible. +- [ ] Schema version matches expected version. +- [ ] Row counts match expected counts within tolerance. +- [ ] Indexes are present and valid. + +### 3. Checksum validation +- [ ] Compute checksums on restored data blocks. +- [ ] Compare against pre-backup checksums. +- [ ] Flag any mismatch for investigation. + +### 4. Application-level validation +- [ ] Run application-level consistency queries: foreign keys resolve, no orphaned records. +- [ ] Compute key aggregates and compare against pre-restore baseline. +- [ ] Execute critical business workflows in a test context against restored data. +- [ ] Verify that application can connect to and query the restored data store. + +### 5. Reconciliation (high-assurance systems) +- [ ] Replay event log against restored state and compare. +- [ ] Compare restored data against an independent replica or audit log. +- [ ] Verify that all committed transactions are present and no uncommitted transactions appear. + +### 6. Sign-off +- [ ] Validation owner reviews all results. +- [ ] Any discrepancy is recorded in the follow-up work ledger. +- [ ] Validation owner signs off or escalates. + +## Common failure modes + +| Failure mode | Detection | Mitigation | +|---|---|---| +| Backup is corrupted | Checksum mismatch | Multiple backup copies; periodic restore testing | +| Backup is incomplete (missing recent data) | Row count below expected; reconciliation gap | Verify backup completeness at time of creation | +| Restore process introduces corruption | Post-restore checksum mismatch | Use verified restore tooling; test restore procedure regularly | +| Schema migration mismatch | Application errors on connect; foreign key failures | Record schema version at backup time; validate compatibility | +| Replica lag captured in backup | Data inconsistency between related tables | Use consistent snapshot or transactionally-consistent backup | diff --git a/resilience-and-recovery/references/discovery-brief.md b/resilience-and-recovery/references/discovery-brief.md new file mode 100644 index 0000000..19c4c69 --- /dev/null +++ b/resilience-and-recovery/references/discovery-brief.md @@ -0,0 +1,77 @@ +# Discovery Brief: Resilience and Recovery + +## Survey scope + +This brief surveys adjacent skills in the agent-skills catalog to define the ownership boundaries of `resilience-and-recovery`. The goal is to own resilience design, exercise evidence, and recovery verification — without duplicating incident command, infrastructure implementation, backup operations, security incident response, release mechanics, or post-incident learning. + +## Skills surveyed + +### site-reliability-engineering + +**What SRE owns:** Incident command, on-call operations, SLO/SLI framework, error budget governance, blameless postmortems, monitoring and alerting, toil elimination, and product-focused reliability. + +**Boundary:** SRE owns the live operational response to incidents — the incident command system, real-time alerting, and post-incident postmortem process. It does not own the pre-incident resilience design method, the exercise-evidence standard, or the recovery-plan template. SRE's postmortem process produces follow-up actions; resilience-and-recovery's exercise method produces follow-up work from game days and restore tests. + +**Routing decision:** Resilience-and-recovery routes live incident command and on-call operations to SRE. SRE's postmortem and incident-command references are the authoritative sources for incident response. Resilience-and-recovery owns the resilience plan, exercise design, and recovery verification — the work that happens before and between incidents. + +### platform-engineering + +**What platform engineering owns:** Infrastructure as code, CI/CD pipeline implementation, container orchestration, service networking, secret management, and cloud architecture. It builds and operates the delivery platform. + +**Boundary:** Platform engineering owns the implementation of infrastructure that satisfies resilience requirements. Resilience-and-recovery owns the resilience requirements themselves: what the infrastructure must withstand, how recovery must behave, and how that behavior is verified through exercises. Platform engineering implements the circuit breaker; resilience-and-recovery defines the degradation path and verifies it in a game day. + +**Routing decision:** Resilience-and-recovery routes infrastructure implementation to platform-engineering. Platform engineering's IaC patterns and service-networking references are the authoritative sources for implementation. Resilience-and-recovery owns the requirements, the exercise design, and the evidence standard. + +### data-engineering + +**What data engineering owns:** Backup strategy implementation, WAL archiving, snapshot management, database migration patterns, data quality monitoring, and storage infrastructure. It operates the data stores. + +**Boundary:** Data engineering owns the backup implementation — the mechanics of taking backups, managing WAL archives, and scheduling snapshots. Resilience-and-recovery owns the restore-testing evidence and data-integrity verification protocol: proving that backups can actually be restored within the RTO, and that restored data is correct and consistent. A backup that passes data engineering's integrity check is necessary but not sufficient; resilience-and-recovery requires a full end-to-end restore exercise with application-level validation. + +**Routing decision:** Resilience-and-recovery routes backup implementation and pipeline operations to data-engineering. Data engineering's backup-and-recovery reference is the authoritative source for backup mechanics. Resilience-and-recovery owns the restore exercise, the data-integrity verification after restore, and the RTO/RPO decision record that the backup strategy must satisfy. + +### secure-software-engineering + +**What secure-software-engineering owns:** Security requirements, threat modeling, secure design, authentication and authorization, input validation, secrets lifecycle, dependency supply chain, and secure code review. It builds security into decisions before defects reach production. + +**Boundary:** Secure-software-engineering owns the security posture of the system — preventing and containing security incidents. Resilience-and-recovery owns the resilience dimension of security incidents: ensuring that recovery capability survives a security event. If a system is compromised, can it be restored from a known-good backup? Is the backup air-gapped or immutable? Has the restore been tested? These are resilience questions that security engineering surfaces but does not own. + +**Routing decision:** Resilience-and-recovery routes security incident containment and forensics to secure-software-engineering. Secure-software-engineering's threat-modeling and incident-learning references are the authoritative sources for security posture. Resilience-and-recovery owns the recovery-from-security-event path: the restore procedure, the data-integrity verification after a security restore, and the exercise that proves it. + +### release-engineering + +**What release-engineering owns:** Release process design, CD pipeline architecture, progressive delivery, feature flags, versioning and artifacts, readiness gates, rollback and recovery planning, change governance, and DORA metrics. + +**Boundary:** Release-engineering owns the release mechanics — how code moves from commit to production, how rollbacks are executed, and how progressive delivery manages risk. Resilience-and-recovery owns the resilience verification of those mechanics: does the rollback actually work under failure conditions? Has the failover been exercised? The release engineer designs the rollback runbook; resilience-and-recovery designs the game day that tests it. + +**Routing decision:** Resilience-and-recovery routes release rollout, progressive delivery, and rollback mechanics to release-engineering. Release-engineering's rollback-planning and progressive-delivery references are the authoritative sources for release mechanics. Resilience-and-recovery owns the exercise that verifies those mechanics and the evidence that results. + +### incident-learning (wave 5, not yet landed) + +**What incident-learning will own:** Separation of observed facts from causal hypotheses and contributing conditions; mapping follow-up work across code, tests, skills, operations, product, and governance; closure defined as verification that the intended change occurred. It is the pipeline that converts incident findings into verified, owned follow-up work. + +**Boundary:** Incident-learning will own the post-incident learning pipeline — taking an incident or near-miss and producing verified, owned follow-up work with closure evidence. Resilience-and-recovery feeds exercise and DR test findings into that pipeline. A game day that exposes a restore gap produces findings; incident-learning tracks those findings to verified closure. + +**Routing decision:** Resilience-and-recovery routes exercise and DR findings to incident-learning for cross-incident pattern analysis and verified follow-up closure. Resilience-and-recovery owns the exercise design and the finding capture; incident-learning owns the verified closure pipeline. + +### production-excellence bundle (wave 6, not yet landed) + +**What production-excellence will own:** Composing production-readiness, migration-engineering, resilience-and-recovery, capacity-and-cost-engineering, and incident-learning into a unified production evidence packet with go/no-go/defer/exception outcomes. + +**Boundary:** Production-excellence will consume resilience evidence — exercise results, RTO/RPO decision records, follow-up work ledgers — as input to production readiness decisions. Resilience-and-recovery produces the resilience dimension of that evidence packet. + +**Routing decision:** Resilience-and-recovery feeds the production-excellence bundle. The resilience evidence this skill produces (exercise results, decision records, follow-up ledgers) is a required input to the production-excellence go/no-go/defer/exception decision. + +## What resilience-and-recovery does NOT own + +- **Incident command**: owned by SRE. Resilience-and-recovery does not manage live incidents or run incident response. +- **Infrastructure implementation**: owned by platform engineering. Resilience-and-recovery does not write Terraform, configure Kubernetes, or build CI/CD pipelines. +- **Backup mechanics**: owned by data engineering. Resilience-and-recovery does not implement backup schedules, WAL archiving, or snapshot management. +- **Security incident containment**: owned by secure-software-engineering. Resilience-and-recovery does not contain threats or perform forensics. +- **Release mechanics**: owned by release-engineering. Resilience-and-recovery does not design release pipelines or progressive-delivery strategies. +- **Post-incident learning pipeline**: owned by incident-learning (wave 5). Resilience-and-recovery feeds findings into that pipeline but does not own the verified-closure process. +- **Production readiness decisions**: owned by the production-excellence bundle (wave 6). Resilience-and-recovery provides resilience evidence as input but does not make the go/no-go call. + +## Summary + +Resilience-and-recovery fills a gap between design-time resilience planning (which SRE, platform, data, security, and release engineering each touch from their own angle) and verified recovery capability. It is the method for designing resilience, exercising it, and proving it — producing evidence that feeds downstream decisions in incident-learning and production-excellence. diff --git a/resilience-and-recovery/references/exercise-design-and-evidence.md b/resilience-and-recovery/references/exercise-design-and-evidence.md new file mode 100644 index 0000000..02794ac --- /dev/null +++ b/resilience-and-recovery/references/exercise-design-and-evidence.md @@ -0,0 +1,70 @@ +# Exercise Design and Evidence + +## Purpose + +Design and execute resilience exercises — game days, restore tests, and failover drills — and record the evidence they produce. Exercises are the standard of proof for resilience claims. A recovery plan without exercise evidence is a design document, not a capability. + +## Exercise types + +| Type | What it tests | Frequency guidance | +|---|---|---| +| **Game day** | System behavior under a specific failure scenario — dependency loss, region failure, resource exhaustion | Per scenario, per release cycle or quarterly | +| **Restore test** | End-to-end restore from backup, including data integrity verification | Monthly for critical systems; quarterly for others | +| **Failover drill** | Failover to a standby or DR region, including application-level validation | Quarterly for systems with DR capability | +| **Chaos experiment** | Continuous or semi-continuous injection of failure in production (or staging) | Ongoing for mature systems; start with game days | + +## Exercise design + +### Pre-exercise + +1. **Define the scenario.** What specific failure is being tested? Be precise: "Primary database region becomes unavailable" not "database failure." +2. **Define success criteria.** Observable, measurable conditions that confirm the system behaved as expected. "Application continues serving reads from replica within 30 seconds" not "system handles failure." +3. **Define the rollback/safety plan.** How to abort the exercise if it goes wrong. Every exercise needs a stop condition and a rollback path. +4. **Notify stakeholders.** Who needs to know the exercise is happening? Use the communication plan from the recovery plan template. +5. **Record the pre-exercise state.** System metrics, data state, configuration — the baseline to compare against. + +### During exercise + +1. **Execute the failure injection.** Follow the scenario precisely. Do not deviate unless the safety plan triggers. +2. **Observe system behavior.** Record: detection time, mitigation activation time, degradation onset, user-visible impact, automated vs. manual responses. +3. **Record anomalies.** Anything unexpected — slower-than-expected recovery, unexpected cascading failures, alerts that did not fire or fired incorrectly. +4. **Abort if safety threshold is crossed.** The exercise is not a martyrdom. If degradation exceeds the pre-defined safety boundary, stop and investigate. + +### Post-exercise + +1. **Restore to pre-exercise state.** Verify the system is back to normal operation. +2. **Compare against success criteria.** Did the system meet every criterion? +3. **Classify findings:** + - **Pass:** All success criteria met. + - **Pass with gaps:** Success criteria met but unexpected behavior observed (e.g., recovery worked but alerts were delayed). + - **Fail:** One or more success criteria not met. +4. **Record evidence.** Date, scenario, participants, observed behavior, metrics, findings, and classification. +5. **Convert findings to follow-up work.** Every gap or failure becomes an entry in the follow-up work ledger with an owner, a target date, and a verification method. + +## Exercise evidence standard + +Exercise evidence is NOT sufficient when it consists only of: + +- Design documentation or architecture diagrams. +- "The plan was reviewed and approved." +- A checklist that was filled out without actual system observation. +- A backup verification report without a restore test. +- A claim that "the system is designed to handle this." + +Exercise evidence IS sufficient when it includes: + +- A timestamped exercise log with observed system behavior. +- Metrics showing the system's actual response (detection time, recovery time, error rates). +- Comparison against pre-defined success criteria with a pass/fail/gaps verdict. +- Findings classified and converted to owned follow-up work. +- The exercise was performed against the actual system (or a faithful replica), not a diagram. + +## Game day scenario examples + +| Scenario | Injection | Expected behavior | Success criteria | +|---|---|---|---| +| Primary database failure | Simulate primary DB unavailability | Application fails over to replica; read-only mode or full service depending on architecture | Failover completes within RTO; no data loss beyond RPO | +| Upstream API timeout | Introduce 30s latency on dependency | Circuit breaker opens; fallback activates; consumers see degraded but acceptable response | Fallback activates within 5s; no error returned to consumers | +| Region loss | Simulate complete AZ/region unavailability | Traffic shifts to secondary region; DR plan activates | Traffic shift completes within RTO; data consistency verified post-failover | +| Disk full | Fill disk on a node to 100% | Monitoring alerts; node drains; service continues on remaining nodes | Alert fires within 2 min; node drains without dropped requests | +| Bad deployment | Deploy a version with a known memory leak | Canary detects degradation; rollback triggers before full rollout | Canary catches degradation within monitoring window; rollback completes without manual intervention | diff --git a/resilience-and-recovery/references/failure-modes-and-dependencies.md b/resilience-and-recovery/references/failure-modes-and-dependencies.md new file mode 100644 index 0000000..822fafe --- /dev/null +++ b/resilience-and-recovery/references/failure-modes-and-dependencies.md @@ -0,0 +1,74 @@ +# Failure Modes and Dependencies + +## Purpose + +Map how a system can fail and what happens when it does. This reference provides the method for identifying failure modes, analyzing dependency behavior under failure, and designing degradation paths with tier-based feature shedding. + +## Failure-mode identification + +### Categories + +| Category | Examples | Detection method | +|---|---|---| +| Component failure | Service crash, OOM kill, deadlock | Health checks, process supervision | +| Dependency failure | Upstream API timeout, database unavailable | Circuit breakers, health indicators | +| Resource exhaustion | Disk full, connection pool saturated, CPU starvation | Resource monitoring, capacity alerts | +| Region/zone failure | Cloud AZ outage, datacenter power loss | Multi-region health checks, DNS failover | +| Data corruption | Bit rot, bad deployment, logical error | Checksums, integrity scans, replica comparison | +| Operator error | Wrong command, wrong target, wrong config | Change reviews, guardrails, dry-run modes | +| Security event | Ransomware, credential leak, unauthorized access | Intrusion detection, anomaly monitoring | +| Cascading failure | Retry storms, thundering herd, dependency chain collapse | Backpressure, circuit breakers, bulkheads | + +### Method + +1. **List every component** in the system boundary. +2. **List every dependency** (upstream and downstream), including external SaaS, internal services, databases, caches, message queues, and third-party APIs. +3. **For each component and dependency**, ask: what happens when it fails? What happens when it is slow? What happens when it returns incorrect data? +4. **Classify each failure** by impact: full outage, degraded but acceptable, degraded and unacceptable, or no impact. +5. **Record assumptions** — especially timeout values, retry behavior, and fallback paths — and verify them in exercises. + +## Dependency behavior under failure + +Every dependency is a failure mode. For each dependency, define: + +| Property | Question | +|---|---| +| **Timeout** | How long does the system wait before treating the dependency as failed? | +| **Retry behavior** | Does the system retry? With what backoff? How many times? Is there a retry budget? | +| **Circuit breaker** | Does the system stop calling a failing dependency after a threshold? What is the threshold? How does it reset? | +| **Fallback** | What does the system do when the dependency is unavailable — cached response, default value, degraded path, or error? | +| **Consumer contract** | What does the system promise ITS consumers when this dependency fails? | + +### Dependency-loss scenarios + +| Scenario | System behavior | Consumer impact | +|---|---|---| +| Dependency unavailable (hard failure) | Circuit breaker opens; fallback activated | Degraded but operational (if fallback exists); error if not | +| Dependency slow (latency degradation) | Timeout triggers after threshold; retries exhaust budget | Slow responses; potential timeout cascades | +| Dependency returns incorrect data | Detection via response validation; circuit breaker may not help (dependency is "up") | Data corruption risk; requires semantic validation | + +## Degradation paths + +### Tier-based feature shedding + +When a dependency or component fails, shed features by tier: + +1. **Tier 3 — Nice-to-have (shed immediately):** Non-critical embellishments — recommendations, social features, cosmetic UI elements, analytics forwarding. These are shed without user-visible impact beyond their absence. +2. **Tier 2 — Enhancing (shed if necessary):** Features that improve experience but are not essential to the core function — advanced search filters, rich formatting, personalization. Shed after Tier 3 if further headroom is needed. +3. **Tier 1 — Core (preserve at all costs):** The system's reason to exist — placing an order, submitting a form, viewing critical data. Must remain available in degraded mode. + +### Degradation design checklist + +- [ ] Which features are shed first, second, and last? +- [ ] What does the user see when a feature is shed — a graceful message, a fallback UI, or nothing? +- [ ] How does the system recover when the dependency returns — automatically or manually? +- [ ] Is there a maximum degraded-operation window after which a full outage is preferable? +- [ ] Has the degradation path been exercised in a game day? + +### Degradation vs. outage decision + +Not every failure justifies degraded operation. Ask: + +- Is degraded operation safer than a clean failure? (e.g., financial systems may prefer to stop than to operate with uncertain data) +- Will degraded operation cause downstream data corruption or inconsistency? +- Can the system detect when degradation is no longer acceptable and fail closed? diff --git a/resilience-and-recovery/references/follow-up-work-ledger.md b/resilience-and-recovery/references/follow-up-work-ledger.md new file mode 100644 index 0000000..2994ca2 --- /dev/null +++ b/resilience-and-recovery/references/follow-up-work-ledger.md @@ -0,0 +1,78 @@ +# Follow-Up Work Ledger + +## Purpose + +Convert every exercise finding into owned, tracked, and verified follow-up work. An exercise that produces findings but no follow-up work is an incomplete exercise. The follow-up work ledger is the bridge between "we found a gap" and "we closed the gap." + +## Ledger template + +| Field | Description | +|---|---| +| **Finding ID** | Unique identifier for this finding | +| **Source exercise** | Which exercise (game day, restore test, failover drill) produced this finding, and when | +| **Finding** | What was observed — be specific: "Restore took 47 minutes against a 30-minute RTO target" not "Restore was slow" | +| **Severity** | blocker (system cannot meet its recovery objective) / gap (system meets objective but with unexpected behavior) / improvement (system meets objective; opportunity to do better) / observation (noted for tracking; no immediate action) | +| **Owner** | Named individual accountable for resolution. Cannot be "the team" or "TBD" | +| **Target date** | When resolution is expected. Must be a specific date, not "next quarter" | +| **Work type** | implementation (code/config change) / test (new or updated test/exercise) / operational (process/procedure change) / documentation (plan/runbook update) | +| **Verification method** | How closure will be verified — re-exercise, automated test, code review, audit | +| **Status** | open / in-progress / verified / escalated / wont-fix (with rationale) | +| **Verification date** | When closure was verified | +| **Verification evidence** | Pointer to evidence — test result, exercise log, commit SHA, audit record | + +## Finding classification + +### Blocker +The finding demonstrates that the system cannot meet a defined recovery objective (RTO, RPO, data integrity). The recovery plan is not valid until this is resolved. + +**Example:** "Restore from backup took 47 minutes. Target RTO is 30 minutes. Gap: 17 minutes." + +### Gap +The system meets its recovery objective but with unexpected behavior — a surprise that did not prevent success but indicates an unknown or unmanaged risk. + +**Example:** "Failover completed within RTO, but alerts for the failover event were delayed by 8 minutes. Operations team was unaware of the failover during that window." + +### Improvement +The system meets its recovery objective. The finding identifies an opportunity to improve beyond the objective. + +**Example:** "Restore completed in 22 minutes against a 30-minute RTO. Parallelizing the restore of two independent data stores could reduce this to under 15 minutes." + +### Observation +Noted for tracking; no immediate action required but worth revisiting. + +**Example:** "During the game day, the team noted that the runbook references a deprecated internal tool name. The procedure works but the documentation is stale." + +## Escalation rules + +Findings that cannot be resolved within the normal follow-up workflow must be escalated: + +| Condition | Escalation | +|---|---| +| No owner can be identified for a blocker finding | Escalate to system owner or engineering manager. The finding cannot remain ownerless. | +| Owner declines ownership without a transfer | Escalate to the owner's manager. Ownership gaps are themselves a finding. | +| Target date passes without resolution | Escalate to the system owner with a revised date or a decision to accept the risk. | +| Blocker finding has no feasible resolution | Escalate to leadership for a risk-acceptance decision. Document the accepted risk. | +| Exercise exposes a gap in another team's ownership (unowned gap) | Escalate to the system owner with the gap documented. The gap must be assigned an owner before the exercise is closed. Do not close the exercise with an unowned gap. | + +## Closure criteria + +A finding is closed (status: verified) only when: + +1. The resolution has been implemented (code change, test addition, process update, documentation fix). +2. The verification method has been executed and passed. +3. Verification evidence is recorded. +4. The owner confirms closure. + +A finding is NOT closed when: + +- A ticket was filed. (Ticket creation is tracking, not verification.) +- "We'll fix it next cycle." (Deferral without a specific date and owner is not closure.) +- "The plan says we handle this." (Design documentation is not verification evidence.) + +## Example entries + +| Finding ID | Source exercise | Finding | Severity | Owner | Target date | Work type | Verification method | Status | +|---|---|---|---|---|---|---|---|---| +| EX-2026-001 | Restore test 2026-03-15 | Restore took 47 min vs 30 min RTO | blocker | jane.chen@example.com | 2026-04-15 | implementation | Re-exercise restore; measure time | open | +| EX-2026-002 | Game day 2026-03-22 | Failover alerts delayed 8 min; ops unaware | gap | marcus.kim@example.com | 2026-04-01 | operational | Monitor alert latency during next game day | in-progress | +| EX-2026-003 | Failover drill 2026-03-29 | Runbook references deprecated tool "backupctl" | observation | priya.patel@example.com | 2026-04-30 | documentation | Review runbook diff | open | diff --git a/resilience-and-recovery/references/recovery-communication.md b/resilience-and-recovery/references/recovery-communication.md new file mode 100644 index 0000000..0b932fa --- /dev/null +++ b/resilience-and-recovery/references/recovery-communication.md @@ -0,0 +1,66 @@ +# Recovery Communication + +## Purpose + +Plan who to notify, when, and through what channels during recovery events. Communication failures during recovery compound technical failures — stakeholders make bad decisions when they lack accurate information. + +## Communication plan template + +### Stakeholder matrix + +| Role | Who | Channel | When to notify | What to communicate | +|---|---|---|---|---| +| **Incident commander** | On-call IC or SRE lead | PagerDuty / Opsgenie / escalation policy | Immediately upon detection | Nature of failure, affected system, initial assessment | +| **System owner** | Named engineering lead for the affected system | Slack, phone | Within 5 minutes of confirmed failure | System status, expected impact, recovery in progress | +| **Engineering team** | On-call engineers for the affected system and dependencies | Slack channel, war room bridge | Within 10 minutes | Technical details, recovery procedure, assistance needed | +| **Product manager** | Named PM for the affected product | Slack, email | Within 15 minutes | User impact, expected duration, customer-facing communication plan | +| **Customer support** | Support team lead or on-call support | Slack, email | Within 15 minutes | What to tell customers, expected resolution time, escalation path | +| **Customers / users** | Status page, in-app notification, email | As defined by SLA | What is affected, what is being done, when to expect update, where to get more information | +| **Executives / leadership** | VP Engineering, CTO, or designated escalation contact | Phone, Slack | For P1/major incidents; within 30 minutes | Business impact, recovery status, estimated resolution, external communication risk | +| **Regulatory / compliance** | Compliance officer, legal | Email, phone | As required by regulation | Incident nature, data affected, containment status, notification timeline | + +### Communication cadence + +| Phase | Frequency | Content | +|---|---|---| +| **Detection to triage** | As information becomes available | What is known, what is unknown, what is being investigated | +| **Active recovery** | Every 15-30 minutes (or per SLA) | Recovery progress, updated ETA, any escalation or blocker | +| **Recovery verification** | At verification completion | Recovery confirmed, data integrity status, any residual impact | +| **Post-recovery** | Once, within 24 hours | Summary of incident, root cause (if known), follow-up actions, next exercise date | + +### Communication templates + +#### Initial notification +``` +Incident: [System] is experiencing [failure type] as of [time]. +Impact: [What users/customers are experiencing]. +Status: Recovery procedure [name] has been initiated. ETA: [estimated recovery time]. +Next update: [time of next scheduled update]. +Contact: [incident commander or recovery lead name and contact]. +``` + +#### Status update +``` +Update: [System] recovery — [time elapsed since start]. +Progress: [What has been done, what is in progress]. +Current status: [System state — degraded, recovering, verifying]. +Revised ETA: [updated estimate if changed]. +Next update: [time]. +``` + +#### Recovery confirmation +``` +Resolved: [System] has been recovered as of [time]. +Verification: [Data integrity confirmed / pending / with gaps]. +Duration: [Total recovery time from detection to verification]. +RTO comparison: [Measured RTO] vs [Target RTO] — [met / exceeded by X]. +RPO comparison: [Measured RPO] vs [Target RPO] — [met / exceeded by X]. +Follow-up: [Link to follow-up work ledger or incident-learning entry]. +``` + +## Anti-patterns + +- **Radio silence during recovery.** Stakeholders fill silence with assumptions. Communicate even when there is no update: "Still investigating; next update at X." +- **Over-communicating to the wrong audience.** Executives need business impact, not technical command output. Engineers need technical detail, not executive summary. Tailor content to audience. +- **No communication plan before the exercise.** If you are figuring out who to notify during a recovery event, the communication plan has already failed. Pre-plan channels and templates. +- **No post-recovery summary.** Stakeholders need closure. A recovery event without a post-recovery summary leaves uncertainty about whether the system is truly recovered. diff --git a/resilience-and-recovery/references/recovery-plan-template.md b/resilience-and-recovery/references/recovery-plan-template.md new file mode 100644 index 0000000..ce0ae04 --- /dev/null +++ b/resilience-and-recovery/references/recovery-plan-template.md @@ -0,0 +1,69 @@ +# Recovery Plan Template + +## Purpose + +A structured template for a system resilience and recovery plan. Every field is required; an empty field is a gap. + +## Template + +### System boundary +- **System name:** +- **System owner (team/individual):** +- **In scope:** Components, services, data stores, and dependencies covered by this plan. +- **Out of scope:** Adjacent systems or components explicitly not covered. + +### Failure modes +| Failure mode | Category | Likelihood | Impact | Detection method | Mitigation | +|---|---|---|---|---|---| +| | component / dependency / resource / region / data / operator / security / cascading | low / medium / high / critical | low / medium / high / critical | | | + +### Dependency map +| Dependency | Direction | Failure behavior | Consumer contract | +|---|---|---|---| +| | upstream (we depend on it) / downstream (it depends on us) | What happens when it fails, is slow, or returns incorrect data | What we promise our consumers when this dependency fails | + +### Degradation choices +| Failure scenario | Tier 3 shed | Tier 2 shed | Tier 1 preserved | User experience | +|---|---|---|---|---| +| | Features shed first | Features shed if necessary | Core function preserved | What the user sees | + +### RTO/RPO decision record +| System / scenario | RTO | RPO | Rationale | Tradeoffs considered | Last verified | +|---|---|---|---|---|---| +| | (context-specific, not a universal value) | (context-specific, not a universal value) | Why this target | Cost, complexity, user impact of alternatives | Date of last exercise | + +### Data integrity +- **Post-restore validation procedure:** Step-by-step verification of data correctness after recovery. +- **Checksum method:** How data integrity is cryptographically verified. +- **Row counts and consistency checks:** Expected counts and cross-table consistency. +- **Reconciliation protocol:** How discrepancies are investigated and resolved. +- **Validation owner:** Who runs the validation and signs off. + +### Recovery procedure +- **Scenario:** +- **Pre-conditions:** What must be true before recovery can begin. +- **Step-by-step procedure:** Numbered, executable steps. +- **Success criteria:** Observable conditions that confirm recovery is complete. +- **Owner:** Who executes the procedure. +- **Last exercise date:** When this procedure was last tested end-to-end. + +### Communication plan +| Scenario | Who to notify | Channel | When | Template | +|---|---|---|---|---| +| | Stakeholder role or name | Email, Slack, PagerDuty, status page | At detection, at start of recovery, at resolution | Link to communication template | + +### Exercise schedule +| Scenario | Last exercised | Result | Findings | Next exercise | +|---|---|---|---|---| +| | Date | pass / fail / pass-with-gaps | Summary of findings | Date | + +### Follow-up work ledger +| Finding ID | Source exercise | Finding | Owner | Target date | Verification method | Status | +|---|---|---|---|---|---|---| +| | | What was found | Named individual | | How closure will be verified | open / in-progress / verified / escalated | + +## Usage notes + +- This template is a starting point. Tailor fields to the system's risk class — higher-risk systems require more detail and more frequent exercise. +- Every field must be populated. A field left empty is a documented gap and must appear in the follow-up work ledger. +- The exercise schedule must show evidence of recent exercises. A plan with no exercise evidence in the last 12 months is a finding, not a passing state. diff --git a/resilience-and-recovery/references/rto-rpo-decision-record.md b/resilience-and-recovery/references/rto-rpo-decision-record.md new file mode 100644 index 0000000..1412b8b --- /dev/null +++ b/resilience-and-recovery/references/rto-rpo-decision-record.md @@ -0,0 +1,69 @@ +# RTO/RPO Decision Record + +## Purpose + +Define Recovery Time Objective (RTO) and Recovery Point Objective (RPO) on a per-system, per-scenario basis using a structured decision record. RTO and RPO are context-dependent; they must never be prescribed as universal values. + +## Why context matters + +RTO and RPO depend on: + +- **System criticality:** A payment processing system has different recovery requirements than an internal wiki. +- **Data classification:** Personally identifiable information, financial records, and audit logs have different RPO constraints than cached or derived data. +- **User impact:** How long can users tolerate unavailability or stale data before the business impact becomes unacceptable? +- **Regulatory requirements:** Some industries mandate specific RTO/RPO ranges (e.g., financial services, healthcare). +- **Cost:** Faster recovery and less data loss cost more. The RTO/RPO decision is a tradeoff between recovery capability and infrastructure cost. +- **Dependency chain:** The system's RTO is bounded by the RTO of its dependencies. You cannot recover faster than the systems you depend on. + +## Examples of context-specific RTO/RPO + +These are examples, not prescriptions. Each system's values must be determined through its own decision record. + +| System type | Example RTO context | Example RPO context | Rationale | +|---|---|---|---| +| Payment processing | May be minutes (financial loss per minute of downtime) | Near-zero (every transaction is a financial record) | Revenue impact per minute; regulatory requirements for transaction records | +| Customer-facing SaaS | May be 1-4 hours (user tolerance for unavailability) | May be 1 hour (acceptable data loss for non-financial data) | User experience impact; no regulatory data-loss constraint | +| Internal wiki | May be 1 business day (non-critical internal tool) | May be 24 hours (document edits are recoverable) | Low user impact; cost of rapid recovery outweighs benefit | +| Batch analytics pipeline | May be 1-2 days (pipeline can be re-run) | May be 24 hours (source data is re-ingestible) | Data is derived; source of truth is elsewhere; re-run is acceptable | +| Healthcare records system | May be minutes (patient safety impact) | Near-zero (clinical data integrity is non-negotiable) | Patient safety; regulatory requirements for data integrity | + +## Decision record template + +### RTO/RPO decision record + +- **System name:** +- **Scenario:** (e.g., primary region loss, data corruption, ransomware) +- **Date of decision:** +- **Decision owner:** + +#### RTO + +- **Target RTO:** (context-specific value with unit — seconds, minutes, hours) +- **Rationale:** Why this target? What is the user/business impact of exceeding it? +- **Tradeoffs considered:** What would it cost (infrastructure, complexity, operational overhead) to achieve a faster RTO? What would a slower RTO cost in user/business impact? +- **Constraints:** Dependencies that bound this RTO (upstream system RTOs, platform capabilities, regulatory minimums). +- **Measurement method:** How is RTO measured in an exercise? Start of outage to verification of recovery. + +#### RPO + +- **Target RPO:** (context-specific value with unit — seconds, minutes, hours) +- **Rationale:** Why this target? What data would be lost if this RPO is exceeded? Is that data loss acceptable? +- **Tradeoffs considered:** What would it cost to achieve a tighter RPO? What data loss would a looser RPO represent? +- **Constraints:** Data classification, regulatory requirements, backup technology limitations. +- **Measurement method:** How is RPO measured in an exercise? Time of last verified backup to time of failure. + +#### Verification + +- **Last exercise date:** +- **Exercise result:** pass / fail / pass-with-gaps +- **Measured RTO:** (actual observed recovery time) +- **Measured RPO:** (actual observed data loss) +- **Gap from target:** (if any — difference between measured and target) +- **Follow-up work:** (if gap exists, reference to follow-up work ledger entry) + +## Anti-patterns + +- **"RTO should be < 1 hour"** — Universal prescription without context. RTO depends on the system. +- **"RPO = 0" without verification** — Claiming zero data loss without an exercise that proves it. +- **Copying RTO/RPO from another system** — Different systems have different failure modes, user impacts, and data classifications. +- **Setting RTO/RPO once and never revisiting** — Systems evolve; recovery objectives must be re-validated as dependencies, data volumes, and user expectations change.