36 KiB
Twenty Years of SRE Lessons Learned & Prodverbs: A Synthesized Analysis
Source material:
- Lessons Learned from Twenty Years of Site Reliability Engineering — Google SRE team
- Prodverbs — Google SRE team
This document synthesizes, analyzes, and cross-references both resources against the canonical Google SRE book. It is not a copy — it distills incident context, practical application, and gaps the original SRE book left open.
Part 1: Eleven Lessons — Synthesized Analysis
Each lesson below follows a consistent structure: Incident Context → The Lesson → What It Means in Practice → How to Apply It.
1. Risk/Mitigation Proportionality
| Aspect | Detail |
|---|---|
| Incident | YouTube caching outage |
| Key insight | A mitigation carries its own risk; a failed mitigation can prolong an outage |
| Original SRE book gap | The book discusses error budgets but not mitigation risk budgeting |
Incident Context: During a YouTube outage caused by a caching layer failure, the SRE team attempted a complex mitigation. When that mitigation failed, it made matters worse — extending the outage significantly. The team had chosen a high-risk, high-reward mitigation without considering what happened if the mitigation itself failed.
The Lesson:
"Monitor the severity of the incident and choose your mitigation risk level accordingly." — Google SRE
In Practice: Every mitigation action has a probability of success and a probability of making things worse. These form a 2x2 matrix:
| Low-severity incident | High-severity incident | |
|---|---|---|
| Low-risk mitigation | Safe choice, standard playbook | May be too slow |
| High-risk mitigation | Unnecessary risk | Potentially justifiable |
How to Apply:
- Classify incident severity within the first 60 seconds using a tier system (S1–S4)
- Predefine what "risk budget" your team has for each severity tier
- Maintain a "mitigation risk register" — track which mitigations have historically backfired
- During an incident, pause before executing a risky mitigation and ask: "If this fails, are we in a worse position than before?"
Counterintuitive insight: Sometimes doing nothing is the correct response to a moderate-severity incident, especially when all available mitigations carry non-trivial failure risk.
2. Test Recovery Mechanisms Before They're Needed
| Aspect | Detail |
|---|---|
| Incident | Generalized — synthesis across multiple Google outages |
| Key insight | "Don't try a ladder during a fire" — untested recovery procedures fail under pressure |
Incident Context: Multiple Google outages revealed that recovery procedures documented in runbooks had not been tested. When SREs attempted them during real incidents, they discovered missing permissions, incorrect scripts, stale database snapshots, or steps that assumed a different state of the system.
The Lesson:
"Test recovery mechanisms during calm periods, not during emergencies." — Google SRE
In Practice: Recovery procedures are code. They rot, they have bugs, and they depend on system state that changes over time. Treating them as documentation rather than executable tests is a recipe for failure.
How to Apply:
- Run quarterly "fire drills" where on-call engineers execute the full recovery procedure in a staging environment
- Automate recovery procedure testing in CI/CD pipelines — if a deploy changes a system, re-validate the recovery path
- Use Chaos Engineering to verify that recovery actually works under degraded conditions
- Track "recovery test coverage" as a metric: what percentage of your failure modes have had their recovery path exercised within the last N months?
3. Canary All Changes
| Aspect | Detail |
|---|---|
| Incident | YouTube config change — 13-minute global outage |
| Key insight | A single misconfigured flag brought down a global service; canarying would have caught it |
Incident Context: A configuration change to YouTube's serving stack was pushed globally without a staged rollout. The change triggered a cascading failure that caused a 13-minute global outage. A canary deployment (to 1–2% of traffic) would have revealed the problem before it reached all users.
The Lesson:
"A small canary deployment is infinitely better than a global outage." — Google SRE
In Practice: The canary strategy is formally described in Google's paper "Introducing the Canary Strategy" but the lesson from YouTube is that even teams who know about canarying skip it. The failure mode is not technical ignorance but process discipline failure.
How to Apply:
- Enforce canary gating at the infrastructure level — not as a human process step
- Define a "canary policy" for every type of change: config changes, code deploys, infrastructure changes, DNS changes
- Canary duration should be based on monitoring signal confidence, not a fixed timer
- Automate rollback when canary metrics breach SLO thresholds
| Change Type | Canary Size | Canary Duration | Metric Gate |
|---|---|---|---|
| Code deploy | 1% → 5% | 5 min | Error rate + latency P99 |
| Config change | 2% | 10 min | Request success rate |
| DNS change | 5% of regions | 30 min | Query resolution latency |
| Infrastructure | 1 cell | Until all health checks pass (min 15 min) | CPU/memory/error rate |
4. The Big Red Button
| Aspect | Detail |
|---|---|
| Incident | Google Calendar near-miss (engineer unplugging their desktop prevented propagation) |
| Key insight | Simple, pre-tested, single-action emergency responses save the day when complex thinking fails |
Incident Context: A Google Calendar incident was approaching critical mass. An engineer, realizing a dangerous rollout was in progress, physically unplugged their desktop from the network. This simple action prevented the bad change from propagating further. The lesson: sometimes the most effective emergency response is a single, well-understood action — not a multi-step procedure.
The Lesson:
"A Big Red Button — a single, pre-tested action to stop or roll back a change — is essential." — Google SRE
In Practice: Under incident pressure, cognitive load is high. Engineers make mistakes executing multi-step procedures. A "Big Red Button" (BRB) is a single action (button click, single command, physical action) that:
- Immediately halts change propagation
- Rolls back the last N minutes of changes
- Notifies the relevant team
How to Apply:
- Identify the most common "oh shit" scenarios for your service
- For each, design a single-action mitigation (one kubectl command, one API call, one button)
- Test the BRB monthly — it must always work
- Guard the BRB against accidental triggering (confirm dialogs, multi-person approval for very destructive actions)
- Log every BRB activation for post-incident review
Design principle: A BRB should be safe to press. If pressing the BRB could cause its own outage, it's not a BRB — it's another dangerous mitigation.
5. Unit Tests Are Insufficient — Integration Testing Is Essential
| Aspect | Detail |
|---|---|
| Incident | Google Calendar outage |
| Key insight | Unit tests passed; the system failed because real-world interaction paths weren't tested |
Incident Context: A Calendar change passed all unit tests with 100% code coverage but caused an outage because the interaction between components — the integration — behaved differently than any component in isolation.
The Lesson:
"Unit tests tell you that components work in isolation. Integration tests tell you the system works. You need both." — Google SRE
In Practice: Unit tests verify that each function/class/module behaves correctly in isolation. But production failures increasingly come from interactions between components: race conditions, protocol mismatches, timeout interactions, data format assumptions, and non-deterministic behavior that only emerges under load.
How to Apply:
- Use the "testing trophy" model (with integration tests as the bulk of your test suite), not the testing pyramid
- Every critical user journey must have an integration test that exercises the full path through the system
- Run integration tests against a realistic environment, not mocks
- Include "dark launch" testing — run new code paths in production without affecting users to observe behavior
6. Backup Communication Channels
| Aspect | Detail |
|---|---|
| Incident | OAuth token incident — 350M users logged out |
| Key insight | When your service goes down, the tools you rely on for incident response may also go down |
Incident Context: A misconfiguration in Google's OAuth system caused approximately 350 million users to be logged out. Crucially, Google's internal incident response tools (Hangouts, Meet, internal dashboards) depended on the same authentication system that had failed. SREs could not communicate via the usual channels to coordinate the response.
The Lesson:
"Design backup communication channels that do not depend on your primary service." — Google SRE
In Practice: This lesson reveals a deep architectural vulnerability: dependency coupling at the tooling layer. If your incident response tools share the same auth system, the same network, or the same infrastructure as your production service, a failure of that shared dependency blinds you at the exact moment you need visibility.
How to Apply:
- Maintain at least one out-of-band communication channel (e.g., a separate Slack workspace on a different provider, a PagerDuty conference bridge on non-Google infrastructure, satellite phones for critical infrastructure)
- Test the backup channel quarterly — just having it isn't enough
- Ensure runbooks are accessible via the backup channel (e.g., printed copies, a static S3 bucket on a different cloud provider)
- Audit your incident response toolchain for shared dependencies with the production service
| Primary Channel | Backup Channel | Dependency Risk |
|---|---|---|
| Google Meet/Hangouts | Twilio conference bridge + Slack on different workspace | Shared OAuth |
| Internal status dashboard | Static site on different provider | Shared authentication |
| Email (Gmail/G Suite) | SMS/pager alerts | Shared identity system |
Key insight from the incident: "We couldn't use our own tools to fix our own tools."
7. Intentionally Degrade Performance Modes
| Aspect | Detail |
|---|---|
| Incident | Generalized — cumulative experience across Google services |
| Key insight | Binary up/down thinking ignores a whole spectrum of useful degraded states |
Incident Context: Google SREs observed that many services were designed as either "fully working" or "completely broken." In reality, most failure modes could be partially mitigated by intentionally degrading performance — serving stale data, disabling non-critical features, reducing recommendation quality, or showing a simpler UI.
The Lesson:
"Move beyond binary up/down thinking. Build graceful degradation into your service." — Google SRE
In Practice: Every feature can be classified along a degradation continuum. The goal is to define, for each feature, what "degraded mode" looks like and how to trigger it.
How to Apply:
- For each feature, define 3–4 degradation levels (full → reduced → minimal → disabled)
- Implement feature flags for every degraded mode
- Build a "degradation dashboard" showing which systems are operating below full capacity
- Test degraded modes in production — your users should know what a degraded experience looks like (it's better than an error page)
- SLOs should have different targets for degraded mode (e.g., normal: 99.9% → degraded: 99.0%)
Practical framework: If your service serves personalized recommendations and the ML pipeline fails, can you serve generic popular items? If your database is overloaded, can you serve stale cache? If your auth system is slow, can you extend session lifetimes?
8. Test for Disaster Resilience
| Aspect | Detail |
|---|---|
| Incident | Generalized — multiple Google disaster scenarios |
| Key insight | Recovery testing ≠ resilience testing |
Incident Context: Google found that teams who had tested recovery procedures (e.g., "restore database from backup") were still caught off guard by resilience failures (e.g., "can the system survive a simultaneous region outage and a DNS failure?"). The two types of testing uncover fundamentally different failure modes.
The Lesson:
"Recovery testing verifies you can fix a problem. Resilience testing verifies you can survive it." — Google SRE
In Practice: Recovery testing is about speed — can you get back to healthy fast enough? Resilience testing is about redundancy — does the system stay healthy when parts fail? Both are necessary.
How to Apply:
- Run tabletop exercises monthly: gather the team, describe a disaster scenario, and trace through what happens. No code changes — just reasoning about failure modes.
- Use chaos engineering (e.g., Chaos Monkey, Litmus) to introduce real failures in staging/production
- Distinguish between:
- Recovery Time Objective (RTO): How fast can you recover?
- Resilience Duration: How long can the system operate in degraded mode before becoming unrecoverable?
- Run a "game day" at least quarterly where a specific disaster scenario is simulated
| Test Type | Example | Frequency |
|---|---|---|
| Tabletop exercise | "What if us-central1 drops off the internet?" | Monthly |
| Chaos experiment | Kill 3 random pods during peak traffic | Bi-weekly |
| Game day | Full simulated region failure + on-call rotation | Quarterly |
9. Automate Mitigations
| Aspect | Detail |
|---|---|
| Incident | 6-day networking outage (March 2023) |
| Key insight | Manual mitigations are too slow; automate the response if the signal is clear |
Incident Context: A Google networking incident in March 2023 stretched for six days. The root cause was complex and took days to fully understand, but the signal that something was wrong was clear within minutes. The team realized that automated mitigations could have ended user impact in minutes rather than days.
The Lesson:
"If you have a clear signal and a known safe action, automate it. Save root-cause analysis for after user impact is resolved." — Google SRE
In Practice: This inverts the traditional priority: most teams prioritize understanding the root cause before acting. The lesson is that mitigation first, diagnosis second is the correct order during an incident. If you can write a rule that reliably detects the failure condition and performs a safe action, automate it.
How to Apply:
- Identify "clear signal + safe action" pairs in your post-incident reviews
- Implement automated mitigations as runbook actions triggered by monitoring alerts
- Use a graduated response: notify (1 min) → suggest action (30 sec) → automated action (immediate)
- Build a "mitigation automation register" — document all automated mitigations, their triggers, their actions, and their safety constraints
Counterintuitive insight: The 6-day outage could have been a 6-minute outage had the mitigation been automated. The root cause took days to find anyway — there was no time saved by not automating.
10. Reduce Time Between Rollouts
| Aspect | Detail |
|---|---|
| Incident | Pokémon GO outage — database field removal |
| Key insight | Slow rollout cadence makes it impossible to reason about the safety of any single change |
Incident Context: When Pokémon GO launched on Google infrastructure, it experienced a major outage caused by removing a database field. The root cause was that the rollout cadence was so slow (weeks between releases) that each change accumulated significant delta from the previous state. When the field removal was rolled out, it was bundled with many other changes — making it impossible to reason about the safety of that specific change in isolation.
The Lesson:
"Release frequently. Small deltas are safe deltas." — Google SRE
In Practice: The relationship between release frequency and change safety is counterintuitive: more frequent releases lead to fewer outages. This is because:
- Each change is smaller and easier to review
- Rollback is faster and has lower blast radius
- The time window for correlated failures to accumulate is shorter
- Each release has lower uncertainty about what it changes
How to Apply:
- Target deployment frequency measured in hours, not weeks
- Each release should change a single logical unit
- If a release takes more than 15 minutes to review, it's too large
- Track "time from commit to production" as a key SRE metric
| Rollout Cadence | Risk Profile | Typical Failure Mode |
|---|---|---|
| Weekly | Low | Small blast radius, easy rollback |
| Monthly | Medium | Moderate change accumulation |
| Quarterly | High | Changes bundle, hard to reason about safety |
| Bi-annual | Very high | Essentially a new system, very high risk |
11. Single Global Hardware Version Is a SPOF
| Aspect | Detail |
|---|---|
| Incident | Networking zero-day (March 2020) |
| Key insight | When all your hardware is the same, one vulnerability takes everything down |
Incident Context: In March 2020, a zero-day vulnerability was discovered in a specific networking chipset that Google used broadly. If Google had standardized on a single networking hardware version globally, a single vulnerability could have taken down the entire network. The incident was mitigated because Google maintained multiple network backbones with diverse hardware.
The Lesson:
"Infrastructure diversity is a resilience strategy. Homogeneity is a single point of failure." — Google SRE
In Practice: This extends beyond hardware. Any single shared dependency — whether hardware, software library, cloud provider, or vendor — is a potential single point of failure. Diversity at the infrastructure layer provides isolation from supply chain attacks, zero-day vulnerabilities, and manufacturing defects.
How to Apply:
- Maintain at least two independent network backbones with different hardware
- Use diversity at every infrastructure layer: compute (different CPU architectures), storage (different storage vendors), networking (different switching/routing hardware)
- For critical software dependencies, have a "second implementation" strategy
- Test failover between diverse infrastructure regularly
Key insight: "Every single point of failure fails eventually" (also a Prodverb — see below). Global hardware uniformity is a hidden SPOF that most teams don't consider.
Lessons Summary Table
| # | Lesson | Novel to 2024? | Gap in Original SRE Book |
|---|---|---|---|
| 1 | Risk/mitigation proportionality | Yes | Book covers error budgets but not mitigation risk |
| 2 | Test recovery mechanisms | Partial | Covered briefly, not emphasized with case study |
| 3 | Canary all changes | No | Well covered in the book |
| 4 | Big Red Button | Yes | Not addressed |
| 5 | Integration testing > unit tests | Partial | Mentioned but not with Calendar case study |
| 6 | Backup communication channels | Yes | Not addressed |
| 7 | Intentional degraded performance | Yes | Not addressed — book is binary up/down focused |
| 8 | Disaster resilience testing | Partial | Tabletop exercises mentioned briefly |
| 9 | Automate mitigations | Partial | Automation covered, but 6-day case study is new |
| 10 | Reduce time between rollouts | Yes | Book assumes fast rollouts; Pokémon GO case is novel |
| 11 | Hardware version diversity | Yes | Not addressed in the original book |
Part 2: Prodverbs — Synthesized Analysis
The Prodverbs page collects pithy, experience-hardened maxims from Google's SRE history. Below, each is analyzed with its full meaning, the scenarios it addresses, and practical application.
"Cascading failures happen, so guard against them."
Full Meaning: A failure in one component can trigger failures in others, which propagate back to the original component in a feedback loop. These are the most dangerous class of failures because they can take down an entire system from a single trigger.
Scenario: A single database replica slows down. The load balancer retries failed queries, increasing load on remaining replicas, which slows them down, triggering more retries. The entire database tier collapses.
Application:
- Implement circuit breakers (fail fast rather than retry into death)
- Use bulkheading — isolate components into failure domains
- Set connection pools with hard limits
- Monitor for "retry storms" — exponential backoff is not optional
"Production is never homogenous."
Full Meaning: Even if you design for uniformity, production will evolve heterogeneity. Different versions of binaries run simultaneously, configs drift, machines age differently, and traffic patterns vary.
Scenario: A team assumes all production servers are identical and deploys a change that only works on one kernel version. Two servers crash; the rest are fine — but nobody knows which ones.
Application:
- Never assume uniformity — build for heterogeneity
- Inventory your production environment regularly and track drift
- Design deployment strategies that work across version skew
- Test against the oldest and newest versions in your fleet
"An empty config doesn't mean you should delete everything."
Full Meaning: A configuration management tool reporting "no config defined" could mean (a) there truly is no config, or (b) the tool can't reach the config source and returned a default empty value. Acting on the latter assumption is catastrophic.
Incident Context: This prodverb is widely attributed to a real Google incident where a configuration management tool lost connectivity to its source of truth, reported "empty config" for a critical service, and an automated cleanup script deleted the service's infrastructure.
Application:
- Before acting on "empty" results, verify the tool's connection to its source of truth
- Implement "delete protection" on production resources
- Use "config validation" — require a signature or checksum on config data
- If a system returns empty, treat it as an alert, not an instruction
Practical pattern: Never trust a tool that returns "empty" when it can't reach its data source. Always distinguish between "known empty" and "unknown state."
"Every single point of failure fails eventually."
Full Meaning: This is the central theorem of reliability engineering. Given enough time, every component, every dependency, every assumption will fail. The question is not if but when.
Application:
- Maintain a "SPOF register" — document every single point of failure in your architecture
- Assign a maximum acceptable risk level to each SPOF
- Prioritize elimination based on impact and likelihood
- Accept that some SPOFs are unavoidable (e.g., power grid) — but have a recovery plan
| SPOF Type | Example | Mitigation |
|---|---|---|
| Hardware | Single network switch | Redundant switches |
| Software | Single cloud provider | Multi-cloud or hot standby |
| Human | Single person with knowledge | Cross-training, runbooks |
| Process | Single deployment pipeline | Backup pipeline, manual fallback |
"First mitigate, then diagnose, then fix."
Full Meaning: The correct order of operations during an incident is: (1) stop the bleeding, (2) understand why the bleeding happened, (3) fix the underlying cause. This is the operational version of triage in medicine.
Common Violation: Engineers are naturally curious. The temptation during an incident is to start diagnosing before mitigating. This prolongs user impact.
Application:
- During an incident, the first action should always be mitigation (rollback, stop traffic, restart)
- Set a timer: 2 minutes for initial mitigation, then diagnose
- Post-incident, distinguish between "mitigation action" and "fix action" in your timeline
- Train on-call engineers to suppress their curiosity during the mitigation phase
"If two systems must agree for them to work, someday they will inevitably disagree."
Full Meaning: Any system that requires two (or more) subsystems to maintain consistent state will eventually experience a state divergence. This is a fundamental property of distributed systems.
Scenario: A primary database and a read replica. The replica falls behind, the primary fails over, the replica is promoted — but it's missing the last 500 writes. Data loss ensues.
Application:
- Design for state divergence — assume consistency will be violated
- Use consensus protocols (Paxos, Raft) where strong consistency is required
- Implement reconciliation processes that detect and repair divergence
- Monitor for divergence metrics (replica lag, clock skew, transaction conflicts)
Practical insight: "Eventually consistent" is a promise about the future, not a guarantee about the present.
"Decrease variance, increase mean."
Full Meaning: Reducing the variance in your system's behavior is often more valuable than improving the average performance. A system with consistent, predictable behavior is easier to operate, debug, and trust.
Application:
- When optimizing, target P99/P999 latency, not P50
- Eliminate "noisy neighbors" through resource isolation
- Standardize deployment patterns, config formats, and monitoring schemas
- Track "operational variance" as a metric: how much does behavior change day-to-day?
| Metric | High Variance | Low Variance |
|---|---|---|
| Deployment time | 5 min – 2 hours | 8–12 min |
| Error rate | 0.01% – 5% | 0.01% – 0.05% |
| P99 latency | 50 ms – 2 seconds | 100–150 ms |
"Backups are only as good as the last restore."
Full Meaning: Having backups is not enough. The only test that matters is whether you can restore from those backups. Untested backups are not backups — they're wishful thinking.
Application:
- Schedule regular restore drills for every backup
- Automate restore verification — don't rely on humans to run it
- Measure "Restore Success Rate" as an SRE metric
- Test restores of the most recent backup, not just a known good one from last month
Corollary: "A backup that has never been restored is a backup that has never been tested."
"If you have no SLOs, toil is your job."
Full Meaning: Service Level Objectives (SLOs) define what "good enough" means. Without them, every minor issue is potentially critical, every degradation requires immediate investigation, and your team spends all its time firefighting. SLOs are the mechanism that transforms toil into engineering.
Application:
- Define SLOs for every service in terms of user-facing metrics
- Use the error budget to decide what deserves immediate attention vs. what can wait
- When SLOs are being met, accept that sub-perfect behavior is acceptable
- Review and adjust SLOs quarterly
"Hope is not a strategy."
Full Meaning: Hoping that a failure won't happen, that a system will recover on its own, that an alert is a false positive — these are all operational failures disguised as optimism. Reliability must be engineered, not wished into existence.
Application:
- Replace "hope" with "plan" for every identified risk
- When you hear "I hope that doesn't happen," ask "What's our plan if it does?"
- Use pre-mortems: assume a future failure, work backward to prevent it
"Scale maintenance sublinearly with the growth of the service."
Full Meaning: As your service grows, operational workload should grow slower than the service itself. If toil grows linearly (or worse, superlinearly) with service size, your team will be overwhelmed. The goal is sublinear scaling of maintenance effort.
Application:
- Automate every operational task that grows with service size
- Invest in tooling that reduces per-server or per-request maintenance overhead
- Track "toil per unit of service" as a metric
- Target: 50% of SRE time on engineering, 50% or less on toil
"May all your incidents be novel."
Full Meaning: This is a benediction from one SRE to another. Novel incidents teach you something new about your system. Repeated incidents (the same failure mode recurring) indicate that your post-incident actions were insufficient.
Application:
- Track incident types and look for repeats
- If the same root cause causes multiple incidents, the corrective actions from the first PIR were inadequate
- Novel incidents are opportunities to learn; repeated incidents are failures of process
Prodverbs Reference Table
| Prodverb | Core Category | Practical Application |
|---|---|---|
| Cascading failures happen | Failure modes | Circuit breakers, bulkheading |
| Production is never homogenous | Operations | Build for heterogeneity |
| Empty config ≠ delete everything | Automation peril | Verify before acting on "empty" |
| Every SPOF fails eventually | Design principle | SPOF register, elimination |
| Mitigate → diagnose → fix | Incident response | Priority ordering during incidents |
| Two agreeing systems will disagree | Distributed systems | Design for state divergence |
| Decrease variance, increase mean | Metrics strategy | Target P99 before P50 |
| Backups = last restore | Data integrity | Regular restore drills |
| No SLOs → toil is your job | SLO importance | Define SLOs for every service |
| Hope is not a strategy | Operations culture | Plans, not prayers |
| Scale maintenance sublinearly | Automation | Toil tracking, automation ROI |
| May all incidents be novel | Learning culture | Track incident repetition |
Part 3: Synthesis — How These Lessons and Prodverbs Complement the Core SRE Book
The original Google SRE book (Beyer et al., 2016) established the foundational framework: SLOs, error budgets, monitoring, toil elimination, and the SRE model itself. The "Twenty Years" lessons and Prodverbs extend this framework in important ways.
What the Original SRE Book Covers Well
The book thoroughly addresses:
- The SRE model and its relationship to traditional ops
- Service Level Objectives, Indicators, and Agreements
- Error budgets as a decision-making mechanism
- Monitoring and alerting (the "four golden signals")
- Automation and the elimination of toil
- Capacity planning and load shedding
- Post-incident review culture
What the New Lessons Add
Novel Additions (Not in the Original Book)
1. Risk/Mitigation Proportionality (Lesson 1): The book discusses accepting risk via error budgets, but doesn't address the risk of mitigations themselves. This is a critical operational insight: mitigations have their own probability of failure, and choosing the wrong mitigation for the severity level can make things worse.
4. Big Red Button (Lesson 4): The concept of a pre-designed, single-action emergency response is entirely absent from the original book. It addresses runbooks and procedures but not the cognitive-load insight that complex procedures fail during emergencies.
6. Backup Communication Channels (Lesson 6): The original book doesn't consider that your incident response tools might fail along with your service. This is a cultural blind spot in the original work — it assumes the tooling layer is always available.
7. Intentional Degraded Performance (Lesson 7): The original book frames reliability in binary terms (available/unavailable). The degraded-mode concept — deliberately serving a reduced experience — is not addressed.
10. Reduce Time Between Rollouts (Lesson 10): While the book assumes fast rollouts, the Pokémon GO case study demonstrates the causal mechanism: slow rollouts produce large deltas, which makes safety reasoning impossible. This is a more precise diagnosis than the book provides.
11. Hardware Version Diversity (Lesson 11): Infrastructure diversity is not discussed in the original book. It assumes homogeneous infrastructure is the goal. The March 2020 zero-day demonstrates that homogeneity is itself a vulnerability.
Reinforcements and Extensions
Lesson 2 (Test Recovery): Reinforces the SRE book's emphasis on automation but adds the specific insight that untested recovery procedures are a form of technical debt.
Lesson 3 (Canary All Changes): Extends the book's canary discussion with a concrete failure case (YouTube 13-minute outage) and the insight that the failure mode is often process discipline, not technical knowledge.
Lesson 5 (Integration Testing): Extends the book's discussion of testing with the Calendar outage case study, which demonstrates that even 100% unit test coverage can miss interaction failures.
Lesson 8 (Disaster Resilience): Extends the book's brief mention of tabletop exercises into a full framework distinguishing recovery testing from resilience testing.
Lesson 9 (Automate Mitigations): Extends the book's automation discussion with the 6-day networking outage case study, which provides the "clear signal → safe action" framework for prioritizing automated mitigation over root cause analysis.
How Prodverbs Fill Gaps
The Prodverbs provide operational wisdom that the book's formal framework doesn't capture:
| Prodverb | Gap in Original Book | How It Fills It |
|---|---|---|
| "Empty config ≠ delete everything" | Book assumes automation is safe; doesn't address automation failure modes | Identifies a specific class of automation-induced disasters |
| "Production is never homogenous" | Book assumes you can enforce uniformity | Acknowledges that production will always drift, and you must design for that |
| "May all your incidents be novel" | Book focuses on PIR process; doesn't address incident repetition | Introduces the concept that repeated incidents indicate process failure |
| "Decrease variance, increase mean" | Book focuses on mean-based metrics | Shifts focus to variance reduction as a primary reliability strategy |
| "If two systems agree, they will disagree" | Book assumes eventual consistency is adequate | Acknowledges that state divergence is not a corner case but a certainty |
Practical Framework for Teams
To integrate these lessons with the original SRE book:
- Start with the SRE book foundation — error budgets, SLOs, monitoring, toil elimination
- Adopt the "Big Red Button" as an operational stopgap (Lesson 4)
- Audit your communication channels for shared dependencies (Lesson 6)
- Design degraded modes for every critical feature (Lesson 7)
- Run quarterly tabletop exercises for disaster scenarios (Lesson 8)
- Automate at least one mitigation per quarter (Lesson 9)
- Track deployment frequency and push toward hourly rollouts (Lesson 10)
- Diversify infrastructure at the hardware and vendor layers (Lesson 11)
- Internalize the Prodverbs — they encode decades of hard-won experience in single sentences
- Review incident types quarterly and flag repeated failures (Prodverb: "May all your incidents be novel")
Closing Thought
"The SRE book gave us the theory. These twenty years of lessons gave us the scars. The Prodverbs gave us the wisdom." — Synthesized from Google SRE community discussions
The original SRE book is a design document — it describes how to build reliable systems. The "Twenty Years" lessons are an after-action report — they describe what still goes wrong despite good design. The Prodverbs are an operational manual — they encode the instincts that keep you out of trouble between the formal processes.
All three are essential. None alone is sufficient.