65 KiB
Google SRE Book — Comprehensive Chapter Summaries
Source: Site Reliability Engineering: How Google Runs Production Systems (Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy, eds.) Online: https://sre.google/sre-book/ Purpose: Quick reference for SRE agents and practitioners — core theses, key principles, actionable takeaways, and direct quotes from each chapter.
Table of Contents
Part I: Introduction
Chapter 1: Introduction
Core Thesis
SRE is "what happens when a software engineer is tasked with designing an operations team." It is a discipline that applies a software engineering mindset to system administration and production operations. The chapter establishes the philosophy, organizational model, and key distinctions between SRE and traditional IT operations.
Key Principles
| # | Principle | Description |
|---|---|---|
| 1 | Hiring from Software Engineering | SREs are software engineers who design and build automation to run systems — they don't manually operate them |
| 2 | Operations as a Software Problem | Toil reduction is achieved through code, not process |
| 3 | Shared Ownership | SRE shares responsibility for system health with product/dev teams |
| 4 | The 50% Cap on Operational Work | SRE teams spend at most 50% of time on ops; the remainder goes to engineering projects that reduce future toil or add reliability features |
| 5 | Error Budget | 100% reliability is the wrong target; acceptable risk is quantified and managed as a budget |
| 6 | SLOs Drive Decisions | Releases are gated on error budget burn rate, not arbitrary freezes |
The SRE <-> Dev Relationship
┌─────────────────────────────────────────────────────────┐
│ DEV TEAM SRE TEAM │
│ ┌─────────┐ ┌─────────┐ │
│ │ Feature │ │Stability│ │
│ │ velocity│ │reliability│ │
│ └────┬────┘ └────┬────┘ │
│ │ │ │
│ └───────── Error ──────────┘ │
│ Budget │
│ │
│ Shared responsibility within error budget tolerance │
└─────────────────────────────────────────────────────────┘
The Seven Levels of SRE Maturity (Informal)
- Dev ops model — devs push code, ops runs it, ad-hoc communication
- SRE is hired — first SREs join, begin measuring and automating
- SLOs defined — explicit reliability targets and error budgets
- Error budget-driven releases — releases blocked when budget exhausted
- Automation replaces toil — >50% of ops work automated
- Product reliability as a feature — SRE expertise embedded in design phase
- Self-healing systems — systems adapt to failure without human intervention
Actionable Takeaways
- For leadership: Hire SREs from software engineering, not sysadmin backgrounds. Enforce the 50% ops cap.
- For teams: Define an error budget before defining SLOs. This changes the conversation from "how do we prevent all outages?" to "how much downtime is acceptable?"
- For engineers: Treat operations as a software engineering problem. If you fix a thing manually three times, automate it.
Important Quotes
"The overriding goal of SRE is to make the systems run themselves."
"SRE is a profession that encompasses both software engineering and operations — it's what happens when a software engineer is tasked with designing an operations team."
"A fundamental principle of SRE is that operations is a software problem."
"At Google, SRE is fundamentally doing work that has two aspects: (1) making Google's system more reliable, and (2) making Google's engineering teams more productive."
"The SRE model is predicated on the idea that a system is never 'done' — it is always evolving, and operations is the day-to-day task of managing that evolution."
Chapter 2: The Production Environment at Google, from the Viewpoint of an SRE
Core Thesis
To understand SRE, one must understand the environment it operates in. This chapter describes Google's hardware infrastructure, the software stack that runs on it, and the lifecycle of a user request — from DNS lookup to serving an ad or search result.
Key Concepts
| Concept | Description |
|---|---|
| Borg | Google's cluster management system (predecessor to Kubernetes). Schedules jobs onto machines, handles failures, manages resources |
| Borg Name Service (BNS) | Naming and service discovery layer used by all Google services |
| GSLB (Global Software Load Balancer) | DNS-based traffic distribution across datacenters |
| Megastore / Spanner | Globally distributed storage systems providing strong consistency |
| Chubby | Distributed lock service (based on Paxos) used for leader election and configuration |
The Lifecycle of a Request
User → DNS → GSLB → Frontend → Backend → Storage
│ │ │ │ │
Latency-based │ RPC load BNS-based Megastore
load balancing │ balancing discovery / Spanner
│ │
HTTP(S) Colocation
termination & caching
Software Infrastructure Layers
- Networking — Jupiter datacenter network fabric, B4 WAN
- Storage — Colossus (GFS v2), Bigtable, Spanner, Megastore
- Compute — Borg, Omega, Kubernetes (later)
- Service Infrastructure — gRPC, Protocol Buffers, Stubby
Key Insight for SREs
The physical infrastructure is abstracted to such a degree that an SRE rarely thinks about individual machines. The unit of management is the service — a set of jobs running on Borg that collectively provide an API. Failures are expected; the system is designed to route around them.
Important Quotes
"The sheer scale of Google's infrastructure is difficult to overstate. More than a billion user requests hit Google's servers in a single minute."
"If you have to think about which machine your code is running on, you're doing it wrong."
"When machines fail — and they do, every day — the system automatically shifts work to healthy machines."
Part II: Principles
Chapter 3: Embracing Risk
Core Thesis
100% reliability is neither achievable nor desirable. The cost of reliability increases non-linearly — the last 0.01% requires orders of magnitude more investment than the first 99.99%. SRE explicitly manages reliability as a trade-off against feature velocity, cost, and complexity.
Core Thesis (Extended)
The key insight: If a service is "too reliable," you are wasting resources that could be spent on features, reducing latency, or improving other services. The goal is to find the optimal level of reliability, not the maximum.
Key Principles
| # | Principle | Explanation |
|---|---|---|
| 1 | Reliability is a spectrum | Not binary (reliable vs. unreliable). SRE defines where on the spectrum a service should sit |
| 2 | Non-linear cost curve | Each "nine" of availability costs roughly 10x the previous one |
| 3 | Error budgets align incentives | Devs want to ship; SREs want stability. The error budget gives both sides a shared, measurable goal |
| 4 | Risk tolerance is service-specific | Gmail can have different SLOs than Google Ads; tier-1 services have tighter budgets |
| 5 | Risk changes over time | A young service may tolerate more risk; a mature service should tighten its SLOs |
Managing Risk Targets (Flowchart)
┌──────────────────────┐
│ Identify service need │
└──────────┬───────────┘
│
┌──────────▼───────────┐
│ What level of │
│ availability does the │
│ service require? │
└──────────┬───────────┘
│
┌──────────▼───────────┐
│ What would 100% cost? │
└──────────┬───────────┘
│
┌──────────▼───────────┐
│ Define SLO & error │
│ budget accordingly │
└──────────┬───────────┘
│
┌──────────▼───────────┐
│ Measure & iterate │
│ (quarterly reviews) │
└──────────────────────┘
Actionable Takeaways
- Define explicit risk tolerance for every service. Write it down. If you can't state it, you can't measure it.
- Calculate error budget monthly/daily:
(1 - SLO) * total requests = allowed errors. Track burn rate. - Use error budget to gate releases: When the budget is spent (or nearly spent), stop all changes until it replenishes.
- Adapt targets to service tier: Tier 1 (user-facing, revenue-critical) gets 99.99%; Tier N (internal tools, batch) might get 99.9% or less.
- Monitor unavailability carefully: Not all downtime is equal. Partial degradation may cause more user harm than a hard outage.
Important Quotes
"If an SRE team spends all their time making a service 100% reliable, they're doing it wrong. The most reliable service is one that doesn't change — but nobody uses it."
"The risk of a system failing is directly proportional to the cost of preventing that failure. The trick is to optimize for the 'acceptable' level of risk, not zero risk."
"An error budget is a tool for rational discussion about risk between product development and SRE."
"Managing risk is about deciding what not to do as much as what to do."
"Google's experience suggests that 100% is the wrong reliability target for (nearly) everything."
Chapter 4: Service Level Objectives
Core Thesis
SLOs are the contract between SRE and the product team — and between the service and its users. This chapter details how to define, measure, and use Service Level Indicators (SLIs), Service Level Objectives (SLOs), and Service Level Agreements (SLAs). Without good SLOs, you cannot know whether the service is "healthy" or not.
Key Definitions
| Term | Definition | Example |
|---|---|---|
| SLI | A carefully defined quantitative measure of some aspect of the service level | Request latency at 95th percentile |
| SLO | A target value or range for an SLI that represents acceptable service | 99% of requests < 200ms at p95 |
| SLA | An explicit or implicit contract with users that includes consequences for breach | 99.9% uptime; failure = service credits |
SLI Selection Framework
┌───────────────────────────────────────┐
│ What matters to users? │
│ ┌──────────┬──────────┬──────────┐ │
│ │ Latency │Throughput│Availabil-│ │
│ │ │ │ ity │ │
│ ├──────────┼──────────┼──────────┤ │
│ │ p50, │ QPS, │ Uptime, │ │
│ │ p95, │ req/s │ success │ │
│ │ p99 │ │ rate │ │
│ └──────────┴──────────┴──────────┘ │
│ ┌──────────┐ │
│ │ Durability│ (for storage systems) │
│ └──────────┘ │
│ ┌──────────┐ │
│ │Correctness│ (for transactions) │
│ └──────────┘ │
└───────────────────────────────────────┘
SLI Measurement Approaches
| Approach | Method | Use Case |
|---|---|---|
| Server-side | Aggregate from application logs | Controlled, accurate |
| Client-side | Synthetic probes (RPCs) | Reflects actual user experience |
| Client-side (real) | Browser instrumentation (e.g., RUM) | Best for latency, hardest to implement |
Common SLIs at Google
| SLI | Definition | Target |
|---|---|---|
| Availability | (successful requests / total requests) * 100 |
≥ 99.9% |
| Latency (p50) | Median response time | < 100ms |
| Latency (p95) | 95th percentile response time | < 200ms |
| Latency (p99) | 99th percentile response time | < 500ms |
| Throughput | Requests per second | Variable |
| Error rate | Count or fraction of 5xx responses | < 0.1% |
SLO Construction Rules
- Choose few SLOs. More than 3-4 SLOs per service leads to dilution of focus.
- Prefer "loose" SLOs initially. You can always tighten them; loosening an SLO means admitting you can't meet your target.
- Don't aim for 100%. Pick a target that reflects actual user needs.
- Use "target ± tolerance" for SLI windows. Example: "p95 latency < 200ms over any rolling 5-minute window."
- SLOs are aspirational, SLAs are contractual. SLO failure is a signal; SLA failure is a financial event.
The SLO Burn Rate Model
┌──────────────────────────────────────────┐
│ Burn Rate = Error Budget consumed / time │
├──────────────────────────────────────────┤
│ Example: │
│ 99.9% SLO = 0.1% error budget / month │
│ 1% errors for 1 hour = ~1.4% of monthly │
│ budget consumed in 1 hour │
├──────────────────────────────────────────┤
│ Alert thresholds: │
│ - Page when burn rate > 2x for 1 hour │
│ - Page when burn rate > 10x for 10 mins │
│ - Ticket when burn rate > 1x for 1 day │
└──────────────────────────────────────────┘
Actionable Takeaways
- Start with 2-3 SLIs per service. Availability and latency cover 90% of use cases.
- Measure from the user's perspective. Server-side metrics alone lie — they don't capture network issues.
- Make SLOs visible. Dashboard them prominently, post them in team rooms, include them in every release review.
- Use burn rate alerts instead of static threshold alerts. They catch problems faster and with less noise.
- Quarterly SLO reviews. Services evolve; their SLOs should too.
Important Quotes
"You can't manage what you don't measure. More importantly, you can't optimize what you don't measure well."
"SLOs are not just numbers — they are a statement of intent about how your service should behave, and they form the basis of conversations between SRE, product development, and management."
"An SLO that users don't notice is probably too tight. An SLO that users complain about is probably too loose."
"A common mistake is to define too many SLOs. The right number is small — ideally 3 (±1) — because each additional SLO reduces the attention paid to each one."
Chapter 5: Eliminating Toil
Core Thesis
Toil is the enemy of SRE. Defined as work that is manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly with service growth. SRE must actively eliminate toil to preserve the engineering time needed to build reliable systems.
The Five Characteristics of Toil
| # | Characteristic | Description |
|---|---|---|
| 1 | Manual | Requires human touch (e.g., SSHing to a machine to restart a process) |
| 2 | Repetitive | Done over and over (e.g., responding to the same alert daily) |
| 3 | Automatable | A computer could do it — you just haven't written the code yet |
| 4 | Tactical | Puts out fires but doesn't improve the system |
| 5 | No enduring value | Once done, the system is no better than before |
| 6 | Scales linearly | Doubling the service size doubles the toil |
The Toil Taxonomy
| Category | Examples | Impact |
|---|---|---|
| Manual operations | Restarting jobs, reprovisioning VMs, clearing queues | Direct time drain |
| Acknowledgment fatigue | Reading non-actionable alerts, acknowledging pages for known false positives | Attentional debt |
| Ticket handling | Duplicate bug reports, user password resets, permission requests | Context switching |
| Incident response | Same outage pattern recurring weekly without a fix | No improvement |
| Data munging | Extracting logs by hand to answer ad-hoc questions | Repeated cognitive load |
Measuring Toil
Toil Percentage = (Hours spent on toil / Total SRE hours) × 100
Target: < 50% (enforced by management)
Warning: 50-65% (needs intervention)
Critical: > 65% (team is in firefighting mode)
Strategies for Eliminating Toil
- Automate relentlessly. Every manual step is a potential automation target. Write scripts, tools, or full systems to handle it.
- Build self-service tooling. Give dev teams access to deploy, rollback, and monitor without SRE intervention.
- Improve system architecture. Redesign components that generate disproportionate toil (e.g., replace stateful services with stateless ones).
- Say "no." Not every request deserves SRE attention. Use the error budget model to push back.
- Instrument toil. Track time spent on toil categories. What gets measured gets reduced.
The 50% Ops Cap in Practice
┌─────────────────────────────────────────────┐
│ SRE Team Time Allocation (Goal) │
├─────────────────────────────────────────────┤
│ ┌─────────────────────────┬───────────────┐ │
│ │ Project Work (≥ 50%) │ Ops (≤ 50%) │ │
│ │ ┌───────────────────┐ │ ┌───────────┐ │ │
│ │ │ Automation │ │ │ Alerts │ │ │
│ │ │ Architecture │ │ │ Tickets │ │ │
│ │ │ Capacity planning │ │ │ On-call │ │ │
│ │ │ Performance │ │ │ Releases │ │ │
│ │ │ Tooling │ │ │ Manual │ │ │
│ │ └───────────────────┘ │ └───────────┘ │ │
│ └─────────────────────────┴───────────────┘ │
└─────────────────────────────────────────────┘
If ops exceeds 50%: The team stops project work and declares operational bankruptcy. The only project work allowed is that which reduces toil.
Actionable Takeaways
- Track toil as a metric. Add it to your team's dashboard. If you can't measure it, you can't cap it.
- One-off fixes are acceptable. The third time you do something manually, automate it.
- Use "cutting the red wire" test: If you're asked to do a manual operation under pressure, ask: "If the building was on fire, would this still need to be done by a human?"
- Refuse to hire your way out of toil. Adding more SREs to a toil-heavy team just increases the toil budget. Fix the system.
Important Quotes
"Toil is the enemy of SRE. It saps the energy, time, and creativity of the people who are supposed to be making the system better."
"If a machine can do it, a machine should do it."
"The 50% rule is the single most important organizational mechanism for making SRE work. Without it, SRE teams inevitably become operations teams."
"Hiring more people to do a job a computer could do is not a strategy."
"Toil is not just boring — it's dangerous. The more toil an SRE team does, the less time they have for the engineering work that prevents future outages."
Chapter 6: Monitoring Distributed Systems
Core Thesis
Monitoring is the foundation of SRE practice. This chapter presents Google's philosophy on monitoring: four golden signals, the difference between alerting, dashboarding, and debugging tools, and how to build monitoring that produces actionable signals — not noise.
The Four Golden Signals
| Signal | What It Measures | Why It Matters |
|---|---|---|
| Latency | Time to service a request | High latency = unhappy users, cascading failures |
| Traffic | Demand on the system (QPS, connections, active users) | Predicts scaling needs |
| Errors | Rate of failed requests (explicit 5xx, implicit bad content, slow responses) | Direct indicator of system health |
| Saturation | How "full" the service is (CPU, memory, disk, connections, quota) | Predicts capacity failure |
The Monitoring Stack at Google
┌──────────────────────────────────────┐
│ Alerting │
│ (Triggers pages, pages SRE on-call) │
├──────────────────────────────────────┤
│ Dashboards │
│ (ROI: answer "what's broken?" fast) │
├──────────────────────────────────────┤
│ Debugging Tools │
│ (Trace analysis, log explorers) │
├──────────────────────────────────────┤
│ Data Pipeline │
│ (Log collection, aggregation, query) │
├──────────────────────────────────────┤
│ Instrumentation │
│ (Application metrics emitted) │
└──────────────────────────────────────┘
Rules for Good Alerting
| Rule | Explanation |
|---|---|
| Rule 1 | Every page must be actionable (a human must do something) |
| Rule 2 | Every page must be urgent (the thing can't wait until morning) |
| Rule 3 | Every page must be novel (not the same problem that pages every night) |
| Rule 4 | Tickets for non-urgent issues; pages only for emergencies |
Categorizing Monitoring Output
| Category | Action | Latency Tolerance | Example |
|---|---|---|---|
| Page | Wake someone up | Seconds to minutes | Service down, error budget burning fast |
| Ticket | Create a work item | Hours to days | Certificates expiring, disk filling up |
| Logging | No immediate action | None | Debugging data for postmortems |
| Dashboard | Visual for ad-hoc review | N/A | Real-time latency heatmap |
Reducing Alert Noise
Common pattern at Google:
- ~10,000 raw monitoring rules defined
- ~100 alerts per day after dedup
- ~5-10 pages per shift (before tuning)
- Target: 0 false-positive pages per shift
Actionable Takeaways
- Monitor symptoms, not causes. "My database is slow" is a cause. "User requests timeout" is a symptom. Alert on symptoms.
- Prefer black-box monitoring (from the user's perspective) over white-box monitoring (internal metrics) for alerting.
- Use white-box monitoring for debugging and capacity planning.
- Apply the "alerts as a product" mindset. Every alert is a product that should be tested, documented, and have an owner.
- Delete alerts that never fire. If an alert hasn't fired in 6 months, remove it. If you need it later, add it back.
Important Quotes
"When a system is on fire, you do not have time to look at a graph. You need a pager."
"The best monitoring is the kind that never pages you because the problem was already fixed by automation."
"If you page a human for every anomaly, you train them to ignore the pager."
"Monitoring should never require a human to interpret 'what's wrong.' The alert should tell them."
"A dashboard that nobody looks at is as useful as no dashboard at all. Build monitoring for the signal, not the noise."
Chapter 7: The Evolution of Automation at Google
Core Thesis
Automation is the SRE's primary tool for eliminating toil, reducing error, and enabling scale. This chapter traces Google's automation journey from manual scripts through Borg to the fully automated release pipeline — and extracts the general principles that apply to any organization.
Levels of Automation Maturity
| Level | Description | Example |
|---|---|---|
| 1 — Manual | No automation. Every step is done by a human | SSH to machine, run command |
| 2 — Scripted | Single steps automated with scripts | Shell script that runs a series of SSH commands |
| 3 — Templated | Repeatable patterns with parameters | Ansible playbooks, deployment templates |
| 4 — Monitored | Automation observes and adjusts | Auto-scaling, self-healing service |
| 5 — Autonomous | System manages itself fully | Borg: schedules work, recovers from failures, allocates resources |
The Automation Decision Framework
Before automating, ask:
┌──────────────────────────────┐
│ 1. Will this save time? │
│ (time saved > time to │
│ build + maintain) │
├──────────────────────────────┤
│ 2. Will this reduce errors? │
│ (human error rate vs. │
│ software error rate) │
├──────────────────────────────┤
│ 3. Will this let humans │
│ focus on higher-value │
│ work? │
├──────────────────────────────┤
│ 4. Does the process change │
│ frequently? │
│ (if yes, automate later) │
└──────────────────────────────┘
Benefits of Automation
| Benefit | Explanation |
|---|---|
| Consistency | The same action is performed identically every time |
| Speed | Machines work faster than humans, especially for multi-step operations |
| Reliability | Well-tested automation fails less than a tired human at 3 AM |
| Scalability | Automation costs do not increase linearly with service growth |
| Human focus | Frees SREs to work on architecture, performance, and features |
Automation Pitfalls
- Automating the wrong thing — Don't automate a process you plan to eliminate.
- Brittle automation — Over-specific scripts break when the environment changes.
- Bypassing expertise — Junior engineers may blindly trust automation without understanding what it does.
- Black box syndrome — Automated systems become so complex that nobody knows how to fix them when they break.
- Loss of skills — When nobody does a task manually anymore, the knowledge of how it works is lost.
The Borg Automation Story (Case Study)
Manual (2003): Engineers submit jobs via email, manually assign machines
│
Scripted (2004): Simple scheduler allocates machines
│
Borg (2005): Full cluster management — scheduling, health checking, restart
│
Omega (2012): Next-gen scheduler with better isolation and performance
│
Kubernetes (2014): Open-source version of Borg's principles
Actionable Takeaways
- Automate in layers. Start with the most painful, highest-frequency manual tasks.
- Don't aim for level 5 immediately. Level 3 (templated) captures 80% of the value.
- Keep automation debuggable. Every automated step should log what it did and why.
- Write automation as services, not scripts. A service can be monitored, tested, and evolved; a script sits on someone's laptop.
- Plan for maintenance. The cost of maintaining automation is real. Factor it into the ROI calculation.
Important Quotes
"Automation is not just about efficiency. It's about enabling humans to do more valuable work."
"The most reliable system is one that never requires a human being to take action."
"Automation is a force multiplier. A single SRE with good automation can manage thousands of machines. Without it, they can manage dozens."
"Beware of automating inefficient processes. You'll just get broken things faster."
"Google's experience has shown that the best automation is the kind that doesn't need to be watched. If you have to monitor your automation, you've just shifted the toil, not eliminated it."
Chapter 8: Release Engineering
Core Thesis
Release engineering is the discipline of building and deploying software reliably, repeatably, and at scale. Google's approach treats the release process as a software engineering problem in its own right — with defined roles, automated pipelines, and rigorous testing at every stage.
The Release Engineering Pipeline
┌─────────┐ ┌──────────┐ ┌─────────┐ ┌──────────┐ ┌───────────┐
│ Dev │→│ Commit │→│ Build │→│ Test │→│ Deploy │
│ Branch │ │ Queue │ │ (Bazel) │ │ Suite │ │ (Sisyphus) │
└─────────┘ └──────────┘ └─────────┘ └──────────┘ └───────────┘
│ │ │
│ ┌───────┴───────┐ │
│ │ Unit / Int │ │
│ │ / System / │ │
│ │ Canary / │ │
│ │ Prod │ │
│ └───────────────┘ │
│ ┌───────┴───────┐
│ │ Canary → 5% │
│ │ → 25% → 50% │
│ │ → 100% │
│ └───────────────┘
Key Concepts
| Concept | Description |
|---|---|
| CI (Continuous Integration) | Every commit is built and tested automatically |
| CD (Continuous Delivery) | Software is always in a deployable state |
| Progressive Rollout | Deploy to small subsets of users first, then expand |
| Rollback | Reverting to a known-good version is as important as rolling forward |
| Bazel | Google's build system — deterministic, hermetic, scalable |
| Sisyphus | Google's deployment system — manages fleet updates with minimal disruption |
The Release Cadence Model
┌─────────────────────────────────────┐
│ Google's approach: │
│ │
│ - Single mainline branch │
│ - Every commit must pass CI │
│ - Release candidates cut on │
│ schedule (e.g., weekly) │
│ - Candidates go through test │
│ pipeline (hours to days) │
│ - If candidate fails, next one │
│ is cut from latest mainline │
│ - Deploy progression: │
│ Canary → 5% → 25% → 50% → 100% │
│ - Rollback is a first-class │
│ operation │
└─────────────────────────────────────┘
Binary vs. Configuration Releases
| Aspect | Binary Release | Configuration Release |
|---|---|---|
| Frequency | Weekly/biweekly | Daily or more |
| Risk | Higher (new code) | Lower (tuning existing code) |
| Rollback | Full deploy pipeline | Instant revert |
| Testing | Full CI/CD pipeline | Config schema validation |
| Tooling | Bazel + Sisyphus | Rapid evaluation + push |
Actionable Takeaways
- Make releases boring. A good release process should be so reliable that nobody worries about it. Drama in releases = process failure.
- Invest in build infrastructure. A slow or unreliable build is a productivity killer.
- Staged rollouts are non-negotiable. Always deploy to a canary. Always. 100% deployment on day one is a crash waiting to happen.
- Test rollbacks as thoroughly as rollouts. If you can't roll back, you can't safely roll forward.
- Treat configuration as code. Version it. Review it. Test it. Roll it back.
Important Quotes
"Release engineering is not an afterthought — it's a first-class engineering discipline."
"A release process that requires manual steps is not a process; it's a ceremony."
"The best release engineer is the one nobody ever has to page."
"If you're not testing your rollback procedure, you don't have a rollback procedure — you have a wish."
"Configuration pushes are the most dangerous thing a team does. A single typo in a config file can take down a global service."
Chapter 9: Simplicity
Core Thesis
Complexity is the root cause of most reliability failures. SREs must actively fight complexity by designing simple systems, reducing accidental complexity, and embracing "boring" technology that is well-understood over novel but fragile solutions.
Essential vs. Accidental Complexity
| Type | Definition | Example |
|---|---|---|
| Essential complexity | Complexity inherent to the problem the system solves | A distributed consensus algorithm (Paxos/Raft) is inherently complex |
| Accidental complexity | Complexity introduced by the implementation choices | Using five different caching layers when one would do |
The Virtues of "Boring" Technology
┌─────────────────────────────────────────────┐
│ Reliable technology = boring technology │
├─────────────────────────────────────────────┤
│ "Boring" means: │
│ ✅ Well-understood by the team │
│ ✅ Extensively deployed in production │
│ ✅ Well-documented failure modes │
│ ✅ Mature tooling and monitoring │
│ ✅ Simple to debug │
│ │
│ Risky choices: │
│ ❌ Cutting-edge but untested │
│ ❌ "Shiny" new framework │
│ ❌ Over-abstracted for the problem │
│ ❌ More than one way to do everything │
│ ❌ Dependencies on unvetted open-source │
└─────────────────────────────────────────────┘
The "Minimal API" Principle
Good API design for reliability:
- Few endpoints (narrow surface area)
- Idempotent operations (safe to retry)
- Fail-fast on invalid input
- Backward compatible by default
- Explicit versioning
Don't design for every future use case.
Design for today's known use cases.
Tomorrow's can add surfaces.
Fighting Complexity: Practical Tactics
| Tactic | Description |
|---|---|
| Code reviews with a simplicity lens | Reviewers should ask "Is this simpler than it needs to be?" |
| Prune dead code | Delete features, flags, and configs no longer in use |
| Standardize | Use one build system, one deployment tool, one monitoring stack — not five |
| Reduce dependencies | Every dependency is a point of failure. Audit them regularly |
| Prefer stateless | Stateless services are simpler to scale, deploy, and debug |
| Use feature flags | Keep code paths explicit and reversible |
The "Negative Lines of Code" Metric
"Software entropy" is real:
- Every feature adds complexity
- Every bug fix adds branches
- Every workaround adds technical debt
Reliability practice:
- Actively remove code that is no longer needed
- Celebrate "negative lines of code" in reviews
- Simpler code has fewer bugs
Actionable Takeaways
- Simplicity is a design goal, not an accident. Explicitly allocate time to simplify existing systems.
- Resist the temptation to over-engineer. Use the simplest thing that works for your current scale.
- Standardize ruthlessly. If you have two tools that do the same thing, deprecate one.
- Say "no" to features. More features = more complexity = less reliability.
- Document complexity. When you can't avoid it, document why — and what would need to change to simplify it.
Important Quotes
"Simplicity is a prerequisite for reliability."
"The best code is code that doesn't exist. The second-best code is code that is simple enough that everyone can understand it."
"Boring technology is reliable technology. If it's boring, it's been around long enough that we know how it fails."
"Complexity is like a gas — it expands to fill whatever container you give it. You must actively fight it."
"Every time you add a new feature, you are making a bet: that the value of this feature outweighs the cost of the complexity it adds. Make sure you're winning that bet."
Part III: Practices
Chapter 10: Practical Alerting
Core Thesis
Alerting is the primary interface between SRE and the production system — but it's often the most broken part of the monitoring stack. This chapter explains Google's approach to designing alerting systems that produce few, high-signal pages rather than constant noise.
The Alerting Design Workflow
┌─────────────────────────────────────────────────────┐
│ Step 1: Understand the service's SLOs │
│ Step 2: Identify what would violate those SLOs │
│ Step 3: Design alerts that detect SLO violation │
│ risk, not symptoms │
│ Step 4: Build runbooks for each alert │
│ Step 5: Test each alert (at least quarterly) │
│ Step 6: Delete alerts that never fire │
│ Step 7: Review and tune alert thresholds │
└─────────────────────────────────────────────────────┘
Types of Alerts
| Alert Type | When It Fires | Action | Target Latency |
|---|---|---|---|
| Page (Urgent) | Error budget burning faster than expected | Wake up SRE and resolve | < 5 minutes |
| Ticket (Non-urgent) | Future risk detected (disk filling, cert expiring) | Create ticket, handle in business hours | < 24 hours |
| Info / Log | Something notable but not actionable | Log for postmortem / analysis | None |
Alert Quality Metrics
| Metric | Description | Target |
|---|---|---|
| Precision | % of pages that lead to a real action | > 90% |
| Recall | % of real incidents captured by an alert | > 95% |
| MTTA | Mean time to acknowledge | < 5 min |
| MTTR | Mean time to resolve | Service-specific |
| False positive rate | Pages that required no action | < 10% |
| Alert fatigue index | Pages per shift per SRE | < 2 |
Alerting Anti-Patterns
| Anti-Pattern | Description | Fix |
|---|---|---|
| The Kitchen Sink | Alert on everything because you might miss something | Only alert on SLO violations |
| The Boy Who Cried Wolf | High false positive rate → alerts ignored | Cut precision to zero; rebuild with strict thresholds |
| The Echo Chamber | Multiple alerts for the same problem | Dedup/group alerts by incident |
| The Spaghetti Graph | Alerts that require interpretation to understand | Write explicit alert text |
| The Zombie Alert | Alert that hasn't fired in 6+ months | Delete it. Add it back only if needed |
Sample Runbook Template
# Alert: High Error Rate (>5% 5xx responses, 5 min window)
## Symptoms
- Users report errors
- Elevated 5xx rate on monitoring dashboard
## Impact
- P0: Error budget depleting rapidly
- Estimated: X% of users affected
## Immediate Steps
1. Acknowledge the page.
2. Check the error dashboard: [link]
3. Identify common characteristics of failed requests:
- Which datacenter?
- Which endpoint?
- Which user cohort?
4. If single datacenter: drain and redirect traffic.
5. If global: check for recent deploy or config push.
6. Rollback the most recent change.
7. Monitor error rate for recovery (expect 5-10 min).
## Escalation
If unresolved after 15 minutes: escalate to {secondary on-call}.
If unresolved after 30 minutes: escalate to {team lead}.
## Postmortem
File a bug with:
- Duration of elevated error rate
- Root cause
- Monitoring gaps discovered
- Action items
Actionable Takeaways
- Every alert must have a runbook. If you need to figure out what to do, it's not a page — it's a multi-car pileup.
- Alert on symptoms, not causes. Alert on "user requests failing", not "database connection pool full."
- Use burn-rate alerts. Instead of "error rate > X", use "error budget consumed > Y% in Z minutes."
- Test your alerts. Set up synthetic failures quarterly to verify alerts fire correctly.
- Track your alert quality metrics. Bad alerting is worse than no alerting.
Important Quotes
"A good alert is one that fires rarely, tells you exactly what's wrong and what to do about it, and lets you go back to sleep."
"If your pager goes off and you don't know why, that's a systems failure, not a personal one."
"The goal of alerting is not to catch every possible problem. It's to catch the problems that require immediate human action."
"Every alert that doesn't result in action is training your team to ignore the pager."
Chapter 11: Being On-Call
Core Thesis
On-call is the most visible, most stressful responsibility of an SRE. This chapter describes Google's practices for structuring on-call rotations, managing the cognitive load, maintaining work-life balance, and creating a sustainable on-call culture.
The On-Call Rotation Model
Google's typical approach:
┌───────────┐ ┌───────────┐ ┌───────────┐
│ Primary │ │ Primary │ │ Primary │ ...
│ (Week 1) │ │ (Week 2) │ │ (Week 3) │
└───────────┘ └───────────┘ └───────────┘
│ │ │
└───────┬───────┘ │
│ │
┌───────▼───────┐ ┌───────▼───────┐
│ Secondary │ │ Secondary │
│ (Same week) │ │ (Same week) │
└───────────────┘ └───────────────┘
Key parameters:
- Shift length: 12-24 hours (day) or full week (24/7 rotation)
- Team size: minimum 6-8 for a 24/7 rotation
- Primary handles pages; secondary is escalation/backup
- No-shift is immediately followed by a rest period
On-Call Balance Rules
| Rule | Description |
|---|---|
| 1. Manage stress | On-call should not be a constant state of emergency. If it is, the system is broken |
| 2. Fair rotation | All team members share the on-call burden equally |
| 3. No retaliation | Pages caused by developer changes do not result in blame on the developer |
| 4. Time off | After a particularly bad on-call shift, a compensatory day off |
| 5. Escalation exists | No one should be expected to handle every incident alone |
| 6. Post-shift decompression | The day after on-call has reduced meeting load |
The "Operational Overload" Threshold
Warning signs that on-call is unsustainable:
[ ] Pages per shift > 5 (alerting is too noisy)
[ ] MTTA > 10 minutes (alerts are being ignored)
[ ] MTTR > 2x baseline (system is too complex to debug)
[ ] SREs have > 1 "bad" shift per month (frequent sleep disruption)
[ ] No runbooks for > 50% of alerts
[ ] On-call rotation smaller than 6 people
Incident Lifecycle During On-Call
┌─────────┐ ┌─────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Receive │→│ Triage │→│ Mitigate │→│ Resolve │→│ Follow │
│ Alert │ │(assess) │ │ (fix now) │ │(permafix)│ │ Up │
└─────────┘ └─────────┘ └──────────┘ └──────────┘ └──────────┘
Key distinctions:
- Mitigate ≠ Resolve
- Mitigation: stop the bleeding (rollback, redirect traffic)
- Resolution: fix the root cause (patch code, rewrite config)
- Post-incident: file bug, update runbook, write postmortem
Actionable Takeaways
- Ensure at least 6-8 people in the rotation. Fewer than 6 creates unsustainable load.
- Keep on-call shifts to 12 hours for daytime, or full week 24/7. Avoid "always on" perma-on-call.
- Create a culture of blamelessness during on-call. The pager fires because the system failed, not the person.
- Track on-call metrics (pages/shift, MTTA, MTTR, false positives) and review them weekly.
- Provide immediate decompression: The morning after a nighttime incident, the on-call SRE should be explicitly excused from morning meetings.
Important Quotes
"Being on-call is not about being permanently available — it's about being reliably available during a defined period."
"A sustainable on-call rotation is one where the team isn't burned out, where incidents are genuinely surprising, and where the postmortem culture leads to system improvements that reduce the on-call burden over time."
"If you're on-call and your pager goes off every night, the problem is not with you — it's with your system."
"The goal of on-call is not to prevent all incidents. The goal is to respond to them effectively and to make them less frequent over time."
Chapter 12: Effective Troubleshooting
Core Thesis
Troubleshooting is a structured discipline, not black magic. This chapter presents Google's systematic approach to debugging production issues: from hypothesis formulation to data collection to controlled experimentation.
The Troubleshooting Workflow
┌──────────────────────────────────────────────┐
│ ┌──────────────┐ │
│ │ Report Issue │ │
│ └──────┬───────┘ │
│ │ │
│ ┌──────▼───────┐ │
│ │ Triage │ │
│ │ (Severity, │ │
│ │ Impact) │ │
│ └──────┬───────┘ │
│ │ │
│ ┌──────▼───────┐ │
│ ┌──────►│ Examine │◄──────┐ │
│ │ │ (Data gather)│ │ │
│ │ └──────┬───────┘ │ │
│ │ │ │ │
│ │ ┌──────▼───────┐ │ │
│ │ │ Hypothesize │ │ │
│ │ └──────┬───────┘ │ │
│ │ │ │ │
│ │ ┌──────▼───────┐ │ │
│ │ │ Test / │ │ │
│ │ │ Experiment ├───────┘ │
│ │ └──────┬───────┘ (hypothesis │
│ │ │ disproven) │
│ │ ┌──────▼───────┐ │
│ │ │ Root Cause │ │
│ │ │ Identified │ │
│ │ └──────┬───────┘ │
│ │ │ │
│ │ ┌──────▼───────┐ │
│ │ │ Mitigate │ │
│ │ └──────┬───────┘ │
│ │ │ │
│ │ ┌──────▼───────┐ │
│ │ │ Resolve │ │
│ │ └──────┬───────┘ │
│ │ │ │
│ │ ┌──────▼───────┐ │
│ └───────┤ Follow Up │ │
│ │ (Bug, Post- │ │
│ │ mortem) │ │
│ └──────────────┘ │
└──────────────────────────────────────────────┘
Troubleshooting Principles
| Principle | Description |
|---|---|
| 1. Don't panic | The system is already broken. Rushing makes it worse |
| 2. Be systematic | Follow a hypothesis → test → refine loop, not random guessing |
| 3. Fix forward, not backward | Prefer rolling forward (fix + deploy) over rolling back when possible |
| 4. One change at a time | Changing multiple things simultaneously makes it impossible to know what fixed the problem |
| 5. Document as you go | Write down what you've tried, what you found, and what you're about to try |
| 6. Know when to escalate | If you're stuck for >15 minutes, loop in a second person |
Hypothesis-Driven Debugging
The scientific method applied to production:
1. Observe the symptom
→ "Users are getting 503 errors on the /search endpoint"
2. Formulate hypotheses (rank by likelihood)
→ H1: Recent deploy introduced a bug (HIGH — deploy 10 min ago)
→ H2: Database is overloaded (MEDIUM — seen before)
→ H3: Network configuration changed (LOW — no config pushes)
3. Design experiments to test each hypothesis
→ H1 test: Rollback the deploy. If errors drop, confirmed.
→ H2 test: Check database query latency dashboard.
→ H3 test: Compare current network config with last known good.
4. Execute experiments, starting with highest probability
5. Repeat until root cause is confirmed
6. Mitigate, then resolve, then document
Common Pitfalls in Troubleshooting
| Pitfall | Description | Prevention |
|---|---|---|
| Jumping to conclusions | Assuming you know the cause without evidence | Write down at least 3 hypotheses before acting |
| Confirmation bias | Interpreting ambiguous data to support your preferred hypothesis | Deliberately try to disprove each hypothesis |
| Changing too many things | Multiple changes at once → can't identify cause | "One change, measure, next change" discipline |
| Fixing symptoms, not causes | Restarting a process when the config is wrong | Always ask "what caused this to break?" |
| Not asking for help | Sinking 30+ minutes into a mystery | Set a timer. At 15 minutes, call for backup. |
Actionable Takeaways
- Use a structured approach (hypothesis → test → refine) for every incident. It prevents panic and cuts MTTR.
- Write a "war room" timeline as you go. It will be invaluable for the postmortem.
- Invest in good debugging tools. A 30-second
curlprobe is worth more than a 30-minute dashboard dive. - Practice incident response. Run tabletop exercises and fire drills quarterly.
- Post-incident, update your runbooks. Every "I didn't know that" is a gap that needs filling.
Important Quotes
"Troubleshooting is not about knowing the answer. It's about knowing how to find the answer."
"The first thing to do when a system breaks is to take a deep breath. Panic is not a debugging strategy."
"If you don't have a hypothesis, you're not troubleshooting — you're guessing."
"The best debugging tool is a recent change log."
"When you've identified the root cause, you haven't finished. You still need to fix it, monitor the fix, and ensure it doesn't happen again."
Chapter 15: Postmortem Culture: Learning from Failure
Core Thesis
A blameless postmortem culture is the single most important driver of long-term reliability improvement. Google treats every significant incident as an opportunity to learn — and the first rule is that the postmortem is blameless. The goal is to understand what the system did wrong, not who did wrong.
The Postmortem Philosophy
┌─────────────────────────────────────────────────────┐
│ DO: │
│ ✅ Ask "What can we learn from this?" │
│ ✅ Focus on system, process, and tool failures │
│ ✅ Assign action items to fix root causes │
│ ✅ Share postmortems widely across the organization │
│ ✅ Celebrate postmortems as learning opportunities │
│ │
│ DON'T: │
│ ❌ Ask "Whose fault was this?" │
│ ❌ Punish the person who triggered the incident │
│ ❌ Blame individuals in postmortem text │
│ ❌ Treat postmortems as disciplinary hearings │
│ ❌ Keep postmortems secret / internal-only │
└─────────────────────────────────────────────────────┘
The Anatomy of a Postmortem
A Google-style postmortem typically includes:
| Section | Description | Example Content |
|---|---|---|
| Title | Short description | "Excessive Error Rate on Search Backend — 2023-03-15" |
| Date and Duration | When the incident occurred and how long it lasted | 2023-03-15 14:23 UTC — 2023-03-15 15:47 UTC |
| Summary | 2-3 sentence high-level description | "A misconfigured load balancer caused 12% of search traffic to be routed to an under-provisioned pool..." |
| Impact | Quantified effect on users and systems | "~50K failed requests (0.02% of total), 84 min elevated latency" |
| Timeline | Chronological log of events | Detailed from first alert to resolution |
| Root Cause | The underlying system/process failure | "Config push for load balancer pool weights was applied without validation" |
| Trigger | What set the chain of events in motion | "SRE X deployed config change CL/12345 to the production load balancer" |
| Resolution | How the incident was mitigated and eventually fixed | "Rolled back config change, errors dropped to baseline within 5 minutes" |
| Detection | How the incident was discovered | "Burn-rate alert fired at p95 latency > 500ms" |
| Action Items | Concrete, prioritized fixes | "Add config validation for pool weight ranges (P0)", "Write runbook for load balancer recovery (P1)" |
| Lessons Learned | What the team learned from the incident | "Config validation is a single point of failure; all configs should be validated" |
| What Went Well | Positive aspects of the response | "Alerting caught the issue within 2 minutes", "Rollback was documented and fast" |
| What Went Wrong | Process/tool failures | "No runbook for this alert", "On-call had multiple simultaneous pages" |
| Where We Got Lucky | Things that could have been worse | "Incident occurred during business hours with senior engineers available" |
Sample Timeline Section
Timeline (all times in UTC):
14:23:17 Burn-rate alert fires: p95 latency exceeds 500ms for 2+ minutes.
14:23:45 Primary on-call acknowledges page.
14:24:30 Dashboard shows elevated error rate (12%) on search-backend-pool-b.
14:26:15 Check: recent deploy? Negative. Last deploy 3 hours ago.
14:27:00 Check: recent config push? Positive. CL/12345 deployed 15 minutes ago.
14:28:30 Hypothesis: config change is causing the issue.
14:29:00 Begin rollback of CL/12345.
14:32:15 Rollback complete. Error rate begins to decline.
14:35:00 Error rate returns to baseline (<0.1%).
14:35:30 Notify stakeholders: incident resolved.
14:36:00 File bug to investigate root cause of CL/12345 failure.
14:40:00 Begin postmortem draft.
Measuring Postmortem Effectiveness
| Metric | Target | Why It Matters |
|---|---|---|
| % of incidents with postmortems | 100% (all P0/P1 incidents) | No learning without documentation |
| Action item closure rate | > 90% within 30 days | Postmortems without follow-through are theater |
| Repeat incident rate | Decreasing YoY | The goal is to fix root causes permanently |
| Postmortem sharing frequency | Shared across org | Maximizes organizational learning |
| Time to postmortem | < 1 week after incident | Fresh details lead to better insights |
The "Blameless" Rule in Practice
Blameless ≠ letting people off the hook.
Blameless = acknowledging that humans will always make errors,
and that the solution is to make systems error-tolerant.
A blameless postmortem asks:
"What about the system made this failure possible?"
Not:
"Who made the mistake?"
Because if you fire the person who made a mistake,
you lose the expertise that person has, AND
the next person will make the same mistake
because the system hasn't been fixed.
Actionable Takeaways
- Write a postmortem for every significant incident (P0/P1). "Significant" = any incident that impacted users or exhausted error budget.
- Write within one week. Fresh details matter. Don't let them decay.
- Make postmortems blameless in both substance and tone. Words like "they failed to" or "X should have" have no place in a postmortem.
- Assign action items with owners and deadlines. Track them in a shared system (bug tracker, ticketing system).
- Share postmortems broadly. The whole organization learns from each incident. A culture of secrecy around failures guarantees they will be repeated.
- Celebrate good postmortems. The best teams hold postmortem reads and publicly thank the author.
Important Quotes
"A blameless postmortem is not just nice-to-have — it's the cornerstone of a learning organization."
"If your postmortem has the word 'should have' in it, you're doing it wrong. The system is what it is. Fix the system, not the person."
"The goal of a postmortem is not to assign blame. The goal is to understand what happened, why it happened, and how to prevent it from happening again."
"Postmortems are not about punishment — they are about progress."
"The most important action item from any postmortem is not the one that fixes the immediate bug. It's the one that changes the process so that this class of bug can never happen again."
"If your organization doesn't have a blameless postmortem culture, you don't have a learning culture — you have a blame culture. And blame cultures produce systems that break in exactly the same way, forever."
Appendix: Cross-Chapter Reference Table
| Concept | Primary Chapter(s) | Related Chapters |
|---|---|---|
| Error Budget | 3, 4 | 10, 11, 15 |
| SLO / SLI / SLA | 3, 4 | 1, 6, 10 |
| Toil | 5 | 1, 7, 11 |
| Automation | 7 | 5, 8, 9 |
| Monitoring | 6 | 10, 12 |
| Alerting | 10 | 6, 11, 12 |
| On-call | 11 | 10, 12, 15 |
| Release Engineering | 8 | 1, 7, 9 |
| Simplicity | 9 | 5, 7 |
| Postmortem | 15 | 11, 12 |
| Troubleshooting | 12 | 6, 10 |
Appendix: Key Formulas
Availability Calculation
Availability = (Successful Requests / Total Requests) × 100%
Example (99.9% SLO):
Monthly total requests: 10,000,000
Maximum allowed failures: 10,000 (0.1%)
Per-day budget: ~333 failures
Error Budget
Error Budget = 1 - SLO
Example:
SLO = 99.9% → Error Budget = 0.1%
Monthly budget = 0.1% of total requests
Burn Rate
Burn Rate = Error Budget Consumed / Time Elapsed
Example:
SLO = 99.9%, 30-day budget = 0.1%
If 1% of requests fail for 1 hour:
Budget consumed = 1% × (1/720) = ~0.0014%
Normalized burn rate = 0.0014% / (0.1%/day) = 1.4x
Toil Percentage
Toil % = (Hours Spent on Toil / Total SRE Hours) × 100
Target: ≤ 50%
- Project work ≥ 50%
- Operations ≤ 50%
End of reference document. Last updated: June 2026.