magnus919_agent-skills/site-reliability-engineering/references/toil-elimination.md

Toil Elimination Reference

Overview

Toil is the enemy of reliable systems and the single largest drain on Site Reliability Engineering productivity. This reference provides a comprehensive framework for identifying, measuring, prioritizing, and eliminating toil from operational workflows. Rooted in Google's SRE practices as documented in the Site Reliability Engineering books, this guide is designed for SRE agents and teams who need a structured approach to reducing operational burden.

---

1. What Is Toil?

Toil is a specific category of operational work that is fundamentally different from engineering work. It is not merely "hard work" or "a lot of work" — it is work that exhibits five defining characteristics.

The Five Characteristics of Toil

1. Manual The work requires human intervention. There is no automated process handling it. A human must SSH into a machine, click a button, run a command, or manually inspect output. If a system could handle the task without human involvement but currently doesn't, that work is toil.

2. Repetitive The work is performed repeatedly. You handle the same kind of ticket, run the same command, or follow the same runbook more than once. A one-off task is not toil, even if it is manual and tedious. Repetition is the key differentiator — if you've done it three times, you will likely do it a hundred more.

3. Automatable The work could be automated by engineering effort. If the task requires genuine human judgment that cannot be encoded in logic, it may not be toil — it may be craft. But if a program or script could perform the action with the same or better result, the work qualifies as automatable and therefore as toil.

4. Tactical The work is reactive and interrupt-driven rather than strategic. It comes in as alerts, tickets, or pages and demands immediate attention. Tactical work displaces proactive engineering improvements because it consumes the time and cognitive energy needed for design and development.

5. No Enduring Value The work does not make your systems permanently better. Once completed, the service is simply back to its previous state. Nothing was improved, no new capability was added, no debt was paid down. The value of the work decays to zero as soon as the immediate issue is resolved.

6. Scales Linearly (The Multiplier Problem) As the service grows, the amount of toil grows proportionally. If you have 100 servers and each requires 10 minutes of manual patching per month, going to 1,000 servers means 100 minutes. Toil does not benefit from economies of scale — it tracks directly with infrastructure growth. Engineering work, by contrast, produces leverage: one automation script can handle 10 servers or 10,000 with negligible additional cost.

What Toil Is NOT

• Engineering work: Designing, building, testing, and deploying new systems or features.
• Craft: Work requiring deep expertise, judgment, and non-repeatable problem-solving.
• One-off efforts: Even if painful, a task that will never repeat is not toil (though it may still warrant postmortem-driven changes).
• On-call response: Incident response is essential operational work. However, excessive on-call toil (page storms, false alarms, manual remediation steps) is a symptom of toil in the monitoring and alerting pipeline.

---

2. The 50% Engineering Time Mandate

The Principle

Google SRE established a foundational rule: an SRE team should spend no more than 50% of its time on operational work (toil + on-call overhead). The remaining 50% must be reserved for engineering projects that reduce toil, improve reliability, or add service capacity.

Why 50%?

Reason	Explanation
Sustainability	Without the 50% cap, toil expands to fill all available time. Teams burn out, attrition spikes, and the service enters a doom loop of increasing operational burden.
Self-Improvement	The engineering half is what makes tomorrow easier than today. Without it, the team can never escape the toil trap.
Reliability	Systems that never receive engineering investment become fragile piles of manual procedures. The more toil a team handles, the less reliable the service becomes, because improvements are not being made.
Career Health	Engineers who spend 80-100% of their time on toil stagnate. They learn nothing new, build nothing durable, and leave. The 50% mandate is a career preservation mechanism.

How to Measure It

Track engineering time using these categories:

Category	Description	Counts Toward
Pure Toil	Manual, repetitive, automatable tasks	Operational (50% bucket)
On-Call Overhead	Incident response, pages, alert triage	Operational (50% bucket)
Engineering Projects	Building automation, tools, platforms, design	Engineering (50% bucket)
Code Reviews & Design	Peer review, architecture discussions, documentation of systems	Engineering (50% bucket)
Meetings & Admin	Stand-ups, planning, 1:1s, HR tasks	Neither (overhead; minimize)
Learning & Growth	Reading, training, conferences, spike experiments	Engineering (50% bucket)

The Breach Response

If operational work exceeds 50% for more than one quarter, the team must:

1. Freeze all new feature work and feature onboarding.
2. Dedicate the entire engineering half to toil reduction projects.
3. Escalate to management that the team is in "toil emergency" mode.
4. Re-evaluate the SLA/SLO — if the team cannot sustain the current reliability target without exceeding 50% toil, the SLO is likely wrong.

---

3. Toil Assessment Methodology

Step 1: Tag and Track

Every task an SRE performs must be tagged with a toil classification. Use the following rubric to score each task on a 1-5 scale across each of the five characteristics.

Toil Assessment Rubric

Characteristic	1 (Not Toil)	2 (Low)	3 (Moderate)	4 (High)	5 (Pure Toil)
Manual	Fully automated, no human touch	One-click approval needed	CLI command with parameters	SSH + multi-step runbook	Physical data center visit
Repetitive	Never done before	Once per quarter	Once per week	Daily	Multiple times per day
Automatable	Requires human judgment (design work)	Would need ML to automate	Could be automated with moderate effort	Trivially scriptable	Already partially automated, needs finishing
Tactical	Proactive, planned project	Scheduled maintenance	On-call ticket during business hours	Page at 2 AM	Interrupts ongoing engineering work
Enduring Value	Permanently improves the system	Adds monitoring or documentation	Fixes a bug permanently	Restores service to previous state (only)	Actively makes the system worse (workaround debt)
Scales Linearly	Cost decreases per-unit with scale	Flat cost regardless of scale	Sub-linear growth	Linear growth	Super-linear growth (more users = more work per user)

Scoring: Sum the six scores. A task scoring 18+ is overwhelmingly toil. A task scoring 6-10 may be legitimate operational work. A task scoring 11-17 is in the grey zone — investigate whether it can be automated or eliminated.

Step 2: Time Sampling

For one sprint (two weeks) every quarter, have every team member log their time against a toil taxonomy. Use the following categories:

Toil Category	Examples
Incident Response	Pages, triage, mitigation, postmortem
Ticket Handling	User requests, access grants, quota increases
Release Engineering	Manual deployments, rollbacks, cherry-picks
Monitoring & Alerting	Tuning thresholds, silencing noise, investigating false positives
Capacity Planning	Manual scaling, provisioning, resource juggling
Data Operations	Manual backups, restores, data migrations, log analysis
Access & Permissions	Granting access, rotating credentials, audit responses
Incident Follow-Up	Manual remediation of known issues, applying known fixes

Step 3: Calculate the Toil Ratio

Toil Ratio = (Hours of toil) / (Total engineering hours) × 100

Target: < 25% toil ratio (leaving room for on-call overhead within the 50% operational cap).

Warning threshold: > 35% toil ratio. Immediate intervention needed.

Step 4: Trend Analysis

Track toil ratio over time. A decreasing trend confirms that toil reduction investments are paying off. An increasing trend means either:

• The service is growing faster than automation efforts
• New sources of toil have been introduced (new features, new integrations)
• Engineering time is being consumed elsewhere (meetings, bureaucratic overhead)

---

4. Automation Decision Tree

Not all toil should be automated. Some toil should be eliminated by redesign. Some should be accepted and budgeted for. Use this decision framework to determine the right response.

Automation Decision Criteria Table

Criteria	Automate Now	Automate Later	Do Not Automate
Frequency	Daily or more	Weekly to monthly	Quarterly or less
Time per occurrence	> 15 minutes	5-15 minutes	< 5 minutes
Error cost	Human errors cause incidents	Human errors cause minor issues	Errors are trivially recoverable
Stability of process	Process is well-understood and stable	Process changes < 2x/year	Process changes frequently or is poorly understood
Number of practitioners	> 3 people do this task	2-3 people do this task	1 person or specialist only
Business impact of delay	Delays cause SLA breaches	Delays cause customer complaints	No time sensitivity
Automation complexity	Can be done in < 2 weeks of effort	2-8 weeks of effort	> 8 weeks of effort or dependency on other teams
Failure mode of automation	Safe failure (script errors are recoverable)	Moderate risk	Automation failures could cause data loss or outages

Decision Tree Logic

Is the task toil? (scores 18+ on rubric)
    |
    +-- No --> Is it valuable engineering work?
    |           +-- Yes --> Keep doing it, document it
    |           +-- No  --> Eliminate or delegate it
    |
    +-- Yes --> Can we eliminate the need entirely?
                |
                +-- Yes --> Redesign the system to make the task irrelevant
                |
                +-- No --> Does it pass the "Automate Now" criteria?
                            |
                            +-- Yes (4+ criteria) --> Build automation
                            |
                            +-- No (0-3 criteria) --> Does it pass "Automate Later"?
                            |                           |
                            |                           +-- Yes --> Track in toil backlog, revisit quarterly
                            |                           |
                            |                           +-- No --> Accept as budgeted toil
                            |
                            +-- Mixed --> Apply cost-benefit analysis:

                    Cost = automation effort (hours)
                    Benefit = time saved per occurrence × frequency × 12 months

                    If Benefit >= 3 × Cost: automate
                    If Benefit < Cost: accept as budgeted toil
                    Otherwise: schedule for next planning cycle

Common Automation Traps

Trap	Why It's Dangerous	Better Approach
Automating a bad process	You just do the wrong thing faster	Fix the process first, then automate
Gold-plating	Building "enterprise-grade" automation for a task that happens twice	Use a simple script; upgrade only if frequency increases
Automating instability	Building automation on top of an unstable system	Stabilize the system first, then automate
The half-automation	Automating 80% but leaving 20% manual	The 20% still requires a human in the loop, defeating much of the benefit
Automating before measuring	You don't know how much time you'll actually save	Measure baseline toil time before and after automation

---

5. Categories of Automation

Toil reduction automation falls into four tiers. Each tier represents increasing leverage and decreasing operational burden.

Tier 1: Script-Based Automation

Single-purpose scripts that replace manual CLI commands or runbook steps.

Example	Tooling	Effort	Impact
Log rotation script	Bash, Python, Go	Hours	Saves minutes per server per week
User access grant script	Python + LDAP library	Days	Saves 15 minutes per ticket
Database index rebuild	SQL script + cron	Hours	Eliminates manual DBA tasks

Best for: High-frequency, well-defined, low-risk tasks. These are the "low-hanging fruit" of toil elimination.

Tier 2: Zero-Touch Automation

Fully automated workflows that require no human initiation or approval. The system detects a condition and acts on it automatically.

Example	Trigger	Action
Auto-scaling	CPU/memory threshold breach	Provision or de-provision instances
Auto-remediation	Specific error pattern in logs	Restart service, clear cache, rotate log
Automated certificate renewal	TTL approaching expiration	Request, validate, and install new certificate
Disk cleanup	Disk usage > 85%	Archive old logs, prune unused Docker images

Best for: Well-understood failure modes with safe, reversible automated responses. Always include circuit breakers and kill switches.

Tier 3: Self-Service Platforms

Web-based or API-driven interfaces that allow stakeholders (developers, product teams, internal users) to perform tasks themselves without SRE involvement.

Example	Users	Toil Eliminated
Self-service deployment portal	Developers	"Can you deploy my branch?" requests
Self-service access dashboard	All employees	Access grant tickets
Service catalog / internal developer portal	All teams	Provisioning requests, environment requests
Self-service data export	Analysts, PMs	"Can you run this query for me?" requests
Incident dashboard	On-call engineers	Manual log digging, status check spreadsheets

Best for: Tasks that are inherently manual because they require context from the requester. The platform shifts the burden from SRE to the requester, who has the context anyway.

Tier 4: Design Elimination

The most powerful form of toil reduction: redesigning the system so the toil cannot exist. Rather than automating a bad process, you eliminate the process entirely.

Example	Before	After
Immutable infrastructure	SSH into servers to patch and debug	Tear down and rebuild from golden images
Blue-green deployments	Manual rollback procedures	Traffic switch: flip a DNS record
Database sharding	Manual data migration scripts	Auto-sharding built into the data layer
Config-driven systems	Per-environment config changes in code	Centralized config service with RBAC
Stateless services	Session state management, drain procedures	Any instance can serve any request; no state to manage

Best for: Systemic sources of toil that cannot be effectively scripted away. Requires architectural investment but pays the highest long-term dividend.

---

6. The Toil-to-Engineering Progression

Toil reduction is not a one-time project — it is a continuous progression. Teams move through distinct phases as they mature.

Phase	Characteristics	Toil Ratio	Key Action
1. Survival	Everything is manual. Pages happen constantly. No automation exists.	80-100%	Measure and triage. Automate the top 3 sources of pages.
2. Awareness	Some scripts exist, but they are fragile and unowned. Still mostly reactive.	60-80%	Build a toil backlog. Assign owners for each automation project.
3. Foundation	Core automation exists. Runbooks are documented. Alerting is being tuned.	40-60%	Implement the 50% engineering time mandate. Formalize toil tracking.
4. Leverage	Self-service platforms emerge. Auto-remediation handles common incidents.	25-40%	Shift focus to design elimination. Build internal developer platform.
5. Optimization	Most operations are zero-touch. SRE time is predominantly engineering.	10-25%	Tune remaining automation. Focus on reliability engineering and feature work.
6. Autopilot	Systems are largely self-healing. SREs write code and design systems full-time.	< 10%	Innovation mode. SRE is a product engineering function with reliability expertise.

The Trap at Each Phase:

• Survival → Awareness: Teams can get stuck in survival mode if management does not grant engineering time. The 50% mandate must be enforced from the top.
• Awareness → Foundation: Teams build a dozen fragile scripts that no one maintains. Script rot becomes a new source of toil. Centralize and standardize.
• Foundation → Leverage: Teams automate everything except the things users ask for. The "last mile" of user requests remains a huge toil sink. Build self-service.
• Leverage → Optimization: Auto-remediation can mask underlying instability. Monitor for "silent failures" where automation hides a worsening system.

---

7. Common Sources of Toil (Ranked by Frequency)

Based on published SRE experience across multiple large-scale organizations, the following are the most common sources of toil, ranked by how frequently they appear in team assessments.

Rank	Source	Typical Toil Ratio Contribution	Primary Automation Tier
1	Alert noise / false positives	15-25%	Tier 2 (auto-remediation) + redesign alerting
2	User / stakeholder requests (access, deployments, queries, config changes)	15-20%	Tier 3 (self-service platforms)
3	Manual release / deployment processes	10-15%	Tier 1 (script) + Tier 4 (CI/CD pipelines)
4	Incident response for known issues	10-15%	Tier 2 (auto-remediation) + Tier 4 (fix root cause)
5	Monitoring and observability maintenance (dashboards, log parsing, threshold tuning)	8-12%	Tier 1 (script) + Tier 3 (self-service dashboards)
6	Capacity management (manual scaling, resource provisioning)	5-10%	Tier 2 (auto-scaling) + Tier 4 (design elimination)
7	Data operations (manual backups, restores, ETL, data migrations)	5-10%	Tier 1 (script) + Tier 4 (architectural changes)
8	Access and permission management	5-8%	Tier 3 (self-service) + Tier 4 (PAM integration)
9	Configuration management (manual config changes across environments)	5-8%	Tier 3 (self-service) + Tier 4 (GitOps)
10	Documentation maintenance	3-5%	Tier 3 (self-service runbooks) + culture shift
11	Postmortem and incident follow-up busywork	2-5%	Tier 1 (templating) + culture shift
12	Compliance and audit evidence gathering	2-5%	Tier 1 (script) + Tier 3 (self-service audit trails)

The Pareto Principle Applied to Toil

In most organizations, 20% of toil sources generate 80% of the operational burden. The top three sources (alert noise, user requests, manual deployments) typically account for 40-60% of all toil. Focusing automation efforts on these three categories first yields the highest return on engineering investment.

---

8. Toil Budget Tracking

The Toil Budget Concept

Treat toil like a financial budget. Each team has a quarterly toil allowance (ideally < 25% of total engineering hours). Once the budget is consumed, no additional toil work is accepted — it must be automated, eliminated, or declined.

How to Implement

Step 1: Set the Budget

• Target toil ratio: 25% (or less for mature teams)
• Hard cap: 35% (if exceeded, trigger a "toil freeze" — all non-critical work stops until toil is back under budget)

Step 2: Track Consumption

• Use a ticketing system where every task is tagged as toil, engineering, or overhead
• Weekly dashboard showing:
  • Current toil ratio
  • Toil remaining in budget (hours)
  • Projected date of budget exhaustion
  • Top 5 toil sources this week
• Monthly report to management with trend data

Step 3: Budget Replenishment (Engineering Investment)

• Every hour of toil eliminated frees up budget for the next quarter
• Track "toil debt" — the accumulated cost of not automating a task

Metric	Formula	Target
Toil Ratio	Toil hours / Total hours	< 25%
Toil Debt	Sum of (time spent on automatable task × frequency) over past 12 months	Should be decreasing
Automation ROI	(Time saved per month × 12) / Engineering hours to build	> 3x
Toil Velocity	Change in toil ratio month-over-month	Negative (decreasing)
Self-Service Adoption	% of user requests handled by self-service vs. tickets	> 80%

Sample Toil Budget Card

Item	Q1	Q2	Q3	Q4
Toil Budget (hours)	200	200	200	200
Toil Consumed	250	220	180	150
Over/Under Budget	-50	-20	+20	+50
Toil Ratio	31%	27%	22%	19%
Status	BREACH	WARNING	ON TRACK	ON TRACK

Automated Toil Tracking

Implement a bot or script that:

1. Scans all closed tickets weekly
2. Classifies them by toil/engineering using keyword matching + manual overrides
3. Posts a summary to the team's chat channel
4. Alerts when the budget is approaching exhaustion (at 75%, 90%, and 100% consumption)

---

9. How to Prioritize Toil Reduction

Not all toil elimination projects are equal. Use the following prioritization framework to decide what to tackle first.

The Toil Reduction Priority Matrix

	High Frequency	Low Frequency
High Impact	CRITICAL — Automate immediately	MODERATE — Schedule within next quarter
Low Impact	MODERATE — Quick wins, script in hours	LOW — Accept as budgeted toil or defer

• Frequency: How often does this task occur?
• Impact: How much total engineering time is consumed? Include both direct task time and context-switching cost.

Prioritization Factors (Weighted)

Factor	Weight	Description
Time Saved	40%	Total engineering hours recovered per month
Error Potential	25%	How often does this task cause incidents or require rework?
Morale Impact	15%	How much do team members hate this task? (Survey or vote)
External Dependency	10%	Does automation require another team to change? (Lower = better)
Visibility	10%	Will reducing this toil be visible to stakeholders? (Helps justify engineering time)

Scoring Method

For each toil source, calculate:

Priority Score = (Time Saved × 0.4) + (Error Potential × 0.25) + (Morale × 0.15) + (Dependency × 0.10) + (Visibility × 0.10)

Each factor scored 1-5 (1 = low, 5 = high). Sort by Priority Score descending. Tackle the top items first.

The "Quick Win" Rule

Any toil item that can be automated in one day or less should be automated immediately, regardless of priority score. These quick wins build momentum and demonstrate the value of toil reduction to the team and management.

Quarterly Toil Review Cadence

Week	Activity
Week 1	Review toil metrics from previous quarter. Update toil budget.
Week 2	Run a team toil estimation session. Identify new toil sources.
Week 3	Prioritize toil reduction projects for the quarter using the matrix.
Week 4+	Execute. One day per sprint dedicated to toil reduction (hack day format).

---

10. Organizational Consequences of Excessive Toil

Excessive toil is not just an operational problem — it is an organizational poison that erodes every aspect of team health.

Career Stagnation

Consequence	Mechanism
No portfolio growth	Engineers who spend 80%+ of their time on toil have no projects to show. Their resume stagnates.
Skill atrophy	Toil does not teach modern engineering practices, architecture, or design. Skills degrade over time.
No mentorship	Senior engineers buried in toil have no bandwidth to mentor juniors. The team's knowledge transfer pipeline breaks.
Promotion dead-end	Most organizations require demonstrated project impact for promotion. Toil produces no project artifacts.

Signs to watch for: Engineers who have been with the company 2+ years but cannot point to a single system they built or improved.

Low Morale

Consequence	Mechanism
Learned helplessness	"Why bother automating? It'll just be torn down next quarter." Chronic toil breeds fatalism.
Loss of ownership	When every day is reactive firefighting, engineers stop feeling ownership of the systems they run.
Boredom	Toil is intellectually unfulfilling. Talented engineers need challenge and creativity.
resentment	"I'm just a ticket monkey" — engineers resent being used as human automation.

Signs to watch for: Low participation in engineering discussions, disengagement in sprint planning, "whatever" responses to architecture questions.

Attrition

Consequence	Mechanism
Talent flight	The best engineers leave first. They have the most options and the lowest tolerance for toil.
Brain drain	When senior engineers leave, their undocumented tribal knowledge goes with them. Toil increases for remaining team members.
Hiring difficulties	Word spreads: "Don't join Team X — it's a firefighting nightmare." Recruitment becomes impossible.
Replacement cost	Each departed SRE costs 6-9 months of salary in recruiting, onboarding, and productivity loss.

Signs to watch for: Voluntary turnover rate > 20% per year, multiple departures to the same competitor, roles open for 6+ months.

The Toil Death Spiral

High toil → Low engineering investment → Poor system reliability
    ↑                                            ↓
More incidents → More manual remediation → More on-call burden
    ↑                                            ↓
Engineers leave ← Low morale ← No improvement projects ←

Breaking the Spiral

The spiral can only be broken by an intentional, enforced decision to prioritize engineering time over operational work, even if that means temporarily accepting lower availability or slower response times.

Intervention	Timeframe	Impact
Enforce 50% engineering time mandate	1 quarter	Immediate protection of engineering time
Toil freeze	1 sprint	Stop all work except P0 incidents and critical toil
Automation sprint	1-2 sprints	Build automation for top 3 toil sources
Self-service platform investment	1-2 quarters	Reduce user-request toil
System redesign (design elimination)	2-4 quarters	Eliminate entire categories of toil

Measuring Organizational Health

Metric	Healthy	Warning	Critical
Toil Ratio	< 25%	25-35%	> 35%
Voluntary Turnover	< 10%/year	10-20%/year	> 20%/year
Engineering Project Completion	80%+ on time	50-80% on time	< 50% on time
On-Call Satisfaction (survey)	> 4/5	3-4/5	< 3/5
Time to Onboard New SREs	< 3 months productive	3-6 months	> 6 months
Number of Active Automation Projects	5+	2-4	0-1

---

11. Practical Anti-Patterns and Pitfalls

The "All Toil Is Bad" Fallacy

Some toil is acceptable and even necessary. A small amount of operational work keeps engineers connected to the reality of their systems. The goal is not zero toil, but managed toil — toil that is within budget, acknowledged, and stable or decreasing.

The "Automate Everything" Trap

Automation is a tool, not a religion. Some things should not be automated:

• Tasks that change too frequently
• Tasks where the cost of automation failure is catastrophic
• Tasks that serve as a forcing function for system improvement (if automating a bad process, fix the process first)

The "Set and Forget" Myth

Automation is not fire-and-forget. Every automated system requires:

• Monitoring (is the automation still working?)
• Maintenance (does it still match the current system state?)
• Circuit breakers (what happens when it fails?)
• Documentation (what does it do and how do we override it?)

Treat automation as a product, not a one-time script.

The "Toil Tax" Trap

Teams that successfully reduce toil are often rewarded with more work — new services, more users, additional responsibilities. Without a formal toil budget that scales with responsibility, the team is punished for efficiency. Ensure that toil reduction gains are protected and reinvested, not consumed by scope growth.

---

12. Quick Reference: Toil Elimination Workflow

1. Identify: Tag every task with the toil rubric. Identify tasks scoring 18+.
2. Quantify: Measure time spent on each toil source. Calculate toil ratio.
3. Categorize: Is this script-based, zero-touch, self-service, or design elimination?
4. Decide: Use the automation decision tree. Should we automate, eliminate, or accept?
5. Prioritize: Score using the priority matrix. Quick wins first.
6. Build: One engineering day per sprint dedicated to toil reduction.
7. Measure: Recalculate toil ratio. Is it going down?
8. Defend: Protect engineering time. Escalate if toil ratio exceeds 35%.
9. Repeat: Quarterly toil review. Never stop.

---

References and Further Reading

• Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (2016). Site Reliability Engineering: How Google Runs Production Systems. O'Reilly Media. — Chapter 6: "Eliminating Toil"
• Beyer, B., Murphy, N. R., Rensin, D. K., Kawahara, K., & Thorne, S. (2018). The Site Reliability Workbook: Practical Ways to Implement SRE. O'Reilly Media. — Chapter 5: "Toil Assessment"
• Google SRE Team. (2023). SRE Fundamentals. Google Cloud Training.
• Jones, C. (2020). Toil: The SRE Killer. USENIX SREcon.
• Adkins, H., Beyer, B., Blankinship, P., Lewandowski, P., Oprea, A., & Stubblefield, A. (2020). Building Secure and Reliable Systems. O'Reilly Media.
• Niall Murphy, et al. (2022). Toil Reduction at Scale: Case Studies from Large Production Environments. ACM Queue, 20(3).