magnus919_agent-skills/release-engineering/templates/rollback-runbook.md

Rollback Runbook — [SERVICE / SYSTEM] v[VERSION]

One runbook per service or release. Fill in the placeholders, then rehearse it — an unrehearsed runbook is fiction. Keep this next to the deploy runbook. Last reviewed: [YYYY-MM-DD] by [name].

1. When This Runbook Applies

Field	Value
Service / system	[name]
Release(s) covered	[e.g., v2.4.0 and any patch on top of it]
Known-good artifact	[image:tag + sha256 digest]
Deploy mechanism	[Argo CD sync / pipeline deploy / manual steps]
Rollback decision authority	[name — on-call lead; decision time-boxed to X minutes]

2. Trigger and Detection Thresholds

Initiate rollback (or flag-off) when any of these holds for the stated window. Base thresholds on pre-release baselines, not guesses.

Signal	Threshold	Window	Tool / Alert
Error rate (HTTP 5xx)	[e.g., > 1.0%, or > 2× baseline]	[10 min]	[Datadog alert]
Latency p95	[e.g., > 300 ms, or > 1.5× baseline]	[15 min]	[Grafana]
Error budget burn	[e.g., > 2% of monthly budget consumed in 1 h]	[1 h]	[burn-rate alert]
Saturation / capacity	[e.g., CPU/memory > 85% on > 50% of instances]	[15 min]	[infra alert]
Data integrity	[e.g., migration verification query fails]	[immediately]	[migration log]
Business signal	[e.g., support-ticket spike about the new feature]	[1 h]	[ticketing]

Gotcha — compare canary vs. control populations, never "before vs. after" (time is a confound). If the defect is gated behind a feature flag, flip the flag off first — it is the fastest and least risky lever.

3. Impact Assessment

Question	Answer
Who is affected?	[users / segments / internal teams]
What is the blast radius?	[service, downstream dependencies, data, clients]
Severity	[SEV-1 / SEV-2 / SEV-3]
Is data at risk?	[yes/no — if yes, stop and involve the DB owner before acting]
Was a schema migration deployed with this release?	[yes/no — if yes, see section 5.3; rollback may be unsafe after finalization]
Is the change client-side (mobile/desktop/IoT)?	[yes/no — if yes, rollback is forward-only; use a kill switch / phased release]

4. Decision Matrix — Rollback vs. Roll-Forward vs. Flag-Off

Situation	Recommended action	Why
Defect is behind a feature flag	Flag off	Seconds, no redeploy, fully reversible, auditable
User-visible or severe defect, flag not involved	Artifact rollback	Returns to a known-good state that has run in production
Minor defect with a trivial, low-risk fix	Roll-forward (hotfix)	Faster than rollback if the fix is certain; still build + test + stage it
Destructive schema change already finalized	Roll-forward with a new migration	Code rollback is broken after finalization — never combine old code with a finalized schema
Data corruption / loss, no forward path	Backup / point-in-time restore (last resort)	Slow and lossy; governed by RPO/RTO — escalate first

Gotcha — git revert is not a rollback. It produces new code that must be rebuilt, retested, and redeployed, and it does not undo migrations, data changes, or flag state that shipped with the reverted commit.

5. Step-by-Step Rollback

5.1 Ordering (microservices)

• Roll consumers back before producers: undo the caller's use of the new behavior before removing the provider's capability.
• Assume any service may roll back one version (N-1 compatibility): never depend on a service that could roll back under you.
• With N-1 contracts in place, services roll back independently — no orchestration needed. Coordinated rollback across services is a design smell.

5.2 Stateless services / artifacts

1. [Announce in #incident / #releases: "Rolling back to — reason: ".]
2. [Re-point the deploy to the known-good artifact — e.g., kubectl set image deployment/<svc> <svc>=<registry>/<svc>:<good-tag>, or Argo CD sync to the previous tag, or pipeline "redeploy release ".]
3. [Enable connection draining / graceful termination so in-flight requests finish.]
4. [Warm caches before restoring full traffic to avoid a latency spike.]
5. [Confirm new pods healthy and traffic shifted.]
6. [Blue/green: rollback is a router change — cut traffic back to the blue environment, verify, then keep the bad green environment for inspection.]

5.3 Stateful services / databases

1. [Identify the migration phase: initial / transition / finalization. Never roll code back past a finalized schema.]
2. [If code rollback is safe (schema still supports the previous release): redeploy the previous binary; the database stays in the transition phase until a patch is released.]
3. [For feature removal: prefer a new forward migration (append-only, idempotent) over "un-applying" the old one.]
4. [If data is corrupted: escalate to the DB owner; plan backup / point-in-time restore with RPO [X min] and RTO [Y min]; get approval before restoring.]
5. [Manual checkpoint before any destructive step — pause and confirm with the on-call lead.]

5.4 Clients / devices (mobile, desktop, IoT)

1. [Rollback is not possible for shipped binaries — use a kill switch / remote config / feature flag to disable the broken behavior.]
2. [Mobile: pause a phased release, then publish the last stable build as a new version with a higher build number, re-signed and re-submitted.]
3. [IoT: rely on A/B (dual-bank) partitions + watchdog auto-revert; validate post-install before switching the active bank.]
4. [Document the version long-tail: some users will keep the bad version for days or indefinitely.]

6. Verification (Rollback Is Complete When ...)

SLI	Target after rollback	Check
Error rate	[back to baseline, e.g., < 0.5%]	[dashboard link]
Latency p95	[back to baseline, e.g., < 250 ms]	[dashboard link]
Error budget	[no longer burning]	[budget dashboard]
Version breakdown	[100% of traffic on the known-good version]	[version-labeled metrics]
Data integrity	[migration / consistency checks green]	[check output]

Gotcha — verify per-version metrics, not aggregate: subtle failures (e.g., errors only for a subset of users) surface only when most instances run the bad version.

7. Communication Plan

Audience	Channel	Message	When
Internal (eng + on-call)	#incident	Decision + observed signal	Immediately
Support	#support	User-facing impact + ETA	Within [15] min
Customers / status page	status page	Outage / degradation notice	Within [30] min
Post-incident	#postmortems	Rollback changelist + timeline	After resolution

8. Post-Rollback Activities

• Quarantine the bad artifact: label/deny it so it cannot be re-promoted (e.g., remove the tag, blocklist the digest).
• Record the rollback changelist describing the observed problem (see change-governance-record.md).
• Open a blameless postmortem within [2] business days; rollbacks are normal, not a failure of the team.
• Fix the pipeline / thresholds that let the defect through (CI gap, missing canary stage, wrong threshold).
• Update this runbook and the readiness checklist with lessons learned.

9. Rehearsal Log

Rehearse "just because" every few weeks — find traps (incompatible versions, broken automation) while the release is healthy. If rehearsal breaks, roll forward and fix the cause.

Date	Rehearsed by	Scenario	Result	Traps found	Follow-up
[YYYY-MM-DD]	[name]	[e.g., canary error-rate spike]	Pass / Fail	[none / description]	[ticket]
[YYYY-MM-DD]	[name]	[e.g., migration rollback window]	Pass / Fail	[none / description]	[ticket]