magnus919_agent-skills/release-engineering/references/release-process-models.md

Release Process Models

How a team branches, integrates, and cuts releases is the first structural decision in release engineering. The model you choose determines merge complexity, how defects flow toward production, whether a bad change can be isolated, and — per DORA's research — how fast and how stable your delivery actually is. The models below are not a menu of equal options: the evidence and the practitioner consensus point hard at trunk-based development for anything cloud-hosted, with the others surviving where constraints (packaged software, regulation, hardware) genuinely demand them.

Trunk-Based Development — the Modern Default

Trunk-based development (TBD) means all developers collaborate on one long-lived branch (main/trunk). Feature branches are short-lived — ideally less than a day and holding a single developer's work — and exist only to carry a code review through CI before merging back. No long-lived feature or release branches are used for artifact creation. Incomplete work is hidden behind feature flags or branch-by-abstraction, never parked on a branch.

Why this matters (and why DORA evidence favors it):

Effect	Mechanism
Small batches	Merging to trunk daily forces small, reviewable increments; DORA correlates small batches with higher performance and lower failure impact
No merge hell	Conflicts are resolved continuously instead of accumulated for a big merge
One source of truth	Every deploy candidate comes from the same integrated line; no "is this fix on the branch?" archaeology
Fast feedback	Every change is integration-tested the moment it lands, not weeks later

TBD is the default recommendation for cloud/SaaS teams that can automate build, test, and deploy, and that can use flags or branch-by-abstraction to decouple deploy from release. If you cannot merge to trunk at least daily, you are not really doing TBD — you are doing batched integration with a trunk-shaped label.

Merge Queues

Under high PR volume, keeping main green is itself a bottleneck. Merge queues (GitHub merge queue, Graphite, Trunk.io) dynamically group ready PRs, run CI against the combined set, and only merge when the group passes — replacing fragile "rebase-and-retest" manual rituals. GitHub's own data is the reference point: its merge queue ships 2,500+ PRs/month into its monorepo from 500+ engineers, and reduced average wait time by ~33% after launch. A merge queue is an enabler of TBD, not a substitute for it: it keeps the trunk green while preserving small, frequent merges.

Branch Protection and Review Workflow

The branching model is only as good as its enforcement layer — the branch-protection rules that make the model structural rather than aspirational:

Rule	Protects against	Applies to
Required status checks (CI must pass)	Merging broken code	All merges to main/trunk
Required PR review (peer review)	Untested, unreviewed changes	All merges to main/trunk
No direct pushes to protected branches	Bypassing the model	main/trunk, release branches
Restrict pushes to release branches	Unauthorized hotfixes	release/* during stabilization
Merge queue on protected branches	CI races and red main	High-volume repos

Branch protection is policy-as-code applied to the repository: it encodes "small, reviewed, CI-passing changes only" so the process does not depend on memory or goodwill. It pairs with the pipeline's own gates (see cd-and-pipeline-stages.md) — the repo protects the merge, the pipeline protects the deploy.

Named example — Google

Roughly 35,000 developers collaborate in a single monorepo on a shared trunk. Most major projects branch from mainline at a revision for a release and never merge back; fixes are cherry-picked forward into the release branch and periodically returned to mainline. This works because builds are hermetic, tests are fast, and feature flags cover in-flight work. Google's scale is the stress test for TBD: if it breaks down anywhere, it is at merge-request review and CI capacity, not at the branching model itself.

GitHub Flow — Lightweight Middle Ground

GitHub Flow is main plus short-lived feature branches merged via pull request, with deployment happening from main. The trunkbaseddevelopment.com reference notes it is "quite similar" to TBD; the difference is mostly where you release from — GitHub Flow keeps the option of releasing straight off main with no release-branch ceremony.

Aspect	GitHub Flow	Trunk-based
Long-lived branch	main only	main/trunk only
Feature branches	Short-lived, PR-merged	Short-lived, PR-merged
Release point	Tag or deploy from main	Tag from trunk, or cut a short-lived release branch
Best fit	Small-to-mid teams, SaaS, low ceremony	Teams needing strict mainline discipline at scale

GitHub Flow is the right default when a team wants most of TBD's benefits but has not yet built the flag/testing machinery to keep incomplete work safely on trunk. The migration path is usually GitHub Flow → TBD as flags and pipeline maturity arrive, not the reverse.

GitFlow — the Legacy Model (When It Still Fits)

GitFlow adds long-lived develop and release branches alongside main, with hotfix branches for urgent patches. It was designed for versioned, packaged software with scheduled releases and parallel maintenance of multiple released versions. That is also its cost: every change travels feature → develop → release → main, and fixes must be merged in multiple directions, which accumulates merge overhead and slows lead time. Practitioner consensus is to prefer TBD or GitHub Flow unless you genuinely ship boxed/versioned software that customers run without your control.

When GitFlow still fits:

• On-premise or customer-managed software where several released major versions are supported in parallel and each must receive security patches.
• Regulated environments that mandate a formal release branch as the artifact source for audit.
• Hardware-coupled or firmware products where field units cannot be upgraded arbitrarily.

Gotcha — GitFlow as cargo cult: Teams adopt GitFlow "because the enterprise template says so," then pay merge-tax on every release while gaining nothing. If you cannot name the customer constraint that requires parallel version maintenance, use trunk-based or GitHub Flow instead.

Release Branches Cut Just-in-Time

Release branches remain a valid tool when they are cut just-in-time from trunk, hardened, and deleted after release — or omitted entirely when you release from trunk with a fix-forward strategy. The just-in-time pattern gives you a stable stabilization surface without the long-lived-branch overhead of GitFlow:

1. Branch cut — at the release-candidate point, create release/v<major>.<minor>.x from a trunk revision you trust.
2. Stabilization — cherry-pick only critical fixes (P0/P1 defects, security, customer-blocking issues) from trunk into the branch.
3. Release — tag and build the artifact from the branch; re-run the full CI suite on the branch, because cherry-picks bypass the integration testing that happened on trunk.
4. Patches — hotfixes go through the same branch, tagged v2.3.1, v2.3.2, ...
5. End of life — archive or delete the branch once all consumers have migrated.

Cherry-pick discipline: every cherry-pick is code that skipped trunk's integration tests. Require a tracking issue, enforce the same code-review standard, re-run CI on the branch after each pick, and keep a log (SHA + description) attached to the release notes. Google rebuilds at the original release revision and pins the build toolchain to that revision, so a compiler change cannot silently alter a hotfixed in-production release.

The merge-back decision: at release time, choose explicitly among merge the branch back, cherry-pick fixes forward, or archive it. Google's model is branch-from-mainline-at-a-revision, never merge back, cherry-pick fixes forward and periodically return them to mainline. The unforgivable option is no decision: a branch that lingers indefinitely, silently accumulating divergence until "which branch has the fix?" becomes archaeology. If a branch will outlive one release, it must have a merge-back or archiving plan before it is cut.

Release Trains and Calendar Releases

Release trains ship on a fixed cadence regardless of which features are ready — the train leaves the station on schedule. This converts coordination cost into a predictable date: teams know when the branch cuts, when stabilization starts, and when the release ships, and can plan cross-team dependencies against it. The tradeoff is that incomplete work must be explicitly deferred to the next train, which only works when features are flag-gated or genuinely shippable in pieces.

Example	Cadence	Notes
Ubuntu	6-month cycle (YY.MM versions)	Feature freeze and release freeze ladder leading up to each release
Chromium	4-week branch cadence	Branch point every 4 weeks, stabilization to stable, patches on the branch
GitLab	Monthly (YY.MM), self-managed release on the third Thursday of the month (one-week delay if needed)	Patch releases for regressions and security fixes between majors
Firefox	2-week cycle (main → beta merge every 2 weeks)	Version-numbered trains with a single beta channel feeding stable; ~5 betas per cycle, RC QA-tagged, uplifts via tracking/approval flags

The freeze ladder is the train's engine: each milestone (feature freeze → release candidate → final release) converts "should we include this?" from a debate into a date. What is not frozen by the freeze date rides the next train — and the only way that is acceptable is feature flags, which let a feature land on the train while exposure waits (see progressive-delivery.md).

SAFe Agile Release Trains (ARTs) are the scaled-agile variant: 5–12+ teams align on a common Program Increment cadence (typically 8–12 weeks) with PI Planning and periodic sync events, coordinated by a Release Train Engineer. ARTs are an organizational scaling construct — they schedule and synchronize many teams — not a deployment technique. They fit large, multi-team, regulated, or hardware-coupled environments where independent per-team cadence would fragment the product. Treat the specific PI length and role names as SAFe-version-dependent rather than fixed doctrine.

Gotcha — the train metaphor taken literally: A release train that ships regardless of readiness forces you to either ship broken features or hold finished ones. The escape valve is feature flags: ride the train, but expose features only when each is ready. Trains without flags are how "it's on the train, so we shipped it" accidents happen.

Feature-Driven vs Time-Based Releases

Model	Trigger	Pros	Cons	Fits
Feature-driven	Ship when a feature is done	User value arrives immediately; no artificial wait	No predictable dates; feature scope creep delays everything	SaaS with flag-based dark launches, small teams
Time-based (train/calendar)	Ship on schedule	Predictable dates; coordination cost falls; scope control via deferral	Ships whatever is ready; needs flags/deferral discipline	Multi-team products, regulated releases, hardware, enterprises

The two are not mutually exclusive: mature teams use calendar cadence for the container and feature-driven release inside it — the train ships on schedule, but each feature is flag-gated and turned on when its own quality bar is met. This is precisely how "deploy on the train, release features on demand" reconciles the models. The container handles coordination; the flags handle readiness.

Single-Repo vs Multi-Repo Cadence

The branching model interacts with repository topology:

• Multi-repo (polyrepo): each repo owns its pipeline and versioning; cross-repo dependencies are mediated by artifact registries and version ranges. Coordination cost moves into dependency upgrades — every breaking change triggers a cascade of downstream releases. Releases are per-service, which is good for independence and bad for atomic cross-cutting changes.
• Monorepo: one repository, one (or per-service) pipeline, atomic cross-cutting changes, synchronized builds. Releases may be fixed/one-version (everything ships together — Google's one-version rule) or independently versioned (changesets/release-please) with affected-build detection to avoid rebuilding everything (see monorepo-polyrepo-release.md).

The cadence question follows: polyrepo teams need explicit cross-repo release coordination (version ranges, deprecation windows); monorepo teams can release atomically and are the natural home for TBD and release trains.

Deployment Risk Profiles

The process model also determines how a release reaches production, and Google's risk-profiled deployment is the reference pattern: most services roll out via exponential cluster expansion (small canary → doubling exposure on health), while sensitive infrastructure (billing, data, anything where a mistake is expensive) extends the rollout over several days, interleaving across geographic regions so a regional failure does not become a global one. The risk profile is a property of the release's blast radius, not of the team's mood: a config change to a payment path and a new README deploy do not deserve the same rollout shape. Choosing the profile is part of choosing the release model (see progressive-delivery.md for the mechanisms).

Ownership and Ceremony

Every process model implies an ownership and ceremony structure, and the two must match:

Model	Ceremony	Owner
Trunk-based	None per deploy — the pipeline is the gate	The deploying team; no release manager
GitHub Flow	PR review + CI; tag-and-deploy on demand	The deploying team
Release branches (JIT)	Branch cut, stabilization, patch flow	Release manager or designated DRI per release
Release train	Branch cut, freeze ladder, go/no-go, comms	Release manager DRI who owns the train's schedule

The rule: ceremony should scale with coordination need. Per-deploy ceremony in a trunk-based team is waste; the absence of ceremony on a multi-team train is how trains derail. The train's DRI owns the calendar, the branch cut, and the go/no-go (see readiness-and-quality-gates.md); the trunk-based team's "ceremony" is the pipeline itself. When in doubt, put the ceremony in the pipeline (automated gates) rather than in the calendar (meetings) — the pipeline enforces 24×7; the meeting enforces once a month. And whatever the model, name the DRI: an unnamed owner is an unowned process.

Decision Guidance

Situation	Model
SaaS/high-throughput team with automated tests and flags	Trunk-based
Small team, low ceremony, deploy from main	GitHub Flow
Versioned, packaged software, parallel major-version support	GitFlow (or just-in-time release branches from trunk)
Regulated enterprise needing a formal artifact branch for audit	Trunk + just-in-time release branches
Multi-team product needing predictable dates	Release train / calendar (SAFe ART at scale)
Firmware/hardware with field units	Time-based trains + just-in-time release branches
High PR volume breaking main	TBD + merge queue (GitHub merge queue, Graphite, Trunk.io)

A useful litmus test: can the current main head ship to production right now? If not, your process model is forcing batches larger than your quality gates can absorb — shrink the model, not the gates.

Choosing: a Checklist

When selecting or auditing a process model, run the team through these questions:

1. Can the head of main ship to production today? (If not, batches are too large or gates too weak.)
2. Can incomplete work live safely on trunk (feature flags / branch-by-abstraction)? (If not, you will be tempted to park it on branches.)
3. How many released versions must be maintained in parallel, and by what obligation (customer contract, regulation, device fleet)?
4. Does any regulator or customer contract require a release-branch artifact as the audit source?
5. Do multiple teams need a common schedule (trains), or does per-team cadence fragment the product?
6. What is the actual merge cost today — conflict rate, PR wait time, stabilization length? (Measure it; do not argue about it.)
7. What ceremony does the model demand, and does the team have the ownership structure to sustain it?

The answers to 1–2 determine whether trunk-based is viable; 3–4 determine whether GitFlow-style parallel-version support is genuinely required; 5 determines whether trains are warranted; 6–7 tell you whether the team is actually operating the model it thinks it has.

Signs the Model Is Wrong

Process-model problems announce themselves before they cause incidents. Watch for these symptoms:

Symptom	Likely cause	Fix
Long-lived branches accumulate divergent fixes that never reach mainline	Branch lifecycle unenforced	Schedule branch deletion/merge-back; track every backport
main is frequently red	Batches too large; incomplete work not flag-gated	Shrink batches; hide in-flight work behind flags
PRs sit unmerged for days	No merge queue; PRs too big	Add a merge queue; split PRs
Every release needs weeks of stabilization	Cadence mismatch or oversize batches	Smaller trains; flag-gated deferral of unfinished work
Cherry-pick rate on release branches stays high for months	Fixes are not merging back to trunk	Enforce merge-back as a release criterion
Merges regularly conflict	Long-lived feature branches	Move to trunk-based / short-lived branches

Any of these persisting is a process-model defect, not a personnel problem — the model is generating the friction.

Gotchas

Gotcha — long-lived release branches: A release branch kept alive past its release accumulates divergent fixes that never reach mainline. Either schedule the branch's deletion, or treat every backport as a first-class change with a tracking issue.

Gotcha — trunk-based without the prerequisites: Merging to trunk daily without fast automated tests and feature flags just moves the pain: broken main becomes everyone's problem. TBD is a contract — merge small, keep tests green, hide incomplete work.

Gotcha — merge queues treated as a magic shield: Merge queues keep main green under volume but do not reduce batch size. If PRs still sit unmerged for days, the queue is masking an integration bottleneck, not fixing one.

Gotcha — cherry-pick archaeology: Release branches that are never merged back force engineers to re-apply fixes by hand. Track every cherry-pick with a ticket and a changelog entry so the "did this fix ship?" question has a machine-answerable record.

Gotcha — trains without a deferral mechanism: A calendar release with no way to say "this feature is not ready" degrades into a quality lottery. The deferral mechanism (flag, exemption process, scope review) is part of the model, not an add-on.

Sources and Further Reading

• Trunk-Based Development (trunkbaseddevelopment.com)
• Google SRE Book — Release Engineering (ch. 8)
• DORA — Working in Small Batches capability
• DORA — The DORA Metrics guide
• GitHub Blog — How GitHub uses merge queue to ship hundreds of changes every day
• Chromium — Release Process
• Ubuntu Project Docs — Release Team Freezes
• SAFe — Agile Release Train