magnus919_agent-skills/release-engineering/evals/evals.json

{"schema_version": 1, "skill_name": "release-engineering", "evals": [{"id": "release-plan-design", "prompt": "We are shipping v2.4.0 of our SaaS product next month and I need a release plan covering the timeline, who does what, and what could go wrong. Can you put one together?", "expected_output": "A complete release plan naming the version and target date, a timeline with milestones and branch cut, owners assigned via RACI, scope items derived from the changelog/commits, risks with mitigations, a rollout plan, a rollback contingency, a communications plan, and recorded sign-offs. The plan follows the skill's release-plan template structure.", "assertions": ["The response names the release version and target date with a milestone timeline including branch cut", "The response assigns named owners for each task using a RACI-style responsibility matrix", "The response derives scope items from changelog entries or commit history rather than guessing", "The response lists risks with mitigations and a rollback contingency for the rollout", "The response includes a communications plan and explicit sign-offs before go-live"]}, {"id": "progressive-delivery-selection", "prompt": "We deploy to production every day and I want to start rolling out changes more gradually instead of flipping the switch all at once. Which progressive delivery strategy should we use and how do we gate it?", "expected_output": "A strategy recommendation comparing canary, blue/green, rings, percentage rollouts, and feature flags on speed, safety, and rollback time, built on the deploy-vs-release distinction. The rollout is gated on SLI-derived metrics and error budget, compares canary vs control concurrently rather than before/after, ramps exposure exponentially with verification between steps, and wires guardrail divergence to auto-rollback.", "assertions": ["The response explicitly separates deploy from release and recommends a concrete strategy from canary, blue/green, rings, percentage, or feature flags with trade-offs", "The response gates rollout steps on SLI-derived metrics and the service's error budget", "The response requires comparing canary/experimental population against a control concurrently, never before/after", "The response ramps exposure exponentially (e.g., 1% to 10% to 50% to 100%) with verification between steps", "The response wires guardrail divergence to pause, auto-rollback, or page rather than continuing the ramp"]}, {"id": "version-bump-determination", "prompt": "I have a changelog with breaking change, feat, and fix commits since v1.9.0 and need to know what the next version should be and why.", "expected_output": "A version bump computed from Conventional Commits semantics: a BREAKING CHANGE footer or feat!/fix! maps to a MAJOR bump, feat to MINOR, fix to PATCH, and no-bump types (chore, docs, refactor, ci) do not advance the version. For a breaking change after v1.9.0 the computed version is 2.0.0, validated against strict SemVer with the bump rule justified per commit type.", "assertions": ["The response maps a BREAKING CHANGE footer or feat!/fix! to a MAJOR bump", "The response maps feat commits to MINOR and fix commits to PATCH bumps", "The response ignores no-bump types such as chore, docs, refactor, or ci", "The response computes 2.0.0 as the next version after 1.9.0 when a breaking change is present", "The response validates the computed version against strict SemVer and justifies each bump rule"]}, {"id": "rollback-plan-stateful-service", "prompt": "Our service shares a Postgres database and the last release included a schema migration that already ran in production. I need a rollback plan for when this release goes wrong.", "expected_output": "A rollback plan built on expand/contract (parallel change) with forward-only migrations: the schema must support the previous release so the binary can be rolled back safely, rollback is safe only before finalization, and after finalization the only path is a new forward migration. The plan rejects git revert as a rollback for schema/data changes, adds manual checkpoints for data-touching operations, and treats backup/restore with RPO/RTO as a last resort.", "assertions": ["The response applies expand/contract (parallel change) so old and new schema coexist across releases", "The response states rollback is safe only before finalization and requires a forward migration afterwards", "The response rejects git revert as a rollback mechanism for schema or data changes", "The response requires forward-only, append-only migrations and manual checkpoints before data-touching operations", "The response treats backup/restore (with RPO/RTO) as a last resort rather than a primary rollback path"]}, {"id": "dora-metrics-computation", "prompt": "I exported our deployment and commit events for the last 30 days and want to compute our DORA metrics from the raw data, not from whatever the vendor dashboard shows.", "expected_output": "A computation of all five DORA metrics from raw events: deployment frequency as successful production deploys per day, change lead time aggregated as the median across changes, change failure rate as failed over total deployments, failed deployment recovery time as median time to recover from change-caused failures, and deployment rework rate as the share of unplanned bug-fix deploys. Metrics are scoped to production only, use medians not means, and flag measurement pitfalls such as PRs vs deploys and ignoring rollbacks.", "assertions": ["The response computes deployment frequency from successful production deployments per day", "The response aggregates change lead time as the median across changes, not the mean", "The response computes change failure rate as failed deployments divided by total deployments", "The response scopes every metric to production or release-to-users and includes failed deployment recovery time and deployment rework rate", "The response flags measurement pitfalls such as counting PR merges as deploys, ignoring rollbacks, or using mean instead of median"]}, {"id": "release-readiness-checklist", "prompt": "Our release candidate is ready and I need a readiness review structure so we can decide go or no-go with actual evidence instead of vibes.", "expected_output": "A readiness checklist organized into the four dimensions — functional, non-functional, operational, and governance — where every item has a named owner and an evidence link, plus a go/no-go decision block. Operational items cover monitoring live before go-live, runbooks, and a rehearsed rollback; governance items cover approval and the ticket-to-verification audit chain.", "assertions": ["The response organizes readiness into the four dimensions: functional, non-functional, operational, and governance", "The response assigns a named owner (a person, not a team) and an evidence link to every checklist item", "The response includes an explicit go/no-go decision block with recorded verdict", "The response requires monitoring and alerting to be live before go-live and a rehearsed rollback path", "The response ties governance items to the audit chain of ticket to PR review to CI to approval to deploy log to verification"]}, {"id": "feature-flag-cleanup-plan", "prompt": "We have dozens of feature flags that have been on for a year and nobody remembers what they do. How do we clean them up safely?", "expected_output": "A flag cleanup plan following the remove-then-archive order: remove all code references first (verified with code-reference scanning), then archive the flag key in the platform — never delete or reuse keys. The plan assigns owners and expiry at creation, uses time-bomb checks to enforce removal, targets a 90-120 day archive cadence, tests both ON and OFF states, and warns about the Knight Capital flag-reuse failure.", "assertions": ["The response requires removing all code references before archiving the flag", "The response says to archive flag keys rather than delete them, and never reuse a key", "The response recommends code-reference scanning and time-bomb expiry checks to enforce removal", "The response targets archiving temporary flags within roughly 90-120 days", "The response warns about the Knight Capital failure mode of reactivating a stale flag and requires testing both ON and OFF states"]}, {"id": "anti-trigger-incident-debugging", "prompt": "Our checkout service is throwing intermittent 500 errors in production and I need help doing root-cause analysis to find the underlying fault.", "expected_output": "The agent declines to apply release-engineering to this request, recognizing that production incident root-cause debugging and fault localization fall outside its scope. It routes the user to the systematic-debugging skill for root-cause analysis and fault localization, and to site-reliability-engineering for on-call and incident response operations.", "assertions": ["The response declines release-engineering as the appropriate skill for production incident root-cause debugging", "The response routes the user to systematic-debugging for fault localization and root-cause analysis", "The response names site-reliability-engineering for on-call and incident-response operations", "The response does not attempt to apply release planning, rollback, or DORA methodology to the debugging task", "The response explains that incident root-cause debugging sits outside the release-engineering negative boundary"]}]}