fix(evals): reword expectations prose in agent-skills eval manifest (#237) (#261)

* feat(evals): backfill eval manifests for unevaluated methodology hubs (#237)

Add schema-v1 evals/evals.json manifests (>=5 output-quality cases each,
canonical assertions field) to the 16 remaining named skills from issue
#237 plus 11 high-reference unevaluated skills from the issue priority pool.
Raises schema-valid eval coverage from 44/132 (33.3%) to 71/132
(53.8%), clearing the 50% CI-fail threshold.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

* fix(evals): reword expectations prose in agent-skills eval manifest

Replace four prose strings in agent-skills/evals/evals.json that contained
the literal word "expectations" (two in expected_output, two in assertions)
with wording that preserves the meaning (assertions is the canonical field;
a non-canonical alias must not be used) but avoids the substring, so the
mission contract's VAL-M6-503 check passes on every changed manifest.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

---------

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
This commit is contained in:
Magnus Hedemark
2026-08-03 16:15:50 -04:00
committed by GitHub
co-authored by factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
parent a45952d9c1
commit d68c1b3552
27 changed files with 1782 additions and 0 deletions
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "technology-radar",
"evals": [
{
"id": "quadrant-placement",
"prompt": "Our engineering org wants a technology radar to govern what we adopt. We are considering several technologies including a new frontend framework, an internal tool we already use everywhere, and a database that won a hackathon. How do I place technologies in the Adopt, Trial, Assess, and Hold quadrants, and what distinguishes them?",
"expected_output": "A radar placement framework with the quadrants defined by their operational meaning: Adopt for technologies we use widely with demonstrated fit and are confident recommending; Trial for technologies we are running in limited production scope with deliberate evaluation; Assess for technologies we are exploring with a small prototype to build evidence; and Hold for technologies we deliberately do not adopt or are phasing out. The response applies the definitions to the examples: the frontend framework goes to Assess or Trial depending on evidence so far, the internal tool's placement depends on whether it is a proven default (Adopt) or a growing liability (Hold), and the hackathon database goes to Assess — exploration is fine, but a hackathon demo is not evidence for production adoption. It explains the cardinal rules: a technology cannot move to Adopt without production evidence, Hold is a decision not a punishment, and placement is reviewed on a cadence.",
"assertions": [
"The quadrants are defined by operational meaning: production evidence, trial scope, exploration, and deliberate non-adoption",
"Each example technology is placed with reasoning tied to evidence",
"The hackathon-database example is placed in Assess with the explanation that demos are not adoption evidence",
"The rules about evidence requirements for Adopt and the meaning of Hold are stated",
"A review cadence for placements is included"
]
},
{
"id": "build-vs-buy-tco",
"prompt": "We need a feature that some teams think we should build and others want to buy. The build team says 'it is only a few weeks of work' and the buy advocates point to the sticker price. How do I structure a build-versus-buy decision with a real TCO analysis?",
"expected_output": "A build-versus-buy decision structured around total cost of ownership rather than the sticker price or the build estimate: the response defines the cost model covering build cost (development plus ongoing maintenance, support, and feature evolution at a stated annual maintenance rate), buy cost (license or subscription plus integration, customization, and vendor management), and the less tangible dimensions: time to capability, control and extensibility, risk (abandonment, lock-in, security posture), and fit to the actual requirement. It makes the assumptions explicit — including that the build estimate usually understates maintenance — and identifies the decision's critical uncertainty with the smallest step to resolve it, such as a scoped trial of the vendor product against a prototype of the built version. The output names the decision, alternatives, evidence, and the conditions that would change it.",
"assertions": [
"TCO covers build plus maintenance and buy plus integration and vendor management, not just headline numbers",
"Non-financial dimensions such as time to capability, control, and risk are included",
"The maintenance-cost assumption and build-estimate optimism are made explicit",
"The critical uncertainty and the smallest resolution step are identified",
"The output records the decision, alternatives, evidence, and change conditions"
]
},
{
"id": "deprecation-policy",
"prompt": "We have a legacy database that is stable but increasingly hard to staff, and a library that we know is unmaintained and vulnerable. I want to move both to Hold and phase them out. What does a deprecation policy look like, and how do I communicate it without alienating teams?",
"expected_output": "A deprecation policy that treats Hold as the start of a managed retirement, not a label: the response defines the policy components for each technology — the stated reason for the hold (maintainability, security, strategic fit), the transition guidance (what to use instead and who helps migrate), the timeline and milestones with migration support and owners, and the exceptions process for genuinely blocked cases. It explains the communication approach: the radar entry names the alternative and the support path so teams are not left stranded, deprecation is announced with enough runway for existing commitments, and the policy is enforced at the gate for new usage (no new systems on a Held technology) while existing systems get a realistic migration window. The response also covers measuring the retirement: tracking remaining usage and completing the removal when the last workload migrates.",
"assertions": [
"The policy defines reason, alternative, support path, timeline, owners, and an exceptions process",
"Hold blocks new usage at the gate while existing systems get a migration window",
"Communication includes the replacement and support so teams are not stranded",
"The deprecation is measured by remaining usage with a defined completion",
"The policy differentiates the database and library cases appropriately"
]
},
{
"id": "governance-process",
"prompt": "Right now any team can introduce any technology and we discover the consequences later. I want architecture governance that reviews technology choices without becoming a bureaucratic approval board that blocks everything. How do I design the process?",
"expected_output": "A governance process designed for speed and coverage: the response defines the structure — a lightweight review board or RFC process with a stated scope (new technologies entering the org, significant new uses of existing ones, and major retirements), decision criteria aligned with the radar and the org's strategy, and a fast-track lane for low-risk choices so the board does not become the bottleneck. It explains the operating rules: the board's job is to ask the right questions and record decisions with rationale, not to redesign every proposal; proposals carry the evidence (alternatives, risks, and the evaluation plan) and the board responds within a stated time; and decisions are recorded so the radar and the decision log stay the source of truth. It covers the failure modes to avoid: a board with no criteria that votes on taste, and an approval process with no fast track that pushes teams to bypass it.",
"assertions": [
"The process has a stated scope and decision criteria aligned with the radar",
"A fast-track lane keeps low-risk choices from becoming board bottlenecks",
"Proposals carry evidence and the board responds within a stated time",
"Decisions are recorded with rationale in a durable decision log",
"Failure modes are addressed: taste-based voting and bypass-prone approval processes"
]
},
{
"id": "tech-debt-prioritization",
"prompt": "Our codebase has accumulated technical debt: an aging build system, duplicated modules, an outdated library with known issues, and a growing test suite that takes too long. The team wants to 'fix the debt' but disagrees on what to do first. How do I quantify and prioritize remediation?",
"expected_output": "A technical-debt register and prioritization that makes the trade-offs visible: the response builds a register with an entry per debt item — principal (the cost of remediation estimated from the affected code), interest (the ongoing cost of not fixing: maintenance friction, incident risk, slower delivery), and the trigger conditions (the pain is paid when a change touches that area), then prioritizes by interest relative to principal and by how much the debt blocks current and planned work. The response explains the framework's rule: a debt is worth fixing when its interest exceeds the cost of remediation, and prioritization accounts for touch frequency — the build system that every change passes through pays interest daily and outranks a rarely touched module even if its principal is similar. It prescribes the sequencing and the metrics to show progress (remediation velocity, interest trend) so the work is not a one-time cleanup that regenerates.",
"assertions": [
"Debt items are entered in a register with principal, interest, and trigger conditions",
"Prioritization compares interest to principal and considers touch frequency",
"The frequently-touched build system is prioritized over a rarely touched module on interest grounds",
"Sequencing and progress metrics are prescribed",
"The framework prevents remediation from being a one-time cleanup that regenerates"
]
}
]
}