skills/magnus919_agent-skills
1
0
Fork 0
mirror of https://github.com/magnus919/agent-skills.git synced 2026-09-11 19:47:12 +03:00
Code Issues Packages Projects Releases Wiki Activity

fix(evals): reword expectations prose in agent-skills eval manifest (#237) (#261)

Browse Source 
* feat(evals): backfill eval manifests for unevaluated methodology hubs (#237)

Add schema-v1 evals/evals.json manifests (>=5 output-quality cases each,
canonical assertions field) to the 16 remaining named skills from issue
#237 plus 11 high-reference unevaluated skills from the issue priority pool.
Raises schema-valid eval coverage from 44/132 (33.3%) to 71/132
(53.8%), clearing the 50% CI-fail threshold.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

* fix(evals): reword expectations prose in agent-skills eval manifest

Replace four prose strings in agent-skills/evals/evals.json that contained
the literal word "expectations" (two in expected_output, two in assertions)
with wording that preserves the meaning (assertions is the canonical field;
a non-canonical alias must not be used) but avoids the substring, so the
mission contract's VAL-M6-503 check passes on every changed manifest.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

---------

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
This commit is contained in:
Magnus Hedemark
2026-08-03 16:15:50 -04:00
committed by GitHub
co-authored by factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
parent a45952d9c1
commit d68c1b3552
27 changed files with 1782 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
adr-authoring/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "adr-authoring",
"evals": [
{
"id": "adr-authoring",
"prompt": "We just decided to switch our service-to-service communication from synchronous REST calls to an event-driven model with a message broker. I need to record this as an architecture decision record so future engineers understand why. What should the ADR contain and how should I write it?",
"expected_output": "An ADR following the standard structure: a status line (proposed, accepted, or superseded with the date and decider), the context that explains the forces and constraints at the time (the coupling pain, the scale trajectory, the team's operational constraints), the decision stated in one unambiguous sentence, the alternatives that were considered with the reasons they were rejected (synchronous REST with retries, a shared database change feed), the consequences of the decision split into positive and negative so the trade-off is visible (decoupling and independent scaling against the new operational burden of message ordering, delivery guarantees, and debugging async flows), and the compliance and follow-up items. The response explains what makes it durable: the context captures the reasoning so the decision survives personnel changes, the alternatives are recorded honestly, and the consequences include what the team must now do differently.",
"assertions": [
"The ADR contains status, context, decision, alternatives, and consequences sections",
"The decision is stated in one unambiguous sentence",
"Alternatives are recorded with the reasons they were rejected",
"Consequences are split into positive and negative with the trade-off visible",
"The context captures the reasoning so the decision survives personnel changes"
]
},
{
"id": "template-selection",
"prompt": "We are starting to write ADRs for a new project and I have seen many formats: the original Nygard format, MADR, and heavier enterprise templates. The team has different preferences. How do I choose a template, and should I even pick one?",
"expected_output": "A template-selection approach that prioritizes consistency and the decision's needs over format loyalty: the response explains that the template catalog exists because decision contexts differ — the original Nygard format suits a focused technical decision, MADR adds structure for incremental updates, and heavier formats carry the governance fields larger organizations need — and prescribes picking one default for the org with the selection made on criteria: the depth of governance required, how the ADRs will be consumed (read by the team, audited by a governance board), and the update pattern (append-only records versus evolving documents). It recommends starting with a lightweight default and migrating to a heavier format only if the governance need appears, and it stresses that the bigger win is a fixed convention — one template, one naming scheme, one location — over the choice of which format, because consistency is what makes ADRs searchable and reliable.",
"assertions": [
"Template choice is tied to the decision context and governance need, not format loyalty",
"The response compares Nygard, MADR, and heavier formats on concrete criteria",
"A lightweight default with optional migration is recommended",
"Consistency of convention is valued above the specific format choice",
"The recommendation covers naming and location conventions"
]
},
{
"id": "adr-lifecycle-governance",
"prompt": "We have an ADR that was accepted, then partially reversed a year later, and now a proposal wants to replace it entirely. Our ADR folder is a flat list of files with no states and nobody knows what is actually in force. How do I manage the ADR lifecycle and status transitions?",
"expected_output": "A lifecycle governance design that makes status the primary way to understand an ADR: the response defines the status model (proposed, accepted, superseded, deprecated, and rejected), the transition rules (an accepted ADR moves to superseded when a new ADR replaces it and links to the replacement; a partially reversed decision is recorded as a new decision or a revision rather than silently editing the original), and the practical mechanics: each ADR carries its status and date in the header, superseded ADRs link to their successor, and the folder has an index (a README or status table) showing what is currently in force. The response prescribes the workflow: decisions flow through review before acceptance, supersession is explicit with a reason, and the index is part of the review so the team can see the current architecture at a glance instead of reading every file.",
"assertions": [
"The status model covers proposed, accepted, superseded, deprecated, and rejected with transition rules",
"Superseded ADRs link to their replacement and record the reason",
"Partial reversals are recorded as new decisions or revisions rather than silent edits",
"An index shows what is currently in force",
"The review workflow keeps statuses and the index current"
]
},
{
"id": "adr-quality-review",
"prompt": "I am reviewing ADRs before we accept them and I keep seeing the same problems: decisions with no alternatives, consequences that only list the positives, and context sections that describe the solution instead of the problem. What should my review checklist check?",
"expected_output": "An ADR review checklist focused on the properties that make a decision durable: the response prescribes checking that the context describes the problem and forces, not the chosen solution; that the decision is a clear statement of what was decided and what was explicitly not decided; that alternatives are real alternatives that were seriously considered, with the rejection reasons recorded; that consequences include the negative and operational costs, not only benefits; and that the ADR records who decided, when, and under what constraints. It also covers the sustainability checks: whether the ADR would still make sense to a reader in two years who does not know the authors, whether the trade-offs are stated in terms that can be revisited when the context changes, and whether it leaves the team with follow-up items or open questions that should be tracked rather than hidden.",
"assertions": [
"The checklist verifies context describes the problem, not the chosen solution",
"Alternatives are verified as seriously considered with rejection reasons",
"Consequences must include negatives and operational costs",
"The ADR records who decided, when, and under what constraints",
"The review assesses durability: would it make sense in two years, and are follow-ups tracked"
]
},
{
"id": "fitness-functions",
"prompt": "We have an accepted ADR mandating that new services must use our standard logging format, but a year later half the services violate it and nobody noticed until an incident. I want the architecture rules enforced automatically. How do I turn ADRs into checkable constraints?",
"expected_output": "A fitness-function approach that turns ADR decisions into automated checks: the response explains the concept — a fitness function is an automated test or check that continuously validates an architectural characteristic, and the ADR maps to one or more functions (a check that scans service code or configuration for the standard logging setup, run in CI or as a periodic audit). It prescribes the workflow: for each ADR with a mechanical consequence, define the check, implement it in the project's test or CI layer, and attach it to the ADR record so the link between decision and enforcement is explicit. The response covers the boundary: not every decision is mechanically checkable (judgment calls stay in review), but anything with a detectable pattern should be checked, and the check must be part of the definition of done for new services, not a retrofit after incidents. It includes an example check shape for the logging standard.",
"assertions": [
"The fitness-function concept is explained as automated enforcement of architectural rules",
"Each mechanical ADR consequence maps to a concrete check",
"The ADR record links to its enforcement check",
"The boundary between checkable and judgment-based decisions is stated",
"Enforcement is part of definition of done, not a post-incident retrofit"
]
}
]
}
agent-evals-and-observability/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "agent-evals-and-observability",
"evals": [
{
"id": "eval-dataset-design",
"prompt": "I want to build an evaluation set for our customer-support agent to judge whether responses are good before every release. What does the dataset design look like: what cases belong in it, how many, and how do I keep it from going stale?",
"expected_output": "An eval dataset design organized around the agent's task contract: the response defines a set of representative tasks sampled from real production traffic (the actual questions and edge cases users ask), each with a task description, the input, and the expected behavior, and it explains the population discipline: cases are drawn from production distributions including the failure modes the team cares about, plus a curated set of edge cases and regression cases from past incidents. The dataset's size is matched to the decision it supports (a small curated set for release gating versus a larger sampling set for tuning), and the response covers the freshness contract: a process for adding new cases from production incidents and removing or re-validating cases that no longer reflect the product, with versioning so results stay comparable across releases.",
"assertions": [
"The dataset is sampled from real production traffic and past failure modes, not invented by the team alone",
"Each case carries a task description, input, and expected behavior",
"Dataset size is matched to the decision it supports",
"A freshness process adds incident-derived cases and retires stale ones",
"Versioning keeps results comparable across releases"
]
},
{
"id": "grader-criteria",
"prompt": "Our eval harness has tasks and recorded agent outputs, but the scoring is a single human judgment of 'looks good.' I want objective, repeatable grading. How do I write grader criteria that multiple reviewers (or a judge model) can apply consistently?",
"expected_output": "Grading criteria written as discrete, checkable requirements rather than holistic impressions: each task has pass-fail or rubric-scored criteria derived from the task contract — required content must be present, required steps must be taken, forbidden behaviors must be absent, and correctness is defined against a reference answer or verifiable facts rather than style. The response explains the rubric design: a small number of criteria (not a long checklist of trivia), each stated so that a reviewer can determine pass or fail without interpretation, with explicit handling of partial credit and a rule for when a single failure fails the whole case (e.g., a safety violation or a wrong factual claim). It also covers calibration: sample-scoring a set of outputs against the rubric, reconciling disagreements, and iterating the rubric until reviewers converge.",
"assertions": [
"Criteria are discrete and checkable, derived from the task contract",
"Required content, required steps, and forbidden behaviors are separated",
"Partial credit and fail-the-case rules are explicit",
"A calibration pass with disagreement reconciliation is prescribed",
"Rubric iteration continues until reviewers converge"
]
},
{
"id": "regression-analysis",
"prompt": "Our agent's overall score went up after a prompt change, but a few individual tasks got much worse, and I suspect they are the ones that matter. How do I analyze eval results across releases to catch regressions rather than a single averaged number?",
"expected_output": "A regression analysis that looks below the aggregate score: the response prescribes comparing per-case results between the baseline and candidate release, separating the cases that improved, regressed, and stayed the same, and weighting the regressed cases by their production frequency and severity so a drop on a high-traffic task outweighs gains on rare ones. It explains the statistical ground rules: small eval sets produce noisy deltas, so the analysis distinguishes meaningful changes from sampling noise (via confidence bounds or a stated sample requirement) and flags regressions for investigation even when the average improves. It also covers bucketing by case category (customer-visible errors, safety, style) so the team can see which behavior class moved, and it prescribes a fix loop: investigate the regressed cases, decide whether the change or the eval is wrong, and re-run.",
"assertions": [
"Analysis compares per-case results between baseline and candidate, not just the average",
"Regressed cases are weighted by production frequency and severity",
"Statistical noise in small eval sets is accounted for with confidence bounds",
"Results are bucketed by behavior class such as safety, errors, and style",
"A fix loop investigates regressed cases and re-runs after changes"
]
},
{
"id": "release-gate-design",
"prompt": "We want to gate releases on eval results so a bad change cannot ship. How do I design the release gate so it blocks real regressions without making every release an evals fire drill?",
"expected_output": "A release-gate design that separates the gating decision from the raw score: the response defines the gate criteria in terms of the regression analysis rather than a single threshold — no regressions on critical-path cases, no regressions beyond a tolerance band on the overall set, and the gate uses the previously established baseline for the same version of the eval set (so set changes do not silently move the goalposts). It prescribes the operational mechanics: the gate runs in CI, produces a comparable report against the merged baseline, blocks on the blocking criteria, and routes borderline results to a human review queue with the evidence attached rather than an automatic pass or fail. It also covers the escape hatch: an explicit override process with a recorded reason and owner, so the gate stays credible.",
"assertions": [
"The gate uses regression-relative criteria, not a single absolute score",
"The baseline is pinned to the same eval-set version so goalposts do not move",
"The gate runs in CI with a comparable report and blocks on critical-path regressions",
"Borderline results route to human review with evidence, not automatic pass or fail",
"An explicit, recorded override process keeps the gate credible"
]
},
{
"id": "incident-to-case-learning",
"prompt": "A customer-facing incident last week traced back to an agent answer we never tested: the agent confidently gave wrong configuration advice. I want the incident to become a permanent eval case so it cannot regress. What is the incident-to-case workflow and how does trajectory review fit in?",
"expected_output": "An incident-to-case workflow that converts the postmortem into durable eval coverage: the response walks the process — extract the failing behavior from the incident (the wrong output, the context that produced it, the harm), turn it into a task case with the correct expected behavior and grader criteria, add it to the regression set, and verify it fails on the current release and passes on the fix. It explains where trajectory review fits: for agent failures, the answer alone may not show the flaw, so the review examines the reasoning trajectory (the steps, tool calls, and sources the agent used) to understand whether the error was a knowledge gap, a retrieval failure, or a reasoning failure, which determines the fix and the right case shape. The response also covers the loop: the new case joins the baseline, so any future release that reintroduces the behavior is blocked.",
"assertions": [
"The workflow converts the incident into a task case with expected behavior and grader criteria",
"The case is verified to fail on the current release and pass on the fix",
"Trajectory review is used to classify the failure as knowledge, retrieval, or reasoning",
"The failure classification drives both the fix and the case shape",
"The new case joins the baseline so the regression is blocked going forward"
]
}
]
}
agent-skills/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "agent-skills",
"evals": [
{
"id": "skill-creation-structure",
"prompt": "I want to create a new skill called 'pdf-tools' that teaches an agent how to merge, split, and extract text from PDFs. What is the required directory structure and what must the SKILL.md contain to be format-compliant?",
"expected_output": "A format-compliant skill scaffold: a directory named pdf-tools containing SKILL.md with YAML frontmatter and a markdown body, a human-facing README.md, and for a new skill an evals/evals.json with at least five output-quality cases. The response specifies the frontmatter contract: name must match the directory name in lowercase hyphenated form, description must be 1-1024 characters, start with an imperative verb, and define both positive triggers and a negative boundary (when not to use this skill), and the body must stay under 500 lines with supporting material in references/ loaded on demand. It also explains why progressive disclosure matters: core instructions in SKILL.md, detail in references/scripts, so activation cost stays low.",
"assertions": [
"The scaffold names the required structure: SKILL.md, README.md, and evals/evals.json for new skills",
"Frontmatter rules are specified: name matches directory, imperative-verb description with negative boundary",
"The 500-line body limit and progressive disclosure structure are stated",
"File references are relative from the skill root",
"The rationale for progressive disclosure (low activation cost) is explained"
]
},
{
"id": "skill-review-compliance",
"prompt": "I need to review a skill in our repository before merging it. The skill has a SKILL.md, a README.md, and a scripts folder, but I am not sure it is format-compliant. What checklist should I run, and what are the most common compliance failures?",
"expected_output": "A review checklist covering the format requirements: name matches the directory, description is 1-1024 characters starting with an imperative verb and defining a negative boundary, body under 500 lines, valid frontmatter with only allowed fields, README.md present with the required human-facing sections, file references resolve to real relative paths, and for new skills an eval manifest with at least five cases using the canonical assertions field. The response lists the most common failures: descriptions that do not start with an imperative verb, missing negative boundaries, frontmatter fields beyond the allowed set, stale or broken relative links, references to skills that do not exist in the catalog, and evals that use a non-canonical alias instead of the canonical assertions field. It prescribes running the repository's validation scripts to catch what the checklist misses.",
"assertions": [
"The checklist covers frontmatter, description rules, body limits, README sections, references, and evals",
"Common failures are enumerated: non-imperative descriptions, missing negative boundaries, bad links, dead routing",
"The canonical assertions field versus the non-canonical alias is called out",
"The review verifies relative references resolve",
"Running the repository validators is prescribed as the final check"
]
},
{
"id": "progressive-disclosure-fix",
"prompt": "A skill I wrote has a 900-line SKILL.md because I put everything in the main file. Agents loading it consume enormous context even when they only need one section. How do I restructure it for progressive disclosure?",
"expected_output": "A restructuring plan that moves the SKILL.md to a lean core: keep the description-driven triggers, the operating instructions, and the routing that tells the agent when to load which reference, then move the detailed material into references/ files organized by concern, templates/ for fillable documents, and scripts/ for executable tooling, each referenced from SKILL.md with the load condition. The response explains the progressive disclosure contract: metadata at startup, instructions on activation, resources on demand, and that the SKILL.md should state when to read each supporting file rather than embedding it. It also covers the practical audit: after the split, the body must be under 500 lines, every reference must resolve, and nothing the core workflow depends on may be left only in a reference the agent is not told to load.",
"assertions": [
"The plan moves detail into references, templates, and scripts while keeping SKILL.md as the lean core",
"The response applies the load-on-demand contract: instructions on activation, resources on demand",
"SKILL.md states when to load each supporting file rather than embedding it",
"The post-split body is verified under 500 lines with all references resolving",
"Core workflow dependencies are not stranded in unlinked references"
]
},
{
"id": "evals-manifest-authoring",
"prompt": "I am adding an eval manifest to an existing skill that has no evals. What must the manifest contain to pass the repository's v1 validation, and what makes the cases actually useful for judging output quality?",
"expected_output": "An eval manifest written to the v1 contract: schema_version 1, skill_name matching the skill directory, and an evals array of at least five cases, each with a stable lowercase-hyphenated id, a realistic prompt, an expected_output describing the correct behavior, and a list of assertions that are observable claims about the output, using the canonical assertions field and never a non-canonical alias. The response explains what makes cases useful: prompts that reflect real activation scenarios for the skill, assertions that are specific enough to fail meaningfully rather than vague quality platitudes, stable IDs that durable evidence references can rely on, and coverage across the skill's main behaviors. It also notes the trigger-only probes belong in a separate harness-specific test set, not in evals.json.",
"assertions": [
"The manifest uses schema_version 1, matching skill_name, and at least five cases",
"Each case has a stable id, realistic prompt, expected_output, and observable assertions",
"The assertions field is canonical and non-canonical aliases are never used",
"Cases reflect real activation scenarios and are specific enough to fail meaningfully",
"Trigger-only probes are excluded from evals.json as harness-specific"
]
},
{
"id": "client-discovery-loading",
"prompt": "I am building an agent client that needs to discover and load skills from a directory of Agent Skills-format skills. How should discovery and loading work so the client respects the format's cost model?",
"expected_output": "A discovery and loading design implementing the three-stage model: at startup the client reads only each skill's name and description from frontmatter as metadata, when a user request matches a description the client loads the full SKILL.md, and supporting files under references/, templates/, and scripts/ are loaded on demand when the skill's instructions say to. The response explains why loading everything at startup defeats the format's purpose (context cost grows with catalog size) and how routing works: the description is the trigger surface, so description quality directly determines whether the right skill activates. It covers the failure modes: skills whose descriptions do not overlap the user's phrasing fail to trigger, and a client that reads beyond frontmatter during discovery pays the cost the format was designed to avoid.",
"assertions": [
"Discovery reads only name and description frontmatter at startup",
"Full SKILL.md is loaded only on trigger match and supporting files on demand",
"The response explains the context-cost rationale for staged loading",
"Description quality is tied to routing correctness",
"The failure modes of eager loading and weak trigger overlap are covered"
]
}
]
}
api-design-and-evolution/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "api-design-and-evolution",
"evals": [
{
"id": "rest-api-contract-design",
"prompt": "We are building a public REST API for our invoicing product. I need the contract for the invoices resource: endpoints, pagination, error handling, and filtering. What should the contract specify and what decisions matter most for consumers?",
"expected_output": "A REST contract design for the invoices resource that makes the consumer-facing decisions explicit: the resource URL structure with consistent plural nouns and stable identifiers, the HTTP methods and their semantics (list, get, create, update, delete) with proper status codes, pagination as a stable mechanism (cursor or offset with a stated default and maximum page size), consistent error responses with a machine-readable error code, message, and correlation ID, idempotency for creation via a client-supplied key, and filtering and sorting parameters that are documented and bounded. The response explains the compatibility discipline: fields and parameters are additive, response shapes are stable, and the OpenAPI document is the source of truth that consumers build against.",
"assertions": [
"The contract defines the full invoice resource surface: methods, status codes, and URL structure",
"Pagination is specified as a stable mechanism with defaults and limits",
"Error responses have a consistent machine-readable shape with codes and correlation IDs",
"Creation is idempotent via a client-supplied key",
"Filtering and sorting are documented and bounded, with OpenAPI as the source of truth"
]
},
{
"id": "versioning-deprecation",
"prompt": "We need to change the response of our customers endpoint from a flat structure to a nested one, which will break current consumers. The API is used by dozens of partners. How do I version this change and manage the deprecation lifecycle responsibly?",
"expected_output": "A versioning and deprecation plan that avoids breaking consumers: the response evaluates versioning options (URL path versioning versus content negotiation versus additive-only evolution) and selects one for the change, then defines the deprecation lifecycle: ship the new version alongside the old, announce the deprecation with a concrete timeline, add Sunset headers and deprecation notices in responses so consumers see it programmatically, migrate the known partners with support, and remove the old version only after the deadline with the usage metrics confirming no remaining traffic. The response explains that a breaking change that could be done additively (new field, old field kept) should not force a version bump, and it sets the policy for when a major version is genuinely warranted.",
"assertions": [
"The response evaluates versioning strategies and picks one for the change",
"The change is assessed for additive compatibility before forcing a major version",
"A deprecation lifecycle is defined: parallel versions, announcement, Sunset headers, timeline",
"Partner migration is supported and removal is gated on usage metrics",
"Deprecation notices are surfaced programmatically to consumers"
]
},
{
"id": "event-interface-asyncapi",
"prompt": "We are adding an events interface so internal services and external partners can subscribe to invoice.created and invoice.paid events. I have never designed an event interface. What does the contract look like and what decisions do I need to make?",
"expected_output": "An event-interface design with an AsyncAPI contract as the source of truth: the event names and their payload schemas (what fields each event carries and the guarantees about them), the delivery semantics (at-least-once with a message ID, deduplication keys, and ordering caveats), the channel or topic naming scheme, and the compatibility rules for evolving payloads (additive fields only, versioned schema for breaking changes). The response explains the core consumer-contract decisions: exactly-once is not provided so consumers must deduplicate, ordering is per-partition not global, and retries need a dead-letter policy. It specifies what the producer guarantees versus what consumers must handle.",
"assertions": [
"The design is documented as an AsyncAPI contract with named events and payload schemas",
"Delivery semantics are explicit: at-least-once, message IDs, deduplication, ordering caveats",
"Channel or topic naming and payload evolution rules are specified",
"The response states that consumers must handle deduplication and that ordering is per-partition",
"Retry and dead-letter handling are part of the consumer contract"
]
},
{
"id": "api-review-existing-contract",
"prompt": "A teammate wrote an OpenAPI spec for a new bookings API and asked for a review before publishing it to partners. What should I look for in a contract review beyond syntax correctness?",
"expected_output": "An API contract review that checks the decisions that create or avoid future breaking changes: naming consistency and URL structure, response envelope consistency, error schema uniformity, pagination on list endpoints, whether create/update are idempotent or need to be, whether required fields are truly required or just asserted, parameter validation and bounds, and whether the spec matches the documented behavior in examples. The review prioritizes findings by consumer impact: anything that forces a breaking change later or that partners will mis-implement gets flagged first, followed by inconsistencies and documentation gaps. The response frames the review output as actionable findings with severity rather than a general comment thread.",
"assertions": [
"The review checks future-compatibility decisions: idempotency, pagination, error uniformity, required-field truthfulness",
"Findings are prioritized by consumer impact and breaking-change risk",
"The review compares the spec against its own examples for consistency",
"Parameter validation and bounds are checked",
"The review produces actionable, severity-ranked findings"
]
},
{
"id": "error-handling-idempotency",
"prompt": "Our mobile app sometimes retries a payment API call and ends up charging customers twice. The API returns 500 on timeouts, and the client retries blindly. How should the API and client coordinate so retries are safe?",
"expected_output": "A retry-safety design centered on idempotency: the API accepts an idempotency key from the client, stores the key with the result of the first attempt, and returns the stored result on any retry with the same key instead of processing again. The response specifies the client contract: generate a key per logical operation, reuse it on retries, and treat 5xx and network timeouts as retryable while 4xx are not. It also covers the API-side decisions: idempotency-key validity window, uniqueness enforcement under concurrency, and what happens when a key is replayed with a different payload, plus the timeout-error shape (409 or a dedicated response that lets the client know the outcome is being determined) so the client does not double-submit.",
"assertions": [
"The design uses client-supplied idempotency keys stored with the first attempt's result",
"Replayed keys return the stored result rather than re-processing",
"The client contract distinguishes retryable (5xx, network) from non-retryable (4xx) failures",
"Key validity, uniqueness under concurrency, and key-replay-with-different-payload are addressed",
"The timeout response shape prevents the double-submit race"
]
}
]
}
artifact-pyramids/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "artifact-pyramids",
"evals": [
{
"id": "pyramid-scaffold",
"prompt": "I am starting a research project on the competitive landscape of the observability market and need to produce durable, agent-consumable research artifacts. How do I scaffold the output as an artifact pyramid, and what goes in each layer?",
"expected_output": "A scaffold following the three-layer pyramid: an L1 summary file that states the research question, the key findings, and the most important implications with links down to the L2 analysis files; L2 analysis files per dimension (market sizing, competitor profiles, technical feasibility) that are self-contained and each link to L3 dossiers; and L3 dossiers holding the raw evidence: source excerpts, data tables, interview notes, and methodology notes. The response explains that production is top-down with recursive gap analysis: write the summary, embed links to analysis files, write analysis files that link to dossiers, and re-check for gaps after each round. It also requires the SOURCES navigation section at the bottom of every file with absolute paths and a description of what each deeper file contains.",
"assertions": [
"The scaffold defines L1 summary, L2 analysis collection, and L3 dossiers with distinct content contracts",
"L1 links down to L2 files and L2 files link down to L3 dossiers",
"Production is top-down with recursive gap analysis between layers",
"Every file carries the SOURCES navigation section with paths and descriptions",
"The layers are applied to the observability-market research topic with concrete example files"
]
},
{
"id": "l1-summary-authoring",
"prompt": "I have completed the research for our market-entry question and need to write the top-layer summary file. What belongs in an L1 summary, what must be excluded, and how do I know when the summary is complete?",
"expected_output": "An L1 summary written to the layer's contract: the research question and its scope, the key findings as conclusions with the most important implications for the decision, and nothing else — no raw data dumps, no methodology narration, and no findings that lack a home in an L2 file. The response explains the boundary rules: anything that is supporting evidence belongs in an L2 analysis or L3 dossier and is linked, not embedded; the summary must be independently consumable by an agent that reads only L1; and completion is checked by the quality gates for the layer: the question is answered, every finding is traceable to a linked L2 file, and no gap remains that the summary papered over.",
"assertions": [
"The L1 summary contains the research question, key findings, and implications only",
"Raw evidence and methodology detail are excluded and routed to L2 or L3 via links",
"The summary is independently consumable without reading the lower layers",
"Every finding is traceable to a linked L2 file",
"Completion is checked against the layer's quality gates, including uncovered gaps"
]
},
{
"id": "l2-analysis-sources",
"prompt": "I am writing the market-analysis layer of a pyramid about the developer-tooling market. Each analysis file must be self-contained and consumable. How do I structure one L2 file, and what is the SOURCES convention that keeps the pyramid navigable?",
"expected_output": "An L2 analysis file structured as a self-contained dimension analysis: the dimension's scope and the question it answers, the analysis with its conclusions, the caveats and uncertainty, and links down to the L3 dossiers holding the underlying evidence. The response demonstrates the SOURCES convention: a section at the bottom of the file listing each referenced dossier with its absolute path and a one-line description of what the consumer will find there, phrased to answer what the deeper file contains rather than just naming it. It explains why the navigation section matters: it is the affordance that lets a consuming agent decide whether to pull the next layer, and it is required on every file at every layer.",
"assertions": [
"The L2 file is structured around the dimension's question, analysis, conclusions, and caveats",
"It links down to the L3 dossiers that hold the evidence",
"The SOURCES section lists absolute paths with descriptions of what each deeper file contains",
"The response explains the navigation section as the agent's pull-decision affordance",
"The convention is applied with a concrete market-analysis example"
]
},
{
"id": "pyramid-audit",
"prompt": "I inherited a research output directory with a 00-index file, several markdown files, and a dump of raw interview transcripts, but nothing links to anything. I suspect it is not a valid artifact pyramid. How do I audit it and what do I fix?",
"expected_output": "An audit against the pyramid's structural contracts: the response checks layer presence (an L1 summary file, L2 analysis files, and L3 dossiers), checks the navigation mechanism (every file carries a SOURCES section with absolute paths), checks content placement (findings belong in L1, analysis in L2, raw evidence in L3 — not a 00-index stuffing findings meant for L1, and not raw transcripts masquerading as analysis), and checks link integrity between layers. The response produces a concrete remediation plan: promote or split the 00-index content into a proper L1 summary, reorganize the raw transcripts into dossiers, rewrite analysis files to be self-contained, and add the SOURCES sections and cross-links. It also notes the validation-script pitfall: structural checks pass on directory shape alone and do not prove content contracts, so the audit must read the files.",
"assertions": [
"The audit checks all three layers exist with the right content contracts",
"It verifies the SOURCES navigation and link integrity between layers",
"Misplaced content (findings in an index, raw transcripts as analysis) is specifically flagged",
"A concrete remediation plan reorganizes the inherited files into a valid pyramid",
"The response warns that directory-shape checks do not prove content compliance"
]
},
{
"id": "composite-synthesis",
"prompt": "I ran three subagent research teams in parallel — one on market, one on competitors, one on technical feasibility — and each returned its own pyramid. I need one root-level deliverable for the decision maker. How do I merge them into a composite pyramid?",
"expected_output": "A composite synthesis procedure: the root pyramid's L1 summary is built from the three subagent pyramids' L1 findings, with a per-dimension L2 analysis layer that each maps to the corresponding subagent pyramid as its evidence source, using the SOURCES convention to reference the subagent pyramids rather than copying their content. The response explains the orchestrator flow: define the root question, map each subagent pyramid to a dimension, verify each subagent pyramid is complete before synthesis, reconcile conflicting findings across teams explicitly rather than silently averaging them, and produce the root SOURCES sections pointing at each sub-pyramid with a description of what it contains.",
"assertions": [
"The root L1 is synthesized from the sub-pyramids' L1 findings per dimension",
"Each root L2 dimension references its subagent pyramid as the evidence source",
"Conflicting findings across teams are reconciled explicitly, not averaged silently",
"Sub-pyramids are verified complete before synthesis",
"The root SOURCES sections point at each sub-pyramid with descriptions"
]
}
]
}
cli-builder/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "cli-builder",
"evals": [
{
"id": "design-agent-friendly-cli",
"prompt": "We are building a new CLI that lets agents manage our deployment environments. I want it designed for AI agent consumption from the start. What are the design rules, and what should the first version of the command surface look like?",
"expected_output": "A CLI design following agent-consumption principles: non-interactive by default with every behavior driven by flags, machine-readable JSON output via a --json flag, an explicit --dry-run preview for any state-changing operation, idempotent commands that can be rerun safely, a clear and stable help surface with progressive discovery, and sensible default output for humans when JSON is not requested. The response sketches the concrete command surface for environment management (list, create, promote, destroy with --dry-run, --json, and confirmation gating on destructive ops), explains why an agent-friendly design omits interactive prompts and colored-only output, and specifies the exit-code and error-output contract an agent relies on.",
"assertions": [
"The design is non-interactive and flag-driven with every behavior reachable without prompts",
"--json machine-readable output and --dry-run preview are part of the core contract",
"Destructive operations require an explicit gate such as a confirmation flag",
"The response sketches a concrete command surface for the environment-management use case",
"The response specifies exit codes and stable error output that agents can rely on"
]
},
{
"id": "refactor-interactive-cli",
"prompt": "We have an existing CLI that asks 'Continue? (y/n)' before every action, prints tables, and exits 0 even when it fails. Agents keep hanging on the prompt or misreading success. How do I refactor it for agent use without rewriting everything?",
"expected_output": "A refactor plan that targets the specific agent-hostile behaviors: replace interactive confirmations with a --yes/--no-confirm flag while keeping the human default, add --json output alongside the human table, fix exit codes so failures are non-zero and errors go to stderr with a stable machine-readable error field, and add a --dry-run that shows what the command would do. The response prioritizes the changes by the failures they fix (prompt removal and exit codes first, JSON second) and shows how to keep backward compatibility for humans, and it includes a verification checklist: run every command with --help, confirm no command blocks on input, confirm exit codes are truthful.",
"assertions": [
"Interactive confirmations are replaced by a flag while human defaults are preserved",
"Exit codes are made truthful with errors on stderr in a stable format",
"--json and --dry-run are added alongside the human output",
"Changes are prioritized by the agent failures they fix",
"A verification checklist confirms no command blocks and exit codes are truthful"
]
},
{
"id": "json-output-contract",
"prompt": "I am adding --json to our status command. What makes JSON output good for agents? I have seen CLIs that dump raw API responses and call it JSON support. What should I actually do?",
"expected_output": "A JSON output design that treats the schema as a contract: a documented, stable, versioned schema with consistent field names and types, values that are normalized (timestamps in ISO-8601, enums spelled consistently, numbers not strings), an object at the top level that always contains the same envelope even for errors, and no stray human text mixed into stdout. The response explains why dumping the upstream API response is a trap (it couples agents to an unstable vendor schema and leaks internal fields), recommends a curated projection of the fields an agent actually needs, and specifies that errors under --json must be structured with a machine-readable code and message rather than only a stack trace. It also covers deterministic ordering and stable IDs so agents can diff outputs.",
"assertions": [
"JSON output is defined as a documented, stable schema with normalized types",
"The response warns against dumping raw upstream API responses and recommends a curated projection",
"Errors under --json are structured with a code and message, not only a trace",
"Output is deterministic with stable ordering and IDs so agents can diff",
"The envelope is consistent across success and failure cases"
]
},
{
"id": "dry-run-idempotency",
"prompt": "Our cleanup script deletes expired sessions when run, and an agent ran it twice and deleted sessions that were renewed between runs. I want --dry-run and idempotent behavior so this cannot happen again. How should I redesign the command?",
"expected_output": "A redesign with a dry-run that shows exactly what the command would change (computed from the current state, listing each session and why it qualifies) and a real run that is idempotent: qualifying sessions are selected and deleted by ID with a guard that re-checks the condition immediately before deletion, so a session renewed between the dry-run and the run is skipped. The response specifies the guard order (select by condition, re-verify per item, delete by ID), the --dry-run exit code and output contract, a --force or confirmation gate for the destructive path, and a rerun test proving the second run reports nothing left to do.",
"assertions": [
"--dry-run computes and displays the exact changes from current state",
"The destructive run re-verifies each item's condition before deleting by ID, preventing stale deletions",
"The response specifies the gate between dry-run and real execution",
"Idempotency is proven by a rerun that reports nothing left to do",
"The redesign addresses the specific race that caused the double-deletion incident"
]
},
{
"id": "debugging-agent-cli-failures",
"prompt": "An agent keeps failing to use our CLI: sometimes it passes flags the CLI does not have, other times it misreads the output, and occasionally it calls the wrong subcommand entirely. How do I debug this and make the tool easier to use correctly?",
"expected_output": "A debugging approach that looks at the tool surface before blaming the agent: the response walks through the failure modes and their tool-side causes — invented flags and wrong subcommands are usually a discoverability problem (help text not surfacing the real command tree, ambiguous names, missing examples), misread output is usually a formatting problem (tables that break parsers, progress bars, no JSON mode, colors obscuring values). The response prescribes concrete fixes: a complete and correct --help with examples for each subcommand, unambiguous naming, stable JSON output with documented fields, and a strict mode that errors on unknown flags instead of silently ignoring them, plus a test harness that replays the agent's failing invocations against the CLI to confirm the fixes.",
"assertions": [
"The response maps each failure mode to a tool-side cause rather than blaming the agent",
"Discoverability fixes include complete help, examples, and unambiguous naming",
"Output readability fixes include JSON mode and removing parser-hostile formatting",
"Unknown flags are rejected in strict mode rather than silently ignored",
"A replay harness verifies the fixes against the agent's actual failing invocations"
]
}
]
}
data-architect/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "data-architect",
"evals": [
{
"id": "quickscan-assessment",
"prompt": "My team's data pipelines keep breaking, the cloud bill is climbing without explanation, and nobody agrees on what 'customer' means across our reports. I do not know where to start. Run a quick scan of our data organization and tell me what the top gaps are.",
"expected_output": "A quick-scan assessment that turns the symptoms into a structured gap list: the response walks the five-minute scan over the common failure areas — pipeline reliability (how data is loaded, where failures happen and whether they are detected), cost (where spend concentrates, whether compute is runaway or rightsized), definitions (whether 'customer' and other core entities are defined once or redefined per report), ownership (who owns each pipeline and what happens when it breaks), and trust (whether anyone can trace a number on a dashboard to its source). It maps each symptom to the likely root gap: breaking pipelines to missing ownership and observability, climbing bills to ungoverned compute, and the customer disagreement to a missing shared semantic layer. It ends with the prioritized gaps and the first concrete action for each.",
"assertions": [
"The scan covers pipeline reliability, cost, definitions, ownership, and trust",
"Each symptom is mapped to a likely root gap rather than a superficial fix",
"The 'customer' disagreement is tied to a missing shared definition layer",
"The output is a prioritized gap list with a first concrete action per gap",
"The response reflects a quick scan and names the deeper review each gap needs"
]
},
{
"id": "pipeline-architecture-review",
"prompt": "We ingest events from our app into a warehouse through a chain of scripts, transform them in the database, and export dashboards. The pipeline works but takes 14 hours and fails often. Review this architecture and tell me what should change.",
"expected_output": "An architecture review that evaluates the pipeline against the properties that matter: the response assesses the current state (script-based orchestration, in-database transforms, batch frequency) and identifies the structural weaknesses — fragile orchestration without retries and monitoring, transformations that run in the serving database and scale poorly, no incremental processing so the 14-hour runtime grows with data volume, and no data-quality checks between stages so failures surface downstream. It proposes the target shape: orchestration with retries and observability, staging and transform layers that separate raw, cleaned, and modeled data, incremental or partitioned processing to bound runtime, and quality gates at each stage. The review is prioritized: the changes that reduce failure and runtime land first, and it identifies which parts of the current architecture can stay (the serving layer, the dashboards) while the plumbing is reworked.",
"assertions": [
"The review identifies the structural weaknesses: fragile orchestration, in-DB transforms, no incremental processing",
"It proposes a layered target with raw, cleaned, and modeled stages and quality gates",
"Incremental or partitioned processing is prescribed to bound the runtime",
"Changes are prioritized by failure-and-runtime reduction",
"Working parts such as the serving layer are retained rather than rewritten wholesale"
]
},
{
"id": "platform-decision-framework",
"prompt": "We need a data platform and are torn between using our existing Postgres for everything, adopting a cloud warehouse, and a newer lakehouse stack. The team has different opinions and the vendors are pushing hard. How do I make this decision properly?",
"expected_output": "A decision framework that defers the platform choice until the requirements are understood: the response identifies the decisions the platform must serve — the workloads (analytics, ML feature access, real-time versus batch), the data volumes and concurrency, the team's skills and operating capacity, and the future direction (lakehouse expansion, streaming). It frames the comparison across the named options on those requirements, including the total cost of ownership (licensing, compute, storage, and the people cost of operating each), and it exposes the vendor-pressure dynamic by grounding the choice in the workload evidence rather than platform enthusiasm. The framework produces a recommendation with the conditions under which the other options would win, and a pilot or proof-of-value step before commitment. It explicitly warns against picking the platform to avoid a later decision.",
"assertions": [
"The framework defines the workloads, volumes, concurrency, and team capacity before comparing platforms",
"Options are compared on total cost of ownership including operating people-cost",
"The choice is grounded in workload evidence rather than vendor momentum",
"The recommendation includes the conditions under which each alternative would win",
"A pilot or proof-of-value step precedes the commitment"
]
},
{
"id": "semantic-layer-governance",
"prompt": "Marketing reports revenue one way, finance reports it another, and the two numbers are different by 12%. I need to fix the definitions and stop the argument. How do I set up a governed semantic layer without freezing all data work?",
"expected_output": "A semantic-layer governance design that treats definitions as owned artifacts: the response establishes a single source of truth for core metrics (revenue, customer, active user) with a written definition, the calculation, and the owner for each, and it explains the governance model — definitions change through a review process with recorded rationale rather than per-report improvisation. The migration path keeps work moving: the semantic layer is introduced for the disputed metrics first, reports are migrated one at a time with a comparison period showing the old and new numbers side by side, and the legacy report is retired only when it matches. The response explains the 12% difference by identifying the likely divergence points (inclusion of refunds, definition of the reporting period, deduplication rules) and prescribes documenting those as part of the definition.",
"assertions": [
"Core metrics get a written definition, calculation, and named owner",
"Definitions change through a review process, not per-report improvisation",
"Migration is incremental with side-by-side comparison before retiring the legacy report",
"The 12% divergence is diagnosed against likely divergence points such as refunds and period definitions",
"The governance model prevents the argument from restarting without freezing data work"
]
},
{
"id": "strategy-roadmap",
"prompt": "Our data team spends all its time firefighting broken pipelines and has no time to build the analytics the business is asking for. Leadership wants a data strategy. What should the strategy and roadmap contain, and how do we get out of firefighting?",
"expected_output": "A data strategy that addresses the firefighting trap structurally: the response frames the strategy around the outcomes the business needs and the capabilities required, then sequences a roadmap that first stabilizes the foundation — ownership for the broken pipelines, observability so failures are detected and repaired fast, and the quick wins that stop the most frequent incidents — before adding net-new analytics. The roadmap is phased with explicit criteria for moving from one phase to the next: the firefighting load must fall below a threshold, not just a calendar date. It covers the governance and staffing implications (who owns the platform, how new requests are triaged), and it names the metrics that show the strategy working: incident rate, time-to-repair, on-time analytics delivery. The response resists a roadmap that schedules the new analytics first while the foundation stays broken.",
"assertions": [
"The strategy is organized around business outcomes and required capabilities",
"The roadmap sequences foundation stabilization before net-new analytics",
"Phase transitions are gated on firefighting-load criteria, not calendar dates",
"Ownership, triage, and governance implications are addressed",
"The metrics showing the strategy works are named, including incident rate and delivery"
]
}
]
}
data-engineering/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "data-engineering",
"evals": [
{
"id": "incremental-load-pipeline-design",
"prompt": "We load a 50 GB orders table from Postgres into our warehouse nightly with a full refresh, and it now takes four hours and is starting to collide with business hours. I want to move to incremental loading with dbt. How should I design this so it stays correct when source rows are updated or deleted, not just appended?",
"expected_output": "An incremental loading design that moves from full refresh to a dbt incremental model with a configurable lookback window. The design distinguishes append-only sources from mutable ones: for append-only data a simple incremental filter on an updated_at or event timestamp works; for mutable rows it combines a timestamp-based incremental window with a full-refresh fallback or a merge strategy (incremental_strategy='merge') keyed on the natural key, or uses a CDC capture layer when upstream changes are frequent. The design addresses idempotency of reruns, backfill procedures when the window logic changes, and a data-quality check that row counts reconcile with the source between full refreshes.",
"assertions": [
"The response recommends incremental models with a configurable lookback window instead of nightly full refresh",
"The response distinguishes append-only sources from mutable sources and picks a merge or CDC strategy for updates and deletes",
"The response covers idempotent reruns and a backfill procedure when the incremental window changes",
"The response includes a reconciliation or row-count check that catches silent data drift",
"The response gives a concrete dbt pattern such as incremental_strategy or incremental_predicates rather than hand-waving"
]
},
{
"id": "schema-migration-plan",
"prompt": "We need to split a users table into users and profiles in our production Postgres database, and the analytics warehouse reads the same table. Several services write to it. How do I plan this migration so the change is safe, reversible, and does not break downstream consumers?",
"expected_output": "A schema migration plan following expand-contract (parallel change): first add the new profiles table and backfill it while writes continue to users; then update writers to dual-write and readers to read from the new structure behind a flag; run a validation job comparing the two paths; finally cut over and drop or freeze the legacy columns. The plan includes a rollback path at each stage, uses transactional DDL where the platform allows it or staged changes otherwise, coordinates with the warehouse sync to avoid mid-migration loads, and names the owners and timing for each step.",
"assertions": [
"The response uses an expand-contract or parallel-change pattern rather than a single destructive migration",
"The response sequences the change: additive schema, dual-write, backfill, cutover, then cleanup",
"The response includes validation between old and new paths and a rollback path at each stage",
"The response coordinates downstream consumers such as the warehouse to avoid loading inconsistent state",
"The response names owners and timing for cutover and legacy-column removal"
]
},
{
"id": "data-quality-monitoring",
"prompt": "Our dashboards recently showed impossible numbers: negative revenue, suddenly empty customer tables, and a 3x spike in distinct users. We have no data-quality monitoring today. What should I set up so these problems are caught at load time rather than discovered by the CEO?",
"expected_output": "A data-quality monitoring design with automated checks at pipeline boundaries: schema and null-rate checks, uniqueness and primary-key checks, freshness (staleness) checks on timestamp columns, row-count anomaly detection against a rolling baseline, and distribution tests for critical metrics (range checks, negative-value detection, ratio sanity like revenue-to-orders). The design wires these checks into the pipeline as gate or warn steps with owners on the failing run, produces a daily quality report, and distinguishes hard failures from anomalies that need human review. It also covers backfilling checks on historical data to find where the breakage started.",
"assertions": [
"The response defines automated checks at pipeline boundaries: freshness, null rates, uniqueness, row-count anomalies",
"The response includes distribution and range checks that catch negative revenue and impossible spikes",
"The response wires checks as gate or warn steps with clear owners on failure",
"The response includes anomaly detection against a rolling baseline rather than only fixed thresholds",
"The response covers backfilling checks on history to locate when data broke"
]
},
{
"id": "sql-analytical-pattern",
"prompt": "I need to compute a weekly retention cohort in SQL over our events table (event_time, user_id, event_name) with columns: signup week, week 0, week 1, week 2 retention. The events table has 300 million rows. How should I write this so it runs in reasonable time?",
"expected_output": "A SQL pattern that computes the cohort table from first-event timestamps rather than scanning all events repeatedly: derive each user's signup week in a CTE, join events back to the signup week, compute weeks_since = date difference bucketed per user-week, and pivot with conditional aggregation. The response includes an incremental or filtered-scope recommendation (only events after cohort start), an index or partition hint for the join columns, and verification steps that the cohort numbers reconcile with a hand-checked subset. It should avoid naive correlated subqueries per cohort and explain the cost difference.",
"assertions": [
"The response derives the signup week once in a CTE and joins events to it rather than scanning per cohort",
"The response computes weeks-since-signup and pivots with conditional aggregation",
"The response limits the scan scope by filtering events after cohort start or using partitions",
"The response includes reconciliation checks against a manually computed subset",
"The response explains the cost and why the naive per-cohort approach is slow"
]
},
{
"id": "vector-database-selection",
"prompt": "We want to add semantic search over 10 million product descriptions for an internal assistant. I see options like pgvector on our existing Postgres, Pinecone, Qdrant, and Weaviate. We already run Postgres in production. How should we choose, and what is the simplest first step?",
"expected_output": "A storage selection recommendation that treats the choice as workload-driven: starts with pgvector on the existing Postgres because it keeps operational surface small, supports hybrid search with existing metadata filters, and handles 10 million vectors comfortably if the index is chosen correctly (HNSW with tuned m/ef_construction), with the caveat that dedicated vector stores add value only when scale, availability, or specialized filtering demands outgrow Postgres. The design includes the migration path, embedding model and dimension choice, index build strategy, and an evaluation harness measuring recall and latency on a labeled set before committing.",
"assertions": [
"The response evaluates the choice against the workload rather than assuming a dedicated vector database is needed",
"The response recommends starting with pgvector on existing Postgres for a small operational surface and explains when to outgrow it",
"The response covers index choice (HNSW parameters) and dimension/embedding-model considerations",
"The response includes an evaluation harness with labeled queries measuring recall and latency",
"The response lays out a migration path from the first step to a dedicated store if needed"
]
}
]
}
data-scientist/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "data-scientist",
"evals": [
{
"id": "ab-test-design-power",
"prompt": "We want to test a new onboarding flow that we believe will increase activation rate from 20% to 22%. How many users do we need in the experiment, how long should it run, and what analysis should we do at the end? I want to be rigorous and avoid a false-positive-driven launch.",
"expected_output": "An experiment design that starts by stating the unit of randomization (user), the metric (activation rate), and the minimum detectable effect (2 percentage points), then computes required sample size per arm using a standard two-proportion power calculation (alpha 0.05, power 0.8), accounting for multiple metrics and multiple variants with a correction if applicable. It covers duration planning from expected daily traffic plus a buffer for novelty effects and seasonality, pre-registers the primary metric and stopping rule, and prescribes the analysis: check sample ratio mismatch, compute confidence interval on the effect, run sensitivity checks, and distinguish statistical significance from practical significance before launch.",
"assertions": [
"The response specifies the randomization unit, primary metric, baseline rate, and minimum detectable effect before computing sample size",
"The response computes sample size with stated alpha, power, and a two-proportion formula",
"The response plans duration from daily traffic including buffers for novelty and seasonality",
"The response pre-registers the primary metric and stopping rule and checks for sample ratio mismatch",
"The response distinguishes statistical from practical significance before recommending launch"
]
},
{
"id": "causal-inference-vs-correlation",
"prompt": "Sales data shows customers who attend our webinars churn 40% less than those who do not. My boss wants to make webinars the centerpiece of the retention strategy based on this. Is that justified, and what would it take to actually establish causality?",
"expected_output": "A response that resists the correlational conclusion: webinar attendees differ systematically from non-attendees (they are more engaged, more likely to be on certain plans, earlier in lifecycle), so the naive comparison suffers from selection bias and confounding. It proposes the hierarchy of evidence for the question: randomized encouragement designs, natural experiments or instrument variables, difference-in-differences using a roll-out, or propensity-score/regression adjustments as weaker alternatives, and specifies what data would be needed to support each. It also states what analysis should be run now to quantify the selection bias (compare observables between groups) before any investment decision.",
"assertions": [
"The response flags selection bias and confounding as the core problem with the observed comparison",
"The response explains why attendees differ systematically from non-attendees and how that undermines the causal claim",
"The response proposes an identification strategy such as randomized encouragement, diff-in-diff, or instrumental variables",
"The response includes a near-term analysis comparing observables between groups to quantify selection",
"The response does not endorse the webinar strategy on the correlation alone"
]
},
{
"id": "model-selection-task",
"prompt": "We need to predict which accounts will churn in the next 30 days so our sales team can intervene. We have 40k accounts, 120 features with lots of missing values, class imbalance (about 5% churn), and the team has been tuning XGBoost for weeks. How should I frame model selection here, and what should drive the final choice?",
"expected_output": "A model-selection framing that leads with the business decision context: churn prediction is a ranking task for intervention, so evaluation should use recall-at-k or precision-at-k at the intervention capacity, not raw accuracy on an imbalanced set. It recommends a baseline (logistic regression or simple heuristic) before complex models, a proper train/validation/test split that respects time ordering (no random split leaking future information), handling of missingness that is validated rather than assumed, and a cost-aware threshold choice based on the cost of a false positive versus a missed churner. The response compares the XGBoost candidate against baselines with the ranking metric and states that the choice is justified by validated lift, not tuning effort.",
"assertions": [
"The response reframes evaluation around ranking metrics (recall-at-k or precision-at-k) tied to intervention capacity",
"The response mandates a time-respecting split rather than a random split",
"The response requires a simple baseline before accepting the tuned model",
"The response treats threshold choice as cost-aware, weighing false positives against missed churners",
"The response rejects accuracy as the evaluation metric on an imbalanced set"
]
},
{
"id": "bayesian-vs-frequentist",
"prompt": "We ran an experiment and the frequentist analysis says the effect is not significant (p=0.09). A colleague says we should switch to a Bayesian analysis because it will let us conclude there is a high probability the change is positive. Is that a valid reason to switch analysis methods?",
"expected_output": "A response that distinguishes the legitimate from the illegitimate uses of Bayesian analysis: switching after peeking because the frequentist result is not convenient is p-hacking by another name, and a Bayesian analysis with a flat prior run after the fact will not manufacture evidence. It explains that a Bayesian approach can add value when designed up front: an informative prior based on prior experiments, a decision rule on the posterior (P(effect > 0) and expected loss), and sequential monitoring that is principled. It notes that the two frameworks answer different questions and that the analysis choice must be pre-registered, and it shows how to compute the posterior probability of a positive effect and the posterior probability of a practically meaningful effect from the observed data.",
"assertions": [
"The response flags switching methods after seeing the p-value as post-hoc analysis rather than principled",
"The response explains that a flat-prior Bayesian analysis run post hoc does not create evidence",
"The response describes when Bayesian analysis is genuinely useful: informative priors, decision rules, principled sequential monitoring",
"The response distinguishes the question each framework answers and requires pre-registration of the analysis plan",
"The response computes or specifies computing P(effect > 0) and the posterior probability of a meaningful effect"
]
},
{
"id": "analysis-report-uncertainty",
"prompt": "I ran a regression analysis on customer spend and found a coefficient for the new pricing plan of +$12/month. I need to write a report for leadership. What should the report contain beyond the coefficient, and how should I communicate the uncertainty?",
"expected_output": "An analysis report structured for decision-makers: the question and the decision it informs, the data and its limitations, the model and its key assumptions stated plainly, and the estimate with a confidence interval rather than a single point, expressed in decision-relevant language (range of plausible effects, probability of the effect being positive or economically meaningful if a Bayesian interpretation is used). The report discloses confounders and omitted-variable risk, checks robustness (alternative model specifications, sensitivity to outliers), and ends with what would change the conclusion. It avoids overprecision and states clearly what is measured versus assumed.",
"assertions": [
"The response structures the report around the decision the analysis informs",
"The response communicates the estimate with a confidence interval rather than a single point",
"The response discloses model assumptions, confounders, and omitted-variable risk",
"The response includes robustness checks such as alternative specifications or sensitivity to outliers",
"The response states what is measured versus assumed and what would change the conclusion"
]
}
]
}
docker-compose/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "docker-compose",
"evals": [
{
"id": "compose-file-design",
"prompt": "I need a compose.yaml for a web service with a Postgres database and a Redis cache, with a worker that consumes the service's job queue. The developer needs this to run locally with one command. What does the compose file look like and what pitfalls should I avoid?",
"expected_output": "A compose.yaml design with separate services for the app, database, cache, and worker; healthchecks on the database and app so dependencies wait correctly (depends_on with condition service_healthy rather than start-order alone); named volumes for Postgres and Redis persistence so data survives restarts; environment configuration via interpolation with sensible defaults in a .env file; and an explanation of the pitfalls: hardcoding secrets in the file, no version pinning on images, ports colliding with other local projects, and the worker being defined as a separate service so it does not get started twice. The response also covers how the app service reaches the database by service name over the default network.",
"assertions": [
"The compose file defines app, database, cache, and worker as separate services",
"Healthchecks and depends_on condition service_healthy are used instead of bare start ordering",
"Named volumes persist database and cache data across restarts",
"Secrets are not hardcoded and configuration uses interpolation",
"The response explains service-name DNS and the port-collision pitfall"
]
},
{
"id": "networking-design",
"prompt": "My compose stack has three services that should talk to each other, one service that must NOT be reachable from outside the container network, and one that should be exposed on a specific port. How do I design the networking?",
"expected_output": "A networking design that explains Compose's default network and when custom networks are needed: services on the same network resolve each other by service name; a database or internal worker should not publish ports to the host (no ports: mapping, or only internal network membership), while the public-facing service publishes the desired host port. The response shows how to define custom networks to segment services (an internal network for db and worker, a frontend network for the public service and any proxies), how aliases and static IPs are avoided in favor of service names, and how to verify reachability between containers. It covers the security point that publishing ports is opt-in and that services left unpublished are unreachable from the host and the internet.",
"assertions": [
"The response explains service-name DNS on the default Compose network",
"Internal services are kept off published ports and segmented onto an internal network",
"Only the public service publishes a host port",
"Custom networks are used to segment internal from external-facing services",
"Reachability verification between containers is included"
]
},
{
"id": "secrets-management",
"prompt": "My compose file currently has the database password and API keys written directly in the YAML, committed to git. The stack runs locally and in CI. How do I move secrets out of the file while keeping the developer experience smooth?",
"expected_output": "A secrets strategy appropriate to Compose's deployment range: environment interpolation from a gitignored .env file for local development, Compose's top-level secrets with file-based secrets for services that support them (file_mount target paths, appropriate uid/gid), and a clear statement that plaintext secrets in the YAML or in images must be removed. The response shows the migration: create the .env file with placeholder keys, reference ${VAR} with defaults where safe, add .env to .gitignore, keep a .env.example documenting the required variables, and for CI show injecting secrets through the CI provider rather than the repository. It flags the residual risk that environment variables are visible in process listings and container inspect, and notes where file-based secrets are the stronger option.",
"assertions": [
"Secrets are moved to a gitignored .env with ${VAR} interpolation and a documented .env.example",
"File-based Compose secrets are used for services that support them",
"The migration removes plaintext secrets from YAML and images",
"CI secrets are injected via the CI provider, not committed",
"The response notes the residual exposure of env-based secrets and where file secrets are stronger"
]
},
{
"id": "profiles-overrides",
"prompt": "I have one compose project used for development and production. Dev needs hot reload, a fake mail server, and exposed debug ports; prod needs none of that and should be lean. How do I structure the compose files so dev and prod stay in sync but behave differently?",
"expected_output": "A structure using the base compose.yaml for the shared topology plus override layers: a compose.override.yaml with dev-specific settings (bind mounts for hot reload, exposed debug ports, the fake mail service) that Compose applies automatically in development, and a compose.prod.yaml or explicit -f invocation for production that keeps the same services but removes dev conveniences. The response explains profiles as the alternative for selectively starting optional services, how to verify the merged effective configuration (docker compose config), and the rule that the base file stays the source of truth for the topology so dev and prod cannot drift into different stacks.",
"assertions": [
"The response uses a base compose.yaml plus an override layer for dev-specific settings",
"Production uses an explicit file or invocation without dev conveniences",
"Profiles are shown for selectively started optional services",
"docker compose config is used to verify the merged effective configuration",
"The base file remains the source of truth so dev and prod do not drift"
]
},
{
"id": "troubleshooting-failing-stack",
"prompt": "My compose stack was working yesterday and today `docker compose up` fails: the app container exits immediately with an error I do not understand, and the database looks fine. Walk me through diagnosing this systematically.",
"expected_output": "A troubleshooting procedure that gathers the actual state before guessing: check which services are up with docker compose ps, read the failing container's logs (docker compose logs app) for the real error, inspect the exit reason with docker compose ps and container inspect, and test the dependency assumption directly (can the app reach the database by service name — network check, credentials, schema). The response distinguishes the common causes: the app's error message is the primary evidence and should be read before restarting; environment changes since yesterday (the .env file, image tags, ports in use, database data volume state) are the likely regression source. It prescribes reproducing after each fix and verifying the stack is healthy rather than just running.",
"assertions": [
"The response starts with state gathering: ps, logs of the failing container, and the real error",
"The dependency assumption is tested directly, such as service-name connectivity and credentials",
"Changes since the last working run are checked as the likely regression source",
"Container inspect or exit reasons are examined before restarting",
"Fixes are verified by reproducing and confirming a healthy stack"
]
}
]
}
go-to-market/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "go-to-market",
"evals": [
{
"id": "dunford-positioning",
"prompt": "We are a workflow-automation product for finance teams and have been describing ourselves as 'the intelligent automation platform for modern finance.' Sales says prospects do not understand what we do. How do I build a proper positioning statement using the Dunford method?",
"expected_output": "A positioning statement built through the April Dunford steps rather than adjectives: first identify the real competitive alternatives the prospect is comparing (spreadsheets and manual work, their existing ERP tooling, internal scripts, or a named competitor), then isolate the unique attributes that actually beat those alternatives, weigh the value of those attributes for the target segment, and identify the best-fit customer for whom those attributes matter most. The response rewrites the vague 'intelligent automation platform' claim into a positioning that names the category context, the target, the alternatives, the differentiators, and the value in prospect terms, and it explicitly rejects features-dump positioning in favor of the few attributes that win the comparison.",
"assertions": [
"The response follows the Dunford steps: alternatives, unique attributes, value, best-fit customer",
"The positioning names the real competitive alternatives prospects compare against",
"The vague 'intelligent automation platform' framing is replaced with a concrete category and differentiator",
"The response rejects feature-dump positioning and limits claims to the winning attributes",
"The resulting statement names target, alternatives, differentiators, and value in prospect terms"
]
},
{
"id": "message-hierarchy",
"prompt": "Our marketing site lists 14 features and every campaign explains all of them. I want to introduce a message hierarchy so every asset tells the same story. How do I build one and what goes in each level?",
"expected_output": "A message hierarchy built top-down from the positioning: the core value proposition at the top that every asset must communicate, the key messages that support it (the few claims that differentiate and convert), and the proof points and feature-level details that substantiate those messages at the bottom. The response explains the discipline: assets lead with the value proposition, each supporting message is backed by at least one proof point, feature lists are demoted below the story, and the hierarchy is used to triage new content (if an asset cannot fit the hierarchy, the asset is wrong, not the hierarchy). It applies the structure to the product with concrete examples for each level.",
"assertions": [
"The hierarchy has distinct levels: value proposition, supporting messages, proof points",
"Each supporting message is backed by proof points",
"Feature lists are demoted below the story rather than leading",
"The hierarchy is used to triage new content so assets stay consistent",
"Concrete examples are given for each level for the product at hand"
]
},
{
"id": "plg-vs-slg-strategy",
"prompt": "We are a developer tool with a free tier that grows by word of mouth, and management is pressuring us to build an enterprise sales team because the biggest accounts are not self-serving. How should I think about PLG versus SLG, and what is the hybrid that actually works here?",
"expected_output": "A PLG-versus-SLG analysis that does not treat them as a binary: the response maps which motions are working (self-serve adoption for the long tail, product-qualified leads emerging from usage data) and which gap the sales team is meant to close (procurement, security review, custom contracts, larger deployments). It recommends a product-led acquisition with sales-assisted expansion: the free tier and onboarding stay self-serve, sales engages product-qualified accounts at the moment usage signals indicate expansion potential, and the handoff is defined by signals, not arbitrary account size. It defines the operating metrics for the hybrid: self-serve activation, PQL qualification rate, sales cycle on assisted deals, and expansion revenue, and it warns against bolting on a sales team that sells before the product-led motion has data to route leads on.",
"assertions": [
"The response treats PLG and SLG as complementary motions rather than an either-or choice",
"The hybrid is defined as product-led acquisition with sales-assisted expansion",
"The sales handoff is driven by product-qualified lead signals, not arbitrary account size",
"Operating metrics are defined for both the self-serve and assisted motions",
"The response warns against selling before the product-led data can route leads effectively"
]
},
{
"id": "growth-modeling-cac-ltv",
"prompt": "We spend across paid search, content, and partnerships and have no idea which channel actually pays back. I want a growth model that allocates budget by channel. What should it contain and what data do I need to feed it?",
"expected_output": "A channel-level growth model with the economics made explicit per channel: CAC (blended and by channel with a consistent attribution definition), payback period against gross margin, LTV computed from cohort retention curves rather than an arbitrary multiple, and contribution per channel. The response distinguishes first-touch attribution noise from the decisions the model actually supports, requires cohort-based LTV so churn is represented honestly, and frames the allocation rule: budget flows to channels within payback constraints, with a test budget for uncertain channels and a guardrail that total spend respects cash and margin. It lists the exact data required (cost by channel, signups by channel, cohort retention, price and margin) and flags which inputs are estimates needing validation.",
"assertions": [
"CAC and LTV are defined with consistent attribution and cohort-based retention",
"Payback period against gross margin is part of the model",
"Budget allocation follows an explicit rule with payback constraints and a test budget",
"The response names the exact data inputs and which are estimates needing validation",
"Total spend is governed by margin and cash guardrails, not channel enthusiasm"
]
},
{
"id": "beachhead-market-entry",
"prompt": "We are a compliance-automation startup deciding whether to expand from fintech into healthcare and manufacturing simultaneously. I think we should pick one beachhead first. How do I make that call and what does a land-and-expand plan look like for the chosen segment?",
"expected_output": "A beachhead analysis that picks one segment by the criteria that matter for a land-and-expand motion: a segment where the product already solves a painful problem without major adaptation, where a referenceable cluster of customers exists, where the buying motion is repeatable, and where adjacent expansion potential is high. The response weighs fintech, healthcare, and manufacturing against those criteria and recommends one, then builds the land-and-expand plan for it: how to win the first reference accounts, the expansion sequence from initial use case to broader footprint within the segment, and the repeatable playbook that will be transferred to the next segment. It explicitly defers the second segment with the trigger conditions for entering it, so expansion is sequenced rather than simultaneous.",
"assertions": [
"The beachhead is chosen against explicit criteria: pain fit, referenceable cluster, repeatable buying, expansion potential",
"The response weighs the named segments against the criteria and picks one",
"The plan covers winning reference accounts and the expansion sequence within the segment",
"The playbook is designed to be repeatable and transferable to the next segment",
"Entry into the second segment is deferred with stated trigger conditions"
]
}
]
}
kubernetes/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "kubernetes",
"evals": [
{
"id": "crashloopbackoff-diagnosis",
"prompt": "A pod in my cluster is in CrashLoopBackOff: it starts and dies every few seconds. I ran kubectl get pods and see the state but nothing else. How do I diagnose this systematically, from the fastest checks to the deeper ones?",
"expected_output": "A systematic CrashLoopBackOff diagnosis ordered by evidence value: describe the pod (kubectl describe pod) to see events, restart counts, and why it is being killed; read the container logs (kubectl logs with the previous container flag for the dying attempt) for the actual error; distinguish the failure classes — application error at startup (bad config, missing env var, crash in code), readiness or liveness probe failures killing a healthy app (probe path, port, or timeout wrong), resource limits being exceeded (OOMKilled in the container status), and image or command problems (imagePullBackOff, wrong entrypoint). The response then maps each class to its fix: fix the config or code for app errors, correct the probe definition for probe failures, raise limits or fix the memory profile for OOM, and check image tags and pull secrets for image errors. It prescribes verifying the fix by watching the pod reach Running and Ready.",
"assertions": [
"Diagnosis starts with describe and logs, including the previous container's logs",
"Failure classes are distinguished: app crash, probe failures, OOM, image problems",
"Each class is mapped to its specific fix",
"Probe-related deaths are identified as a distinct class where the app may be healthy",
"Verification is by watching the pod reach Running and Ready"
]
},
{
"id": "rbac-networkpolicy-design",
"prompt": "I am deploying a three-tier app: frontend, API, and database. I want least-privilege access: the frontend may talk only to the API, the API only to the database, and nothing external may reach the database. How do I implement this with Kubernetes RBAC and NetworkPolicy?",
"expected_output": "A design that uses NetworkPolicy as the primary isolation mechanism and RBAC for control-plane access: the response defines a deny-by-default posture with policies that allow the specific flows (frontend to API on the API port, API to database on the database port) using pod selectors and ports, and a policy for the database that only the API pods can reach. For RBAC it specifies the identities and roles: separate service accounts per tier, role bindings scoped to what each workload needs, and the principle that pods get credentials only through their own service accounts. The response explains the practical gotchas: NetworkPolicy is enforced by the CNI (default-deny requires explicit policies, and a namespace with no policies allows all), ingress and egress policy fields work independently, and selector-based policy must match the actual pod labels. It prescribes verifying with a connectivity test between tiers and from outside.",
"assertions": [
"NetworkPolicy is used for data-plane isolation with a deny-by-default posture",
"Policies specify exact pod selectors and ports for each allowed flow",
"RBAC uses per-tier service accounts with scoped role bindings",
"The response explains that NetworkPolicy enforcement depends on the CNI and default behavior",
"Verification via connectivity tests between tiers and from outside is prescribed"
]
},
{
"id": "ingress-routing-troubleshoot",
"prompt": "Traffic to my service works when I port-forward but returns 503 through the Ingress. The ingress controller is running. What should I check to find where the path breaks?",
"expected_output": "A routing diagnosis that walks the path layer by layer: check the ingress resource itself (host and path rules match the request, the service name and port in the ingress backend are correct, annotations are valid), check the service (selector matches the pod labels, the targetPort exists, endpoints are populated — a service with no endpoints returns 503), and check the controller (ingress class matches the controller, the controller can reach the pods). The response distinguishes 503 from 404 (503 means the ingress found the route but the backend was unreachable, pointing at service endpoints, while 404 points at the routing rules), and it explains the port-forward-works-but-ingress-fails pattern: the service selector or namespace mismatch is the usual culprit since port-forward bypasses the service. It prescribes checking kubectl get endpoints as the fastest discriminator.",
"assertions": [
"The diagnosis walks ingress resource, service, and controller layers in order",
"The 503-versus-404 distinction is explained and used to narrow the cause",
"Service selector and endpoint population are checked as the prime suspect",
"The port-forward-works-but-ingress-fails pattern is explained",
"kubectl get endpoints is prescribed as the fast discriminator"
]
},
{
"id": "upgrade-planning",
"prompt": "We run a self-managed cluster on k3s with a few production workloads and are several minor versions behind. I want to plan an upgrade that does not take down the workloads. What does a safe upgrade plan look like?",
"expected_output": "An upgrade plan that treats version drift as the primary risk and sequencing as the control: the response starts by inventorying the current versions (server, kubelet, and the client tools) and the Kubernetes minor-version skew policy, checks the target version's deprecations against the workloads' API usage (verify the resource API versions the manifests use are still served), and upgrades in controlled stages: back up etcd state first, upgrade one node or a non-production cluster as a rehearsal, then roll the control plane and worker nodes, draining nodes before upgrades and uncordoning after. It includes the rollback path (restore backup, or downgrade within supported bounds) and verification at each stage: node versions, workload health, and the API-version compatibility checks. It also flags the common failure: upgrading the control plane without checking deprecated APIs breaks workloads after the upgrade, and using the in-place k3s install script without reading the release notes.",
"assertions": [
"The plan inventories versions and checks the skew policy before upgrading",
"Deprecated API usage in the workloads' manifests is checked against the target version",
"Upgrades are staged with etcd backup, a rehearsal environment, and drain-then-upgrade node rolling",
"A rollback path is defined for each stage",
"Verification at each stage covers node versions, workload health, and API compatibility"
]
},
{
"id": "autoscaling-rightsizing",
"prompt": "Our API pods run at 30% CPU average but the cluster sometimes spikes and the HPA scales to 20 replicas that mostly sit idle. I want autoscaling that matches demand without waste. How should I configure it?",
"expected_output": "An autoscaling and rightsizing design: the response starts with the workloads' actual profile — measure sustained CPU and memory percentiles over a representative period (not the average), set resource requests from the p95/p99 so the scheduler reserves honestly without over-reserving, and configure the HPA on the metric that reflects demand (CPU utilization relative to requests, or a custom metric such as request latency or queue depth if CPU is a poor proxy). It explains the HPA tuning knobs: min/max replicas chosen from the measured demand curve, target utilization set so it scales before latency degrades but not on noise, and the scaling-delay parameters (stabilization window, cooldown) that prevent the thrash between 5 and 20 replicas. The response also covers the interplay with cluster autoscaling: the HPA's ceiling must be reconcilable with node capacity, and idle spike behavior is addressed by the utilization target and scale-down stabilization rather than more replicas.",
"assertions": [
"Rightsizing starts from measured percentile CPU and memory, with requests set from the high percentile",
"The HPA metric is chosen to match demand, with custom metrics considered over CPU alone",
"Min/max replicas and target utilization come from the measured demand curve",
"Stabilization windows are used to prevent replica thrash",
"The HPA ceiling and cluster autoscaling are reconciled"
]
}
]
}
langgraph/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "langgraph",
"evals": [
{
"id": "pattern-selection",
"prompt": "We are building an agent that handles customer support tickets: it needs to classify the ticket, call a specialist tool for the issue type, and sometimes escalate to a human. I keep reading about supervisor patterns, swarm patterns, and hierarchical patterns. How do I choose the right orchestration pattern?",
"expected_output": "A pattern-selection analysis grounded in the workflow's structure rather than pattern-name enthusiasm: the response examines the workflow's control flow — the ticket must be routed by type, specialist sub-agents do bounded work, and human escalation is an interrupt — and maps it to the pattern that fits: a supervisor pattern where a router node decides the specialist and checks the result, with explicit state passing, rather than a free-running swarm where agents autonomously hand off work, because the workflow has a defined decision point and bounded sub-tasks. It explains the distinguishing questions: is the next step decided centrally (supervisor) or by agents themselves (swarm), does the graph have a fixed skeleton with choices (state graph with conditional edges) or recursive spawning (hierarchical), and where human-in-the-loop interrupts live. It prescribes sketching the control flow before choosing the pattern and names the gotcha: reaching for a swarm when the workflow is a deterministic pipeline adds nondeterminism and observability cost.",
"assertions": [
"The response maps the workflow's control flow to a concrete pattern choice",
"Supervisor and swarm patterns are distinguished by who decides the next step",
"Human-in-the-loop interrupts are placed in the design",
"The response warns against free-running swarm patterns for deterministic routed workflows",
"It prescribes sketching control flow before pattern selection"
]
},
{
"id": "state-schema-design",
"prompt": "I am designing a LangGraph agent that researches a topic, drafts a report, and revises it after review. The agents need to share the research findings and the draft, but I keep hearing that shared mutable state causes bugs in LangGraph. How should I design the state schema?",
"expected_output": "A state-schema design that separates the concerns of shared data from per-step data: the response explains LangGraph's state model — a shared state object that nodes annotate, with reducers controlling how updates merge — and prescribes modeling the fields the whole graph needs (the topic, the research findings, the draft, review feedback) with typed annotations and explicit reducers where messages or lists accumulate, while transient per-node data that should not persist stays local to the node. It explains the common bugs: using a plain list field without a reducer so each node overwrites prior messages, mutating shared state in place instead of returning updates, and stuffing node-local scratch data into the shared state where it pollutes downstream nodes. The response prescribes the reducer choice (add for accumulating lists, replace for single-value updates, and a custom reducer for merging dicts) and shows how to inspect the state at each step for debugging.",
"assertions": [
"The response separates graph-shared state from per-node transient data",
"Reducers are explained and prescribed for accumulating or merging fields",
"The overwriting-list bug and in-place mutation pitfall are called out",
"Typed annotations with reducer behavior are part of the design",
"State inspection per step is prescribed for debugging"
]
},
{
"id": "subgraph-composition",
"prompt": "My agent has three independent phases — research, drafting, and review — and each phase is itself a multi-node graph. I want to compose them so each phase stays reusable and testable. How do I structure this with subgraphs, and where do I get it wrong?",
"expected_output": "A subgraph-composition design where each phase is a self-contained graph with its own internal nodes and a narrow contract with the parent: the response prescribes defining each phase as a compiled subgraph whose input and output are explicit typed states, then composing them in the parent graph as single nodes that pass and receive only the agreed fields. It explains the common failure modes: coupling phases through the shared state by reading fields the phase does not own, making the subgraph's internal nodes reachable from outside (breaking encapsulation and making tests brittle), and mismatched state schemas between the parent and subgraph that surface as silent drops or type errors. The response covers testing each phase independently with its own fixtures and the parent test that verifies the handoff between phases, and it shows how the composition stays legible when each subgraph is treated as a node.",
"assertions": [
"Each phase is a self-contained subgraph with a narrow input-output contract",
"The parent graph composes subgraphs as single nodes passing only agreed fields",
"Encapsulation failures (external access to internal nodes, shared-state coupling) are flagged",
"State-schema mismatches between parent and subgraph are called out",
"Independent phase tests plus a handoff test are prescribed"
]
},
{
"id": "human-in-the-loop-interrupt",
"prompt": "My agent drafts an expense report and should pause for a human to approve it before submitting. If the approval fails or the reviewer edits the draft, the agent must revise and re-pause. How do I implement this interrupt pattern without losing the agent's state?",
"expected_output": "A human-in-the-loop implementation built on interrupts and checkpoints: the response prescribes using the graph's interrupt mechanism at the approval node so the graph pauses with its state intact, then resumes when the human decision arrives, with the checkpointing layer persisting the full state so a process restart resumes the same run. It covers the design decisions: the interrupt payload (what the human sees and the structured decision input they return), validating the resumed input before continuing, the branch after the interrupt (approved proceeds, rejected or edited returns to the revision node with the feedback added to state), and the guardrails against the loop spinning: an explicit revision budget with a cap on re-pauses. It also explains the debugging angle: after an interrupt, inspecting the checkpointed state is how you verify nothing was lost.",
"assertions": [
"The interrupt mechanism is used with checkpointing so state survives pauses and restarts",
"The interrupt payload and structured human decision input are designed",
"Resumed input is validated before the graph continues",
"The approve-revise-repause branch is implemented with a revision budget cap",
"Checkpointed state inspection verifies nothing is lost"
]
},
{
"id": "production-debugging",
"prompt": "Our LangGraph agent works in tests but in production it sometimes ends in the wrong node: a tool result is missing from state, and a conditional edge routes to the error path even though the tool succeeded. How do I debug state and routing issues in a running graph?",
"expected_output": "A debugging procedure that makes the graph's execution observable: the response prescribes inspecting the state and event stream at each step — using the graph's streaming or state-inspection facilities to see the exact state before and after each node, verifying what the tool call actually returned versus what the node wrote to state, and checking the conditional edge's routing function against that state to see why it chose the error path. It separates the failure classes: a node that did not write its output to state (missing field or wrong key), a reducer that overwrote a previous value, a conditional router reading the wrong field or applying the wrong predicate, and a tool result that never landed because the tool node errored or was skipped. The response prescribes reproducing with the production-shaped input, adding targeted logging at the state transitions, and writing a regression test that pins the routing decision.",
"assertions": [
"The debug procedure inspects state and events at each node transition",
"It verifies what the tool returned versus what the node wrote to state",
"Conditional routing functions are checked against the actual state",
"Failure classes are separated: missing writes, reducer overwrites, wrong routing field",
"A regression test pins the routing decision"
]
}
]
}
linear/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "linear",
"evals": [
{
"id": "smallest-read-query",
"prompt": "I need to list the open issues assigned to me across all teams in our Linear workspace, with their titles and states. The API can return huge amounts of data, so I want the smallest query that gets exactly what I need. What does the GraphQL query look like?",
"expected_output": "A minimal Linear GraphQL query that requests only the needed fields: a viewer query with the current user, issues filtered by assignee and state (filter for state.type equal to 'started' or 'unstarted' or by status name) with pagination handled via after and first, requesting only id, title, and state name (plus updatedAt if ordering matters). The response explains the field-selection discipline: requesting only the fields used keeps the response small and avoids GraphQL over-fetching, and it shows how pagination works with the issues connection (first and after, pageInfo.hasNextPage and endCursor) rather than assuming all results come back at once. It includes the practical details: authentication via an API key header and the base endpoint.",
"assertions": [
"The query requests only the needed fields (id, title, state) with the correct filter",
"Pagination via first and after with pageInfo is used correctly",
"The response explains why minimal field selection matters for GraphQL",
"Authentication and endpoint details for the Linear API are included",
"The query is scoped to the current user's assigned issues"
]
},
{
"id": "create-update-issue",
"prompt": "I want a CLI action that creates a Linear issue for a bug report and then moves it to the correct team and project, but I want to avoid creating duplicate issues when the action is run twice. How do I design the create-and-update operation safely?",
"expected_output": "A safe create-and-update design: the response recommends checking for an existing issue first (search by the deduplication key such as the bug title or a stored external ID) before creating, then creating with only the required fields (teamId, title, description) and updating state, assignee, or project in a follow-up mutation only if needed. It explains Linear's mutation pattern: mutations are issued with an input object and return the updated issue, and since creation is not inherently idempotent, the client must implement deduplication or store the created issue ID. It covers the state-change semantics: moving an issue between states uses the workflow's state IDs, and it prescribes verifying the result after the mutation by re-querying the issue rather than trusting the mutation response alone.",
"assertions": [
"The design checks for an existing issue before creating to avoid duplicates",
"Creation sends only required fields and updates are separate mutations",
"The response explains that Linear mutations are not inherently idempotent and deduplication is client-side",
"State changes use the workflow's state IDs",
"The result is verified by re-querying the created issue"
]
},
{
"id": "document-lookup",
"prompt": "A teammate shared a Linear document link with me, but I only remember the title fragment and that it lives in a project. I need to find the document and read its content from the CLI. What is the smallest set of queries to find and read it?",
"expected_output": "A document lookup path: first a search query against documents filtered by the title fragment and project (using the searchDocuments or documents connection with a filter on title), then, once the document id is known, a query for that document fetching the content field. The response explains the differences between Linear document types (project documents versus organization documents) and the fields available (title, content, project), and it prescribes reading only the content field needed for the task rather than fetching attachments and metadata. It also covers handling the not-found case: empty results mean the title fragment or project filter is wrong, and the response suggests loosening the filter before concluding the document does not exist.",
"assertions": [
"The lookup uses a title-fragment search scoped to the project first",
"The read step fetches the document by id requesting only the content field",
"Document types and fields (title, content, project) are explained",
"The not-found case is handled by loosening filters before concluding absence",
"The response stays minimal: two queries, not a crawl of the workspace"
]
},
{
"id": "cycle-management",
"prompt": "I manage a team that runs two-week cycles. I need to see the current cycle's workload, what is unassigned, and how full the cycle is. What queries should I use and what should I not assume about how cycles work in Linear?",
"expected_output": "A cycle-management query set: fetch the current cycle for the team (cycles connection filtered by state 'active' or by name/date window), then query its issues with assignee and state, computing workload from the issues' estimates if the team uses them. The response explains what not to assume: cycles are team-scoped and may overlap with previous cycles' leftovers, issue estimates are optional and may be absent, and completion percentage is derived from the issues' states, not stored as a field. It prescribes using the cycle's issues connection with only the fields needed (assignee, state, estimate, completedAt) and computing the picture client-side, plus handling teams that do not use cycles at all (the query returns empty and the response should say so rather than inventing one).",
"assertions": [
"The current cycle is found via the team's cycles connection filtered to active state",
"Workload is computed from the cycle's issues with assignee, state, and estimates where present",
"The response states that completion is derived from issue states, not a stored field",
"The assumption that all teams use cycles or estimates is explicitly rejected",
"The empty-cycle case is handled honestly"
]
},
{
"id": "error-recovery",
"prompt": "My script that syncs Linear issues to a spreadsheet started failing today with rate-limit errors, and sometimes the API returns an error that does not say whether my mutation applied. How should I handle API errors and verify state after failures?",
"expected_output": "An error-handling design for the Linear API: the response distinguishes retryable failures (rate limits with a Retry-After or reset timestamp, transient network errors) from permanent ones (authentication failures, invalid input, unknown identifiers) and prescribes backoff with the rate-limit reset rather than blind retries. For ambiguous mutations — an error where it is unclear whether the change applied — the response prescribes re-querying the affected entity to observe the actual state before retrying, and it explains Linear's typical behavior: mutations are synchronous and return the object on success, but network-level uncertainty means verification is the reliable recovery path. It also covers tracking request IDs or identifiers so the retry logic can deduplicate.",
"assertions": [
"Retryable failures (rate limits, transient network) are distinguished from permanent errors",
"Rate-limit handling uses the reset window with backoff",
"Ambiguous mutation failures are resolved by re-querying the entity's actual state",
"The response explains Linear's synchronous mutation behavior and its limits under network uncertainty",
"Retry logic can deduplicate via identifiers or request tracking"
]
}
]
}
product-design-and-ux/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "product-design-and-ux",
"evals": [
{
"id": "task-flow-design",
"prompt": "We are designing the invite-teammates flow for our collaboration app. Users need to add multiple teammates, choose roles, and receive invites by email. What does the task flow look like, and what should I decide before the design handoff to engineering?",
"expected_output": "A task flow design that walks the complete user path including the decisions at each step: entry points into invite, the invite composition step (email addresses, roles, optional message), validation states for invalid or duplicate addresses, the confirmation and what happens next for the inviter, and the invitee-side journey (email receipt, acceptance, account creation or login, role assignment). The response makes the interface decisions explicit: whether invites can be sent partially (some valid, some invalid), whether resending and revoking are supported, the failure and error states, and the boundary conditions (max batch size, deduplication). It ends with the unresolved decision points that need product input before engineering handoff.",
"assertions": [
"The response covers both the inviter-side and invitee-side journeys end to end",
"Validation and error states are specified for invalid, duplicate, and partial batches of addresses",
"The response resolves or flags decisions on resend, revoke, deduplication, and batch limits",
"The flow includes the post-acceptance state such as role assignment and account creation",
"Unresolved decision points are explicitly listed for product input before handoff"
]
},
{
"id": "state-recovery-model",
"prompt": "Our checkout form loses all user input when the page reloads or the session times out, and users abandon the flow. I want a state and recovery model so this stops happening. What states does the form have and how should recovery work in each?",
"expected_output": "A state and recovery model that enumerates the form's states: initial empty, partial input, validation error, submitting, submitted, and the failure states (network error, session expiry, server-side validation rejection, duplicate submission). For each state the response defines what is preserved and what is recoverable: input autosaved locally and restored on reload, session expiry handled by re-authentication that returns the user to the same step with data intact, idempotency so a retried submit does not double-charge, and explicit error recovery paths that tell the user what happened and what to do. The response also defines the timeout boundary after which recovery is no longer possible and the data is cleared with a clear message.",
"assertions": [
"The response enumerates the full state set including partial input, submitting, network failure, and session expiry",
"Input is preserved and restored on reload through autosave or local persistence",
"Session expiry recovery returns the user to the same step with data intact",
"Retried submissions are idempotent and cannot double-charge",
"The response defines when recovery is no longer possible and how that is communicated"
]
},
{
"id": "interface-contract-handoff",
"prompt": "I am handing the search-results page design to engineering. Every handoff before has produced drift: different spacing, wrong empty states, unclear loading behavior. What should the interface contract contain so the implementation matches the design?",
"expected_output": "An interface contract that specifies behavior rather than only aesthetics: the component inventory with names matching the design system, spacing and sizing tokens with concrete values, the states each component must render (loading, empty, error, populated, end-of-results), interaction behavior (debounce timing, keyboard navigation, focus management), and the data contract the page consumes (fields, ordering, pagination model). The response explains how to make the contract verifiable: reference screenshots or fixtures for key states, named tokens instead of pixel values repeated ad hoc, and a checklist the engineer uses to confirm each state before the page is considered done.",
"assertions": [
"The contract covers behavior and states, not only visual appearance",
"Components map to the design system with concrete spacing and sizing tokens",
"All states including loading, empty, error, and end-of-results are specified",
"Interaction behavior such as debounce, keyboard navigation, and focus is specified",
"The contract is verifiable through named tokens, fixtures, and a state checklist"
]
},
{
"id": "usability-study-plan",
"prompt": "We are about to redesign our dashboard navigation and want to test the new information architecture before building it. How do I plan a usability study that gives us signal we can act on, without over-engineering the research?",
"expected_output": "A usability study plan scoped to the decision at hand: a task-based test of the navigation where participants are asked to find specific information in the new IA, with a small set of well-chosen tasks that cover the highest-frequency user goals. The plan specifies participant criteria (a pragmatic small set of current users across the main personas, not a large panel), the protocol (moderated sessions with think-aloud, task success and time as measures, plus observation of where people look first), and the analysis: task success rates, the places where participants get lost, and a ranked list of IA problems with severity. The response right-sizes the study: 5-8 participants for a formative IA test, task selection driven by analytics data on what users actually do, and a report format that leads with the actionable problems.",
"assertions": [
"The study is task-based and tests the new IA against high-frequency user goals",
"Participant criteria are pragmatic and grounded in real usage, with a small formative sample",
"The protocol specifies moderated think-aloud sessions and the measures taken",
"Analysis produces ranked, severity-ordered problems from task success and lostness data",
"The response right-sizes the study and justifies the sample size for a formative test"
]
},
{
"id": "information-architecture-review",
"prompt": "Our app has grown from 5 to 40 screens and navigation is now a maze. Users cannot find features that exist, and teams keep adding entries to the sidebar. How should I review the information architecture and fix it systematically?",
"expected_output": "An information architecture review that starts from evidence, not opinion: card sorting or tree-testing results if available, analytics on navigation paths and search queries, and a content inventory grouped by user task frequency. The response proposes a hierarchy organized around the top user jobs with a small number of top-level destinations, applies the rules of thumb for where items belong (frequency and importance drive depth, related tasks cluster), and defines the governance fix: a stated policy for when a new feature earns a navigation entry versus living inside an existing destination, so the maze does not grow back. It sequences the work: audit, propose IA, validate with tree-testing or a quick study, then migrate with redirects.",
"assertions": [
"The review is driven by evidence such as navigation analytics, search data, and a content inventory",
"The proposed hierarchy is organized around top user jobs with few top-level destinations",
"Placement rules are explicit, using frequency and importance to drive depth",
"The response includes navigation governance so the structure does not regrow into a maze",
"The response sequences audit, redesign, validation, and migration with redirects"
]
}
]
}
product-discovery/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "product-discovery",
"evals": [
{
"id": "stakeholder-map",
"prompt": "We are starting discovery for a billing-system overhaul. I want to map who to talk to before scheduling any interviews. What does a complete stakeholder map look like and how do I decide who belongs in it?",
"expected_output": "A stakeholder map that covers the full set of people whose needs and constraints shape the outcome: decision-makers who fund and approve scope, end users who operate the system day to day, operators and support staff who handle billing escalations, downstream teams who consume billing data (finance, sales ops, data), and adjacent system owners whose systems integrate with billing. For each stakeholder the map records their role, what they need from the system, what constraints they impose, and how their input could conflict with others. The response explains the selection criteria: include anyone whose unmet need would block adoption or whose assumptions would silently break the design if left unspoken.",
"assertions": [
"The response builds a stakeholder map covering decision-makers, end users, operators, downstream consumers, and adjacent system owners",
"For each stakeholder the response captures needs, constraints, and potential conflicts",
"The response gives selection criteria for who belongs in the map rather than an arbitrary list",
"The response identifies where stakeholder interests conflict and how to surface those conflicts in discovery",
"The response is specific to the billing-system context rather than a generic template"
]
},
{
"id": "surface-hidden-assumptions",
"prompt": "During interviews for our new reporting feature, stakeholders keep saying 'just like the current export, but better.' Nobody can define what better means. What questions should I ask to surface the hidden assumptions behind that phrase?",
"expected_output": "A set of discovery questions designed to make the unstated assumptions explicit: what specific pain drives the request (which current export behavior is broken or slow), what outcome would count as success in measurable terms, who uses the export and for what decision, what the edge cases are that the current export handles badly (large files, empty data, formatting, timestamps in different timezones), what they would accept as a v1 versus what cannot wait, and what they would NOT want to change. The response frames each question to expose the underlying job and acceptance criteria, and it flags the pattern where vague phrasing signals either an unexamined assumption or an unstated constraint, then shows how to test the assumption by restating it and asking for confirmation.",
"assertions": [
"The response provides concrete questions that force definition of 'better' into measurable success criteria",
"The questions probe who uses the output, for what decision, and which current behaviors are broken",
"The questions surface edge cases and constraints the current export handles",
"The response distinguishes v1 scope from deferred wants",
"The response shows how to test an assumption by restating it and asking for confirmation"
]
},
{
"id": "conflict-resolution",
"prompt": "Two stakeholders disagree on the new checkout redesign: the payments team wants fewer steps to reduce abandonment, while the fraud team wants more verification to reduce chargebacks. Discovery is stuck. How do I resolve this without picking a winner politically?",
"expected_output": "A conflict-resolution approach that treats the disagreement as a design constraint to be understood, not a battle to be won: first the response maps each stakeholder's underlying goal and the evidence behind it (abandonment data versus chargeback data), then looks for a resolution space that satisfies both — differentiating verification by risk tier, moving verification off the critical path to a background check, or adding a post-purchase verification step — and evaluates the options against both goals with the trade-off made explicit. The response keeps both stakeholders in the decision, documents the trade-off in the discovery output, and escalates only when the trade-off is genuinely unresolvable and requires a product decision, at which point it frames the decision for the accountable owner with the evidence on both sides.",
"assertions": [
"The response reframes the conflict as constraints to be designed against, not personalities to be managed",
"The response maps each side's underlying goal and the evidence supporting it",
"The response generates resolution options that serve both goals, such as risk-tiered verification or off-critical-path checks",
"The response documents the trade-off and keeps both stakeholders in the decision",
"The response frames escalation to the accountable owner as a last resort with evidence on both sides"
]
},
{
"id": "gap-detection",
"prompt": "We have written requirements for a mobile app feature but I suspect we are missing whole scenarios. The requirements cover the happy path in detail. How do I systematically find the gaps before we commit to a spec?",
"expected_output": "A gap-detection procedure that walks the requirements against structured scenario categories rather than brainstorming: failure and recovery paths (what happens when the network drops, a payment fails, a sync conflicts), permission and entitlement states (users without access, expired sessions, shared accounts), boundary and empty states (no data, zero results, maximum data volume), multi-user and concurrency cases (two people editing the same object), and time-dependent behavior (timezones, midnight boundaries, scheduled jobs). For each category the response produces probe questions that turn into concrete scenarios, and it flags the highest-risk gaps for the feature at hand, such as offline-first behavior for a mobile app, and requires that the discovered gaps be added to the discovery output before spec sign-off.",
"assertions": [
"The response uses structured scenario categories such as failure paths, empty states, permissions, concurrency, and time boundaries",
"Each category is turned into concrete probe scenarios rather than generic advice",
"The response identifies the highest-risk gaps specific to a mobile app, such as offline and sync behavior",
"The response requires discovered gaps to be captured in the discovery output before spec sign-off",
"The response covers boundary states like zero results, maximum volume, and conflicting edits"
]
},
{
"id": "translate-to-sdd-spec",
"prompt": "Discovery is done for the notifications-center feature. I have interview notes, stakeholder maps, and a list of validated scenarios. Now I need to hand this off so it becomes a proper spec for the build pipeline. What should the handoff contain and how do I structure it?",
"expected_output": "A structured handoff that translates discovery evidence into the inputs a spec pipeline needs: a problem statement grounded in the stakeholder evidence, the validated user scenarios written as concrete given-when-then behavior, explicit scope boundaries listing what is out of scope and why, unresolved questions and the owners for each, and the acceptance criteria per scenario that the implementation phase can verify against. The response explains the mapping from discovery artifacts to spec sections so the evidence trail stays intact — each requirement traceable to the interview or scenario that produced it — and it notes where the discovery output is incomplete and should not be silently papered over in the spec.",
"assertions": [
"The response structures the handoff around problem statement, validated scenarios, scope boundaries, and acceptance criteria",
"Scenarios are written in concrete given-when-then behavior an implementation pipeline can verify",
"The response keeps each requirement traceable to the discovery evidence that produced it",
"Unresolved questions are listed with owners rather than silently resolved",
"The response flags incomplete discovery areas instead of papering over them"
]
}
]
}
product-methodology/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "product-methodology",
"evals": [
{
"id": "rice-prioritization",
"prompt": "We have six candidate features for next quarter and a team of four engineers. I want to prioritize them. Our data team can estimate reach and impact, and we know our confidence in each estimate varies. Walk me through a RICE scoring session and how to turn the scores into a quarter plan.",
"expected_output": "A RICE prioritization that computes reach x impact x confidence / effort for each candidate, with the inputs sourced and assumptions stated for each estimate rather than invented. The response handles the confidence axis honestly: low-confidence estimates are either discounted or flagged for validation before the quarter, and the response explains how to treat scores that are close together (within rounding noise) as a tie to be resolved by strategic fit, dependencies, or sequencing, not by score precision. It translates the ranked list into a quarter plan that respects team capacity, sequences dependencies, and reserves slack for discovery and validation work.",
"assertions": [
"The response computes RICE scores with reach, impact, confidence, and effort for each candidate",
"Each estimate is sourced or explicitly flagged as an assumption rather than invented",
"The response handles low-confidence estimates and near-tie scores honestly instead of trusting precision",
"The response converts the ranked list into a capacity-aware quarter plan with dependencies and slack",
"The response explains when scores should be treated as ties resolved by strategic fit or sequencing"
]
},
{
"id": "opportunity-solution-tree",
"prompt": "Activation is flat even though we keep shipping features. I want to structure the thinking before we plan more work. How do I build an opportunity solution tree for improving activation, and what makes it different from just a feature list?",
"expected_output": "An opportunity solution tree that starts from the desired outcome (higher activation) and branches into the opportunities where the outcome could be unlocked, then into candidate solutions for each opportunity, keeping the tree connected to evidence: each opportunity is stated as an unmet user need or gap with evidence behind it, and each solution links back to the opportunity it serves. The response explains the discipline that solutions are generated only for evidence-backed opportunities, that the tree makes it visible when the team is building solutions for opportunities that are not actually blocking the outcome, and that it gives a shared language for killing weak branches before they become features. It applies the structure to the activation problem with concrete example branches.",
"assertions": [
"The response structures the tree as outcome to opportunities to solutions, each level connected",
"Opportunities are stated as evidence-backed user needs, not feature ideas",
"Each solution is traced back to the opportunity it serves",
"The response explains how the tree makes solution-for-its-own-sake visible and killable",
"The response includes concrete example branches for the activation problem"
]
},
{
"id": "moscow-scoping",
"prompt": "Our stakeholders all marked everything as 'must have' for the reporting dashboard. The team can realistically ship half of it. How do I run a MoSCoW session that produces a defensible scope instead of a fight?",
"expected_output": "A MoSCoW scoping approach that establishes the rules before categorization: must-have means the release fails its core promise without it, should-have and could-have are valuable but have clear release-date trade-offs, and won't-have is explicit this time. The response resolves the everything-is-must-have pattern by forcing a dependency test (what breaks if this is missing at launch), an alternative test (what already satisfies this need), and a cost-benefit test against the release date. It records the rationale for each category so the scope is defensible, sequences should-haves into a follow-up commitment so they are not lost, and produces a scope the team commits to with a definition of done for the release.",
"assertions": [
"The response establishes category rules before categorization, including a strict test for must-have",
"The response uses dependency, alternative, and cost-benefit tests to break the everything-is-must-have pattern",
"Category decisions are recorded with rationale so the scope is defensible",
"Should-haves are sequenced into a follow-up commitment rather than dropped",
"The session ends with a team-committed scope and a release definition of done"
]
},
{
"id": "decision-log-entry",
"prompt": "We just decided to drop the iOS build of the admin app in favor of a responsive web version, reversing a decision we made last quarter. I want this recorded properly so future teams understand why. What should the decision log entry contain?",
"expected_output": "A decision log entry that records the decision in a durable, queryable format: date, deciders, the decision in one sentence, the context and evidence at the time, the alternatives considered with the reasons they were rejected, the anticipated consequences and how they will be monitored, and explicit links to the superseded decision from last quarter with the reason the context changed. The response treats the reversal honestly as a response to changed evidence (support costs, team skills, adoption data) rather than as an inconsistency, and it specifies where the entry lives and how the superseded entry is marked so the log tells a coherent story.",
"assertions": [
"The entry records date, deciders, the decision, context, alternatives, and consequences",
"The entry links to and marks the superseded prior decision, explaining what evidence changed",
"The response frames the reversal as evidence-driven rather than inconsistent",
"The entry specifies how consequences will be monitored",
"The response names where the log entry lives and how superseded entries are marked"
]
},
{
"id": "audience-specific-communication",
"prompt": "We decided to move the roadmap from committed-date promises to a themes-based model. I need to communicate this to three audiences: the sales team, the C-suite, and existing customers. They will each react differently. What should I say to each, and what should I deliberately not promise?",
"expected_output": "Audience-specific communication that adapts the message while keeping it consistent: sales gets the honest mechanics (which commitments remain, how roadmap items are now framed in deals, where the risk lies), the C-suite gets the strategic rationale, the risk the change manages (missed-date commitments damaging trust), and the metrics that will show it working, and customers get the benefit (fewer broken dates, clearer priorities) plus what explicitly will not change for them. The response deliberately avoids promising specific dates the new model does not support, prepares answers for the skeptical questions each audience will ask, and sequences the communication so internal audiences hear it before customers.",
"assertions": [
"The response produces distinct messaging for sales, C-suite, and customers that stays internally consistent",
"Sales messaging covers how roadmap items are framed in deals and where risk remains",
"C-suite messaging gives strategic rationale and the metrics that will show success",
"Customer messaging names benefits and explicitly what will not change",
"The response identifies promises to avoid and sequences internal communication before customer communication"
]
}
]
}
product-strategy/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "product-strategy",
"evals": [
{
"id": "north-star-metric",
"prompt": "We are a B2B analytics product and need a North Star metric to align the company. The team is proposing daily active users, but I worry it rewards cheap usage over delivered value. How should we define our North Star metric and how do we keep it honest?",
"expected_output": "A North Star metric definition that starts from the core value users receive rather than the cheapest engagement signal: for a B2B analytics product, a metric such as weekly reporting frequency per workspace or number of teams with a produced report, tied to the moment a user gets value from the product. The response explains why raw DAU is risky as a North Star for a B2B product (it rewards logging in, not outcomes) and shows how to validate the chosen metric against retention and paid-plan correlation before committing. It defines the guardrail metrics that prevent gaming (if the North Star rises while activation or retention falls, the metric is wrong) and explains how the metric cascades into team-level metrics without every team inheriting the same number.",
"assertions": [
"The response ties the North Star to delivered value rather than cheap engagement such as DAU",
"The proposed metric is validated against retention or paid-plan correlation before adoption",
"The response explains why raw DAU is a risky North Star for a B2B product",
"Guardrail metrics are defined so a rising North Star with falling health signals is caught",
"The response cascades the metric into team-level metrics without forcing one number everywhere"
]
},
{
"id": "competitive-positioning-analysis",
"prompt": "A well-funded competitor just launched a cheaper version of our product. I need to understand whether this is a real threat and how to position against it, before we react by cutting price. What analysis should I run?",
"expected_output": "A competitive analysis that separates signal from noise: an assessment of where the competitor actually wins (feature set, price, distribution, customer segment), an honest capability comparison against our product across the dimensions customers care about, and a segment analysis of which customers the cheaper offering genuinely threatens versus which are underserved by it. The response explicitly pushes back on reflexive price-cutting by analyzing whether the competitor's customers are price-driven segments we do not currently serve or core customers who would leave for features, not price. It positions the response around our defensible differentiators and the customers whose needs the competitor does not meet, and it sets up monitoring for the signs that the threat is real (share loss in our core segment, win-rate changes).",
"assertions": [
"The response analyzes which segments the competitor genuinely threatens versus which they underserve",
"It compares capabilities on the dimensions customers actually care about",
"The response pushes back on reflexive price-cutting and analyzes what would drive real defection",
"Positioning is built around defensible differentiators, not reactive pricing",
"The response sets up monitoring signals such as win-rate and segment share changes"
]
},
{
"id": "tam-sam-som-sizing",
"prompt": "We are a team-collaboration tool and need a market-size estimate for an investor deck. How do I build TAM, SAM, and SOM credibly without inventing numbers, and what should the numbers actually claim?",
"expected_output": "A market sizing built top-down and bottom-up with the numbers reconciled: TAM from a defensible unit model (number of knowledge workers or teams in the target geographies times a credible per-seat spending benchmark), SAM narrowed by the segments the product actually serves (region, company size, category budget), and SOM grounded in what the business can actually capture within a stated horizon given go-to-market capacity and observed win rates. The response explains the source and assumption for each number, cross-checks the top-down estimate against a bottom-up calculation from customer counts and pricing, and states the sizing claims in ranges with the key assumptions exposed so the deck number is defensible rather than aspirational.",
"assertions": [
"The response builds TAM, SAM, and SOM with a stated unit model for each",
"Top-down estimates are cross-checked against a bottom-up calculation",
"SOM is grounded in go-to-market capacity and observed win rates within a stated horizon",
"Assumptions and sources are exposed for each number",
"The sizing is presented as ranges with defensible claims, not single aspirational figures"
]
},
{
"id": "roadmap-prioritization-framework",
"prompt": "Our roadmap is a collection of what the loudest customer asked for last. I want to introduce a prioritization framework across strategy, product, and engineering without turning planning into a bureaucracy. How do I choose and run one?",
"expected_output": "A prioritization approach that picks the framework by decision type rather than applying one everywhere: strategic bets by the executive team (using a framework suited to options and trade-offs), feature-level prioritization by product using a scored model like RICE or a weighted opportunity model, and engineering sequencing by cost and dependency. The response explains how to run it without bureaucracy: a single shared backlog with the scoring inputs visible, scores treated as input to a discussion rather than a verdict, a monthly cadence where the framework output is reviewed and adjusted, and a rule that anyone can propose but the scoring inputs must be evidence-backed. It covers how the framework connects strategy to roadmap so top-level bets constrain what gets prioritized.",
"assertions": [
"The response matches frameworks to decision types: strategy, feature priority, and engineering sequencing",
"The process keeps scoring inputs visible and treats scores as discussion input, not verdict",
"The cadence is lightweight, such as a monthly review, without heavy process",
"Proposals require evidence-backed scoring inputs",
"Strategic bets constrain feature-level prioritization so the roadmap follows strategy"
]
},
{
"id": "product-market-fit-assessment",
"prompt": "We have been selling our developer tool for eight months. Usage is growing but churn is noticeable. Investors ask if we have product-market fit. How do I assess this rigorously rather than with vibes?",
"expected_output": "A product-market-fit assessment built from evidence across the standard signals: a Sean Ellis-style survey of active users (the share who would be very disappointed without the product, with the 40% benchmark contextualized for a developer tool), retention cohort analysis showing whether usage stabilizes or decays for each acquisition cohort, the qualitative pattern of how users found and adopted the product (organic pull versus sales push), and the economic test of whether the value delivered exceeds acquisition cost per retained user. The response explains how to interpret mixed signals honestly: a product can be loved by a segment and fail on others, so fit is assessed per segment, and it prescribes what to do next based on where the evidence lands rather than declaring fit from a single metric.",
"assertions": [
"The response uses multiple signals: disappointment surveys, retention cohorts, organic adoption, and unit economics",
"The 40% benchmark is contextualized for the product type rather than applied mechanically",
"Fit is assessed per segment, allowing for mixed signals",
"Retention cohort analysis shows whether usage stabilizes or decays",
"The response prescribes next steps from the evidence pattern rather than declaring a verdict from one metric"
]
}
]
}
secure-software-engineering/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "secure-software-engineering",
"evals": [
{
"id": "threat-model-feature",
"prompt": "We are adding a file-upload endpoint to our web application that accepts images from unauthenticated users and serves them back. Before we build it, I want a threat model. What do I consider, and how should I structure the threat modeling session?",
"expected_output": "A threat model for the upload feature organized around the system's assets, trust boundaries, and attackers: the response walks the data flow (upload, storage, validation, serving) and identifies the threats at each boundary — malicious file content (malware and polyglot files), denial of service (unbounded size, decompression bombs, resource exhaustion), stored cross-site scripting when files are served inline, path traversal and overwrite of existing files, content-type spoofing bypassing image validation, and abuse of the storage as a hosting vector. For each threat it proposes controls mapped to the threat: strict content validation by magic bytes plus re-encoding, size and count limits, serving from a separate origin or with Content-Disposition and no inline HTML, randomized storage keys never derived from user input, and rate limits. The response uses a structured method (e.g., STRIDE or a data-flow walk) and records decisions so the review is auditable.",
"assertions": [
"The threat model walks the data flow and identifies threats at each trust boundary",
"Upload-specific threats are covered: malware, decompression bombs, stored XSS, path traversal, content-type spoofing",
"Controls are mapped to each threat, including magic-byte validation, size limits, and randomized storage keys",
"Serving mitigations such as separate origin or Content-Disposition are specified",
"A structured method is used and decisions are recorded for auditability"
]
},
{
"id": "security-requirements",
"prompt": "We are designing a new customer-facing API that exposes account data, and security keeps being an afterthought. I want security requirements written into the design before implementation. What requirements should the design carry?",
"expected_output": "A security-requirements set written as testable design constraints, not slogans: authentication requirements (which mechanism, token lifetime and refresh policy, where tokens are stored), authorization requirements (least privilege, per-resource access checks at the data layer rather than hidden in the UI), data-handling requirements (encryption in transit and at rest, what sensitive fields are collected at all and the minimization rule), input and output requirements (validation of untrusted input, no sensitive data in logs or error messages), and operational requirements (secret management, audit logging of access to sensitive data, key rotation). The response explains how each requirement is verified during implementation and review, and it prioritizes the requirements by the harm they prevent so the team knows what cannot be deferred.",
"assertions": [
"Requirements are written as testable constraints covering authentication, authorization, and data handling",
"Authorization is specified as least-privilege with data-layer access checks",
"Data minimization, encryption, and sensitive-data-in-logs rules are explicit",
"Operational requirements cover secret management, audit logging, and key rotation",
"Requirements are prioritized by the harm they prevent"
]
},
{
"id": "authn-authz-review",
"prompt": "In a code review I noticed our new endpoint checks 'is the user logged in?' but not 'is this user allowed to see this specific document?'. The frontend hides buttons based on role, and the API trusts that. What is the risk and what should the design enforce?",
"expected_output": "A review finding that separates authentication from authorization: the response explains the risk precisely — hiding buttons in the frontend is not a security control, and an API that trusts UI state allows direct requests to access documents the caller should not see, which is an insecure-direct-object-reference or missing-object-level-authorization pattern. It prescribes the fix: every API handler must independently check authorization against the resource (the caller's identity and their relationship to the specific document) at the data-access boundary, not in the controller only, with deny-by-default behavior and tests that hit the endpoint directly without the UI to prove access is denied. It also covers the general principle: authorization checks belong where the data is read, and the frontend's role-based UI is a UX concern, not a control.",
"assertions": [
"The response identifies the missing object-level authorization as the core risk",
"It explains why frontend button-hiding is not a security control and direct API access bypasses it",
"The fix enforces per-resource authorization at the data-access boundary with deny-by-default",
"Tests are prescribed that hit the endpoint without the UI to prove denial",
"The response distinguishes authentication from authorization and UI-state from security control"
]
},
{
"id": "untrusted-input-secrets",
"prompt": "Our service parses user-supplied YAML files, runs some of the fields through a templating engine, and stores API keys in a config file committed to the repository. I know both are wrong but I need a concrete plan to fix them. What do I do?",
"expected_output": "A hardening plan for both problems with the risks stated precisely: YAML parsing is unsafe for untrusted input (aliases and object construction can execute code), so the response prescribes parsing with a safe configuration (no arbitrary object instantiation) or moving to a stricter format with a schema, and treating the templating engine as code execution by design with input sandboxed and never fed raw user content. The second half addresses the committed API keys: rotate the exposed keys immediately, remove them from history-aware secrets management going forward (inject via environment or a secret store, never the repository), and scan the repository to confirm no other secrets remain. The response sequences the work by urgency: rotate exposed keys first, then fix parsing, then templating, with verification for each step.",
"assertions": [
"The YAML risk is stated precisely and fixed with safe parsing or a stricter schema",
"The templating engine is treated as code execution and hardened against untrusted input",
"Exposed API keys are rotated immediately and moved to environment or secret-store injection",
"A repository scan confirms no other secrets remain",
"The plan is sequenced by urgency with verification at each step"
]
},
{
"id": "dependency-evaluation",
"prompt": "A teammate wants to add a new npm package to our backend service. It is popular, but we have been burned before by abandoned dependencies and supply-chain surprises. What is a proper dependency evaluation before we accept it?",
"expected_output": "A dependency evaluation covering the dimensions that matter for supply-chain safety: maintenance and community health (release cadence, response to issues, bus factor), the dependency's own dependency tree (transitive bloat and known vulnerabilities), provenance and integrity (published from a verified account, signatures, and the maintainer's reputation), license compatibility, the security-relevant surface (does it parse untrusted input, does it touch the network or filesystem), and the fallback cost if it is abandoned (how much code would need to be forked). The response produces a decision framework: acceptable with a pinned version and periodic review, acceptable only behind isolation, or rejected, and it prescribes the ongoing controls: lockfiles, automated vulnerability scanning in CI, and a review cadence for critical-path dependencies.",
"assertions": [
"The evaluation covers maintenance health, transitive dependencies, and known vulnerabilities",
"Provenance, integrity, and license compatibility are checked",
"The security-relevant surface of the package is assessed",
"The decision framework includes rejection and isolation options, not just acceptance",
"Ongoing controls are prescribed: lockfiles, CI scanning, and review cadence"
]
}
]
}
security-audit-methodology/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "security-audit-methodology",
"evals": [
{
"id": "authorized-audit-plan",
"prompt": "We have been asked to review the security posture of an internal web application before it is exposed to customers. Where do I start: what authorization do I need, what is in scope, and what does the plan look like?",
"expected_output": "An audit plan that establishes authorization and scope before any assessment activity: the response starts with the written authorization for the review (who requested it, what systems are in scope, what actions are permitted, and the boundaries of the engagement), then defines the assessment scope by system and threat surface, and outlines the plan phases: architecture and threat-model review, configuration and dependency review, and targeted testing of the in-scope controls. The response explicitly distinguishes a defensive, authorized review from offensive operations and states the rule that anything outside the written scope is out of bounds until re-authorized. It also covers the deliverable structure: a findings report with severity, evidence, and remediation guidance, and it prescribes confirming authorization details with the requester before starting.",
"assertions": [
"The plan requires written authorization defining scope, permitted actions, and boundaries before assessment",
"Scope is defined by system and threat surface, with out-of-scope items explicit",
"The plan phases cover architecture review, configuration and dependency review, and targeted control testing",
"The response distinguishes authorized defensive review from offensive operations",
"The deliverable is a findings report with severity, evidence, and remediation"
]
},
{
"id": "threat-modeling-session",
"prompt": "Our payments service is getting a new API endpoint that accepts webhooks from a third-party provider. I want to threat-model this addition before it ships. How do I run the session and what should come out of it?",
"expected_output": "A structured threat-modeling session for the webhook endpoint: the response defines the system boundary and trust zones (the third-party provider, the internet, the API service, internal systems), enumerates the assets and data flows into and out of the endpoint, and walks the threats against each flow — unauthenticated or spoofed webhooks triggering actions, replay of captured webhook requests, payload injection into downstream processing, denial of service via volume or large payloads, and the webhook endpoint being used to probe internal behavior. For each threat the session produces a decision: accepted with justification, mitigated with a specific control (signature verification, timestamp and nonce replay protection, payload schema validation, rate limiting, idempotent processing), or flagged for follow-up. The output is a recorded threat register with owners, so the session is not a discussion but a decision artifact.",
"assertions": [
"The session defines trust zones and data flows for the webhook endpoint",
"Webhook-specific threats are covered: spoofing, replay, injection, and abuse via volume",
"Each threat is resolved to accept, mitigate, or follow-up with a named control",
"Mitigations include signature verification, replay protection, and schema validation",
"The output is a recorded threat register with owners"
]
},
{
"id": "architecture-audit",
"prompt": "I need to review the architecture of a legacy application that will be internet-facing for the first time. It was built for an internal network. What are the highest-priority architectural security questions to answer?",
"expected_output": "An architecture audit structured around the highest-priority security properties: the response identifies the critical questions — where trust boundaries are and what crosses them (input that was trusted internally but will now be attacker-controlled), how authentication and authorization are enforced and whether they are centralized or scattered, how secrets are stored and rotated, what data the application handles and how it is protected in transit and at rest, how the application is isolated from other systems it shares a network with, and how failures are observed (logging and alerting on security events). The response prioritizes the questions by the risk of moving from internal to internet-facing: any trust assumption that was safe on an internal network is now an exposure, and it explains how each answer maps to a finding or a remediation decision in the audit report.",
"assertions": [
"The audit leads with trust-boundary questions: what was trusted internally that will now be attacker-controlled",
"Authentication and authorization enforcement points are examined for centralization",
"Secrets handling, data protection, and network isolation are covered",
"Observability of security events is part of the audit",
"Questions are prioritized by the internal-to-internet risk shift"
]
},
{
"id": "dependency-audit",
"prompt": "Our application uses 200+ dependencies and we have never audited them. There are known CVEs reported in the vulnerability scanner but the team ignores them because 'everything has CVEs.' How do I run a dependency audit that produces an actionable outcome?",
"expected_output": "A dependency audit that moves from a raw CVE list to a risk-ranked decision list: the response prescribes triaging the scanner output by reachability (does the vulnerable code path actually run in this application), exploitability (public exploits, attacker-accessible input), and the version gap, so findings are ordered by real exposure rather than count. It explains how to verify reachability with the dependency graph and code paths instead of trusting the scanner's default severity, and it separates the actionable buckets: patch now (reachable, exploitable), patch in the next cycle (reachable but lower risk), and track (not reachable today but should be monitored or the dependency scheduled for removal). The response also covers the structural fixes: a policy for adding dependencies (the review before adoption), upgrading cadence, and retiring abandoned dependencies that accumulate risk without being used.",
"assertions": [
"Audit findings are triaged by reachability, exploitability, and version gap, not scanner severity alone",
"Reachability is verified with the dependency graph and code paths",
"Findings are bucketed into patch-now, next-cycle, and track categories",
"A dependency-adoption policy and upgrade cadence are prescribed",
"Abandoned or unused dependencies are identified for removal"
]
},
{
"id": "vulnerability-classification",
"prompt": "Our audit found a list of issues: an exposed admin panel, a session cookie without secure flags, an SQL injection in a search endpoint, and a missing rate limit on login. I need to write the findings report. How do I classify and communicate these so the team fixes the right things first?",
"expected_output": "A findings report with classification and communication that drives correct prioritization: the response assigns severity per finding using a consistent method that combines exploitability and impact — the SQL injection is exploitable, attacker-reachable, and leads to data exposure so it is critical and must be fixed first; the exposed admin panel is high because it is internet-reachable but requires an account or adds exposure; the login rate-limit absence is high for brute-force risk; the cookie flag is medium-low because it requires a separate attack to be useful. Each finding gets the evidence needed to reproduce or locate it, the affected component, and a concrete remediation. The report separates what must be fixed before launch from what can be scheduled, and it avoids severity inflation that trains teams to ignore findings. The response also explains how to present the report to engineering so remediation is actionable, not defensive.",
"assertions": [
"Severity is assigned with a consistent method combining exploitability and impact",
"Each finding includes evidence, affected component, and concrete remediation",
"Findings are separated into launch-blocking and schedule-able categories",
"The response explains why the SQL injection outranks the cookie flag with reasoning",
"Severity inflation is avoided so the report drives action"
]
}
]
}
site-reliability-engineering/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "site-reliability-engineering",
"evals": [
{
"id": "slo-error-budget-policy",
"prompt": "We have no SLOs and every service team defines reliability differently. I want to introduce SLOs with error budgets for our API platform. How do I pick the first SLOs, set targets, and define what the error budget actually controls?",
"expected_output": "An SLO design grounded in user-facing reliability rather than internal metrics: the response identifies the user journeys that matter (API availability and latency percentiles for the core request path), picks SLOs on the metrics users actually experience (e.g., availability and latency at p95/p99 measured from the edge), sets targets that are ambitious but realistic given current performance, and defines the error budget as 100% minus the SLO target over a rolling window. It specifies how the budget governs action: when burn is high, releases freeze or changes require review; when the budget is healthy, velocity proceeds. It also covers alerting on error-budget burn rate rather than waiting for budget exhaustion, and the review cadence where targets are revisited with evidence.",
"assertions": [
"SLOs are chosen on user-facing metrics such as availability and latency percentiles for core journeys",
"Targets are set relative to current performance and the error budget is defined as a rolling window",
"Error-budget policy governs action: release freezes or review gates when burn is high",
"Burn-rate alerting is prescribed instead of alerting only on budget exhaustion",
"A review cadence revisits targets with evidence"
]
},
{
"id": "incident-command",
"prompt": "We just had a major outage: the checkout service is down, the on-call engineer is overwhelmed, and everyone is jumping into the chat with suggestions. I have been told to take over incident command. What do I do in the first ten minutes and how do I run the response?",
"expected_output": "An incident-command response that establishes structure under pressure: declare the incident, assign roles (incident commander, communications lead, and operations leads for investigation and mitigation) so the commander is not also debugging, and set up a dedicated channel and timeline. The commander's first actions: confirm the blast radius and current status, stabilize with the fastest safe mitigation while investigation continues in parallel, and drive communication with one consistent narrative to stakeholders. The response explains the commander's core discipline: decide who does what, watch the clock, and do not get pulled into individual debugging threads, plus the post-mitigation sequence: verify recovery, declare the incident over, and schedule the postmortem with the timeline captured while fresh.",
"assertions": [
"Roles are assigned (commander, communications, operations) so the commander is not debugging",
"The first actions confirm blast radius and stabilize with the fastest safe mitigation",
"One consistent communications narrative is maintained for stakeholders",
"The commander's discipline of not joining individual debugging threads is explicit",
"The response covers verification of recovery, declaring the incident over, and scheduling the postmortem"
]
},
{
"id": "burn-rate-alerting",
"prompt": "Our current alerting pages someone only when the error rate crosses 5% for five minutes, and we are constantly paged for noise or miss slow burn entirely. I want alerting driven by the error budget instead. How do I design it?",
"expected_output": "A burn-rate alerting design tied to the SLO error budget: the response defines burn rate as the ratio of actual error consumption to budgeted consumption over a window and sets up multi-window alerts — a fast-burn window (e.g., 14x budget over 1 hour) for immediate pages and a slow-burn window (e.g., 2x over 6 hours or 1x over days) for gradual degradation, so both sudden spikes and slow creeping failures page appropriately. It explains the rationale: the 5%-for-5-minutes rule is decoupled from the SLO and cannot distinguish a budget-destroying event from a blip. It covers severity routing (page for budget-destroying burn, ticket for moderate burn), the runbooks tied to each alert, and calibration so noisy pages are reduced.",
"assertions": [
"Burn rate is defined as error consumption relative to the budgeted rate",
"Multi-window alerts distinguish fast burn from slow burn",
"The response explains why fixed threshold alerting is decoupled from the SLO and misses slow burn",
"Severity routing maps budget-destroying burn to pages and moderate burn to tickets",
"Alert calibration to reduce noise is part of the design"
]
},
{
"id": "capacity-operational-review",
"prompt": "Every Black Friday our services degrade because traffic triples and we are always caught short. We scale reactively. I want a capacity process that prevents this and covers the day itself. What does the operational plan look like?",
"expected_output": "A capacity and operational-readiness plan built on evidence: the response starts with demand forecasting from historical traffic patterns, planned growth, and marketing calendars, then defines capacity requirements per service with headroom targets, load-testing the expected peak before the event, and the scaling plan (autoscaling policies, provisioned capacity, and the manual levers if automation fails). The operational plan for the event covers the runbook: pre-event checks, live dashboards with the capacity signals, a paging and escalation structure for the day, and explicit decision rules for shedding load or degrading gracefully under saturation. It ends with the post-event review: what the forecast got wrong, what headroom was actually needed, and the adjustments carried into the next cycle.",
"assertions": [
"Demand forecasting is grounded in historical patterns, growth, and event calendars",
"Capacity requirements include headroom targets and load testing before the event",
"The scaling plan covers automation plus the manual levers if automation fails",
"The event-day plan has dashboards, escalation structure, and load-shedding decision rules",
"A post-event review feeds corrections into the next capacity cycle"
]
},
{
"id": "error-budget-decision",
"prompt": "Our payment service is burning through its error budget three times faster than expected this quarter due to a known flaky dependency. The team wants to ship a big feature this week, and the error budget is nearly exhausted. How do I make the call with the budget?",
"expected_output": "A decision made through the error-budget policy rather than a gut call: the response walks the analysis — how fast the budget is burning, what the burn implies about user impact, whether the flaky dependency is being addressed with an owner and timeline, and what the policy says about releases under high burn. It explains the two honest paths: if the budget is nearly exhausted, the policy gates the release (freeze or require exceptional approval), and if the team believes the feature will not worsen the burn, that is a hypothesis to support with evidence, not an exception to negotiate. The response treats the exhausted budget as the forcing function to fix the dependency, and it distinguishes a one-off exceptional release with a stated owner and deadline from repeatedly ignoring the budget, which makes the policy meaningless.",
"assertions": [
"The decision follows the error-budget policy with an analysis of burn rate and user impact",
"High burn gates the release rather than being negotiated around",
"Claims that the feature will not worsen burn are treated as evidence-backed hypotheses",
"The dependency causing the burn gets an owner and timeline",
"One-off exceptions with stated owners are distinguished from policy-ignoring patterns"
]
}
]
}
spec-driven-development/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "spec-driven-development",
"evals": [
{
"id": "spec-authoring",
"prompt": "We are starting an AI-assisted software project and I need to write the first specification for a feature that imports customer data from CSV files with validation and deduplication. What does a good spec look like in this pipeline, and what must it contain before implementation can begin?",
"expected_output": "A specification that functions as the contract for the pipeline: a clear problem statement and scope, the functional behavior written as concrete given-when-then scenarios (valid import, malformed rows, duplicate detection, partial failures), explicit edge cases and error handling, acceptance criteria per scenario that a verifier can check mechanically, and the boundaries of what is out of scope. The response explains why the spec is the input to implementation rather than documentation produced after, why ambiguity in the spec becomes divergence in the code, and it includes the Gherkin-style scenarios so an agent can implement and verify against them directly.",
"assertions": [
"The spec contains a problem statement, scope, and out-of-scope boundaries",
"Functional behavior is written as concrete given-when-then scenarios",
"Edge cases and error handling for malformed input and partial failures are specified",
"Each scenario has mechanically checkable acceptance criteria",
"The spec is structured to be the direct input to implementation, not post-hoc documentation"
]
},
{
"id": "quality-gate-review",
"prompt": "Our implementation gate just rejected a feature because the generated code does not match the spec: the error messages differ, and one validation rule was implemented differently than specified. The spec and code disagree in small ways. How do I run the review and decide what gets fixed?",
"expected_output": "A gate review that treats spec-code divergence as the primary defect signal: the response classifies each discrepancy (behavioral mismatch, cosmetic difference, ambiguous spec that allowed two readings, missing edge case) and routes them correctly — behavior and validation mismatches are code fixes against the spec, ambiguity is a spec revision before re-implementation, and cosmetic differences are ignored unless they affect observability. It prescribes re-running the verification against the corrected code, keeping the spec as the source of truth, and records the review outcome so the loop is auditable. It also warns against weakening the gate to pass the code instead of fixing the mismatch.",
"assertions": [
"Each discrepancy is classified as behavioral, cosmetic, or spec-ambiguity with a different routing",
"Behavioral mismatches are fixed in code against the spec as source of truth",
"Ambiguity is resolved by revising the spec before re-implementation",
"Verification is re-run after fixes and the outcome is recorded",
"The response warns against weakening the gate instead of fixing the mismatch"
]
},
{
"id": "decomposition-into-tasks",
"prompt": "I have a spec for a two-week feature but it is one large block of work. The team implements it as a single prompt and gets back a mess that does not verify. How should I decompose the spec so each unit of implementation is verifiable?",
"expected_output": "A decomposition that slices the spec along verify-able seams rather than by file or guesswork: each task maps to a subset of the spec's scenarios with its own acceptance criteria, dependencies between tasks are explicit, and the order is chosen so early tasks establish the contract (schema and interfaces) that later tasks implement against. The response explains the rule that a task is complete when its scenarios pass the gate, that decomposition follows the spec structure rather than the code structure, and that interfaces between tasks are themselves specified so tasks integrate without renegotiation. It shows the slice boundaries for the import feature and what each slice's verification looks like.",
"assertions": [
"Decomposition slices the spec along scenario boundaries with per-task acceptance criteria",
"Dependencies between tasks are explicit and ordering is contract-first",
"Interfaces between tasks are specified so integration does not require renegotiation",
"A task is defined as complete only when its scenarios pass the gate",
"Concrete slice boundaries are shown for the import feature"
]
},
{
"id": "gate-recovery-revision",
"prompt": "The implementation gate failed a feature three times. Each retry was a fresh generation from the full spec and each produced different, still-failing code. The team wants to rewrite the spec from scratch. How should we run the revision loop properly?",
"expected_output": "A revision-loop diagnosis that distinguishes the failure cause before rewriting: the response examines why three independent generations failed differently, which points to a spec problem (ambiguity, contradictions, missing edge cases) rather than luck, and prescribes patching the spec at the specific failure points instead of a full rewrite, because a full rewrite discards the parts that already verified and resets the convergence. It explains the patch-not-rewrite discipline: keep the spec's stable core, tighten only the failing scenarios, re-review the patch scope, and re-run the gate. It also covers re-reviewing the affected implementation slice rather than re-generating everything.",
"assertions": [
"The response diagnoses spec-level causes of divergent repeated failures before rewriting",
"The revision is scoped as a patch to the failing scenarios, not a full spec rewrite",
"The response explains why full rewrites discard verified progress and slow convergence",
"The patch is re-reviewed and the gate re-run",
"Only the affected implementation slice is reworked, not the whole feature"
]
},
{
"id": "pipeline-mode-selection",
"prompt": "We have a tiny one-file script change and a brand-new multi-service feature, and I am told to run both through the same spec pipeline. The tiny change is drowning in process. How do I decide the pipeline mode for a given change?",
"expected_output": "A pipeline-mode decision that scales the process to the change's risk and size: the response defines the criteria (behavioral surface area, blast radius, number of integration points, reversibility) and maps them to modes — a trivial mechanical change gets a lightweight path with a short spec and direct verification, while a new multi-service feature gets the full pipeline with decomposition and gates at each phase. It explains that the pipeline's purpose is enforcing correctness where divergence is expensive, and that over-applying full ceremony to trivial changes erodes trust in the process. It also covers the guardrail that the mode decision itself is recorded so it can be audited.",
"assertions": [
"The response defines explicit criteria for selecting the pipeline mode by risk and size",
"Trivial changes get a lightweight path while new features get the full pipeline",
"The rationale ties pipeline depth to where divergence is expensive",
"The response warns that over-ceremony on trivial changes erodes process trust",
"The mode decision is recorded for auditability"
]
}
]
}
strategy-frameworks/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "strategy-frameworks",
"evals": [
{
"id": "strategic-direction",
"prompt": "Our company has grown reactively for years and the leadership team cannot agree on where we are going. I have been asked to structure a strategic-direction conversation. How do I run it and what artifact should come out?",
"expected_output": "A strategic-direction process that uses a framework to make the conversation structured rather than free-form: the response selects the strategic-planning approach and runs it through the working method — state the decision and its owner, articulate the current situation and the constraints, generate the direction options with the assumptions behind each, and record the trade-offs and the conditions that would change the choice. The output artifact names the direction, the alternatives considered, the evidence, the key assumptions, and the next validation step. The response explains the discipline: the framework produces questions and options, not a verdict, and strategic logic is paired with financial, customer, and operational analysis rather than asserted. It also covers where the output lands in the organization's planning cycle so the direction is not a document that dies on a shelf.",
"assertions": [
"The process is framework-structured with a stated decision, owner, and constraints",
"Direction options are generated with assumptions and evidence made visible",
"The artifact names the decision, alternatives, evidence, assumptions, and next validation step",
"The response states that frameworks produce questions and options, not verdicts",
"The output connects to the planning cycle so it is acted on"
]
},
{
"id": "competitive-analysis",
"prompt": "We are entering a market with two strong incumbents and several startups. I need to understand the competitive landscape to decide whether entering is even wise and how to position. What analysis should I do and what framework fits?",
"expected_output": "A competitive analysis using industry-structure frameworks to examine the attractiveness of the market before the positioning question: the response applies a five-forces-style analysis to the market (rivalry intensity, threat of entry, supplier and buyer power, substitutes) to surface where the profit pool is and how hard it is to capture, then layers a competitor-positioning view (who serves which segments, on what dimensions they compete) to identify an underserved position. It explains the working method: state the decision (enter or not, and how), make the assumptions about competitors' likely responses explicit, and record the trade-offs and the evidence that would change the recommendation. The output names the decision, the alternatives, the evidence, and the next validation step rather than producing a framework-labeled verdict.",
"assertions": [
"Market attractiveness is assessed with an industry-structure framework before positioning",
"Competitor mapping identifies an underserved position rather than a crowded me-too one",
"Assumptions about competitor responses are explicit",
"The working method records evidence, trade-offs, and the decision",
"The output includes the next validation step, not just a framework classification"
]
},
{
"id": "growth-strategy-options",
"prompt": "Our core product is mature and growth has flattened. The options on the table are expanding into adjacent segments, moving upmarket, or building a platform play. How do I structure the growth-strategy decision so we compare the options fairly?",
"expected_output": "A growth-option analysis that compares the candidates on common criteria rather than their internal enthusiasm: the response frames each option (adjacent segments, upmarket, platform) with its market evidence, the capabilities it requires and whether they exist, the capital and timeline involved, and the risk profile, then evaluates them against the company's current position and constraints. The framework is used to generate questions and options: for each candidate the response identifies the critical uncertainty and the smallest experiment or evidence step that would validate or kill it. The output names the recommended option with its alternatives, assumptions, trade-offs, and the next validation step, and the response explicitly rejects picking an option because it is more exciting rather than better evidenced.",
"assertions": [
"Each growth option is assessed on common criteria: evidence, capabilities, capital, timeline, risk",
"The framework generates the critical uncertainty and the smallest validation step per option",
"Options are compared fairly rather than by internal enthusiasm",
"The output names the recommendation, alternatives, assumptions, and trade-offs",
"The next validation step is explicit for the chosen path"
]
},
{
"id": "resource-allocation",
"prompt": "Our company has three business units and limited capital for next year. Each unit is asking for more than it got last year, and the requests together exceed what we can fund. How do I structure the capital-allocation decision across the portfolio?",
"expected_output": "A resource-allocation process that compares the units on the economics of the choice rather than last year's budget: the response defines the decision criteria (return prospects, strategic fit, risk, and the funding required to actually move the needle for each unit), gathers comparable evidence for each unit against those criteria, and frames the options including the portfolio-level trade-offs — funding one unit fully versus spreading thin, or investing in new growth versus defending the core. It records the assumptions and the conditions under which the allocation would change, and it names the decision owner for the final call. The response explains the discipline that allocation decisions are about marginal return, not fairness or history, and it structures the conversation so the leadership team sees the trade-offs explicitly instead of negotiating from last year's numbers.",
"assertions": [
"Units are compared on marginal-return criteria, not last year's budget or fairness",
"Common evidence is gathered for each unit against the defined criteria",
"Portfolio-level trade-offs are framed explicitly, including defend-core versus new-growth",
"Assumptions and change conditions are recorded",
"A named decision owner makes the final call"
]
},
{
"id": "portfolio-choice",
"prompt": "We are considering acquiring a small competitor to close a capability gap, and separately a team inside is proposing to build the same capability from scratch. How do I structure the build-versus-buy portfolio decision?",
"expected_output": "A build-versus-buy analysis that compares the two options on the dimensions that matter: time to capability, total cost including integration and ongoing ownership, risk (acquisition integration risk versus build delivery risk), strategic control, and the option value each path creates. The response gathers comparable evidence for both options, makes the assumptions explicit (what the acquisition would actually cost including integration, what the build timeline really requires, what talent is available), and identifies the critical uncertainty that should decide between them plus the smallest step to resolve it, such as a scoped pilot or deeper diligence. It records the trade-offs and the decision owner, and it explicitly rejects defaulting to build because it feels cheaper or acquire because it is faster without comparing the full cost picture.",
"assertions": [
"Build and acquire are compared on common dimensions: time, total cost, risk, control, option value",
"Assumptions about integration cost and delivery timelines are made explicit",
"The critical uncertainty and the smallest step to resolve it are identified",
"The trade-offs and decision owner are recorded",
"Neither path is chosen by default without the full cost comparison"
]
}
]
}
systematic-debugging/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "systematic-debugging",
"evals": [
{
"id": "resist-quick-fix",
"prompt": "Our API started returning 500s after last night's deploy. The obvious suspect is the new rate-limiting middleware that was added in that deploy, and my teammate wants to roll it back immediately. What should we do before touching anything, and how do I prove the cause?",
"expected_output": "A response that follows the iron law: understand the bug before fixing. It starts by reproducing the failure consistently and narrowing when it began (checking the deploy window, logs, and error rates), gathering evidence before acting: the exact error payload, the stack trace, request patterns, and a minimal reproduction. It explicitly resists the rollback-before-diagnosis instinct by checking whether the rate limiter actually appears in the failing path and what evidence links it, and it prescribes the smallest experiment that confirms or refutes the hypothesis (compare a request that bypasses the middleware) before any rollback. It also checks recent changes beyond the middleware, because 'obvious suspect' framing usually comes from deploy proximity, not causality.",
"assertions": [
"The response resists immediate rollback and requires a consistent reproduction first",
"Evidence gathering includes the exact error, stack trace, request patterns, and deploy-window correlation",
"The response designs a minimal experiment that confirms or refutes the middleware hypothesis",
"It checks recent changes beyond the obvious suspect rather than assuming deploy proximity means causality",
"The response states what must be proven before acting on the fix"
]
},
{
"id": "test-failure-root-cause",
"prompt": "A unit test that passed for months started failing this morning. The test asserts a function returns a sorted list, and it now returns nearly-sorted. Nobody remembers changing the function. How do I find the real cause?",
"expected_output": "A root-cause investigation that treats the failing test as a signal to trace back to a change: check recent commits touching the function, its inputs, or shared dependencies (a locale, timezone, or Python-version change can flip sort behavior), reproduce with the exact failing input, and isolate by testing the function in isolation versus through the changed path. The response explicitly suspects environment and dependency drift, not just source edits: a date-parsing change, a different locale sort, or a dependency upgrade can alter behavior while the function is untouched. It prescribes bisecting the change history, checking the environment between the last pass and first failure, and writing a regression test that pins the previously-passing behavior once the cause is confirmed.",
"assertions": [
"The response traces the failure to a change via git history and the first-failure time window",
"It checks environment and dependency drift such as locale, timezone, or version changes",
"It reproduces with the exact failing input and isolates the function from the changed path",
"Bisecting the change history is part of the procedure",
"A regression test pins the previously-passing behavior once the cause is confirmed"
]
},
{
"id": "performance-regression",
"prompt": "Our checkout endpoint slowed from 120 ms to 900 ms over the last two weeks without a single obvious change. Users are complaining. I have profiler output but do not know where to start. How do I investigate a slow regression systematically?",
"expected_output": "A systematic performance investigation that establishes the baseline and the shape of the regression first: which percentile slowed, whether it is latency spikes or uniform slowdown, which call path the profiler attributes time to, and when the slope started (two weeks suggests gradual drift such as growing data or accumulating state, not a single deploy). The response ranks hypotheses by evidence: growing table sizes and missing index usage, connection-pool exhaustion, cache misses, new work added to the hot path, and background load. It prescribes measuring before optimizing: capture a flame graph under realistic load, compare against the 120 ms baseline, verify each candidate cause with a targeted experiment, and fix with a regression test or benchmark that prevents the slowdown from returning.",
"assertions": [
"The response characterizes the regression shape: percentiles, spikes versus uniform slowdown, and when it began",
"Gradual-drift causes such as growing data, state accumulation, and pool exhaustion are ranked as hypotheses",
"The response mandates measurement (flame graph, baseline comparison) before optimization",
"Each candidate cause is verified with a targeted experiment",
"A benchmark or regression test guards against the slowdown returning"
]
},
{
"id": "multi-component-evidence",
"prompt": "An end-to-end purchase flow fails intermittently across our mobile app, API gateway, payment provider, and background job pipeline. Each team says their component looks fine. Where do I start looking for evidence in a multi-component system?",
"expected_output": "A cross-component investigation that follows the data flow and the failure's shape instead of starting at any one team's logs: the response correlates the failure across components by tracing a single failing request end to end (trace IDs, timestamps across services), establishes the failure distribution (which steps fail, at what rate, correlated with what), and looks for the boundary conditions that single-component views miss: timeouts at handoff points, mismatched payload schemas between services, retry storms, and clock or concurrency mismatches. It prescribes building the end-to-end picture from one trace first, then comparing the failing trace against a successful one to find the divergence point, and only then narrowing to the owning team.",
"assertions": [
"The response traces a single failing request end to end before judging any component",
"It establishes the failure distribution and correlations across the system",
"Boundary conditions such as timeouts at handoffs, schema mismatches, and retry storms are explicitly checked",
"A failing trace is diffed against a successful trace to find the divergence point",
"Narrowing to an owning team happens only after the cross-component picture is built"
]
},
{
"id": "schema-environment-divergence",
"prompt": "The same service behaves differently in staging and production: features that work in staging fail in prod with validation errors. The code and config are supposedly identical. What could differ, and how do I find the divergence?",
"expected_output": "A schema-and-environment divergence investigation: the response enumerates what actually differs between environments despite identical code — database schema drift (a migration ran in staging but not prod, or vice versa), environment variables and feature flags, secret rotation, dependency versions resolved differently, and data itself (prod data hitting validation paths staging data never exercises). It prescribes diffing the real artifacts: schema migrations applied in each database, the resolved dependency lockfiles, the environment configuration, and the actual data shapes hitting the validation code. It warns that 'identical config' is usually an assumption, and the first step is to verify the assumption by diffing the environments rather than re-reading the code.",
"assertions": [
"The response enumerates real divergence sources: schema drift, flags and env vars, secrets, dependency resolution, data shapes",
"It mandates diffing the applied migrations in each database rather than trusting the code is identical",
"Environment variables, feature flags, and resolved dependencies are compared",
"The response treats 'identical config' as an assumption to verify by diffing, not a fact",
"Prod-specific data shapes are checked against the validation paths that reject them"
]
}
]
}
technology-radar/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "technology-radar",
"evals": [
{
"id": "quadrant-placement",
"prompt": "Our engineering org wants a technology radar to govern what we adopt. We are considering several technologies including a new frontend framework, an internal tool we already use everywhere, and a database that won a hackathon. How do I place technologies in the Adopt, Trial, Assess, and Hold quadrants, and what distinguishes them?",
"expected_output": "A radar placement framework with the quadrants defined by their operational meaning: Adopt for technologies we use widely with demonstrated fit and are confident recommending; Trial for technologies we are running in limited production scope with deliberate evaluation; Assess for technologies we are exploring with a small prototype to build evidence; and Hold for technologies we deliberately do not adopt or are phasing out. The response applies the definitions to the examples: the frontend framework goes to Assess or Trial depending on evidence so far, the internal tool's placement depends on whether it is a proven default (Adopt) or a growing liability (Hold), and the hackathon database goes to Assess — exploration is fine, but a hackathon demo is not evidence for production adoption. It explains the cardinal rules: a technology cannot move to Adopt without production evidence, Hold is a decision not a punishment, and placement is reviewed on a cadence.",
"assertions": [
"The quadrants are defined by operational meaning: production evidence, trial scope, exploration, and deliberate non-adoption",
"Each example technology is placed with reasoning tied to evidence",
"The hackathon-database example is placed in Assess with the explanation that demos are not adoption evidence",
"The rules about evidence requirements for Adopt and the meaning of Hold are stated",
"A review cadence for placements is included"
]
},
{
"id": "build-vs-buy-tco",
"prompt": "We need a feature that some teams think we should build and others want to buy. The build team says 'it is only a few weeks of work' and the buy advocates point to the sticker price. How do I structure a build-versus-buy decision with a real TCO analysis?",
"expected_output": "A build-versus-buy decision structured around total cost of ownership rather than the sticker price or the build estimate: the response defines the cost model covering build cost (development plus ongoing maintenance, support, and feature evolution at a stated annual maintenance rate), buy cost (license or subscription plus integration, customization, and vendor management), and the less tangible dimensions: time to capability, control and extensibility, risk (abandonment, lock-in, security posture), and fit to the actual requirement. It makes the assumptions explicit — including that the build estimate usually understates maintenance — and identifies the decision's critical uncertainty with the smallest step to resolve it, such as a scoped trial of the vendor product against a prototype of the built version. The output names the decision, alternatives, evidence, and the conditions that would change it.",
"assertions": [
"TCO covers build plus maintenance and buy plus integration and vendor management, not just headline numbers",
"Non-financial dimensions such as time to capability, control, and risk are included",
"The maintenance-cost assumption and build-estimate optimism are made explicit",
"The critical uncertainty and the smallest resolution step are identified",
"The output records the decision, alternatives, evidence, and change conditions"
]
},
{
"id": "deprecation-policy",
"prompt": "We have a legacy database that is stable but increasingly hard to staff, and a library that we know is unmaintained and vulnerable. I want to move both to Hold and phase them out. What does a deprecation policy look like, and how do I communicate it without alienating teams?",
"expected_output": "A deprecation policy that treats Hold as the start of a managed retirement, not a label: the response defines the policy components for each technology — the stated reason for the hold (maintainability, security, strategic fit), the transition guidance (what to use instead and who helps migrate), the timeline and milestones with migration support and owners, and the exceptions process for genuinely blocked cases. It explains the communication approach: the radar entry names the alternative and the support path so teams are not left stranded, deprecation is announced with enough runway for existing commitments, and the policy is enforced at the gate for new usage (no new systems on a Held technology) while existing systems get a realistic migration window. The response also covers measuring the retirement: tracking remaining usage and completing the removal when the last workload migrates.",
"assertions": [
"The policy defines reason, alternative, support path, timeline, owners, and an exceptions process",
"Hold blocks new usage at the gate while existing systems get a migration window",
"Communication includes the replacement and support so teams are not stranded",
"The deprecation is measured by remaining usage with a defined completion",
"The policy differentiates the database and library cases appropriately"
]
},
{
"id": "governance-process",
"prompt": "Right now any team can introduce any technology and we discover the consequences later. I want architecture governance that reviews technology choices without becoming a bureaucratic approval board that blocks everything. How do I design the process?",
"expected_output": "A governance process designed for speed and coverage: the response defines the structure — a lightweight review board or RFC process with a stated scope (new technologies entering the org, significant new uses of existing ones, and major retirements), decision criteria aligned with the radar and the org's strategy, and a fast-track lane for low-risk choices so the board does not become the bottleneck. It explains the operating rules: the board's job is to ask the right questions and record decisions with rationale, not to redesign every proposal; proposals carry the evidence (alternatives, risks, and the evaluation plan) and the board responds within a stated time; and decisions are recorded so the radar and the decision log stay the source of truth. It covers the failure modes to avoid: a board with no criteria that votes on taste, and an approval process with no fast track that pushes teams to bypass it.",
"assertions": [
"The process has a stated scope and decision criteria aligned with the radar",
"A fast-track lane keeps low-risk choices from becoming board bottlenecks",
"Proposals carry evidence and the board responds within a stated time",
"Decisions are recorded with rationale in a durable decision log",
"Failure modes are addressed: taste-based voting and bypass-prone approval processes"
]
},
{
"id": "tech-debt-prioritization",
"prompt": "Our codebase has accumulated technical debt: an aging build system, duplicated modules, an outdated library with known issues, and a growing test suite that takes too long. The team wants to 'fix the debt' but disagrees on what to do first. How do I quantify and prioritize remediation?",
"expected_output": "A technical-debt register and prioritization that makes the trade-offs visible: the response builds a register with an entry per debt item — principal (the cost of remediation estimated from the affected code), interest (the ongoing cost of not fixing: maintenance friction, incident risk, slower delivery), and the trigger conditions (the pain is paid when a change touches that area), then prioritizes by interest relative to principal and by how much the debt blocks current and planned work. The response explains the framework's rule: a debt is worth fixing when its interest exceeds the cost of remediation, and prioritization accounts for touch frequency — the build system that every change passes through pays interest daily and outranks a rarely touched module even if its principal is similar. It prescribes the sequencing and the metrics to show progress (remediation velocity, interest trend) so the work is not a one-time cleanup that regenerates.",
"assertions": [
"Debt items are entered in a register with principal, interest, and trigger conditions",
"Prioritization compares interest to principal and considers touch frequency",
"The frequently-touched build system is prioritized over a rarely touched module on interest grounds",
"Sequencing and progress metrics are prescribed",
"The framework prevents remediation from being a one-time cleanup that regenerates"
]
}
]
}
three/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "three",
"evals": [
{
"id": "basic-scene-setup",
"prompt": "I want to build a first Three.js scene that shows a rotating cube on a colored background in the browser. What is the minimal correct setup: renderer, scene, camera, geometry, and the render loop?",
"expected_output": "A working minimal Three.js scene setup: a WebGLRenderer created and appended to the DOM with a chosen clear color and size that matches the container, a Scene, a PerspectiveCamera positioned at a reasonable distance looking at the origin, a geometry with a MeshBasicMaterial or MeshStandardMaterial plus lighting if needed, a renderer.render call, and an animation loop driven by requestAnimationFrame that rotates the cube and renders each frame. The response explains why the render loop must call requestAnimationFrame continuously for animation and why the camera must look at the object after being positioned. It should present the code as a complete, copy-pasteable example rather than fragments, and note the WebGL context requirements for the page.",
"assertions": [
"The setup creates renderer, scene, camera, geometry, and material with correct wiring",
"The renderer is appended to the DOM and sized to its container",
"The camera is positioned and oriented toward the object",
"The animation loop uses requestAnimationFrame to rotate and render each frame",
"The example is complete enough to copy and run"
]
},
{
"id": "animation-loop",
"prompt": "My cube appears but does not move. I added rotation in the code but nothing animates. What is the usual cause of a static scene, and how do I structure the animation loop correctly?",
"expected_output": "A diagnosis of the static-scene problem with the loop structure as the core fix: the response explains that renderer.render must be called inside a requestAnimationFrame callback that schedules itself, so a single render outside the loop produces a still frame, and that rotation applied once before a single render is invisible. It prescribes the standard pattern: a function that updates object properties based on time (using clock.getDelta or elapsed time for frame-rate-independent speed), calls renderer.render, and schedules the next frame with requestAnimationFrame. It also covers the common secondary causes: the renderer or canvas is behind another element, the camera does not actually face the object, or rotation is applied to the wrong object, and it suggests checking the browser console for context errors.",
"assertions": [
"The static-scene cause is diagnosed as rendering outside a self-scheduling requestAnimationFrame loop",
"Time-based updates are prescribed so motion is frame-rate independent",
"The loop structure is shown as a complete pattern",
"Secondary causes such as camera orientation and canvas stacking are checked",
"Console context errors are suggested as a diagnostic step"
]
},
{
"id": "resize-handling",
"prompt": "My Three.js scene looks right when the window loads but distorts or crops when I resize the browser window. The canvas does not track the container. How do I handle resize correctly?",
"expected_output": "A resize-handling pattern that keeps the renderer and camera consistent with the container: the response prescribes listening for the resize event (or using a ResizeObserver on the container for layout-driven changes), updating the renderer size with renderer.setSize using the new pixel dimensions with updateStyle handling, and updating the camera's aspect ratio with camera.aspect and camera.updateProjectionMatrix before the next render. It explains the distortion mechanics: without updating aspect, the projection matrix stays from the old size and the scene stretches, and it covers device-pixel-ratio handling with renderer.setPixelRatio so the scene stays sharp on high-DPI displays without the canvas being enormous.",
"assertions": [
"The resize handler updates renderer size and camera aspect and calls updateProjectionMatrix",
"A ResizeObserver is suggested for container-driven layout changes",
"The mechanics of aspect mismatch causing distortion are explained",
"Pixel-ratio handling keeps the scene sharp without oversized canvases",
"The pattern is integrated with the render loop"
]
},
{
"id": "blank-canvas-debug",
"prompt": "My scene renders nothing — just a black or blank canvas. There are no console errors. The code looks right to me. How do I debug a Three.js scene that silently renders nothing?",
"expected_output": "A systematic debug procedure for a silently blank scene: verify the canvas is actually in the DOM and sized (a zero-height container or display:none parent produces nothing), verify the camera is inside the scene and looking at the geometry with correct near/far planes, verify the geometry has a material that is not transparent or fully dark under the current lighting (a MeshStandardMaterial without lights renders black, a MeshBasicMaterial does not need lights), verify the object is inside the camera frustum by position and scale, and check for a scene that never receives renderer.render. The response walks these checks as a decision tree ordered by likelihood and includes quick probes: temporarily using MeshBasicMaterial to rule out lighting, logging the camera-to-object distance, and inspecting the canvas size via the DOM.",
"assertions": [
"The debug procedure checks canvas presence and sizing first, including zero-height containers",
"Camera setup including frustum and orientation is verified",
"Material and lighting interaction is tested by switching to MeshBasicMaterial",
"Object position and scale inside the camera frustum are verified",
"The response orders checks by likelihood and includes concrete probes"
]
},
{
"id": "raycaster-interaction",
"prompt": "I want users to click on 3D objects in my scene to select them. I have several meshes in the scene and a camera that can move. How do I implement click-to-select with raycasting correctly?",
"expected_output": "A raycasting implementation that maps the click correctly: the response derives the normalized device coordinates from the mouse event using the renderer's viewport size, creates a raycaster, sets it from the camera with the NDC coordinates, intersects against the selectable meshes (only objects in the intersected set, not the whole scene graph unnecessarily), and handles the results: nearest intersection wins, highlighting the selected mesh and clearing previous selection. The response covers the pitfalls: forgetting to account for canvas position when computing NDC if the canvas is not fullscreen, raycasting before the renderer size is updated, and intersecting with invisible helpers or materials. It presents the complete pattern including the mousemove or click listener and cleanup of the previous highlight.",
"assertions": [
"Normalized device coordinates are computed from the event relative to the canvas",
"The raycaster is set from the camera and intersects a scoped set of objects",
"Nearest-intersection selection with highlight and clear-previous is implemented",
"Canvas-position and renderer-size pitfalls are handled",
"The pattern is complete with event listeners and cleanup"
]
}
]
}
traefik/evals/evals.json
+66
View File
@@ -0,0 +1,66 @@
{
"schema_version": 1,
"skill_name": "traefik",
"evals": [
{
"id": "http-routing-config",
"prompt": "I am setting up Traefik v3 as a reverse proxy in front of two services: a web app on port 3000 and an API on port 8080. Requests to app.example.com should go to the web app and api.example.com to the API. How do I configure the routers and services?",
"expected_output": "A Traefik configuration with the routing model explained: the response defines HTTP routers keyed on host rules (Host(`app.example.com`) and Host(`api.example.com`)), each router with a rule, a priority understanding for overlapping rules (Traefik v3 uses rule specificity, with explicit priorities only when needed), and each router pointing to a service defined with its load balancer and the correct backend port. The response explains the provider model: the same routing can be declared in a static config with a file provider or via the Docker provider's labels, and it shows a concrete YAML example for the file provider covering entryPoints, routers, and services, plus the corresponding Docker labels. It covers the operational details: which entrypoint the routers attach to, how to verify with a healthcheck or curl through the proxy, and the common mistake of a router rule that matches too broadly.",
"assertions": [
"Routers are keyed on host rules with correct rule syntax for the two domains",
"Services are defined with the right load balancer backend ports",
"The response shows the file-provider YAML and the Docker-label equivalent",
"Entrypoints and rule-priority behavior are explained",
"Verification via curl or healthcheck is included"
]
},
{
"id": "tls-acme",
"prompt": "My Traefik proxy is up and routing works over HTTP, but I need HTTPS with automatic certificates for two domains. I have a DNS provider with an API for verification. How do I configure TLS with ACME, and how do I know the certificates are being issued correctly?",
"expected_output": "An ACME configuration for Traefik: the response prescribes the certificatesResolver in the static config with the ACME storage path, the challenge type chosen for the setup (DNS challenge via the DNS provider's API when HTTP challenge is not viable, e.g., for wildcards or restricted inbound ports), and the provider credentials wired securely via environment or secrets rather than plaintext in the config. It explains how certificates attach to routers: routers with TLS enabled automatically request certificates for their Host rules through the resolver, and it covers the operational verification: checking the ACME storage for the issued certificates, confirming the certificate's SANs and expiry, testing the HTTPS handshake with curl, and monitoring renewal behavior. It flags the common failures: a resolver with no storage file permissions, wrong challenge provider credentials, and the ACME staging versus production endpoint confusion.",
"assertions": [
"The certificatesResolver is configured with storage and the DNS challenge using the provider API",
"Provider credentials are wired via environment or secrets, not plaintext",
"TLS-enabled routers automatically obtain certificates for their host rules",
"Verification covers the ACME store, SANs, expiry, and an HTTPS handshake test",
"Common ACME failures and staging-versus-production confusion are addressed"
]
},
{
"id": "middleware-chain",
"prompt": "I need to protect my API with rate limiting, add security headers to responses, and require a client certificate or basic auth for an admin path. How do I use Traefik middlewares for this, and how do they chain together?",
"expected_output": "A middleware design showing how middlewares compose: the response defines each middleware (rateLimit with a source criterion and burst, headers middleware adding the security headers, basicAuth with a hashed credentials file or forwardAuth for a client certificate requirement) and chains them on the routers that need them, explaining the ordering semantics — middleware chains apply in declared order and the response explains where auth sits relative to rate limiting so unauthenticated floods are rejected before hitting the backend. It covers the provider-agnostic mechanics: middlewares are defined once and referenced by name from routers (file provider or Docker labels), and the response flags the traps: middleware definitions that are declared but never referenced, rate limit source criteria that are too coarse, and basicAuth credentials stored as plaintext instead of hashed. It includes a concrete chain example with verification.",
"assertions": [
"Each middleware is defined with correct parameters: rateLimit, headers, basicAuth or forwardAuth",
"Middleware chains are ordered with rationale, including auth before backend access",
"Middlewares are defined once and referenced by routers",
"Common traps are flagged: unreferenced middlewares, plaintext credentials, coarse rate-limit keys",
"A concrete chained example with verification is included"
]
},
{
"id": "docker-provider-labels",
"prompt": "I want Traefik to discover my containers automatically: every container with a label should get a route without me editing a central config file. How do I use the Docker provider labels, and what labels do I need for a container to be routed?",
"expected_output": "A Docker-provider setup with the discovery model explained: the response prescribes enabling the docker provider in the static config (with the docker socket mounted and the provider enabled, noting the socket permission model), then shows the container labels that define routing: traefik.enable=true, the router rule (Host(...)), the entrypoint, the service port, and optionally middlewares and TLS settings. It explains the label conventions: the double-label form for router and service definitions, how labels map to the same router-service model as the file provider, and the provider's default behaviors such as network selection and when a container is excluded. The response flags the security and operational gotchas: the docker socket access grants proxy privileges, containers with multiple networks need the right network selected, and label typos silently produce no route. Verification is prescribed by hitting the generated route and inspecting the Traefik dashboard or API.",
"assertions": [
"The Docker provider is enabled with the socket mount and its permission model explained",
"The required labels for a routed container are listed: enable, rule, entrypoint, port",
"Label-to-router-service mapping and double-label conventions are explained",
"Socket security and multi-network selection gotchas are flagged",
"Verification via the generated route and dashboard or API is prescribed"
]
},
{
"id": "bad-gateway-troubleshoot",
"prompt": "My Traefik setup was working and now one route returns 502 Bad Gateway while others work. The backend container is running and the app inside responds on localhost. Where do I look?",
"expected_output": "A 502 diagnosis that walks the proxy-to-backend path: the response explains that 502 means Traefik could not reach the backend and directs the checks in order — is the backend actually listening on the address Traefik uses (containers on the same Docker network resolve and connect by service name, so localhost-only listeners fail), does the service definition use the correct port and scheme (http versus https against the backend), is the container on the network Traefik is configured to watch, and did the backend restart and change IP with the service pointing at a stale address. It also covers the health-check angle: if a healthcheck is defined, Traefik only routes to healthy backends, so an unhealthy backend yields a 502 even though the container runs. The response prescribes testing connectivity from inside the proxy's context and checking the Traefik logs for the specific dial error, which usually names the failing address.",
"assertions": [
"The 502 cause is explained as backend unreachability, not a routing-rule failure",
"The localhost-listener pitfall is called out: Traefik connects by service name over the container network",
"Service port, scheme, and network membership are checked",
"Healthcheck routing to healthy-backends-only is covered as a 502 source",
"Traefik logs are prescribed as the source of the specific dial error"
]
}
]
}