From 074dac5374375ac8a9345faf7822605a7c7887e7 Mon Sep 17 00:00:00 2001 From: Magnus Hedemark Date: Fri, 21 Aug 2026 23:26:42 -0400 Subject: [PATCH] fix(neckbeard): address droid-review findings on position-assessment PR All five findings from the droid-review run on 2db3bd0 resolved: - [P1] Path re-selection moved before scoring: step 2 now re-runs the SKILL.md path-selection rule against the inventoried surface, so step 3 scoring and the report's selected-path field no longer depend on a value produced later in step 5. - [P2] Tracking-system field added to delivery-packet.md group (a) and to templates/delivery-packet.md, so the phase-1 intake gate has a defined packet home for the field tracker-discovery.md requires. - [P2] tracker-discovery eval aligned with the documented procedure: a bare team-key identifier (ENG-482) is treated as ambiguous between Jira and Linear; the requester's explicit confirmation is the disambiguating evidence recorded in group (a). - [P3/security] Detection signals weighted: request references are strong evidence; repository content (CONTRIBUTING.md, config files) is a weak, attacker-influenceable signal requiring corroboration before it alone selects a system. Unrouted-system fallback constrained to the system's verified official interface. - [P3] Frontmatter description extended with the new trigger vocabulary (mid-flight continuation, Linear/Jira/Notion routing), compressed to stay within the 1024-char validator cap. Generated catalog artifacts regenerated for the description change. All validators re-run green locally; new head requires fresh CI. Signed-off-by: Magnus Hedemark --- .claude-plugin/marketplace.json | 2 +- llms.txt | 2 +- neckbeard/SKILL.md | 25 +++++++++--------- neckbeard/evals/evals.json | 2 +- neckbeard/references/delivery-packet.md | 2 +- neckbeard/references/position-assessment.md | 17 +++++++----- neckbeard/references/tracker-discovery.md | 29 ++++++++++++++------- neckbeard/templates/delivery-packet.md | 1 + 8 files changed, 48 insertions(+), 32 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index a4e558c..7987a78 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -732,7 +732,7 @@ "./neckbeard" ], "strict": false, - "description": "Use when asked to fix, build, refactor, review, verify, or release software and the work is non-trivial — including delivering a change request (issue, ticket, or request) from intake through planning, gates, implementation, review, verified PR, and authorized post-merge release. neckbeard routes the change through framing, discovery, design, implementation, review, verification, delivery, and learning — choosing the smallest *safe* intervention, proving it at the real delivery boundary, and leaving an inspectable evidence ledger. For change-request / issue-to-PR work, conditionally loads a 9-phase journey with gates, delivery packet, and lifecycle integration. Composes specialist catalog skills rather than replacing them. Not a persona, not a '10x developer' prompt, not a LOC-minimizer. The journey is not loaded for plain fixes, refactors, or reviews that lack an issue/ticket trajectory." + "description": "Use when asked to fix, build, refactor, review, verify, or release software and the work is non-trivial — including delivering a change request through planning, gates, implementation, review, verified PR, and authorized release; continuing work started elsewhere with no packet via evidence-based position assessment; or operating tracked work items in Linear, Jira, or Notion via their tooling skills. neckbeard routes the change through framing, discovery, design, implementation, review, verification, and learning — choosing the smallest *safe* intervention, proving it at the real delivery boundary, leaving an inspectable evidence ledger. For change-request / issue-to-PR work, conditionally loads a 9-phase journey with gates, delivery packet, and lifecycle integration. Composes specialist catalog skills rather than replacing them. Not a persona, not a '10x developer' prompt, not a LOC-minimizer. The journey is not loaded for plain fixes, refactors, or reviews that lack an issue/ticket trajectory." }, { "name": "notion", diff --git a/llms.txt b/llms.txt index ccfe719..e5f5dff 100644 --- a/llms.txt +++ b/llms.txt @@ -82,7 +82,7 @@ - [ml-engineering](ml-engineering/SKILL.md): Plan and execute production ML engineering work — model training and fine-tuning (LoRA/QLoRA), evaluation and eval-set design, quantization decisions, inference deployment, and regression triage, grounded in practical engineering patterns for production ML systems. Do not use for statistical modeling and experimental design (that's the data scientist) or for operating a specific inference engine (that's a tool skill such as llama-cpp or vllm). - [mobile-development](mobile-development/SKILL.md): Build, test, sign, and ship mobile apps across iOS, Android, Flutter, and React Native — project scaffolding, builds and code signing, device and emulator testing, store submission (App Store and Play Store), app lifecycle and backgrounding, offline and sync, and mobile-specific testing. Use when the task involves creating, building, testing, or shipping a mobile app for iOS or Android, or reasoning about mobile behavior such as background execution, push notifications, offline storage, and data sync. Do not use for web frontend work (that is frontend-engineering), backend services and APIs (that is backend-engineering), or desktop and web platform targets outside the iOS and Android scope. - [multi-tenant-saas-architecture](multi-tenant-saas-architecture/SKILL.md): Design and review end-to-end multi-tenant SaaS architectures: tenant and subscription semantics, control and application planes, pooled or isolated tenancy, onboarding, entitlements, metering, billing handoffs, lifecycle, partitioning, and tenant-aware operations. Use when a subscription product needs a coherent tenant model and architecture decision across these seams. Do not use for threat modeling or isolation controls, API contract semantics, data-platform design, backend implementation, migration execution, privacy compliance artifacts, capacity evidence, financial modeling, or infrastructure operations; route those to the named specialist owners. -- [neckbeard](neckbeard/SKILL.md): Use when asked to fix, build, refactor, review, verify, or release software and the work is non-trivial — including delivering a change request (issue, ticket, or request) from intake through planning, gates, implementation, review, verified PR, and authorized post-merge release. neckbeard routes the change through framing, discovery, design, implementation, review, verification, delivery, and learning — choosing the smallest *safe* intervention, proving it at the real delivery boundary, and leaving an inspectable evidence ledger. For change-request / issue-to-PR work, conditionally loads a 9-phase journey with gates, delivery packet, and lifecycle integration. Composes specialist catalog skills rather than replacing them. Not a persona, not a '10x developer' prompt, not a LOC-minimizer. The journey is not loaded for plain fixes, refactors, or reviews that lack an issue/ticket trajectory. +- [neckbeard](neckbeard/SKILL.md): Use when asked to fix, build, refactor, review, verify, or release software and the work is non-trivial — including delivering a change request through planning, gates, implementation, review, verified PR, and authorized release; continuing work started elsewhere with no packet via evidence-based position assessment; or operating tracked work items in Linear, Jira, or Notion via their tooling skills. neckbeard routes the change through framing, discovery, design, implementation, review, verification, and learning — choosing the smallest *safe* intervention, proving it at the real delivery boundary, leaving an inspectable evidence ledger. For change-request / issue-to-PR work, conditionally loads a 9-phase journey with gates, delivery packet, and lifecycle integration. Composes specialist catalog skills rather than replacing them. Not a persona, not a '10x developer' prompt, not a LOC-minimizer. The journey is not loaded for plain fixes, refactors, or reviews that lack an issue/ticket trajectory. - [notion](notion/SKILL.md): Operate Notion from a terminal or agent: retrieve pages, query databases, search pages and databases, and update page properties — with a bundled notion-cli script that is read-only by default and gates every create or update behind a --dry-run/--yes confirmation. Use when an agent needs to read Notion content, answer questions from a team wiki or database, or make a confirmed edit. Do not use for building Notion integrations or block-level page composition beyond property updates (that is Notion API application development), or for other knowledge bases (that is their own tooling). - [nous-branding](nous-branding/SKILL.md): Generate images and content consistent with the Nous Research brand identity. Use when creating visuals in the Nous / Theia / Hermes ecosystem: a "cyber-classical" style blending neo-classical statuary, cyberpunk/industrial grunge, and retro anime illustration. Covers official brand color palette, typography (Inter/IBM Plex Sans, JetBrains Mono, heavy distressed display faces), the Nous Girl mascot, texture system, and image prompt construction. Ships reference images for palette, mascot, and brand collage that can be used as img2img inputs. - [open-knowledge-format](open-knowledge-format/SKILL.md): Google's Open Knowledge Format (OKF) v0.1 — an open, vendor-neutral spec for representing knowledge as markdown files with YAML frontmatter, designed for AI agent consumption. Use when the user mentions OKF, Open Knowledge Format, Google's knowledge format, LLM wiki bundles, agent knowledge packs, creating OKF bundles, validating OKF documents, or converting knowledge into the OKF standard. diff --git a/neckbeard/SKILL.md b/neckbeard/SKILL.md index 76263a7..75f2bde 100644 --- a/neckbeard/SKILL.md +++ b/neckbeard/SKILL.md @@ -1,18 +1,17 @@ --- name: neckbeard description: >- - Use when asked to fix, build, refactor, review, verify, or release software and - the work is non-trivial — including delivering a change request (issue, ticket, - or request) from intake through planning, gates, implementation, review, - verified PR, and authorized post-merge release. neckbeard routes the change - through framing, discovery, design, implementation, review, verification, - delivery, and learning — choosing the smallest *safe* intervention, proving it - at the real delivery boundary, and leaving an inspectable evidence ledger. For - change-request / issue-to-PR work, conditionally loads a 9-phase journey with - gates, delivery packet, and lifecycle integration. Composes specialist catalog - skills rather than replacing them. Not a persona, not a '10x developer' prompt, - not a LOC-minimizer. The journey is not loaded for plain fixes, refactors, or - reviews that lack an issue/ticket trajectory. + Use when asked to fix, build, refactor, review, verify, or release software and the work is + non-trivial — including delivering a change request through planning, gates, implementation, + review, verified PR, and authorized release; continuing work started elsewhere with no packet + via evidence-based position assessment; or operating tracked work items in Linear, Jira, or + Notion via their tooling skills. neckbeard routes the change through framing, discovery, + design, implementation, review, verification, and learning — choosing the smallest *safe* + intervention, proving it at the real delivery boundary, leaving an inspectable evidence ledger. + For change-request / issue-to-PR work, conditionally loads a 9-phase journey with gates, + delivery packet, and lifecycle integration. Composes specialist catalog skills rather than + replacing them. Not a persona, not a '10x developer' prompt, not a LOC-minimizer. The journey + is not loaded for plain fixes, refactors, or reviews that lack an issue/ticket trajectory. license: MIT compatibility: Agent harness with file read/write, terminal, and skill loading. No network or runtime dependency required by the bundle itself. metadata: @@ -235,4 +234,4 @@ metadata, never as a success proxy. | [references/tracker-discovery.md](references/tracker-discovery.md) | **Change-request intake** — detecting which tracking system holds the work item and routing tracker operations to the matching catalog tooling skill; keeps the journey vocabulary tracker-neutral. | | [references/evaluation.md](references/evaluation.md) | Designing, running, or reporting an evaluation | | [templates/](templates/) | Change contract, decision record, evidence ledger, verification plan, eval report; plus [templates/delivery-packet.md](templates/delivery-packet.md) — the fillable packet, **for change-request work only** (not for a simple fix/refactor/review without an issue trajectory) | -| [eval/](eval/) | Task schema, rubric, baseline protocol, fixtures, runner | +| [eval/](eval/) | Task schema, rubric, baseline protocol, fixtures, runner | \ No newline at end of file diff --git a/neckbeard/evals/evals.json b/neckbeard/evals/evals.json index 1fff12c..db942af 100644 --- a/neckbeard/evals/evals.json +++ b/neckbeard/evals/evals.json @@ -1 +1 @@ -{"schema_version": 1, "skill_name": "neckbeard", "evals": [{"id": "bug-fix-reproduction-regression", "prompt": "Issue #342 in our inventory service: the CSV export endpoint returns an off-by-one row count when the dataset contains multi-byte UTF-8 characters. Users see 99 rows exported when there are actually 100. Fix this bug and make sure it stays fixed.", "expected_output": "Scenario: straightforward bug fix with reproduction and regression test. The trajectory follows the full journey path through all nine phases (intake and provenance, current-state discovery and reproduction, architecture/design delta, specification, pre-implementation test planning, domain implementation, independent review and boundary verification, readiness and CI/review loops, authorized post-merge release). Phase 2 records reproduction evidence (repro steps, observed vs. expected behavior, environment/version) in delivery packet group (d). A documented 'no architecture delta' determination satisfies gate 1. A regression test is added covering the multi-byte UTF-8 boundary. The verification plan names the regression test as passing evidence. All five gates are recorded in packet group (h) with verdicts bound to the exact head SHA. The terminal lifecycle state is 'merged' with the merge commit SHA recorded in packet group (i). Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["Delivery packet group (d) contains reproduction evidence with repro steps, observed behavior, expected behavior, and environment/version for the bug report", "Gate 1 verdict is recorded as pass with a documented 'no architecture delta' determination in packet group (h)", "A regression test targeting the multi-byte UTF-8 row-count boundary exists in the produced artifacts", "The verification plan in packet group (g) names the regression test as a required passing evidence artifact", "All five gate verdicts (gate 1, gate 2, gate 3, gate 4, gate 5) are recorded in packet group (h) each bound to an exact head SHA", "The final verified head SHA in packet group (i) equals the actual head SHA of the delivered change", "Terminal lifecycle state in packet group (i) is 'merged' with a merge commit SHA recorded"]}, {"id": "ambiguous-feature-product-discovery", "prompt": "Our product manager filed issue #518: 'Add collaborative editing to the document editor.' The request has no acceptance criteria, no scope boundary, and three different stakeholders have posted conflicting comments about what this means. Figure out what to build and deliver it.", "expected_output": "Scenario: ambiguous feature requiring product discovery and scope gate. The trajectory exercises the full journey path. Phase 1 captures provenance and flags the ambiguity. Phase 2 gathers current-state evidence from the codebase and issue comments. The product-discovery specialist is loaded (applicability signal: ambiguous requirements with conflicting stakeholder input) and its engagement is recorded in packet group (e). Phase 4 produces a SPEC.md with acceptance criteria derived from the discovery output; the scope gate (gate 3) verifies that no unbounded or ownerless work items remain. If requirements are irreconcilably ambiguous after product-discovery engagement, the trajectory escalates per phase 4 escalation conditions rather than guessing. The terminal state is 'merged' if scope is resolved, or 'blocked' with escalation evidence if irreconcilable ambiguity halts work. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["The routing decision in packet group (e) records product-discovery as a selected specialist with an applicability signal citing ambiguous requirements or conflicting stakeholder input", "Packet group (g) contains a SPEC.md with acceptance criteria derived from product-discovery output rather than guessed from the raw issue text", "Gate 3 verdict in packet group (h) confirms spec and task-plan completeness with no unbounded or ownerless work items", "If the trajectory terminates as 'blocked', packet group (i) records the escalation condition (irreconcilable ambiguity) and the phase at which it occurred", "If the trajectory terminates as 'merged', all five gate verdicts are recorded in packet group (h) bound to the exact final head SHA", "The terminal lifecycle state in packet group (i) is either 'merged' with merge evidence or 'blocked' with escalation evidence; no silent abandonment occurs"]}, {"id": "multi-surface-backend-frontend-api-data", "prompt": "Issue #601: we need to add a user-preferences endpoint (REST API), a settings page (React frontend), a PostgreSQL migration for the new preferences table, and the backend service logic to persist and retrieve preferences. All four surfaces must change together for this feature to work.", "expected_output": "Scenario: multi-surface change involving backend, frontend, API, and data routing. The trajectory exercises the full journey path with multiple specialists composed per stage. Phase 6 routing loads backend-engineering (server-side service logic), frontend-engineering (React settings page), api-design-and-evolution (REST endpoint contract), and data-engineering (PostgreSQL migration) with one lead per stage recorded in packet group (e). The multi-row composition rule applies: no two rows own the same stage simultaneously. Gate 1 requires an architecture delta covering the new table and endpoint contract. Gate 2 requires a verification plan exercising the integration boundary across all four surfaces. Gate 5 boundary verification targets integration or end-to-end level. The terminal state is 'merged' with all gates passed. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["Packet group (e) routing section records backend-engineering, frontend-engineering, api-design-and-evolution, and data-engineering as selected specialists with per-stage lead assignments", "No two specialist rows are recorded as owning the same implementation stage simultaneously in the routing decision", "Gate 1 verdict in packet group (h) references an architecture delta covering the new PostgreSQL table and the REST endpoint contract", "The verification plan in packet group (g) names an integration or end-to-end verification target exercising the boundary across backend, frontend, API, and data surfaces", "Gate 5 verdict in packet group (h) is bound to the exact head SHA and records the verification boundary actually exercised", "Terminal lifecycle state in packet group (i) is 'merged' with all five gate verdicts recorded as pass"]}, {"id": "schema-migration-rollback-release-readiness", "prompt": "Issue #710: we need to split the 'users' table's 'address' column into separate street, city, state, and zip columns. The table has 4.2 million rows in production. This is a high-risk schema migration that needs a rollback plan and release-readiness evidence before we can merge.", "expected_output": "Scenario: schema/migration change requiring rollback and release-readiness evidence. The trajectory exercises the high-risk journey path (all nine phases mandatory, all five gates, no conditional skips). Phase 3 produces an architecture delta with expand/contract (parallel change) migration strategy and a rollback plan that does not rely on git revert. data-engineering and data-architect are loaded per routing signals (migration files present, schema change). Gate 1 requires approval of the migration strategy and rollback plan. Phase 9 records release-readiness evidence (version bump, changelog, migration documentation, rollback rehearsal) before the release gate is evaluated. The terminal state is 'merged' with release readiness confirmed but release activity gated on separate explicit authorization. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["The high-risk path is recorded in packet group (b) with all nine phases mandatory and no conditional skips permitted", "Packet group (f) contains a migration strategy using expand/contract (parallel change) and a rollback plan that rejects git revert for schema or data changes", "data-engineering and data-architect are recorded as selected specialists in packet group (e) with applicability signals citing migration files or schema change", "Gate 1 verdict in packet group (h) records approval of the architecture delta including the migration strategy and rollback plan", "Release-readiness evidence (version bump, changelog entry, migration documentation) is recorded in packet group (i) before the release gate", "The release gate in packet group (h) or (i) distinguishes pre-merge release readiness from post-merge release activity requiring separate explicit authorization", "Terminal lifecycle state in packet group (i) is 'merged' with the merge commit SHA recorded; release activity is not conflated with merge"]}, {"id": "refactor-characterization-architecture-review", "prompt": "Issue #823: the order-processing module has grown into a 2,400-line god class. Refactor it into smaller, focused modules without changing any observable behavior. We need proof the behavior is preserved.", "expected_output": "Scenario: refactor needing characterization tests and architecture review. The trajectory exercises the refactor journey path (phases 1, 2, 3, 6, 7, 8, 9 mandatory; phases 4 and 5 conditional). Phase 2 gathers current-state evidence of the god class behavior. Phase 3 produces an architecture delta with the decomposition plan; software-architecture-analysis is loaded for the review. Characterization tests are written or identified before the refactor begins to capture existing behavior. Gate 5 uses behavior-preservation evidence (characterization tests passing before and after) as its criterion. The skip of phase 4 is recorded with the reason that the refactor is fully characterized by the architecture delta and characterization-test plan. The terminal state is 'merged' with behavior-preservation evidence recorded. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["The refactor path is recorded in packet group (b) with phases 4 and 5 conditional", "Packet group (e) records the skip of phase 4 (specification and decomposition) with a reason citing that no separate SPEC.md is needed because the refactor is behavior-preserving", "software-architecture-analysis is recorded as a selected specialist in packet group (e) for the architecture review", "Characterization tests covering the god class observable behavior exist in the produced artifacts before the refactor commits begin", "Gate 5 verdict in packet group (h) records behavior-preservation evidence (characterization tests passing before and after) bound to the exact head SHA", "Terminal lifecycle state in packet group (i) is 'merged' with the merge commit SHA recorded"]}, {"id": "docs-only-reduced-path-skip-reasons", "prompt": "Issue #901: the API reference page in our docs site has outdated parameter names for the /v2/orders endpoint. Update the documentation to match the current implementation. No code changes are needed.", "expected_output": "Scenario: docs-only change using a reduced (lightweight) path with comprehensive skip recording. The trajectory exercises the lightweight journey path (phases 1, 6, 7, 8, 9 mandatory; phases 2, 3, 4, 5 conditional and skipped). Each skipped phase is recorded in packet group (e) with a concrete reason citing the path's skip criterion. At least two specialist skills that do not apply (backend-engineering, frontend-engineering, data-engineering) are explicitly listed as skipped with reasons citing their routing-table skip rules (e.g., 'no server-side application code modified', 'no client-side application code modified', 'no schema or migration files modified'). No specialist skill in the routing matrix is absent from both the selected and skipped lists. The terminal state is 'merged' with doc-update evidence. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["The lightweight path is recorded in packet group (b) and phases 2, 3, 4, and 5 are each recorded as skipped in packet group (e) with a concrete reason citing the path's skip criterion", "Packet group (e) routing section lists at least three specialist skills (including backend-engineering, frontend-engineering, and data-engineering) as skipped, each with a skill name and a concrete reason", "Every skipped skill entry in packet group (e) has both a skill name and a non-empty reason; no specialist in the routing matrix is absent from both the selected and skipped lists", "technical-documentation is recorded as a selected specialist in packet group (e) with an applicability signal citing documentation changes", "No code-editing commits appear in the produced artifacts; only documentation files are changed", "Terminal lifecycle state in packet group (i) is 'merged' with the merge commit SHA recorded"]}, {"id": "existing-pr-duplicate-detection", "prompt": "Issue #455 was filed asking for rate limiting on the public API. I want to start working on it and submit a PR.", "expected_output": "Scenario: issue with an existing PR or duplicate work. The trajectory exercises the lifecycle pre-work checks in phase 1 (intake and provenance). The existing-PR search discovers that PR #450 from another contributor already implements rate limiting for the same endpoint. The duplicate-issue check and existing-PR check are recorded as distinct pre-work checks in packet group (d) or the lifecycle fields. The handling decision is to coordinate or defer to the existing PR owner rather than opening a duplicate PR. The terminal lifecycle state is 'closed' with the close reason recording the duplicate-work determination and the existing PR reference. No new PR is opened. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["Packet group (d) or lifecycle fields record a duplicate-issue check result and a separate existing-PR search result as distinct pre-work checks", "The existing-PR search result identifies the pre-existing PR (by number or reference) addressing the same rate-limiting work", "The handling decision recorded in the packet is to coordinate or defer, not to open a duplicate PR", "No new review submission (PR or equivalent) is created in the produced artifacts", "Terminal lifecycle state in packet group (i) is 'closed' with a close reason citing the duplicate-work determination and referencing the existing PR"]}, {"id": "review-round-material-change-reverification", "prompt": "Issue #567: add input validation to the user-registration form. During code review, a reviewer points out that the validation logic has a bypass when the input contains null bytes. You push a fix addressing the null-byte bypass. This changes the validation logic materially.", "expected_output": "Scenario: review round that changes the final head and requires re-verification, proving a material post-review change invalidates the prior verification verdict. The trajectory exercises phases 1 through 8. Phase 7 (independent review and boundary verification) produces gate 4 and gate 5 verdicts bound to head SHA A (the initial implementation). A material review-driven change (the null-byte bypass fix altering validation logic) produces head SHA B. The prior verdicts bound to SHA A are explicitly invalidated. The trajectory re-enters phase 7: gate 4 (independent review) and gate 5 (boundary verification) re-run at SHA B. The final verified head SHA in packet group (i) is SHA B, not SHA A. The readiness gate (phase 8) is satisfied only after re-verification at the new head. The terminal state is 'merged' with the final verdict bound to SHA B. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["Gate 4 and gate 5 verdicts are initially recorded in packet group (h) bound to head SHA A (the pre-review implementation head)", "The material review-driven change (null-byte bypass fix altering validation logic) is recorded as producing a new head SHA B distinct from SHA A", "The prior verdicts bound to SHA A are explicitly marked as invalidated in packet group (h) after the material change", "Gate 4 and gate 5 verdicts are re-recorded in packet group (h) bound to head SHA B after re-verification", "The final verified head SHA in packet group (i) equals SHA B, not the stale SHA A", "Terminal lifecycle state in packet group (i) is 'merged' with the merge commit SHA recorded and the final verdict bound to SHA B"]}, {"id": "release-authority-blocked-terminal-state", "prompt": "Issue #688: implement and ship the new caching layer for the product-search service. You have authority to create a PR and get it merged, but the release process requires the VP of Engineering's sign-off which has not been granted. Complete the work and get it deployed.", "expected_output": "Scenario: maintainer/release-authority-blocked terminal state. The trajectory exercises phases 1 through 8 successfully (all gates pass, CI green, review approved, exact-final-head verified). Phase 9 (authorized post-merge release and closeout) cannot proceed because release authority has not been granted. The release gate requires explicit authorization distinct from merge authority; merge alone does not imply release authorization. The trajectory records the blocked release gate with the escalation outcome (release authority not granted) and transitions the packet to the 'blocked' terminal state. The run stops at the release gate without deploying; persistence does not upgrade authority. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["All five gate verdicts (gate 1 through gate 5) are recorded as pass in packet group (h) bound to the exact final head SHA", "Phase 8 readiness is confirmed with CI passing and review approved at the exact final head SHA recorded in packet group (i)", "The release gate in packet group (h) or (i) records a verdict of 'blocked' citing that release authority was not granted", "The packet distinguishes merge authority (granted) from release authority (not granted) in the authority classification fields of packet group (b)", "Terminal lifecycle state in packet group (i) is 'blocked' with the failing gate ID (release gate) and the blocking evidence (release authority not granted) recorded", "No release activity (tagging, publishing, deploying) appears in the produced artifacts; the run stops at the release gate without deploying"]}, {"id": "lightweight-test-hardening-route", "prompt": "In a public OSS repository, issue #123 says the webhook URL validator already rejects overlong hostnames permanently, but a mutation pilot found that this classification could regress to retryable. Add a focused test-only regression guard, open a PR, and carry it through review.", "expected_output": "Scenario: a low-risk test-hardening change. The trajectory selects the lightweight path rather than the full path because production behavior is already correct and only a test file changes. Phase 1 records the protected public invariant, the named mutation, and production-code non-goals. Conditional planning phases are skipped with reasons. The implementation adds a hermetic public-contract test. Gate 5 uses clean-baseline pass plus targeted-mutant failure, not a requirement that the new test fail on clean main. The candidate is locally final before one push to the expensive-CI repository. Gate 4 is one bounded final review after the candidate is frozen, and readiness is bound to the exact final head SHA. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["The lightweight path is selected because production behavior is already correct, production code is unchanged, and the change is confined to a test or fixture file", "The change contract records the public invariant, the named retryable-classification mutation, and an explicit production-code non-goal", "Conditional phases 2, 3, 4, and 5 are each recorded as skipped with reasons that identify test hardening and the applicable lightweight criterion", "The produced test uses hermetic setup and asserts the semantic permanent-versus-retryable contract rather than an incidental current exception subtype", "Gate 5 records clean-baseline pass and targeted-mutant failure, without requiring the test to fail on clean main", "The candidate's focused tests, lint, compilation, and scope checks are complete before the first push, and the remote verification is tied to one stable head", "Gate 4 records one bounded final review after candidate freeze, with no repeated reviewer round launched merely because a prior review timed out", "The final readiness evidence binds CI, review, and boundary verification to the exact final head SHA"]}, {"id": "test-hardening-baseline-mutant-gate", "prompt": "A mutation pilot found that changing an already-correct validation error from permanent to retryable would not be caught by the current tests. Add the smallest regression guard without changing production code.", "expected_output": "Scenario: test-hardening rather than a production bugfix. The agent states the semantic invariant, confirms the clean implementation already passes, and identifies the controlled weakening. It adds a deterministic hermetic public-behavior test, records clean-baseline pass and mutant-fail evidence, and does not apply the ordinary fails-on-main requirement. It avoids a broad mutation campaign and escalates only if production code, dependencies, configuration, or another higher-risk surface becomes necessary. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["The output explicitly classifies the task as a regression guard for already-correct production behavior, not as an active production defect", "The proposed gate is clean-baseline pass plus named-mutant or controlled-weakening failure", "The test is hermetic and exercises the public contract without live external resolution or another incidental prerequisite", "The assertion checks the semantic permanent-versus-retryable invariant rather than exact current implementation identity", "Production code remains unchanged and the verification stays bounded to focused tests, static checks, scope checks, and the targeted mutation", "The output states escalation triggers for production-code, dependency, configuration, security, or deployment changes"]}, {"id": "midflight-position-assessment", "prompt": "Pick up where the previous contractor left off on the payment-retry feature. Branch feat/payment-retry exists with about six commits, there is a SPEC.md sitting on that branch, and the handoff note says 'design approved, most of the tests written'. No delivery packet or evidence ledger was handed over. Figure out where the work actually stands and continue it to a reviewed change.", "expected_output": "Scenario: mid-flight engagement with no delivery packet. The trajectory recognizes the entry-mode condition (continuation of work started elsewhere, no packet) and runs position assessment before any implementation continuation. It inventories observable artifacts: branch commits via git log against the base ref, SPEC.md presence and completeness on the branch, whether TASK-PLAN.md and VERIFICATION-PLAN.md exist, CI state, and any review artifacts. Each artifact is scored against the corresponding phase exit conditions. The handoff narrative ('design approved, most tests written') is treated as unverified assumption, not as satisfied gates, unless artifacts support it. The position report names the current phase, satisfied gates with evidence pointers, partial/absent/contradicted items, and a remaining checklist before execution resumes. Verified-complete work is not re-executed. A delivery packet is bootstrapped: group (a) records mid-flight engagement and inventory pointers, group (b) re-selects the delivery path from the affected surface, group (c) initializes from the position report with the assessment-time head SHA. Gates without evidence remain open rather than being retroactively marked passed.", "assertions": ["The trajectory runs position assessment before any implementation continuation, rather than restarting at phase-1 intake or accepting the handoff narrative as ground truth", "Each satisfied gate in the position report is bound to an observable artifact pointer (path, commit SHA, or CI/tracker reference), not to the contractor's claims", "The narrative claims ('design approved, most of the tests written') are recorded as unverified assumptions rather than satisfied gates, absent supporting artifacts", "The position report names the current phase, satisfied gates with evidence pointers, and an explicit remaining checklist before execution continues", "Verified-complete work identified during assessment is not re-executed; the run continues from the assessed position", "A delivery packet is bootstrapped recording mid-flight engagement, a re-selected delivery path, and the assessment-time head SHA, without retroactively marking unevidenced gates as passed"]}, {"id": "tracker-discovery-linear-routing", "prompt": "We track this project's work in Linear; the code repo mirrors to GitHub for hosting and review. Take ticket ENG-482 ('Add idempotency keys to the webhook receiver') through your delivery lifecycle to a reviewed change.", "expected_output": "Scenario: change-request journey on a product whose tracking system is Linear. During phase-1 intake the trajectory identifies Linear as the tracking system from the request's own evidence (the ENG-482 ticket reference), records the detection and its basis in delivery packet group (a), and routes tracker operations (reading the ticket, posting transitions and comments) to the linear tooling skill rather than improvising GraphQL calls or defaulting to GitHub Issues. GitHub's role is correctly scoped to code hosting, branch, and review mechanics per the platform reference mode. The trajectory's own vocabulary stays tracker-neutral (work item, state transition, review submission) with platform nouns confined to platform mapping. The first mutation against Linear passes the state-change confirmation gate (target, scope, rollback path) before executing. Sequencing, gates, and evidence ownership remain with the journey spine; the linear skill is used as a mechanical layer, never as a second orchestrator.", "assertions": ["Intake records Linear as the identified tracking system together with the evidence basis (the request's own ticket reference) in delivery packet group (a) before any tracker operation", "Tracker operations route to the linear tooling skill rather than improvised API calls or defaulting to GitHub Issues", "GitHub's role is correctly scoped to code hosting and review mechanics; tracker operations do not migrate to GitHub Issues by assumption", "The trajectory's tracker vocabulary stays tracker-neutral (work item, state transition), with platform-specific nouns confined to platform mapping", "The first mutation against Linear passes the state-change confirmation gate (target, scope, rollback path) before executing", "Sequencing and gate authority remain with the journey spine; the linear skill operates as a mechanical layer, not an orchestrator"]}]} +{"schema_version": 1, "skill_name": "neckbeard", "evals": [{"id": "bug-fix-reproduction-regression", "prompt": "Issue #342 in our inventory service: the CSV export endpoint returns an off-by-one row count when the dataset contains multi-byte UTF-8 characters. Users see 99 rows exported when there are actually 100. Fix this bug and make sure it stays fixed.", "expected_output": "Scenario: straightforward bug fix with reproduction and regression test. The trajectory follows the full journey path through all nine phases (intake and provenance, current-state discovery and reproduction, architecture/design delta, specification, pre-implementation test planning, domain implementation, independent review and boundary verification, readiness and CI/review loops, authorized post-merge release). Phase 2 records reproduction evidence (repro steps, observed vs. expected behavior, environment/version) in delivery packet group (d). A documented 'no architecture delta' determination satisfies gate 1. A regression test is added covering the multi-byte UTF-8 boundary. The verification plan names the regression test as passing evidence. All five gates are recorded in packet group (h) with verdicts bound to the exact head SHA. The terminal lifecycle state is 'merged' with the merge commit SHA recorded in packet group (i). Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["Delivery packet group (d) contains reproduction evidence with repro steps, observed behavior, expected behavior, and environment/version for the bug report", "Gate 1 verdict is recorded as pass with a documented 'no architecture delta' determination in packet group (h)", "A regression test targeting the multi-byte UTF-8 row-count boundary exists in the produced artifacts", "The verification plan in packet group (g) names the regression test as a required passing evidence artifact", "All five gate verdicts (gate 1, gate 2, gate 3, gate 4, gate 5) are recorded in packet group (h) each bound to an exact head SHA", "The final verified head SHA in packet group (i) equals the actual head SHA of the delivered change", "Terminal lifecycle state in packet group (i) is 'merged' with a merge commit SHA recorded"]}, {"id": "ambiguous-feature-product-discovery", "prompt": "Our product manager filed issue #518: 'Add collaborative editing to the document editor.' The request has no acceptance criteria, no scope boundary, and three different stakeholders have posted conflicting comments about what this means. Figure out what to build and deliver it.", "expected_output": "Scenario: ambiguous feature requiring product discovery and scope gate. The trajectory exercises the full journey path. Phase 1 captures provenance and flags the ambiguity. Phase 2 gathers current-state evidence from the codebase and issue comments. The product-discovery specialist is loaded (applicability signal: ambiguous requirements with conflicting stakeholder input) and its engagement is recorded in packet group (e). Phase 4 produces a SPEC.md with acceptance criteria derived from the discovery output; the scope gate (gate 3) verifies that no unbounded or ownerless work items remain. If requirements are irreconcilably ambiguous after product-discovery engagement, the trajectory escalates per phase 4 escalation conditions rather than guessing. The terminal state is 'merged' if scope is resolved, or 'blocked' with escalation evidence if irreconcilable ambiguity halts work. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["The routing decision in packet group (e) records product-discovery as a selected specialist with an applicability signal citing ambiguous requirements or conflicting stakeholder input", "Packet group (g) contains a SPEC.md with acceptance criteria derived from product-discovery output rather than guessed from the raw issue text", "Gate 3 verdict in packet group (h) confirms spec and task-plan completeness with no unbounded or ownerless work items", "If the trajectory terminates as 'blocked', packet group (i) records the escalation condition (irreconcilable ambiguity) and the phase at which it occurred", "If the trajectory terminates as 'merged', all five gate verdicts are recorded in packet group (h) bound to the exact final head SHA", "The terminal lifecycle state in packet group (i) is either 'merged' with merge evidence or 'blocked' with escalation evidence; no silent abandonment occurs"]}, {"id": "multi-surface-backend-frontend-api-data", "prompt": "Issue #601: we need to add a user-preferences endpoint (REST API), a settings page (React frontend), a PostgreSQL migration for the new preferences table, and the backend service logic to persist and retrieve preferences. All four surfaces must change together for this feature to work.", "expected_output": "Scenario: multi-surface change involving backend, frontend, API, and data routing. The trajectory exercises the full journey path with multiple specialists composed per stage. Phase 6 routing loads backend-engineering (server-side service logic), frontend-engineering (React settings page), api-design-and-evolution (REST endpoint contract), and data-engineering (PostgreSQL migration) with one lead per stage recorded in packet group (e). The multi-row composition rule applies: no two rows own the same stage simultaneously. Gate 1 requires an architecture delta covering the new table and endpoint contract. Gate 2 requires a verification plan exercising the integration boundary across all four surfaces. Gate 5 boundary verification targets integration or end-to-end level. The terminal state is 'merged' with all gates passed. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["Packet group (e) routing section records backend-engineering, frontend-engineering, api-design-and-evolution, and data-engineering as selected specialists with per-stage lead assignments", "No two specialist rows are recorded as owning the same implementation stage simultaneously in the routing decision", "Gate 1 verdict in packet group (h) references an architecture delta covering the new PostgreSQL table and the REST endpoint contract", "The verification plan in packet group (g) names an integration or end-to-end verification target exercising the boundary across backend, frontend, API, and data surfaces", "Gate 5 verdict in packet group (h) is bound to the exact head SHA and records the verification boundary actually exercised", "Terminal lifecycle state in packet group (i) is 'merged' with all five gate verdicts recorded as pass"]}, {"id": "schema-migration-rollback-release-readiness", "prompt": "Issue #710: we need to split the 'users' table's 'address' column into separate street, city, state, and zip columns. The table has 4.2 million rows in production. This is a high-risk schema migration that needs a rollback plan and release-readiness evidence before we can merge.", "expected_output": "Scenario: schema/migration change requiring rollback and release-readiness evidence. The trajectory exercises the high-risk journey path (all nine phases mandatory, all five gates, no conditional skips). Phase 3 produces an architecture delta with expand/contract (parallel change) migration strategy and a rollback plan that does not rely on git revert. data-engineering and data-architect are loaded per routing signals (migration files present, schema change). Gate 1 requires approval of the migration strategy and rollback plan. Phase 9 records release-readiness evidence (version bump, changelog, migration documentation, rollback rehearsal) before the release gate is evaluated. The terminal state is 'merged' with release readiness confirmed but release activity gated on separate explicit authorization. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["The high-risk path is recorded in packet group (b) with all nine phases mandatory and no conditional skips permitted", "Packet group (f) contains a migration strategy using expand/contract (parallel change) and a rollback plan that rejects git revert for schema or data changes", "data-engineering and data-architect are recorded as selected specialists in packet group (e) with applicability signals citing migration files or schema change", "Gate 1 verdict in packet group (h) records approval of the architecture delta including the migration strategy and rollback plan", "Release-readiness evidence (version bump, changelog entry, migration documentation) is recorded in packet group (i) before the release gate", "The release gate in packet group (h) or (i) distinguishes pre-merge release readiness from post-merge release activity requiring separate explicit authorization", "Terminal lifecycle state in packet group (i) is 'merged' with the merge commit SHA recorded; release activity is not conflated with merge"]}, {"id": "refactor-characterization-architecture-review", "prompt": "Issue #823: the order-processing module has grown into a 2,400-line god class. Refactor it into smaller, focused modules without changing any observable behavior. We need proof the behavior is preserved.", "expected_output": "Scenario: refactor needing characterization tests and architecture review. The trajectory exercises the refactor journey path (phases 1, 2, 3, 6, 7, 8, 9 mandatory; phases 4 and 5 conditional). Phase 2 gathers current-state evidence of the god class behavior. Phase 3 produces an architecture delta with the decomposition plan; software-architecture-analysis is loaded for the review. Characterization tests are written or identified before the refactor begins to capture existing behavior. Gate 5 uses behavior-preservation evidence (characterization tests passing before and after) as its criterion. The skip of phase 4 is recorded with the reason that the refactor is fully characterized by the architecture delta and characterization-test plan. The terminal state is 'merged' with behavior-preservation evidence recorded. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["The refactor path is recorded in packet group (b) with phases 4 and 5 conditional", "Packet group (e) records the skip of phase 4 (specification and decomposition) with a reason citing that no separate SPEC.md is needed because the refactor is behavior-preserving", "software-architecture-analysis is recorded as a selected specialist in packet group (e) for the architecture review", "Characterization tests covering the god class observable behavior exist in the produced artifacts before the refactor commits begin", "Gate 5 verdict in packet group (h) records behavior-preservation evidence (characterization tests passing before and after) bound to the exact head SHA", "Terminal lifecycle state in packet group (i) is 'merged' with the merge commit SHA recorded"]}, {"id": "docs-only-reduced-path-skip-reasons", "prompt": "Issue #901: the API reference page in our docs site has outdated parameter names for the /v2/orders endpoint. Update the documentation to match the current implementation. No code changes are needed.", "expected_output": "Scenario: docs-only change using a reduced (lightweight) path with comprehensive skip recording. The trajectory exercises the lightweight journey path (phases 1, 6, 7, 8, 9 mandatory; phases 2, 3, 4, 5 conditional and skipped). Each skipped phase is recorded in packet group (e) with a concrete reason citing the path's skip criterion. At least two specialist skills that do not apply (backend-engineering, frontend-engineering, data-engineering) are explicitly listed as skipped with reasons citing their routing-table skip rules (e.g., 'no server-side application code modified', 'no client-side application code modified', 'no schema or migration files modified'). No specialist skill in the routing matrix is absent from both the selected and skipped lists. The terminal state is 'merged' with doc-update evidence. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["The lightweight path is recorded in packet group (b) and phases 2, 3, 4, and 5 are each recorded as skipped in packet group (e) with a concrete reason citing the path's skip criterion", "Packet group (e) routing section lists at least three specialist skills (including backend-engineering, frontend-engineering, and data-engineering) as skipped, each with a skill name and a concrete reason", "Every skipped skill entry in packet group (e) has both a skill name and a non-empty reason; no specialist in the routing matrix is absent from both the selected and skipped lists", "technical-documentation is recorded as a selected specialist in packet group (e) with an applicability signal citing documentation changes", "No code-editing commits appear in the produced artifacts; only documentation files are changed", "Terminal lifecycle state in packet group (i) is 'merged' with the merge commit SHA recorded"]}, {"id": "existing-pr-duplicate-detection", "prompt": "Issue #455 was filed asking for rate limiting on the public API. I want to start working on it and submit a PR.", "expected_output": "Scenario: issue with an existing PR or duplicate work. The trajectory exercises the lifecycle pre-work checks in phase 1 (intake and provenance). The existing-PR search discovers that PR #450 from another contributor already implements rate limiting for the same endpoint. The duplicate-issue check and existing-PR check are recorded as distinct pre-work checks in packet group (d) or the lifecycle fields. The handling decision is to coordinate or defer to the existing PR owner rather than opening a duplicate PR. The terminal lifecycle state is 'closed' with the close reason recording the duplicate-work determination and the existing PR reference. No new PR is opened. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["Packet group (d) or lifecycle fields record a duplicate-issue check result and a separate existing-PR search result as distinct pre-work checks", "The existing-PR search result identifies the pre-existing PR (by number or reference) addressing the same rate-limiting work", "The handling decision recorded in the packet is to coordinate or defer, not to open a duplicate PR", "No new review submission (PR or equivalent) is created in the produced artifacts", "Terminal lifecycle state in packet group (i) is 'closed' with a close reason citing the duplicate-work determination and referencing the existing PR"]}, {"id": "review-round-material-change-reverification", "prompt": "Issue #567: add input validation to the user-registration form. During code review, a reviewer points out that the validation logic has a bypass when the input contains null bytes. You push a fix addressing the null-byte bypass. This changes the validation logic materially.", "expected_output": "Scenario: review round that changes the final head and requires re-verification, proving a material post-review change invalidates the prior verification verdict. The trajectory exercises phases 1 through 8. Phase 7 (independent review and boundary verification) produces gate 4 and gate 5 verdicts bound to head SHA A (the initial implementation). A material review-driven change (the null-byte bypass fix altering validation logic) produces head SHA B. The prior verdicts bound to SHA A are explicitly invalidated. The trajectory re-enters phase 7: gate 4 (independent review) and gate 5 (boundary verification) re-run at SHA B. The final verified head SHA in packet group (i) is SHA B, not SHA A. The readiness gate (phase 8) is satisfied only after re-verification at the new head. The terminal state is 'merged' with the final verdict bound to SHA B. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["Gate 4 and gate 5 verdicts are initially recorded in packet group (h) bound to head SHA A (the pre-review implementation head)", "The material review-driven change (null-byte bypass fix altering validation logic) is recorded as producing a new head SHA B distinct from SHA A", "The prior verdicts bound to SHA A are explicitly marked as invalidated in packet group (h) after the material change", "Gate 4 and gate 5 verdicts are re-recorded in packet group (h) bound to head SHA B after re-verification", "The final verified head SHA in packet group (i) equals SHA B, not the stale SHA A", "Terminal lifecycle state in packet group (i) is 'merged' with the merge commit SHA recorded and the final verdict bound to SHA B"]}, {"id": "release-authority-blocked-terminal-state", "prompt": "Issue #688: implement and ship the new caching layer for the product-search service. You have authority to create a PR and get it merged, but the release process requires the VP of Engineering's sign-off which has not been granted. Complete the work and get it deployed.", "expected_output": "Scenario: maintainer/release-authority-blocked terminal state. The trajectory exercises phases 1 through 8 successfully (all gates pass, CI green, review approved, exact-final-head verified). Phase 9 (authorized post-merge release and closeout) cannot proceed because release authority has not been granted. The release gate requires explicit authorization distinct from merge authority; merge alone does not imply release authorization. The trajectory records the blocked release gate with the escalation outcome (release authority not granted) and transitions the packet to the 'blocked' terminal state. The run stops at the release gate without deploying; persistence does not upgrade authority. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["All five gate verdicts (gate 1 through gate 5) are recorded as pass in packet group (h) bound to the exact final head SHA", "Phase 8 readiness is confirmed with CI passing and review approved at the exact final head SHA recorded in packet group (i)", "The release gate in packet group (h) or (i) records a verdict of 'blocked' citing that release authority was not granted", "The packet distinguishes merge authority (granted) from release authority (not granted) in the authority classification fields of packet group (b)", "Terminal lifecycle state in packet group (i) is 'blocked' with the failing gate ID (release gate) and the blocking evidence (release authority not granted) recorded", "No release activity (tagging, publishing, deploying) appears in the produced artifacts; the run stops at the release gate without deploying"]}, {"id": "lightweight-test-hardening-route", "prompt": "In a public OSS repository, issue #123 says the webhook URL validator already rejects overlong hostnames permanently, but a mutation pilot found that this classification could regress to retryable. Add a focused test-only regression guard, open a PR, and carry it through review.", "expected_output": "Scenario: a low-risk test-hardening change. The trajectory selects the lightweight path rather than the full path because production behavior is already correct and only a test file changes. Phase 1 records the protected public invariant, the named mutation, and production-code non-goals. Conditional planning phases are skipped with reasons. The implementation adds a hermetic public-contract test. Gate 5 uses clean-baseline pass plus targeted-mutant failure, not a requirement that the new test fail on clean main. The candidate is locally final before one push to the expensive-CI repository. Gate 4 is one bounded final review after the candidate is frozen, and readiness is bound to the exact final head SHA. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["The lightweight path is selected because production behavior is already correct, production code is unchanged, and the change is confined to a test or fixture file", "The change contract records the public invariant, the named retryable-classification mutation, and an explicit production-code non-goal", "Conditional phases 2, 3, 4, and 5 are each recorded as skipped with reasons that identify test hardening and the applicable lightweight criterion", "The produced test uses hermetic setup and asserts the semantic permanent-versus-retryable contract rather than an incidental current exception subtype", "Gate 5 records clean-baseline pass and targeted-mutant failure, without requiring the test to fail on clean main", "The candidate's focused tests, lint, compilation, and scope checks are complete before the first push, and the remote verification is tied to one stable head", "Gate 4 records one bounded final review after candidate freeze, with no repeated reviewer round launched merely because a prior review timed out", "The final readiness evidence binds CI, review, and boundary verification to the exact final head SHA"]}, {"id": "test-hardening-baseline-mutant-gate", "prompt": "A mutation pilot found that changing an already-correct validation error from permanent to retryable would not be caught by the current tests. Add the smallest regression guard without changing production code.", "expected_output": "Scenario: test-hardening rather than a production bugfix. The agent states the semantic invariant, confirms the clean implementation already passes, and identifies the controlled weakening. It adds a deterministic hermetic public-behavior test, records clean-baseline pass and mutant-fail evidence, and does not apply the ordinary fails-on-main requirement. It avoids a broad mutation campaign and escalates only if production code, dependencies, configuration, or another higher-risk surface becomes necessary. Claims are scoped to the harness, model, fixtures, and revision under test.", "assertions": ["The output explicitly classifies the task as a regression guard for already-correct production behavior, not as an active production defect", "The proposed gate is clean-baseline pass plus named-mutant or controlled-weakening failure", "The test is hermetic and exercises the public contract without live external resolution or another incidental prerequisite", "The assertion checks the semantic permanent-versus-retryable invariant rather than exact current implementation identity", "Production code remains unchanged and the verification stays bounded to focused tests, static checks, scope checks, and the targeted mutation", "The output states escalation triggers for production-code, dependency, configuration, security, or deployment changes"]}, {"id": "midflight-position-assessment", "prompt": "Pick up where the previous contractor left off on the payment-retry feature. Branch feat/payment-retry exists with about six commits, there is a SPEC.md sitting on that branch, and the handoff note says 'design approved, most of the tests written'. No delivery packet or evidence ledger was handed over. Figure out where the work actually stands and continue it to a reviewed change.", "expected_output": "Scenario: mid-flight engagement with no delivery packet. The trajectory recognizes the entry-mode condition (continuation of work started elsewhere, no packet) and runs position assessment before any implementation continuation. It inventories observable artifacts: branch commits via git log against the base ref, SPEC.md presence and completeness on the branch, whether TASK-PLAN.md and VERIFICATION-PLAN.md exist, CI state, and any review artifacts. Each artifact is scored against the corresponding phase exit conditions. The handoff narrative ('design approved, most tests written') is treated as unverified assumption, not as satisfied gates, unless artifacts support it. The position report names the current phase, satisfied gates with evidence pointers, partial/absent/contradicted items, and a remaining checklist before execution resumes. Verified-complete work is not re-executed. A delivery packet is bootstrapped: group (a) records mid-flight engagement and inventory pointers, group (b) re-selects the delivery path from the affected surface, group (c) initializes from the position report with the assessment-time head SHA. Gates without evidence remain open rather than being retroactively marked passed.", "assertions": ["The trajectory runs position assessment before any implementation continuation, rather than restarting at phase-1 intake or accepting the handoff narrative as ground truth", "Each satisfied gate in the position report is bound to an observable artifact pointer (path, commit SHA, or CI/tracker reference), not to the contractor's claims", "The narrative claims ('design approved, most of the tests written') are recorded as unverified assumptions rather than satisfied gates, absent supporting artifacts", "The position report names the current phase, satisfied gates with evidence pointers, and an explicit remaining checklist before execution continues", "Verified-complete work identified during assessment is not re-executed; the run continues from the assessed position", "A delivery packet is bootstrapped recording mid-flight engagement, a re-selected delivery path, and the assessment-time head SHA, without retroactively marking unevidenced gates as passed"]}, {"id": "tracker-discovery-linear-routing", "prompt": "We track this project's work in Linear; the code repo mirrors to GitHub for hosting and review. Take ticket ENG-482 ('Add idempotency keys to the webhook receiver') through your delivery lifecycle to a reviewed change.", "expected_output": "Scenario: change-request journey on a product whose tracking system is Linear. During phase-1 intake the trajectory identifies Linear as the tracking system. The bare ENG-482 key is treated as an ambiguous signal (team-key identifiers match both Jira and Linear per tracker-discovery detection signals); the disambiguating evidence is the requester's explicit confirmation ('we track work in Linear'). Intake records Linear as requester-confirmed with that basis in delivery packet group (a), then routes tracker operations (reading the ticket, posting transitions and comments) to the linear tooling skill rather than improvising GraphQL calls or defaulting to GitHub Issues. GitHub's role is correctly scoped to code hosting, branch, and review mechanics per the platform reference mode. The trajectory's own vocabulary stays tracker-neutral (work item, state transition, review submission) with platform nouns confined to platform mapping. The first mutation against Linear passes the state-change confirmation gate (target, scope, rollback path) before executing. Sequencing, gates, and evidence ownership remain with the journey spine; the linear skill is used as a mechanical layer, never as a second orchestrator.", "assertions": ["Intake records Linear as the identified tracking system with its evidence basis recorded in delivery packet group (a) before any tracker operation, treating the bare ENG-482 key as ambiguous and the requester's explicit confirmation as the disambiguating evidence", "Tracker operations route to the linear tooling skill rather than improvised API calls or defaulting to GitHub Issues", "GitHub's role is correctly scoped to code hosting and review mechanics; tracker operations do not migrate to GitHub Issues by assumption", "The trajectory's tracker vocabulary stays tracker-neutral (work item, state transition), with platform-specific nouns confined to platform mapping", "The first mutation against Linear passes the state-change confirmation gate (target, scope, rollback path) before executing", "Sequencing and gate authority remain with the journey spine; the linear skill operates as a mechanical layer, not an orchestrator"]}]} diff --git a/neckbeard/references/delivery-packet.md b/neckbeard/references/delivery-packet.md index ee62f83..6b07d5f 100644 --- a/neckbeard/references/delivery-packet.md +++ b/neckbeard/references/delivery-packet.md @@ -29,7 +29,7 @@ into another or left to implication. | Group | Name | Fields | |---|---|---| -| **(a)** | **Change-request provenance** | Change-request URL or number; source type (issue / ticket / email / verbal); repository; base ref; issue/comment snapshot (request text, comments, and linked work captured at intake); head SHA at intake. | +| **(a)** | **Change-request provenance** | Change-request URL or number; source type (issue / ticket / email / verbal); repository; base ref; issue/comment snapshot (request text, comments, and linked work captured at intake); tracking system identified at intake per [tracker-discovery.md](tracker-discovery.md), with its evidence basis (repository-detected or requester-confirmed); head SHA at intake. | | **(b)** | **Granted authority + workflow mode/path** | Authority class granted (Explore / Modify / Publish / Deploy / Merge — see [risk-authority-gates.md](risk-authority-gates.md)); selected workflow mode (GitHub reference mode or enterprise mode); selected path (lightweight / full / refactor / high-risk). | | **(c)** | **Resumable phase/gate state + current head SHA** | Current phase name; current gate name; the head SHA at which the last successful gate verdict was recorded; the current lifecycle state (see below). | | **(d)** | **Problem / baseline evidence / scope / non-goals / affected surfaces** | User-visible problem; baseline (pre-change) evidence with boundary labels; in-scope work; explicit non-goals; affected surfaces (paths, contracts, boundaries). | diff --git a/neckbeard/references/position-assessment.md b/neckbeard/references/position-assessment.md index d0787b2..611bca8 100644 --- a/neckbeard/references/position-assessment.md +++ b/neckbeard/references/position-assessment.md @@ -72,6 +72,11 @@ Inventory reads are read-only discovery; they need no confirmation. Record where each artifact was found (path, SHA, URL) — pointers become the evidence references in the report. +**Re-select the delivery path here, before scoring.** Run the path-selection +rule ([../SKILL.md](../SKILL.md) § Path selection) against the affected surface +and risk observed in the inventory; do not inherit a path assumption from the +prior work. The selected path determines which phases are mandatory to score. + ### Step 3 — Score against phase exit conditions Compare each inventoried artifact with the exit condition and gate definition @@ -87,9 +92,9 @@ of its phase ([journey.md](journey.md); gate semantics in Scoring constraints: -- Score the selected delivery path's mandatory phases; conditional phases are - scored only if their artifacts exist (an absent conditional phase with a - legitimate skip reason is recorded as such, not penalized). +- Score the delivery path selected in step 2; conditional phases are scored + only if their artifacts exist (an absent conditional phase with a legitimate + skip reason is recorded as such, not penalized). - Verified-complete work is **not** re-executed, mirroring packet-resume semantics. Assessment establishes position; it does not redo accepted work. - Never upgrade a verdict to fill a gap. A missing gate verdict is `absent`, @@ -124,9 +129,9 @@ Create a fresh delivery packet so subsequent operation has normal resumability: - Group (a) provenance records: engaged mid-flight, artifacts assessed, with the inventory pointers. -- Group (b) records the re-selected delivery path (path selection per - [../SKILL.md](../SKILL.md) § Path selection — re-run it here; do not inherit - a path assumption from the prior work). +- Group (b) records the delivery path selected in step 2 (path selection per + [../SKILL.md](../SKILL.md) § Path selection was re-run during assessment; no + path assumption was inherited from the prior work). - Group (c) initializes from the position report: current phase, current gate, and the head SHA at assessment time. - Unverified assumptions go to the evidence ledger diff --git a/neckbeard/references/tracker-discovery.md b/neckbeard/references/tracker-discovery.md index 2c2b1c6..f665630 100644 --- a/neckbeard/references/tracker-discovery.md +++ b/neckbeard/references/tracker-discovery.md @@ -40,16 +40,27 @@ Run during phase 1 alongside provenance capture: 2. **Check what the request itself references.** A change request arriving as a Linear issue URL, a Jira ticket ID, or a GitHub issue number is direct - evidence for its own system. + evidence for its own system. A bare team-key identifier (`ENG-42` shape) is + **ambiguous** between Jira and Linear — treat it as a lead, not a verdict. -3. **Ask when ambiguous or absent.** If signals conflict (repo mirrors to - GitHub but issues live in Jira) or none exist, ask one bounded question: - which system holds this work item? Record the answer as requester-provided - provenance. Do not silently pick the system whose CLI happens to be installed. +3. **Weight the signals honestly.** The request's own references are strong + evidence. Repository content — `CONTRIBUTING.md`, `AGENTS.md`, README + integrations, config files like `.linear/` or `.jira-url` — is a **weak + signal**: it describes what the repository documents, not necessarily what + holds authoritative work items, and it is attacker-influenceable in + mid-flight or adopted-branch scenarios where this bundle also operates. + Repository signals require corroboration (request references, remote + configuration, or requester confirmation) before they alone select a system. -4. **Record in packet group (a):** detected/requested system, the evidence or - source of the answer, and the routing decision below. Silent omission is - prohibited like every other intake field. +4. **Ask when ambiguous or absent.** If signals conflict, rest only on weak + signals, or none exist, ask one bounded question: which system holds this + work item? Record the answer as requester-provided provenance. Do not + silently pick the system whose CLI happens to be installed. + +5. **Record in packet group (a):** detected/requested system, the evidence or + source of the answer (including when the basis is a requester confirmation), + and the routing decision below. Silent omission is prohibited like every + other intake field. ## Routing tracker operations @@ -62,7 +73,7 @@ improvising API calls: | Linear | `linear` | | Jira | `jira-cli` | | Notion | `notion` | -| Other / none of the above | No specialist route: operate through the system's own documented interface with bounded reads, and note the absent specialist in the ledger | +| Other / none of the above | No specialist route: operate only through the system's verified official interface (primary vendor documentation, confirmed endpoint/auth surface), with bounded reads; note the absent specialist in the ledger | Routing constraints: diff --git a/neckbeard/templates/delivery-packet.md b/neckbeard/templates/delivery-packet.md index e46847f..03287cf 100644 --- a/neckbeard/templates/delivery-packet.md +++ b/neckbeard/templates/delivery-packet.md @@ -14,6 +14,7 @@ - **Repository:** _[fill: repo path or URL, e.g. `https://github.com/org/repo`]_ - **Base ref:** _[fill: branch or ref the change is based on, e.g. `main`]_ - **Issue/comment snapshot:** _[fill: request text, comments, and linked work captured at intake — paste inline or link to a snapshot file]_ +- **Tracking system:** _[fill: system identified per ../references/tracker-discovery.md, e.g. `Linear (requester-confirmed)` / `GitHub Issues (detected: issue URL in request)`]_ - **Head SHA at intake:** _[fill: `git rev-parse HEAD` at intake, e.g. `a1b2c3d`]_ ## (b) Granted authority + workflow mode/path