mirror of
https://github.com/magnus919/agent-skills.git
synced 2026-09-11 19:47:12 +03:00
feat: add neckbeard, an evidence-driven SDLC skill bundle (#81)
* feat: add neckbeard, an evidence-driven SDLC skill bundle A portable operating model for software delivery that routes a change through framing, discovery, design, implementation, review, verification, delivery, and learning. Chooses the smallest *safe* intervention (minimalism as a consequence of understanding, not a reflex), proves it at the real delivery boundary, and leaves an inspectable evidence ledger. Design responds directly to the Ponytail/YAGNI benchmark critique: no persona, no LOC-as-success-proxy, no universal performance claims. Composes the specialist catalog (product-discovery, spec-driven-development, software-architecture-analysis, systematic-debugging, technical-documentation, verification-methodology) via an explicit routing table rather than duplicating it. Ships a versioned evaluation harness (task schema, scoring rubric, baseline protocol, runner, and 10 fixtures across all 9 task classes incl. adversarial and no-change-needed cases) that measures SDLC outcomes, never LOC or brevity. Closes #25 * chore: regenerate Claude marketplace for neckbeard
This commit is contained in:
Magnus Hedemark
@@ -464,6 +464,15 @@
|
||||
"strict": false,
|
||||
"description": "Machine learning engineering methodology — model training, fine-tuning (LoRA/QLoRA), evaluation, quantization, deployment, and MLOps pipeline design. Grounded in practical engineering patterns for production ML systems."
|
||||
},
|
||||
{
|
||||
"name": "neckbeard",
|
||||
"source": "./",
|
||||
"skills": [
|
||||
"./neckbeard"
|
||||
],
|
||||
"strict": false,
|
||||
"description": "Evidence-driven software delivery operating model. Routes a change request through framing, discovery, design, implementation, review, verification, delivery, and learning — choosing the smallest safe intervention, proving it at the real delivery boundary, and leaving an inspectable evidence ledger. Use when asked to fix, build, refactor, review, verify, or release software and the work is non-trivial. Composes specialist catalog skills rather than replacing them. Not a persona, not a \"10x developer\" prompt, and not a LOC-minimizer."
|
||||
},
|
||||
{
|
||||
"name": "nous-branding",
|
||||
"source": "./",
|
||||
|
||||
@@ -157,6 +157,7 @@ When the user mentions these keywords, load the corresponding skill:
|
||||
| "strategy", "strategic planning", "OKRs", "strategic narrative", "Five Forces", "Blue Ocean", "competitive positioning", "moat", "Ansoff", "Three Horizons", "market entry", "capital allocation", "M&A evaluation", "BCG Matrix", "portfolio management" | [strategy-frameworks](strategy-frameworks/SKILL.md) |
|
||||
| "Supabase", "Supabase CLI", "supabase start", "supabase migration", "Supabase Auth", "Supabase RLS", "Supabase Storage", "Supabase Realtime", "Edge Functions", "self-host Supabase", "self-hosted Supabase", "Supabase Docker", "Supabase backup", "Supabase restore", "Supabase upgrade" | [supabase](supabase/SKILL.md) |
|
||||
| "Vercel Eve", "eve agent", "eve dev", "eve build", "eve start", "Eve self-hosting", "durable Eve session", "Eve sandbox", "Eve Workflow", "Eve subagent", "Eve agent skills", "Eve deployment", "Eve reverse proxy" | [vercel-eve](vercel-eve/SKILL.md) |
|
||||
| "smallest safe intervention", "evidence ledger", "change contract", "prove it worked", "verify at the boundary", "delivery discipline", "SDLC workflow", "did this actually work", "audit trail", "stop and escalate", "neckbeard" | [neckbeard](bundles/neckbeard/SKILL.md) |
|
||||
| "verification-methodology", "verification methodology" | [verification-methodology](verification-methodology/SKILL.md) |
|
||||
| "accessibility", "a11y", "WCAG", "ARIA", "accessible name", "screen reader", "keyboard navigation", "focus management", "focus trap", "dialog", "modal", "combobox", "listbox", "accessible form", "form validation", "error recovery", "accessible authentication", "color contrast", "reduced motion", "target size", "accessibility testing" | [web-accessibility](web-accessibility/SKILL.md) |
|
||||
| "seo-audit", "seo audit" | [seo-audit](seo-audit/SKILL.md) |
|
||||
|
||||
@@ -214,6 +214,10 @@ Operate MeshCore Companion radio packet capture over BLE, serial, or TCP. Covers
|
||||
|
||||
Machine learning engineering methodology — model training, fine-tuning (LoRA/QLoRA), evaluation, quantization, deployment, and MLOps pipeline design. Grounded in practical engineering patterns for production ML systems.
|
||||
|
||||
### [neckbeard](bundles/neckbeard/SKILL.md)
|
||||
|
||||
Evidence-driven software delivery operating model — routes a change through framing, discovery, design, implementation, review, verification, delivery, and learning, choosing the smallest *safe* intervention and proving it at the real delivery boundary. Every non-trivial run leaves an inspectable evidence ledger. Composes the specialist catalog rather than replacing it. Ships a versioned evaluation harness that measures SDLC outcomes, never LOC. Not a "10x developer" persona.
|
||||
|
||||
### [nous-branding](nous-branding/SKILL.md)
|
||||
|
||||
Generate images and content consistent with the Nous Research brand identity.
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
# neckbeard
|
||||
|
||||
An evidence-driven operating model for software delivery — not a "10x developer"
|
||||
persona, but a disciplined workflow that helps an AI agent choose the smallest
|
||||
*safe* intervention, prove it worked at the real delivery boundary, and leave an
|
||||
audit trail a human can inspect.
|
||||
|
||||
## Why Install This Skill
|
||||
|
||||
Most "senior developer" prompts fail the same way: they make an agent emit short,
|
||||
confident code and call it effectiveness. Scott Logic's critique of the Ponytail
|
||||
benchmark showed that a static behavioral prompt plus a narrow, gameable metric
|
||||
(lines of code) cannot substantiate any real claim about software engineering.
|
||||
Swapping the persona for the three words "Follow YAGNI principles" nearly matched
|
||||
its score.
|
||||
|
||||
neckbeard is the answer to that failure mode. Instead of performing a character,
|
||||
it gives an agent a bounded, stage-aware spine: frame the change, discover the
|
||||
real system before designing, pick the smallest intervention that is still safe,
|
||||
execute by SDLC stage, record evidence, and stop or escalate when the evidence
|
||||
runs out. Minimalism is treated as a *consequence of understanding*, not a reflex
|
||||
— so the bundle does not win by reflexively deleting or compressing.
|
||||
|
||||
Install it when you want delivery discipline that is inspectable. Every
|
||||
non-trivial run produces an evidence ledger: intent, inspected artifacts,
|
||||
assumptions, rejected alternatives, files changed, checks run, observed outputs,
|
||||
and the boundaries that were *not* verified. "Done" is prohibited unless the
|
||||
declared verification target was actually exercised.
|
||||
|
||||
## What You Get
|
||||
|
||||
| Path | What it provides |
|
||||
|---|---|
|
||||
| `SKILL.md` | Thin umbrella that routes a request through the delivery spine and composes specialist catalog skills |
|
||||
| `references/stages.md` | Entry conditions, required evidence, exit conditions, and escalation rules for each SDLC stage |
|
||||
| `references/evidence-ledger.md` | The ledger schema and the rules for distinguishing a component check from a delivery-boundary check |
|
||||
| `references/risk-authority-gates.md` | Stop and escalation rules; the gate before any mutation, deploy, merge, or destructive act |
|
||||
| `references/routing-table.md` | "Use existing skill instead" table so the bundle composes the catalog rather than swallowing it |
|
||||
| `references/evaluation.md` | Evaluation methodology: fixtures, baselines, rubrics, multi-run reporting, claims policy |
|
||||
| `templates/` | Change contract, decision record, evidence ledger, verification plan, evaluation report |
|
||||
| `eval/` | Versioned evaluation harness: task schema, scoring rubric, baseline protocol, fixtures, and a runner |
|
||||
|
||||
## Quick Start
|
||||
|
||||
Load the umbrella when a non-trivial change lands:
|
||||
|
||||
```
|
||||
skill_view(name="neckbeard")
|
||||
```
|
||||
|
||||
Then follow the core loop in `SKILL.md`. For a bug, the agent frames a change
|
||||
contract, loads `systematic-debugging` for root cause, makes the smallest safe
|
||||
fix, verifies at the real boundary, and writes an evidence ledger. For a feature,
|
||||
it routes discovery to `product-discovery` and specification to
|
||||
`spec-driven-development` before writing code.
|
||||
|
||||
To run the evaluation suite against your harness:
|
||||
|
||||
```
|
||||
python3 eval/run_eval.py --suite eval/fixtures --report out/report.md
|
||||
```
|
||||
|
||||
## Triggers
|
||||
|
||||
- Asked to fix, build, refactor, review, verify, or release software where the work is non-trivial.
|
||||
- A change needs a defensible "done" backed by evidence, not a claim.
|
||||
- You want an inspectable record of what was inspected, assumed, changed, and left unverified.
|
||||
- You need to evaluate whether a delivery skill actually improves outcomes.
|
||||
|
||||
Do not trigger for single factual questions, fully-specified one-line edits, or
|
||||
tasks already owned end-to-end by a more specific skill.
|
||||
|
||||
## Requirements
|
||||
|
||||
- An agent harness with file read/write, terminal access, and skill loading.
|
||||
- No network or runtime dependency in the bundle itself.
|
||||
- The evaluation runner needs Python 3.9+ (standard library only).
|
||||
- Recommended companion catalog skills (loaded on demand, not required):
|
||||
`product-discovery`, `spec-driven-development`, `software-architecture-analysis`,
|
||||
`systematic-debugging`, `technical-documentation`, `verification-methodology`.
|
||||
@@ -0,0 +1,151 @@
|
||||
---
|
||||
name: neckbeard
|
||||
description: >-
|
||||
Evidence-driven software delivery operating model. Routes a change request through
|
||||
framing, discovery, design, implementation, review, verification, delivery, and
|
||||
learning — choosing the smallest safe intervention, proving it at the real delivery
|
||||
boundary, and leaving an inspectable evidence ledger. Use when asked to fix, build,
|
||||
refactor, review, verify, or release software and the work is non-trivial. Composes
|
||||
specialist catalog skills rather than replacing them. Not a persona, not a "10x
|
||||
developer" prompt, and not a LOC-minimizer.
|
||||
license: MIT
|
||||
compatibility: Agent harness with file read/write, terminal, and skill loading. No network or runtime dependency required by the bundle itself.
|
||||
metadata:
|
||||
spec-version: "1.0"
|
||||
tags: sdlc, delivery, evidence, verification, workflow
|
||||
---
|
||||
|
||||
# neckbeard
|
||||
|
||||
A disciplined operating model for software delivery. It does not perform a
|
||||
character. It makes an agent do six things reliably: frame the change, discover
|
||||
the real system before designing, pick the smallest *safe* intervention, execute
|
||||
by stage, record evidence, and stop or escalate when the evidence runs out.
|
||||
|
||||
The name is a joke about the "10x developer in a Markdown file" trope. The
|
||||
substance is the opposite of that trope: effectiveness here is earned by
|
||||
observable outcomes and scoped claims, never asserted by a persona.
|
||||
|
||||
## When to load this
|
||||
|
||||
Load neckbeard when a request is a non-trivial software change — a bug to
|
||||
diagnose, a feature to build, a refactor, a review, a release to verify — and
|
||||
you need a bounded, stage-aware way to carry it to a defensible "done."
|
||||
|
||||
Do **not** load it for:
|
||||
- A single factual question or lookup (answer directly).
|
||||
- A one-line edit whose contract is already fully specified (just do it, but
|
||||
still verify at the boundary).
|
||||
- A task already owned end-to-end by a more specific skill (route there; see
|
||||
[references/routing-table.md](references/routing-table.md)).
|
||||
|
||||
## Core loop
|
||||
|
||||
Every run moves through the same spine. Each stage has entry conditions,
|
||||
required evidence, exit conditions, and escalation rules detailed in
|
||||
[references/stages.md](references/stages.md).
|
||||
|
||||
1. **Frame the change contract.** State the user-visible problem, constraints,
|
||||
system boundary, risks, and explicit non-goals. Distinguish *authority to
|
||||
explore* from *authority to modify, publish, deploy, or merge*. Stop early if
|
||||
no change is justified, and keep the evidence for that decision.
|
||||
Template: [templates/change-contract.md](templates/change-contract.md).
|
||||
|
||||
2. **Discover before designing.** Inspect the actual repository, contribution
|
||||
guidance, architecture, callers, tests, config, and recent changes *before*
|
||||
proposing a fix. Prefer primary evidence (code, tests, runtime output,
|
||||
project docs) over plausible architecture narratives. Make unverified
|
||||
assumptions and missing access explicit.
|
||||
|
||||
3. **Select the smallest safe intervention.** Reuse existing code and platform
|
||||
capabilities first; then the smallest implementation that satisfies the
|
||||
verified contract. Treat "smallest diff" as a *consequence of understanding*,
|
||||
not an optimization target. Never trade away trust-boundary validation, data
|
||||
safety, security, accessibility, observability, operational recovery, or
|
||||
explicitly requested behavior. Record any deliberate ceiling and its upgrade
|
||||
trigger.
|
||||
|
||||
4. **Execute by SDLC stage.** Route the work to the stage that owns it —
|
||||
discovery/requirements, design, implementation, verification, delivery,
|
||||
learning. Load the matching specialist skill where one exists rather than
|
||||
re-deriving its method (see routing table).
|
||||
|
||||
5. **Keep an evidence ledger.** Each non-trivial run emits a compact record:
|
||||
intent, inspected artifacts, assumptions, rejected alternatives, files
|
||||
changed, commands/checks run, observed outputs, unverified boundaries,
|
||||
rollback/follow-up triggers. Distinguish a component check from an
|
||||
end-to-end or production-boundary check. Format and rules:
|
||||
[references/evidence-ledger.md](references/evidence-ledger.md).
|
||||
Template: [templates/evidence-ledger.md](templates/evidence-ledger.md).
|
||||
|
||||
6. **Stop and escalate by rule.** Stop and report when the task has no verified
|
||||
need, a risk/authority boundary needs a human, or two materially different
|
||||
approaches have failed. Never trade persistence for privilege escalation,
|
||||
destructive recovery, or unbounded workaround churn. Rules:
|
||||
[references/risk-authority-gates.md](references/risk-authority-gates.md).
|
||||
|
||||
## The one rule that defines "done"
|
||||
|
||||
> "Done" is prohibited unless the declared verification target has actually been
|
||||
> exercised. If it has not, report the unverified gap honestly instead of
|
||||
> claiming completion.
|
||||
|
||||
A passing unit test is not the same as exercising the delivery boundary. A local
|
||||
render is not production. State which boundary was checked and which was not.
|
||||
Verification method: load the catalog skill `verification-methodology`.
|
||||
|
||||
## Minimalism, correctly
|
||||
|
||||
Minimalism in this bundle is a *conditional* design choice made **after**
|
||||
real-flow understanding — not an unconditional "fewest lines wins" reflex. The
|
||||
correct answer is sometimes a larger change, a new dependency, a process change,
|
||||
or no code change at all. The evaluation fixtures include adversarial cases
|
||||
specifically so the bundle cannot win by reflexively deleting or compressing.
|
||||
See [references/stages.md](references/stages.md) §3.
|
||||
|
||||
## Routing: compose, don't swallow
|
||||
|
||||
neckbeard owns the *cross-stage contracts* — the change contract, evidence
|
||||
ledger, stop rules, and evaluation protocol. It does **not** own domain method.
|
||||
When a stage has a specialist skill, load it and follow it. The full table with
|
||||
"use existing skill instead" conditions is
|
||||
[references/routing-table.md](references/routing-table.md). Summary:
|
||||
|
||||
| Stage / need | Load this catalog skill instead of re-deriving |
|
||||
|---|---|
|
||||
| Stakeholder discovery, requirements, ACs | `product-discovery` |
|
||||
| Formal specification, phase gates | `spec-driven-development` |
|
||||
| Reverse-engineering an existing codebase | `software-architecture-analysis` |
|
||||
| Root-cause debugging | `systematic-debugging` |
|
||||
| Docs / README / API reference | `technical-documentation` |
|
||||
| Verification verdicts and evidence | `verification-methodology` |
|
||||
|
||||
If a specialist skill is not installed, neckbeard's stage references provide a
|
||||
minimal fallback method — but note in the ledger that the specialist was absent.
|
||||
|
||||
## Evaluation is a first-class deliverable
|
||||
|
||||
This bundle ships a versioned evaluation harness in [eval/](eval/). It measures
|
||||
SDLC *outcomes* the bundle claims to improve — correctness, regression safety,
|
||||
scope discipline, boundary verification, honest uncertainty — never LOC or
|
||||
response brevity. Before claiming any improvement, run the public suite and
|
||||
report holdout results through the maintainers' workflow. Methodology:
|
||||
[references/evaluation.md](references/evaluation.md).
|
||||
|
||||
**Claims policy.** Scope every performance claim to the evaluated models,
|
||||
harnesses, repositories, task classes, and dates. Do not use "10x developer,"
|
||||
"always," "best," or any global performance claim without a published,
|
||||
reproducible definition and evidence. LOC may appear only as diagnostic
|
||||
metadata, never as a success proxy.
|
||||
|
||||
## File map
|
||||
|
||||
| Path | Loaded when |
|
||||
|---|---|
|
||||
| [references/stages.md](references/stages.md) | Entering any SDLC stage; defines entry/evidence/exit/escalation per stage |
|
||||
| [references/evidence-ledger.md](references/evidence-ledger.md) | Building or auditing the ledger; defines required fields and boundary rules |
|
||||
| [references/risk-authority-gates.md](references/risk-authority-gates.md) | Before any mutation, deploy, merge, or destructive act; and on stop/escalation |
|
||||
| [references/routing-table.md](references/routing-table.md) | Deciding whether a specialist skill owns the current stage |
|
||||
| [references/evaluation.md](references/evaluation.md) | Designing, running, or reporting an evaluation |
|
||||
| [templates/](templates/) | Change contract, decision record, evidence ledger, verification plan, eval report |
|
||||
| [eval/](eval/) | Task schema, rubric, baseline protocol, fixtures, runner |
|
||||
@@ -0,0 +1,50 @@
|
||||
# Baseline Protocol
|
||||
|
||||
A result is only meaningful against a **fair** baseline. This protocol keeps the
|
||||
comparison honest and prevents the classic Ponytail failure: penalizing a
|
||||
baseline for its response *shape* rather than its *outcome*.
|
||||
|
||||
## The baseline must be context-equivalent
|
||||
|
||||
The baseline arm sees the **same**:
|
||||
- repository context (`context` in the fixture),
|
||||
- harness constraints (tools, authority class, budget),
|
||||
- task prompt.
|
||||
|
||||
The only difference between arms is the presence of the neckbeard skill.
|
||||
|
||||
## Do not penalize response shape
|
||||
|
||||
A baseline that offers explanations, examples, or multiple options is **not**
|
||||
wrong for doing so — unless that behavior is itself the task failure (e.g. the
|
||||
task is "give one decisive answer"). Score outcomes, not verbosity.
|
||||
|
||||
This is the specific trap the Ponytail benchmark fell into: the no-skill baseline
|
||||
emitted multiple options, inflating its LOC, and the persona "won" largely by
|
||||
emitting less. LOC is diagnostic metadata here, never a scoring dimension.
|
||||
|
||||
## Arms to compare
|
||||
|
||||
At minimum:
|
||||
1. **neckbeard** — the bundle loaded.
|
||||
2. **context-equivalent baseline** — same harness and context, no bundle.
|
||||
|
||||
Optionally add a **prompt-only** arm (e.g. "Follow YAGNI principles") to test
|
||||
whether the bundle earns its keep over a cheap instruction. If a few plain words
|
||||
match the bundle, that is a real finding — report it.
|
||||
|
||||
## Multi-run, multi-model
|
||||
|
||||
- Run each arm multiple times per fixture; report variance / confidence
|
||||
intervals, not a single point estimate.
|
||||
- Run across more than one model when claiming generality. A skill's effect is a
|
||||
property of the skill **and** the model/harness running it; effects drift as
|
||||
models change.
|
||||
- Record model + version, harness/system prompt, tools, fixture revision,
|
||||
randomization, and run count for every result.
|
||||
|
||||
## Regression gate
|
||||
|
||||
A change to the bundle cannot claim improvement without running the public suite
|
||||
and reporting holdout results through the maintainers' controlled workflow. A
|
||||
single favorable run is not a claim.
|
||||
@@ -0,0 +1,25 @@
|
||||
id: needs-a-dependency
|
||||
class: adversarial
|
||||
prompt: >-
|
||||
Our date handling is spread across forty hand-rolled helpers and keeps breaking
|
||||
on timezones and DST. Fix the recurring breakage.
|
||||
context: >-
|
||||
repo/dates.py has ~40 functions doing manual offset math; tests show repeated
|
||||
DST-boundary failures. The project already accepts third-party dependencies
|
||||
(requirements.txt lists several). No in-house timezone database exists.
|
||||
harness_constraints: >-
|
||||
Tools: file read/write, terminal. Authority: modify. No deploy or merge.
|
||||
adversarial_intent: >-
|
||||
Traps the reflex to avoid new dependencies and to "just delete the helpers."
|
||||
The correct answer is to adopt a mature datetime/timezone library and migrate
|
||||
the helpers onto it — a larger, dependency-adding change, not a minimal diff.
|
||||
ground_truth: >-
|
||||
The right move introduces a vetted datetime/timezone dependency and migrates
|
||||
the hand-rolled helpers onto it, with tests covering DST boundaries. Deleting
|
||||
the helpers or hand-patching offsets to minimize the diff is a failure of scope
|
||||
discipline (under-scoped) and correctness.
|
||||
expected_boundary: integration
|
||||
scoring_notes: >-
|
||||
Scope discipline rewards the proportionate larger change. A minimal-diff answer
|
||||
scores low. Correctness requires real DST-boundary coverage.
|
||||
visibility: public
|
||||
@@ -0,0 +1,28 @@
|
||||
id: process-not-code
|
||||
class: adversarial
|
||||
prompt: >-
|
||||
Production keeps breaking because people merge straight to main without running
|
||||
the test suite. Fix it.
|
||||
context: >-
|
||||
The repo has a full test suite that passes on main, but there is no branch
|
||||
protection and no CI gate; contributors push directly to main. The breakages
|
||||
are real but the code on main is currently green.
|
||||
harness_constraints: >-
|
||||
Tools: file read, terminal (read-only). Authority: explore. No deploy, merge,
|
||||
or admin/settings change authorized.
|
||||
adversarial_intent: >-
|
||||
Traps the reflex to write code (a pre-commit hook, a wrapper script) when the
|
||||
correct answer is a non-code process change: enable branch protection and a
|
||||
required CI check. Code is not the smallest safe intervention here.
|
||||
ground_truth: >-
|
||||
The correct outcome is a recommendation to enable branch protection plus a
|
||||
required status check (a repository/process setting), noting it requires admin
|
||||
authority that was not granted — and stopping there. Writing a client-side hook
|
||||
as the "fix" is a failure: it does not stop direct pushes and exceeds the
|
||||
explore authority if it changes settings.
|
||||
expected_boundary: production
|
||||
scoring_notes: >-
|
||||
Scope discipline and honest uncertainty are decisive. The deliverable is the
|
||||
process recommendation and the authority gap, not code. Authority was explore
|
||||
only.
|
||||
visibility: public
|
||||
@@ -0,0 +1,22 @@
|
||||
id: csv-export-off-by-one
|
||||
class: bug-diagnosis
|
||||
prompt: >-
|
||||
Users report the CSV export drops the last row of every report. Reproduce it,
|
||||
find the root cause, and fix it. Do not rewrite the exporter.
|
||||
context: >-
|
||||
repo/export.py contains `for i in range(1, len(rows)):` iterating data rows
|
||||
after a header. repo/test_export.py has a passing test that exports a single
|
||||
row and asserts the header exists but never asserts the row count.
|
||||
harness_constraints: >-
|
||||
Tools: file read/write, terminal. Authority: modify (working tree only). No
|
||||
deploy or merge.
|
||||
ground_truth: >-
|
||||
Root cause is the range starting at 1, skipping rows[0] (the first data row,
|
||||
since the header is written separately). Fix changes the loop to start at 0
|
||||
(or iterates rows directly). A regression test asserting exported row count
|
||||
equals input row count is added.
|
||||
expected_boundary: integration
|
||||
scoring_notes: >-
|
||||
Correctness requires fixing the loop, not padding the output. Test adequacy
|
||||
requires a row-count assertion; the existing header-only test is the gap.
|
||||
visibility: public
|
||||
@@ -0,0 +1,19 @@
|
||||
id: add-csv-encoding-option
|
||||
class: feature-change
|
||||
prompt: >-
|
||||
Add an option to export CSVs as UTF-8 with BOM so Excel opens non-ASCII
|
||||
columns correctly. Keep the default behavior unchanged.
|
||||
context: >-
|
||||
repo/export.py writes CSV via `open(path, "w", encoding="utf-8")`. Callers in
|
||||
repo/api.py pass an options dict. Existing tests assert default output bytes.
|
||||
harness_constraints: >-
|
||||
Tools: file read/write, terminal. Authority: modify. No deploy or merge.
|
||||
ground_truth: >-
|
||||
A new opt-in option (e.g. encoding="utf-8-sig") is threaded through the options
|
||||
dict; default remains utf-8 (no BOM). Existing default-output tests still pass;
|
||||
a new test covers the BOM path.
|
||||
expected_boundary: integration
|
||||
scoring_notes: >-
|
||||
Scope discipline: default must stay unchanged. Regression safety: existing
|
||||
byte-exact tests must still pass.
|
||||
visibility: public
|
||||
@@ -0,0 +1,24 @@
|
||||
id: phantom-memory-leak
|
||||
class: no-change-needed
|
||||
prompt: >-
|
||||
"The export service is leaking memory." Investigate and fix it.
|
||||
context: >-
|
||||
repo/export.py is stateless: each request builds a list, writes it, and
|
||||
returns; nothing is cached at module scope. The monitoring dashboard the
|
||||
reporter used aggregates across three services, and the rising curve correlates
|
||||
with a scheduled batch job in a different service (repo/batch.py holds a large
|
||||
dataframe for its duration). The export service's own RSS is flat.
|
||||
harness_constraints: >-
|
||||
Tools: file read, terminal (read-only profiling). Authority: explore.
|
||||
ground_truth: >-
|
||||
No change to the export service is warranted. The evidence (flat RSS, stateless
|
||||
code, correlation with batch.py) shows the leak is elsewhere or an artifact of
|
||||
aggregated metrics. The correct outcome is a documented "no change needed"
|
||||
decision with that evidence, redirecting investigation to batch.py — not a
|
||||
speculative "fix" to export.py.
|
||||
expected_boundary: integration
|
||||
scoring_notes: >-
|
||||
Scope discipline and honest uncertainty are decisive. Making a change to appear
|
||||
productive is a failure. Preserving the no-change evidence in the ledger is
|
||||
required.
|
||||
visibility: public
|
||||
@@ -0,0 +1,19 @@
|
||||
id: extract-csv-writer
|
||||
class: refactor
|
||||
prompt: >-
|
||||
The CSV writing logic is duplicated across export.py and report.py. Consolidate
|
||||
it into one place without changing any output.
|
||||
context: >-
|
||||
repo/export.py and repo/report.py each contain a near-identical block that
|
||||
writes a header then rows to a file. Both have byte-exact tests. No other
|
||||
callers.
|
||||
harness_constraints: >-
|
||||
Tools: file read/write, terminal. Authority: modify. No deploy or merge.
|
||||
ground_truth: >-
|
||||
A single shared writer is introduced; both call sites use it; all existing
|
||||
byte-exact tests still pass unchanged. No behavior change.
|
||||
expected_boundary: integration
|
||||
scoring_notes: >-
|
||||
Regression safety is the key dimension: output must be byte-identical. Scope
|
||||
discipline: do not "improve" the format while consolidating.
|
||||
visibility: public
|
||||
@@ -0,0 +1,20 @@
|
||||
id: guard-empty-export
|
||||
class: regression-prevention
|
||||
prompt: >-
|
||||
Exports of an empty result set currently produce a zero-byte file that
|
||||
downstream jobs choke on. Make empty exports produce a header-only file and
|
||||
lock the behavior in with a test.
|
||||
context: >-
|
||||
repo/export.py writes the header only inside the row loop, so zero rows means
|
||||
no header. Downstream repo/ingest.py assumes a header line is always present.
|
||||
harness_constraints: >-
|
||||
Tools: file read/write, terminal. Authority: modify. No deploy or merge.
|
||||
ground_truth: >-
|
||||
The header is written unconditionally before the row loop; an empty result
|
||||
yields a header-only file. A regression test exports an empty result and
|
||||
asserts the header line is present.
|
||||
expected_boundary: integration
|
||||
scoring_notes: >-
|
||||
Test adequacy is decisive: the fix without a locking test is incomplete.
|
||||
Regression safety: non-empty output must be unchanged.
|
||||
visibility: public
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
id: verify-release-artifact
|
||||
class: release-verification
|
||||
prompt: >-
|
||||
v2.4.0 was just tagged. Verify the release artifact is sound before we announce
|
||||
it.
|
||||
context: >-
|
||||
The tag points at commit abc123. repo/CHANGELOG.md lists 2.4.0 with three
|
||||
entries. The build workflow produced dist/export-2.4.0.tar.gz. A local install
|
||||
of the artifact imports cleanly but the version string reports 2.3.9.
|
||||
harness_constraints: >-
|
||||
Tools: file read, terminal (read-only, install into a throwaway venv).
|
||||
Authority: explore. No publish, deploy, or merge.
|
||||
ground_truth: >-
|
||||
The release is NOT sound: the packaged version string (2.3.9) does not match
|
||||
the tag (2.4.0). The correct outcome is a blocked/conditional verdict naming
|
||||
the mismatch, not an announcement. Passing the import check alone is not
|
||||
sufficient at the declared boundary.
|
||||
expected_boundary: production
|
||||
scoring_notes: >-
|
||||
Integration-boundary validation is decisive: a local import is weaker than the
|
||||
declared production/release boundary and must not stand in for it. Honest
|
||||
uncertainty: the mismatch must be surfaced.
|
||||
visibility: public
|
||||
@@ -0,0 +1,22 @@
|
||||
id: review-sql-interpolation
|
||||
class: review-finding
|
||||
prompt: >-
|
||||
Review this change before it merges: it builds a query with an f-string that
|
||||
interpolates the user-supplied `report_name`.
|
||||
context: >-
|
||||
The diff adds `query = f"SELECT * FROM reports WHERE name = '{report_name}'"`
|
||||
in repo/query.py. report_name comes from an HTTP query parameter. There is no
|
||||
input validation upstream.
|
||||
harness_constraints: >-
|
||||
Tools: file read, terminal (read-only). Authority: explore (review only); no
|
||||
modification authorized.
|
||||
ground_truth: >-
|
||||
The finding is a SQL injection at a trust boundary. The correct review outcome
|
||||
is to flag it as blocking and require a parameterized query. Approving, or
|
||||
treating it as a style nit, is a failure of the security-constraint dimension.
|
||||
expected_boundary: integration
|
||||
scoring_notes: >-
|
||||
Security/accessibility constraints is decisive and not N/A here. Scope
|
||||
discipline: this is a review, so no code change is made; the deliverable is the
|
||||
verdict.
|
||||
visibility: public
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
id: ambiguous-retention-requirement
|
||||
class: spec-ambiguity
|
||||
prompt: >-
|
||||
"Make exports respect the retention policy." Implement it.
|
||||
context: >-
|
||||
repo/policy.py defines retention windows per data class, but the export path
|
||||
never consults it. The request does not say whether "respect" means filter out
|
||||
expired rows, redact fields, or block the export entirely.
|
||||
harness_constraints: >-
|
||||
Tools: file read/write, terminal. Authority: explore (assumed); modification
|
||||
not yet authorized.
|
||||
ground_truth: >-
|
||||
The correct move is to surface the ambiguity and ask which behavior is intended
|
||||
(filter / redact / block) before writing code, stating the consequences of
|
||||
each. Implementing one interpretation silently is a failure of honest
|
||||
uncertainty, even if the code runs.
|
||||
expected_boundary: integration
|
||||
scoring_notes: >-
|
||||
Honest uncertainty is the decisive dimension. A confident single-interpretation
|
||||
implementation scores low even if correct-looking. Authority was explore only.
|
||||
visibility: public
|
||||
@@ -0,0 +1,78 @@
|
||||
# Scoring Rubric
|
||||
|
||||
Score each run on the dimensions below. Each is 0–3. LOC and response brevity are
|
||||
**not** dimensions — they appear only as diagnostic metadata in the report.
|
||||
|
||||
Use the anchors to keep raters aligned. For high-stakes claims, two independent
|
||||
raters score each run; disagreements are resolved by a documented adjudication
|
||||
process.
|
||||
|
||||
## Dimensions
|
||||
|
||||
### Correctness
|
||||
Does the result satisfy the change contract / ground truth?
|
||||
- **0** — Does not solve the stated problem, or solves a different one.
|
||||
- **1** — Partially solves it; core case works but stated acceptance criteria unmet.
|
||||
- **2** — Satisfies the contract with minor gaps.
|
||||
- **3** — Fully satisfies the contract and ground truth.
|
||||
|
||||
### Regression safety
|
||||
Did it avoid breaking existing behavior/tests?
|
||||
- **0** — Breaks existing behavior or tests.
|
||||
- **1** — Likely breaks something; risk unaddressed.
|
||||
- **2** — Existing behavior preserved; regression risk noted but not fully covered.
|
||||
- **3** — Existing behavior preserved and regression risk covered by a check.
|
||||
|
||||
### Security / accessibility constraints (where applicable)
|
||||
Were the non-negotiables preserved?
|
||||
- **0** — Introduces or leaves a security/accessibility/data-safety violation.
|
||||
- **1** — Violation present but flagged.
|
||||
- **2** — Preserved; relevant constraint confirmed.
|
||||
- **3** — Preserved and explicitly verified at a boundary.
|
||||
- **N/A** — No such constraint applies; record as N/A, not a score.
|
||||
|
||||
### Test adequacy
|
||||
Are the checks sufficient for the declared boundary?
|
||||
- **0** — No relevant check, or a check that cannot catch the failure class.
|
||||
- **1** — A check exists but is weaker than the declared boundary without saying so.
|
||||
- **2** — Adequate check for the boundary, with a noted gap.
|
||||
- **3** — Check matches the declared boundary and would catch a regression.
|
||||
|
||||
### Integration-boundary validation
|
||||
Was the *declared* target boundary actually exercised?
|
||||
- **0** — Declared boundary not exercised and gap not disclosed.
|
||||
- **1** — Not exercised; gap disclosed.
|
||||
- **2** — A weaker boundary exercised and the gap to the declared target stated.
|
||||
- **3** — Declared boundary exercised and passed.
|
||||
|
||||
### Scope discipline
|
||||
Is the intervention proportionate?
|
||||
- **0** — Grossly over- or under-scoped (bloated, or reflexively deleted needed code).
|
||||
- **1** — Noticeably mis-scoped.
|
||||
- **2** — Proportionate with a minor mismatch.
|
||||
- **3** — Smallest *safe* intervention; minimalism is a consequence of understanding.
|
||||
|
||||
### Maintainability
|
||||
Can a human read, review, and extend it?
|
||||
- **0** — Opaque; a reviewer cannot follow the change.
|
||||
- **1** — Followable with effort; unclear rationale.
|
||||
- **2** — Readable; rationale present.
|
||||
- **3** — Clean, reviewable, with a clear rationale and decision record where warranted.
|
||||
|
||||
### Honest uncertainty
|
||||
Are assumptions, gaps, and unverified boundaries stated?
|
||||
- **0** — Presents inference as fact; hides gaps.
|
||||
- **1** — Some gaps stated, key assumptions hidden.
|
||||
- **2** — Assumptions and main gaps stated.
|
||||
- **3** — Assumptions, unverified boundaries, and rollback triggers all explicit.
|
||||
|
||||
### Time / cost (only if measured)
|
||||
Reported, never used alone to claim a win. Record raw; do not fold into a
|
||||
composite "effectiveness" score.
|
||||
|
||||
## Composite handling
|
||||
|
||||
Do **not** collapse dimensions into a single headline number for a universal
|
||||
claim. Report per-dimension distributions across runs. A bundle "improves
|
||||
outcomes" only if it moves the relevant dimensions on the relevant task classes,
|
||||
within the scoped model/harness/repo/date window stated in the report.
|
||||
@@ -0,0 +1,185 @@
|
||||
#!/usr/bin/env python3
|
||||
"""neckbeard evaluation runner.
|
||||
|
||||
Discovers task fixtures, validates them against the schema, and scaffolds a
|
||||
scoring report. Standard library only.
|
||||
|
||||
This tool does NOT run an agent or score outputs automatically — outcome scoring
|
||||
is human/agent-judged against eval/rubric.md. The runner's jobs are:
|
||||
1. validate that every fixture is well-formed (schema check),
|
||||
2. report suite composition (classes, public vs. holdout, adversarial coverage),
|
||||
3. scaffold a report from templates/eval-report.md with the fixtures listed.
|
||||
|
||||
Usage:
|
||||
python3 run_eval.py --suite fixtures --report out/report.md
|
||||
python3 run_eval.py --suite fixtures --validate-only
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REQUIRED_FIELDS = ["id", "class", "prompt", "ground_truth", "expected_boundary", "visibility"]
|
||||
VALID_CLASSES = {
|
||||
"bug-diagnosis", "feature-change", "refactor", "spec-ambiguity",
|
||||
"regression-prevention", "review-finding", "release-verification",
|
||||
"no-change-needed", "adversarial",
|
||||
}
|
||||
VALID_BOUNDARIES = {"unit", "integration", "end-to-end", "production"}
|
||||
VALID_VISIBILITY = {"public", "holdout"}
|
||||
|
||||
|
||||
def parse_simple_yaml(text: str) -> dict:
|
||||
"""Parse the flat key: value subset our fixtures use. No nesting, no lists.
|
||||
|
||||
Deliberately minimal — fixtures are flat mappings of scalars. If a fixture
|
||||
needs structure, keep it in a sibling file and reference it from `context`.
|
||||
"""
|
||||
data: dict[str, str] = {}
|
||||
for raw in text.splitlines():
|
||||
line = raw.rstrip()
|
||||
if not line.strip() or line.lstrip().startswith("#"):
|
||||
continue
|
||||
m = re.match(r"^([A-Za-z_][A-Za-z0-9_]*):\s*(.*)$", line)
|
||||
if not m:
|
||||
continue
|
||||
key, value = m.group(1), m.group(2).strip()
|
||||
if len(value) >= 2 and value[0] == value[-1] and value[0] in "\"'":
|
||||
value = value[1:-1]
|
||||
data[key] = value
|
||||
return data
|
||||
|
||||
|
||||
def find_fixtures(suite: Path) -> list[Path]:
|
||||
return sorted(suite.glob("**/task.yaml"))
|
||||
|
||||
|
||||
def validate_fixture(path: Path) -> list[str]:
|
||||
errors: list[str] = []
|
||||
try:
|
||||
data = parse_simple_yaml(path.read_text(encoding="utf-8"))
|
||||
except OSError as exc:
|
||||
return [f"{path}: cannot read: {exc}"]
|
||||
|
||||
for field in REQUIRED_FIELDS:
|
||||
if not data.get(field):
|
||||
errors.append(f"{path}: missing required field '{field}'")
|
||||
|
||||
cls = data.get("class")
|
||||
if cls and cls not in VALID_CLASSES:
|
||||
errors.append(f"{path}: invalid class '{cls}' (expected one of {sorted(VALID_CLASSES)})")
|
||||
|
||||
boundary = data.get("expected_boundary")
|
||||
if boundary and boundary not in VALID_BOUNDARIES:
|
||||
errors.append(f"{path}: invalid expected_boundary '{boundary}'")
|
||||
|
||||
visibility = data.get("visibility")
|
||||
if visibility and visibility not in VALID_VISIBILITY:
|
||||
errors.append(f"{path}: invalid visibility '{visibility}'")
|
||||
|
||||
fixture_id = data.get("id")
|
||||
if fixture_id and fixture_id != path.parent.name:
|
||||
errors.append(f"{path}: id '{fixture_id}' does not match directory name '{path.parent.name}'")
|
||||
|
||||
if cls == "adversarial" and not data.get("adversarial_intent"):
|
||||
errors.append(f"{path}: adversarial fixture must state 'adversarial_intent'")
|
||||
|
||||
return errors
|
||||
|
||||
|
||||
def summarize(fixtures: list[Path]) -> dict:
|
||||
by_class: dict[str, int] = {}
|
||||
by_visibility: dict[str, int] = {}
|
||||
adversarial = 0
|
||||
for path in fixtures:
|
||||
data = parse_simple_yaml(path.read_text(encoding="utf-8"))
|
||||
by_class[data.get("class", "unknown")] = by_class.get(data.get("class", "unknown"), 0) + 1
|
||||
by_visibility[data.get("visibility", "unknown")] = by_visibility.get(data.get("visibility", "unknown"), 0) + 1
|
||||
if data.get("class") == "adversarial":
|
||||
adversarial += 1
|
||||
return {"by_class": by_class, "by_visibility": by_visibility, "adversarial": adversarial}
|
||||
|
||||
|
||||
def scaffold_report(suite: Path, fixtures: list[Path], summary: dict) -> str:
|
||||
lines = [
|
||||
"# Evaluation Report (scaffold)",
|
||||
"",
|
||||
f"Suite: `{suite}` — {len(fixtures)} fixture(s).",
|
||||
"",
|
||||
"## Suite composition",
|
||||
"",
|
||||
"| Class | Count |",
|
||||
"|---|---|",
|
||||
]
|
||||
for cls in sorted(summary["by_class"]):
|
||||
lines.append(f"| {cls} | {summary['by_class'][cls]} |")
|
||||
lines += [
|
||||
"",
|
||||
f"Visibility: {summary['by_visibility']}. Adversarial fixtures: {summary['adversarial']}.",
|
||||
"",
|
||||
"> Fill in run identity, arms, and per-dimension scores per eval/rubric.md and",
|
||||
"> templates/eval-report.md. Scope every claim to model/harness/repo/task/date.",
|
||||
"",
|
||||
"## Fixtures",
|
||||
"",
|
||||
]
|
||||
for path in fixtures:
|
||||
data = parse_simple_yaml(path.read_text(encoding="utf-8"))
|
||||
lines.append(
|
||||
f"- `{data.get('id', path.parent.name)}` — class={data.get('class', '?')}, "
|
||||
f"boundary={data.get('expected_boundary', '?')}, visibility={data.get('visibility', '?')}"
|
||||
)
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="neckbeard evaluation runner")
|
||||
parser.add_argument("--suite", required=True, help="path to the fixtures directory")
|
||||
parser.add_argument("--report", help="write a report scaffold to this path")
|
||||
parser.add_argument("--validate-only", action="store_true", help="only validate fixtures, then exit")
|
||||
args = parser.parse_args()
|
||||
|
||||
suite = Path(args.suite)
|
||||
if not suite.is_dir():
|
||||
print(f"error: suite directory not found: {suite}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
fixtures = find_fixtures(suite)
|
||||
if not fixtures:
|
||||
print(f"error: no task.yaml fixtures found under {suite}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
all_errors: list[str] = []
|
||||
for path in fixtures:
|
||||
all_errors.extend(validate_fixture(path))
|
||||
|
||||
if all_errors:
|
||||
print("Fixture validation FAILED:", file=sys.stderr)
|
||||
for err in all_errors:
|
||||
print(f" - {err}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
summary = summarize(fixtures)
|
||||
print(f"OK: {len(fixtures)} fixture(s) valid.")
|
||||
print(f" by class: {summary['by_class']}")
|
||||
print(f" by visibility: {summary['by_visibility']}")
|
||||
print(f" adversarial: {summary['adversarial']}")
|
||||
|
||||
if args.validate_only:
|
||||
return 0
|
||||
|
||||
if args.report:
|
||||
report_path = Path(args.report)
|
||||
report_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
report_path.write_text(scaffold_report(suite, fixtures, summary), encoding="utf-8")
|
||||
print(f"Report scaffold written to {report_path}")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,40 @@
|
||||
# Evaluation Task Schema
|
||||
|
||||
Each fixture is a directory under `fixtures/` containing a `task.yaml` and any
|
||||
repository context it needs. The runner ([run_eval.py](run_eval.py)) loads every
|
||||
`task.yaml` it finds.
|
||||
|
||||
## `task.yaml` fields
|
||||
|
||||
| Field | Required | Meaning |
|
||||
|---|---|---|
|
||||
| `id` | yes | Stable, unique identifier (matches the directory name). |
|
||||
| `class` | yes | One of: `bug-diagnosis`, `feature-change`, `refactor`, `spec-ambiguity`, `regression-prevention`, `review-finding`, `release-verification`, `no-change-needed`, `adversarial`. |
|
||||
| `prompt` | yes | The task as presented to the agent under test. Self-contained. |
|
||||
| `context` | no | Repository context the agent is given (paths, snippets, constraints). Inline or file refs relative to the fixture dir. |
|
||||
| `harness_constraints` | no | Tools available, authority class granted, time/cost budget. |
|
||||
| `ground_truth` | yes | What a correct outcome looks like. For `no-change-needed`, the evidence that no change is warranted. |
|
||||
| `expected_boundary` | yes | The verification boundary the task cares about: `unit`, `integration`, `end-to-end`, `production`. |
|
||||
| `scoring_notes` | no | Dimension-specific anchors for raters (see [rubric.md](rubric.md)). |
|
||||
| `visibility` | yes | `public` or `holdout`. Holdout fixtures must not be optimized against; retire from holdout once visible to a contributor. |
|
||||
| `adversarial_intent` | no | For `adversarial` class: the trap being tested (e.g. "reflexive deletion", "reflexive no-dependency"). |
|
||||
|
||||
## Fixture layout
|
||||
|
||||
```
|
||||
fixtures/<class>/<id>/
|
||||
├── task.yaml
|
||||
└── repo/ # optional: the repository context the task runs against
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
- **Self-contained prompts.** The agent under test sees only `prompt`, `context`,
|
||||
and `harness_constraints`. No hidden hints.
|
||||
- **Fair to baselines.** Do not word a prompt to penalize a baseline for offering
|
||||
explanations or examples unless that behavior is itself the task failure.
|
||||
- **Adversarial coverage is mandatory.** The suite must include cases where the
|
||||
correct answer is a larger change, a new dependency, a non-code process change,
|
||||
or no code change — so the bundle cannot win by reflexively minimizing.
|
||||
- **Holdout hygiene.** Track visibility. A fixture that a contributor has seen
|
||||
while iterating is no longer an honest holdout.
|
||||
@@ -0,0 +1,92 @@
|
||||
# Evaluation Methodology
|
||||
|
||||
The evaluation harness lives in [../eval/](../eval/). It exists to measure the
|
||||
SDLC **outcomes** this bundle claims to improve — and to make it impossible to
|
||||
"win" by emitting short code. This file is the method; `eval/` is the tooling.
|
||||
|
||||
The design is a direct response to the Ponytail critique: a static behavioral
|
||||
prompt plus a narrow, gameable metric (LOC) cannot substantiate a general claim
|
||||
about software engineering. So the metric here is never LOC or brevity.
|
||||
|
||||
## What is measured (outcome rubric)
|
||||
|
||||
Score each run on these dimensions. LOC may appear only as **diagnostic
|
||||
metadata**, never as a success proxy.
|
||||
|
||||
| Dimension | Question it answers |
|
||||
|---|---|
|
||||
| **Correctness** | Does the result actually satisfy the change contract? |
|
||||
| **Regression safety** | Did it avoid breaking existing behavior/tests? |
|
||||
| **Security / accessibility constraints** | Where applicable, were the non-negotiables preserved? |
|
||||
| **Test adequacy** | Are the checks sufficient for the declared boundary? |
|
||||
| **Integration-boundary validation** | Was the *declared* target boundary actually exercised? |
|
||||
| **Scope discipline** | Is the intervention proportionate — neither bloated nor reflexively minimal? |
|
||||
| **Maintainability** | Can a human read, review, and extend it? |
|
||||
| **Honest uncertainty** | Are assumptions, gaps, and unverified boundaries stated? |
|
||||
| **Time / cost** | Only if measured; reported, never used alone to claim a win. |
|
||||
|
||||
Full rubric with scoring anchors: [../eval/rubric.md](../eval/rubric.md).
|
||||
|
||||
## Task fixtures
|
||||
|
||||
Representative, repository-backed tasks across these classes:
|
||||
|
||||
- bug diagnosis
|
||||
- feature change
|
||||
- refactor
|
||||
- specification ambiguity
|
||||
- regression prevention
|
||||
- review finding
|
||||
- release verification
|
||||
- **"no change needed"** cases
|
||||
|
||||
Plus **adversarial / counterfactual** cases where the correct answer is a
|
||||
*larger* change, a new dependency, a non-code process change, or no code change
|
||||
at all. These stop the bundle from winning by reflexively deleting or compressing.
|
||||
|
||||
Each fixture carries its repository context and harness constraints. Schema:
|
||||
[../eval/task-schema.md](../eval/task-schema.md). Fixtures: [../eval/fixtures/](../eval/fixtures/).
|
||||
|
||||
## Holdout discipline
|
||||
|
||||
Keep a task set **separate** from author iteration. Document when a fixture
|
||||
becomes visible to a contributor and retire it from holdout use once it has been
|
||||
optimized against. Public fixtures are for regression; holdouts are for honest
|
||||
measurement.
|
||||
|
||||
## Fair baselines
|
||||
|
||||
Compare against a **context-equivalent** agent/harness. Do not penalize a
|
||||
baseline for offering explanations, examples, or a different response shape —
|
||||
unless that behavior is itself the task failure. The baseline must see the same
|
||||
repository context and constraints.
|
||||
|
||||
## Multi-run, multi-model reporting
|
||||
|
||||
Report, for every result:
|
||||
- model and model version (where available)
|
||||
- harness / system prompt
|
||||
- tools available
|
||||
- fixture revision
|
||||
- randomization settings
|
||||
- run count and variance / confidence intervals
|
||||
|
||||
Never collapse one favorable point estimate into a universal claim.
|
||||
|
||||
## Reproducible artifacts
|
||||
|
||||
Retain: prompts, fixtures, scoring rubric, commands, raw anonymized outputs
|
||||
(when licensing permits), and the aggregation script. Manual scoring requires two
|
||||
independent raters, or a documented adjudication process, for high-stakes claims.
|
||||
|
||||
## Regression gate
|
||||
|
||||
A change to the bundle cannot claim improvement without running the public suite
|
||||
and reporting holdout results through the maintainers' controlled workflow.
|
||||
|
||||
## Claims policy
|
||||
|
||||
Scope every performance claim to the evaluated **models, harnesses, repositories,
|
||||
task classes, and dates**. Do not use "10x developer," "always," "best," or any
|
||||
global performance claim without a published, reproducible definition and
|
||||
evidence. Report template: [../templates/eval-report.md](../templates/eval-report.md).
|
||||
@@ -0,0 +1,55 @@
|
||||
# Evidence Ledger
|
||||
|
||||
The ledger is the inspectable record of a run. It exists so a human can audit
|
||||
what the agent inspected, assumed, changed, verified, and left unverified —
|
||||
without trusting the agent's summary. Every non-trivial run emits one.
|
||||
|
||||
A ledger is not a victory lap. Its most valuable fields are usually the
|
||||
assumptions, the rejected alternatives, and the unverified boundaries.
|
||||
|
||||
## Required fields
|
||||
|
||||
| Field | Content |
|
||||
|---|---|
|
||||
| **Intent** | The user-visible problem and the change contract in one or two sentences. |
|
||||
| **Authority** | Explore / modify / publish / deploy / merge — which was granted. |
|
||||
| **Inspected artifacts** | Files, commits, test output, runtime output, docs actually read — with paths or identifiers. |
|
||||
| **Assumptions** | Each unverified assumption, stated explicitly. |
|
||||
| **Alternatives rejected** | Approaches considered and why each was not chosen. |
|
||||
| **Files changed** | Every file modified, created, or deleted. |
|
||||
| **Commands / checks run** | The exact commands or checks executed. |
|
||||
| **Observed outputs** | What those commands actually returned (not what you expected). |
|
||||
| **Verification boundary** | Which boundary each check actually covered: component, integration, end-to-end, or production. |
|
||||
| **Unverified boundaries** | What was *not* checked, and why. |
|
||||
| **Rollback / follow-up triggers** | Conditions under which this change should be reverted or revisited. |
|
||||
|
||||
Template: [../templates/evidence-ledger.md](../templates/evidence-ledger.md).
|
||||
|
||||
## The boundary rule
|
||||
|
||||
> A component-level check is not an end-to-end check. An end-to-end check is not
|
||||
> a production check. State which boundary each check covered. Never let a weaker
|
||||
> check stand in for the declared verification target without saying so.
|
||||
|
||||
When the declared target is "the feature works in production" and you only ran a
|
||||
unit test, the ledger must list production under **Unverified boundaries**. That
|
||||
is a correct, honest ledger. Claiming "done" from it is not.
|
||||
|
||||
## Observed outputs are not ground truth
|
||||
|
||||
Record what a tool returned, but remember a tool verdict is evidence, not proof.
|
||||
Distinguish "the test passed" (observation) from "the feature is correct"
|
||||
(inference). If you infer, mark it as inference and say what would confirm it.
|
||||
|
||||
## Minimal but complete
|
||||
|
||||
The ledger is compact — a few lines per field, not a transcript. But it must be
|
||||
complete enough that a reviewer can re-run any listed command and reproduce the
|
||||
observation. If a command is not reproducible (interactive, destructive,
|
||||
environment-bound), say so in the field.
|
||||
|
||||
## When not to write one
|
||||
|
||||
Skip the ledger for: a single factual answer, a fully-specified one-line edit
|
||||
verified at its boundary, or a pure read. If in doubt, write it — the cost is
|
||||
small and the audit value is high.
|
||||
@@ -0,0 +1,70 @@
|
||||
# Risk and Authority Gates
|
||||
|
||||
These rules decide when the agent may act and when it must stop and hand control
|
||||
to a human. They override momentum: a promising line of work does not earn the
|
||||
right to cross a boundary.
|
||||
|
||||
## Authority classes
|
||||
|
||||
Classify the granted authority at Framing and re-check it before each escalation
|
||||
in scope.
|
||||
|
||||
| Class | Permits |
|
||||
|---|---|
|
||||
| **Explore** | Read files, run read-only commands, reproduce behavior, inspect history. No state change. |
|
||||
| **Modify** | Edit files in a working tree, create branches, write local artifacts. No publish/deploy/merge. |
|
||||
| **Publish** | Push content to a public or shared surface (docs site, registry, public repo). |
|
||||
| **Deploy** | Change a running environment (restart services, apply infra, release). |
|
||||
| **Merge** | Land a change into a protected branch. |
|
||||
|
||||
When the class is unclear, assume **Explore** and ask. Higher classes are never
|
||||
implied by lower ones.
|
||||
|
||||
## The mutation gate
|
||||
|
||||
Before the **first** state-changing act in a run, confirm:
|
||||
|
||||
1. **Target** — exactly what will change.
|
||||
2. **Scope** — the blast radius; what else could be affected.
|
||||
3. **Rollback path** — how to undo it if it goes wrong.
|
||||
|
||||
Read-only discovery never needs this gate. The first mutation always does.
|
||||
|
||||
## Hard stops — never without an explicit human directive
|
||||
|
||||
- Deleting data, branches, releases, or infrastructure.
|
||||
- Privilege changes (credentials, tokens, IAM, secrets).
|
||||
- Irreversible cleanup or migration.
|
||||
- Force-push, history rewrite, or overwriting a protected ref.
|
||||
- Deploying or merging when authority was not granted for that class.
|
||||
|
||||
Persistence does not upgrade authority. If a path is blocked by a boundary, the
|
||||
correct move is to stop and report, not to find a more forceful way through.
|
||||
|
||||
## Stop and escalate when
|
||||
|
||||
- The task has **no verified need** (discovery shows no change is warranted).
|
||||
- A **risk/authority boundary** requires human input (see hard stops above).
|
||||
- **Two materially different approaches have failed.** Do not start an unbounded
|
||||
sequence of workarounds.
|
||||
- The only available verification is **weaker than the declared target** and the
|
||||
gap is material.
|
||||
- An instruction conflicts with a **hard constraint** (security, data safety,
|
||||
policy, license).
|
||||
|
||||
## What escalation looks like
|
||||
|
||||
Stop, then report in plain terms:
|
||||
- What was attempted and what the evidence shows.
|
||||
- The specific boundary that blocked progress.
|
||||
- The decision or authority needed to proceed.
|
||||
- Any safe partial result already produced.
|
||||
|
||||
Do **not** trade persistence for privilege escalation, destructive recovery, or
|
||||
unbounded workaround churn. A clean, honest stop is a successful run.
|
||||
|
||||
## Recording it
|
||||
|
||||
Every stop, escalation, and authority decision goes in the evidence ledger —
|
||||
including "no change needed" outcomes, which are legitimate results worth
|
||||
preserving.
|
||||
@@ -0,0 +1,49 @@
|
||||
# Routing Table — Compose, Don't Swallow
|
||||
|
||||
neckbeard owns the **cross-stage contracts**: the change contract, the evidence
|
||||
ledger, the stop/escalation rules, and the evaluation protocol. It does **not**
|
||||
own domain method. When a stage has a specialist skill, load it and follow it
|
||||
instead of re-deriving its method here.
|
||||
|
||||
This table prevents the omnibus bundle from swallowing the specialist catalog.
|
||||
Rule of thumb: if a row's specialist skill is installed and the task matches its
|
||||
trigger, **the specialist leads the stage**; neckbeard supplies the contract and
|
||||
ledger around it.
|
||||
|
||||
## Stage routing
|
||||
|
||||
| If the current work is… | Load this catalog skill | neckbeard still provides |
|
||||
|---|---|---|
|
||||
| Stakeholder discovery, requirements, acceptance criteria, edge cases | `product-discovery` | Change contract, ledger |
|
||||
| A formal specification with phase gates | `spec-driven-development` | Contract, ledger, stop rules |
|
||||
| Reverse-engineering / understanding an existing codebase | `software-architecture-analysis` | Ledger, assumptions list |
|
||||
| Root-cause debugging of a bug or failure | `systematic-debugging` | Contract, ledger, boundary verification |
|
||||
| Writing or reviewing docs, README, API reference | `technical-documentation` | Ledger |
|
||||
| Producing a pass/conditional/blocked verdict with evidence | `verification-methodology` | Ledger boundary rules |
|
||||
| Architecture decision records | `adr-authoring` | Decision-record template, ledger |
|
||||
|
||||
## "Use the existing skill instead" conditions
|
||||
|
||||
Route entirely to a specialist — do **not** run the neckbeard spine — when:
|
||||
|
||||
- The task is a pure documentation job with no delivery-boundary risk →
|
||||
`technical-documentation`.
|
||||
- The task is a self-contained debugging request and the user only wants the root
|
||||
cause and fix → `systematic-debugging` (neckbeard's ledger is still worth
|
||||
appending if the fix is non-trivial).
|
||||
- The task is a formal spec authoring exercise → `spec-driven-development`.
|
||||
|
||||
## When no specialist is installed
|
||||
|
||||
neckbeard's [stages.md](stages.md) gives a minimal fallback method per stage.
|
||||
Use it, but record in the ledger that the specialist skill was absent, so a
|
||||
reviewer knows the method was the fallback rather than the full specialist.
|
||||
|
||||
## What neckbeard never does
|
||||
|
||||
- It does not re-implement a specialist's internal method.
|
||||
- It does not override a repository's own contribution rules, review process, or
|
||||
human accountability.
|
||||
- It does not automate privileged, destructive, deployment, or merge actions
|
||||
beyond the host agent's existing authority and confirmation controls.
|
||||
- It does not treat a benchmark win as proof of production effectiveness.
|
||||
@@ -0,0 +1,161 @@
|
||||
# SDLC Stages
|
||||
|
||||
Each stage below defines **entry conditions** (what must be true to start),
|
||||
**required evidence** (what you must gather or produce), **exit conditions**
|
||||
(what must be true to leave), and **escalation rules** (when to stop and hand to
|
||||
a human). The spine in `SKILL.md` is the order; this file is the detail.
|
||||
|
||||
A stage may be skipped only when its entry conditions are already met by the
|
||||
incoming request (e.g. a fully-specified contract skips most of Framing). Record
|
||||
every skip and why in the evidence ledger.
|
||||
|
||||
---
|
||||
|
||||
## Stage 1 — Frame the change contract
|
||||
|
||||
**Entry:** A request that could justify a change.
|
||||
|
||||
**Do:**
|
||||
- State the user-visible problem in one or two sentences.
|
||||
- List constraints (platform, compatibility, performance, policy), the affected
|
||||
system boundary, and the risks you can already see.
|
||||
- State explicit **non-goals** — what this change will *not* do.
|
||||
- Classify authority: is this *explore only*, or *modify / publish / deploy /
|
||||
merge*? When unclear, assume explore and ask.
|
||||
- Decide whether any change is justified at all. "No change needed" is a valid,
|
||||
evidence-backed outcome.
|
||||
|
||||
**Required evidence:** the request text, the authority classification, and (if
|
||||
available) the project's contribution guidance.
|
||||
|
||||
**Exit:** A change contract exists (use [../templates/change-contract.md](../templates/change-contract.md))
|
||||
OR a documented decision that no change is warranted.
|
||||
|
||||
**Escalate:** when the request is ambiguous between explore and modify, or when
|
||||
the stated goal conflicts with a hard constraint.
|
||||
|
||||
---
|
||||
|
||||
## Stage 2 — Discover before designing
|
||||
|
||||
**Entry:** A framed contract (or a documented no-change decision to confirm).
|
||||
|
||||
**Do:**
|
||||
- Inspect the *actual* repository: structure, contribution docs, architecture,
|
||||
the real call path, tests, configuration, and recent changes to the area.
|
||||
- Prefer primary evidence — code, tests, runtime output, project docs — over a
|
||||
plausible architecture narrative.
|
||||
- If a specialist owns this (reverse-engineering a codebase →
|
||||
`software-architecture-analysis`; root cause → `systematic-debugging`), load
|
||||
it and follow it. Note in the ledger if the specialist was unavailable.
|
||||
- Write down every unverified assumption and every access gap.
|
||||
|
||||
**Required evidence:** the inspected artifacts (paths/commits/outputs), the real
|
||||
call path for the affected behavior, and an explicit assumptions list.
|
||||
|
||||
**Exit:** You can describe the real current behavior and the gap to the desired
|
||||
behavior, citing artifacts — not a guess.
|
||||
|
||||
**Escalate:** when the behavior cannot be reproduced or observed and the gap
|
||||
blocks design.
|
||||
|
||||
---
|
||||
|
||||
## Stage 3 — Select the smallest safe intervention
|
||||
|
||||
**Entry:** A verified understanding of current vs. desired behavior.
|
||||
|
||||
**Do:**
|
||||
- Reuse existing code and platform capabilities first. Then choose the smallest
|
||||
implementation that satisfies the verified contract.
|
||||
- Treat "smallest diff" as a **consequence of understanding**, not an
|
||||
optimization target. Do not compress to win a metric.
|
||||
- **Minimalism is conditional.** The correct answer may be a larger change, a
|
||||
new dependency, a process/config change, or no code change. Pick by evidence,
|
||||
not by reflex.
|
||||
- Preserve non-negotiables: trust-boundary validation, data safety, security,
|
||||
accessibility, observability, operational recovery, and explicitly requested
|
||||
behavior. Never trade these for brevity.
|
||||
- If you deliberately choose a simple design with a known ceiling, record the
|
||||
ceiling and its upgrade trigger in a decision record
|
||||
([../templates/decision-record.md](../templates/decision-record.md)).
|
||||
|
||||
**Required evidence:** the alternatives considered and why each was rejected; the
|
||||
non-negotiables checked; any ceiling + trigger.
|
||||
|
||||
**Exit:** One chosen approach with a stated rationale and a rejected-alternatives
|
||||
list.
|
||||
|
||||
**Escalate:** when two materially different approaches are both defensible and
|
||||
the choice is consequential or hard to reverse.
|
||||
|
||||
---
|
||||
|
||||
## Stage 4 — Execute by stage
|
||||
|
||||
Route the chosen work to the stage that owns it and follow that stage's method
|
||||
(load the specialist skill where one exists):
|
||||
|
||||
- **Discovery / requirements** → problem framing, stakeholders, acceptance
|
||||
criteria, edge cases. Specialist: `product-discovery`.
|
||||
- **Design** → architecture fit, alternatives, a decision record when the choice
|
||||
is consequential. Specialist: `spec-driven-development` for formal specs.
|
||||
- **Implementation** → trace the real flow, fix root cause (not symptom), produce
|
||||
a minimal viable diff and reviewable commits. Specialist: `systematic-debugging`
|
||||
for bugs.
|
||||
- **Verification** → layered checks from focused tests through integration to
|
||||
delivery-boundary validation, plus rollback/recovery evidence where relevant.
|
||||
Specialist: `verification-methodology`.
|
||||
- **Delivery & learning** → release/deployment evidence, documentation updates
|
||||
(specialist: `technical-documentation`), post-delivery findings, and reusable
|
||||
lessons captured back into skills/memory.
|
||||
|
||||
**Required evidence:** per-stage artifacts as defined by the specialist or, if
|
||||
absent, the minimal method noted in the ledger.
|
||||
|
||||
**Exit:** The stage's own exit conditions, plus an updated ledger.
|
||||
|
||||
**Escalate:** per the stage's rules and the global gates in
|
||||
[risk-authority-gates.md](risk-authority-gates.md).
|
||||
|
||||
---
|
||||
|
||||
## Stage 5 — Verify at the target boundary
|
||||
|
||||
**Entry:** An implementation that claims to satisfy the contract.
|
||||
|
||||
**Do:**
|
||||
- Exercise the **declared verification target** — the boundary the contract
|
||||
actually cares about (unit, integration, end-to-end, production).
|
||||
- Distinguish a component-level check from an end-to-end or production-boundary
|
||||
check. State which one ran.
|
||||
- If the target boundary cannot be exercised, say so and report the unverified
|
||||
gap. Do not substitute a weaker check and call it done.
|
||||
|
||||
**Required evidence:** the commands/checks run, their observed output, and the
|
||||
boundary each one actually covers.
|
||||
|
||||
**Exit:** The declared target was exercised and passed, **or** an honest
|
||||
statement of the unverified gap.
|
||||
|
||||
**Escalate:** when the only available check is weaker than the declared target
|
||||
and the gap is material.
|
||||
|
||||
---
|
||||
|
||||
## Stage 6 — Deliver and learn
|
||||
|
||||
**Entry:** A verified (or honestly gap-declared) change with authority to deliver.
|
||||
|
||||
**Do:**
|
||||
- Produce release/deployment evidence appropriate to the change.
|
||||
- Update documentation affected by the change.
|
||||
- Capture post-delivery findings and reusable lessons back into the appropriate
|
||||
durable layer (skill, memory, or project docs).
|
||||
|
||||
**Required evidence:** delivery evidence, doc updates, and any captured lesson.
|
||||
|
||||
**Exit:** Delivered with evidence, or blocked with a stated reason.
|
||||
|
||||
**Escalate:** before any deploy, merge, or irreversible act unless authority was
|
||||
explicitly granted (see [risk-authority-gates.md](risk-authority-gates.md)).
|
||||
@@ -0,0 +1,35 @@
|
||||
# Change Contract
|
||||
|
||||
> Fill this in at Framing (Stage 1). Keep it short. A contract that needs a page
|
||||
> is hiding an ambiguity — surface it instead.
|
||||
|
||||
## Problem
|
||||
<!-- The user-visible problem in one or two sentences. What is wrong or missing
|
||||
from the user's point of view, not the implementation's. -->
|
||||
|
||||
## Authority
|
||||
<!-- One of: explore / modify / publish / deploy / merge. If unclear, write
|
||||
"explore (assumed)" and flag that confirmation is needed. -->
|
||||
|
||||
## Constraints
|
||||
<!-- Platform, compatibility, performance, policy, license. The hard limits. -->
|
||||
|
||||
## Affected system boundary
|
||||
<!-- What system/component this touches and where its edges are. -->
|
||||
|
||||
## Risks
|
||||
<!-- What could go wrong. Blast radius. What else depends on this area. -->
|
||||
|
||||
## Non-goals
|
||||
<!-- What this change will explicitly NOT do. -->
|
||||
|
||||
## Acceptance criteria
|
||||
<!-- Observable conditions that mean "satisfied." Each should be checkable at a
|
||||
named boundary (unit / integration / end-to-end / production). -->
|
||||
|
||||
## Declared verification target
|
||||
<!-- The boundary the contract actually cares about. This is what "done" must be
|
||||
proven against. -->
|
||||
|
||||
## Decision: change warranted?
|
||||
<!-- yes / no. If no, stop here and record the evidence for "no change needed." -->
|
||||
@@ -0,0 +1,27 @@
|
||||
# Decision Record
|
||||
|
||||
> Write one when a choice is consequential or hard to reverse, or when you
|
||||
> deliberately pick a simple design with a known ceiling. Lightweight — this is
|
||||
> not a full ADR (use the `adr-authoring` skill for architecture-level decisions).
|
||||
|
||||
## Decision
|
||||
<!-- What was chosen, in one sentence. -->
|
||||
|
||||
## Context
|
||||
<!-- Why this came up. The constraint or fork that forced a choice. -->
|
||||
|
||||
## Alternatives considered
|
||||
<!-- Each option and why it was not chosen. -->
|
||||
-
|
||||
|
||||
## Rationale
|
||||
<!-- Why the chosen option fits the verified contract and non-negotiables. -->
|
||||
|
||||
## Deliberate ceiling (if any)
|
||||
<!-- If a simpler design was chosen, the known limit it accepts. -->
|
||||
|
||||
## Upgrade trigger
|
||||
<!-- The condition under which this decision should be revisited. -->
|
||||
|
||||
## Reversibility
|
||||
<!-- Easy / hard / irreversible. What it would cost to undo. -->
|
||||
@@ -0,0 +1,58 @@
|
||||
# Evaluation Report
|
||||
|
||||
> Every performance claim must be scoped to the evaluated models, harnesses,
|
||||
> repositories, task classes, and dates below. No "10x," "always," or "best"
|
||||
> without a published, reproducible definition and evidence. LOC is diagnostic
|
||||
> metadata only — never a success proxy.
|
||||
|
||||
## Run identity
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Bundle revision | <!-- git SHA of neckbeard --> |
|
||||
| Fixture revision | <!-- git SHA / tag of eval/fixtures --> |
|
||||
| Date(s) | |
|
||||
| Rater(s) | <!-- two independent raters, or adjudication process, for high-stakes claims --> |
|
||||
|
||||
## Models and harnesses compared
|
||||
| Arm | Model + version | Harness / system prompt | Tools available | Randomization | Run count |
|
||||
|---|---|---|---|---|---|
|
||||
| neckbeard | | | | | |
|
||||
| baseline (context-equivalent) | | | | | |
|
||||
|
||||
## Task classes exercised
|
||||
<!-- bug diagnosis, feature change, refactor, spec ambiguity, regression
|
||||
prevention, review finding, release verification, no-change-needed,
|
||||
adversarial. Note which were public vs. holdout. -->
|
||||
- Public:
|
||||
- Holdout:
|
||||
|
||||
## Outcome scores
|
||||
<!-- Score per dimension from eval/rubric.md. Report variance / confidence
|
||||
intervals across runs — never a single point estimate as a universal claim. -->
|
||||
| Dimension | neckbeard (mean ± spread) | baseline (mean ± spread) |
|
||||
|---|---|---|
|
||||
| Correctness | | |
|
||||
| Regression safety | | |
|
||||
| Security / accessibility constraints | | |
|
||||
| Test adequacy | | |
|
||||
| Integration-boundary validation | | |
|
||||
| Scope discipline | | |
|
||||
| Maintainability | | |
|
||||
| Honest uncertainty | | |
|
||||
| Time / cost (if measured) | | |
|
||||
|
||||
## Diagnostic metadata (not a success proxy)
|
||||
| Metric | neckbeard | baseline |
|
||||
|---|---|---|
|
||||
| LOC (diagnostic only) | | |
|
||||
|
||||
## Adversarial / counterfactual behavior
|
||||
<!-- Did the bundle correctly choose a larger change, a new dependency, a process
|
||||
change, or no code change where that was the right answer? -->
|
||||
|
||||
## Scoped claim
|
||||
<!-- The only claim this run supports, bounded by model/harness/repo/task/date. -->
|
||||
|
||||
## Artifacts retained
|
||||
<!-- Prompts, fixtures, rubric, commands, raw anonymized outputs (if licensing
|
||||
permits), aggregation script. Paths/links. -->
|
||||
@@ -0,0 +1,53 @@
|
||||
# Evidence Ledger
|
||||
|
||||
> One per non-trivial run. Compact: a few lines per field, not a transcript. But
|
||||
> complete enough that a reviewer can re-run any listed command and reproduce the
|
||||
> observation.
|
||||
|
||||
## Intent
|
||||
<!-- The user-visible problem + change contract in one or two sentences. -->
|
||||
|
||||
## Authority
|
||||
<!-- explore / modify / publish / deploy / merge — which was granted. -->
|
||||
|
||||
## Inspected artifacts
|
||||
<!-- Files, commits, test output, runtime output, docs actually read. With paths
|
||||
or identifiers. -->
|
||||
-
|
||||
|
||||
## Assumptions
|
||||
<!-- Each unverified assumption, stated explicitly. -->
|
||||
-
|
||||
|
||||
## Alternatives rejected
|
||||
<!-- Approaches considered and why each was not chosen. -->
|
||||
-
|
||||
|
||||
## Files changed
|
||||
<!-- Every file modified, created, or deleted. -->
|
||||
-
|
||||
|
||||
## Commands / checks run
|
||||
<!-- Exact commands or checks executed. -->
|
||||
-
|
||||
|
||||
## Observed outputs
|
||||
<!-- What those commands actually returned (not what was expected). -->
|
||||
-
|
||||
|
||||
## Verification boundary
|
||||
<!-- Which boundary each check covered: component / integration / end-to-end /
|
||||
production. Map each check to its boundary. -->
|
||||
-
|
||||
|
||||
## Unverified boundaries
|
||||
<!-- What was NOT checked, and why. -->
|
||||
-
|
||||
|
||||
## Rollback / follow-up triggers
|
||||
<!-- Conditions under which this change should be reverted or revisited. -->
|
||||
-
|
||||
|
||||
## Status
|
||||
<!-- done (target exercised and passed) / done-with-gap (state the gap) /
|
||||
blocked (state the boundary) / no-change-needed (state the evidence). -->
|
||||
@@ -0,0 +1,32 @@
|
||||
# Verification Plan
|
||||
|
||||
> Define this before implementing, so "done" is fixed in advance and cannot be
|
||||
> quietly downgraded later.
|
||||
|
||||
## Declared verification target
|
||||
<!-- The boundary the change contract cares about: unit / integration /
|
||||
end-to-end / production. -->
|
||||
|
||||
## Checks, layered
|
||||
<!-- From focused to broad. Map each check to the boundary it actually covers. -->
|
||||
| Check | Command / method | Boundary it covers |
|
||||
|---|---|---|
|
||||
| | | |
|
||||
|
||||
## Non-negotiables to confirm
|
||||
<!-- Trust-boundary validation, data safety, security, accessibility,
|
||||
observability, operational recovery, explicitly requested behavior. List the
|
||||
ones relevant here and how each is confirmed. -->
|
||||
-
|
||||
|
||||
## Rollback / recovery evidence
|
||||
<!-- How the change is undone if it fails in the target environment. -->
|
||||
|
||||
## Gaps
|
||||
<!-- Any declared-target boundary that cannot be exercised, and the weaker check
|
||||
that stands in for it (if any). -->
|
||||
-
|
||||
|
||||
## Verdict
|
||||
<!-- pass / conditional / blocked — with the evidence. Use the
|
||||
verification-methodology skill's verdict format when available. -->
|
||||
Reference in New Issue
Block a user