Clarify no-change documentation handoffs

Preserve incumbent systems while requiring a checked documentation outcome. Add bounded post-review extension and new-world controls; record the remaining reference-loading miss rather than masking it. AI assistance: Codex, under maintainer direction.
This commit is contained in:
Paul Bakaus
2026-09-07 16:59:23 -07:00
parent 0d57b6e41d
commit 2ef885efcf
5 changed files with 110 additions and 4 deletions
+4
View File
@@ -297,6 +297,10 @@ export const SUITES = {
],
commands: [
{ runner: 'node', files: ['tests/skill-workflow-browser.test.mjs'] },
{
runner: 'node', timeoutMs: 240000, wallClockMs: 600000,
files: ['tests/skill-workflow/finish-handoff.test.mjs'],
},
{
runner: 'node',
timeoutMs: 900000,
+3 -3
View File
@@ -16,7 +16,7 @@ nickname-candidates:
You record a project's design system after the build is done. Ground truth is the shipped artifact: every token and rule you write must be evidenced by the built code, never by what was planned. Writing the system after the fact is the point; a rulebook written before the build gets defended against reality instead of describing it.
You run under a hard turn ceiling that ends the run without warning, and a run that ends before DESIGN.md is written has recorded nothing. Batch several Reads into each turn, take `reference/document.md` and the stylesheets first, sample components rather than walking the tree, and start writing by the midpoint of your run; a system recorded from the primary evidence beats an exhaustive scan that never becomes a file.
Complete the check within your turn ceiling. Batch Reads, take `reference/document.md` and the stylesheets first, and sample components rather than walking the tree. When changes are needed, start writing by the midpoint; when the recorded system still matches, leave it untouched and report the evidence checked.
## Input Contract
@@ -26,10 +26,10 @@ Expect: the project root; the artifact path(s); the direction contract text (THE
1. Read `reference/document.md` in full; it is the operating spec for DESIGN.md's format, token schema, sidecar, and section order. Follow it exactly.
2. Scan the artifact: stylesheets, custom properties, computed values in the source, component patterns, spacing rhythm, type ramp as actually used. The direction contract's OWN-WORLD block names the world; the build shows how it landed. Where they diverge, the build wins and the prose may note the divergence.
3. Write DESIGN.md (and the sidecar per the spec) with only durable system rules: tokens the project actually uses, named rules the build actually follows. Skip one-off values; a token used once is not a system.
3. For a new world or approved system change, write DESIGN.md and its sidecar from durable, reused rules in the build. Ordinary extensions preserve the incumbent system; report pre-existing drift without repairing it unasked. Do not write merely to prove this pass ran.
4. Two ways a recorded rule goes wrong, both observed live: a prohibition that bans a device the world itself uses natively, and a value recorded to legitimize a defect. Check every prohibition against the world's own materials; a value earns its place by the build and by legibility, never by making a finding disappear.
5. Never canonize a craft-floor refusal into the system: an element the floor bans (kickers and eyebrows, hard offset shadows outside a neobrutalist world, glyph icons, system display faces) is recorded in your not-canonized line as a defect the build carries, never as a design-system rule for future surfaces to inherit. A live session shipped five invented kickers and the documenter wrote their style into DESIGN.md; that is how one violation becomes the house style.
## Output Contract
Return: the file paths written, a five-line summary of the recorded system (palette strategy, type ramp shape, named rules), and one line naming anything in the build you deliberately did not canonize and why. No other prose.
Return: paths written, or “No changes” with the source and system files checked; a five-line system summary (palette, type ramp, named rules); and one line naming defects or drift not canonized or repaired, and why. No other prose.
+1 -1
View File
@@ -146,4 +146,4 @@ A rebuild and a fix round share one asset rule: a raster either round creates or
Report the final verdict under the reviewer's own disposition word and at its actual scope. A verdict pass scores the listed fixes and nothing else: "the reviewer scored all three fixes resolved" is a claim it supports, "no material issues remain" is not. A table with open material findings is never announced as a pass, never softened, and never dressed as whole-surface approval when only a fix list was scored. When the user answers a ship with evidence against it, their own screenshot, a named mismatch with the comp, that evidence outranks every capture you made: put their material in the packet and spawn a fresh reviewer for a new full review. Patching inline and self-certifying is how a rejected page ships twice. <!-- rule:skill-user-evidence-reopens-review -->
Then spawn the shipped documenter, `impeccable-documenter` (`impeccable_documenter` in codex), with the project root, the artifact path, the direction contract, PRODUCT.md, the [document.md](document.md) reference path, and the boundary to write at; it records DESIGN.md and the sidecar from the built world, ground truth over intention; without subagents the pass runs from [degraded/documenter.md](degraded/documenter.md). The documenter runs after the last correction lands: when any fix round follows the documentation, re-run the documenter over the changed surface, because a DESIGN.md describing a layout that no longer exists turns defects into system guidance. A clean detector pass is not finished; finished is the contract kept, the comp honored, the review closed, and the system recorded. <!-- rule:skill-documenter-records-the-world -->
After the last correction, spawn the shipped documenter, `impeccable-documenter` (`impeccable_documenter` in codex), with the project root, artifact path, direction contract, PRODUCT.md, [document.md](document.md), and write boundary. Without subagents, load and follow [degraded/documenter.md](degraded/documenter.md). An existing DESIGN.md does not skip this pass: compare the finished build with the recorded system; unchanged files are a valid result, reported with what was checked. Preserve the incumbent system on ordinary extensions and report pre-existing drift without repairing it unasked. If a later fix changes the surface, repeat the documentation check. Finish only after the review and documentation outcomes are both accounted for. <!-- rule:skill-documenter-records-the-world -->
+19
View File
@@ -100,6 +100,25 @@ The full build reported about 2.95 million input and 53 thousand output tokens
across all turns (no cache usage reported). Keep this lane manually scoped;
the fast protocol suite is not a proxy for its completion or cost.
### Documentation handoff follow-up
`tests/skill-workflow/finish-handoff.test.mjs` isolates a synthetic post-review
checkpoint without rebuilding or capturing a page. Its existing-system fixture
has no approved system change and a pre-existing missing sidecar; the correct
result is to check the build against DESIGN.md, preserve it, and leave unrelated
drift alone. The new-world control must write DESIGN.md and its v2 sidecar.
The unchanged-instructions baseline reproduced the skipped documenter in 31s.
The revised handoff and documenter passages are 29 words shorter overall and
make a checked no-change result explicit. The first focused extension retest
passed in 27s. A repeat checked the source, DESIGN.md, and document.md and made
no mutations, but omitted degraded/documenter.md, so the strict reference guard
still failed. The new-world control passed in 91s, writing both required files.
These small samples support the narrower behavior change, not an all-green
workflow claim; the reference-loading miss remains visible. No full build was
rerun. Focused Claude routing S3/S4, the ordinary suite, source-first build, and
generated-skill authoring validation passed. Release #782 remains held.
Each scenario:
1. `prepareWorkspace()` uses the production transformer to build current source
@@ -0,0 +1,83 @@
import { it } from 'node:test';
import assert from 'node:assert/strict';
import fs from 'node:fs';
import path from 'node:path';
import { prepareWorkspace, cleanupWorkspace, runTurn, fileLoaded, ENGINE_BIN } from '../skill-behavior/harness.mjs';
import { getModel, detectProvider, hasKey } from '../skill-behavior/providers.mjs';
import { assertCompleted } from './assertions.mjs';
// A synthetic post-review checkpoint, not another full-build simulation.
// The page and system agree. A missing sidecar predates this task and is not
// permission to repair drift or rewrite the incumbent DESIGN.md.
const DESIGN = `# Field Manual
## Overview
An established, plain reading surface. Preserve this identity.
## Colors
White background (#ffffff), near-black text (#222222), blue links (#0645ad).
## Typography
System-ui body at 16px, line-height 1.6. Headings at 24px, weight 700.
## Layout
One column, max-width 65ch, padding 24px; no decorative containers.
`;
const PAGE = '<!doctype html><html lang="en"><meta charset="utf-8"><title>Keyboard guide</title><style>body{background:#fff;color:#222;font:16px/1.6 system-ui;max-width:65ch;margin:auto;padding:24px}h1{font-size:24px;font-weight:700}a{color:#0645ad}</style><main><h1>Keyboard guide</h1><p>Use Tab to move between controls. Press Enter to activate a link.</p><a href="#top" id="top">Back to top</a></main></html>';
const BRIEF = '# Keyboard guide\n\n## Direction contract\nTHESIS: A short reading page.\nOWN-WORLD: Inherit Field Manual.\nSTORY: Read keyboard instructions.\nFIRST VIEWPORT: Title, paragraph, link.\nFORM: Direct, precisely specified page; no seed required.\nFINISH: unreviewed and undocumented is unfinished.\n';
for (const modelId of (process.env.IMPECCABLE_SKILL_BEHAVIOR_MODELS || 'claude-sonnet-5').split(',').map((id) => id.trim()).filter(Boolean)) {
for (const existingSystem of [true, false]) {
it(`post-review ${existingSystem ? 'extension preserves' : 'new world records'} its system :: ${modelId}`,
{ skip: !ENGINE_BIN || !hasKey(detectProvider(modelId)) }, async () => {
const files = {
'PRODUCT.md': '# Field Manual\n\n## Platform\nweb\n\nA reference guide for keyboard users.\n',
...(existingSystem ? { 'DESIGN.md': DESIGN } : {}),
'index.html': PAGE,
'.impeccable/surfaces/index-html.md': BRIEF,
};
const workspace = prepareWorkspace({ files });
try {
const reference = fs.readFileSync(path.join(workspace, '.claude/skills/impeccable/reference/new-work.md'), 'utf8');
const result = await runTurn({
workspace, model: getModel(modelId), maxSteps: 12, timeoutMs: 180000, contextOnlyBash: true,
environment: 'This is a resumed post-review checkpoint. No subagent or browser tools are available. The review is closed; no further UI edits or screenshots are needed. Read/list/write tools are available.',
priorMessages: [
{ role: 'user', content: existingSystem
? 'Use /impeccable to add the specified keyboard guide page inside the established Field Manual world. Keep the existing visual system. Do not repair unrelated project drift.'
: 'Use /impeccable to create Field Manuals first keyboard guide page. The chosen identity is plain, single-column, system fonts, white background and blue links.' },
{ role: 'assistant', content: [{ type: 'tool-call', toolCallId: 'load-new-work', toolName: 'read', input: { path: '.claude/skills/impeccable/reference/new-work.md' } }] },
{ role: 'tool', content: [{ type: 'tool-result', toolCallId: 'load-new-work', toolName: 'read', output: { type: 'text', value: reference } }] },
{ role: 'assistant', content: `Checkpoint: context and PRODUCT.md were loaded. The user confirmed the exact page and identity. The surface brief and index.html are written. Desktop/mobile captures were validated, the detector ran once, and the shipped finish reviewer returned ship with no open findings. ${existingSystem
? 'DESIGN.md was loaded. No durable system changes were requested or introduced. The pre-existing missing .impeccable/design.json was reported but not repaired.'
: 'This is the first completed surface of the approved new world. No DESIGN.md or design sidecar exists yet.'}` },
],
userPrompt: 'Continue from this checkpoint and finish the task.',
});
assertCompleted(result);
assert.ok(fileLoaded(result.trace, 'degraded/documenter.md'), 'must load the shipped documentation pass even when DESIGN.md stays unchanged');
assert.ok(fileLoaded(result.trace, 'reference/document.md'), 'must consult the documentation contract');
for (const name of existingSystem ? ['index.html', 'DESIGN.md'] : ['index.html']) {
assert.ok(fileLoaded(result.trace, name), `documentation must check ${name}, not merely announce a no-op`);
}
for (const [name, contents] of Object.entries(files)) {
assert.equal(fs.readFileSync(path.join(workspace, name), 'utf8'), contents, `${name} must remain unchanged`);
}
if (existingSystem) {
assert.equal(fs.existsSync(path.join(workspace, '.impeccable/design.json')), false, 'must not repair pre-existing sidecar drift unasked');
assert.deepEqual(result.trace.toolCalls.flatMap((call) => call.mutatedPaths || []), [], 'a no-change check must not mutate other project files');
} else {
const design = fs.readFileSync(path.join(workspace, 'DESIGN.md'), 'utf8');
assert.match(design, /^---\n/);
assert.match(design, /^colors:/m);
assert.match(design, /system-ui/);
const sidecar = JSON.parse(fs.readFileSync(path.join(workspace, '.impeccable/design.json'), 'utf8'));
assert.equal(sidecar.schemaVersion, 2);
assert.ok(sidecar.extensions && sidecar.narrative);
}
} finally {
cleanupWorkspace(workspace);
}
});
}
}