hooks: direction-contract audit in the Stop deep pass

The skill's decide-then-build step opens the built HTML artifact with a
DIRECTION CONTRACT comment (UNIQUE / NOT-TEMPLATE / OWN-WORLD / STORY /
FIRST VIEWPORT / FORM). Until now nothing ever judged the finished build
against that contract; the eval harness proved sample contracts promised
radical compositions while the build shipped the standard template anyway.

The Stop deep pass now extracts the leading contract comment from each
session-touched HTML file (marker match in the first 200 chars, body
capped at 1800 chars) and appends a contract-audit section after the
detector findings: audit the render promise by promise, naming the two
observed failure shapes (a promise not in the pixels; a contract whose
own plan is the standard template wearing the concept's nouns). Zero
extra API calls; the audit rides the existing single Stop emission and
fires at most once per file per session via a contractAudited flag on
the same session cache entry the finding dedupe uses.

Ported from the eval harness reference implementation
(extractDirectionContract / composeContractAuditMessage in
impeccable-evals runner/workers/anthropic-native.ts). No hooks.json
changes needed: Claude Code and Codex both already dispatch Stop to
hook.mjs.

Tests: 163 -> 179 in tests/hook.test.mjs (extraction unit coverage plus
Stop-pass integration: present/absent/once-per-session/non-HTML/
malformed/oversized). hook-build 18/18, build:skills prose gate clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Paul Bakaus
2026-07-14 08:58:19 -07:00
co-authored by Claude Fable 5
parent 4c7a3651d5
commit 313b641361
3 changed files with 346 additions and 7 deletions
+114 -6
View File
@@ -26,6 +26,7 @@
* loadDetector() -> Promise<{ detectText, detectHtml }>
* matchesAnyGlob(filePath, globs)
* normalizeScanTargets(primaryTargets, projectCwd)
* extractDirectionContract(content) / renderContractAudit(entries, opts)
* runHook(deps) -> { exitCode, stdout, audit, reason? }
* runStopHook(deps) -> { exitCode, stdout, audit, emission? }
*
@@ -1842,6 +1843,76 @@ export async function runHook({ stdinJson, env = {}, cwd = process.cwd(), now =
}
}
// ── Direction-contract audit ─────────────────────────────────────────────
// The skill's decide-then-build step opens the built HTML artifact with a
// DIRECTION CONTRACT comment (UNIQUE / NOT-TEMPLATE / OWN-WORLD / STORY /
// FIRST VIEWPORT / FORM blocks). At Stop time the deep pass extracts that
// comment and feeds it back so the model audits the render against its own
// promises. Proven in the eval harness: sample contracts promised radical
// compositions and the build shipped the standard template anyway, because
// nothing ever judged the build against the contract. Zero extra API calls:
// this is a message, not a judge. It fires at most once per file per session
// (a `contractAudited` flag on the session cache entry, the same state the
// deep pass uses for finding dedupe) and rides the Stop pass's existing
// emission rather than adding another block round.
// How far into the file to look for the leading comment. A contract lives at
// the very top of the artifact; anything deeper is not the contract.
export const CONTRACT_HEAD_CHARS = 6000;
// The contract/concept marker must appear early in the comment body, so a
// license header or unrelated note does not get mistaken for a contract.
export const CONTRACT_MARKER_CHARS = 200;
// Cap the extracted contract so a rambling comment cannot blow up the
// Stop message.
export const CONTRACT_MAX_CHARS = 1800;
// Cap contract sections per Stop emission so many touched artifacts cannot
// stack an unbounded message.
export const CONTRACT_AUDIT_MAX_FILES = 3;
/**
* Extract the artifact's own direction-contract comment: the first HTML
* comment in the head of the file, when its opening chars identify it as a
* contract/concept block. Returns the trimmed, length-capped body, or null
* when the file carries none (no comment, unclosed comment, marker missing,
* or the comment starts past the head window).
*/
export function extractDirectionContract(content) {
if (typeof content !== 'string' || !content) return null;
const head = content.slice(0, CONTRACT_HEAD_CHARS);
const m = /<!--([\s\S]*?)-->/.exec(head);
if (!m) return null;
const body = m[1].trim();
if (!body || !/contract|concept/i.test(body.slice(0, CONTRACT_MARKER_CHARS))) return null;
return body.slice(0, CONTRACT_MAX_CHARS);
}
/**
* Render the contract-audit section of the Stop message. `entries` is
* [{ filePath, contract }]; at most CONTRACT_AUDIT_MAX_FILES are shown.
* The two failure shapes named here are the ones observed in practice:
* a promise the pixels do not deliver, and a contract whose own plan is
* the standard template wearing the concept's nouns.
*/
export function renderContractAudit(entries, opts = {}) {
if (!Array.isArray(entries) || entries.length === 0) return '';
const cwd = opts.cwd || process.cwd();
const shown = entries.slice(0, CONTRACT_AUDIT_MAX_FILES);
const blocks = shown.map(({ filePath, contract }) => {
const display = relativize(filePath, cwd);
return `${display} opens with this direction contract, written when the direction was decided:\n\n${contract}`;
});
return [
`${ENVELOPE_PREFIX} Direction-contract audit. Before finishing, audit the rendered page against the contract it opens with, promise by promise.`,
'',
blocks.join('\n\n'),
'',
'Two failure shapes to check honestly:',
'1. A promise that is not in the pixels: the contract describes a composition or structure the built page does not deliver, because the build fell back to a standard arrangement, possibly one the contract explicitly rejects. Rebuild that part until the render matches the promise.',
"2. A contract whose own section plan is the standard template wearing the concept's nouns: if a neighboring product could ship the same sequence of sections under different labels, revise the plan and the page together.",
'State each promise and whether the render delivers it, and fix every gap before finishing.',
].join('\n');
}
// Cap on files the Stop deep pass will scan. The touched-file list is
// session-scoped and already capped per edit, but a very long session could
// accumulate more than the 30s hook timeout comfortably covers.
@@ -1850,7 +1921,10 @@ export const STOP_MAX_FILES = 20;
/**
* Run the Stop-event deep pass: the FULL detector rule set over every UI
* file touched this session, surfaced once, deduped against everything the
* per-edit hook already reported. Same result contract as runHook():
* per-edit hook already reported. Touched HTML artifacts that open with a
* direction-contract comment additionally get a one-time contract-audit
* section appended after the detector findings (see extractDirectionContract
* above). Same result contract as runHook():
* { exitCode, stdout, audit, emission? }
*
* Never throws; exits silent (and fast) when the session touched no UI
@@ -1917,6 +1991,7 @@ export async function runStopHook({ stdinJson, env = {}, cwd = process.cwd(), no
const scanOptions = designSystemOptions(config, det, projectCwd);
const freshGroups = [];
const contractEntries = [];
let scanned = 0;
for (const filePath of touched) {
if (scanned >= STOP_MAX_FILES) break;
@@ -1937,6 +2012,22 @@ export async function runStopHook({ stdinJson, env = {}, cwd = process.cwd(), no
const useHtmlEngine = configuredExt
? configuredExt.engine === 'html'
: (ext === '.html' || ext === '.htm');
// Direction-contract audit: HTML artifacts only, at most once per file
// per session. The flag lives on the same session cache entry the
// finding dedupe uses, so a second Stop fire stays quiet about it.
if (useHtmlEngine) {
const fileEntry = ensureFile(cache, sessionId, filePath);
if (!fileEntry.contractAudited) {
const contract = extractDirectionContract(content);
if (contract) {
fileEntry.contractAudited = true;
ensureSession(cache, sessionId).updatedAt = Date.now();
contractEntries.push({ filePath, contract });
}
}
}
if (useHtmlEngine && typeof det.detectHtml === 'function') {
try { findings = await det.detectHtml(filePath, scanOptions); } catch { findings = []; }
} else {
@@ -1955,24 +2046,41 @@ export async function runStopHook({ stdinJson, env = {}, cwd = process.cwd(), no
}
audit.scannedFiles = scanned;
if (freshGroups.length === 0) {
if (freshGroups.length === 0 && contractEntries.length === 0) {
return result({ emitted: false, skipped: 'stop-clean', durationMs: Date.now() - started });
}
// Fresh findings earn the cache write; they also mark this batch as
// surfaced so the next Stop fire is silent unless new issues appear.
// Fresh findings and first-time contract audits earn the cache write;
// both mark this batch as surfaced so the next Stop fire is silent
// unless new issues appear.
persistCache(projectCwd, cache);
const text = appendDesignSystemNote(renderGroupedTemplate(freshGroups, config, { cwd: projectCwd }), scanOptions);
// Detector findings first, then the contract audit. Both ride the same
// single Stop emission: the audit never adds an extra block round.
const parts = [];
if (freshGroups.length > 0) {
parts.push(renderGroupedTemplate(freshGroups, config, { cwd: projectCwd }));
}
if (contractEntries.length > 0) {
parts.push(renderContractAudit(contractEntries, { cwd: projectCwd }));
}
const text = appendDesignSystemNote(parts.join('\n\n'), scanOptions);
return {
exitCode: 0,
stdout: payload(text, 'Stop', harness),
emission: { kind: 'stop-deep-pass', groups: freshGroups },
emission: {
kind: 'stop-deep-pass',
groups: freshGroups,
...(contractEntries.length > 0
? { contractFiles: contractEntries.map((entry) => entry.filePath) }
: {}),
},
audit: {
...audit,
emitted: true,
freshFiles: freshGroups.length,
freshFindings: freshGroups.reduce((sum, group) => sum + group.findings.length, 0),
...(contractEntries.length > 0 ? { contractAudits: contractEntries.length } : {}),
chars: text.length,
durationMs: Date.now() - started,
},
+3 -1
View File
@@ -10,7 +10,9 @@
* `hookSpecificOutput.additionalContext` when findings exist.
* - Stop: runs the FULL detector rule set over every UI file touched this
* session (the deep pass), deduped against what the per-edit pass already
* surfaced, and emits once via the Stop additionalContext channel.
* surfaced, and emits once via the Stop additionalContext channel. Touched
* HTML artifacts opening with a direction-contract comment get a one-time
* contract-audit section appended to the same emission.
*
* Contract: never break a turn. Always exit 0. Clean files emit a small ack
* unless quiet mode is enabled; a clean Stop pass is silent.
+229
View File
@@ -53,6 +53,11 @@ import {
IMMEDIATE_TIER_RULES,
splitFindingsByTier,
perEditTieringActive,
extractDirectionContract,
renderContractAudit,
CONTRACT_MAX_CHARS,
CONTRACT_HEAD_CHARS,
CONTRACT_AUDIT_MAX_FILES,
payload,
extractFindingIgnoreValue,
resolveProjectPlatform,
@@ -2773,3 +2778,227 @@ describe('runStopHook()', () => {
assert.equal(reentrant.stdout, '');
});
});
describe('extractDirectionContract()', () => {
const contractComment = [
'<!--',
'DIRECTION CONTRACT',
'UNIQUE: the page is a boarding pass, read top to bottom like a gate agent would.',
'NOT-TEMPLATE: no centered hero, no three-card feature row.',
'-->',
].join('\n');
it('extracts the leading contract comment body', () => {
const html = `${contractComment}\n<!doctype html><html><body>hi</body></html>`;
const body = extractDirectionContract(html);
assert.ok(body);
assert.match(body, /^DIRECTION CONTRACT/);
assert.match(body, /boarding pass/);
assert.doesNotMatch(body, /<!--|-->/);
});
it('finds the contract even after a doctype line', () => {
const html = `<!doctype html>\n${contractComment}\n<html></html>`;
assert.match(extractDirectionContract(html) || '', /boarding pass/);
});
it('returns null when there is no comment at all', () => {
assert.equal(extractDirectionContract('<!doctype html><html></html>'), null);
assert.equal(extractDirectionContract(''), null);
assert.equal(extractDirectionContract(null), null);
});
it('returns null when the first comment is not a contract', () => {
const html = '<!-- Copyright 2026 Example Corp. All rights reserved. -->\n<html></html>';
assert.equal(extractDirectionContract(html), null);
});
it('requires the marker word inside the first 200 chars of the comment', () => {
const padding = 'x'.repeat(250);
const html = `<!-- ${padding} contract -->\n<html></html>`;
assert.equal(extractDirectionContract(html), null);
});
it('caps the extracted body at CONTRACT_MAX_CHARS', () => {
const longBody = `DIRECTION CONTRACT\n${'promise '.repeat(600)}END-MARKER`;
const html = `<!--${longBody}-->\n<html></html>`;
const body = extractDirectionContract(html);
assert.ok(body);
assert.equal(body.length, CONTRACT_MAX_CHARS);
assert.doesNotMatch(body, /END-MARKER/);
});
it('returns null for an unclosed (malformed) comment', () => {
const html = '<!-- DIRECTION CONTRACT: never closed\n<html><body></body></html>';
assert.equal(extractDirectionContract(html), null);
});
it('returns null when the comment starts past the head window', () => {
const html = `${' '.repeat(CONTRACT_HEAD_CHARS)}${contractComment}<html></html>`;
assert.equal(extractDirectionContract(html), null);
});
it('renderContractAudit shows at most CONTRACT_AUDIT_MAX_FILES sections', () => {
const entries = Array.from({ length: CONTRACT_AUDIT_MAX_FILES + 2 }, (_, i) => ({
filePath: `/tmp/page-${i}.html`,
contract: `DIRECTION CONTRACT ${i}`,
}));
const text = renderContractAudit(entries, { cwd: '/tmp' });
assert.match(text, new RegExp(`DIRECTION CONTRACT ${CONTRACT_AUDIT_MAX_FILES - 1}`));
assert.doesNotMatch(text, new RegExp(`DIRECTION CONTRACT ${CONTRACT_AUDIT_MAX_FILES}\\b`));
assert.equal(renderContractAudit([], {}), '');
});
});
describe('runStopHook() — direction-contract audit', () => {
let cwd;
beforeEach(() => { cwd = mkTmp(); });
afterEach(() => fs.rmSync(cwd, { recursive: true, force: true }));
const CONTRACT_HTML = [
'<!--',
'DIRECTION CONTRACT',
'UNIQUE: the page is a boarding pass.',
'FIRST VIEWPORT: gate number dominates.',
'-->',
'<!doctype html><html><body><h1>Gate 12</h1></body></html>',
].join('\n');
function write(rel, body) {
const abs = path.join(cwd, rel);
fs.mkdirSync(path.dirname(abs), { recursive: true });
fs.writeFileSync(abs, body);
return abs;
}
function editEvent(file, sessionId) {
return {
session_id: sessionId,
cwd,
hook_event_name: 'PostToolUse',
tool_name: 'Edit',
tool_input: { file_path: file },
};
}
function stopEvent(sessionId) {
return {
session_id: sessionId,
cwd,
hook_event_name: 'Stop',
stop_hook_active: false,
};
}
// Seed the session's touched-file list. `.impeccable/` must exist so a
// clean per-edit pass still persists the cache (see runHook write gating).
async function touch(file, sid, det) {
fs.mkdirSync(path.join(cwd, '.impeccable'), { recursive: true });
await runHook({ stdinJson: JSON.stringify(editEvent(file, sid)), env: {}, cwd, detector: det });
}
it('appends the contract audit after the detector findings', async () => {
const sid = 'contract-with-findings';
const file = write('index.html', CONTRACT_HTML);
const det = fakeDetector([finding('em-dash-overuse', 3)]);
await touch(file, sid, det);
const stop = await runStopHook({ stdinJson: JSON.stringify(stopEvent(sid)), env: {}, cwd, detector: det });
assert.equal(stop.audit.emitted, true);
assert.equal(stop.audit.contractAudits, 1);
const text = JSON.parse(stop.stdout).hookSpecificOutput.additionalContext;
assert.match(text, /em-dash-overuse/);
assert.match(text, /Direction-contract audit/);
assert.match(text, /boarding pass/);
assert.match(text, /not in the pixels/);
assert.ok(
text.indexOf('em-dash-overuse') < text.indexOf('Direction-contract audit'),
'detector findings come before the contract audit',
);
assert.deepEqual(stop.emission.contractFiles, [file]);
// Included exactly once.
assert.equal(text.match(/Direction-contract audit/g).length, 1);
});
it('emits the audit even when the detector deep pass is clean', async () => {
const sid = 'contract-clean-detector';
const file = write('index.html', CONTRACT_HTML);
const det = fakeDetector([]);
await touch(file, sid, det);
const stop = await runStopHook({ stdinJson: JSON.stringify(stopEvent(sid)), env: {}, cwd, detector: det });
assert.equal(stop.audit.emitted, true);
const out = JSON.parse(stop.stdout);
assert.equal(out.hookSpecificOutput.hookEventName, 'Stop');
assert.match(out.hookSpecificOutput.additionalContext, /Direction-contract audit/);
assert.doesNotMatch(out.hookSpecificOutput.additionalContext, /findings requiring review/);
});
it('adds no audit section when the HTML has no contract comment', async () => {
const sid = 'no-contract';
const file = write('index.html', '<!doctype html><html><body>plain</body></html>');
const det = fakeDetector([finding('em-dash-overuse', 3)]);
await touch(file, sid, det);
const stop = await runStopHook({ stdinJson: JSON.stringify(stopEvent(sid)), env: {}, cwd, detector: det });
const text = JSON.parse(stop.stdout).hookSpecificOutput.additionalContext;
assert.match(text, /em-dash-overuse/);
assert.doesNotMatch(text, /Direction-contract audit/);
assert.equal(stop.audit.contractAudits, undefined);
assert.equal(stop.emission.contractFiles, undefined);
});
it('a second Stop never repeats the audit, even with fresh detector findings', async () => {
const sid = 'contract-once';
const file = write('index.html', CONTRACT_HTML);
await touch(file, sid, fakeDetector([]));
const first = await runStopHook({ stdinJson: JSON.stringify(stopEvent(sid)), env: {}, cwd, detector: fakeDetector([]) });
assert.match(first.stdout, /Direction-contract audit/);
// No new findings: fully silent.
const second = await runStopHook({ stdinJson: JSON.stringify(stopEvent(sid)), env: {}, cwd, detector: fakeDetector([]) });
assert.equal(second.stdout, '');
assert.equal(second.audit.skipped, 'stop-clean');
// New findings ride a fresh emission, but the audit does not come back.
const third = await runStopHook({ stdinJson: JSON.stringify(stopEvent(sid)), env: {}, cwd, detector: fakeDetector([finding('side-tab', 7)]) });
const text = JSON.parse(third.stdout).hookSpecificOutput.additionalContext;
assert.match(text, /side-tab/);
assert.doesNotMatch(text, /Direction-contract audit/);
});
it('ignores contract-looking comments in non-HTML touched files', async () => {
const sid = 'contract-non-html';
const file = write('src/Card.tsx', '{/* stub */}\n// <!-- DIRECTION CONTRACT: not an artifact -->\nexport default 1;\n');
const det = fakeDetector([finding('em-dash-overuse', 3)]);
await touch(file, sid, det);
const stop = await runStopHook({ stdinJson: JSON.stringify(stopEvent(sid)), env: {}, cwd, detector: det });
const text = JSON.parse(stop.stdout).hookSpecificOutput.additionalContext;
assert.match(text, /em-dash-overuse/);
assert.doesNotMatch(text, /Direction-contract audit/);
});
it('stays silent for a malformed (unclosed) contract comment on a clean pass', async () => {
const sid = 'contract-malformed';
const file = write('index.html', '<!-- DIRECTION CONTRACT never closed\n<html><body></body></html>');
await touch(file, sid, fakeDetector([]));
const stop = await runStopHook({ stdinJson: JSON.stringify(stopEvent(sid)), env: {}, cwd, detector: fakeDetector([]) });
assert.equal(stop.stdout, '');
assert.equal(stop.audit.skipped, 'stop-clean');
});
it('truncates an oversized contract in the emitted message', async () => {
const sid = 'contract-huge';
const huge = `<!--\nDIRECTION CONTRACT\n${'promise '.repeat(600)}TAIL-MARKER\n-->\n<html></html>`;
const file = write('index.html', huge);
await touch(file, sid, fakeDetector([]));
const stop = await runStopHook({ stdinJson: JSON.stringify(stopEvent(sid)), env: {}, cwd, detector: fakeDetector([]) });
const text = JSON.parse(stop.stdout).hookSpecificOutput.additionalContext;
assert.match(text, /Direction-contract audit/);
assert.doesNotMatch(text, /TAIL-MARKER/);
});
});