mirror of
https://github.com/pbakaus/impeccable.git
synced 2026-09-12 06:06:37 +03:00
Add automatic design hook install and exceptions (#170)
* docs: add PRD for design detector hook integration Plans a PostToolUse hook for Claude Code and Codex that runs the existing design detector after every relevant file write and feeds findings back to the agent as advisory system-reminder context. No implementation in this commit; covers UX, technical design, build pipeline changes, distribution, coverage tradeoffs, and rollout. Co-authored-by: Cursor <cursoragent@cursor.com> * docs: revise hook PRD with best-practices review Folds in the P0/P1/P2 findings from an online best-practices critique against the official Claude Code and Codex hook references plus 10+ 2026 community guides and similar prior-art tools (claw-hooks, claude-code-hooks-mastery). Key changes: - Exec form everywhere (Codex snippet was shell form), with Windows rationale. - Default timeout dropped from 10s to 5s. - Re-entrancy guard (CLAUDE_HOOK_DEPTH) and per-file edit counter. - Session-scoped finding dedup promoted from open question to v1. - Per-language inline-ignore syntax map (HTML/JSX/CSS/JS). - Hard-skip rules for sensitive paths and generated/lock files. - Honest framing about Claude Code lacking per-plugin hook disable. - Honest framing about Bash-written files being invisible in v1. - Codex Windows-not-supported call-out, feature flag note, trust ceremony detail. - Optional NDJSON audit log via IMPECCABLE_HOOK_LOG. - Findings cap lowered 8 → 5 with attention-budget rationale. - Versioned envelope ([impeccable@1]) on rendered template. - Expanded test plan, decision log, and stdin payload appendix. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(hooks): ship the design detector hook for Claude Code and Codex Implements docs/hooks-prd.md: a PostToolUse hook that runs the impeccable design detector after every Edit/Write/MultiEdit on a UI file and pushes findings into the agent's next-turn context as a short system reminder. Silent on clean files. Never blocks an edit. Why this matters: today, design slop (side-tab borders, gradient text, purple/cyan palettes, bounce easing, etc.) only gets caught when a human notices or someone explicitly runs /impeccable audit. The hook closes the loop at the moment slop is written. What ships in v1 - skill/scripts/hook.mjs: PostToolUse entry. Reads stdin, runs the detector in-process (no `npx impeccable` cold start), emits hookSpecificOutput.additionalContext when fresh findings exist. - skill/scripts/hook-lib.mjs: extracted helpers (config, cache, filter, render, audit log, runHook orchestrator). 100% unit-testable. - skill/scripts/hook-session-start.mjs: SessionStart greeting, gated by a project-scannable probe + 30-day throttle. - skill/scripts/hook-admin.mjs: backs /impeccable hooks on/off/status/ignore-rule/ignore-file/reset. Hardening built in - Re-entrancy guard (IMPECCABLE_HOOK_DEPTH) so the hook can never recursively spawn itself. - Hard-skip regexes for sensitive paths (.env, .pem, id_rsa, secrets, credentials, .git) and generated/lock/build output. These fire before the file is even read; cannot be turned off via config. - Path-traversal check on the inbound file_path. - Session-scoped dedup keyed by (session, file, rule, line) so the same finding never lands in context twice. Prevents the ~12.5K wasted tokens per chatty session called out in the PRD. - Per-(session, file) edit counter with a one-shot suppression notice on the 7th edit, silent after. - Fail-open contract: every error path returns exit 0 with no stdout. Optional NDJSON audit log via IMPECCABLE_HOOK_LOG. Three kill switches (precedence high to low): 1. IMPECCABLE_HOOK_DISABLED env var (1/true/yes/on, case-insensitive) 2. .impeccable/hook.json `enabled: false` 3. /impeccable hooks off slash command (writes the JSON) Inline ignores are language-aware. `// impeccable: ignore <rule>` for JS/TS, `<!-- impeccable: ignore <rule> -->` for HTML/Vue/Svelte/Astro, `{/* impeccable: ignore <rule> */}` for JSX/TSX, `/* impeccable: ignore <rule> */` for CSS. `*` matches any rule. Directive applies to the next non-blank line. Same shape as ESLint, Stylelint, Biome. Build pipeline - scripts/lib/transformers/hooks.js: per-provider hooks.json builders, plus the slim .codex-plugin/plugin.json manifest. - providers.js: emitHooks: 'claude' for claude-code, emitHooks: 'codex' for codex and agents. Codex also emits emitCodexPlugin. - factory.js: emits hooks/hooks.json next to the skills tree. - build.js: syncs hooks/ into harness roots and into the slim plugin/ subtree; writes .codex-plugin/plugin.json. Build is idempotent (verified: 98 staged files unchanged across two runs). Claude Code wiring uses exec form (command + args) and the ${CLAUDE_PLUGIN_ROOT} placeholder. Matcher: Edit|Write|MultiEdit. `if:` glob filters to UI extensions before spawning Node. PostToolUse timeout 5s, SessionStart timeout 3s. Codex wiring uses ${PLUGIN_ROOT} (Codex's native placeholder), matcher Edit|Write|apply_patch, no `if:` analog (the script does the extension filter). macOS and Linux only; hooks are disabled on Windows in current Codex builds. The trust ceremony and feature flag are documented in README.md. Routing - /impeccable hooks lives outside the 23-command router table on purpose: it is plumbing, not a design skill. The hidden routing slot is added to SKILL.md alongside pin/unpin so the LLM knows to dispatch it. The 23-command count and all stale-count validators remain happy. Tests - tests/hook.test.mjs: 38 unit tests covering env parsing, config load + defaults + malformed, cache round-trip + GC, ignoreRules/minSeverity/inline ignores (all four languages), globbing with **/*/{a,b}, render template with cap + clamp + 0-line prefix drop, audit log NDJSON, payload event-name parameterization, re-entrancy, kill switches, sensitive-path + generated-path + traversal skips, allowlist filter, config ignoreFiles, edit counter cycle including the 7th-edit notice, MultiEdit and apply_patch payload shapes, detector throw swallow, malformed stdin, missing file race. - tests/hook-build.test.mjs: 18 integration tests covering hook manifest shape (matcher, timeouts, exec form, if: glob, placeholders), Codex differences (${PLUGIN_ROOT}, no if:, no SessionStart), Codex plugin manifest (no inline hooks field to avoid the duplicate-file error), routing across the hooksJsonFor table, and presence of all three committed artifacts plus the bundled detector the runtime relative-import path depends on. Full suite: 175 bun tests + 186 node tests, all green. Docs - README.md: new "Design hook" section explaining default behavior, per-project / global / inline disable paths, the JSON schema knobs, the audit log debug flag, and the slop / a11y coverage split. - HARNESSES.md: flips the `hooks` row for Codex from No -> Yes (Claude was already Yes), adds a per-harness hook-surface table with the manifest location and matcher each provider uses. Open questions from the PRD intentionally deferred to v2: Bash-write blind spot, effort-aware suppression, Stop-hook session summary, per-rule severity, async hook mode. None block v1. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix Codex hook scanning: apply_patch paths and co-located stylesheets Parse file targets from Codex apply_patch command bodies, co-scan imported and sibling CSS when UI components are edited, drop the git-sweep PostToolUse group, and align Codex SessionStart manifest and trust docs with the official hooks spec. Co-authored-by: Cursor <cursoragent@cursor.com> * Gitignore hook session cache and drop local test HTML Hook dedup/throttle state in .impeccable/hook.cache.json is per-project runtime data like other .impeccable/ sidecars. Remove an untracked bad-nested-flexbox scratch page from site/public/. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix Claude Code hook: drop Edit-only if filter so Write/MultiEdit fire Claude's if permission rule binds to one tool name, so Edit(*.{…}) never spawned the hook on Write or MultiEdit despite the matcher listing them. Extension filtering now lives in hook-lib on both Claude and Codex. Co-authored-by: Cursor <cursoragent@cursor.com> * Surface Cursor design findings via stop-hook followup Replace dropped postToolUse additional_context with afterFileEdit recording and a one-shot stop followup_message so anti-pattern nudges reach the agent. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix design hook packaging and scans * Fix Cursor hook pending bucket fallback * Fix Sass hook scan coverage * Fix Cursor hook review findings * Fix session start dead hook normalization * Fix hook config and relative scan paths * Remove SessionStart design hook * Remove redundant afterFileEdit normalization * Fix Cursor suppression and module style scans * Fix sensitive path hook filter * Fix disabled Cursor stop hook emission * Refresh hook harness artifacts * Fix Cursor hook manifest install * Add hook ignore-value support * Ignore hook runtime files locally * Fix Codex plugin hook packaging * fix: address PR review bot findings Block numeric hook depth counters from re-entering. Avoid following stylesheet imports from traversal-looking hook targets. * fix: gate ignore-value suggestions by supported rules Only render exact ignore-value commands when the same finding can be suppressed by ignoreValues. * Package Codex plugin as hook-only * Remove Codex plugin packaging * Recover hook install probe plumbing * Remove Codex hook packaging follow-up doc * Remove extra hook docs and skill wording changes * Install real design hooks via skills CLI * Add provider hook smoke runner * Fix Cursor hook delivery with preToolUse gate * Simplify Cursor hook install to preToolUse * Clarify confirmed hook exceptions * Persist hook ignores in shared config * Guard font hook exceptions * Fix hook install after main rebase * Fix hook scan target handling * fix: address hook review findings * Address hook review feedback * Stabilize DeepSeek insert live fixture * Fix Cursor hook Python shell write bypass --------- Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -21,6 +21,7 @@ import { fileURLToPath } from 'url';
|
||||
import { readSourceFiles, readPatterns, stashPerProjectArtifacts, restorePerProjectArtifacts } from './lib/utils.js';
|
||||
import { generateApiData } from './lib/api-data.js';
|
||||
import { createTransformer, PROVIDERS } from './lib/transformers/index.js';
|
||||
import { hooksJsonFor } from './lib/transformers/hooks.js';
|
||||
import { createAllZips } from './lib/zip.js';
|
||||
import { ANTIPATTERNS } from '../cli/engine/registry/antipatterns.mjs';
|
||||
// Sub-page generation is now handled by Astro content collections.
|
||||
@@ -403,6 +404,21 @@ function copyDirSync(src, dest) {
|
||||
}
|
||||
}
|
||||
|
||||
function syncRootHookManifests(rootDir) {
|
||||
const synced = [];
|
||||
for (const config of Object.values(PROVIDERS)) {
|
||||
if (!config.emitHooks) continue;
|
||||
const manifest = hooksJsonFor(config.emitHooks);
|
||||
if (!manifest) continue;
|
||||
const rel = config.hooksManifestRel || path.join('hooks', 'hooks.json');
|
||||
const dest = path.join(rootDir, config.configDir, rel);
|
||||
fs.mkdirSync(path.dirname(dest), { recursive: true });
|
||||
fs.writeFileSync(dest, JSON.stringify(manifest, null, 2) + '\n');
|
||||
synced.push(path.join(config.configDir, rel).split(path.sep).join('/'));
|
||||
}
|
||||
return synced;
|
||||
}
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
const __dirname = path.dirname(__filename);
|
||||
const ROOT_DIR = path.resolve(__dirname, '..');
|
||||
@@ -657,6 +673,29 @@ async function build() {
|
||||
}
|
||||
}
|
||||
|
||||
const syncedHooks = syncRootHookManifests(ROOT_DIR);
|
||||
if (syncedHooks.length > 0) {
|
||||
console.log(`🪝 Synced hook manifests to: ${syncedHooks.join(', ')}`);
|
||||
}
|
||||
|
||||
// Remove deprecated skill stubs from local harness dirs. They exist
|
||||
// in dist/ so the cleanup script can redirect users, but they should
|
||||
// not clutter the repo's own skill directories.
|
||||
const deprecatedLocalSkills = [
|
||||
'frontend-design', 'teach-impeccable',
|
||||
'arrange', 'normalize', 'onboard', 'extract',
|
||||
// v3.0 consolidation: standalone skills -> /impeccable sub-commands
|
||||
'adapt', 'animate', 'audit', 'bolder', 'clarify', 'colorize',
|
||||
'critique', 'delight', 'distill', 'harden', 'layout', 'optimize',
|
||||
'overdrive', 'polish', 'quieter', 'shape', 'typeset',
|
||||
];
|
||||
for (const { configDir } of syncConfigs) {
|
||||
for (const name of deprecatedLocalSkills) {
|
||||
const p = path.join(ROOT_DIR, configDir, 'skills', name);
|
||||
if (fs.existsSync(p)) fs.rmSync(p, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`📋 Synced skills to: ${syncConfigs.map(p => p.configDir).join(', ')}`);
|
||||
|
||||
// Build the Claude Code plugin subtree at ./plugin/.
|
||||
|
||||
@@ -10,6 +10,7 @@ import {
|
||||
stripRuleMarkers,
|
||||
} from '../utils.js';
|
||||
import { SKILL_CATEGORIES, CATEGORY_ORDER } from '../sub-pages-data.js';
|
||||
import { hooksJsonFor } from './hooks.js';
|
||||
|
||||
/**
|
||||
* Map from frontmatter field name to extraction spec.
|
||||
@@ -300,10 +301,26 @@ export function createTransformer(config) {
|
||||
}
|
||||
}
|
||||
|
||||
// Emit the provider hook manifest when the provider opts in.
|
||||
// Claude Code uses `.claude/settings.json`, Codex uses project-local
|
||||
// `.codex/hooks.json`, and Cursor uses `.cursor/hooks.json`.
|
||||
let hooksEmitted = false;
|
||||
if (config.emitHooks) {
|
||||
const manifest = hooksJsonFor(config.emitHooks);
|
||||
if (manifest) {
|
||||
const hooksRel = config.hooksManifestRel || path.join('hooks', 'hooks.json');
|
||||
writeFile(path.join(providerDir, configDir, hooksRel), JSON.stringify(manifest, null, 2) + '\n');
|
||||
hooksEmitted = true;
|
||||
}
|
||||
}
|
||||
|
||||
const skillWord = skills.length === 1 ? 'skill' : 'skills';
|
||||
const refInfo = refCount > 0 ? ` (${refCount} reference files)` : '';
|
||||
const scriptInfo = scriptCount > 0 ? ` (${scriptCount} script files)` : '';
|
||||
const agentInfo = agentCount > 0 ? ` (${agentCount} agent files)` : '';
|
||||
console.log(`✓ ${displayName}: ${skills.length} ${skillWord}${refInfo}${scriptInfo}${agentInfo}`);
|
||||
const hooksInfo = hooksEmitted
|
||||
? ` (${config.hooksManifestRel || path.join('hooks', 'hooks.json')})`
|
||||
: '';
|
||||
console.log(`✓ ${displayName}: ${skills.length} ${skillWord}${refInfo}${scriptInfo}${agentInfo}${hooksInfo}`);
|
||||
};
|
||||
}
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
/**
|
||||
* Build-pipeline emitters for the Impeccable design hook.
|
||||
*
|
||||
* The hook install path in this PR is project-local:
|
||||
* - Claude Code: `.claude/settings.json`
|
||||
* - Codex: `.codex/hooks.json`
|
||||
* - Cursor: `.cursor/hooks.json`
|
||||
*
|
||||
* No provider marketplace or Codex plugin packaging is emitted here.
|
||||
*/
|
||||
|
||||
export const IMPECCABLE_HOOK_COMMAND_MARKER = 'skills/impeccable/scripts/hook.mjs';
|
||||
|
||||
const TIMEOUT_SECONDS = 5;
|
||||
const CLAUDE_PROJECT_HOOK = '${CLAUDE_PROJECT_DIR}/.claude/skills/impeccable/scripts/hook.mjs';
|
||||
const CODEX_PROJECT_HOOK = '$(git rev-parse --show-toplevel)/.agents/skills/impeccable/scripts/hook.mjs';
|
||||
const CURSOR_BEFORE_EDIT_SCRIPT = '.cursor/skills/impeccable/scripts/hook-before-edit.mjs';
|
||||
|
||||
export function buildClaudeSettingsManifest() {
|
||||
return {
|
||||
description: 'Impeccable design detector: runs after Edit/Write/MultiEdit on UI files and surfaces findings as system reminders.',
|
||||
hooks: {
|
||||
PostToolUse: [
|
||||
{
|
||||
matcher: 'Edit|Write|MultiEdit',
|
||||
hooks: [
|
||||
{
|
||||
type: 'command',
|
||||
command: `node "${CLAUDE_PROJECT_HOOK}"`,
|
||||
timeout: TIMEOUT_SECONDS,
|
||||
statusMessage: 'Scanning design',
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export function buildCodexHooksManifest() {
|
||||
return {
|
||||
description: 'Impeccable design detector: runs after Edit/Write/apply_patch on UI files and surfaces findings as system reminders.',
|
||||
hooks: {
|
||||
PostToolUse: [
|
||||
{
|
||||
matcher: 'Edit|Write|apply_patch',
|
||||
hooks: [
|
||||
{
|
||||
type: 'command',
|
||||
command: `node "${CODEX_PROJECT_HOOK}"`,
|
||||
timeout: TIMEOUT_SECONDS,
|
||||
statusMessage: 'Scanning design',
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export function buildCursorHooksManifest() {
|
||||
return {
|
||||
version: 1,
|
||||
hooks: {
|
||||
preToolUse: [
|
||||
{
|
||||
command: `node "${CURSOR_BEFORE_EDIT_SCRIPT}"`,
|
||||
timeout: TIMEOUT_SECONDS,
|
||||
},
|
||||
],
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export function hooksJsonFor(provider) {
|
||||
switch (provider) {
|
||||
case 'claude':
|
||||
return buildClaudeSettingsManifest();
|
||||
case 'codex':
|
||||
return buildCodexHooksManifest();
|
||||
case 'cursor':
|
||||
return buildCursorHooksManifest();
|
||||
default:
|
||||
return null;
|
||||
}
|
||||
}
|
||||
@@ -16,6 +16,9 @@ export const PROVIDERS = {
|
||||
configDir: '.cursor',
|
||||
displayName: 'Cursor',
|
||||
frontmatterFields: ['license', 'compatibility', 'metadata'],
|
||||
emitHooks: 'cursor',
|
||||
// Cursor reads `.cursor/hooks.json`, not `.cursor/hooks/hooks.json`.
|
||||
hooksManifestRel: 'hooks.json',
|
||||
},
|
||||
'claude-code': {
|
||||
provider: 'claude-code',
|
||||
@@ -24,6 +27,9 @@ export const PROVIDERS = {
|
||||
displayName: 'Claude Code',
|
||||
frontmatterFields: ['user-invocable', 'argument-hint', 'license', 'compatibility', 'metadata', 'allowed-tools'],
|
||||
agentFormat: 'claude-md',
|
||||
emitHooks: 'claude',
|
||||
// Project-local Claude Code hooks live in `.claude/settings.json`.
|
||||
hooksManifestRel: 'settings.json',
|
||||
},
|
||||
gemini: {
|
||||
provider: 'gemini',
|
||||
@@ -43,6 +49,9 @@ export const PROVIDERS = {
|
||||
// No agentFormat: the Codex subagent ships nested inside the skill's own
|
||||
// agents/ folder (see CODEX_SKILL_PROVIDERS in factory.js), which Codex
|
||||
// auto-discovers on install. No top-level .codex/agents/ sidecar is emitted.
|
||||
emitHooks: 'codex',
|
||||
// Codex discovers project-local hooks at `.codex/hooks.json`.
|
||||
hooksManifestRel: 'hooks.json',
|
||||
},
|
||||
agents: {
|
||||
provider: 'agents',
|
||||
|
||||
+4
-2
@@ -8,7 +8,9 @@
|
||||
|
||||
import path from 'path';
|
||||
import { createWriteStream, existsSync, statSync } from 'fs';
|
||||
import { ZipArchive } from 'archiver';
|
||||
import * as archiverModule from 'archiver';
|
||||
|
||||
const createArchiver = archiverModule.default || archiverModule.create || archiverModule;
|
||||
|
||||
/**
|
||||
* Create ZIP file for a provider directory
|
||||
@@ -28,7 +30,7 @@ export async function createProviderZip(providerDir, distDir, providerName) {
|
||||
try {
|
||||
await new Promise((resolve, reject) => {
|
||||
const output = createWriteStream(zipPath);
|
||||
const archive = new ZipArchive({ zlib: { level: 9 } });
|
||||
const archive = createArchiver('zip', { zlib: { level: 9 } });
|
||||
|
||||
output.on('close', resolve);
|
||||
archive.on('error', reject);
|
||||
|
||||
@@ -0,0 +1,979 @@
|
||||
#!/usr/bin/env node
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import {
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
readFileSync,
|
||||
readdirSync,
|
||||
rmSync,
|
||||
writeFileSync,
|
||||
} from 'node:fs';
|
||||
import { homedir, tmpdir } from 'node:os';
|
||||
import { dirname, join, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const prRoot = resolve(__dirname, '..');
|
||||
const defaultBundle = join(prRoot, 'dist', 'universal.zip');
|
||||
const defaultProviders = ['direct', 'claude', 'codex', 'cursor'];
|
||||
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
if (args.help || args.h || !args.repo) {
|
||||
const usage = [
|
||||
'Usage: bun run smoke:hooks -- --repo <target-repo> [--bundle dist/universal.zip] [--providers direct,claude,codex,cursor]',
|
||||
'',
|
||||
'The target repo must be explicit so this local smoke does not depend on one contributor machine path.',
|
||||
].join('\n');
|
||||
if (args.help || args.h) {
|
||||
console.log(usage);
|
||||
process.exit(0);
|
||||
}
|
||||
console.error(usage);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const targetRepo = resolve(args.repo);
|
||||
const bundlePath = resolve(args.bundle || defaultBundle);
|
||||
const selectedProviders = (args.providers || defaultProviders.join(','))
|
||||
.split(',')
|
||||
.map((provider) => provider.trim().toLowerCase())
|
||||
.filter(Boolean);
|
||||
const smokeDir = join(targetRepo, '.impeccable', 'provider-smoke');
|
||||
const summaryPath = join(smokeDir, 'summary.json');
|
||||
const smokeFiles = {
|
||||
direct: 'src/__impeccable_provider_smoke_direct.html',
|
||||
claude: 'src/__impeccable_provider_smoke_claude.html',
|
||||
codex: 'src/__impeccable_provider_smoke_codex.html',
|
||||
cursor: 'src/__impeccable_provider_smoke_cursor.html',
|
||||
confirmedClaude: 'src/__impeccable_provider_smoke_confirmed_claude.html',
|
||||
confirmedCodex: 'src/__impeccable_provider_smoke_confirmed_codex.html',
|
||||
confirmedCursor: 'src/__impeccable_provider_smoke_confirmed_cursor.html',
|
||||
agentChoiceClaude: 'src/__impeccable_provider_smoke_font_choice_claude.html',
|
||||
agentChoiceCodex: 'src/__impeccable_provider_smoke_font_choice_codex.html',
|
||||
agentChoiceCursor: 'src/__impeccable_provider_smoke_font_choice_cursor.html',
|
||||
};
|
||||
|
||||
const results = [];
|
||||
|
||||
main().catch((error) => {
|
||||
if (!results.some((result) => !result.pass)) {
|
||||
record('fatal', false, String(error?.message || error), 'fatal');
|
||||
}
|
||||
writeSummary();
|
||||
console.error(error?.stack || error);
|
||||
process.exit(1);
|
||||
});
|
||||
|
||||
async function main() {
|
||||
assertPath(targetRepo, 'target repo');
|
||||
assertPath(bundlePath, 'universal bundle');
|
||||
mkdirSync(smokeDir, { recursive: true });
|
||||
ensureTargetGitExclude();
|
||||
|
||||
cleanSmokeArtifacts();
|
||||
await checked('fresh install/update', 'install shape', reinstallFresh);
|
||||
checked('install shape', 'install shape', verifyInstallShape);
|
||||
|
||||
if (selectedProviders.includes('direct')) checked('direct script contracts', 'direct script failed', runDirectContractChecks);
|
||||
if (selectedProviders.some((provider) => ['claude', 'codex', 'cursor'].includes(provider))) {
|
||||
checked('confirmed exception persistence', 'confirmed exception persistence failed', runConfirmedExceptionPersistenceChecks);
|
||||
checked('agent-chosen font exception', 'agent-chosen font exception failed', runAgentChosenFontExceptionChecks);
|
||||
}
|
||||
if (selectedProviders.includes('claude')) checked('claude provider', 'provider did not fire or did not surface output', runClaudeProviderSmoke);
|
||||
if (selectedProviders.includes('codex')) checked('codex provider', 'provider did not fire or did not surface output', runCodexProviderSmoke);
|
||||
if (selectedProviders.includes('cursor')) checked('cursor provider', 'provider did not fire or did not surface output', runCursorProviderSmoke);
|
||||
|
||||
cleanSmokeFiles();
|
||||
clearRuntimeState();
|
||||
writeSummary();
|
||||
|
||||
const failed = results.filter((result) => !result.pass);
|
||||
if (failed.length > 0) {
|
||||
console.error(`Provider smoke failed: ${failed.map((r) => r.name).join(', ')}`);
|
||||
process.exit(1);
|
||||
}
|
||||
console.log(`Provider smoke passed. Summary: ${summaryPath}`);
|
||||
}
|
||||
|
||||
async function checked(name, classification, fn) {
|
||||
try {
|
||||
return await fn();
|
||||
} catch (error) {
|
||||
record(name, false, String(error?.message || error), error?.classification || classification);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
function parseArgs(argv) {
|
||||
const out = {};
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const arg = argv[i];
|
||||
if (!arg.startsWith('--')) continue;
|
||||
const eq = arg.indexOf('=');
|
||||
if (eq !== -1) {
|
||||
out[arg.slice(2, eq)] = arg.slice(eq + 1);
|
||||
} else {
|
||||
out[arg.slice(2)] = argv[i + 1] && !argv[i + 1].startsWith('--') ? argv[++i] : 'true';
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function assertPath(path, label) {
|
||||
if (!existsSync(path)) throw new Error(`${label} does not exist: ${path}`);
|
||||
}
|
||||
|
||||
function record(name, pass, detail = '', classification = '') {
|
||||
const result = { name, pass, classification, detail };
|
||||
results.push(result);
|
||||
console.log(`${pass ? 'PASS' : 'FAIL'} ${name}${classification ? ` [${classification}]` : ''}${detail ? `: ${detail}` : ''}`);
|
||||
}
|
||||
|
||||
function writeSummary() {
|
||||
mkdirSync(smokeDir, { recursive: true });
|
||||
writeFileSync(summaryPath, `${JSON.stringify({
|
||||
targetRepo,
|
||||
bundlePath,
|
||||
providers: selectedProviders,
|
||||
results,
|
||||
}, null, 2)}\n`);
|
||||
}
|
||||
|
||||
function run(cmd, cmdArgs, {
|
||||
cwd = targetRepo,
|
||||
env = {},
|
||||
input = undefined,
|
||||
logName,
|
||||
timeoutMs = 10 * 60 * 1000,
|
||||
allowFailure = false,
|
||||
} = {}) {
|
||||
const fallbackPath = '/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Codex.app/Contents/Resources';
|
||||
const inheritedPath = process.env.PATH || fallbackPath;
|
||||
const fullEnv = {
|
||||
...process.env,
|
||||
PATH: `${join(homedir(), '.local', 'bin')}:${inheritedPath}:${fallbackPath}`,
|
||||
...env,
|
||||
};
|
||||
const res = spawnSync(cmd, cmdArgs, {
|
||||
cwd,
|
||||
env: fullEnv,
|
||||
input,
|
||||
encoding: 'utf8',
|
||||
maxBuffer: 50 * 1024 * 1024,
|
||||
timeout: timeoutMs,
|
||||
});
|
||||
const output = [
|
||||
`$ ${cmd} ${cmdArgs.map(shellQuote).join(' ')}`,
|
||||
`exit=${res.status ?? 'null'} signal=${res.signal ?? ''}`,
|
||||
'--- stdout ---',
|
||||
res.stdout || '',
|
||||
'--- stderr ---',
|
||||
res.stderr || '',
|
||||
res.error ? `--- error ---\n${res.error.stack || res.error.message || res.error}` : '',
|
||||
].join('\n');
|
||||
if (logName) writeFileSync(join(smokeDir, logName), output);
|
||||
if (!allowFailure && (res.error || res.status !== 0)) {
|
||||
const message = res.error
|
||||
? `${cmd} failed: ${res.error.message}`
|
||||
: `${cmd} exited ${res.status}`;
|
||||
throw Object.assign(new Error(message), { output, status: res.status });
|
||||
}
|
||||
return { ...res, output };
|
||||
}
|
||||
|
||||
function shellQuote(value) {
|
||||
const s = String(value);
|
||||
return /^[A-Za-z0-9_/:=.,@%+-]+$/.test(s) ? s : JSON.stringify(s);
|
||||
}
|
||||
|
||||
async function reinstallFresh() {
|
||||
cleanInstalledImpeccable();
|
||||
const packDir = makeTempDir('impeccable-provider-smoke-pack-');
|
||||
const pack = run('npm', ['pack', '--pack-destination', packDir], {
|
||||
cwd: prRoot,
|
||||
logName: 'npm-pack.log',
|
||||
timeoutMs: 2 * 60 * 1000,
|
||||
});
|
||||
const packName = (pack.stdout || '').trim().split('\n').pop();
|
||||
const tarball = join(packDir, packName);
|
||||
assertPath(tarball, 'local npm tarball');
|
||||
|
||||
const env = { IMPECCABLE_BUNDLE_PATH: bundlePath };
|
||||
run('npx', ['--yes', '--package', tarball, 'impeccable', 'skills', 'install', '-y', '--force', '--providers=claude,cursor,codex'], {
|
||||
cwd: targetRepo,
|
||||
env,
|
||||
logName: 'skills-install.log',
|
||||
timeoutMs: 5 * 60 * 1000,
|
||||
});
|
||||
run('npx', ['--yes', '--package', tarball, 'impeccable', 'skills', 'update', '-y'], {
|
||||
cwd: targetRepo,
|
||||
env,
|
||||
logName: 'skills-update.log',
|
||||
timeoutMs: 5 * 60 * 1000,
|
||||
});
|
||||
record('fresh install/update', true, 'installed through local npx + IMPECCABLE_BUNDLE_PATH');
|
||||
}
|
||||
|
||||
function makeTempDir(prefix) {
|
||||
const dir = join(tmpdir(), `${prefix}${Date.now()}-${Math.random().toString(16).slice(2)}`);
|
||||
mkdirSync(dir, { recursive: true });
|
||||
return dir;
|
||||
}
|
||||
|
||||
function cleanInstalledImpeccable() {
|
||||
clearRuntimeState();
|
||||
cleanSmokeFiles();
|
||||
|
||||
for (const rel of [
|
||||
'.claude/skills/impeccable',
|
||||
'.cursor/skills/impeccable',
|
||||
'.agents/skills/impeccable',
|
||||
'.claude/hooks/hooks.json',
|
||||
'.agents/hooks',
|
||||
'.agents/plugins/marketplace.json',
|
||||
'.cursor/pre-log.mjs',
|
||||
'.cursor/rules/impeccable-design-hook.mdc',
|
||||
'plugin-codex',
|
||||
]) {
|
||||
rmSync(join(targetRepo, rel), { recursive: true, force: true });
|
||||
}
|
||||
|
||||
for (const rel of ['.claude/settings.json', '.cursor/hooks.json', '.codex/hooks.json']) {
|
||||
stripManifest(rel);
|
||||
}
|
||||
|
||||
run('claude', ['plugin', 'uninstall', 'impeccable@impeccable', '--scope', 'user'], {
|
||||
cwd: targetRepo,
|
||||
logName: 'claude-plugin-uninstall.log',
|
||||
allowFailure: true,
|
||||
timeoutMs: 60 * 1000,
|
||||
});
|
||||
run('claude', ['plugin', 'marketplace', 'remove', 'impeccable', '--scope', 'user'], {
|
||||
cwd: targetRepo,
|
||||
logName: 'claude-marketplace-remove.log',
|
||||
allowFailure: true,
|
||||
timeoutMs: 60 * 1000,
|
||||
});
|
||||
run('codex', ['plugin', 'remove', 'impeccable@impeccable'], {
|
||||
cwd: targetRepo,
|
||||
logName: 'codex-plugin-remove.log',
|
||||
allowFailure: true,
|
||||
timeoutMs: 60 * 1000,
|
||||
});
|
||||
run('codex', ['plugin', 'marketplace', 'remove', 'impeccable'], {
|
||||
cwd: targetRepo,
|
||||
logName: 'codex-marketplace-remove.log',
|
||||
allowFailure: true,
|
||||
timeoutMs: 60 * 1000,
|
||||
});
|
||||
|
||||
for (const abs of [
|
||||
join(homedir(), '.claude/plugins/cache/impeccable'),
|
||||
join(homedir(), '.claude/plugins/data/impeccable-impeccable'),
|
||||
join(homedir(), '.codex/plugins/cache/impeccable'),
|
||||
join(homedir(), '.codex/plugins/data/impeccable-impeccable'),
|
||||
]) {
|
||||
rmSync(abs, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
function ensureTargetGitExclude() {
|
||||
const excludePath = join(targetRepo, '.git', 'info', 'exclude');
|
||||
if (!existsSync(dirname(excludePath))) return;
|
||||
const block = [
|
||||
'# impeccable-provider-smoke-start',
|
||||
'.impeccable/provider-smoke/',
|
||||
'src/__impeccable_provider_smoke_*.html',
|
||||
'# impeccable-provider-smoke-end',
|
||||
].join('\n');
|
||||
const current = readMaybe(excludePath);
|
||||
const next = current.includes('# impeccable-provider-smoke-start')
|
||||
? current.replace(/# impeccable-provider-smoke-start[\s\S]*?# impeccable-provider-smoke-end/g, block)
|
||||
: `${current.replace(/\s*$/, '')}\n${block}\n`;
|
||||
if (next !== current) writeFileSync(excludePath, next);
|
||||
}
|
||||
|
||||
function stripManifest(rel) {
|
||||
const file = join(targetRepo, rel);
|
||||
if (!existsSync(file)) return;
|
||||
let json;
|
||||
try {
|
||||
json = JSON.parse(readFileSync(file, 'utf8'));
|
||||
} catch {
|
||||
rmSync(file, { force: true });
|
||||
return;
|
||||
}
|
||||
const hooks = json.hooks && typeof json.hooks === 'object' && !Array.isArray(json.hooks) ? json.hooks : {};
|
||||
const nextHooks = {};
|
||||
for (const [event, entries] of Object.entries(hooks)) {
|
||||
const preserved = Array.isArray(entries)
|
||||
? entries.map(stripImpeccableHookEntry).filter(Boolean)
|
||||
: entries;
|
||||
if (Array.isArray(preserved) ? preserved.length > 0 : Boolean(preserved)) nextHooks[event] = preserved;
|
||||
}
|
||||
const next = { ...json, hooks: nextHooks };
|
||||
if (Object.keys(nextHooks).length === 0) {
|
||||
rmSync(file, { force: true });
|
||||
} else {
|
||||
writeFileSync(file, `${JSON.stringify(next, null, 2)}\n`);
|
||||
}
|
||||
}
|
||||
|
||||
function stripImpeccableHookEntry(entry) {
|
||||
if (!entry || typeof entry !== 'object') return entry;
|
||||
if (containsImpeccableHook(entry)) return null;
|
||||
if (Array.isArray(entry.hooks)) {
|
||||
const hooks = entry.hooks.map(stripImpeccableHookEntry).filter(Boolean);
|
||||
if (hooks.length === 0 && entry.hooks.some(containsImpeccableHook)) return null;
|
||||
return { ...entry, hooks };
|
||||
}
|
||||
return entry;
|
||||
}
|
||||
|
||||
function containsImpeccableHook(value) {
|
||||
if (typeof value === 'string') return value.includes('skills/impeccable/scripts/hook') || value.includes('.cursor/pre-log.mjs');
|
||||
if (Array.isArray(value)) return value.some(containsImpeccableHook);
|
||||
if (value && typeof value === 'object') return Object.values(value).some(containsImpeccableHook);
|
||||
return false;
|
||||
}
|
||||
|
||||
function verifyInstallShape() {
|
||||
const claude = readText('.claude/settings.json');
|
||||
const codex = readText('.codex/hooks.json');
|
||||
const cursor = readText('.cursor/hooks.json');
|
||||
assertCount(claude, '.claude/skills/impeccable/scripts/hook.mjs', 1, 'Claude hook.mjs');
|
||||
assertCount(codex, '.agents/skills/impeccable/scripts/hook.mjs', 1, 'Codex hook.mjs');
|
||||
assertCount(cursor, '.cursor/skills/impeccable/scripts/hook-before-edit.mjs', 1, 'Cursor preToolUse');
|
||||
assertCount(cursor, '.cursor/skills/impeccable/scripts/hook-after-edit.mjs', 0, 'Cursor afterFileEdit');
|
||||
assertCount(cursor, '.cursor/skills/impeccable/scripts/hook-stop.mjs', 0, 'Cursor stop');
|
||||
for (const text of [claude, codex, cursor]) {
|
||||
if (text.includes('hook-probe.mjs')) throw new Error('hook-probe.mjs still appears in hook manifests');
|
||||
}
|
||||
for (const rel of [
|
||||
'.claude/skills/impeccable/scripts/hook.mjs',
|
||||
'.claude/skills/impeccable/scripts/hook-lib.mjs',
|
||||
'.claude/skills/impeccable/scripts/detector/cli/main.mjs',
|
||||
'.agents/skills/impeccable/scripts/hook.mjs',
|
||||
'.agents/skills/impeccable/scripts/hook-lib.mjs',
|
||||
'.agents/skills/impeccable/scripts/detector/cli/main.mjs',
|
||||
'.cursor/skills/impeccable/scripts/hook-before-edit.mjs',
|
||||
'.cursor/skills/impeccable/scripts/hook-lib.mjs',
|
||||
'.cursor/skills/impeccable/scripts/detector/cli/main.mjs',
|
||||
]) {
|
||||
assertPath(join(targetRepo, rel), rel);
|
||||
}
|
||||
for (const rel of [
|
||||
'.cursor/skills/impeccable/scripts/hook-after-edit.mjs',
|
||||
'.cursor/skills/impeccable/scripts/hook-stop.mjs',
|
||||
]) {
|
||||
if (existsSync(join(targetRepo, rel))) throw new Error(`${rel} should not exist in Cursor payload`);
|
||||
}
|
||||
if (findFiles(['.claude', '.cursor', '.agents'], 'hook-probe.mjs').length > 0) {
|
||||
throw new Error('hook-probe.mjs still exists in installed payloads');
|
||||
}
|
||||
assertNoPluginInstall();
|
||||
record('install shape', true, 'real hook manifests and payloads installed; no probe/plugin leftovers');
|
||||
}
|
||||
|
||||
function readText(rel) {
|
||||
return readFileSync(join(targetRepo, rel), 'utf8');
|
||||
}
|
||||
|
||||
function assertCount(text, needle, expected, label) {
|
||||
const actual = text.split(needle).length - 1;
|
||||
if (actual !== expected) throw new Error(`${label}: expected ${expected}, got ${actual}`);
|
||||
}
|
||||
|
||||
function findFiles(roots, filename) {
|
||||
const found = [];
|
||||
for (const root of roots) {
|
||||
walk(join(targetRepo, root), (file) => {
|
||||
if (file.endsWith(`/${filename}`)) found.push(file);
|
||||
});
|
||||
}
|
||||
return found;
|
||||
}
|
||||
|
||||
function walk(dir, visit) {
|
||||
if (!existsSync(dir)) return;
|
||||
for (const entry of readdirSync(dir, { withFileTypes: true })) {
|
||||
const full = join(dir, entry.name);
|
||||
if (entry.isDirectory()) walk(full, visit);
|
||||
else if (entry.isFile()) visit(full);
|
||||
}
|
||||
}
|
||||
|
||||
function assertNoPluginInstall() {
|
||||
const claude = run('claude', ['plugin', 'list'], { allowFailure: true, logName: 'claude-plugin-list.log', timeoutMs: 60 * 1000 });
|
||||
const codex = run('codex', ['plugin', 'list'], { allowFailure: true, logName: 'codex-plugin-list.log', timeoutMs: 60 * 1000 });
|
||||
const codexMarket = run('codex', ['plugin', 'marketplace', 'list'], { allowFailure: true, logName: 'codex-marketplace-list.log', timeoutMs: 60 * 1000 });
|
||||
for (const [name, text] of [
|
||||
['Claude plugin list', `${claude.stdout}\n${claude.stderr}`],
|
||||
['Codex plugin list', `${codex.stdout}\n${codex.stderr}`],
|
||||
['Codex marketplace list', `${codexMarket.stdout}\n${codexMarket.stderr}`],
|
||||
]) {
|
||||
if (/impeccable@impeccable|Marketplace `impeccable`|impeccable-design-hook-impl/.test(text)) {
|
||||
throw new Error(`${name} still contains Impeccable plugin install`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function runDirectContractChecks() {
|
||||
clearRuntimeState();
|
||||
const file = writeBadFixture(smokeFiles.direct);
|
||||
const env = { IMPECCABLE_HOOK_LOG: join(smokeDir, 'direct.ndjson') };
|
||||
const claude = run('node', ['.claude/skills/impeccable/scripts/hook.mjs'], {
|
||||
cwd: targetRepo,
|
||||
env,
|
||||
logName: 'direct-claude.log',
|
||||
input: JSON.stringify(postToolUseEvent('direct-claude', file, 'Edit')),
|
||||
});
|
||||
requireFinding('direct Claude hook', `${claude.stdout}\n${readMaybe(join(smokeDir, 'direct.ndjson'))}`);
|
||||
|
||||
clearRuntimeState();
|
||||
const codex = run('node', ['.agents/skills/impeccable/scripts/hook.mjs'], {
|
||||
cwd: targetRepo,
|
||||
env,
|
||||
logName: 'direct-codex.log',
|
||||
input: JSON.stringify(postToolUseEvent('direct-codex', file, 'apply_patch')),
|
||||
});
|
||||
requireFinding('direct Codex hook', `${codex.stdout}\n${readMaybe(join(smokeDir, 'direct.ndjson'))}`);
|
||||
|
||||
clearRuntimeState();
|
||||
const pre = run('node', ['.cursor/skills/impeccable/scripts/hook-before-edit.mjs'], {
|
||||
cwd: targetRepo,
|
||||
env,
|
||||
logName: 'direct-cursor-before.log',
|
||||
input: JSON.stringify({
|
||||
hook_event_name: 'preToolUse',
|
||||
cwd: targetRepo,
|
||||
tool_name: 'Write',
|
||||
tool_input: {
|
||||
file_path: join(targetRepo, smokeFiles.direct),
|
||||
content: badFixtureContent(),
|
||||
},
|
||||
}),
|
||||
});
|
||||
requireFinding('direct Cursor preToolUse hook', `${pre.stdout}\n${readMaybe(join(smokeDir, 'direct.ndjson'))}`);
|
||||
|
||||
record('direct script contracts', true, 'Claude, Codex, and Cursor preToolUse scripts detect side-tab');
|
||||
}
|
||||
|
||||
function runConfirmedExceptionPersistenceChecks() {
|
||||
const providers = selectedProviders.filter((provider) => ['claude', 'codex', 'cursor'].includes(provider));
|
||||
for (const provider of providers) {
|
||||
runConfirmedExceptionForProvider(provider);
|
||||
}
|
||||
record('confirmed exception persistence', true, `${providers.join(', ')} ignored confirmed overused-font values through shared hook.json, not source comments`);
|
||||
}
|
||||
|
||||
function runConfirmedExceptionForProvider(provider) {
|
||||
clearRuntimeState();
|
||||
const rel = confirmedSmokeFile(provider);
|
||||
const file = writeConfirmedFixture(rel);
|
||||
const configPath = join(targetRepo, '.impeccable', 'hook.json');
|
||||
const beforeLog = `${provider}-confirmed-before.ndjson`;
|
||||
const afterLog = `${provider}-confirmed-after.ndjson`;
|
||||
|
||||
rmSync(join(smokeDir, beforeLog), { force: true });
|
||||
rmSync(join(smokeDir, afterLog), { force: true });
|
||||
|
||||
const first = runInstalledProviderHook(provider, file, beforeLog);
|
||||
requireRuleFinding(`${provider} confirmed exception first hook`, `${first.stdout}\n${first.stderr}\n${readMaybe(join(smokeDir, beforeLog))}`, 'overused-font');
|
||||
if (existsSync(configPath)) {
|
||||
throw new Error(`${provider} hook wrote .impeccable/hook.json before explicit confirmation`);
|
||||
}
|
||||
|
||||
run('node', [
|
||||
providerAdminScript(provider),
|
||||
'ignore-value',
|
||||
'overused-font',
|
||||
'Roboto',
|
||||
'--shared',
|
||||
'--reason',
|
||||
`Provider smoke confirmed Roboto is intentional for ${provider}`,
|
||||
], {
|
||||
cwd: targetRepo,
|
||||
logName: `${provider}-confirmed-admin.log`,
|
||||
timeoutMs: 60 * 1000,
|
||||
});
|
||||
|
||||
const config = readJson(configPath);
|
||||
assertSpecificFontIgnoreConfig(provider, config);
|
||||
|
||||
clearTransientHookState();
|
||||
const second = runInstalledProviderHook(provider, file, afterLog);
|
||||
if (provider === 'cursor') {
|
||||
const payload = JSON.parse(second.stdout || '{}');
|
||||
if (payload.permission !== 'allow') {
|
||||
throw new Error('Cursor confirmed ignore-value did not allow the proposed write');
|
||||
}
|
||||
} else if (/overused-font|Required design corrections/.test(second.stdout || '')) {
|
||||
throw new Error(`${provider} confirmed ignore-value emitted a correction after persistence`);
|
||||
}
|
||||
|
||||
const afterEvents = readAuditEvents(join(smokeDir, afterLog));
|
||||
const suppressed = afterEvents.some((event) =>
|
||||
event.file === file
|
||||
&& (
|
||||
(Number(event.findings) > 0 && Number(event.freshFindings) === 0)
|
||||
|| (Number(event.findings) > 0 && Number(event.blockedFindings) === 0)
|
||||
)
|
||||
);
|
||||
if (!suppressed) {
|
||||
throw new Error(`${provider} confirmed ignore-value did not produce suppression audit evidence`);
|
||||
}
|
||||
|
||||
clearRuntimeState();
|
||||
}
|
||||
|
||||
function runAgentChosenFontExceptionChecks() {
|
||||
const providers = selectedProviders.filter((provider) => ['claude', 'codex', 'cursor'].includes(provider));
|
||||
for (const provider of providers) {
|
||||
runAgentChosenFontExceptionForProvider(provider);
|
||||
}
|
||||
record('agent-chosen font exception', true, `${providers.join(', ')} persisted Roboto as ignoreValues and did not write ignoreRules`);
|
||||
}
|
||||
|
||||
function runAgentChosenFontExceptionForProvider(provider) {
|
||||
clearRuntimeState();
|
||||
const rel = agentChoiceSmokeFile(provider);
|
||||
const file = writeConfirmedFixture(rel);
|
||||
const configPath = join(targetRepo, '.impeccable', 'hook.json');
|
||||
const beforeLog = `${provider}-agent-choice-before.ndjson`;
|
||||
const afterLog = `${provider}-agent-choice-after.ndjson`;
|
||||
|
||||
rmSync(join(smokeDir, beforeLog), { force: true });
|
||||
rmSync(join(smokeDir, afterLog), { force: true });
|
||||
|
||||
const first = runInstalledProviderHook(provider, file, beforeLog);
|
||||
requireRuleFinding(`${provider} agent-choice first hook`, `${first.stdout}\n${first.stderr}\n${readMaybe(join(smokeDir, beforeLog))}`, 'overused-font');
|
||||
if (existsSync(configPath)) {
|
||||
throw new Error(`${provider} hook wrote .impeccable/hook.json before explicit confirmation`);
|
||||
}
|
||||
|
||||
runProviderAgentFontException(provider, rel);
|
||||
|
||||
const config = readJson(configPath);
|
||||
assertSpecificFontIgnoreConfig(provider, config);
|
||||
|
||||
clearTransientHookState();
|
||||
const second = runInstalledProviderHook(provider, file, afterLog);
|
||||
if (provider === 'cursor') {
|
||||
const payload = JSON.parse(second.stdout || '{}');
|
||||
if (payload.permission !== 'allow') {
|
||||
throw new Error('Cursor agent-chosen ignore-value did not allow the proposed write');
|
||||
}
|
||||
} else if (/overused-font|Required design corrections/.test(second.stdout || '')) {
|
||||
throw new Error(`${provider} agent-chosen ignore-value emitted a correction after persistence`);
|
||||
}
|
||||
|
||||
clearRuntimeState();
|
||||
}
|
||||
|
||||
function runProviderAgentFontException(provider, rel) {
|
||||
const prompt = fontExceptionPrompt(provider, rel);
|
||||
if (provider === 'claude') {
|
||||
run('claude', [
|
||||
'-p',
|
||||
'--setting-sources', 'project',
|
||||
'--permission-mode', 'acceptEdits',
|
||||
'--tools', 'Read,Bash',
|
||||
'--allowedTools', 'Read Bash',
|
||||
'--debug', 'hooks',
|
||||
'--debug-file', join(smokeDir, 'claude-agent-choice-debug.log'),
|
||||
prompt,
|
||||
], {
|
||||
cwd: targetRepo,
|
||||
logName: 'claude-agent-choice.log',
|
||||
timeoutMs: 10 * 60 * 1000,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (provider === 'codex') {
|
||||
run('codex', [
|
||||
'exec',
|
||||
'-C', targetRepo,
|
||||
'--dangerously-bypass-hook-trust',
|
||||
'--dangerously-bypass-approvals-and-sandbox',
|
||||
'--json',
|
||||
prompt,
|
||||
], {
|
||||
cwd: targetRepo,
|
||||
logName: 'codex-agent-choice.log',
|
||||
timeoutMs: 10 * 60 * 1000,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (provider === 'cursor') {
|
||||
ensureCursorAgent();
|
||||
const res = run('agent', [
|
||||
'-p',
|
||||
'--force',
|
||||
'--trust',
|
||||
'--workspace', targetRepo,
|
||||
'--output-format', 'stream-json',
|
||||
prompt,
|
||||
], {
|
||||
cwd: targetRepo,
|
||||
logName: 'cursor-agent-choice.log',
|
||||
timeoutMs: 10 * 60 * 1000,
|
||||
allowFailure: true,
|
||||
});
|
||||
if (res.error || res.status !== 0) {
|
||||
const output = `${res.stdout}\n${res.stderr}\n${res.error?.message || ''}`;
|
||||
if (/Authentication required|agent login|CURSOR_API_KEY/i.test(output)) {
|
||||
const err = new Error('Cursor CLI authentication required. Run `agent login` or set CURSOR_API_KEY, then rerun `bun run smoke:hooks -- --providers=cursor`.');
|
||||
err.classification = 'cursor auth required';
|
||||
throw err;
|
||||
}
|
||||
throw new Error(res.error ? `agent failed: ${res.error.message}` : `agent exited ${res.status}`);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
throw new Error(`Unsupported agent-choice provider: ${provider}`);
|
||||
}
|
||||
|
||||
function assertSpecificFontIgnoreConfig(provider, config) {
|
||||
if (Array.isArray(config.ignoreRules) && config.ignoreRules.includes('overused-font')) {
|
||||
throw new Error(`${provider} wrote ignoreRules["overused-font"]; specific fonts must use ignoreValues`);
|
||||
}
|
||||
const ignoredValue = Array.isArray(config.ignoreValues)
|
||||
&& config.ignoreValues.some((entry) => entry.rule === 'overused-font' && entry.value === 'roboto');
|
||||
if (!ignoredValue) {
|
||||
throw new Error(`${provider} did not persist overused-font=roboto in ignoreValues`);
|
||||
}
|
||||
}
|
||||
|
||||
function runInstalledProviderHook(provider, file, logName) {
|
||||
const env = { IMPECCABLE_HOOK_LOG: join(smokeDir, logName) };
|
||||
if (provider === 'claude') {
|
||||
return run('node', ['.claude/skills/impeccable/scripts/hook.mjs'], {
|
||||
cwd: targetRepo,
|
||||
env,
|
||||
logName: `direct-${provider}-confirmed-${logName.replace(/\.ndjson$/, '.log')}`,
|
||||
input: JSON.stringify(postToolUseEvent(`confirmed-${provider}`, file, 'Edit')),
|
||||
});
|
||||
}
|
||||
if (provider === 'codex') {
|
||||
return run('node', ['.agents/skills/impeccable/scripts/hook.mjs'], {
|
||||
cwd: targetRepo,
|
||||
env,
|
||||
logName: `direct-${provider}-confirmed-${logName.replace(/\.ndjson$/, '.log')}`,
|
||||
input: JSON.stringify(postToolUseEvent(`confirmed-${provider}`, file, 'apply_patch')),
|
||||
});
|
||||
}
|
||||
if (provider === 'cursor') {
|
||||
return run('node', ['.cursor/skills/impeccable/scripts/hook-before-edit.mjs'], {
|
||||
cwd: targetRepo,
|
||||
env,
|
||||
logName: `direct-${provider}-confirmed-${logName.replace(/\.ndjson$/, '.log')}`,
|
||||
input: JSON.stringify({
|
||||
hook_event_name: 'preToolUse',
|
||||
cwd: targetRepo,
|
||||
tool_name: 'Write',
|
||||
tool_input: {
|
||||
file_path: file,
|
||||
content: readFileSync(file, 'utf8'),
|
||||
},
|
||||
}),
|
||||
});
|
||||
}
|
||||
throw new Error(`Unsupported confirmed exception provider: ${provider}`);
|
||||
}
|
||||
|
||||
function confirmedSmokeFile(provider) {
|
||||
if (provider === 'claude') return smokeFiles.confirmedClaude;
|
||||
if (provider === 'codex') return smokeFiles.confirmedCodex;
|
||||
if (provider === 'cursor') return smokeFiles.confirmedCursor;
|
||||
throw new Error(`Unsupported confirmed exception provider: ${provider}`);
|
||||
}
|
||||
|
||||
function agentChoiceSmokeFile(provider) {
|
||||
if (provider === 'claude') return smokeFiles.agentChoiceClaude;
|
||||
if (provider === 'codex') return smokeFiles.agentChoiceCodex;
|
||||
if (provider === 'cursor') return smokeFiles.agentChoiceCursor;
|
||||
throw new Error(`Unsupported agent-choice provider: ${provider}`);
|
||||
}
|
||||
|
||||
function providerAdminScript(provider) {
|
||||
if (provider === 'claude') return '.claude/skills/impeccable/scripts/hook-admin.mjs';
|
||||
if (provider === 'codex') return '.agents/skills/impeccable/scripts/hook-admin.mjs';
|
||||
if (provider === 'cursor') return '.cursor/skills/impeccable/scripts/hook-admin.mjs';
|
||||
throw new Error(`Unsupported admin provider: ${provider}`);
|
||||
}
|
||||
|
||||
function runClaudeProviderSmoke() {
|
||||
clearRuntimeState();
|
||||
const env = { IMPECCABLE_HOOK_LOG: join(smokeDir, 'claude.ndjson') };
|
||||
const prompt = providerPrompt(smokeFiles.claude);
|
||||
const res = run('claude', [
|
||||
'-p',
|
||||
'--setting-sources', 'project',
|
||||
'--permission-mode', 'acceptEdits',
|
||||
'--tools', 'Read,Write,Edit',
|
||||
'--allowedTools', 'Read Write Edit',
|
||||
'--debug', 'hooks',
|
||||
'--debug-file', join(smokeDir, 'claude-debug.log'),
|
||||
prompt,
|
||||
], {
|
||||
cwd: targetRepo,
|
||||
env,
|
||||
logName: 'claude-provider.log',
|
||||
timeoutMs: 10 * 60 * 1000,
|
||||
});
|
||||
const evidence = `${res.stdout}\n${res.stderr}\n${readMaybe(join(smokeDir, 'claude.ndjson'))}\n${readMaybe(join(smokeDir, 'claude-debug.log'))}`;
|
||||
requireFile(smokeFiles.claude, 'Claude provider fixture');
|
||||
requireFinding('Claude provider hook', evidence);
|
||||
if (!/PostToolUse|hook/i.test(evidence)) throw new Error('Claude provider evidence lacks hook/PostToolUse marker');
|
||||
record('claude provider', true, 'Claude edit triggered PostToolUse hook and side-tab detection');
|
||||
}
|
||||
|
||||
function runCodexProviderSmoke() {
|
||||
clearRuntimeState();
|
||||
const env = { IMPECCABLE_HOOK_LOG: join(smokeDir, 'codex.ndjson') };
|
||||
const prompt = `Use apply_patch to ${providerPrompt(smokeFiles.codex)}`;
|
||||
const res = run('codex', [
|
||||
'exec',
|
||||
'-C', targetRepo,
|
||||
'--dangerously-bypass-hook-trust',
|
||||
'--dangerously-bypass-approvals-and-sandbox',
|
||||
'--json',
|
||||
prompt,
|
||||
], {
|
||||
cwd: targetRepo,
|
||||
env,
|
||||
logName: 'codex-provider.log',
|
||||
timeoutMs: 10 * 60 * 1000,
|
||||
});
|
||||
const evidence = `${res.stdout}\n${res.stderr}\n${readMaybe(join(smokeDir, 'codex.ndjson'))}`;
|
||||
const cacheEvidence = `${readMaybe(join(targetRepo, '.impeccable', 'hook.cache.json'))}\n${readMaybe(join(targetRepo, '.impeccable', 'hook.pending.json'))}`;
|
||||
requireFile(smokeFiles.codex, 'Codex provider fixture');
|
||||
requireFinding('Codex provider hook', `${evidence}\n${cacheEvidence}`);
|
||||
record('codex provider', true, 'Codex apply_patch triggered project hook and side-tab detection');
|
||||
}
|
||||
|
||||
function runCursorProviderSmoke() {
|
||||
ensureCursorAgent();
|
||||
clearRuntimeState();
|
||||
const env = { IMPECCABLE_HOOK_LOG: join(smokeDir, 'cursor.ndjson') };
|
||||
const prompt = providerPrompt(smokeFiles.cursor);
|
||||
const res = run('agent', [
|
||||
'-p',
|
||||
'--force',
|
||||
'--trust',
|
||||
'--workspace', targetRepo,
|
||||
'--output-format', 'stream-json',
|
||||
prompt,
|
||||
], {
|
||||
cwd: targetRepo,
|
||||
env,
|
||||
logName: 'cursor-provider.log',
|
||||
timeoutMs: 10 * 60 * 1000,
|
||||
allowFailure: true,
|
||||
});
|
||||
if (res.error || res.status !== 0) {
|
||||
const output = `${res.stdout}\n${res.stderr}\n${res.error?.message || ''}`;
|
||||
if (/Authentication required|agent login|CURSOR_API_KEY/i.test(output)) {
|
||||
const err = new Error('Cursor CLI authentication required. Run `agent login` or set CURSOR_API_KEY, then rerun `bun run smoke:hooks -- --providers=cursor`.');
|
||||
err.classification = 'cursor auth required';
|
||||
throw err;
|
||||
}
|
||||
throw new Error(res.error ? `agent failed: ${res.error.message}` : `agent exited ${res.status}`);
|
||||
}
|
||||
const evidence = `${res.stdout}\n${res.stderr}\n${readMaybe(join(smokeDir, 'cursor.ndjson'))}\n${readMaybe(join(targetRepo, '.impeccable', 'hook.pending.json'))}\n${readMaybe(join(targetRepo, '.impeccable', 'hook.cache.json'))}`;
|
||||
requireFinding('Cursor provider hook', evidence);
|
||||
const auditEvents = readAuditEvents(join(smokeDir, 'cursor.ndjson'));
|
||||
if (!auditEvents.some((event) => event.event === 'preToolUse' && event.blocked === true)) {
|
||||
throw new Error('Cursor provider evidence lacks a preToolUse audit entry with blocked=true');
|
||||
}
|
||||
const fixturePath = join(targetRepo, smokeFiles.cursor);
|
||||
const intentionalIgnore = auditEvents.some((event) =>
|
||||
event.event === 'preToolUse'
|
||||
&& event.file === fixturePath
|
||||
&& event.skipped === 'config-ignore-file'
|
||||
);
|
||||
if (existsSync(fixturePath)) {
|
||||
const fixtureContent = readFileSync(fixturePath, 'utf8');
|
||||
if (/border-left\s*:\s*[2-9]\d*px/i.test(fixtureContent)) {
|
||||
if (!intentionalIgnore || !/ignoreFiles|ignore-file/i.test(evidence)) {
|
||||
throw new Error('Cursor provider left the blocked side-tab fixture on disk without an explicit Impeccable ignore-file escape hatch');
|
||||
}
|
||||
}
|
||||
}
|
||||
record('cursor provider', true, 'Cursor agent triggered preToolUse hook, blocked side-tab, and only proceeded through explicit ignore handling for the intentional fixture');
|
||||
}
|
||||
|
||||
function ensureCursorAgent() {
|
||||
const version = run('agent', ['--version'], {
|
||||
cwd: targetRepo,
|
||||
logName: 'cursor-agent-version-before.log',
|
||||
allowFailure: true,
|
||||
timeoutMs: 60 * 1000,
|
||||
});
|
||||
if (version.status === 0) return;
|
||||
|
||||
run('/bin/bash', ['-c', '/usr/bin/curl https://cursor.com/install -fsS | /bin/bash'], {
|
||||
cwd: targetRepo,
|
||||
logName: 'cursor-agent-install.log',
|
||||
timeoutMs: 5 * 60 * 1000,
|
||||
});
|
||||
run('agent', ['--version'], {
|
||||
cwd: targetRepo,
|
||||
logName: 'cursor-agent-version-after.log',
|
||||
timeoutMs: 60 * 1000,
|
||||
});
|
||||
}
|
||||
|
||||
function writeBadFixture(rel) {
|
||||
const abs = join(targetRepo, rel);
|
||||
mkdirSync(dirname(abs), { recursive: true });
|
||||
writeFileSync(abs, badFixtureContent());
|
||||
return abs;
|
||||
}
|
||||
|
||||
function writeConfirmedFixture(rel) {
|
||||
const abs = join(targetRepo, rel);
|
||||
mkdirSync(dirname(abs), { recursive: true });
|
||||
writeFileSync(abs, confirmedFixtureContent());
|
||||
return abs;
|
||||
}
|
||||
|
||||
function badFixtureContent() {
|
||||
return [
|
||||
'<!doctype html>',
|
||||
'<html>',
|
||||
' <body>',
|
||||
' <style>',
|
||||
' .card {',
|
||||
' border-left: 4px solid #7c3aed;',
|
||||
' border-radius: 16px;',
|
||||
' padding: 16px;',
|
||||
' }',
|
||||
' </style>',
|
||||
' <div class="card">Temporary Impeccable provider smoke fixture</div>',
|
||||
' </body>',
|
||||
'</html>',
|
||||
'',
|
||||
].join('\n');
|
||||
}
|
||||
|
||||
function confirmedFixtureContent() {
|
||||
return [
|
||||
'<!doctype html>',
|
||||
'<html>',
|
||||
' <body>',
|
||||
' <style>',
|
||||
' /* impeccable: ignore overused-font */',
|
||||
' body { font-family: "Roboto", ui-sans-serif, system-ui, sans-serif; }',
|
||||
' .card { border-radius: 16px; padding: 16px; }',
|
||||
' </style>',
|
||||
' <div class="card">Temporary Impeccable confirmed exception fixture using Roboto.</div>',
|
||||
' </body>',
|
||||
'</html>',
|
||||
'',
|
||||
].join('\n');
|
||||
}
|
||||
|
||||
function providerPrompt(rel) {
|
||||
return [
|
||||
`Create ${rel} as an intentionally bad Impeccable hook test fixture.`,
|
||||
'The file must be HTML and include a rounded card with exactly this CSS pattern: border-left: 4px solid #7c3aed; border-radius: 16px; padding: 16px.',
|
||||
'This is intentionally bad UI for hook verification, so do not fix or remove the side border if a hook warns about it.',
|
||||
'After creating the file, stop.',
|
||||
].join(' ');
|
||||
}
|
||||
|
||||
function fontExceptionPrompt(provider, rel) {
|
||||
return [
|
||||
`Read the installed Impeccable hooks reference for ${provider}, then persist a confirmed hook exception for Roboto specifically in ${rel}.`,
|
||||
'The user confirms Roboto is intentional for this fixture, but did not ask to ignore overused fonts generally.',
|
||||
'Use the /impeccable hooks / hook-admin flow; do not edit .impeccable/hook.json by hand and do not edit the source fixture.',
|
||||
'The final config must use ignoreValues for overused-font=roboto and must not add overused-font to ignoreRules.',
|
||||
'After updating the config, stop.',
|
||||
].join(' ');
|
||||
}
|
||||
|
||||
function postToolUseEvent(sessionId, file, toolName) {
|
||||
return {
|
||||
session_id: sessionId,
|
||||
cwd: targetRepo,
|
||||
hook_event_name: 'PostToolUse',
|
||||
tool_name: toolName,
|
||||
tool_input: { file_path: file },
|
||||
};
|
||||
}
|
||||
|
||||
function requireFile(rel, label) {
|
||||
const abs = join(targetRepo, rel);
|
||||
if (!existsSync(abs)) throw new Error(`${label} was not created: ${rel}`);
|
||||
}
|
||||
|
||||
function requireFinding(label, text) {
|
||||
requireRuleFinding(label, text, 'side-tab');
|
||||
}
|
||||
|
||||
function requireRuleFinding(label, text, rule) {
|
||||
const rulePattern = new RegExp(rule.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'));
|
||||
if (!rulePattern.test(text) || !/Required design corrections|findings?|antipattern|side-tab|overused-font/.test(text)) {
|
||||
throw new Error(`${label} did not show ${rule} detector evidence`);
|
||||
}
|
||||
}
|
||||
|
||||
function readAuditEvents(path) {
|
||||
return readMaybe(path)
|
||||
.split('\n')
|
||||
.map((line) => line.trim())
|
||||
.filter(Boolean)
|
||||
.map((line) => {
|
||||
try { return JSON.parse(line); } catch { return null; }
|
||||
})
|
||||
.filter(Boolean);
|
||||
}
|
||||
|
||||
function readMaybe(path) {
|
||||
try {
|
||||
return readFileSync(path, 'utf8');
|
||||
} catch {
|
||||
return '';
|
||||
}
|
||||
}
|
||||
|
||||
function readJson(path) {
|
||||
return JSON.parse(readFileSync(path, 'utf8'));
|
||||
}
|
||||
|
||||
function cleanSmokeArtifacts() {
|
||||
rmSync(smokeDir, { recursive: true, force: true });
|
||||
mkdirSync(smokeDir, { recursive: true });
|
||||
cleanSmokeFiles();
|
||||
}
|
||||
|
||||
function cleanSmokeFiles() {
|
||||
for (const rel of Object.values(smokeFiles)) {
|
||||
rmSync(join(targetRepo, rel), { force: true });
|
||||
}
|
||||
}
|
||||
|
||||
function clearRuntimeState() {
|
||||
for (const rel of [
|
||||
'.impeccable/hook.cache.json',
|
||||
'.impeccable/hook.pending.json',
|
||||
'.impeccable/hook.json',
|
||||
'.impeccable/hook.local.json',
|
||||
]) {
|
||||
rmSync(join(targetRepo, rel), { force: true });
|
||||
}
|
||||
}
|
||||
|
||||
function clearTransientHookState() {
|
||||
for (const rel of [
|
||||
'.impeccable/hook.cache.json',
|
||||
'.impeccable/hook.pending.json',
|
||||
]) {
|
||||
rmSync(join(targetRepo, rel), { force: true });
|
||||
}
|
||||
}
|
||||
@@ -25,11 +25,11 @@ export const SUITES = {
|
||||
triggers: [
|
||||
...COMMON_INFRA_PATTERNS,
|
||||
/^scripts\/(?!benchmark-detector|build-browser-detector|build-extension)/,
|
||||
/^skill\/(SKILL\.src\.md|agents\/|reference\/|scripts\/(context|context-signals|critique-storage|design-parser|impeccable-paths|is-generated))/,
|
||||
/^skill\/(SKILL\.src\.md|agents\/|reference\/|scripts\/(cleanup-deprecated|context|context-signals|critique-storage|design-parser|hook|impeccable-paths|is-generated))/,
|
||||
/^site\/(pages|content|components|layouts)\//,
|
||||
/^README(\.npm)?\.md$/,
|
||||
/^cli\/bin\//,
|
||||
/^tests\/(build|context|context-signals|critique-storage|design-parser|docs-integrity|impeccable-paths|skills-cli|test-suites|windows-path-fix)\.test\.(js|mjs)$/,
|
||||
/^tests\/(build|cleanup-deprecated|context|context-signals|critique-storage|design-parser|docs-integrity|hook|hook-build|impeccable-paths|skills-cli|test-suites|windows-path-fix)\.test\.(js|mjs)$/,
|
||||
/^tests\/lib\//,
|
||||
],
|
||||
commands: [
|
||||
@@ -55,6 +55,8 @@ export const SUITES = {
|
||||
'tests/context-signals.test.mjs',
|
||||
'tests/critique-storage.test.mjs',
|
||||
'tests/design-parser.test.mjs',
|
||||
'tests/hook-build.test.mjs',
|
||||
'tests/hook.test.mjs',
|
||||
'tests/impeccable-paths.test.mjs',
|
||||
'tests/test-suites.test.mjs',
|
||||
],
|
||||
|
||||
Reference in New Issue
Block a user