mirror of
https://github.com/pbakaus/impeccable.git
synced 2026-09-11 21:57:14 +03:00
Keep critique's report out of the question's message
The critique report and the AskUserQuestion call shipped in one assistant message, so the report stayed hidden until the user answered the picker and the command read as if it had never run. Reorder critique's persistence steps so the temp-file cleanup runs after the report and trend line are sent. That cleanup now ends the message carrying the report, leaving the questions to open a fresh one. Both critique.md and overdrive.md state the constraint and why it exists, so the ordering is not an unexplained sequence a model can optimize away. Overdrive additionally moves its direction descriptions inside the question options, where the user is actually reading them. Also fix the ask_instruction splices. The placeholder is a complete sentence, but five call sites spliced it mid-sentence and shipped text like "stop and STOP and call the AskUserQuestion tool to clarify. before expanding it". Every call site is now sentence-initial and the twelve lowercase provider values are capitalized to match, with a comment in utils.js pinning the contract. Record a workflow-contract baseline for the current model lineup. The two failures seen while validating this change are pre-existing: bolder refinement fails on deepseek-v4-flash identically with bolder.md reverted to HEAD, and redesign replaces DESIGN is flaky on assertions driven by new-work.md, which this change does not touch. Prepared with AI assistance (Claude Code). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Paul Bakaus
co-authored by
Claude Opus 5
Claude Opus 5
parent
bd25359748
commit
0e5c6cbe17
+13
-11
@@ -465,31 +465,33 @@ export const PROVIDER_PLACEHOLDERS = {
|
||||
'cursor': {
|
||||
model: 'the model',
|
||||
config_file: '.cursorrules',
|
||||
ask_instruction: 'ask the user directly to clarify what you cannot infer.',
|
||||
ask_instruction: 'Ask the user directly to clarify what you cannot infer.',
|
||||
command_prefix: '/'
|
||||
},
|
||||
'gemini': {
|
||||
model: 'Gemini',
|
||||
config_file: 'GEMINI.md',
|
||||
ask_instruction: 'ask the user directly to clarify what you cannot infer.',
|
||||
ask_instruction: 'Ask the user directly to clarify what you cannot infer.',
|
||||
command_prefix: '/'
|
||||
},
|
||||
'codex': {
|
||||
model: 'GPT',
|
||||
config_file: 'AGENTS.md',
|
||||
// Every {{ask_instruction}} call site is sentence-initial, so each value is a
|
||||
// complete capitalized sentence. Keep it that way when adding a provider.
|
||||
ask_instruction: "STOP and use Codex's structured user-input/question tool when available; if unavailable, ask directly in chat to clarify what you cannot infer.",
|
||||
command_prefix: '$'
|
||||
},
|
||||
'agents': {
|
||||
model: 'the model',
|
||||
config_file: '.github/copilot-instructions.md',
|
||||
ask_instruction: 'ask the user directly to clarify what you cannot infer.',
|
||||
ask_instruction: 'Ask the user directly to clarify what you cannot infer.',
|
||||
command_prefix: '/'
|
||||
},
|
||||
'kiro': {
|
||||
model: 'Claude',
|
||||
config_file: '.kiro/settings.json',
|
||||
ask_instruction: 'ask the user directly to clarify what you cannot infer.',
|
||||
ask_instruction: 'Ask the user directly to clarify what you cannot infer.',
|
||||
command_prefix: '/'
|
||||
},
|
||||
opencode: {
|
||||
@@ -501,31 +503,31 @@ export const PROVIDER_PLACEHOLDERS = {
|
||||
'pi': {
|
||||
model: 'the model',
|
||||
config_file: 'AGENTS.md',
|
||||
ask_instruction: 'ask the user directly to clarify what you cannot infer.',
|
||||
ask_instruction: 'Ask the user directly to clarify what you cannot infer.',
|
||||
command_prefix: '/'
|
||||
},
|
||||
'qoder': {
|
||||
model: 'the model',
|
||||
config_file: 'AGENTS.md',
|
||||
ask_instruction: 'ask the user directly to clarify what you cannot infer.',
|
||||
ask_instruction: 'Ask the user directly to clarify what you cannot infer.',
|
||||
command_prefix: '/'
|
||||
},
|
||||
'trae': {
|
||||
model: 'the model',
|
||||
config_file: 'RULES.md',
|
||||
ask_instruction: 'ask the user directly to clarify what you cannot infer.',
|
||||
ask_instruction: 'Ask the user directly to clarify what you cannot infer.',
|
||||
command_prefix: '/'
|
||||
},
|
||||
'rovo-dev': {
|
||||
model: 'Rovo Dev',
|
||||
config_file: 'AGENTS.md',
|
||||
ask_instruction: 'ask the user directly to clarify what you cannot infer.',
|
||||
ask_instruction: 'Ask the user directly to clarify what you cannot infer.',
|
||||
command_prefix: '/'
|
||||
},
|
||||
'vibe': {
|
||||
model: 'Mistral',
|
||||
config_file: 'AGENTS.md',
|
||||
ask_instruction: 'ask the user directly to clarify what you cannot infer.',
|
||||
ask_instruction: 'Ask the user directly to clarify what you cannot infer.',
|
||||
command_prefix: '/'
|
||||
},
|
||||
'grok': {
|
||||
@@ -537,7 +539,7 @@ export const PROVIDER_PLACEHOLDERS = {
|
||||
'antigravity': {
|
||||
model: 'Gemini',
|
||||
config_file: 'AGENTS.md',
|
||||
ask_instruction: 'ask the user directly to clarify what you cannot infer.',
|
||||
ask_instruction: 'Ask the user directly to clarify what you cannot infer.',
|
||||
command_prefix: '/'
|
||||
},
|
||||
'hermes': {
|
||||
@@ -546,7 +548,7 @@ export const PROVIDER_PLACEHOLDERS = {
|
||||
// for harnesses without a vendor-fixed assistant name.
|
||||
model: 'the model',
|
||||
config_file: 'AGENTS.md',
|
||||
ask_instruction: 'ask the user directly to clarify what you cannot infer.',
|
||||
ask_instruction: 'Ask the user directly to clarify what you cannot infer.',
|
||||
command_prefix: '/'
|
||||
}
|
||||
};
|
||||
|
||||
@@ -6,7 +6,7 @@ An open direction round owns the word first: "bolder" said while a direction dec
|
||||
|
||||
## Scope is sovereign
|
||||
|
||||
"Everything else stays" is a literal instruction. Touch only the named target. Do not restyle its neighbors, do not migrate the page to a new idea, do not add colors, fonts, radii, shadows, or system primitives the surface does not already own. If the existing system genuinely cannot express the direction, stop and {{ask_instruction}} before expanding it, naming the exact addition and the job it would do.
|
||||
"Everything else stays" is a literal instruction. Touch only the named target. Do not restyle its neighbors, do not migrate the page to a new idea, do not add colors, fonts, radii, shadows, or system primitives the surface does not already own. If the existing system genuinely cannot express the direction, do not expand it on your own. {{ask_instruction}} Name the exact addition and the job it would do.
|
||||
|
||||
## Why it reads flat
|
||||
|
||||
|
||||
@@ -221,15 +221,13 @@ Skip this step if the Setup slug was null (vague or root-level target).
|
||||
```
|
||||
`max_score` is the applicable maximum from the heuristic table (40 when every heuristic applied), so a later run can tell a renormalized total from a full one. The helper prints the absolute path it wrote.
|
||||
|
||||
3. **Delete the temp body file** after the write attempt completes, whether the write succeeded or failed. If deletion fails, mention `temp-file cleanup failed: <reason>` briefly in the final output, but do not block the critique.
|
||||
|
||||
4. **Read the trend** for context:
|
||||
3. **Read the trend** for context:
|
||||
```bash
|
||||
node {{scripts_path}}/critique-storage.mjs trend "<resolved target>" 5
|
||||
```
|
||||
This returns a JSON array of the last 5 frontmatter entries (including the one you just wrote).
|
||||
|
||||
5. **Append a single line to the user-visible output**, after the report and before the questions:
|
||||
4. **Append a single line to the user-visible output**, after the report and before the questions:
|
||||
|
||||
> **Trend for `<slug>` (last 5 runs): 24 → 28 → 32 → 29 → 32 (out of 40)**
|
||||
> Wrote `.impeccable/critique/<filename>`.
|
||||
@@ -238,12 +236,16 @@ Skip this step if the Setup slug was null (vague or root-level target).
|
||||
|
||||
If this is the first run for the slug, the trend is just one score; say so: "First run for this target, no trend yet."
|
||||
|
||||
5. **Send the report and trend line to the user, then delete the temp body file**, whether the write succeeded or failed. The order is deliberate: the deletion is the last thing in the message that carries the report, so the questions below open a fresh one. Do not hold the report back to bundle it with the questions. If deletion fails, mention `temp-file cleanup failed: <reason>` briefly in the final output, but do not block the critique.
|
||||
|
||||
This is fire-and-forget. Do not show the user the helper's JSON output; only the human-readable trend line and the written path. Failures here should not block the rest of the flow; print the error and move on.
|
||||
|
||||
### Ask the User
|
||||
|
||||
**After presenting findings**, use targeted questions based on what was actually found. {{ask_instruction}} These answers will shape the action plan.
|
||||
|
||||
The report must already be sent before you ask. A structured question blocks the message it rides in until the user answers, so a report bundled with the questions stays invisible until they have answered, and the critique reads as if it never ran. The temp-file cleanup that closed the persistence step ends the report's message; ask once it returns, and put nothing but the questions in that turn.
|
||||
|
||||
Ask questions along these lines (adapt to the specific findings; do NOT ask generic questions):
|
||||
|
||||
1. **Priority direction**: Based on the issues found, ask which category matters most to the user right now. For example: "I found problems with visual hierarchy, color usage, and information overload. Which area should we tackle first?" Offer the top 2-3 issue categories as options.
|
||||
|
||||
@@ -21,7 +21,7 @@ Analyze what makes the design feel complex or cluttered:
|
||||
- What can be removed, hidden, or combined?
|
||||
- What's the 20% that delivers 80% of value?
|
||||
|
||||
If any of these are unclear from the codebase, {{ask_instruction}}
|
||||
If any of these are unclear from the codebase, do not guess. {{ask_instruction}}
|
||||
|
||||
**CRITICAL**: Simplicity is not about removing features. It's about removing obstacles between users and their goals. Every element should justify its existence.
|
||||
|
||||
|
||||
@@ -68,7 +68,7 @@ Omit irrelevant sections rather than filling them with invented rules. Put respo
|
||||
- An existing `DESIGN.md` is stale (the design has drifted).
|
||||
- Before a large redesign, to capture the current state as a reference.
|
||||
|
||||
If a `DESIGN.md` already exists, **do not silently overwrite it**. Show the user the existing file and {{ask_instruction}} whether to refresh, overwrite, or merge.
|
||||
If a `DESIGN.md` already exists, **do not silently overwrite it**. Show the user the existing file, then {{ask_instruction}} The choice is refresh, overwrite, or merge.
|
||||
|
||||
## Two paths
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ Identify reusable patterns, components, and design tokens, then extract and cons
|
||||
|
||||
Find the design system, component library, or shared UI directory. Understand its structure: component organization, naming conventions, design token structure, import/export conventions.
|
||||
|
||||
**CRITICAL**: If no design system exists, {{ask_instruction}} before creating one. Understand the preferred location and structure first.
|
||||
**CRITICAL**: If no design system exists, do not create one yet. {{ask_instruction}} Understand the preferred location and structure first.
|
||||
|
||||
## Step 2: Identify Patterns
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ Push an interface past conventional limits. This isn't just about visual effects
|
||||
This command has the highest potential to misfire. Do NOT jump straight into implementation. You MUST:
|
||||
|
||||
1. **Think through 2-3 different directions**: consider different techniques, levels of ambition, and aesthetic approaches. For each direction, briefly describe what the result would look and feel like.
|
||||
2. **{{ask_instruction}}** to present these directions and get the user's pick before writing any code. Explain trade-offs (browser support, performance cost, complexity).
|
||||
2. **Get the user's pick before writing any code.** {{ask_instruction}} Carry each direction's description and its trade-offs (browser support, performance cost, complexity) inside the option itself, so the user is choosing between things they can read. A structured question blocks the message it rides in until the user answers, so directions written alongside the question stay invisible while the user is being asked to choose between them.
|
||||
3. Only proceed with the direction the user confirms.
|
||||
|
||||
Skipping this step risks building something embarrassing that needs to be thrown away.
|
||||
|
||||
@@ -28,7 +28,7 @@ Analyze what makes the design feel too intense:
|
||||
- What's working? (Don't throw away good ideas)
|
||||
- What's the core message? (Preserve what matters)
|
||||
|
||||
If any of these are unclear from the codebase, {{ask_instruction}}
|
||||
If any of these are unclear from the codebase, do not guess. {{ask_instruction}}
|
||||
|
||||
**CRITICAL**: "Quieter" doesn't mean boring or generic. It means refined and easier on the eyes. Think luxury, not laziness.
|
||||
|
||||
|
||||
@@ -614,7 +614,7 @@ describe('replacePlaceholders', () => {
|
||||
expect(result).toBe('STOP and call the AskUserQuestion tool to clarify.');
|
||||
|
||||
const cursorResult = replacePlaceholders('{{ask_instruction}}', 'cursor');
|
||||
expect(cursorResult).toBe('ask the user directly to clarify what you cannot infer.');
|
||||
expect(cursorResult).toBe('Ask the user directly to clarify what you cannot infer.');
|
||||
});
|
||||
|
||||
test('should replace {{available_commands}} with command list', () => {
|
||||
|
||||
@@ -63,6 +63,63 @@ an initialized natural build request, replacement-world redesign, and scope-pres
|
||||
refinement. It checks question order and context/artifact writes rather than
|
||||
only reference-file loading.
|
||||
|
||||
## Workflow-contract baseline (2026-08-13, current lineup)
|
||||
|
||||
Measured on `claude-sonnet-5`, `gpt-5.6-luna`, `gemini-3.5-flash`, and
|
||||
`deepseek-v4-flash` while checking whether an `{{ask_instruction}}` rewrite had
|
||||
regressed anything. Two of the four workflow-contract scenarios fail for reasons
|
||||
that predate that change. Treat both as the known floor; a regression is a
|
||||
failure beyond these.
|
||||
|
||||
| Scenario | claude-sonnet-5 | gpt-5.6-luna | gemini-3.5-flash | deepseek-v4-flash |
|
||||
|---|---|---|---|---|
|
||||
| attended fresh init | not measured | not measured | not measured | not measured |
|
||||
| initialized natural build | not measured | not measured | not measured | not measured |
|
||||
| redesign replaces DESIGN | flaky | not measured | not measured | not measured |
|
||||
| bolder refinement | not measured | pass | pass | **fail** |
|
||||
|
||||
`not measured` means exactly that: the cell was never run in isolation on this
|
||||
lineup. Only the two failing scenarios were scoped per model, because the
|
||||
investigation was about whether a specific edit had regressed them. The rows are
|
||||
worth keeping anyway, since a scenario absent from the table is easy to mistake
|
||||
for a scenario that passed.
|
||||
|
||||
**`bolder refinement`, deepseek-v4-flash.** The model runs `context.mjs`, reads
|
||||
`bolder.md`, `craft-floor.md`, and `current.html`, then ends its turn without
|
||||
editing anything: empty `writePaths`, no `ask_user_question` call, well short of
|
||||
the 16-step cap. Confirmed identical on HEAD with `bolder.md` reverted, so it is
|
||||
not a skill-text problem. Same shape as the gpt-5.4-mini scenario 6/7 failures
|
||||
below: the model consumes the references and then declines to act.
|
||||
|
||||
**`redesign replaces DESIGN`, flaky.** It has failed on two different assertions
|
||||
across runs (`designWrite > question` and `implementation > designWrite`), and on
|
||||
one run claude-sonnet-5 exhausted the 300s per-test timeout instead of asserting.
|
||||
The traces never load `document.md`; the ordering under test comes from
|
||||
`new-work.md`. Re-run before believing a single red result here. Which model
|
||||
produced which failure was not pinned down, so the row records only that the
|
||||
scenario is unstable.
|
||||
|
||||
The `bolder` claude-sonnet-5 cell is unmeasured for a specific reason: the scoped
|
||||
run that produced this table used a 180s cap, which sonnet exceeded. That is a
|
||||
timeout, not a failure, and it is why the guidance below insists on 300000.
|
||||
|
||||
### Scoping a run while investigating
|
||||
|
||||
Both files honor `--test-name-pattern`, which is much cheaper than a full sweep
|
||||
when bisecting one scenario:
|
||||
|
||||
```bash
|
||||
IMPECCABLE_QUESTION_DISABLED=1 CI=1 IMPECCABLE_SKILL_BEHAVIOR_MODELS=deepseek-v4-flash \
|
||||
node --test --test-timeout=300000 --test-force-exit \
|
||||
--test-name-pattern="bolder refinement" tests/skill-behavior/workflow-contract.test.mjs
|
||||
```
|
||||
|
||||
Keep `--test-timeout` at 300000. A tighter cap turns claude-sonnet-5's slower
|
||||
runs into timeouts that look like failures. Set `IMPECCABLE_QUESTION_DISABLED=1`
|
||||
and `CI=1` so `serve-question.mjs` cannot open a browser window on the host. Pipe
|
||||
to a file rather than `tail`; node prints the failing-test summary at the end,
|
||||
and truncating it costs you the per-model attribution.
|
||||
|
||||
## Baseline state (2026-05-20, previous cheap tier)
|
||||
|
||||
> **Historical record.** The default models are now `claude-sonnet-5`,
|
||||
|
||||
Reference in New Issue
Block a user