Give Claude routing checks a measured setup budget

AI assistance: Codex, under maintainer direction.
This commit is contained in:
Paul Bakaus
2026-09-07 16:06:31 -07:00
parent cf9b35d544
commit 61925821b3
2 changed files with 19 additions and 6 deletions
+14
View File
@@ -63,6 +63,8 @@ its genuine structured-question fallback; browser decisions have separate E2E.
Set `IMPECCABLE_SKILL_BEHAVIOR_TRACE_DIR=<directory>` to retain per-turn JSON
with model, prompt, tool results, response ordering, usage, and finish reason.
Progress and failed turns retain their tool traces too; only completed turns
carry the full response sequence and final usage.
These are local diagnostic artifacts; inspect before sharing. Successful reads
or full reference content in shell output count as loading; filename mentions,
denied commands, and failed reads do not.
@@ -98,6 +100,18 @@ An intermediate candidate run scored 8/9 because the control rejected valid
rerun passed 9/9. This is one measured run per variant, not a reliability estimate
or an all-workflow pass. Broader routing and workflow results remain separate.
On the resolved fixture, Gemini's workflow run passed 4/5: the completed new
page omitted user confirmation. Tightening the existing question paragraph
made its focused build lifecycle rerun pass 1/1. OpenAI passed all five workflow
cases with the fixture corrections alone. These are incremental measurements,
not one full sweep on the final candidate.
Claude's three-step routing sweep cut off two setup cases before loading
`new-work.md`. Both loaded it in bounded six-step diagnostics. Claude now has
the same six-step setup allowance as Gemini; reference and edit-order assertions
are unchanged. A full-build baseline separately hit the existing 840-second
deadline and is not counted as a pass.
## Scenarios
| # | Setup | Assertion |
+5 -6
View File
@@ -101,12 +101,11 @@ for (const modelId of resolveModelList()) {
return;
}
const model = getModel(modelId);
// Gemini Flash tends to inspect one file at a time, while the production
// Anthropic/OpenAI models batch setup reads and then begin implementation.
// Keep the latter tightly bounded so this routing suite does not turn into
// a page-generation benchmark, but leave Gemini enough room to reach the
// same required reference.
const setupMaxSteps = provider === 'google' ? 6 : 3;
// Claude and Gemini may inspect the workspace before loading references.
// Three steps truncated valid Claude setup; six-step diagnostics reached
// the same required references. Keep the budget bounded, not a requirement
// that every provider batches its tool calls like OpenAI.
const setupMaxSteps = provider === 'openai' ? 3 : 6;
it('scenario 1: no PRODUCT.md / DESIGN.md', async () => {
const workspace = prepareWorkspace({ files: {} });