diff --git a/tests/skill-behavior/README.md b/tests/skill-behavior/README.md index 2860e1c71..e6183fb13 100644 --- a/tests/skill-behavior/README.md +++ b/tests/skill-behavior/README.md @@ -63,6 +63,8 @@ its genuine structured-question fallback; browser decisions have separate E2E. Set `IMPECCABLE_SKILL_BEHAVIOR_TRACE_DIR=` to retain per-turn JSON with model, prompt, tool results, response ordering, usage, and finish reason. +Progress and failed turns retain their tool traces too; only completed turns +carry the full response sequence and final usage. These are local diagnostic artifacts; inspect before sharing. Successful reads or full reference content in shell output count as loading; filename mentions, denied commands, and failed reads do not. @@ -98,6 +100,18 @@ An intermediate candidate run scored 8/9 because the control rejected valid rerun passed 9/9. This is one measured run per variant, not a reliability estimate or an all-workflow pass. Broader routing and workflow results remain separate. +On the resolved fixture, Gemini's workflow run passed 4/5: the completed new +page omitted user confirmation. Tightening the existing question paragraph +made its focused build lifecycle rerun pass 1/1. OpenAI passed all five workflow +cases with the fixture corrections alone. These are incremental measurements, +not one full sweep on the final candidate. + +Claude's three-step routing sweep cut off two setup cases before loading +`new-work.md`. Both loaded it in bounded six-step diagnostics. Claude now has +the same six-step setup allowance as Gemini; reference and edit-order assertions +are unchanged. A full-build baseline separately hit the existing 840-second +deadline and is not counted as a pass. + ## Scenarios | # | Setup | Assertion | diff --git a/tests/skill-behavior/scenarios.test.mjs b/tests/skill-behavior/scenarios.test.mjs index ed1f30f54..9ae7888fc 100644 --- a/tests/skill-behavior/scenarios.test.mjs +++ b/tests/skill-behavior/scenarios.test.mjs @@ -101,12 +101,11 @@ for (const modelId of resolveModelList()) { return; } const model = getModel(modelId); - // Gemini Flash tends to inspect one file at a time, while the production - // Anthropic/OpenAI models batch setup reads and then begin implementation. - // Keep the latter tightly bounded so this routing suite does not turn into - // a page-generation benchmark, but leave Gemini enough room to reach the - // same required reference. - const setupMaxSteps = provider === 'google' ? 6 : 3; + // Claude and Gemini may inspect the workspace before loading references. + // Three steps truncated valid Claude setup; six-step diagnostics reached + // the same required references. Keep the budget bounded, not a requirement + // that every provider batches its tool calls like OpenAI. + const setupMaxSteps = provider === 'openai' ? 3 : 6; it('scenario 1: no PRODUCT.md / DESIGN.md', async () => { const workspace = prepareWorkspace({ files: {} });