mirror of
https://github.com/pbakaus/impeccable.git
synced 2026-09-17 00:26:41 +03:00
Give Claude routing checks a measured setup budget
AI assistance: Codex, under maintainer direction.
This commit is contained in:
@@ -63,6 +63,8 @@ its genuine structured-question fallback; browser decisions have separate E2E.
|
||||
|
||||
Set `IMPECCABLE_SKILL_BEHAVIOR_TRACE_DIR=<directory>` to retain per-turn JSON
|
||||
with model, prompt, tool results, response ordering, usage, and finish reason.
|
||||
Progress and failed turns retain their tool traces too; only completed turns
|
||||
carry the full response sequence and final usage.
|
||||
These are local diagnostic artifacts; inspect before sharing. Successful reads
|
||||
or full reference content in shell output count as loading; filename mentions,
|
||||
denied commands, and failed reads do not.
|
||||
@@ -98,6 +100,18 @@ An intermediate candidate run scored 8/9 because the control rejected valid
|
||||
rerun passed 9/9. This is one measured run per variant, not a reliability estimate
|
||||
or an all-workflow pass. Broader routing and workflow results remain separate.
|
||||
|
||||
On the resolved fixture, Gemini's workflow run passed 4/5: the completed new
|
||||
page omitted user confirmation. Tightening the existing question paragraph
|
||||
made its focused build lifecycle rerun pass 1/1. OpenAI passed all five workflow
|
||||
cases with the fixture corrections alone. These are incremental measurements,
|
||||
not one full sweep on the final candidate.
|
||||
|
||||
Claude's three-step routing sweep cut off two setup cases before loading
|
||||
`new-work.md`. Both loaded it in bounded six-step diagnostics. Claude now has
|
||||
the same six-step setup allowance as Gemini; reference and edit-order assertions
|
||||
are unchanged. A full-build baseline separately hit the existing 840-second
|
||||
deadline and is not counted as a pass.
|
||||
|
||||
## Scenarios
|
||||
|
||||
| # | Setup | Assertion |
|
||||
|
||||
@@ -101,12 +101,11 @@ for (const modelId of resolveModelList()) {
|
||||
return;
|
||||
}
|
||||
const model = getModel(modelId);
|
||||
// Gemini Flash tends to inspect one file at a time, while the production
|
||||
// Anthropic/OpenAI models batch setup reads and then begin implementation.
|
||||
// Keep the latter tightly bounded so this routing suite does not turn into
|
||||
// a page-generation benchmark, but leave Gemini enough room to reach the
|
||||
// same required reference.
|
||||
const setupMaxSteps = provider === 'google' ? 6 : 3;
|
||||
// Claude and Gemini may inspect the workspace before loading references.
|
||||
// Three steps truncated valid Claude setup; six-step diagnostics reached
|
||||
// the same required references. Keep the budget bounded, not a requirement
|
||||
// that every provider batches its tool calls like OpenAI.
|
||||
const setupMaxSteps = provider === 'openai' ? 3 : 6;
|
||||
|
||||
it('scenario 1: no PRODUCT.md / DESIGN.md', async () => {
|
||||
const workspace = prepareWorkspace({ files: {} });
|
||||
|
||||
Reference in New Issue
Block a user