skills/pbakaus_impeccable
1
0
Fork 0
mirror of https://github.com/pbakaus/impeccable.git synced 2026-09-16 16:16:32 +03:00
Code Issues Packages Projects Releases Wiki Activity

Bump skill-behavior test lineup: gemini-3.7-flash (#598)

Browse Source 
* Bump skill-behavior google lineup to gemini-3.7-flash

gemini-3.7-flash replaces gemini-3.6-flash in DEFAULT_MODELS. The
README notes that the recorded gemini baseline cells were measured on
3.6-flash (or 3.5-flash where marked) and count as unmeasured on 3.7
per the suite's own cross-version rule, to be re-run on the next Setup
or routing change.

AI-assisted change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This commit is contained in:
Paul Bakaus
2026-08-15 19:03:18 -07:00
committed by GitHub
co-authored by Claude Fable 5 Copilot Autofix powered by AI
parent 7b646bafd6
commit 36457e191f
2 changed files with 7 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/skill-behavior/README.md
+6
View File
@@ -90,6 +90,12 @@ Against the current default lineup, two cells are the known floor:
`redesign replaces DESIGN` is flaky on every model, and `critique closes` is
flaky on gemini-3.6-flash. A regression is a failure beyond those two.
The Google slot in `DEFAULT_MODELS` moved to `gemini-3.7-flash` on 2026-08-15.
Every Gemini cell in the tables below was measured on 3.6-flash (or 3.5-flash
where marked), and per the cross-version rule further down, those results are
unmeasured on 3.7, not inherited. Re-run the sweep on the next Setup or routing
change and update the tables to the new column.
**Read any failure against the clock before calling it behavior.** The suite ran
at a 300s per-test timeout until 2026-08-13, and for the workflow-contract
scenarios that cap was below the runtime of a correct run. `initialized natural
tests/skill-behavior/providers.mjs
+1 -1
View File
@@ -127,7 +127,7 @@ export function getProviderOptions(modelId) {
* its own floor:
* IMPECCABLE_SKILL_BEHAVIOR_MODELS=gpt-5.6-luna,deepseek-v4-flash
*/
export const DEFAULT_MODELS = ['claude-sonnet-5', 'gpt-5.6-terra', 'gemini-3.6-flash'];
export const DEFAULT_MODELS = ['claude-sonnet-5', 'gpt-5.6-terra', 'gemini-3.7-flash'];
export function resolveModelList() {
const override = process.env.IMPECCABLE_SKILL_BEHAVIOR_MODELS;