Free profile: explore, the judge and the code review run on models that don't reason (run 89) #218

Closed
opened 2026-10-19 17:10:36 -04:00 by cmoriarty · 1 comment
Owner

What happened

On run 89 (Free profile), the explore step and the judge ran on gemini/gemini-3.5-flash-lite, which reports no reasoning tokens:

  • Explore (reader tier) made the run's design decisions on its own: it chose a directional arrow and to change only src/game/screens.ts. The proposal then recorded these as settled.
  • The judge (judge tier) took 155 s with 0 reasoning tokens and passed the change on the strength of tests that it had not seen finish (see the shell time-limit issue).
  • The code review started on gemini/gemini-3.8-flash, which ran out of its 20 free requests a day partway through. The step finished on a mix of Flash-Lite and kilo/reflection/beam-free, and after 22 minutes it found nothing to fix.

Proposal, design, apply and the spec tests ran on kilo/reflection/beam-free, which does reason (thousands of reasoning tokens a step), and those parts of the run held up best.

Suggested fixes

  • Keep models that don't reason out of the judge tier, and out of the reader tier when it serves explore. Where nothing else has quota, wait for the earliest reset rather than fall back.
  • Don't start a step on a model with only a few requests left for the day. Let the remaining quota choose the model at step start, so the step doesn't change models halfway through.

The review lenses' side of this is on #198.

## What happened On run 89 (Free profile), the explore step and the judge ran on `gemini/gemini-3.5-flash-lite`, which reports no reasoning tokens: - **Explore** (reader tier) made the run's design decisions on its own: it chose a directional arrow and to change only `src/game/screens.ts`. The proposal then recorded these as settled. - **The judge** (judge tier) took 155 s with 0 reasoning tokens and passed the change on the strength of tests that it had not seen finish (see the shell time-limit issue). - **The code review** started on `gemini/gemini-3.8-flash`, which ran out of its 20 free requests a day partway through. The step finished on a mix of Flash-Lite and `kilo/reflection/beam-free`, and after 22 minutes it found nothing to fix. Proposal, design, apply and the spec tests ran on `kilo/reflection/beam-free`, which does reason (thousands of reasoning tokens a step), and those parts of the run held up best. ## Suggested fixes - Keep models that don't reason out of the judge tier, and out of the reader tier when it serves explore. Where nothing else has quota, wait for the earliest reset rather than fall back. - Don't start a step on a model with only a few requests left for the day. Let the remaining quota choose the model at step start, so the step doesn't change models halfway through. The review lenses' side of this is on #198.
Author
Owner

Shipped in f2b4f10 (change judge-and-reader-need-models-that-reason, commit 3bb6112). Gemini 3.5 and 3.1 Flash-Lite are out of the judge and reader tiers (strong, fast and vision keep them); a tier with nothing left that has quota waits for the earliest reset. A model with fewer than 10 requests left in a counted window goes behind the tier's others when a chain is put, never out. Note: a Free run with a Gemini key alone is now refused at start, since the reader tier needs Kilo's beam, OVHcloud's Qwen or another key (real runs have Kilo and OVH, which need no key). Simulated run free-judge-skips-flash-lite; free-review-looks-again now reads again on North Mini Code.

Shipped in f2b4f10 (change `judge-and-reader-need-models-that-reason`, commit 3bb6112). Gemini 3.5 and 3.1 Flash-Lite are out of the judge and reader tiers (strong, fast and vision keep them); a tier with nothing left that has quota waits for the earliest reset. A model with fewer than 10 requests left in a counted window goes behind the tier's others when a chain is put, never out. Note: a Free run with a Gemini key alone is now refused at start, since the reader tier needs Kilo's beam, OVHcloud's Qwen or another key (real runs have Kilo and OVH, which need no key). Simulated run `free-judge-skips-flash-lite`; `free-review-looks-again` now reads again on North Mini Code.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#218
No description provided.