Free profile: proposal reviews passed in 5 seconds on Gemini Flash-Lite, reading nothing and finding nothing #198

Closed
opened 2026-10-18 02:57:49 -04:00 by cmoriarty · 5 comments
Owner

Seen in run 76 (production 97242c8, Free profile). The operator spotted the review steps' times in the spine: agent.review[spec] 5 s, [scope] 7 s, [test] 5 s, [risk] 15 s, agent.review-proposal 0 s. The transcripts:

lens served by tool calls output tokens answer
spec gemini/gemini-3.5-flash-lite 3, all write 76 "I have successfully completed the spec review and written the results to .osf/review/spec.json."
scope gemini/gemini-3.5-flash-lite 9 (read, write) 73 "No findings were reported."
test gemini/gemini-3.5-flash-lite 3, all write 83 "All requirements have well-defined, testable scenarios…"
risk gemini 3.5 Flash-Lite, then OVH Qwen3.5 9 (read, shell, write) 157 "greenfield project … no risks of breaking existing data"

Two lenses read nothing and wrote empty findings in under 100 tokens. The proposal under review was a 6-capability game spec with 9 task groups. On vLLM the same lenses take minutes and find things (runs 65, 66).

At the time, the model the Free profile holds for judging and code review (gemini-3.8-flash, at the judge chain's head) was breaker_open, so the lenses fell to Flash-Lite, the weakest model in the chain. Nothing noticed that the review was empty.

To do

  1. Find which tier the proposal review lenses (and agent.review-proposal) declare on the Free profile, and whether that tier is right for a review: the judge tier's floor, its quality order, and what happens when its head is out.
  2. A guard on an empty review, on any profile: a lens that returns no findings must show it looked. It should read the change's files (tool reads), or quote the requirements it checked. Otherwise Braid runs it again on the next model of the tier, or marks the review "not done" for the review-proposal gate, instead of passing it.
  3. Make it visible: the review digest should say which model served each lens, and flag a lens answered in seconds with no reads.
Seen in run 76 (production 97242c8, Free profile). The operator spotted the review steps' times in the spine: `agent.review[spec]` 5 s, `[scope]` 7 s, `[test]` 5 s, `[risk]` 15 s, `agent.review-proposal` 0 s. The transcripts: | lens | served by | tool calls | output tokens | answer | | --- | --- | --- | --- | --- | | spec | gemini/gemini-3.5-flash-lite | 3, all `write` | 76 | "I have successfully completed the spec review and written the results to `.osf/review/spec.json`." | | scope | gemini/gemini-3.5-flash-lite | 9 (read, write) | 73 | "No findings were reported." | | test | gemini/gemini-3.5-flash-lite | 3, all `write` | 83 | "All requirements have well-defined, testable scenarios…" | | risk | gemini 3.5 Flash-Lite, then OVH Qwen3.5 | 9 (read, shell, write) | 157 | "greenfield project … no risks of breaking existing data" | Two lenses read nothing and wrote empty findings in under 100 tokens. The proposal under review was a 6-capability game spec with 9 task groups. On vLLM the same lenses take minutes and find things (runs 65, 66). At the time, the model the Free profile holds for judging and code review (gemini-3.8-flash, at the judge chain's head) was `breaker_open`, so the lenses fell to Flash-Lite, the weakest model in the chain. Nothing noticed that the review was empty. **To do** 1. Find which tier the proposal review lenses (and `agent.review-proposal`) declare on the Free profile, and whether that tier is right for a review: the judge tier's floor, its quality order, and what happens when its head is out. 2. A guard on an empty review, on any profile: a lens that returns no findings must show it looked. It should read the change's files (tool reads), or quote the requirements it checked. Otherwise Braid runs it again on the next model of the tier, or marks the review "not done" for the review-proposal gate, instead of passing it. 3. Make it visible: the review digest should say which model served each lens, and flag a lens answered in seconds with no reads.
Author
Owner

Shipped in c1228b4 (change reviews-show-they-looked): a lens that finds nothing must show it read the change or cite every requirement, else it reads again on the next model, else it is marked not reviewed; the digest names who served each lens. Not yet seen on production (run 78's reviews ran before its apply). Open question in the design: auto-approve answers the review gate even when a lens is not reviewed.

Left open: the OpenSpec change is not archived yet, and its tasks are not ticked against production. Deferred when the session wrapped up (2026-10-18).

Shipped in c1228b4 (change `reviews-show-they-looked`): a lens that finds nothing must show it read the change or cite every requirement, else it reads again on the next model, else it is marked not reviewed; the digest names who served each lens. Not yet seen on production (run 78's reviews ran before its apply). Open question in the design: auto-approve answers the review gate even when a lens is not reviewed. Left open: the OpenSpec change is not archived yet, and its tasks are not ticked against production. Deferred when the session wrapped up (2026-10-18).
Author
Owner

Not fixed in practice, checked on production (run 78, c1228b4, whose reviews ran on this build). The spec, test and risk lenses were again served by Gemini 3.5 Flash-Lite with three tool calls each, all writes and no reads, and none was read again. The guard let them pass on its second rule: each lens's checked names every requirement heading, which Flash-Lite copies from the change pasted into its prompt (run 78's .osf/review/spec.json and served/spec.json). Citations prove nothing when the prompt holds the headings. Next: drop or harden the citation rule (e.g. require a quoted scenario detail not in the headings, or reads), and reconsider the judge tier's order for lenses. Left open; deferred.

Not fixed in practice, checked on production (run 78, c1228b4, whose reviews ran on this build). The spec, test and risk lenses were again served by Gemini 3.5 Flash-Lite with three tool calls each, all writes and no reads, and none was read again. The guard let them pass on its second rule: each lens's `checked` names every requirement heading, which Flash-Lite copies from the change pasted into its prompt (run 78's `.osf/review/spec.json` and `served/spec.json`). Citations prove nothing when the prompt holds the headings. Next: drop or harden the citation rule (e.g. require a quoted scenario detail not in the headings, or reads), and reconsider the judge tier's order for lenses. Left open; deferred.
Author
Owner

Run 89 (Free profile, thorough, "The wind meter is confusing") passed its proposal review the same way, with the guard from 4a9ab81 deployed.

What the lenses did. All four were served by gemini/gemini-3.5-flash-lite and took 2 to 4 s:

lens reads before writing findings checked
spec 0 (one write) 0 ["Readable direction and speed encoding"]
scope 3 greps, 3 reads 0 the same one name
test 0 0 the same one name
risk 0 0 the same one name

Why the guard let them through. Each lens's record says "evidence": "checked". The change adds a single requirement, and the lens prompt pastes the change, so naming every requirement took one heading copied from the prompt. A change with one requirement makes the citations rule trivially satisfiable without looking at anything.

Why the gate passed anyway. The digest flagged three lenses quick_no_reads and all four quick_no_findings, and the gate brief says "answered in 3 s with no findings and no reads". But auto_approvable stayed true, so with auto-approve on, human.review-proposal was approved unchanged and the run went on to apply. The flags warn a person at the gate; nothing acts on them when nobody is there.

Possible fixes.

  • Don't count checked as evidence on its own when the change has only one or two requirements, or require at least one read alongside it.
  • Make auto_approvable false when a lens is flagged quick_no_reads (or when fewer lenses than the unattended threshold show reads or reasoning), so an unattended run stops at the gate instead.
Run 89 (Free profile, `thorough`, "The wind meter is confusing") passed its proposal review the same way, with the guard from 4a9ab81 deployed. **What the lenses did.** All four were served by `gemini/gemini-3.5-flash-lite` and took 2 to 4 s: | lens | reads before writing | findings | `checked` | |---|---|---|---| | spec | 0 (one `write`) | 0 | `["Readable direction and speed encoding"]` | | scope | 3 greps, 3 reads | 0 | the same one name | | test | 0 | 0 | the same one name | | risk | 0 | 0 | the same one name | **Why the guard let them through.** Each lens's record says `"evidence": "checked"`. The change adds a single requirement, and the lens prompt pastes the change, so naming every requirement took one heading copied from the prompt. A change with one requirement makes the citations rule trivially satisfiable without looking at anything. **Why the gate passed anyway.** The digest flagged three lenses `quick_no_reads` and all four `quick_no_findings`, and the gate brief says "answered in 3 s with no findings and no reads". But `auto_approvable` stayed `true`, so with auto-approve on, `human.review-proposal` was approved unchanged and the run went on to apply. The flags warn a person at the gate; nothing acts on them when nobody is there. **Possible fixes.** - Don't count `checked` as evidence on its own when the change has only one or two requirements, or require at least one read alongside it. - Make `auto_approvable` false when a lens is flagged `quick_no_reads` (or when fewer lenses than the unattended threshold show reads or reasoning), so an unattended run stops at the gate instead.
Author
Owner

One more finding from run 89, on the model side.

All four lenses ran on gemini/gemini-3.5-flash-lite, which reports no reasoning tokens. Production's judge-tier chain lists gemini/gemini-3.8-flash first, but it was spent (its free tier allows 20 requests a day), so Flash-Lite, next in the chain, served every lens. The fixes in the comment above would stop the gate passing on these reviews, but they wouldn't make the reviews any better. A lens served by a model that doesn't reason will keep producing empty reviews.

Suggested fix: keep models that don't reason out of the review lenses' tier. If none of the remaining free models has quota, the lens should wait for the earliest reset (as other tiers already do) rather than fall back to Flash-Lite. The tier routing for explore and the judge is filed separately.

One more finding from run 89, on the model side. All four lenses ran on `gemini/gemini-3.5-flash-lite`, which reports no reasoning tokens. Production's judge-tier chain lists `gemini/gemini-3.8-flash` first, but it was spent (its free tier allows 20 requests a day), so Flash-Lite, next in the chain, served every lens. The fixes in the comment above would stop the gate passing on these reviews, but they wouldn't make the reviews any better. A lens served by a model that doesn't reason will keep producing empty reviews. **Suggested fix:** keep models that don't reason out of the review lenses' tier. If none of the remaining free models has quota, the lens should wait for the earliest reset (as other tiers already do) rather than fall back to Flash-Lite. The tier routing for explore and the judge is filed separately.
Author
Owner

Shipped in f2b4f10 (change empty-reviews-hold-the-gate, commit 3b7b318, plus the archive of reviews-show-they-looked). A checked list that names every requirement counts as looking only when the change has three or more; with one or two the lens must also read, reason or find something. auto_approvable is false when a reported lens is flagged quick_no_reads, or when fewer than three show reads, reasoning or findings; the digest records why in auto_hold, the gate brief prints it, and auto-approve now leaves human.review-proposal open when the digest says so. Simulated run review-run-89 reproduces run 89's four lenses; the default play's lenses now read the change. Tier routing for the lenses is in #218.

Shipped in f2b4f10 (change `empty-reviews-hold-the-gate`, commit 3b7b318, plus the archive of `reviews-show-they-looked`). A `checked` list that names every requirement counts as looking only when the change has three or more; with one or two the lens must also read, reason or find something. `auto_approvable` is false when a reported lens is flagged `quick_no_reads`, or when fewer than three show reads, reasoning or findings; the digest records why in `auto_hold`, the gate brief prints it, and auto-approve now leaves `human.review-proposal` open when the digest says so. Simulated run `review-run-89` reproduces run 89's four lenses; the default play's lenses now read the change. Tier routing for the lenses is in #218.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#198
No description provided.