Free profile: proposal reviews passed in 5 seconds on Gemini Flash-Lite, reading nothing and finding nothing #198
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Seen in run 76 (production
97242c8, Free profile). The operator spotted the review steps' times in the spine:agent.review[spec]5 s,[scope]7 s,[test]5 s,[risk]15 s,agent.review-proposal0 s. The transcripts:write.osf/review/spec.json."writeTwo lenses read nothing and wrote empty findings in under 100 tokens. The proposal under review was a 6-capability game spec with 9 task groups. On vLLM the same lenses take minutes and find things (runs 65, 66).
At the time, the model the Free profile holds for judging and code review (gemini-3.8-flash, at the judge chain's head) was
breaker_open, so the lenses fell to Flash-Lite, the weakest model in the chain. Nothing noticed that the review was empty.To do
agent.review-proposal) declare on the Free profile, and whether that tier is right for a review: the judge tier's floor, its quality order, and what happens when its head is out.Shipped in
c1228b4(changereviews-show-they-looked): a lens that finds nothing must show it read the change or cite every requirement, else it reads again on the next model, else it is marked not reviewed; the digest names who served each lens. Not yet seen on production (run 78's reviews ran before its apply). Open question in the design: auto-approve answers the review gate even when a lens is not reviewed.Left open: the OpenSpec change is not archived yet, and its tasks are not ticked against production. Deferred when the session wrapped up (2026-10-18).
Not fixed in practice, checked on production (run 78,
c1228b4, whose reviews ran on this build). The spec, test and risk lenses were again served by Gemini 3.5 Flash-Lite with three tool calls each, all writes and no reads, and none was read again. The guard let them pass on its second rule: each lens'scheckednames every requirement heading, which Flash-Lite copies from the change pasted into its prompt (run 78's.osf/review/spec.jsonandserved/spec.json). Citations prove nothing when the prompt holds the headings. Next: drop or harden the citation rule (e.g. require a quoted scenario detail not in the headings, or reads), and reconsider the judge tier's order for lenses. Left open; deferred.Run 89 (Free profile,
thorough, "The wind meter is confusing") passed its proposal review the same way, with the guard from4a9ab81deployed.What the lenses did. All four were served by
gemini/gemini-3.5-flash-liteand took 2 to 4 s:checkedwrite)["Readable direction and speed encoding"]Why the guard let them through. Each lens's record says
"evidence": "checked". The change adds a single requirement, and the lens prompt pastes the change, so naming every requirement took one heading copied from the prompt. A change with one requirement makes the citations rule trivially satisfiable without looking at anything.Why the gate passed anyway. The digest flagged three lenses
quick_no_readsand all fourquick_no_findings, and the gate brief says "answered in 3 s with no findings and no reads". Butauto_approvablestayedtrue, so with auto-approve on,human.review-proposalwas approved unchanged and the run went on to apply. The flags warn a person at the gate; nothing acts on them when nobody is there.Possible fixes.
checkedas evidence on its own when the change has only one or two requirements, or require at least one read alongside it.auto_approvablefalse when a lens is flaggedquick_no_reads(or when fewer lenses than the unattended threshold show reads or reasoning), so an unattended run stops at the gate instead.One more finding from run 89, on the model side.
All four lenses ran on
gemini/gemini-3.5-flash-lite, which reports no reasoning tokens. Production's judge-tier chain listsgemini/gemini-3.8-flashfirst, but it was spent (its free tier allows 20 requests a day), so Flash-Lite, next in the chain, served every lens. The fixes in the comment above would stop the gate passing on these reviews, but they wouldn't make the reviews any better. A lens served by a model that doesn't reason will keep producing empty reviews.Suggested fix: keep models that don't reason out of the review lenses' tier. If none of the remaining free models has quota, the lens should wait for the earliest reset (as other tiers already do) rather than fall back to Flash-Lite. The tier routing for explore and the judge is filed separately.
Shipped in
f2b4f10(changeempty-reviews-hold-the-gate, commit3b7b318, plus the archive ofreviews-show-they-looked). Acheckedlist that names every requirement counts as looking only when the change has three or more; with one or two the lens must also read, reason or find something.auto_approvableis false when a reported lens is flaggedquick_no_reads, or when fewer than three show reads, reasoning or findings; the digest records why inauto_hold, the gate brief prints it, and auto-approve now leaveshuman.review-proposalopen when the digest says so. Simulated runreview-run-89reproduces run 89's four lenses; the default play's lenses now read the change. Tier routing for the lenses is in #218.