Free profile: no model can read screenshots, so UI changes pass with no one looking (run 89) #217
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happened
Run 89 (Free profile, issue "The wind meter is confusing") replaced the game's wind legend, a purely visual change, and no model ever looked at the result. The judge's report says "browser check skipped: no free model could read screenshots", and the pull request lists "browser screenshot inspection" as not done.
Why
Production's Free profile has no vision tier:
/api/free/quotalists no model under it. The catalog's vision chain (gemini-3.5-flash-lite,gemini-3.1-flash-lite,zai/glm-4.6v-flash) is emptied by the probes insrc/osf/free/catalog.py:gemini/gemini-3.5-flash-lite: failed, "a screenshot inside a tool result does not reach it through GoModel"zai/glm-4.6v-flash: failed, overloaded through the probeSo on Free,
browser_holderreturns nobody and every UI change goes unchecked. The Gemini models accept images; the screenshot is lost because it arrives inside a tool result, which GoModel doesn't pass through.Possible fixes
#205 lists
free-no-visionamong the simulated runs not yet written. It would cover the last point.Part shipped in
f2b4f10(changeui-change-nobody-looked-at-stops-for-a-person, commite313ce8): a Free run with no model that can drive a browser no longer passes a UI change on "browser check skipped". When the judge was due and the round would pass, the closing check opens the recovery gate "Nobody looked, round N", which auto-approve leaves to a person ("Pass it unlooked" / "Stop here"). Simulated runsfree-no-visionandfree-no-vision-passed(the one #205 lists). Still open here: delivering the screenshot to the model as an image part of a user message instead of inside a tool result, and re-running the vision probe (tools/free_probe.py) and updating PROBES. That needs a live GoModel and a vision model's key, so it was not attempted.The rest shipped in
a92dab0(changescreenshots-reach-free-models-through-look, commitc93b95d), closing this.The cause, measured on 2026-10-20 against the pinned GoModel 0.1.103 with the Gemini key: the same image, asked "what word is in it?", is read correctly when sent as an image part of a user message (3.5 and 3.1 Flash-Lite both answered ZEPHYR) and answered wrongly when it sits inside a tool result (RECOGNITION, HOPE). GoModel drops an image in a tool result without an error, and a Playwright screenshot arrives in one.
The fix: a
looktool, registered in a Free run whose vision tier has a model. It reads a screenshot file from the run's screenshots or tmp folder and sends it to the vision tier as an image part of a user message with the caller's question, returning the answer as text; a path outside those folders, a non-image, or a failed/empty answer comes back as a failure that tells the caller not to guess. Thebrowsersubagent and the judge are told the browser tools' own images are not shown to them and thatlookis how they see one. The vision probe (tools/free_probe.py look) now sends the image that way; Gemini 3.5 and 3.1 Flash-Lite pass it 3 of 3 and are the vision tier again, so a Free run with a Gemini key has the Playwright server, thebrowsersubagent andlook. Without a Gemini key the "Nobody looked" gate from the first part still applies. Simulated runsfree-look(the fake vision model reports how the image arrived) and the twofree-no-visionruns.Not yet seen on a real Free run: the first UI change on Free after this deploy is the proof; GLM-4.6V-Flash stays marked failed (overloaded) and was not re-probed.