Free profile: no model can read screenshots, so UI changes pass with no one looking (run 89) #217

Closed
opened 2026-10-19 17:10:36 -04:00 by cmoriarty · 2 comments
Owner

What happened

Run 89 (Free profile, issue "The wind meter is confusing") replaced the game's wind legend, a purely visual change, and no model ever looked at the result. The judge's report says "browser check skipped: no free model could read screenshots", and the pull request lists "browser screenshot inspection" as not done.

Why

Production's Free profile has no vision tier: /api/free/quota lists no model under it. The catalog's vision chain (gemini-3.5-flash-lite, gemini-3.1-flash-lite, zai/glm-4.6v-flash) is emptied by the probes in src/osf/free/catalog.py:

  • gemini/gemini-3.5-flash-lite: failed, "a screenshot inside a tool result does not reach it through GoModel"
  • zai/glm-4.6v-flash: failed, overloaded through the probe

So on Free, browser_holder returns nobody and every UI change goes unchecked. The Gemini models accept images; the screenshot is lost because it arrives inside a tool result, which GoModel doesn't pass through.

Possible fixes

  • Give the screenshot to the model in a form GoModel passes on, for example as an image part of a user message, rather than inside a tool result. Then re-run the vision probe.
  • Or fix or report the tool-result image handling in GoModel itself.
  • Until there's a vision tier, a UI change on Free should stop at a human gate, saying no model looked at it, rather than pass on "browser check skipped".

#205 lists free-no-vision among the simulated runs not yet written. It would cover the last point.

## What happened Run 89 (Free profile, issue "The wind meter is confusing") replaced the game's wind legend, a purely visual change, and no model ever looked at the result. The judge's report says "browser check skipped: no free model could read screenshots", and the pull request lists "browser screenshot inspection" as not done. ## Why Production's Free profile has no vision tier: `/api/free/quota` lists no model under it. The catalog's vision chain (`gemini-3.5-flash-lite`, `gemini-3.1-flash-lite`, `zai/glm-4.6v-flash`) is emptied by the probes in `src/osf/free/catalog.py`: - `gemini/gemini-3.5-flash-lite`: failed, "a screenshot inside a tool result does not reach it through GoModel" - `zai/glm-4.6v-flash`: failed, overloaded through the probe So on Free, `browser_holder` returns nobody and every UI change goes unchecked. The Gemini models accept images; the screenshot is lost because it arrives inside a tool result, which GoModel doesn't pass through. ## Possible fixes - Give the screenshot to the model in a form GoModel passes on, for example as an image part of a user message, rather than inside a tool result. Then re-run the vision probe. - Or fix or report the tool-result image handling in GoModel itself. - Until there's a vision tier, a UI change on Free should stop at a human gate, saying no model looked at it, rather than pass on "browser check skipped". #205 lists `free-no-vision` among the simulated runs not yet written. It would cover the last point.
Author
Owner

Part shipped in f2b4f10 (change ui-change-nobody-looked-at-stops-for-a-person, commit e313ce8): a Free run with no model that can drive a browser no longer passes a UI change on "browser check skipped". When the judge was due and the round would pass, the closing check opens the recovery gate "Nobody looked, round N", which auto-approve leaves to a person ("Pass it unlooked" / "Stop here"). Simulated runs free-no-vision and free-no-vision-passed (the one #205 lists). Still open here: delivering the screenshot to the model as an image part of a user message instead of inside a tool result, and re-running the vision probe (tools/free_probe.py) and updating PROBES. That needs a live GoModel and a vision model's key, so it was not attempted.

Part shipped in f2b4f10 (change `ui-change-nobody-looked-at-stops-for-a-person`, commit e313ce8): a Free run with no model that can drive a browser no longer passes a UI change on "browser check skipped". When the judge was due and the round would pass, the closing check opens the recovery gate "Nobody looked, round N", which auto-approve leaves to a person ("Pass it unlooked" / "Stop here"). Simulated runs `free-no-vision` and `free-no-vision-passed` (the one #205 lists). Still open here: delivering the screenshot to the model as an image part of a user message instead of inside a tool result, and re-running the vision probe (`tools/free_probe.py`) and updating PROBES. That needs a live GoModel and a vision model's key, so it was not attempted.
Author
Owner

The rest shipped in a92dab0 (change screenshots-reach-free-models-through-look, commit c93b95d), closing this.

The cause, measured on 2026-10-20 against the pinned GoModel 0.1.103 with the Gemini key: the same image, asked "what word is in it?", is read correctly when sent as an image part of a user message (3.5 and 3.1 Flash-Lite both answered ZEPHYR) and answered wrongly when it sits inside a tool result (RECOGNITION, HOPE). GoModel drops an image in a tool result without an error, and a Playwright screenshot arrives in one.

The fix: a look tool, registered in a Free run whose vision tier has a model. It reads a screenshot file from the run's screenshots or tmp folder and sends it to the vision tier as an image part of a user message with the caller's question, returning the answer as text; a path outside those folders, a non-image, or a failed/empty answer comes back as a failure that tells the caller not to guess. The browser subagent and the judge are told the browser tools' own images are not shown to them and that look is how they see one. The vision probe (tools/free_probe.py look) now sends the image that way; Gemini 3.5 and 3.1 Flash-Lite pass it 3 of 3 and are the vision tier again, so a Free run with a Gemini key has the Playwright server, the browser subagent and look. Without a Gemini key the "Nobody looked" gate from the first part still applies. Simulated runs free-look (the fake vision model reports how the image arrived) and the two free-no-vision runs.

Not yet seen on a real Free run: the first UI change on Free after this deploy is the proof; GLM-4.6V-Flash stays marked failed (overloaded) and was not re-probed.

The rest shipped in a92dab0 (change `screenshots-reach-free-models-through-look`, commit c93b95d), closing this. **The cause, measured** on 2026-10-20 against the pinned GoModel 0.1.103 with the Gemini key: the same image, asked "what word is in it?", is read correctly when sent as an image part of a user message (3.5 and 3.1 Flash-Lite both answered ZEPHYR) and answered wrongly when it sits inside a tool result (RECOGNITION, HOPE). GoModel drops an image in a tool result without an error, and a Playwright screenshot arrives in one. **The fix:** a `look` tool, registered in a Free run whose vision tier has a model. It reads a screenshot file from the run's screenshots or tmp folder and sends it to the vision tier as an image part of a user message with the caller's question, returning the answer as text; a path outside those folders, a non-image, or a failed/empty answer comes back as a failure that tells the caller not to guess. The `browser` subagent and the judge are told the browser tools' own images are not shown to them and that `look` is how they see one. The vision probe (`tools/free_probe.py look`) now sends the image that way; Gemini 3.5 and 3.1 Flash-Lite pass it 3 of 3 and are the vision tier again, so a Free run with a Gemini key has the Playwright server, the `browser` subagent and `look`. Without a Gemini key the "Nobody looked" gate from the first part still applies. Simulated runs `free-look` (the fake vision model reports how the image arrived) and the two `free-no-vision` runs. Not yet seen on a real Free run: the first UI change on Free after this deploy is the proof; GLM-4.6V-Flash stays marked failed (overloaded) and was not re-probed.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#217
No description provided.