On Free, the Visual check spirals: a flash-lite browser subagent misses its targets and the judge keeps sending more #338

Closed
opened 2026-10-30 12:14:15 -04:00 by cmoriarty · 1 comment
Owner

What happens
Run 159 (Stocks, Free, subagents on): the Visual check ran 27 min and wrote no report before it was stopped. All 167 Free requests were answered, so the router was fine.

  • The browser subagent runs on the vision tier (Gemini flash-lite). It addressed elements as textbox "Stock search", ref=e26 or [ref=e38] instead of e26, and 8 of its 21 clicks failed, as did nearly every type.
  • Its verdicts didn't follow what it saw (PASS after it couldn't type the symbol), so the judge sent 7 subagents, restarted the app and curled the API.
  • Separately, osfd polls the router's /v1/models without a key: a 401 every 15 s.

Fix

  1. On Free, the judge drives the browser itself, with look for screenshots, even when subagents are on.
  2. The tool guard reads ref=e26 and [ref=e38] as e26.
  3. A Visual check gets at most a few browser subagents, then says "unclear".
  4. The router probe sends the key.

Done when
A Free run's Visual check, with subagents on, drives the browser from the judge and writes its report. A sim covers the holder and the ref targets.

**What happens** Run 159 (Stocks, Free, subagents on): the Visual check ran 27 min and wrote no report before it was stopped. All 167 Free requests were answered, so the router was fine. - The browser subagent runs on the vision tier (Gemini flash-lite). It addressed elements as `textbox "Stock search"`, `ref=e26` or `[ref=e38]` instead of `e26`, and 8 of its 21 clicks failed, as did nearly every type. - Its verdicts didn't follow what it saw (PASS after it couldn't type the symbol), so the judge sent 7 subagents, restarted the app and curled the API. - Separately, osfd polls the router's `/v1/models` without a key: a 401 every 15 s. **Fix** 1. On Free, the judge drives the browser itself, with `look` for screenshots, even when subagents are on. 2. The tool guard reads `ref=e26` and `[ref=e38]` as `e26`. 3. A Visual check gets at most a few browser subagents, then says "unclear". 4. The router probe sends the key. **Done when** A Free run's Visual check, with subagents on, drives the browser from the judge and writes its report. A sim covers the holder and the ref targets.
Author
Owner

Shipped in #339 (deployed with e22494f) and archived as free-judge-holds-the-browser.

Verified by replaying run 159's Visual check (attempt 1) on the laptop with tools/replay.sh (#340):

production, attempt 1 Free replay, today's code
browser subagents 7 0: the judge drove the browser
vision requests 136 6, look only
ended stopped by hand after 66 min, no report report written after 47 min
tool errors many ref= clicks 2 of 153 calls

The scripted replay of the same turns had the 7 browser subagent calls refused. The router probe's 401s were already gone on production (0 in 30 min after the deploy).

Shipped in #339 (deployed with e22494f) and archived as `free-judge-holds-the-browser`. **Verified by replaying run 159's Visual check (attempt 1) on the laptop** with `tools/replay.sh` (#340): | | production, attempt 1 | Free replay, today's code | |---|---|---| | browser subagents | 7 | 0: the judge drove the browser | | vision requests | 136 | 6, `look` only | | ended | stopped by hand after 66 min, no report | report written after 47 min | | tool errors | many `ref=` clicks | 2 of 153 calls | The scripted replay of the same turns had the 7 `browser` subagent calls refused. The router probe's 401s were already gone on production (0 in 30 min after the deploy).
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#338
No description provided.