A browser check takes 5 to 7 minutes #419

Closed
opened 2026-11-08 00:04:10 -05:00 by cmoriarty · 5 comments
Owner

What happens. run 181's judge waited 37 minutes on six checks (297 to 407 s each); run 174's were similar.

Fix. measure where a check's time goes (its own turns at xhigh, screenshots it looks at, waits) before changing anything; the likely levers are the check's effort tier and the number of screenshots it takes.

Done when. a measurement comments on this issue with the next step.

From run 181, the verifying run after #407; the log is docs/design/measurements/work/run-180/findings.md.

**What happens.** run 181's judge waited 37 minutes on six checks (297 to 407 s each); run 174's were similar. **Fix.** measure where a check's time goes (its own turns at `xhigh`, screenshots it looks at, waits) before changing anything; the likely levers are the check's effort tier and the number of screenshots it takes. **Done when.** a measurement comments on this issue with the next step. From run 181, the verifying run after #407; the log is `docs/design/measurements/work/run-180/findings.md`.
Author
Owner

Measured on run 181's round 1 (six browser checks, 36.8 min). Script: docs/design/measurements/work/run-180/browser_checks.py (git-ignored work folder).

check min turns model min tool min screenshots evaluates reads+grep+glob reasoning tokens
Lobby landing page 5.0 60 4.0 1.1 5 7 13 13,856
Lobby entry + deal 6.7 60 5.7 1.9 7 15 20 17,278
Mid-game table 6.7 45 5.9 1.4 5 20 15 21,947
End game + effect 6.1 27 5.5 1.4 1 7 9 20,875
Phone portrait 5.6 45 4.9 0.8 4 16 11 19,209
Dealer + help + motion 6.8 60 5.8 1.2 4 14 12 20,317

Where the time goes. Model time is 87% of a check (31.9 of 36.8 min); tools 21% (overlapping). 297 turns at 7.4 s wall each, 6.5 s of it the model: about 50k tokens of context prefilled and a short answer each turn (382 reasoning tokens a turn on average, 115 output). So a check is many short turns, not a few long thoughts: lowering the effort would save little, and the screenshots (26 in all, 4 a check) are not it either.

What the turns are. 350 tool calls: 79 evaluate, 43 find, 26 screenshots, 25 clicks, and 80 file reads (53 read, 16 grep, 11 glob) by a subagent whose job is to look at a page it was handed the URL and hook of. Three of six checks ran to the subagent's 60-step cap and were cut.

Next step. Take the file tools away from browser (read, grep, glob): it is briefed with the URL, the hook's names and the probe, and anything it needs about the code is the judge's to put in the brief. That removes about a quarter of the turns and the cut-at-60 endings. Then measure again on the next run. Going into the judge-mechanics change of this batch.

**Measured on run 181's round 1 (six `browser` checks, 36.8 min).** Script: `docs/design/measurements/work/run-180/browser_checks.py` (git-ignored work folder). | check | min | turns | model min | tool min | screenshots | evaluates | reads+grep+glob | reasoning tokens | |---|---|---|---|---|---|---|---|---| | Lobby landing page | 5.0 | 60 | 4.0 | 1.1 | 5 | 7 | 13 | 13,856 | | Lobby entry + deal | 6.7 | 60 | 5.7 | 1.9 | 7 | 15 | 20 | 17,278 | | Mid-game table | 6.7 | 45 | 5.9 | 1.4 | 5 | 20 | 15 | 21,947 | | End game + effect | 6.1 | 27 | 5.5 | 1.4 | 1 | 7 | 9 | 20,875 | | Phone portrait | 5.6 | 45 | 4.9 | 0.8 | 4 | 16 | 11 | 19,209 | | Dealer + help + motion | 6.8 | 60 | 5.8 | 1.2 | 4 | 14 | 12 | 20,317 | **Where the time goes.** Model time is 87% of a check (31.9 of 36.8 min); tools 21% (overlapping). 297 turns at 7.4 s wall each, 6.5 s of it the model: about 50k tokens of context prefilled and a short answer each turn (382 reasoning tokens a turn on average, 115 output). So a check is many short turns, not a few long thoughts: lowering the effort would save little, and the screenshots (26 in all, 4 a check) are not it either. **What the turns are.** 350 tool calls: 79 `evaluate`, 43 `find`, 26 screenshots, 25 clicks, and **80 file reads (53 `read`, 16 `grep`, 11 `glob`)** by a subagent whose job is to look at a page it was handed the URL and hook of. Three of six checks ran to the subagent's 60-step cap and were cut. **Next step.** Take the file tools away from `browser` (`read`, `grep`, `glob`): it is briefed with the URL, the hook's names and the probe, and anything it needs about the code is the judge's to put in the brief. That removes about a quarter of the turns and the cut-at-60 endings. Then measure again on the next run. Going into the judge-mechanics change of this batch.
Author
Owner

Part shipped in PR #421 (ff9381d8), change judge-round-two-mechanics; staying open for the re-measurement. The browser subagent's permissions now deny read, grep, glob and list; its prompt says it cannot read the source and reports UNCLEAR naming what the brief left out; the judge's and implementers' guidance say to brief it with the hook's names and the selectors it needs. Next: after the next Scratch run, docs/design/measurements/work/run-180/browser_checks.py on its judge steps, and a comment here with the check times against run 181's 5 to 7 minutes.

**Part shipped in PR #421 (ff9381d8), change `judge-round-two-mechanics`; staying open for the re-measurement.** The `browser` subagent's permissions now deny `read`, `grep`, `glob` and `list`; its prompt says it cannot read the source and reports UNCLEAR naming what the brief left out; the judge's and implementers' guidance say to brief it with the hook's names and the selectors it needs. Next: after the next Scratch run, `docs/design/measurements/work/run-180/browser_checks.py` on its judge steps, and a comment here with the check times against run 181's 5 to 7 minutes.
Author
Owner

Re-measured on run 182 (ten browser checks, 63.2 min, after PR #421 took the file tools away).

check min turns model min tool min screenshots evaluates waits
Lobby: landing, create, join 13.9 60 13.4 12.5 2 0 26
Lobby continuation 1.6 29 1.3 0.5 6 2 0
Table: mid-game state 6.2 31 5.5 0.9 4 11 2
Scoreboard, help, mute 3.0 36 2.5 0.8 4 11 1
Join room as Ada (×3) 0.4–1.0 10–15 1–3 1–2 1–2
Capture game table state 10.5 60 9.4 1.4 2 31 2
Capture table and panels 9.5 60 8.4 1.2 6 30 1
Animations, mute, keyboard, reduced motion 16.8 5 0.1 11.2 0 3 1

What changed. No check read a file (run 181: 80 reads in six checks). A short, well-briefed check now takes under a minute ("Join room as Ada", three times). The per-turn cost is unchanged (8 s of model a turn, 372 reasoning tokens), so length is turn count.

What is left. Three checks still ran to the 60-turn cap: the lobby check on 26 wait_for calls for players its brief promised and the judge had not started, and two "capture" checks on 30 evaluates each, which is a check asked to drive a game rather than look at a state. One check took 16.8 min in 5 turns: 11 min inside tools, a wait or a page script that ran its full limit.

Next step (for the next batch): a cap on wait_for per check (three, say) with the refusal naming what the caller should start itself, and the judge's guidance to brief one state per check, with the state reached by the judge's own script. Then measure again.

**Re-measured on run 182 (ten `browser` checks, 63.2 min, after PR #421 took the file tools away).** | check | min | turns | model min | tool min | screenshots | evaluates | waits | |---|---|---|---|---|---|---|---| | Lobby: landing, create, join | 13.9 | 60 | 13.4 | 12.5 | 2 | 0 | 26 | | Lobby continuation | 1.6 | 29 | 1.3 | 0.5 | 6 | 2 | 0 | | Table: mid-game state | 6.2 | 31 | 5.5 | 0.9 | 4 | 11 | 2 | | Scoreboard, help, mute | 3.0 | 36 | 2.5 | 0.8 | 4 | 11 | 1 | | Join room as Ada (×3) | 0.4–1.0 | 10–15 | | | 1–3 | 1–2 | 1–2 | | Capture game table state | 10.5 | 60 | 9.4 | 1.4 | 2 | 31 | 2 | | Capture table and panels | 9.5 | 60 | 8.4 | 1.2 | 6 | 30 | 1 | | Animations, mute, keyboard, reduced motion | 16.8 | 5 | 0.1 | 11.2 | 0 | 3 | 1 | **What changed.** No check read a file (run 181: 80 reads in six checks). A short, well-briefed check now takes under a minute ("Join room as Ada", three times). The per-turn cost is unchanged (8 s of model a turn, 372 reasoning tokens), so length is turn count. **What is left.** Three checks still ran to the 60-turn cap: the lobby check on 26 `wait_for` calls for players its brief promised and the judge had not started, and two "capture" checks on 30 `evaluate`s each, which is a check asked to drive a game rather than look at a state. One check took 16.8 min in 5 turns: 11 min inside tools, a wait or a page script that ran its full limit. **Next step (for the next batch):** a cap on `wait_for` per check (three, say) with the refusal naming what the caller should start itself, and the judge's guidance to brief one state per check, with the state reached by the judge's own script. Then measure again.
Author
Owner

Wait cap shipped in PR #437 (merged 6a8ee372), archived in PR #438 (browser-wait-cap); the issue stays open for the next run's measurement. A browser check waits at most 6 times, 5 s each: boundWait clamps a browser_wait_for of time, browserWaits counts per browser session, the seventh wait is refused with the words to look at the page now and report what has not come as UNCLEAR. The browser prompt and the judge's briefs quote the numbers and say a brief names one state the script has reached, never a person it has yet to bring in.

Evidence: tests/agent/test_path_guard.py (clamp, text waits untouched, seventh wait refused, second session at nought) and tests/engine/test_browser_delegation.py (both briefs and the plugin carry the numbers). Next: browser_checks.py on the next Scratch run's judge step, against run 182's 63 min over ten checks.

**Wait cap shipped in PR #437 (merged 6a8ee372), archived in PR #438 (`browser-wait-cap`); the issue stays open for the next run's measurement.** A `browser` check waits at most 6 times, 5 s each: `boundWait` clamps a `browser_wait_for` of time, `browserWaits` counts per browser session, the seventh wait is refused with the words to look at the page now and report what has not come as UNCLEAR. The `browser` prompt and the judge's briefs quote the numbers and say a brief names one state the script has reached, never a person it has yet to bring in. Evidence: `tests/agent/test_path_guard.py` (clamp, text waits untouched, seventh wait refused, second session at nought) and `tests/engine/test_browser_delegation.py` (both briefs and the plugin carry the numbers). Next: `browser_checks.py` on the next Scratch run's judge step, against run 182's 63 min over ten checks.
Author
Owner

Re-measured on run 183, after the wait cap (PR 437).

Run 183's first-look Visual check started no browser check at all: it looked through its own browser tools. Its round 3 started five, 41.8 of the round's 83 minutes:

check min turns what it mostly did
Portrait play 5.4 38 14 evaluates, 7 finds, 6 clicks, 3 waits
Waiting-room leave 1.2 23 6 finds, 4 screenshots
End-screen leave and sound 11.2 29 10 evaluates, 3 waits
End-screen leave and sound, again 2.3 25 9 finds, 3 screenshots
Shared win end screen 21.7 50 42 evaluates
  • The wait cap was never reached: 3, 2, 3, 0 and 0 waits.
  • A check that looks at a state its caller reached takes one to two minutes.
  • The long ones drove the app to a state. "Shared win" is the tie the Visual check could not stage: 42 page scripts in 50 turns.

Next step: none of its own. Since braid#443 a state the check cannot reach is marked cannot_stage and becomes a test, and a script over the app's modules is refused, so the staging that made these checks long now leaves the Visual check. Run 185 (Greeter) finished its Visual check in 5.6 minutes with no browser check. Closing on the measurement, as the issue asks; a long check on a later run is a new ticket.

**Re-measured on run 183, after the wait cap (PR 437).** Run 183's first-look Visual check started no `browser` check at all: it looked through its own browser tools. Its round 3 started five, 41.8 of the round's 83 minutes: | check | min | turns | what it mostly did | |---|---:|---:|---| | Portrait play | 5.4 | 38 | 14 evaluates, 7 finds, 6 clicks, 3 waits | | Waiting-room leave | 1.2 | 23 | 6 finds, 4 screenshots | | End-screen leave and sound | 11.2 | 29 | 10 evaluates, 3 waits | | End-screen leave and sound, again | 2.3 | 25 | 9 finds, 3 screenshots | | Shared win end screen | 21.7 | 50 | 42 evaluates | - The wait cap was never reached: 3, 2, 3, 0 and 0 waits. - A check that looks at a state its caller reached takes one to two minutes. - The long ones drove the app to a state. "Shared win" is the tie the Visual check could not stage: 42 page scripts in 50 turns. **Next step: none of its own.** Since braid#443 a state the check cannot reach is marked `cannot_stage` and becomes a test, and a script over the app's modules is refused, so the staging that made these checks long now leaves the Visual check. Run 185 (Greeter) finished its Visual check in 5.6 minutes with no `browser` check. Closing on the measurement, as the issue asks; a long check on a later run is a new ticket.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#419
No description provided.