A browser check takes 5 to 7 minutes #419
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happens. run 181's judge waited 37 minutes on six checks (297 to 407 s each); run 174's were similar.
Fix. measure where a check's time goes (its own turns at
xhigh, screenshots it looks at, waits) before changing anything; the likely levers are the check's effort tier and the number of screenshots it takes.Done when. a measurement comments on this issue with the next step.
From run 181, the verifying run after #407; the log is
docs/design/measurements/work/run-180/findings.md.Measured on run 181's round 1 (six
browserchecks, 36.8 min). Script:docs/design/measurements/work/run-180/browser_checks.py(git-ignored work folder).Where the time goes. Model time is 87% of a check (31.9 of 36.8 min); tools 21% (overlapping). 297 turns at 7.4 s wall each, 6.5 s of it the model: about 50k tokens of context prefilled and a short answer each turn (382 reasoning tokens a turn on average, 115 output). So a check is many short turns, not a few long thoughts: lowering the effort would save little, and the screenshots (26 in all, 4 a check) are not it either.
What the turns are. 350 tool calls: 79
evaluate, 43find, 26 screenshots, 25 clicks, and 80 file reads (53read, 16grep, 11glob) by a subagent whose job is to look at a page it was handed the URL and hook of. Three of six checks ran to the subagent's 60-step cap and were cut.Next step. Take the file tools away from
browser(read,grep,glob): it is briefed with the URL, the hook's names and the probe, and anything it needs about the code is the judge's to put in the brief. That removes about a quarter of the turns and the cut-at-60 endings. Then measure again on the next run. Going into the judge-mechanics change of this batch.Part shipped in PR #421 (
ff9381d8), changejudge-round-two-mechanics; staying open for the re-measurement. Thebrowsersubagent's permissions now denyread,grep,globandlist; its prompt says it cannot read the source and reports UNCLEAR naming what the brief left out; the judge's and implementers' guidance say to brief it with the hook's names and the selectors it needs. Next: after the next Scratch run,docs/design/measurements/work/run-180/browser_checks.pyon its judge steps, and a comment here with the check times against run 181's 5 to 7 minutes.Re-measured on run 182 (ten
browserchecks, 63.2 min, after PR #421 took the file tools away).What changed. No check read a file (run 181: 80 reads in six checks). A short, well-briefed check now takes under a minute ("Join room as Ada", three times). The per-turn cost is unchanged (8 s of model a turn, 372 reasoning tokens), so length is turn count.
What is left. Three checks still ran to the 60-turn cap: the lobby check on 26
wait_forcalls for players its brief promised and the judge had not started, and two "capture" checks on 30evaluates each, which is a check asked to drive a game rather than look at a state. One check took 16.8 min in 5 turns: 11 min inside tools, a wait or a page script that ran its full limit.Next step (for the next batch): a cap on
wait_forper check (three, say) with the refusal naming what the caller should start itself, and the judge's guidance to brief one state per check, with the state reached by the judge's own script. Then measure again.Wait cap shipped in PR #437 (merged
6a8ee372), archived in PR #438 (browser-wait-cap); the issue stays open for the next run's measurement. Abrowsercheck waits at most 6 times, 5 s each:boundWaitclamps abrowser_wait_forof time,browserWaitscounts per browser session, the seventh wait is refused with the words to look at the page now and report what has not come as UNCLEAR. Thebrowserprompt and the judge's briefs quote the numbers and say a brief names one state the script has reached, never a person it has yet to bring in.Evidence:
tests/agent/test_path_guard.py(clamp, text waits untouched, seventh wait refused, second session at nought) andtests/engine/test_browser_delegation.py(both briefs and the plugin carry the numbers). Next:browser_checks.pyon the next Scratch run's judge step, against run 182's 63 min over ten checks.Re-measured on run 183, after the wait cap (PR 437).
Run 183's first-look Visual check started no
browsercheck at all: it looked through its own browser tools. Its round 3 started five, 41.8 of the round's 83 minutes:Next step: none of its own. Since braid#443 a state the check cannot reach is marked
cannot_stageand becomes a test, and a script over the app's modules is refused, so the staging that made these checks long now leaves the Visual check. Run 185 (Greeter) finished its Visual check in 5.6 minutes with nobrowsercheck. Closing on the measurement, as the issue asks; a long check on a later run is a new ticket.