browser subagents run synchronously, hit the 40-step cap, and the apply quota counts sessions: 76 min idle or improvising in run 172 #381

Closed
opened 2026-11-04 17:31:20 -05:00 by cmoriarty · 3 comments
Owner

What happens
In run 172's openspec.apply[all] the agent made six browser subagent calls (238, 162, 147, 350, 967 and 503 s); every one ended with "Maximum steps reached", and the main agent sat idle for each (a 980 s step at 09:49:47). The seventh call was refused: the step "has had 4 browser checks". The quota counted sessions, not calls, and bit after six. With nothing telling it to stop verifying, the agent then spent 10:02-10:33 writing its own Playwright harness: three race bugs in the harness, ten sleep polls from 5 to 280 s (about 14 min asleep). The judge rounds show the same cap: 25 browser subagents over three rounds, 4-5 hit the cap and needed a continuation brief of 2-5k chars written by the slow main model ("Finish check 2 (free retry)"), and a respawn resets the quota (r3 used 10).

Fix

  • Let browser run with background: true like test, so the main agent keeps working.
  • Count the cap in actions (clicks, navigations, evaluates), not find/tabs/screenshot calls, or raise it to ~60.
  • Make the quota count calls and survive a respawn; when it is spent, say "stop verifying: the lanes and the Visual check follow" instead of refusing.

Done when
An apply step's browser checks finish in the background, a pass is not cut mid-check, and a spent quota ends verification instead of starting a hand-written harness. ~45 + 31 min on run 172.

From the run 172 transcript review (Scratch on vLLM + MTP k=4, 2026-11-04; report in the owner's ~/repos/vLLM/reports/). Related: braid#378, braid#379.

**What happens** In run 172's `openspec.apply[all]` the agent made six `browser` subagent calls (238, 162, 147, 350, 967 and 503 s); every one ended with "Maximum steps reached", and the main agent sat idle for each (a 980 s step at 09:49:47). The seventh call was refused: the step "has had 4 browser checks". The quota counted sessions, not calls, and bit after six. With nothing telling it to stop verifying, the agent then spent 10:02-10:33 writing its own Playwright harness: three race bugs in the harness, ten `sleep` polls from 5 to 280 s (about 14 min asleep). The judge rounds show the same cap: 25 browser subagents over three rounds, 4-5 hit the cap and needed a continuation brief of 2-5k chars written by the slow main model ("Finish check 2 (free retry)"), and a respawn resets the quota (r3 used 10). **Fix** - Let `browser` run with `background: true` like `test`, so the main agent keeps working. - Count the cap in actions (clicks, navigations, evaluates), not `find`/`tabs`/`screenshot` calls, or raise it to ~60. - Make the quota count calls and survive a respawn; when it is spent, say "stop verifying: the lanes and the Visual check follow" instead of refusing. **Done when** An apply step's browser checks finish in the background, a pass is not cut mid-check, and a spent quota ends verification instead of starting a hand-written harness. ~45 + 31 min on run 172. _From the run 172 transcript review (Scratch on vLLM + MTP k=4, 2026-11-04; report in the owner's `~/repos/vLLM/reports/`). Related: braid#378, braid#379._
Author
Owner

Shipped in PR #390, production ddd4b6b.

  • A building step's brief says to start browser in the background.
  • A check has 60 turns, up from 40, and an attempt has six checks across its sessions.

Measured on run 172: browser already offered background: true, but the brief said it only of test. All 24 calls in apply and rounds 2 and 3 ran in the foreground. Apply's one test call did use the background.

Evidence: tests/engine/test_browser_delegation.py, and sims browser-checks-capped and browser-check-out-of-turns.

Staying open: no model has been seen using the background. A Free run has no browser subagent, since its primary drives the browser itself, so a Free replay cannot show it. A Free replay of run 172's apply confirmed that: it succeeded in 17.4 minutes with no subagent calls. trogdor was busy elsewhere.

Done when: on a vLLM run, apply's browser calls carry background: true and the step keeps working while they run.

**Shipped** in PR #390, production ddd4b6b. - A building step's brief says to start `browser` in the background. - A check has 60 turns, up from 40, and an attempt has six checks across its sessions. **Measured on run 172:** `browser` already offered `background: true`, but the brief said it only of `test`. All 24 calls in apply and rounds 2 and 3 ran in the foreground. Apply's one `test` call did use the background. **Evidence:** `tests/engine/test_browser_delegation.py`, and sims `browser-checks-capped` and `browser-check-out-of-turns`. **Staying open: no model has been seen using the background.** A Free run has no `browser` subagent, since its primary drives the browser itself, so a Free replay cannot show it. A Free replay of run 172's apply confirmed that: it succeeded in 17.4 minutes with no subagent calls. trogdor was busy elsewhere. **Done when:** on a vLLM run, apply's `browser` calls carry `background: true` and the step keeps working while they run.
Author
Owner

Shown on a Free replay of run 172's openspec.apply[all], on the build of PR #400 (braid#399 gives a Free run a browser subagent on its judge tier):

  • succeeded in 11.1 min, 86 tool calls, one browser subagent
  • the primary started it, said "Waiting for the browser check to finish" and ended its turn; the report came back as a <subagent state="completed"> message, which is the background path, not a tool result
  • the check ran on the judge tier (hetzner/Qwen3.8-27B ×4, ovh/Qwen3.5-397B ×2), never Flash-Lite, and reported PASS with a screenshot

The remaining step was "on a vLLM run, apply's browser calls carry background: true and the step keeps working". The Free replay shows the same brief doing that. Closing once #400 merges.

**Shown on a Free replay** of run 172's `openspec.apply[all]`, on the build of PR #400 (braid#399 gives a Free run a `browser` subagent on its judge tier): - succeeded in 11.1 min, 86 tool calls, one `browser` subagent - the primary started it, said "Waiting for the browser check to finish" and ended its turn; the report came back as a `<subagent state="completed">` message, which is the background path, not a tool result - the check ran on the judge tier (hetzner/Qwen3.8-27B ×4, ovh/Qwen3.5-397B ×2), never Flash-Lite, and reported PASS with a screenshot The remaining step was "on a vLLM run, apply's `browser` calls carry `background: true` and the step keeps working". The Free replay shows the same brief doing that. Closing once #400 merges.
Author
Owner

Closing on the Free replay above: with PR #400 merged (2b6f451b), a Free run's apply starts browser in the background and keeps working. The next vLLM run is the soak test.

Closing on the Free replay above: with PR #400 merged (2b6f451b), a Free run's apply starts `browser` in the background and keeps working. The next vLLM run is the soak test.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#381
No description provided.