The agent shell's 120 s limit cuts off long test suites, and a judge passed on a cut-off run (run 89) #219

Closed
opened 2026-10-19 17:10:37 -04:00 by cmoriarty · 1 comment
Owner

What happened

The agent shell's 120 s limit cut off every full Playwright run in run 89. This project's suite takes about four minutes (Braid's own e2e lane measured 224 s on run 86).

  • Apply group 4: npx playwright test e2e/hole1.spec.ts cut off twice; adding timeout 300 didn't help, because the shell tool's limit applies first.
  • agent.write-spec-tests: cut off three times, then started the suite in the background and polled it with sleep 60. The step took 12 minutes.
  • agent.code-review: ran npx playwright test three times, cut off each time; the step took 22 minutes.
  • agent.judge: its one npx playwright test was cut off after 2 of 15 tests. It still wrote "All automated unit and end-to-end acceptance tests passed successfully" and gave a pass.

Why it matters

The time is lost on any backend, but the bigger problem is the false evidence. A judge that passes the change on a cut-off command looks just like one that saw the suite pass.

Suggested fixes

  • Let a test command run longer than 120 s while it keeps producing output, with a hard limit only as a backstop. Or tell the agent to use Braid's lane for the full suite.
  • When a shell call ended on the time limit, say so in the judge's prompt or tool result. Then reject a pass whose evidence cites a command that didn't finish.
## What happened The agent shell's 120 s limit cut off every full Playwright run in run 89. This project's suite takes about four minutes (Braid's own e2e lane measured 224 s on run 86). - **Apply group 4:** `npx playwright test e2e/hole1.spec.ts` cut off twice; adding `timeout 300` didn't help, because the shell tool's limit applies first. - **agent.write-spec-tests:** cut off three times, then started the suite in the background and polled it with `sleep 60`. The step took 12 minutes. - **agent.code-review:** ran `npx playwright test` three times, cut off each time; the step took 22 minutes. - **agent.judge:** its one `npx playwright test` was cut off after 2 of 15 tests. It still wrote "All automated unit and end-to-end acceptance tests passed successfully" and gave a pass. ## Why it matters The time is lost on any backend, but the bigger problem is the false evidence. A judge that passes the change on a cut-off command looks just like one that saw the suite pass. ## Suggested fixes - Let a test command run longer than 120 s while it keeps producing output, with a hard limit only as a backstop. Or tell the agent to use Braid's lane for the full suite. - When a shell call ended on the time limit, say so in the judge's prompt or tool result. Then reject a pass whose evidence cites a command that didn't finish.
Author
Owner

Shipped in f2b4f10 (change shell-commands-follow-their-output, commit 21c70b8). opencode's shell tool cuts a foreground command off at 120 s (a constant, no setting), so Braid's guards plugin now runs each foreground shell command through idle_run.py: it stops the command only after 120 s of silence or 1,800 s in all, and says in the output that it did not finish (OSF_SHELL_IDLE_S, OSF_SHELL_HARD_S). A judge report that passes on a command that was cut off (Braid's note or opencode's own limit) is held as unfinished and the step is tried again with what was held in its brief. Simulated run shell-long-command. Not measured on a real four-minute Playwright suite yet.

Shipped in f2b4f10 (change `shell-commands-follow-their-output`, commit 21c70b8). opencode's shell tool cuts a foreground command off at 120 s (a constant, no setting), so Braid's guards plugin now runs each foreground shell command through `idle_run.py`: it stops the command only after 120 s of silence or 1,800 s in all, and says in the output that it did not finish (`OSF_SHELL_IDLE_S`, `OSF_SHELL_HARD_S`). A judge report that passes on a command that was cut off (Braid's note or opencode's own limit) is held as unfinished and the step is tried again with what was held in its brief. Simulated run `shell-long-command`. Not measured on a real four-minute Playwright suite yet.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#219
No description provided.