The agent shell's 120 s limit cuts off long test suites, and a judge passed on a cut-off run (run 89) #219
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happened
The agent shell's 120 s limit cut off every full Playwright run in run 89. This project's suite takes about four minutes (Braid's own e2e lane measured 224 s on run 86).
npx playwright test e2e/hole1.spec.tscut off twice; addingtimeout 300didn't help, because the shell tool's limit applies first.sleep 60. The step took 12 minutes.npx playwright testthree times, cut off each time; the step took 22 minutes.npx playwright testwas cut off after 2 of 15 tests. It still wrote "All automated unit and end-to-end acceptance tests passed successfully" and gave a pass.Why it matters
The time is lost on any backend, but the bigger problem is the false evidence. A judge that passes the change on a cut-off command looks just like one that saw the suite pass.
Suggested fixes
Shipped in
f2b4f10(changeshell-commands-follow-their-output, commit21c70b8). opencode's shell tool cuts a foreground command off at 120 s (a constant, no setting), so Braid's guards plugin now runs each foreground shell command throughidle_run.py: it stops the command only after 120 s of silence or 1,800 s in all, and says in the output that it did not finish (OSF_SHELL_IDLE_S,OSF_SHELL_HARD_S). A judge report that passes on a command that was cut off (Braid's note or opencode's own limit) is held as unfinished and the step is tried again with what was held in its brief. Simulated runshell-long-command. Not measured on a real four-minute Playwright suite yet.