Step time limits cut working agents: replace hard limits with supervision #303

Closed
opened 2026-10-29 14:04:52 -04:00 by cmoriarty · 3 comments
Owner

What happens
Every agent step has a fixed time limit set per kind of step (timeout_s: code review 1 h, apply 7.5 h). At a tenth of the limit Braid sends a "last call" telling the agent to write its files now, and at the limit it stops the attempt, whatever the agent is doing.

Run 139's code review was 47 min in and steadily productive (a browser playtest reporting each check) when the console read "15m left". A limit cuts healthy work as surely as it catches a loop. Models need time as much as anything; mid and small models also fall into hopeless loops, so some watch is still needed.

Ideas to explore

  • At the limit, a supervisor agent on a spare slot reads the transcript and decides: interrupt it, or give it more time.
  • Ask the agent instead of ordering it: why is this taking long, and does it need more time?
  • Limits that follow observed progress (tool calls, subagent results, files written, tasks ticked), with a hard cap only as a far backstop.
  • Loop signs rather than time: the same tool call, edit or error repeating, no new files, a growing context with nothing written.
  • Tell a person rather than stop, with the evidence (see #223's nudge).

Done when
A proposal compares these on runs 134 to 139's transcripts (time limits hit, loops seen), and picks a design in which a productive step is never stopped for time alone, and a looping one is still caught.

From run 139 (breakout-planned, vLLM). Related: #223, #295 (a slow prefill redone as a stall).

**What happens** Every agent step has a fixed time limit set per kind of step (`timeout_s`: code review 1 h, apply 7.5 h). At a tenth of the limit Braid sends a "last call" telling the agent to write its files now, and at the limit it stops the attempt, whatever the agent is doing. Run 139's code review was 47 min in and steadily productive (a browser playtest reporting each check) when the console read "15m left". A limit cuts healthy work as surely as it catches a loop. Models need time as much as anything; mid and small models also fall into hopeless loops, so some watch is still needed. **Ideas to explore** - At the limit, a supervisor agent on a spare slot reads the transcript and decides: interrupt it, or give it more time. - Ask the agent instead of ordering it: why is this taking long, and does it need more time? - Limits that follow observed progress (tool calls, subagent results, files written, tasks ticked), with a hard cap only as a far backstop. - Loop signs rather than time: the same tool call, edit or error repeating, no new files, a growing context with nothing written. - Tell a person rather than stop, with the evidence (see #223's nudge). **Done when** A proposal compares these on runs 134 to 139's transcripts (time limits hit, loops seen), and picks a design in which a productive step is never stopped for time alone, and a looping one is still caught. _From run 139 (breakout-planned, vLLM). Related: #223, #295 (a slow prefill redone as a stall)._
Author
Owner

It happened minutes after filing. Run 139's agent.code-review attempt 1:

  • 13:02 started, limit 1 h; most of the hour was a browser playtest through window.__app, reporting each check as it passed.
  • 13:56 the last call (6 min left).
  • 14:02:32 timed_out: "stopped at its 1h time limit; .osf/review/code.json is missing or empty".
  • 14:02:33 failure handling continued the same session as attempt 2, with a 2 h limit.

So the review was not looping: it was cut for time, failed, and was restarted under a longer limit, which is a flat limit doubling itself rather than a judgement about the work.

**It happened minutes after filing.** Run 139's `agent.code-review` attempt 1: - 13:02 started, limit 1 h; most of the hour was a browser playtest through `window.__app`, reporting each check as it passed. - 13:56 the last call (6 min left). - 14:02:32 `timed_out`: "stopped at its 1h time limit; `.osf/review/code.json` is missing or empty". - 14:02:33 failure handling continued the same session as attempt 2, with a 2 h limit. So the review was not looping: it was cut for time, failed, and was restarted under a longer limit, which is a flat limit doubling itself rather than a judgement about the work.
Author
Owner

Seen through the engine in simulated runs (PR #316, merged 41f2cad). Before this, only tests of its parts had run the wait loop's progress check. These scenarios all run quick-fix with a 20 s implementer limit:

  • limit-more-time: still writing files at its deadline, the step got extended_s: 40 ("still working: 2 file write(s) in the last ten minutes"), finished at 27 s, and the run succeeded. The console's transcript shows ⏱ 40s more: still working: 2 file write(s) in the last ten minutes. With the extension switched off, the scenario fails: the step is stopped at 20 s.
  • limit-loop: the same command over and over, nothing written. No more time; stopped at its deadline.
  • limit-backstop: writing every few seconds past three times its limit. More time once, then stopped at the backstop.

Not covered: a limit under ten minutes gets no message before its deadline, so these show the check at the deadline, not the one a little before it. This stays open until a step on production shows it. Run 147's steps all ended well inside their limits.

**Seen through the engine in simulated runs (PR #316, merged 41f2cad).** Before this, only tests of its parts had run the wait loop's progress check. These scenarios all run quick-fix with a 20 s implementer limit: - `limit-more-time`: still writing files at its deadline, the step got `extended_s: 40` ("still working: 2 file write(s) in the last ten minutes"), finished at 27 s, and the run succeeded. The console's transcript shows `⏱ 40s more: still working: 2 file write(s) in the last ten minutes`. With the extension switched off, the scenario fails: the step is stopped at 20 s. - `limit-loop`: the same command over and over, nothing written. No more time; stopped at its deadline. - `limit-backstop`: writing every few seconds past three times its limit. More time once, then stopped at the backstop. **Not covered:** a limit under ten minutes gets no message before its deadline, so these show the check at the deadline, not the one a little before it. This stays open until a step on production shows it. Run 147's steps all ended well inside their limits.
Author
Owner

Closing on the simulated runs as proof, the operator's call: "the sim can serve as proof". limit-more-time, limit-loop and limit-backstop run with every simulated set from now on.

Closing on the simulated runs as proof, the operator's call: "the sim can serve as proof". `limit-more-time`, `limit-loop` and `limit-backstop` run with every simulated set from now on.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#303
No description provided.