A step that hits its time limit fails with no reason: name the cause, show it in the console, and show the clock #116

Closed
opened 2026-10-01 23:12:37 -04:00 by cmoriarty · 1 comment
Owner

On production, run run_01M3WX1VW43K77HWNQ328RBGJK (git-flow, cmoriarty/scratch#5, run #36 in the console) failed at agent.test.browser, and the console gave no reason.

The step ran its whole one-hour limit (timeout_s 3600) and the engine stopped it at 3,603 s, before the agent wrote .osf/test/browser.json. The record says only that the file is missing or empty. The attempt row's error_class and error_text are null; the cause, reason: "timed_out", is only on the osf.attempt.closed event, and nothing in the UI reads it. The console shows ✗ interrupted, which reads like a person stopped it, then ✗ agent.test.browser failed. Step details says "#1 failed". The UI never renders error_text, a running step shows no clock against its limit, and the failure email was not sent (no SMTP credentials on the box).

It is the first timeout in production to fail a step (1 of 650 attempts; the other, on 2026-09-28, still passed its file check). The output limit had the same shape in #101 and was fixed for that cause only.

Asked for:

  1. Record the cause. A step that ends on its time limit or its context budget records error_class and error_text on the attempt ("stopped at its 1h limit after 1h 00m"), and the step's failure leads with it, as #101 did for the output limit.
  2. Show it. The console prints error_text under the failed row in the transcript, on the attempt in Step details, and as a one-line reason beside the Failed chip. When the engine stopped the step, the row before the stop says so instead of ✗ interrupted. The notification headline names the step, not its id.
  3. Show the clock. A running agent step shows how long it has run against its limit, turning amber in the last ten minutes.
On production, run run_01M3WX1VW43K77HWNQ328RBGJK (git-flow, cmoriarty/scratch#5, run #36 in the console) failed at agent.test.browser, and the console gave no reason. The step ran its whole one-hour limit (`timeout_s` 3600) and the engine stopped it at 3,603 s, before the agent wrote `.osf/test/browser.json`. The record says only that the file is missing or empty. The attempt row's `error_class` and `error_text` are null; the cause, `reason: "timed_out"`, is only on the `osf.attempt.closed` event, and nothing in the UI reads it. The console shows `✗ interrupted`, which reads like a person stopped it, then `✗ agent.test.browser failed`. Step details says "#1 failed". The UI never renders `error_text`, a running step shows no clock against its limit, and the failure email was not sent (no SMTP credentials on the box). It is the first timeout in production to fail a step (1 of 650 attempts; the other, on 2026-09-28, still passed its file check). The output limit had the same shape in #101 and was fixed for that cause only. Asked for: 1. **Record the cause.** A step that ends on its time limit or its context budget records `error_class` and `error_text` on the attempt ("stopped at its 1h limit after 1h 00m"), and the step's failure leads with it, as #101 did for the output limit. 2. **Show it.** The console prints `error_text` under the failed row in the transcript, on the attempt in Step details, and as a one-line reason beside the Failed chip. When the engine stopped the step, the row before the stop says so instead of `✗ interrupted`. The notification headline names the step, not its id. 3. **Show the clock.** A running agent step shows how long it has run against its limit, turning amber in the last ten minutes.
Author
Owner

Shipped and deployed, and archived as step-failure-cause. Commits f8405da and 543e54d, live since ac2caaf (2026-10-02 05:12 UTC); production now runs 3bc80ce.

  • A step that reaches its time limit or its context budget records the cause on its attempt (timed_out, budget_exceeded) and its failure leads with it: stopped at its 1h time limit; .osf/test/browser.json is missing or empty. The engine appends osf.step.timed_out before it aborts a step, so the console no longer draws that abort as interrupted, which read as the operator's own Stop.
  • The deadline is recorded (osf.step.deadline, and again whenever the clock restarts; migration 9 adds attempt.timeout_s and attempt.deadline_at). Step details has a time gauge, and the spine's line adds · 13m left in the last quarter of the limit, never earlier than the last quarter of an hour.
  • The console shows the reason under the failed row, on the attempt in Step details and under the Failed chip in the run bar. A failure notification names the step and says why.

Checked on production after the deploy: the database is at version 9 (backed up first), the console of run #36 loads, and its resumed agent.test.browser showed 59m left of 1h in Step details. Attempts from before this carry no reason and no gauge.

Shipped and deployed, and archived as `step-failure-cause`. Commits f8405da and 543e54d, live since ac2caaf (2026-10-02 05:12 UTC); production now runs 3bc80ce. - A step that reaches its time limit or its context budget records the cause on its attempt (`timed_out`, `budget_exceeded`) and its failure leads with it: `stopped at its 1h time limit; .osf/test/browser.json is missing or empty`. The engine appends `osf.step.timed_out` before it aborts a step, so the console no longer draws that abort as `interrupted`, which read as the operator's own Stop. - The deadline is recorded (`osf.step.deadline`, and again whenever the clock restarts; migration 9 adds `attempt.timeout_s` and `attempt.deadline_at`). Step details has a time gauge, and the spine's line adds `· 13m left` in the last quarter of the limit, never earlier than the last quarter of an hour. - The console shows the reason under the failed row, on the attempt in Step details and under the Failed chip in the run bar. A failure notification names the step and says why. Checked on production after the deploy: the database is at version 9 (backed up first), the console of run #36 loads, and its resumed `agent.test.browser` showed `59m left of 1h` in Step details. Attempts from before this carry no reason and no gauge.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cmoriarty/braid#116
No description provided.