The image self-check now and then loses a scenario's osfd with no log of why #347

Closed
opened 2026-10-30 15:16:30 -04:00 by cmoriarty · 2 comments
Owner

What happens
Now and then one smoke scenario in the image's self-check loses its osfd mid-run: osfd hung up on GET /api/runs/…: Server disconnected. osfd's log just stops, with no traceback, and the VM's kernel log shows no OOM or segfault. Once, solo's opencode also vanished in the same run (solo still passed). A rerun passes. This failed deploy 292 (c3407b8) and 2 of 5 local self-checks of that image. Deploys 161, 243 and 260 also failed their self-check and passed on a rerun.

Fix
Have the sim runner record how osfd ended when it hangs up (exit code or signal). Then find what sent the signal: a process-group kill that reaches another scenario's processes is the first suspect.

Done when
The self-check passes 10 times in a row in the image.

**What happens** Now and then one smoke scenario in the image's self-check loses its osfd mid-run: `osfd hung up on GET /api/runs/…: Server disconnected`. osfd's log just stops, with no traceback, and the VM's kernel log shows no OOM or segfault. Once, solo's opencode also vanished in the same run (solo still passed). A rerun passes. This failed deploy 292 (c3407b8) and 2 of 5 local self-checks of that image. Deploys 161, 243 and 260 also failed their self-check and passed on a rerun. **Fix** Have the sim runner record how osfd ended when it hangs up (exit code or signal). Then find what sent the signal: a process-group kill that reaches another scenario's processes is the first suspect. **Done when** The self-check passes 10 times in a row in the image.
Author
Owner

A correction on the local evidence: when the laptop's two self-checks failed, its Docker disk had only 74 MB free. Replay image rebuilds had filled it (a uv download later failed with "No space left on device"). After freeing 12 GB, three self-checks of the same image passed. So those two failures may be the full disk, not this flake. Deploy 292 on the CI host is still unexplained, as are 161, 243 and 260.

A correction on the local evidence: when the laptop's two self-checks failed, its Docker disk had only 74 MB free. Replay image rebuilds had filled it (a uv download later failed with "No space left on device"). After freeing 12 GB, three self-checks of the same image passed. So those two failures may be the full disk, not this flake. Deploy 292 on the CI host is still unexplained, as are 161, 243 and 260.
Author
Owner

Fixed in PR 362. osfd was never killed.

Cause: Stack.stop slept on the runner's one event loop while it killed a process group. Two stacks stopping in a row froze every scenario's polling for 8 s. uvicorn closes a connection idle for 5 s, so the next poll went out on a connection being closed, which reads as "Server disconnected". A trace of os.kill and os.killpg in a private image showed no signal reaching osfd at that moment. The later SIGTERM in the log is the scenario's own clean-up.

Fix: kills run on a thread, and the polling client keeps no idle connection. A scenario also now says how its osfd ended, and osfd logs which signal told it to exit.

Evidence: self-check in the image, 2 of 15 failed before (about 53 s each), 30 of 30 passed after (about 35 s each).

Fixed in PR 362. osfd was never killed. **Cause:** `Stack.stop` slept on the runner's one event loop while it killed a process group. Two stacks stopping in a row froze every scenario's polling for 8 s. uvicorn closes a connection idle for 5 s, so the next poll went out on a connection being closed, which reads as "Server disconnected". A trace of `os.kill` and `os.killpg` in a private image showed no signal reaching osfd at that moment. The later SIGTERM in the log is the scenario's own clean-up. **Fix:** kills run on a thread, and the polling client keeps no idle connection. A scenario also now says how its osfd ended, and osfd logs which signal told it to exit. **Evidence:** self-check in the image, 2 of 15 failed before (about 53 s each), 30 of 30 passed after (about 35 s each).
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#347
No description provided.