The image self-check now and then loses a scenario's osfd with no log of why #347
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happens
Now and then one smoke scenario in the image's self-check loses its osfd mid-run:
osfd hung up on GET /api/runs/…: Server disconnected. osfd's log just stops, with no traceback, and the VM's kernel log shows no OOM or segfault. Once, solo's opencode also vanished in the same run (solo still passed). A rerun passes. This failed deploy 292 (c3407b8) and 2 of 5 local self-checks of that image. Deploys 161, 243 and 260 also failed their self-check and passed on a rerun.Fix
Have the sim runner record how osfd ended when it hangs up (exit code or signal). Then find what sent the signal: a process-group kill that reaches another scenario's processes is the first suspect.
Done when
The self-check passes 10 times in a row in the image.
A correction on the local evidence: when the laptop's two self-checks failed, its Docker disk had only 74 MB free. Replay image rebuilds had filled it (a uv download later failed with "No space left on device"). After freeing 12 GB, three self-checks of the same image passed. So those two failures may be the full disk, not this flake. Deploy 292 on the CI host is still unexplained, as are 161, 243 and 260.
Fixed in PR 362. osfd was never killed.
Cause:
Stack.stopslept on the runner's one event loop while it killed a process group. Two stacks stopping in a row froze every scenario's polling for 8 s. uvicorn closes a connection idle for 5 s, so the next poll went out on a connection being closed, which reads as "Server disconnected". A trace ofos.killandos.killpgin a private image showed no signal reaching osfd at that moment. The later SIGTERM in the log is the scenario's own clean-up.Fix: kills run on a thread, and the polling client keeps no idle connection. A scenario also now says how its osfd ended, and osfd logs which signal told it to exit.
Evidence: self-check in the image, 2 of 15 failed before (about 53 s each), 30 of 30 passed after (about 35 s each).