A simulated run that fails in the image self-check leaves no log of why #285

Closed
opened 2026-10-28 18:45:35 -04:00 by cmoriarty · 1 comment
Owner

What happens

Deploy run #243 (8c527d6) failed its image self-check: the git-flow simulated run reported the runner failed: RemoteProtocolError: Server disconnected without sending a response at 0.0 s. That is the scenario's own osfd closing the connection on the first POST /api/runs, so the process most likely died mid-request. The self-check keeps no osfd.log for a failed scenario, so why can't be told. The same commit passed the smoke set locally (7 of 7), and a re-dispatched deploy passed. An hour earlier, the PR's CI on the same runner had seven unrelated tests 3 to 10 times over their time limits.

Fix

  • When a scenario fails in the self-check, print the tail of its osfd log (and the fake model's) into the job log.
  • Say "osfd hung up" rather than "the runner failed" when the stack's own osfd drops a request.

Done when

A failed self-check scenario's job log shows why its osfd stopped.

**What happens** Deploy run #243 (8c527d6) failed its image self-check: the `git-flow` simulated run reported `the runner failed: RemoteProtocolError: Server disconnected without sending a response` at 0.0 s. That is the scenario's own osfd closing the connection on the first `POST /api/runs`, so the process most likely died mid-request. The self-check keeps no `osfd.log` for a failed scenario, so why can't be told. The same commit passed the smoke set locally (7 of 7), and a re-dispatched deploy passed. An hour earlier, the PR's CI on the same runner had seven unrelated tests 3 to 10 times over their time limits. **Fix** - When a scenario fails in the self-check, print the tail of its osfd log (and the fake model's) into the job log. - Say "osfd hung up" rather than "the runner failed" when the stack's own osfd drops a request. **Done when** A failed self-check scenario's job log shows why its osfd stopped.
Author
Owner

Shipped in PR #297 (1d98b25). A scenario whose osfd drops a request fails with "osfd hung up on POST /api/runs: RemoteProtocolError: …", and keeps the last 40 lines of its osfd log. python -m osf.sim --logs prints them under the scenario, and the image self-check passes --logs.

The production deploy's self-check passed this time, so the new output had nothing to show; the test drives a scenario whose osfd hangs up on POST /api/runs and checks the failure line and the 40-line log tail.

Shipped in PR #297 (1d98b25). A scenario whose osfd drops a request fails with "osfd hung up on POST /api/runs: RemoteProtocolError: …", and keeps the last 40 lines of its osfd log. `python -m osf.sim --logs` prints them under the scenario, and the image self-check passes `--logs`. The production deploy's self-check passed this time, so the new output had nothing to show; the test drives a scenario whose osfd hangs up on `POST /api/runs` and checks the failure line and the 40-line log tail.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#285
No description provided.