A scheduler test hung for 30 s once on CI: the helper in the background #561

Closed
opened 2026-11-19 16:35:32 -05:00 by cmoriarty · 1 comment
Owner

What happens

tests/engine/test_scheduler.py::test_a_helper_in_the_background_holds_the_next_agent_step hung once on CI, on PR 560's run 561, attempt 1. pytest-timeout's 30 s thread timeout ended the worker ("worker 'gw0' crashed", "node down: Not properly terminated"), so no stack reached the log. The backend stage took 40.9 s, against 9 to 11 s on every other run that day. The re-run of the same commit passed, and so did 8 runs in a row on a laptop.

What it is not

PR 560 touches nothing on this test's path: helper places and admission (grant_helper_place, sessions_at_work, release_helper_place). The same test passed on its first CI run.

Fix

Find what the test waits on when it hangs: each agents.reached.wait(). Make it fail with a message instead of hanging, for example by waiting with a bound well under 30 s. Then fix the race it shows. docs/testing.md asks for a flake to be fixed the day it is seen, or deleted. The test protects braid#535, so deleting it is the owner's call.

Done when

The test cannot hang, and it fails with a message that says what it waited for.

From PR 560's CI, 2026-11-19.

**What happens** `tests/engine/test_scheduler.py::test_a_helper_in_the_background_holds_the_next_agent_step` hung once on CI, on PR 560's run 561, attempt 1. pytest-timeout's 30 s thread timeout ended the worker ("worker 'gw0' crashed", "node down: Not properly terminated"), so no stack reached the log. The backend stage took 40.9 s, against 9 to 11 s on every other run that day. The re-run of the same commit passed, and so did 8 runs in a row on a laptop. **What it is not** PR 560 touches nothing on this test's path: helper places and admission (`grant_helper_place`, `sessions_at_work`, `release_helper_place`). The same test passed on its first CI run. **Fix** Find what the test waits on when it hangs: each `agents.reached.wait()`. Make it fail with a message instead of hanging, for example by waiting with a bound well under 30 s. Then fix the race it shows. docs/testing.md asks for a flake to be fixed the day it is seen, or deleted. The test protects braid#535, so deleting it is the owner's call. **Done when** The test cannot hang, and it fails with a message that says what it waited for. *From PR 560's CI, 2026-11-19.*
Author
Owner

Shipped in pull request 566 (change scheduler-test-cannot-hang-561).

What it was: not a race in the scheduler. Since Python 3.12 a gather of tasks that have all ended completes without yielding. The test's drain, while sched._running: await gather(...), went round forever while a finished attempt's own done callback, which takes it off _running, waited for a turn of the loop. Production waits on attempts with asyncio.wait and a timeout, and is not affected.

Fix: one turn of the loop after each gather. Every wait in these tests now goes through tests/support/waits.py, which gives up after 10 s naming what it waited for and where each running task is stopped. A drain that goes round 50 times names what is still in flight.

Evidence: in python:3.13-slim on two cores, 1,500 runs at eight workers: 9 to 14 hangs before; with the bounded waits, each failed in under a second with "the attempts in flight never ended: ('stp_two', 1) … done"; after the fix, 3,000 of 3,000 passed.

Archived in pull request 568.

Shipped in pull request 566 (change `scheduler-test-cannot-hang-561`). **What it was:** not a race in the scheduler. Since Python 3.12 a gather of tasks that have all ended completes without yielding. The test's drain, `while sched._running: await gather(...)`, went round forever while a finished attempt's own done callback, which takes it off `_running`, waited for a turn of the loop. Production waits on attempts with `asyncio.wait` and a timeout, and is not affected. **Fix:** one turn of the loop after each gather. Every wait in these tests now goes through `tests/support/waits.py`, which gives up after 10 s naming what it waited for and where each running task is stopped. A drain that goes round 50 times names what is still in flight. **Evidence:** in `python:3.13-slim` on two cores, 1,500 runs at eight workers: 9 to 14 hangs before; with the bounded waits, each failed in under a second with "the attempts in flight never ended: ('stp_two', 1) … done"; after the fix, 3,000 of 3,000 passed. Archived in pull request 568.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#561
No description provided.