A scheduler test hung for 30 s once on CI: the helper in the background #561
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happens
tests/engine/test_scheduler.py::test_a_helper_in_the_background_holds_the_next_agent_stephung once on CI, on PR 560's run 561, attempt 1. pytest-timeout's 30 s thread timeout ended the worker ("worker 'gw0' crashed", "node down: Not properly terminated"), so no stack reached the log. The backend stage took 40.9 s, against 9 to 11 s on every other run that day. The re-run of the same commit passed, and so did 8 runs in a row on a laptop.What it is not
PR 560 touches nothing on this test's path: helper places and admission (
grant_helper_place,sessions_at_work,release_helper_place). The same test passed on its first CI run.Fix
Find what the test waits on when it hangs: each
agents.reached.wait(). Make it fail with a message instead of hanging, for example by waiting with a bound well under 30 s. Then fix the race it shows. docs/testing.md asks for a flake to be fixed the day it is seen, or deleted. The test protects braid#535, so deleting it is the owner's call.Done when
The test cannot hang, and it fails with a message that says what it waited for.
From PR 560's CI, 2026-11-19.
Shipped in pull request 566 (change
scheduler-test-cannot-hang-561).What it was: not a race in the scheduler. Since Python 3.12 a gather of tasks that have all ended completes without yielding. The test's drain,
while sched._running: await gather(...), went round forever while a finished attempt's own done callback, which takes it off_running, waited for a turn of the loop. Production waits on attempts withasyncio.waitand a timeout, and is not affected.Fix: one turn of the loop after each gather. Every wait in these tests now goes through
tests/support/waits.py, which gives up after 10 s naming what it waited for and where each running task is stopped. A drain that goes round 50 times names what is still in flight.Evidence: in
python:3.13-slimon two cores, 1,500 runs at eight workers: 9 to 14 hangs before; with the bounded waits, each failed in under a second with "the attempts in flight never ended: ('stp_two', 1) … done"; after the fix, 3,000 of 3,000 passed.Archived in pull request 568.