test_interrupted: attempts keep dispatching after a run is stopped (flaky under load) #237

Closed
opened 2026-10-21 19:59:28 -04:00 by cmoriarty · 2 comments
Owner

Flaky under load, seen in CI and locally; the scheduler, not the test's timing, looks responsible.

  • CI run 197 (braid-ci image) and run 191 (run #99): tests/unit/test_interrupted.py::test_a_step_stopped_with_its_run_is_interrupted_by_the_run -> drain() raises AssertionError: attempts never settled (tests/unit/test_scheduler.py:237). drain gathers sched._running 200 times, so this means attempts kept being dispatched after stop_run, not that one hung.
  • Captured log each time: scheduler.py:3547 entry checkpoint for impl#1 failed (GitError: git rev-parse ... not a git repository).
  • Local probe (pytest -n 6 tests/unit/test_interrupted.py tests/unit/test_scheduler.py): 1 of 15 runs failed with no load; with 8 cores loaded, 4 of 5 runs failed and 1 hung for more than 10 min. Under load it also failed test_a_stop_ends_a_command_the_shell_forked_and_leaves_none_of_it_running and test_a_scripted_step_stopped_with_its_run_is_interrupted_not_failed_on_its_exit_code, ending in WriterDeadError from _run_attempt -> release (scheduler.py:2492) after the db closed.
  • Also seen in braid run #99's local unit lane (round 2).

Worth checking: whether a stopped run's step can be re-dispatched (a retry scheduled before the stop lands), and whether an attempt can outlive the test's db.

Flaky under load, seen in CI and locally; the scheduler, not the test's timing, looks responsible. - CI run 197 (braid-ci image) and run 191 (run #99): `tests/unit/test_interrupted.py::test_a_step_stopped_with_its_run_is_interrupted_by_the_run` -> `drain()` raises `AssertionError: attempts never settled` (tests/unit/test_scheduler.py:237). `drain` gathers `sched._running` 200 times, so this means attempts kept being dispatched after `stop_run`, not that one hung. - Captured log each time: `scheduler.py:3547 entry checkpoint for impl#1 failed (GitError: git rev-parse ... not a git repository)`. - Local probe (`pytest -n 6 tests/unit/test_interrupted.py tests/unit/test_scheduler.py`): 1 of 15 runs failed with no load; with 8 cores loaded, 4 of 5 runs failed and 1 hung for more than 10 min. Under load it also failed `test_a_stop_ends_a_command_the_shell_forked_and_leaves_none_of_it_running` and `test_a_scripted_step_stopped_with_its_run_is_interrupted_not_failed_on_its_exit_code`, ending in `WriterDeadError` from `_run_attempt -> release (scheduler.py:2492)` after the db closed. - Also seen in braid run #99's local unit lane (round 2). Worth checking: whether a stopped run's step can be re-dispatched (a retry scheduled before the stop lands), and whether an attempt can outlive the test's db.
Author
Owner

Another one of this family, on a YAML-only change (PR #240, Actions run 220): xdist worker gw5 crashed in tests/unit/test_runtime.py::test_an_operators_control_event_reaches_the_running_executor[human.abort-interrupt] with RuntimeError: Event loop is closed from asyncio's pipe teardown (unix_events._close -> call_soon on a closed loop). Like the test_interrupted failures: an interrupt path whose subprocess or task outlives the test's event loop.

Another one of this family, on a YAML-only change (PR #240, Actions run 220): xdist worker `gw5` crashed in `tests/unit/test_runtime.py::test_an_operators_control_event_reaches_the_running_executor[human.abort-interrupt]` with `RuntimeError: Event loop is closed` from asyncio's pipe teardown (`unix_events._close` -> `call_soon` on a closed loop). Like the `test_interrupted` failures: an interrupt path whose subprocess or task outlives the test's event loop.
Author
Owner

What shipped

  • An attempt now opens only in a write that finds its step still ready and its run still live (queued, running or blocked). A tick holding an old list of ready steps can no longer start anything for a stopped run.

Test
tests/engine/test_scheduler.py: a tick that read a run before it was stopped opens nothing for it.

The old flaky tests went with the suite rebuild (#241); this fixes the cause they pointed at.

Shipped in 5598ddf (OpenSpec change reliable-dispatch-merge-and-ci-evidence, archived).

**What shipped** - An attempt now opens only in a write that finds its step still ready and its run still live (queued, running or blocked). A tick holding an old list of ready steps can no longer start anything for a stopped run. **Test** `tests/engine/test_scheduler.py`: a tick that read a run before it was stopped opens nothing for it. The old flaky tests went with the suite rebuild (#241); this fixes the cause they pointed at. _Shipped in 5598ddf (OpenSpec change `reliable-dispatch-merge-and-ci-evidence`, archived)._
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#237
No description provided.