test_interrupted: attempts keep dispatching after a run is stopped (flaky under load) #237
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Flaky under load, seen in CI and locally; the scheduler, not the test's timing, looks responsible.
tests/unit/test_interrupted.py::test_a_step_stopped_with_its_run_is_interrupted_by_the_run->drain()raisesAssertionError: attempts never settled(tests/unit/test_scheduler.py:237).draingatherssched._running200 times, so this means attempts kept being dispatched afterstop_run, not that one hung.scheduler.py:3547 entry checkpoint for impl#1 failed (GitError: git rev-parse ... not a git repository).pytest -n 6 tests/unit/test_interrupted.py tests/unit/test_scheduler.py): 1 of 15 runs failed with no load; with 8 cores loaded, 4 of 5 runs failed and 1 hung for more than 10 min. Under load it also failedtest_a_stop_ends_a_command_the_shell_forked_and_leaves_none_of_it_runningandtest_a_scripted_step_stopped_with_its_run_is_interrupted_not_failed_on_its_exit_code, ending inWriterDeadErrorfrom_run_attempt -> release (scheduler.py:2492)after the db closed.Worth checking: whether a stopped run's step can be re-dispatched (a retry scheduled before the stop lands), and whether an attempt can outlive the test's db.
Another one of this family, on a YAML-only change (PR #240, Actions run 220): xdist worker
gw5crashed intests/unit/test_runtime.py::test_an_operators_control_event_reaches_the_running_executor[human.abort-interrupt]withRuntimeError: Event loop is closedfrom asyncio's pipe teardown (unix_events._close->call_soonon a closed loop). Like thetest_interruptedfailures: an interrupt path whose subprocess or task outlives the test's event loop.What shipped
Test
tests/engine/test_scheduler.py: a tick that read a run before it was stopped opens nothing for it.The old flaky tests went with the suite rebuild (#241); this fixes the cause they pointed at.
Shipped in
5598ddf(OpenSpec changereliable-dispatch-merge-and-ci-evidence, archived).