The per-test time limit fails fast tests on a busy machine #393

Closed
opened 2026-11-05 01:28:39 -05:00 by cmoriarty · 1 comment
Owner

What happens
./tools/test.sh goes red on a busy machine with nothing wrong. On 2026-11-05 it failed 3 of 5 runs at a load of 5 to 13: 6 failures in 4 tests, all N s against its 1 s limit, none an assertion. CI went red the same way in 7 runs (231 to 252), before #299's tmpfs. None has since, but a busy runner could do it again.

Why
The limit reads the wall clock. At a load of about 30, test_switching_profiles_brings_back_each_ones_run_settings took 4.2 s on the clock, and 3.7 s of that was waiting for a free core. Its own time was 0.6 s.

Fix
Charge a test for its own time: the wall clock less what the OS reports as waiting for a core. A real wait still counts.

Done when
The suite stays green at a measured load of about 30, and a test made truly slow (a long loop, a long wait) still fails.

Found while verifying #392.

**What happens** `./tools/test.sh` goes red on a busy machine with nothing wrong. On 2026-11-05 it failed 3 of 5 runs at a load of 5 to 13: 6 failures in 4 tests, all `N s against its 1 s limit`, none an assertion. CI went red the same way in 7 runs (231 to 252), before #299's tmpfs. None has since, but a busy runner could do it again. **Why** The limit reads the wall clock. At a load of about 30, `test_switching_profiles_brings_back_each_ones_run_settings` took 4.2 s on the clock, and 3.7 s of that was waiting for a free core. Its own time was 0.6 s. **Fix** Charge a test for its own time: the wall clock less what the OS reports as waiting for a core. A real wait still counts. **Done when** The suite stays green at a measured load of about 30, and a test made truly slow (a long loop, a long wait) still fails. *Found while verifying #392.*
Author
Owner

Fixed in #397, not merged yet, so this stays open until it is. The change is archived on that branch.

The rule: a test is charged its wall clock, less the time the OS reports it waited for a core, less 8 s per CPU-second its children used, and never less than the CPU they used.

Green under load (backend, all 454 tests):

Load Today Now
16 spinners, load up to 33 28 over in 3 runs 3 of 3 clean
32 spinners, load 36 to 51 not run 3 of 3 clean
3 other suites, load up to 33 29 over in 3 runs 3 of 3 clean
Linux container, 16 spinners, load 9 to 22 2, 8, 9 over 8 of 8 clean

Truly slow still fails: a 1.5 s loop, Event().wait(2), asyncio.sleep(2), a child on sleep 5, a child with 4 s of work, and a child with 0.5 s of work then an 8 s wait all fail, quiet and at a load of about 30.

Not met in full: the done-when says the suite stays green. The whole suite passed 5 of 6 loaded runs. The red one was a browser stall that also happens with nothing else running: #396.

Also found: every read of the settings re-parsed the built-in pipeline. That is why test_switching_profiles_brings_back_each_ones_run_settings kept tripping. It is parsed once now.

Fixed in #397, not merged yet, so this stays open until it is. The change is archived on that branch. **The rule:** a test is charged its wall clock, less the time the OS reports it waited for a core, less 8 s per CPU-second its children used, and never less than the CPU they used. **Green under load (backend, all 454 tests):** | Load | Today | Now | |---|---|---| | 16 spinners, load up to 33 | 28 over in 3 runs | 3 of 3 clean | | 32 spinners, load 36 to 51 | not run | 3 of 3 clean | | 3 other suites, load up to 33 | 29 over in 3 runs | 3 of 3 clean | | Linux container, 16 spinners, load 9 to 22 | 2, 8, 9 over | 8 of 8 clean | **Truly slow still fails:** a 1.5 s loop, `Event().wait(2)`, `asyncio.sleep(2)`, a child on `sleep 5`, a child with 4 s of work, and a child with 0.5 s of work then an 8 s wait all fail, quiet and at a load of about 30. **Not met in full:** the done-when says the suite stays green. The whole suite passed 5 of 6 loaded runs. The red one was a browser stall that also happens with nothing else running: #396. **Also found:** every read of the settings re-parsed the built-in pipeline. That is why `test_switching_profiles_brings_back_each_ones_run_settings` kept tripping. It is parsed once now.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#393
No description provided.