The time limit charges a test its own time, not the machine's (#393 #395) #397

Merged
cmoriarty merged 8 commits from claude/test-limit-load into main 2026-11-05 08:54:19 -05:00
Owner

A fast backend test no longer fails because the machine is busy, and a test that really is slow still does. Fixes #393 and #395. Merging this deploys, so it is left for you.

What changed

  • The time limit (#393). A test is charged its wall clock, less the time the operating system reports it waited for a core, less 8 s per CPU-second its children used, and never less than the CPU they used. A real wait stays charged. Details are in docs/testing.md and tests/support/timing.py.
  • Save (#395). A real console bug: after a save, the Settings page's Save button turned itself back on, because free_tiers was compared by identity.
  • Settings reads. Every read of /api/settings parsed the built-in pipeline again. It is now parsed once, which also saves production about 30 ms per read.
  • Two tests. The tab journey assumed no other run had a tab. The test that reads all 83 scenario files is now medium.

Evidence

Backend runs clean, at this change's final settings, against the same load on today's limit.

Load Today's limit This change
Laptop, 16 spinners, load up to 33 28 over in 3 runs 3 of 3 clean
Laptop, 32 spinners, load 36 to 51 not run 3 of 3 clean
Laptop, 8 spinners, load 9 to 37 not run 3 of 3 clean
Laptop, 3 other backend suites, load up to 33 29 over in 3 runs 3 of 3 clean
Linux container, 16 spinners, load 9 to 22 2, 8 and 9 over in 3 rounds 8 of 8 clean

Six tests made truly slow (a 1.5 s loop, Event().wait(2), asyncio.sleep(2), a child on sleep 5, a child with 4 s of work, a child with 0.5 s of work then an 8 s wait) all fail on their limit, quiet and at a load of about 30, and a fast test beside them passes. The new hook test failed against today's guard before the fix. CI's run 333 on this branch passed in 44 s.

What this does not settle

  • The whole suite is not always green under load. It passed 5 of 6 runs with 8 spinners or 3 other suites alongside. The red one was a browser stall that also happens with nothing else running, filed as #396, cause unknown. It is not the time limit.
  • At 16 spinners, load near 100 with Chromium on top, the browser journeys fail on their own 10 s waits.
  • The trade-off: a medium test whose children use 0.5 s of CPU can hide a wait of about 6 s, on a quiet machine too. A child that does nothing earns no allowance.

🤖 Generated with Claude Code

A fast backend test no longer fails because the machine is busy, and a test that really is slow still does. Fixes #393 and #395. Merging this deploys, so it is left for you. ## What changed - **The time limit (#393).** A test is charged its wall clock, less the time the operating system reports it waited for a core, less 8 s per CPU-second its children used, and never less than the CPU they used. A real wait stays charged. Details are in `docs/testing.md` and `tests/support/timing.py`. - **Save (#395).** A real console bug: after a save, the Settings page's Save button turned itself back on, because `free_tiers` was compared by identity. - **Settings reads.** Every read of `/api/settings` parsed the built-in pipeline again. It is now parsed once, which also saves production about 30 ms per read. - **Two tests.** The tab journey assumed no other run had a tab. The test that reads all 83 scenario files is now `medium`. ## Evidence Backend runs clean, at this change's final settings, against the same load on today's limit. | Load | Today's limit | This change | |---|---|---| | Laptop, 16 spinners, load up to 33 | 28 over in 3 runs | 3 of 3 clean | | Laptop, 32 spinners, load 36 to 51 | not run | 3 of 3 clean | | Laptop, 8 spinners, load 9 to 37 | not run | 3 of 3 clean | | Laptop, 3 other backend suites, load up to 33 | 29 over in 3 runs | 3 of 3 clean | | Linux container, 16 spinners, load 9 to 22 | 2, 8 and 9 over in 3 rounds | 8 of 8 clean | Six tests made truly slow (a 1.5 s loop, `Event().wait(2)`, `asyncio.sleep(2)`, a child on `sleep 5`, a child with 4 s of work, a child with 0.5 s of work then an 8 s wait) all fail on their limit, quiet and at a load of about 30, and a fast test beside them passes. The new hook test failed against today's guard before the fix. CI's run 333 on this branch passed in 44 s. ## What this does not settle - **The whole suite is not always green under load.** It passed 5 of 6 runs with 8 spinners or 3 other suites alongside. The red one was a browser stall that also happens with nothing else running, filed as #396, cause unknown. It is not the time limit. - **At 16 spinners, load near 100 with Chromium on top,** the browser journeys fail on their own 10 s waits. - **The trade-off:** a medium test whose children use 0.5 s of CPU can hide a wait of about 6 s, on a quiet machine too. A child that does nothing earns no allowance. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The limit read the wall clock, so a busy machine failed tests that were fast. At a load of 25 to
42 a small test took 5.35 s on the clock and 0.63 s of its own: the rest it sat ready to run with
no core free. A test is now charged its wall clock less that wait, as the operating system reports
it, less an allowance for its child processes' same wait, and never less than the CPU they used.
A real wait stays charged, which a limit on CPU time alone could not do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`same()` compared `free_tiers` by identity, and every read of the settings brings a new `{}`. So
the refetch after a save made the form dirty again, Save turned itself back on, and the next save
sent `free_tiers: {}`. The journey at operator.spec.ts:84 only passed when it saw the button before
the refetch landed: 5 of 30 failed at 4 workers, and 30 of 30 pass now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Any run under way gets a tab, and an earlier test in the same worker can leave one, so `[` from
`second` can land on that run instead of `first`. A scratch journey that forced this order failed
as the suite did. `[` still moves one tab left, which is what this now checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It spends its time on real work, PyYAML's pure-Python parser reading 83 files, so the new charge
rightly counts it: 0.2 s of CPU on a quiet laptop, and 1.0 to 1.4 s in a Linux VM on a busy host,
the only test still over its limit there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every GET and POST of /api/settings parsed the default pipeline's YAML again just for its effort
map, about 30 ms each. `test_switching_profiles_brings_back_each_ones_run_settings` did it 8
times, which made it the test that tripped its limit in every red run, and on Linux under load
still did after the wait for a core was taken off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With the load held at about 45, two medium tests still went over, charged 3.3 s for 5.4 s and
4.5 s on the clock. Their children were off a core for longer than 4 s per CPU-second, as the
measurements had shown they could be: up to 8.8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs(openspec): what the load runs showed for the time limit (#393)
All checks were successful
test / suite (pull_request) Successful in 44s
e02c859681
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chore(openspec): archive the time limit that charges a test its own time (#393)
All checks were successful
test / suite (pull_request) Successful in 42s
d8545f3978
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid!397
No description provided.