Test suite: the full lane takes ~27 minutes, can hang, and is red on main (tier 1 fixes) #227
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The full lane takes about 27 minutes (measured 2026-10-20, 8 cores shared with other sessions): backend 3:01, ui types/unit/build 15 s, Playwright 6:06 (365 tests, 4 workers), sim 18:04 (68 scenarios, 4 jobs). deploy.yml runs no tests (image build and braid-selfcheck only).
Findings behind this issue:
quick-fixtakes 21 s alone and 154 s inside the lane.plan-to-completionfailed on its 240 s cap under load.CLAUDE.mdand tests/sim/README.md still say "about a minute".free-providersrequirement "Only probed models are routed" (653 characters, limit 500).loaddist).braid-sim-*directories were left in the temp folder.First tier (this issue):
--dist worksteal; sim--jobsfollows core count and load.free-providersrequirement; settle test_metrics.py (run it or delete it).Later tiers, separate issues: a fixture backend per Playwright worker (8 workers, no serial project), Playwright spec consolidation, in-process scripted steps in the sim, template repos for the backend tests, CI running the fast lane on push. Related: #193 (flaky browser spec under the full lane).
Shipped on main (
8d3854b, archive to follow in the same push), with #228 and #230 beside it.What landed
pytest-timeout, 120 s by thread: a hung test fails with every thread's stack instead of holding the lane. The suites run by hand against a real server get 600 s.tools/lane-lock.sh: the full and sim lanes hold a machine-wide lock, wait naming the holder, take over a dead holder; the fast lane never waits (OSF_LANE_LOCK=0turns it off).--dist worksteal: backend step 157 s to 131 s, alone.smoke: trueon one scenario per pipeline family,python -m osf.sim --smoke(78 s alone) at the end of the fast lane;OSF_SIM_JOBS; a bound failure prints the load average; stacks older than a day are swept. Every scenario at 4 jobs is 645 s alone (3 jobs was slower), so the default stays at 4.free-providers' long requirement is under 500 characters, sotest_spec_limitsis green on main again;test_metrics.pyis skipped for #10 instead of deselected on every command line.a92dab0from the #217 session. The timeout bounds that kind of thing, the fixture cures this one.Times, alone on the 8-core Mac (the morning's figures were inside a lane running beside another session's work): backend 165 s, UI 13 s, browser specs 150 s (was 366), simulated runs about 11 min (was 18 inside the contended lane). The full lane is about 18 minutes now; the fast lane about 7, of which 78 s is the smoke sims.
Left for later, not in this issue: the sim stage is now most of the full lane, and its cost is the real work of each scenario (osfd, opencode, git, the project's own tests), not the harness.