Test suite: the full lane takes ~27 minutes, can hang, and is red on main (tier 1 fixes) #227

Closed
opened 2026-10-20 17:22:22 -04:00 by cmoriarty · 1 comment
Owner

The full lane takes about 27 minutes (measured 2026-10-20, 8 cores shared with other sessions): backend 3:01, ui types/unit/build 15 s, Playwright 6:06 (365 tests, 4 workers), sim 18:04 (68 scenarios, 4 jobs). deploy.yml runs no tests (image build and braid-selfcheck only).

Findings behind this issue:

  • A full lane hung for 33 minutes at 97% of the backend suite, every xdist worker idle in a lock wait. There is no per-test timeout, and no guard against two lanes running at once (the hang appeared while two were running; not reproduced alone).
  • The sim lane is contention-bound: quick-fix takes 21 s alone and 154 s inside the lane. plan-to-completion failed on its 240 s cap under load. CLAUDE.md and tests/sim/README.md still say "about a minute".
  • The full lane is red on main: tests/unit/test_spec_limits.py fails on the free-providers requirement "Only probed models are routed" (653 characters, limit 500).
  • tests/unit/test_metrics.py is deselected in tools/test.sh for good, though its docstring describes an acceptance test written before the code.
  • Backend workers average 370% of 800% CPU; one 15 s test sets the tail (xdist default load dist).
  • 135 braid-sim-* directories were left in the temp folder.

First tier (this issue):

  1. Per-test timeout (pytest-timeout) so a hang fails with a stack.
  2. A lane lock in tools/test.sh so two lanes do not run at once on one machine.
  3. xdist --dist worksteal; sim --jobs follows core count and load.
  4. Fix the over-long free-providers requirement; settle test_metrics.py (run it or delete it).
  5. A sim smoke subset (one scenario per pipeline family) in the fast lane, all scenarios in the full lane; correct the stale timing statements.
  6. Sim temp directories removed after a pass.

Later tiers, separate issues: a fixture backend per Playwright worker (8 workers, no serial project), Playwright spec consolidation, in-process scripted steps in the sim, template repos for the backend tests, CI running the fast lane on push. Related: #193 (flaky browser spec under the full lane).

The full lane takes about 27 minutes (measured 2026-10-20, 8 cores shared with other sessions): backend 3:01, ui types/unit/build 15 s, Playwright 6:06 (365 tests, 4 workers), sim 18:04 (68 scenarios, 4 jobs). deploy.yml runs no tests (image build and braid-selfcheck only). Findings behind this issue: - A full lane hung for 33 minutes at 97% of the backend suite, every xdist worker idle in a lock wait. There is no per-test timeout, and no guard against two lanes running at once (the hang appeared while two were running; not reproduced alone). - The sim lane is contention-bound: `quick-fix` takes 21 s alone and 154 s inside the lane. `plan-to-completion` failed on its 240 s cap under load. `CLAUDE.md` and tests/sim/README.md still say "about a minute". - The full lane is red on main: tests/unit/test_spec_limits.py fails on the `free-providers` requirement "Only probed models are routed" (653 characters, limit 500). - tests/unit/test_metrics.py is deselected in tools/test.sh for good, though its docstring describes an acceptance test written before the code. - Backend workers average 370% of 800% CPU; one 15 s test sets the tail (xdist default `load` dist). - 135 `braid-sim-*` directories were left in the temp folder. First tier (this issue): 1. Per-test timeout (pytest-timeout) so a hang fails with a stack. 2. A lane lock in tools/test.sh so two lanes do not run at once on one machine. 3. xdist `--dist worksteal`; sim `--jobs` follows core count and load. 4. Fix the over-long `free-providers` requirement; settle test_metrics.py (run it or delete it). 5. A sim smoke subset (one scenario per pipeline family) in the fast lane, all scenarios in the full lane; correct the stale timing statements. 6. Sim temp directories removed after a pass. Later tiers, separate issues: a fixture backend per Playwright worker (8 workers, no serial project), Playwright spec consolidation, in-process scripted steps in the sim, template repos for the backend tests, CI running the fast lane on push. Related: #193 (flaky browser spec under the full lane).
Author
Owner

Shipped on main (8d3854b, archive to follow in the same push), with #228 and #230 beside it.

What landed

  • pytest-timeout, 120 s by thread: a hung test fails with every thread's stack instead of holding the lane. The suites run by hand against a real server get 600 s.
  • tools/lane-lock.sh: the full and sim lanes hold a machine-wide lock, wait naming the holder, take over a dead holder; the fast lane never waits (OSF_LANE_LOCK=0 turns it off).
  • --dist worksteal: backend step 157 s to 131 s, alone.
  • Simulated runs: smoke: true on one scenario per pipeline family, python -m osf.sim --smoke (78 s alone) at the end of the fast lane; OSF_SIM_JOBS; a bound failure prints the load average; stacks older than a day are swept. Every scenario at 4 jobs is 645 s alone (3 jobs was slower), so the default stays at 4.
  • free-providers' long requirement is under 500 characters, so test_spec_limits is green on main again; test_metrics.py is skipped for #10 instead of deselected on every command line.
  • The hang itself: a parallel-groups test left a scheduler task pending and the xdist worker wedged at exit; the teardown fix is a92dab0 from the #217 session. The timeout bounds that kind of thing, the fixture cures this one.

Times, alone on the 8-core Mac (the morning's figures were inside a lane running beside another session's work): backend 165 s, UI 13 s, browser specs 150 s (was 366), simulated runs about 11 min (was 18 inside the contended lane). The full lane is about 18 minutes now; the fast lane about 7, of which 78 s is the smoke sims.

Left for later, not in this issue: the sim stage is now most of the full lane, and its cost is the real work of each scenario (osfd, opencode, git, the project's own tests), not the harness.

Shipped on main (`8d3854b`, archive to follow in the same push), with #228 and #230 beside it. **What landed** - `pytest-timeout`, 120 s by thread: a hung test fails with every thread's stack instead of holding the lane. The suites run by hand against a real server get 600 s. - `tools/lane-lock.sh`: the full and sim lanes hold a machine-wide lock, wait naming the holder, take over a dead holder; the fast lane never waits (`OSF_LANE_LOCK=0` turns it off). - `--dist worksteal`: backend step 157 s to 131 s, alone. - Simulated runs: `smoke: true` on one scenario per pipeline family, `python -m osf.sim --smoke` (78 s alone) at the end of the fast lane; `OSF_SIM_JOBS`; a bound failure prints the load average; stacks older than a day are swept. Every scenario at 4 jobs is 645 s alone (3 jobs was slower), so the default stays at 4. - `free-providers`' long requirement is under 500 characters, so `test_spec_limits` is green on main again; `test_metrics.py` is skipped for #10 instead of deselected on every command line. - The hang itself: a parallel-groups test left a scheduler task pending and the xdist worker wedged at exit; the teardown fix is `a92dab0` from the #217 session. The timeout bounds that kind of thing, the fixture cures this one. **Times, alone on the 8-core Mac** (the morning's figures were inside a lane running beside another session's work): backend 165 s, UI 13 s, browser specs 150 s (was 366), simulated runs about 11 min (was 18 inside the contended lane). The full lane is about 18 minutes now; the fast lane about 7, of which 78 s is the smoke sims. Left for later, not in this issue: the sim stage is now most of the full lane, and its cost is the real work of each scenario (osfd, opencode, git, the project's own tests), not the harness.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#227
No description provided.