Replace the test suite: every test in about a minute, no flakes (#241) #242

Merged
cmoriarty merged 1 commit from test/rebuild-suite into main 2026-10-22 17:42:19 -04:00
Owner

Closes #241. OpenSpec change: openspec/changes/rebuild-the-test-suite/ (proposal, design, specs, tasks). The standard: docs/testing.md.

Before: 4,234 pytest + ~957 vitest + ~403 Playwright tests and a 5,640-line fixture backend; 23 minutes on CI (run 184), intermittent timing failures (runs 173, 178).

After: 118 backend tests, 40 vitest, 13 browser journeys. tools/test.sh runs all of them in ~15 s on the laptop, and 34 to 40 s with every core busy (yes ×8). Three clean runs and three runs under load all passed, and the browser suite passed 65/65 repeated five times under load.

  • One command, no lanes. CI fails the suite past 180 s (OSF_TEST_BUDGET_S), timeout-minutes: 4.
  • Small tests are hermetic by construction: the guard in tests/conftest.py makes subprocess, non-loopback sockets and time.sleep raise. Medium tests (real git) are marked. Per-test limits are 1 s and 3 s.
  • Browser journeys run the real create_app on a fresh SQLite store, a fake scheduler and an OS-chosen port per worker, with runs seeded through the real event log (tests/support/e2e_server.py). Web-first assertions only, retries: 0.
  • Every test names the requirement / design section / issue it protects; tests/test_repo.py fails on a tag that names nothing.
  • Simulated runs move to sim/ and tools/sim.sh; the image self-check reads sim/.

Bug found by the new tests: a retried POST /api/runs with the same client id started a second run (_prior_seq matched run_id IS NULL). Fixed.

After merge: make test a required check on main.

Closes #241. OpenSpec change: `openspec/changes/rebuild-the-test-suite/` (proposal, design, specs, tasks). The standard: `docs/testing.md`. **Before:** 4,234 pytest + ~957 vitest + ~403 Playwright tests and a 5,640-line fixture backend; 23 minutes on CI (run 184), intermittent timing failures (runs 173, 178). **After:** 118 backend tests, 40 vitest, 13 browser journeys. `tools/test.sh` runs all of them in ~15 s on the laptop, and 34 to 40 s with every core busy (`yes` ×8). Three clean runs and three runs under load all passed, and the browser suite passed 65/65 repeated five times under load. - One command, no lanes. CI fails the suite past 180 s (`OSF_TEST_BUDGET_S`), `timeout-minutes: 4`. - Small tests are hermetic by construction: the guard in `tests/conftest.py` makes subprocess, non-loopback sockets and `time.sleep` raise. Medium tests (real git) are marked. Per-test limits are 1 s and 3 s. - Browser journeys run the real `create_app` on a fresh SQLite store, a fake scheduler and an OS-chosen port per worker, with runs seeded through the real event log (`tests/support/e2e_server.py`). Web-first assertions only, `retries: 0`. - Every test names the requirement / design section / issue it protects; `tests/test_repo.py` fails on a tag that names nothing. - Simulated runs move to `sim/` and `tools/sim.sh`; the image self-check reads `sim/`. **Bug found by the new tests:** a retried `POST /api/runs` with the same client id started a second run (`_prior_seq` matched `run_id IS NULL`). Fixed. After merge: make `test` a required check on `main`.
test: replace the suite with one that runs in a minute (#241)
All checks were successful
ci-image / build (push) Successful in 43s
test / suite (pull_request) Successful in 57s
2280a20c05
The old suite (4,234 pytest, ~957 vitest, ~403 Playwright tests and a
5,640-line fixture backend) took 23 minutes on CI and failed on timing.
It is replaced whole, by a written standard (docs/testing.md) and a
small suite chosen for risk:

- one command, every test: tools/test.sh, about 15 s on a laptop; CI
  fails the suite past 180 s and is a required check
- small tests cannot start a process, reach the network or sleep (a
  guard in tests/conftest.py); medium tests may run git; 1 s / 3 s limits
- browser journeys run the real create_app on a fresh store with a fake
  scheduler, on an OS-chosen port per worker, seeded through the event
  log; web-first assertions only, no retries
- every test names the requirement, design section or issue it protects,
  and tests/test_repo.py fails on a tag that names nothing
- the simulated runs become a tool (tools/sim.sh, scenarios in sim/)

The new API tests found that a retried POST /api/runs with the same
client id started a second run: the dedupe matched `run_id IS NULL`,
which a run's created event never has. Fixed in routes._prior_seq.
cmoriarty deleted branch test/rebuild-suite 2026-10-22 17:42:19 -04:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid!242
No description provided.