Replace the test suite: every test in under 3 minutes on CI, and none flaky #241

Closed
opened 2026-10-22 16:55:53 -04:00 by cmoriarty · 1 comment
Owner

The suite grew to ~5,600 tests (4,234 pytest, ~957 vitest, ~403 Playwright) plus a 5,640-line fixture backend (ui/dev/fakeosfd.py). CI run 184 took 23 minutes (backend 235 s, browser specs 733 s, smoke sims ~5 min) and runs fail intermittently on timing (runs 173, 178).

Replace it whole rather than audit it:

  • one lane, every test, under 3 minutes on CI; it becomes the gate
  • hermetic by construction: no sleeps or wall-clock waits, no fixed ports, real subprocesses only in a small marked tier
  • Playwright runs against the real API with a fake engine and a seeded store, a few critical journeys only
  • tests chosen for the risk they cover, tagged with the requirement they protect, not for coverage
  • the simulated runs become a tool run by hand, not a test
  • a written standard (docs/testing.md + openspec/config.yaml rules) so the suite stays small and fast
The suite grew to ~5,600 tests (4,234 pytest, ~957 vitest, ~403 Playwright) plus a 5,640-line fixture backend (`ui/dev/fakeosfd.py`). CI run 184 took 23 minutes (backend 235 s, browser specs 733 s, smoke sims ~5 min) and runs fail intermittently on timing (runs 173, 178). Replace it whole rather than audit it: - one lane, every test, under 3 minutes on CI; it becomes the gate - hermetic by construction: no sleeps or wall-clock waits, no fixed ports, real subprocesses only in a small marked tier - Playwright runs against the real API with a fake engine and a seeded store, a few critical journeys only - tests chosen for the risk they cover, tagged with the requirement they protect, not for coverage - the simulated runs become a tool run by hand, not a test - a written standard (docs/testing.md + openspec/config.yaml rules) so the suite stays small and fast
Author
Owner

Shipped in #242 (archived in #243).

  • One suite, one command: tools/test.sh. 118 backend tests, 40 vitest, 13 browser journeys. ~15 s on the laptop, 34-40 s with every core busy; CI job 41-57 s including setup (budget 180 s, enforced).
  • Hermetic by construction: small tests cannot spawn processes, reach the network or sleep (guard in tests/conftest.py); medium tests (real git) are marked; 1 s / 3 s limits; no retries anywhere.
  • Browser journeys run the real create_app with a seeded store and a fake scheduler on an OS-chosen port per worker.
  • Every test names the requirement / design section / issue it protects, checked by tests/test_repo.py.
  • The standard for writing tests going forward: docs/testing.md, and the rules are in openspec/config.yaml.
  • Simulated runs moved to sim/ + tools/sim.sh (not part of the suite); the deploy self-check uses them and passed.
  • test / suite (pull_request) is now a required check on main.
  • Bug found by the new tests and fixed: a retried POST /api/runs with the same client id started a second run.
Shipped in #242 (archived in #243). - One suite, one command: `tools/test.sh`. 118 backend tests, 40 vitest, 13 browser journeys. ~15 s on the laptop, 34-40 s with every core busy; **CI job 41-57 s** including setup (budget 180 s, enforced). - Hermetic by construction: small tests cannot spawn processes, reach the network or sleep (guard in `tests/conftest.py`); medium tests (real git) are marked; 1 s / 3 s limits; no retries anywhere. - Browser journeys run the real `create_app` with a seeded store and a fake scheduler on an OS-chosen port per worker. - Every test names the requirement / design section / issue it protects, checked by `tests/test_repo.py`. - The standard for writing tests going forward: `docs/testing.md`, and the rules are in `openspec/config.yaml`. - Simulated runs moved to `sim/` + `tools/sim.sh` (not part of the suite); the deploy self-check uses them and passed. - `test / suite (pull_request)` is now a required check on `main`. - Bug found by the new tests and fixed: a retried `POST /api/runs` with the same client id started a second run.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#241
No description provided.