Browser tests share a server within a job, and one fails for it now and then #461

Closed
opened 2026-11-09 11:56:39 -05:00 by cmoriarty · 1 comment
Owner

What happens

The browser tests share state within a CI job. Playwright runs four workers; each worker starts one test server with one database and runs its tests against it in turn. A run an earlier test seeded stays in that database, and the console gives every run under way a tab.

ui/e2e/console.spec.ts:22, "runs open as tabs, [ moves between them and x closes one without touching its run", fails now and then for it: PR 435 (2026-11-08) and PR 460 (2026-11-09), on changes that do not touch it. In the failing job the tab list held five runs, the test's own two and three left by earlier tests on worker 3, and after x the closed run's tab was still there when the 10 s poll gave up. It passes on a rerun and 40 times in a row on the laptop, where the same tests do not share a worker in the same order. Each failure costs a rerun by hand and a merge that waits on it.

The runner is not the cause. Each job starts a fresh container, and the host is at full speed (measured 2026-11-09: SHA-256 of 200 MB in 600 ms on Braid's VM, 3,939 ms on the runner in October).

Fix

Both, decided by the owner on 2026-11-09, who accepts about a second more per test:

  1. Every browser test starts fresh. A test gets a server and database of its own, or the worker's is reset to empty before each test, so no run, tab, gate or setting of an earlier test is there.
  2. Every browser test stands on its own. A test asserts on what it seeded and nothing else: no assumption about what else is in a list, which tab is selected, or the order tests ran in. console.spec.ts:22 first.

docs/testing.md says the browser tests are isolated per test, and the time budget is moved to match.

Done when

  • The browser suite passes with its tests in a shuffled order and with one worker, 20 times in a row, and console.spec.ts:22 passes 50 times in a row with the workers CI uses.
  • A test that seeds a run and a later test on the same worker: the later one sees an empty console.
  • ./tools/test.sh still fits its budget, with the new figure written down.
## What happens The browser tests share state within a CI job. Playwright runs four workers; each worker starts one test server with one database and runs its tests against it in turn. A run an earlier test seeded stays in that database, and the console gives every run under way a tab. `ui/e2e/console.spec.ts:22`, "runs open as tabs, [ moves between them and x closes one without touching its run", fails now and then for it: PR 435 (2026-11-08) and PR 460 (2026-11-09), on changes that do not touch it. In the failing job the tab list held five runs, the test's own two and three left by earlier tests on worker 3, and after `x` the closed run's tab was still there when the 10 s poll gave up. It passes on a rerun and 40 times in a row on the laptop, where the same tests do not share a worker in the same order. Each failure costs a rerun by hand and a merge that waits on it. The runner is not the cause. Each job starts a fresh container, and the host is at full speed (measured 2026-11-09: SHA-256 of 200 MB in 600 ms on Braid's VM, 3,939 ms on the runner in October). ## Fix Both, decided by the owner on 2026-11-09, who accepts about a second more per test: 1. **Every browser test starts fresh.** A test gets a server and database of its own, or the worker's is reset to empty before each test, so no run, tab, gate or setting of an earlier test is there. 2. **Every browser test stands on its own.** A test asserts on what it seeded and nothing else: no assumption about what else is in a list, which tab is selected, or the order tests ran in. `console.spec.ts:22` first. `docs/testing.md` says the browser tests are isolated per test, and the time budget is moved to match. ## Done when - The browser suite passes with its tests in a shuffled order and with one worker, 20 times in a row, and `console.spec.ts:22` passes 50 times in a row with the workers CI uses. - A test that seeds a run and a later test on the same worker: the later one sees an empty console. - `./tools/test.sh` still fits its budget, with the new figure written down.
cmoriarty changed title from The run-tabs browser test fails in CI now and then (console.spec.ts:22) to Browser tests share a server within a job, and one fails for it now and then 2026-11-09 12:13:06 -05:00
Author
Owner

Shipped in PR 468 (main ddfd8a93), archived as batch-2026-11-09b (PR 469).

What changed. Every browser test starts its own server and database and stops it after (ui/e2e/fixtures.ts, the server fixture per test, for test and testOnFree). The run-tabs journey asserts on its two runs alone. A serial pair in console.spec.ts seeds a run in one test and shows the next test on the same worker finds /api/runs empty and no tab. docs/testing.md says the browser tests are isolated per test and carries the figures.

Evidence on the laptop.

  • The suite with one worker, files in a shuffled order: 20 runs in a row, all green (48 to 51 tests, 53 s to 1.3 min each).
  • console.spec.ts:22 with the four workers CI uses: 50 of 50.
  • ./tools/test.sh: green in 46 s; the browser step 29 s on the laptop with four workers (was 13 s before the change on the same laptop, so about a second a test, as expected). CI's browser step on PR 468 is within its 60 s budget.
Shipped in PR 468 (main `ddfd8a93`), archived as `batch-2026-11-09b` (PR 469). **What changed.** Every browser test starts its own server and database and stops it after (`ui/e2e/fixtures.ts`, the `server` fixture per test, for `test` and `testOnFree`). The run-tabs journey asserts on its two runs alone. A serial pair in `console.spec.ts` seeds a run in one test and shows the next test on the same worker finds `/api/runs` empty and no tab. `docs/testing.md` says the browser tests are isolated per test and carries the figures. **Evidence on the laptop.** - The suite with one worker, files in a shuffled order: 20 runs in a row, all green (48 to 51 tests, 53 s to 1.3 min each). - `console.spec.ts:22` with the four workers CI uses: 50 of 50. - `./tools/test.sh`: green in 46 s; the browser step 29 s on the laptop with four workers (was 13 s before the change on the same laptop, so about a second a test, as expected). CI's browser step on PR 468 is within its 60 s budget.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#461
No description provided.