A run-bar test missed a stop and a resume once on CI: the bar counted from 0 #567

Closed
opened 2026-11-19 23:36:49 -05:00 by cmoriarty · 3 comments
Owner

What happens

ui/e2e/operator.spec.ts:340, "the bar's elapsed time counts what the run ran and not the time it stood stopped", failed once on CI, on pull request 566's run 569. It opens a running run, then appends a stop at +2 h and a resume. The bar never took them in: it counted 0s, 1s … 10s until the 10 s wait ran out, where it should have read 2h. The suite passed on a laptop the same night, and pull request 566 touches neither the bar nor the stream.

Likely cause

The test's events can land before the page's stream has opened. braid#396 fixed that case for the test below it, which holds the stream on purpose. This test does not hold the stream, so on a slow runner it may meet the same window from the other side.

Fix

Reproduce it under load, for example by repeating the test with the runner's CPU limited. Then make the bar take the events whatever the order, or make the test wait for the stream before appending. docs/testing.md asks for a flake to be fixed the day it is seen, or deleted.

Done when

The test passes 200 times in a row on two cores, and a version that appends before the stream opens still reaches 2h.

From pull request 566's CI, 2026-11-20.

**What happens** `ui/e2e/operator.spec.ts:340`, "the bar's elapsed time counts what the run ran and not the time it stood stopped", failed once on CI, on pull request 566's run 569. It opens a running run, then appends a stop at +2 h and a resume. The bar never took them in: it counted `0s`, `1s` … `10s` until the 10 s wait ran out, where it should have read `2h`. The suite passed on a laptop the same night, and pull request 566 touches neither the bar nor the stream. **Likely cause** The test's events can land before the page's stream has opened. braid#396 fixed that case for the test below it, which holds the stream on purpose. This test does not hold the stream, so on a slow runner it may meet the same window from the other side. **Fix** Reproduce it under load, for example by repeating the test with the runner's CPU limited. Then make the bar take the events whatever the order, or make the test wait for the stream before appending. docs/testing.md asks for a flake to be fixed the day it is seen, or deleted. **Done when** The test passes 200 times in a row on two cores, and a version that appends before the stream opens still reaches `2h`. *From pull request 566's CI, 2026-11-20.*
Author
Owner

Seen again on pull request 571's run 575 (2026-11-20), this time by the test below it: operator.spec.ts:353, "a stop and a resume that land before the stream opens still reach the bar". It failed the same way: the bar counted 0s … 10s and never read 2h.

That test holds the stream until both events are in, so the order of the stream and the events is not the whole story. The bar can also miss a stop and a resume that the stream's snapshot carries.

Pull request 571 touches neither the bar nor the stream: it changes the transcript's reveal hand-over (#569). It passed 20 times out of 20 on the laptop, with four workers.

**Seen again** on pull request 571's run 575 (2026-11-20), this time by the test below it: `operator.spec.ts:353`, "a stop and a resume that land before the stream opens still reach the bar". It failed the same way: the bar counted `0s` … `10s` and never read `2h`. That test holds the stream until both events are in, so the order of the stream and the events is not the whole story. The bar can also miss a stop and a resume that the stream's snapshot carries. Pull request 571 touches neither the bar nor the stream: it changes the transcript's reveal hand-over (#569). It passed 20 times out of 20 on the laptop, with four workers.
Author
Owner

Cause found, in run 569's trace. It is not the test's timing:

  • 0.29 s: the page asks for the run's detail, and the busy server takes 210 ms to answer.
  • 0.41 s and 0.50 s: the test's stop and resume commit, while that read is still running.
  • 0.54 s: the stream opens. The page never reads the run again.

Database.read promises one snapshot (docs/design/contract.md §3), but its connection is autocommit, so each query sees the log as it is at that moment. The detail read the run's intervals before the stop, and the latest seq after the resume, so it claimed to be as new as the resume. The stream's snapshot then looked no newer, so the page skipped its re-read (#396) and the bar kept counting. Load widens the window. It does not make it.

The fix is proposed as the change reads-are-one-snapshot-567: every read runs in one read transaction. A store test makes the race happen without load. Run 575's trace was lost when its CI run was re-run.

**Cause found**, in run 569's trace. It is not the test's timing: - 0.29 s: the page asks for the run's detail, and the busy server takes 210 ms to answer. - 0.41 s and 0.50 s: the test's stop and resume commit, while that read is still running. - 0.54 s: the stream opens. The page never reads the run again. `Database.read` promises one snapshot (`docs/design/contract.md` §3), but its connection is autocommit, so each query sees the log as it is at that moment. The detail read the run's intervals before the stop, and the latest seq after the resume, so it claimed to be as new as the resume. The stream's snapshot then looked no newer, so the page skipped its re-read (#396) and the bar kept counting. Load widens the window. It does not make it. The fix is proposed as the change `reads-are-one-snapshot-567`: every read runs in one read transaction. A store test makes the race happen without load. Run 575's trace was lost when its CI run was re-run.
Author
Owner

Shipped in #573 (change reads-are-one-snapshot-567, archived in #574). Live as 4abd6d1 since 21:22 UTC on 2026-11-20.

Every Database.read now runs its function in one read transaction, so all of a read's queries see the same moment of the log, as contract §3 says. The run detail's seq is then true of what it returns, so a stop and a resume that land while the page is still reading the run reach the bar. A read inside another read joins it, and long scans through reader() stay outside a transaction.

Found while checking: under load, one repeat of the held-stream journey failed with ECONNRESET on its own POST /_test/append. Its page had taken 6.5 s to load, and the test server's uvicorn closes an idle kept-alive connection after 5 s. The test server now keeps its connections for the one test it serves.

Evidence

  • A new store test commits a write in the middle of a read. On main the read's first query saw seq 1 and its second saw seq 2. With the fix both agree.
  • Both run-bar journeys ran 200 times each under 12 CPU busy loops with CI's limits:
    • before the keep-alive fix, 399 of 400 (the one ECONNRESET);
    • after it, 400 of 400.
  • ./tools/test.sh passed in 68 s.

On the laptop, two cores could not be pinned as this issue asked. The CI image needs a registry login, which would have stored the token.

docs/testing.md now has a row in its table of flakes: a race that load widens. Read the failed run's trace, and download its playwright-results artifact before re-running.

**Shipped** in #573 (change `reads-are-one-snapshot-567`, archived in #574). Live as 4abd6d1 since 21:22 UTC on 2026-11-20. Every `Database.read` now runs its function in one read transaction, so all of a read's queries see the same moment of the log, as contract §3 says. The run detail's seq is then true of what it returns, so a stop and a resume that land while the page is still reading the run reach the bar. A read inside another read joins it, and long scans through `reader()` stay outside a transaction. Found while checking: under load, one repeat of the held-stream journey failed with `ECONNRESET` on its own `POST /_test/append`. Its page had taken 6.5 s to load, and the test server's uvicorn closes an idle kept-alive connection after 5 s. The test server now keeps its connections for the one test it serves. **Evidence** - A new store test commits a write in the middle of a read. On main the read's first query saw seq 1 and its second saw seq 2. With the fix both agree. - Both run-bar journeys ran 200 times each under 12 CPU busy loops with CI's limits: - before the keep-alive fix, 399 of 400 (the one `ECONNRESET`); - after it, 400 of 400. - `./tools/test.sh` passed in 68 s. On the laptop, two cores could not be pinned as this issue asked. The CI image needs a registry login, which would have stored the token. `docs/testing.md` now has a row in its table of flakes: a race that load widens. Read the failed run's trace, and download its `playwright-results` artifact before re-running.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#567
No description provided.