A run-bar test missed a stop and a resume once on CI: the bar counted from 0 #567
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happens
ui/e2e/operator.spec.ts:340, "the bar's elapsed time counts what the run ran and not the time it stood stopped", failed once on CI, on pull request 566's run 569. It opens a running run, then appends a stop at +2 h and a resume. The bar never took them in: it counted0s,1s…10suntil the 10 s wait ran out, where it should have read2h. The suite passed on a laptop the same night, and pull request 566 touches neither the bar nor the stream.Likely cause
The test's events can land before the page's stream has opened. braid#396 fixed that case for the test below it, which holds the stream on purpose. This test does not hold the stream, so on a slow runner it may meet the same window from the other side.
Fix
Reproduce it under load, for example by repeating the test with the runner's CPU limited. Then make the bar take the events whatever the order, or make the test wait for the stream before appending. docs/testing.md asks for a flake to be fixed the day it is seen, or deleted.
Done when
The test passes 200 times in a row on two cores, and a version that appends before the stream opens still reaches
2h.From pull request 566's CI, 2026-11-20.
Seen again on pull request 571's run 575 (2026-11-20), this time by the test below it:
operator.spec.ts:353, "a stop and a resume that land before the stream opens still reach the bar". It failed the same way: the bar counted0s…10sand never read2h.That test holds the stream until both events are in, so the order of the stream and the events is not the whole story. The bar can also miss a stop and a resume that the stream's snapshot carries.
Pull request 571 touches neither the bar nor the stream: it changes the transcript's reveal hand-over (#569). It passed 20 times out of 20 on the laptop, with four workers.
Cause found, in run 569's trace. It is not the test's timing:
Database.readpromises one snapshot (docs/design/contract.md§3), but its connection is autocommit, so each query sees the log as it is at that moment. The detail read the run's intervals before the stop, and the latest seq after the resume, so it claimed to be as new as the resume. The stream's snapshot then looked no newer, so the page skipped its re-read (#396) and the bar kept counting. Load widens the window. It does not make it.The fix is proposed as the change
reads-are-one-snapshot-567: every read runs in one read transaction. A store test makes the race happen without load. Run 575's trace was lost when its CI run was re-run.Shipped in #573 (change
reads-are-one-snapshot-567, archived in #574). Live as4abd6d1since 21:22 UTC on 2026-11-20.Every
Database.readnow runs its function in one read transaction, so all of a read's queries see the same moment of the log, as contract §3 says. The run detail's seq is then true of what it returns, so a stop and a resume that land while the page is still reading the run reach the bar. A read inside another read joins it, and long scans throughreader()stay outside a transaction.Found while checking: under load, one repeat of the held-stream journey failed with
ECONNRESETon its ownPOST /_test/append. Its page had taken 6.5 s to load, and the test server's uvicorn closes an idle kept-alive connection after 5 s. The test server now keeps its connections for the one test it serves.Evidence
ECONNRESET);./tools/test.shpassed in 68 s.On the laptop, two cores could not be pinned as this issue asked. The CI image needs a registry login, which would have stored the token.
docs/testing.mdnow has a row in its table of flakes: a race that load widens. Read the failed run's trace, and download itsplaywright-resultsartifact before re-running.