The reconnect backfill test still times out on a loaded CI runner #299

Closed
opened 2026-10-28 22:42:23 -04:00 by cmoriarty · 1 comment
Owner

What happens
test_a_reconnect_backfill_adds_nothing_the_stream_already_delivered still fails on CI after #262's one-write fix. It failed on CI run 246 (PR #297, [tool-error]) and on run 249, the push of #297's merge, which overlapped the deploy ([question] and [cut-off]). Each case ran 1.6 s against the 1 s limit; locally they take 0.01 s. The same 1.6 s in three cases on two runs looks like a fixed wait hit on a slow runner, not general slowness. Main's push run is red because of it.

Fix
Find the wait the slow path hits (a reconnect or backfill timeout near 1.5 s) and make the test not depend on it. If there is none, mark it medium with a reason.

Done when
Five CI runs in a row pass it, including a push that overlaps a deploy.

**What happens** `test_a_reconnect_backfill_adds_nothing_the_stream_already_delivered` still fails on CI after #262's one-write fix. It failed on CI run 246 (PR #297, `[tool-error]`) and on run 249, the push of #297's merge, which overlapped the deploy (`[question]` and `[cut-off]`). Each case ran **1.6 s** against the 1 s limit; locally they take 0.01 s. The same 1.6 s in three cases on two runs looks like a fixed wait hit on a slow runner, not general slowness. Main's push run is red because of it. **Fix** Find the wait the slow path hits (a reconnect or backfill timeout near 1.5 s) and make the test not depend on it. If there is none, mark it medium with a reason. **Done when** Five CI runs in a row pass it, including a push that overlaps a deploy.
Author
Owner

Fixed in #300 (merge 12e8c66, live on production).

Cause: the runner's disk, not a wait in the code. A small test that creates or closes a SQLite file syncs it to disk, and on a busy runner a sync took a second or more. Moving the reconnect test's store into a fixture only moved the failure: run 252 then failed test_migrating_twice_changes_nothing at 1.0 s.

Fix: the CI job's /tmp, where every test's files go, is now a 1 GB tmpfs. A first step fails the job if it isn't. The reconnect test also uses the shared store fixture now, like every other store test.

Done when, met: five runs in a row passed, with the tmpfs check passing in each.

run backend
run 253 5.19 s PR
run 254 5.10 s PR
run 255 5.12 s PR
run 256 5.11 s PR
run 258 5.27 s push to main, while deploy run 257 ran (02:05:59–02:07:49)

Before the tmpfs, the same 198 tests took 6.1–12.0 s (runs 246, 249, 251, 252).

Fixed in #300 (merge 12e8c66, live on production). **Cause:** the runner's disk, not a wait in the code. A small test that creates or closes a SQLite file syncs it to disk, and on a busy runner a sync took a second or more. Moving the reconnect test's store into a fixture only moved the failure: run 252 then failed `test_migrating_twice_changes_nothing` at 1.0 s. **Fix:** the CI job's `/tmp`, where every test's files go, is now a 1 GB tmpfs. A first step fails the job if it isn't. The reconnect test also uses the shared `store` fixture now, like every other store test. **Done when, met:** five runs in a row passed, with the tmpfs check passing in each. | run | backend | | |---|---|---| | run 253 | 5.19 s | PR | | run 254 | 5.10 s | PR | | run 255 | 5.12 s | PR | | run 256 | 5.11 s | PR | | run 258 | 5.27 s | push to main, while deploy run 257 ran (02:05:59–02:07:49) | Before the tmpfs, the same 198 tests took 6.1–12.0 s (runs 246, 249, 251, 252).
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#299
No description provided.