A link's clock follows a source that falls behind steadily (#179) #184

Merged
cmoriarty merged 2 commits from link-clock-drift into main 2026-11-16 18:10:47 -05:00
Owner

Fixes #179.

The fault

A link's clock was the largest segment end timestamp − time first listed of the last 10 minutes. That holds while a stream's delay is steady. On 2026-11-16, shd247.world's sources listed media at 0.90× real time, so their delay grew about 6 s a minute.

  • The game page showed "5.2 s ahead" for a stream about 30 s behind. The swap onto it, confirmed by the sound, went back 43 s.
  • Once the drift passed 30 s, it was taken for "a new stream behind the link" ("its clock moved −34.3 s", "−41.3 s"). The link's pairs were dropped, then found again a second later at the same relation.

Joins were never wrong: they go by the timestamp relation.

The fix

  • offsets._drift measures how fast a link's clock falls behind from its own readings: the slope over the window, once there are three readings a minute apart. It is 0 for a steady source or one catching up.
  • offsets._freshest carries each reading forward by that drift before taking the largest.
  • clock_of (the seconds shown) and _listed's restart check both go by it. A steady drift is no restart; a sudden jump still is.
  • A steady link reads exactly as before.
  • The stream-supervision spec says so: "a source falling steadily behind real time SHALL not count as a restart", with the scenario "A source slower than real time".

Checks

  • New test test_a_source_slower_than_real_time_keeps_its_pairs_and_shows_its_lead_as_it_is_now: a standby at 0.9×, starting 5 s behind.

    • Before the fix it read −5.6 s at t = 200 for a stream 22.5 s behind.
    • Now it is within 1.5 s of the truth at t = 200 and t = 400 (42.5 s behind by then). Its pair is kept and trusted, and nothing is taken for a new stream.
  • The restart tests still drop the pairs on a real jump. All 25 offsets tests pass.

  • make test: 1,485 passed, 4 failed. All four failures were timeouts, while another project's Playwright run on the same Mac held the load average near 150:

    • test_bare_machine_start ("the service process did not answer in time");
    • test_click_to_play_works_in_the_bare_run;
    • test_hand_picked_game_starts_playing_on_click;
    • test_a_comparison_that_runs_too_long_is_said_with_what_it_does_and_the_busy_threads.

    Run alone, the last two passed. test_bare_machine_start fails the same way on #183's branch (no clock change) under the same load, though it passed in that branch's full lane earlier. The lane will be run again once the load drops, and the result added here.

🤖 Generated with Claude Code

Fixes #179. ## The fault A link's clock was the largest `segment end timestamp − time first listed` of the last 10 minutes. That holds while a stream's delay is steady. On 2026-11-16, shd247.world's sources listed media at 0.90× real time, so their delay grew about 6 s a minute. - The game page showed "5.2 s ahead" for a stream about 30 s behind. The swap onto it, confirmed by the sound, went back 43 s. - Once the drift passed 30 s, it was taken for "a new stream behind the link" ("its clock moved −34.3 s", "−41.3 s"). The link's pairs were dropped, then found again a second later at the same relation. Joins were never wrong: they go by the timestamp relation. ## The fix - `offsets._drift` measures how fast a link's clock falls behind from its own readings: the slope over the window, once there are three readings a minute apart. It is 0 for a steady source or one catching up. - `offsets._freshest` carries each reading forward by that drift before taking the largest. - `clock_of` (the seconds shown) and `_listed`'s restart check both go by it. A steady drift is no restart; a sudden jump still is. - A steady link reads exactly as before. - The stream-supervision spec says so: "a source falling steadily behind real time SHALL not count as a restart", with the scenario "A source slower than real time". ## Checks - New test `test_a_source_slower_than_real_time_keeps_its_pairs_and_shows_its_lead_as_it_is_now`: a standby at 0.9×, starting 5 s behind. - Before the fix it read −5.6 s at t = 200 for a stream 22.5 s behind. - Now it is within 1.5 s of the truth at t = 200 and t = 400 (42.5 s behind by then). Its pair is kept and trusted, and nothing is taken for a new stream. - The restart tests still drop the pairs on a real jump. All 25 offsets tests pass. - `make test`: 1,485 passed, 4 failed. All four failures were timeouts, while another project's Playwright run on the same Mac held the load average near 150: - `test_bare_machine_start` ("the service process did not answer in time"); - `test_click_to_play_works_in_the_bare_run`; - `test_hand_picked_game_starts_playing_on_click`; - `test_a_comparison_that_runs_too_long_is_said_with_what_it_does_and_the_busy_threads`. Run alone, the last two passed. `test_bare_machine_start` fails the same way on #183's branch (no clock change) under the same load, though it passed in that branch's full lane earlier. The lane will be run again once the load drops, and the result added here. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
A link's clock was the largest `segment end timestamp - time first
listed` of the last 10 minutes, which holds while a stream's delay is
steady. On 2026-11-16 shd247.world's sources listed media at 0.90x, so
their delay grew about 6 s a minute: the game page showed "5.2 s ahead"
for a stream about 30 s behind, and once the drift passed 30 s it was
taken for a new stream behind the link and the link's pairs were
dropped, to be found again a second later at the same relation.

offsets._drift measures how fast a link's clock falls behind from its
own readings (the slope over the window, once there are three readings
a minute apart; 0 for a steady source or one catching up), and
_freshest carries each reading forward by it before taking the largest.
clock_of (the seconds shown) and _listed's restart check go by it, so a
steady drift is no restart; a sudden jump still is. A steady link reads
exactly as before.

The stream-supervision spec says so, with a scenario and its test: a
standby at 0.9x read -5.6 s at t=200 for a stream 22.5 s behind; now it
is within 1.5 s of the truth throughout, and its pair is kept and
trusted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Owner

Second make test run, 22:07 UTC on 6353c09: 1,448 passed, 41 failed, in 11 min, with the Mac's load average at 384 from another project's Playwright run.

  • Two tests timed out: test_operator_opens_the_app and test_click_to_play_works_in_the_bare_run, both browser tests.
  • 33 of the others failed with "asyncio.run() cannot be called from a running event loop" or "Cannot run the event loop while another loop is running". A test that times out inside its own event loop leaves every later test in that worker failing that way.
  • test_watch_offsets.py failed the same way, not on any clock assertion. It passed in the targeted run (test_offsets.py and test_watch_offsets.py: 25 passed).

Not merged on that result. The branch now has main merged in (03a7040, after #182 and #183). The lane runs again once the 5-minute load average is under 20, and its result will be added here.

Second `make test` run, 22:07 UTC on 6353c09: 1,448 passed, 41 failed, in 11 min, with the Mac's load average at 384 from another project's Playwright run. - Two tests timed out: `test_operator_opens_the_app` and `test_click_to_play_works_in_the_bare_run`, both browser tests. - 33 of the others failed with "asyncio.run() cannot be called from a running event loop" or "Cannot run the event loop while another loop is running". A test that times out inside its own event loop leaves every later test in that worker failing that way. - `test_watch_offsets.py` failed the same way, not on any clock assertion. It passed in the targeted run (`test_offsets.py` and `test_watch_offsets.py`: 25 passed). Not merged on that result. The branch now has `main` merged in (03a7040, after #182 and #183). The lane runs again once the 5-minute load average is under 20, and its result will be added here.
Author
Owner

Third make test run, 23:10 UTC on 03a7040 (this branch with main merged in, after #182 and #183), with the load average at 1.7: 1,493 passed, 1 skipped, in 27 s. The earlier failures were the machine load.

Third `make test` run, 23:10 UTC on 03a7040 (this branch with `main` merged in, after #182 and #183), with the load average at 1.7: 1,493 passed, 1 skipped, in 27 s. The earlier failures were the machine load.
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/sports-breaker!184
No description provided.