Recovery: a copy of the failed stream goes first, the standbys were not reordered, and a slow source starves the search #101
Labels
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
cmoriarty/sports-breaker#101
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
From UTEP vs. Hawai'i on 2026-11-07, the first live game after milestone 36 ("Ready means watched", PR #99) was deployed.
Worked live: the background channel was warm before the tune and the tune came up quickly; the grades went Unproven → Steady; the buffer deepened for a bursty stream (3, then 5, then 6 segments); the standby was recognised as a copy; and at 23:29:55 the degrading channel stayed put ("the standby powerstream.club is the same stream as the one playing; staying").
The timeline (ET):
stream_idc3dbb952…).Three gaps
A failure's recovery takes a copy of the failed stream first. Step 1 of
Channel._recovertakes the ready standby whatever it is. When the standby is the same stream as the one that failed and the second standby is a different one, the different stream should go first. Between copies, prefer one whose playlist is on another host than the failed one (powerstream failed at the same instant as givemestreams, so they probably shared a CDN).The standbys were not put in order before the failure.
background.keep_best→_order_standbysshould have swapped the copy (powerstream) behind sportsframe within 15 s of 23:27:00. It did not. Either sportsframe was judged the same stream as givemestreams (by clock or stream id), which would make the order correct but means a "copy" can still survive a delivery failure, or the ordering misses this case (a played game that is also a background channel). The saved state no longer holds givemestreams' and powerstream's marks, so which one is unknown. Log each link's mark when it is verified and when the order is checked, then reproduce.Recoveries on a slow source starve the search for a different stream. A stream at 0.83× real time drains any buffer, so the channel recovers every 45 s or so, each time by re-reading the same link. Each urgent re-read interrupts the background read that could find a different stream (#77's preemption). For a link graded Choppy whose re-read lands on the same stream again, let the background search finish (or run it as the recovery's next step) instead of re-reading the same link round after round. The Admin card should also say the source itself is slow, not just "recovered".
Related: #91 (stream identity), #95 (best of the different streams), #97 (stay put), #96 (background channels).
Later the same game (23:33–23:55):
stream_idc3dbb952796e65497ce764a3780bc7a4, clock 62695.8 (from the readiness snapshot). The others weren't kept.Deployed 2026-11-08 08:48Z as 1.2.0: PR #113 (merge
654a9a7), released throughrelease/1.2.0(tagv1.2.0), container b0f1f361509d. The OpenSpec changesettings-and-recoveriesis archived ind013eb4.Live now:
_order_standbysput C first within one round every time, so the order check wasn't the bug. The copy came from the recovery's first step, which took the first ready standby without looking at it. That is the part fixed in 1.Left open until a live game shows a recovery between two standbys or a slow-source recovery.
Live check, 2026-11-08 15:23–15:45Z, C Palace vs. Liverpool (watched in Emby Web): the slow-source part never started, although the source was slow the whole time. The owner saw regular freezes and skips.
What happened:
Why the slow-source path never ran:
_slow_sourcewaits for a Choppy grade.watch.gradecalls a link Choppy only once it needs more than 30 s of buffer within its 180 s window, which takes a pace under about 0.83×.The fix, under this issue:
Deployed 2026-11-08 16:02:56Z: PR #116 (merge
9afc14a), container 4799fa3b3239, 37 s down. The OpenSpec changeslow-source-paceis archived in5ae23c4.Live now:
slow.First look after the deploy: Falcons vs. Bengals in the background, on hls.lshls.st again, now ran at 0.98× and was graded Steady, not slow, so there was no false alarm. The container log shows no errors.
Still to see live: a source held under real time graded slow, with the card and the recovery following.
Closing: the fixes are merged to develop in PR #113 (
654a9a7) and PR #116 (9afc14a, the slow-source grade), both deployed.Seen live on 2026-11-08: a degrading swap; a recovery onto a different stream in 3 s; and the standby marks logged at verify.
Not seen yet: a recovery choosing between two ready standbys, and a source held under real time graded slow, with the card and the recovery following. A new issue if either misbehaves.