Recovery: a copy of the failed stream goes first, the standbys were not reordered, and a slow source starves the search #101

Closed
opened 2026-11-07 23:33:02 -05:00 by cmoriarty · 5 comments
Owner

From UTEP vs. Hawai'i on 2026-11-07, the first live game after milestone 36 ("Ready means watched", PR #99) was deployed.

Worked live: the background channel was warm before the tune and the tune came up quickly; the grades went Unproven → Steady; the buffer deepened for a bursty stream (3, then 5, then 6 segments); the standby was recognised as a copy; and at 23:29:55 the degrading channel stayed put ("the standby powerstream.club is the same stream as the one playing; staying").

The timeline (ET):

  • 23:23:56 Ready on givemestreams.cfd. 23:24:06 kept warm. Tuned soon after.
  • 23:25:35 standby powerstream.club, a copy of the same stream.
  • 23:27:00 second standby sportsframe.site (edgestream3.pro packed JSON, stream_id c3dbb952…).
  • 23:29:55 givemestreams degrading (behind); the channel stays put.
  • 23:30:23 givemestreams' playlist answers 503; recovery lands on powerstream (the copy), which fails at once; 23:30:26 recovered on sportsframe.
  • sportsframe then grades Choppy (pace 0.83, would need about 34 s of buffer). Recoveries at 23:31:10 and 23:31:57 re-read sportsframe and land on it again, each interrupting a background check ("check stopped for a watched game's recovery"). By 23:33: 4 recoveries, 16 s of filler, longest stall 12 s.

Three gaps

  1. A failure's recovery takes a copy of the failed stream first. Step 1 of Channel._recover takes the ready standby whatever it is. When the standby is the same stream as the one that failed and the second standby is a different one, the different stream should go first. Between copies, prefer one whose playlist is on another host than the failed one (powerstream failed at the same instant as givemestreams, so they probably shared a CDN).

  2. The standbys were not put in order before the failure. background.keep_best → _order_standbys should have swapped the copy (powerstream) behind sportsframe within 15 s of 23:27:00. It did not. Either sportsframe was judged the same stream as givemestreams (by clock or stream id), which would make the order correct but means a "copy" can still survive a delivery failure, or the ordering misses this case (a played game that is also a background channel). The saved state no longer holds givemestreams' and powerstream's marks, so which one is unknown. Log each link's mark when it is verified and when the order is checked, then reproduce.

  3. Recoveries on a slow source starve the search for a different stream. A stream at 0.83× real time drains any buffer, so the channel recovers every 45 s or so, each time by re-reading the same link. Each urgent re-read interrupts the background read that could find a different stream (#77's preemption). For a link graded Choppy whose re-read lands on the same stream again, let the background search finish (or run it as the recovery's next step) instead of re-reading the same link round after round. The Admin card should also say the source itself is slow, not just "recovered".

Related: #91 (stream identity), #95 (best of the different streams), #97 (stay put), #96 (background channels).

From UTEP vs. Hawai'i on 2026-11-07, the first live game after milestone 36 ("Ready means watched", PR #99) was deployed. **Worked live:** the background channel was warm before the tune and the tune came up quickly; the grades went Unproven → Steady; the buffer deepened for a bursty stream (3, then 5, then 6 segments); the standby was recognised as a copy; and at 23:29:55 the degrading channel stayed put ("the standby powerstream.club is the same stream as the one playing; staying"). **The timeline (ET):** - 23:23:56 Ready on givemestreams.cfd. 23:24:06 kept warm. Tuned soon after. - 23:25:35 standby powerstream.club, a copy of the same stream. - 23:27:00 second standby sportsframe.site (edgestream3.pro packed JSON, `stream_id` c3dbb952…). - 23:29:55 givemestreams degrading (behind); the channel stays put. - 23:30:23 givemestreams' playlist answers 503; recovery lands on powerstream (the copy), which fails at once; 23:30:26 recovered on sportsframe. - sportsframe then grades Choppy (pace 0.83, would need about 34 s of buffer). Recoveries at 23:31:10 and 23:31:57 re-read sportsframe and land on it again, each interrupting a background check ("check stopped for a watched game's recovery"). By 23:33: 4 recoveries, 16 s of filler, longest stall 12 s. ## Three gaps 1. **A failure's recovery takes a copy of the failed stream first.** Step 1 of `Channel._recover` takes the ready standby whatever it is. When the standby is the same stream as the one that failed and the second standby is a different one, the different stream should go first. Between copies, prefer one whose playlist is on another host than the failed one (powerstream failed at the same instant as givemestreams, so they probably shared a CDN). 2. **The standbys were not put in order before the failure.** `background.keep_best` → `_order_standbys` should have swapped the copy (powerstream) behind sportsframe within 15 s of 23:27:00. It did not. Either sportsframe was judged the same stream as givemestreams (by clock or stream id), which would make the order correct but means a "copy" can still survive a delivery failure, or the ordering misses this case (a played game that is also a background channel). The saved state no longer holds givemestreams' and powerstream's marks, so which one is unknown. Log each link's mark when it is verified and when the order is checked, then reproduce. 3. **Recoveries on a slow source starve the search for a different stream.** A stream at 0.83× real time drains any buffer, so the channel recovers every 45 s or so, each time by re-reading the same link. Each urgent re-read interrupts the background read that could find a different stream (#77's preemption). For a link graded Choppy whose re-read lands on the same stream again, let the background search finish (or run it as the recovery's next step) instead of re-reading the same link round after round. The Admin card should also say the source itself is slow, not just "recovered". Related: #91 (stream identity), #95 (best of the different streams), #97 (stay put), #96 (background channels).
Author
Owner

Later the same game (23:33–23:55):

  • sportsframe.site went from Choppy (pace 0.83, would need about 34 s of buffer) back to Steady (pace 0.98–1.0, about 10 s) within about ten minutes, with no recovery after 23:32. A slow stretch, not a slow link.
  • The next standby, tvsportslive.fr, was again a copy of the playing stream (graded Steady). No second standby was found before I stopped watching, so gap 2 (the ordering) didn't come up again.
  • Fingerprints: sportsframe.site's playlist is edgestream3.pro's packed JSON, stream_id c3dbb952796e65497ce764a3780bc7a4, clock 62695.8 (from the readiness snapshot). The others weren't kept.
Later the same game (23:33–23:55): - sportsframe.site went from Choppy (pace 0.83, would need about 34 s of buffer) back to Steady (pace 0.98–1.0, about 10 s) within about ten minutes, with no recovery after 23:32. A slow stretch, not a slow link. - The next standby, tvsportslive.fr, was again a copy of the playing stream (graded Steady). No second standby was found before I stopped watching, so gap 2 (the ordering) didn't come up again. - Fingerprints: sportsframe.site's playlist is edgestream3.pro's packed JSON, `stream_id` c3dbb952796e65497ce764a3780bc7a4, clock 62695.8 (from the readiness snapshot). The others weren't kept.
Author
Owner

Deployed 2026-11-08 08:48Z as 1.2.0: PR #113 (merge 654a9a7), released through release/1.2.0 (tag v1.2.0), container b0f1f361509d. The OpenSpec change settings-and-recoveries is archived in d013eb4.

Live now:

  1. Copy first. A recovery chooses between the two ready standbys. A different stream goes before a copy of the one that failed. Between copies, it takes one whose playlist is on another host. When it takes the second standby first, the log says why.
  2. The order. I reproduced UTEP's layout in tests: kept A, a copy B standing first, a different stream C second. I ran it as a played game, a played background channel and an unwatched one. _order_standbys put C first within one round every time, so the order check wasn't the bug. The copy came from the recovery's first step, which took the first ready standby without looking at it. That is the part fixed in 1.
    • Each standby's mark (site, playlist host, stream id, clock) is now logged when it is verified.
    • The order check logs the three marks and its decision whenever they change. The next case can be read back from the container log.
  3. A slow source. When the stream that failed was graded Choppy:
    • its re-read waits for a slot instead of interrupting the game's own background search;
    • landing back on the same stream wakes the standby search at once;
    • the event and the run say "the source is slow, 0.83×", and the Playing card shows a trend-down mark on that run.

Left open until a live game shows a recovery between two standbys or a slow-source recovery.

Deployed 2026-11-08 08:48Z as **1.2.0**: PR #113 (merge 654a9a7), released through `release/1.2.0` (tag `v1.2.0`), container b0f1f361509d. The OpenSpec change `settings-and-recoveries` is archived in d013eb4. Live now: 1. **Copy first.** A recovery chooses between the two ready standbys. A different stream goes before a copy of the one that failed. Between copies, it takes one whose playlist is on another host. When it takes the second standby first, the log says why. 2. **The order.** I reproduced UTEP's layout in tests: kept A, a copy B standing first, a different stream C second. I ran it as a played game, a played background channel and an unwatched one. `_order_standbys` put C first within one round every time, so the order check wasn't the bug. The copy came from the recovery's first step, which took the first ready standby without looking at it. That is the part fixed in 1. - Each standby's mark (site, playlist host, stream id, clock) is now logged when it is verified. - The order check logs the three marks and its decision whenever they change. The next case can be read back from the container log. 3. **A slow source.** When the stream that failed was graded Choppy: - its re-read waits for a slot instead of interrupting the game's own background search; - landing back on the same stream wakes the standby search at once; - the event and the run say "the source is slow, 0.83×", and the Playing card shows a trend-down mark on that run. Left open until a live game shows a recovery between two standbys or a slow-source recovery.
Author
Owner

Live check, 2026-11-08 15:23–15:45Z, C Palace vs. Liverpool (watched in Emby Web): the slow-source part never started, although the source was slow the whole time. The owner saw regular freezes and skips.

What happened:

  • The CDN: every link that worked (topsurge.st, powerstream.club, bally-sports.lat) carries hls.lshls.st.
  • Its publish rate: it lists a 2 s segment every 2.15–2.32 s, that is 0.86–0.93× real time. Measured over 90 s from outside the app's VPN it was 0.93×, and the app's own waits agree. Segments download in about 1 s, so this is publishing, not throttling.
  • The Bengals stream on the same CDN ran at 0.77×.
  • The cost: 3 swaps, 1 recovery and 5 fillers of 4 s. Every swap landed on hls.lshls.st again, and the one standby on another CDN (shd247.world, instreams.xyz) was worse at 0.66×.
  • What the card said: Healthy, graded Steady, with the verdict "Nothing to do".

Why the slow-source path never ran:

  • _slow_source waits for a Choppy grade.
  • watch.grade calls a link Choppy only once it needs more than 30 s of buffer within its 180 s window, which takes a pace under about 0.83×.
  • A source held under real time never catches up, so no buffer saves it, but at 0.86–0.93× it is graded Steady.

The fix, under this issue:

  1. A pace held under 0.95× over the watch window grades the link Choppy. The pace is measured between listings, so a burst still due doesn't count against it.
  2. When the stream left was slow, swaps and recoveries prefer a standby whose playlist is on another host, and the standby order does the same.
  3. The card's verdict says the source is slow instead of "Nothing to do".
**Live check, 2026-11-08 15:23–15:45Z, C Palace vs. Liverpool (watched in Emby Web):** the slow-source part never started, although the source was slow the whole time. The owner saw regular freezes and skips. **What happened:** - **The CDN:** every link that worked (topsurge.st, powerstream.club, bally-sports.lat) carries hls.lshls.st. - **Its publish rate:** it lists a 2 s segment every 2.15–2.32 s, that is 0.86–0.93× real time. Measured over 90 s from outside the app's VPN it was 0.93×, and the app's own waits agree. Segments download in about 1 s, so this is publishing, not throttling. - **The Bengals stream** on the same CDN ran at 0.77×. - **The cost:** 3 swaps, 1 recovery and 5 fillers of 4 s. Every swap landed on hls.lshls.st again, and the one standby on another CDN (shd247.world, instreams.xyz) was worse at 0.66×. - **What the card said:** Healthy, graded Steady, with the verdict "Nothing to do". **Why the slow-source path never ran:** - `_slow_source` waits for a Choppy grade. - `watch.grade` calls a link Choppy only once it needs more than 30 s of buffer within its 180 s window, which takes a pace under about 0.83×. - A source held under real time never catches up, so no buffer saves it, but at 0.86–0.93× it is graded Steady. **The fix, under this issue:** 1. A pace held under 0.95× over the watch window grades the link Choppy. The pace is measured between listings, so a burst still due doesn't count against it. 2. When the stream left was slow, swaps and recoveries prefer a standby whose playlist is on another host, and the standby order does the same. 3. The card's verdict says the source is slow instead of "Nothing to do".
Author
Owner

Deployed 2026-11-08 16:02:56Z: PR #116 (merge 9afc14a), container 4799fa3b3239, 37 s down. The OpenSpec change slow-source-pace is archived in 5ae23c4.

Live now:

  • The grade: a link whose pace is held under 0.95× across two minutes of listings is graded Choppy, and its grade says slow.
  • Recovery and standbys: a slow stream's recovery takes a standby on another playlist host first, and the standby order does the same while the kept stream is slow.
  • The card: the verdict says Let it work with the source's pace instead of "Nothing to do", and the grade's info says the source is slower than real time.

First look after the deploy: Falcons vs. Bengals in the background, on hls.lshls.st again, now ran at 0.98× and was graded Steady, not slow, so there was no false alarm. The container log shows no errors.

Still to see live: a source held under real time graded slow, with the card and the recovery following.

Deployed 2026-11-08 16:02:56Z: PR #116 (merge 9afc14a), container 4799fa3b3239, 37 s down. The OpenSpec change `slow-source-pace` is archived in 5ae23c4. **Live now:** - **The grade:** a link whose pace is held under 0.95× across two minutes of listings is graded Choppy, and its grade says `slow`. - **Recovery and standbys:** a slow stream's recovery takes a standby on another playlist host first, and the standby order does the same while the kept stream is slow. - **The card:** the verdict says Let it work with the source's pace instead of "Nothing to do", and the grade's info says the source is slower than real time. **First look after the deploy:** Falcons vs. Bengals in the background, on hls.lshls.st again, now ran at 0.98× and was graded Steady, not slow, so there was no false alarm. The container log shows no errors. Still to see live: a source held under real time graded slow, with the card and the recovery following.
Author
Owner

Closing: the fixes are merged to develop in PR #113 (654a9a7) and PR #116 (9afc14a, the slow-source grade), both deployed.

Seen live on 2026-11-08: a degrading swap; a recovery onto a different stream in 3 s; and the standby marks logged at verify.

Not seen yet: a recovery choosing between two ready standbys, and a source held under real time graded slow, with the card and the recovery following. A new issue if either misbehaves.

Closing: the fixes are merged to develop in PR #113 (654a9a7) and PR #116 (9afc14a, the slow-source grade), both deployed. **Seen live on 2026-11-08:** a degrading swap; a recovery onto a different stream in 3 s; and the standby marks logged at verify. **Not seen yet:** a recovery choosing between two ready standbys, and a source held under real time graded slow, with the card and the recovery following. A new issue if either misbehaves.
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/sports-breaker#101
No description provided.