A swap or recovery lands on a stream that just failed, through another site #91

Closed
opened 2026-11-05 22:17:59 -05:00 by cmoriarty · 4 comments
Owner

What happens

When a link fails, the app cools that one player page (Ladder.fail, for 10 min). It doesn't remember which stream failed. Other sites that carry the same stream stay eligible as the standby, as the second standby and on the ladder. So a swap or a recovery can land straight back on a stream that just failed, through a different site.

Ravens vs. Jaguars, 2026-11-05 (the swaps are listed in #90). Behind 9 sites there were 4 streams, A to D. Matching each stream's own clock in the splice lines showed which site carried which:

Time (ET) What happened Landed on Why it was a bad pick It lasted
9:53:28 degrading swap nfl-castnow.lat (A) A had failed on givemestreams.cfd 76 s earlier. The standby read verified this other site of A 63 s after that failure. 3 m 22 s
10:01:31 recovery sportsframe.site (A) A had failed 4 m 41 s earlier. sportsframe was read at 9:56 as a copy of A while A was playing, the #74 fallback. When A failed, s2watch (B) took over and the copy of the stream that had just failed moved up to standby. 25 s
10:07:54 degrading swap buff-streams.buzz (C, probably) The playing link, topsurge.st, was C. buff-streams.buzz had been read at 10:02 as the second standby, a copy of the standby topsurge. When topsurge was promoted, buff-streams moved up to standby as a copy of the playing stream. The degrading swap then went to it anyway. 9 s
10:08:03 recovery givemestreams.cfd (A) A had failed 6 min earlier, on sportsframe. 1 m 13 s

Four of the 11 swaps landed on a stream that had failed in the last six minutes, or on the stream being left. Together those four played about 5 minutes. The rest of the time between them was spent on filler and further swaps.

Where in the code:

  • Only the page cools. resolution.invalidate cools the failed player page, not its stream. verify_standby's avoid holds only the streams kept right now (the link playing, the standby, the second standby), not the streams that just failed.
  • The second standby moves up unchecked. When the standby is promoted, _standby_gone moves the second standby up without checking it against the new main link. A copy of the second standby becomes a copy of the stream now playing.
  • The degrading swap doesn't look at the stream. Channel._maybe_swap compares scores only. A same-stream standby always scores "healthy" against a degrading main link, so it wins.

What it costs

A swap to the same stream, or to one that just failed, rarely helps. The viewer gets another splice, often filler, and the channel spends its standby. After that, the recovery falls to the ladder or a re-read. On a night with only a few streams to choose from, this is what the "rotating through the same streams" looks like.

The idea

  1. Remember failed streams per game. When a link fails, or a degrading swap leaves it, record its stream (stream_identity.identity) and the time. For the cooldown, every link of that stream counts as cooling:
    • the standby read and the second-standby read put it in avoid;
    • the ladder orders it after the other streams.
  2. Re-check after a promotion. When the second standby moves up, check it against the new main link and against the streams that just failed. If it matches either, keep it only as a fallback and wake a standby read for a different stream.
  3. The degrading swap skips a copy. A standby that is the same stream as the playing link doesn't win a degrading swap. The second standby is tried instead if it is a different stream; otherwise the channel stays where it is and the card says so.
  4. The recovery prefers a fresh stream. A ready standby that is a stream that failed in the last few minutes ranks after a ready one that didn't.

Defaults:

  • Window: a failed stream is remembered for the link cooldown (cooldown_seconds, 10 min).
  • Never filler instead of a picture. A recently failed stream is still used when nothing else is ready. It moves down the order; it isn't excluded.
  • The re-read stays. Re-reading the same link after a failure (step 2 of the recovery) is kept: at 10:09:16 it landed on A again and A has held since. An expired URL plays again this way too.
  • Logged in the check's links, the way #74's copies are, and in the container log: "skipped : the same stream as , which failed N min ago".

For the design to check: whether the URL identity alone caught tonight's repeats. All of A's sites may not have used the same CDN URL. The source's own clock (the first segment's timestamp, which the verify probe already reads) could be a second signal: two links whose clocks agree to within a few seconds are the same stream.

Not part of this: showing the swaps on the card. That is #90, and with it this rule's skips can be seen.

Related: #74 (a different stream for the standby), #64 (the second standby), #65 (a relapse to the standby).

## What happens When a link fails, the app cools that one player page (`Ladder.fail`, for 10 min). It doesn't remember which **stream** failed. Other sites that carry the same stream stay eligible as the standby, as the second standby and on the ladder. So a swap or a recovery can land straight back on a stream that just failed, through a different site. **Ravens vs. Jaguars, 2026-11-05** (the swaps are listed in #90). Behind 9 sites there were 4 streams, A to D. Matching each stream's own clock in the splice lines showed which site carried which: | Time (ET) | What happened | Landed on | Why it was a bad pick | It lasted | |---|---|---|---|---| | 9:53:28 | degrading swap | nfl-castnow.lat (A) | A had failed on givemestreams.cfd 76 s earlier. The standby read verified this other site of A 63 s after that failure. | 3 m 22 s | | 10:01:31 | recovery | sportsframe.site (A) | A had failed 4 m 41 s earlier. sportsframe was read at 9:56 as a copy of A while A was playing, the #74 fallback. When A failed, s2watch (B) took over and the copy of the stream that had just failed moved up to standby. | 25 s | | 10:07:54 | degrading swap | buff-streams.buzz (C, probably) | The playing link, topsurge.st, was C. buff-streams.buzz had been read at 10:02 as the second standby, a copy of the standby topsurge. When topsurge was promoted, buff-streams moved up to standby as a copy of the playing stream. The degrading swap then went to it anyway. | 9 s | | 10:08:03 | recovery | givemestreams.cfd (A) | A had failed 6 min earlier, on sportsframe. | 1 m 13 s | Four of the 11 swaps landed on a stream that had failed in the last six minutes, or on the stream being left. Together those four played about 5 minutes. The rest of the time between them was spent on filler and further swaps. **Where in the code:** - **Only the page cools.** `resolution.invalidate` cools the failed player page, not its stream. `verify_standby`'s `avoid` holds only the streams kept right now (the link playing, the standby, the second standby), not the streams that just failed. - **The second standby moves up unchecked.** When the standby is promoted, `_standby_gone` moves the second standby up without checking it against the new main link. A copy of the second standby becomes a copy of the stream now playing. - **The degrading swap doesn't look at the stream.** `Channel._maybe_swap` compares scores only. A same-stream standby always scores "healthy" against a degrading main link, so it wins. ## What it costs A swap to the same stream, or to one that just failed, rarely helps. The viewer gets another splice, often filler, and the channel spends its standby. After that, the recovery falls to the ladder or a re-read. On a night with only a few streams to choose from, this is what the "rotating through the same streams" looks like. ## The idea 1. **Remember failed streams per game.** When a link fails, or a degrading swap leaves it, record its stream (`stream_identity.identity`) and the time. For the cooldown, every link of that stream counts as cooling: - the standby read and the second-standby read put it in `avoid`; - the ladder orders it after the other streams. 2. **Re-check after a promotion.** When the second standby moves up, check it against the new main link and against the streams that just failed. If it matches either, keep it only as a fallback and wake a standby read for a different stream. 3. **The degrading swap skips a copy.** A standby that is the same stream as the playing link doesn't win a degrading swap. The second standby is tried instead if it is a different stream; otherwise the channel stays where it is and the card says so. 4. **The recovery prefers a fresh stream.** A ready standby that is a stream that failed in the last few minutes ranks after a ready one that didn't. **Defaults:** - **Window:** a failed stream is remembered for the link cooldown (`cooldown_seconds`, 10 min). - **Never filler instead of a picture.** A recently failed stream is still used when nothing else is ready. It moves down the order; it isn't excluded. - **The re-read stays.** Re-reading the same link after a failure (step 2 of the recovery) is kept: at 10:09:16 it landed on A again and A has held since. An expired URL plays again this way too. - **Logged** in the check's links, the way #74's copies are, and in the container log: "skipped <host>: the same stream as <host>, which failed N min ago". **For the design to check:** whether the URL identity alone caught tonight's repeats. All of A's sites may not have used the same CDN URL. The source's own clock (the first segment's timestamp, which the verify probe already reads) could be a second signal: two links whose clocks agree to within a few seconds are the same stream. **Not part of this:** showing the swaps on the card. That is #90, and with it this rule's skips can be seen. Related: #74 (a different stream for the standby), #64 (the second standby), #65 (a relapse to the standby).
Author
Owner

Ole Miss vs. Georgia, 2026-11-07: every swap landed on the same stream

The channel was tuned at 7:41 pm ET. Between 7:42 and 7:58 it made 12 degrading swaps and 3 recoveries across 10 sites: s2watch.cyou, powerstream.club, givemestreams.cfd, iptvcast.lol, fast-streams.biz, nfl-castnow.lat, sportsframe.site, bally-sports.lat, tvsportslive.fr and watchsports.online. All of them carried one stream. The splice lines' raw clock runs straight through every swap, from 10188 s to 11280 s.

That stream was the problem: it published about 0.8× real time (#94). No swap could have helped, and each one cost a splice and usually filler:

  • By 7:55 the channel was 58 s behind the source.
  • The 7:56 recovery landed at the live edge and skipped about 84 s of game. The Playing card's Trouble line still said skipped_seconds: 0.

Added to this issue (milestone "Ready means watched")

  1. Name a stream by what it carries, not only by its URL. The #74 rule (the playlist's file name without the token) can't see a copy served under another name. Two signals catch it:
    • The disguised JSON playlists carry a stream_id. Tonight's was e84dab19….
    • The stream's own clock. Two links carry the same stream when their newest segments' timestamps, measured against the wall clock, agree within a segment. That is how the splice lines were matched by hand on 2026-11-05 and again tonight. verify() already downloads the newest segment, so the timestamp costs nothing extra.
  2. Remember failed and degraded streams per game (the original ask). For a while, a stream that failed or was swapped away from is avoided as the standby, as the second standby, by recovery and on the ladder, through any site.
  3. A degrading swap never lands on the stream it is leaving.
    • A copy can still stand by for a failed link, for example one edge server going down.
    • A degrading stream is degraded at its source, so a copy can't fix it.
    • When only copies are left, the channel stays where it is (#97).
  4. The check and channel events name each link's stream. This helps #90.

Not confirmed: whether tonight's sites resolved to one playlist file name. The URL is in neither the INFO log nor the events. So it is open whether #74's rule matched them and the standby fell back to a copy, or whether the rule missed them.

## Ole Miss vs. Georgia, 2026-11-07: every swap landed on the same stream The channel was tuned at 7:41 pm ET. Between 7:42 and 7:58 it made **12 degrading swaps and 3 recoveries** across 10 sites: s2watch.cyou, powerstream.club, givemestreams.cfd, iptvcast.lol, fast-streams.biz, nfl-castnow.lat, sportsframe.site, bally-sports.lat, tvsportslive.fr and watchsports.online. **All of them carried one stream.** The splice lines' raw clock runs straight through every swap, from 10188 s to 11280 s. That stream was the problem: it published about 0.8× real time (#94). No swap could have helped, and each one cost a splice and usually filler: - By 7:55 the channel was 58 s behind the source. - The 7:56 recovery landed at the live edge and **skipped about 84 s of game**. The Playing card's Trouble line still said `skipped_seconds: 0`. ## Added to this issue (milestone "Ready means watched") 1. **Name a stream by what it carries, not only by its URL.** The #74 rule (the playlist's file name without the token) can't see a copy served under another name. Two signals catch it: - **The disguised JSON playlists carry a `stream_id`.** Tonight's was `e84dab19…`. - **The stream's own clock.** Two links carry the same stream when their newest segments' timestamps, measured against the wall clock, agree within a segment. That is how the splice lines were matched by hand on 2026-11-05 and again tonight. `verify()` already downloads the newest segment, so the timestamp costs nothing extra. 2. **Remember failed and degraded streams per game** (the original ask). For a while, a stream that failed or was swapped away from is avoided as the standby, as the second standby, by recovery and on the ladder, through any site. 3. **A degrading swap never lands on the stream it is leaving.** - A copy can still stand by for a *failed* link, for example one edge server going down. - A *degrading* stream is degraded at its source, so a copy can't fix it. - When only copies are left, the channel stays where it is (#97). 4. **The check and channel events name each link's stream.** This helps #90. **Not confirmed:** whether tonight's sites resolved to one playlist file name. The URL is in neither the INFO log nor the events. So it is open whether #74's rule matched them and the standby fell back to a copy, or whether the rule missed them.
Author
Owner

Built in PR #99 (merged as 732ab0a), deployed 2026-11-08 04:19Z (container 52cd495f611a); the OpenSpec change is archived in 038f7a9. This stays open until it is seen working on a live followed game: a grade before the tune, a Warm background channel, and a Choppy stream that stays put.

Built in PR #99 (merged as 732ab0a), deployed 2026-11-08 04:19Z (container 52cd495f611a); the OpenSpec change is archived in 038f7a9. This stays open until it is seen working on a live followed game: a grade before the tune, a Warm background channel, and a Choppy stream that stays put.
Author
Owner

Live test, UTEP vs. Hawai'i, 2026-11-07 23:22–23:55 ET (first followed game after the deploy). Seen working: the standby (powerstream.club, later tvsportslive.fr) was recognised as a copy of the playing stream, and at 23:29:55 the degrading channel refused it ("the same stream as the one playing; staying").
Not working: when the stream then failed outright (a 503 on givemestreams.cfd's playlist), recovery landed on the copy first, which failed 3 s later. Filed as #101.

Live test, UTEP vs. Hawai'i, 2026-11-07 23:22–23:55 ET (first followed game after the deploy). Seen working: the standby (powerstream.club, later tvsportslive.fr) was recognised as a copy of the playing stream, and at 23:29:55 the degrading channel refused it ("the same stream as the one playing; staying"). Not working: when the stream then failed outright (a 503 on givemestreams.cfd's playlist), recovery landed on the copy first, which failed 3 s later. Filed as #101.
Author
Owner

Closed at the owner's word (2026-11-08): the fix is merged to develop in PR #99 (merge 732ab0a) and deployed. A new issue will be opened if a problem shows up.

Closed at the owner's word (2026-11-08): the fix is merged to develop in PR #99 (merge 732ab0a) and deployed. A new issue will be opened if a problem shows up.
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/sports-breaker#91
No description provided.