A channel keeps pace with a source that limits each connection: segments fetched in parallel, polls on a schedule #56

Closed
opened 2026-10-30 19:01:21 -04:00 by cmoriarty · 1 comment
Owner

What happened (2026-10-30, Wizards vs. Hornets, Emby on the Shield)

The picture froze again and again from the moment it was tuned, and Emby finally quit. The source was fine: the channel could not download it as fast as it played. Times are local (UTC−4).

  • 18:47:34: tuned on topsurge.st. The CDN is hls.lshls.st: 1152×648 at 59.94 fps, 4 s segments of about 0.7 MB, and a playlist window of 10 segments.
  • 18:48:33–18:48:49: 16 s without a new segment. Emby killed its direct stream and started a VAAPI→QSV transcode.
  • 18:53:00: the channel swapped to the standby givemestreams.cfd (1280×720 at 30 fps). The transcode died at the splice (Error reinitializing filters!, #39), and the Shield left the stream.

What we measured

  • The source kept time. Polled once a second from 18:52:20 for 43 s, its media sequence climbed 751 → 762: one segment about every 4 s.
  • The channel did not. Its own sequence climbed 48 → 52 over the same 43 s. Across the whole session it served 52 segments, about 208 s of game, in 323 s: 0.64× real time, with 18 stalls of 6–18 s.
  • The CDN limits each connection, not the total.
    • One segment at a time took 2.8–3.3 s from the Mac.
    • Three at once took 2.4–2.7 s each, all three finished within 2.7 s.
    • From the VM, the channel's segments landed 4.0–4.6 s apart inside a batch, slower than they play.
  • Content was skipped at every tenth segment. The clock logged splices at seq 11, 21, 31, 42 and 52, each a jump of 8–20 s in the source's own timestamps.

Why

  • _prefetch downloads one segment at a time. _pull_once waits for all of them before the round ends.
  • _run adds its sleep on top of the work. It sleeps half a target duration (2 s) after each round, so a round costs sleep + playlist fetch + N downloads. At about 4 s per download, the channel never catches up.
  • A late round fetches a whole window. It appends all ten listed segments and takes about 45 s to fetch them.
  • By then the next segment has left the source's window. append_upstream clamps its start to the oldest segment still listed, so the segments in between are dropped without any signal. That is the jump at every tenth seq.

Proposed change

  • Download new segments in parallel, two or three at once, while keeping the clock, the audio step and the listing in playlist order.
  • Schedule polls from the start of each round, and poll again at once when the last round found more than one new segment.
  • When the channel falls more than a few segments behind, jump to near the live edge once and mark a discontinuity, rather than chasing a window it cannot reach.
  • Show how far behind the channel is on the Admin card: the source's newest media sequence minus the newest one fetched, and the segments skipped. Health can use it (separate issue).

Done when

  • A fake upstream that limits each connection to 0.9× real time keeps pace in a test: a new segment listed every target duration, and nothing skipped.
  • A fake upstream with a 10-segment window and 4.5 s downloads does not skip content.
  • Replaying tonight's conditions (one connection at about 0.75× real time, a window of 10) shows no stall longer than one target duration.
## What happened (2026-10-30, Wizards vs. Hornets, Emby on the Shield) The picture froze again and again from the moment it was tuned, and Emby finally quit. The source was fine: the channel could not download it as fast as it played. Times are local (UTC−4). - **18:47:34:** tuned on topsurge.st. The CDN is `hls.lshls.st`: 1152×648 at 59.94 fps, 4 s segments of about 0.7 MB, and a playlist window of 10 segments. - **18:48:33–18:48:49:** 16 s without a new segment. Emby killed its direct stream and started a VAAPI→QSV transcode. - **18:53:00:** the channel swapped to the standby givemestreams.cfd (1280×720 at 30 fps). The transcode died at the splice (`Error reinitializing filters!`, #39), and the Shield left the stream. ## What we measured - **The source kept time.** Polled once a second from 18:52:20 for 43 s, its media sequence climbed 751 → 762: one segment about every 4 s. - **The channel did not.** Its own sequence climbed 48 → 52 over the same 43 s. Across the whole session it served 52 segments, about 208 s of game, in 323 s: **0.64× real time**, with 18 stalls of 6–18 s. - **The CDN limits each connection, not the total.** - One segment at a time took 2.8–3.3 s from the Mac. - Three at once took 2.4–2.7 s each, all three finished within 2.7 s. - From the VM, the channel's segments landed 4.0–4.6 s apart inside a batch, slower than they play. - **Content was skipped at every tenth segment.** The clock logged splices at seq 11, 21, 31, 42 and 52, each a jump of 8–20 s in the source's own timestamps. ## Why - **`_prefetch` downloads one segment at a time.** `_pull_once` waits for all of them before the round ends. - **`_run` adds its sleep on top of the work.** It sleeps half a target duration (2 s) after each round, so a round costs sleep + playlist fetch + N downloads. At about 4 s per download, the channel never catches up. - **A late round fetches a whole window.** It appends all ten listed segments and takes about 45 s to fetch them. - **By then the next segment has left the source's window.** `append_upstream` clamps its start to the oldest segment still listed, so the segments in between are dropped without any signal. That is the jump at every tenth seq. ## Proposed change - **Download new segments in parallel**, two or three at once, while keeping the clock, the audio step and the listing in playlist order. - **Schedule polls from the start of each round**, and poll again at once when the last round found more than one new segment. - **When the channel falls more than a few segments behind, jump to near the live edge once** and mark a discontinuity, rather than chasing a window it cannot reach. - **Show how far behind the channel is** on the Admin card: the source's newest media sequence minus the newest one fetched, and the segments skipped. Health can use it (separate issue). ## Done when - A fake upstream that limits each connection to 0.9× real time keeps pace in a test: a new segment listed every target duration, and nothing skipped. - A fake upstream with a 10-segment window and 4.5 s downloads does not skip content. - Replaying tonight's conditions (one connection at about 0.75× real time, a window of 10) shows no stall longer than one target duration.
Author
Owner

Closing: deployed 2026-10-31 at develop 2e1e618 (PR #60, then #61 for what the first live run found).

Seen live, with a Jellyfin-style ffmpeg reader on each game for 6–7 min: Lakers vs. Warriors, then Kent State vs. Sacramento State.

  • Parallel downloads: segments download up to three at once, on a poll schedule that never waits for them.
  • Catching up: each landing's cold-start backlog went from 7 segments behind the source to 0 in about 30 s. After that the channel stayed 0–1 behind.
  • The viewer side: no game was skipped, and the viewer was served at 1.0× real time with an 8 s buffer.

Found live and fixed in #61:

  • A false frozen failure: a full eight-segment backlog appended nothing for a few polls, and the frozen rule failed a stream that was playing. The source's live edge advancing now counts.
  • 429 from parallel downloads: the CDN behind givemestreams.cfd now and then answered a parallel download with 429. Such a source now drops to one download at a time until the next landing. Seen working live: the channel kept pace one at a time.

Tests: a virtual-time replay of the 2026-10-30 source keeps pace with three download slots and falls behind with one (test_channel_pace.py).

Closing: deployed 2026-10-31 at develop 2e1e618 (PR #60, then #61 for what the first live run found). **Seen live**, with a Jellyfin-style ffmpeg reader on each game for 6–7 min: Lakers vs. Warriors, then Kent State vs. Sacramento State. - **Parallel downloads:** segments download up to three at once, on a poll schedule that never waits for them. - **Catching up:** each landing's cold-start backlog went from 7 segments behind the source to 0 in about 30 s. After that the channel stayed 0–1 behind. - **The viewer side:** no game was skipped, and the viewer was served at 1.0× real time with an 8 s buffer. **Found live and fixed in #61:** - **A false frozen failure:** a full eight-segment backlog appended nothing for a few polls, and the frozen rule failed a stream that was playing. The source's live edge advancing now counts. - **429 from parallel downloads:** the CDN behind givemestreams.cfd now and then answered a parallel download with 429. Such a source now drops to one download at a time until the next landing. Seen working live: the channel kept pace one at a time. **Tests:** a virtual-time replay of the 2026-10-30 source keeps pace with three download slots and falls behind with one (`test_channel_pace.py`).
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/sports-breaker#56
No description provided.