A channel notices and acts on what its viewer gets: stalls in the served playlist, not only the source's downloads #57

Closed
opened 2026-10-30 19:12:43 -04:00 by cmoriarty · 2 comments
Owner

What happened (2026-10-30, Wizards vs. Hornets, Emby on the Shield)

The player froze 18 times in five minutes, for 6–18 s each, and finally quit. Throughout, the Admin card said Healthy · Nothing to do and the trouble count showed 0 stalls. Every remedy the channel has (recovery, the standby, the filler card, the buffer) was in place, and none of them acted until 18:53. When one finally did, Emby crashed. Times are local (UTC−4).

  • 18:47:31: tuned on topsurge.st.
  • 18:47–18:53: 52 segments served in 323 s, about 208 s of game: 0.64× real time.
    • Stalls of 9, 6, 10, 16, 10, 6, 8, 10, 6, 10, 8, 14, 8, 8, 6, 12, 6 and 18 s.
    • 8–20 s of game skipped five times.
  • 18:48:49: after a 16 s stall, Emby killed its direct stream and started a VAAPI→QSV transcode.
  • 18:53:02: a degrading rule finally held for the 60 s that a swap to another picture size waits. The channel swapped to givemestreams.cfd (1280×720 at 30 fps, after 1152×648 at 59.94 fps). The transcode died at the splice (Error reinitializing filters!, #39), and the Shield left the stream.

Tonight's cause was the channel's own downloads (#56): the source kept time. This issue is about why the channel could not see that and act on it.

Why nothing acted

Every trigger judges the source as the downloader (the puller) sees it, and none of them judges what the viewer gets.

  • Recovery never started. It needs FAILED, and the rule that should have caught this is "no new segment for two target durations" (8 s).
    • classify runs only in tick(), after _pull_once() returns.
    • _pull_once() returns only after the last segment of its round has landed, and segment_in() has just reset the clock.
    • So the rule only looks at the one moment it cannot be true. A 16 s stall the viewer sat through was 0 s old by the time it was checked.
    • supervise(), which classifies on its own, is called only from the tests. The tests check the rules, not when production runs them.
  • The degrading rules sat just under their thresholds.
    • Downloads took 3–4.5 s for 4 s of video, so the slow rule counted 1–2 of the last 5.
    • Each stall made one release late, so the late rule counted 2 of 5.
    • The channel ran at 0.64× real time and still read as Healthy.
  • Filler never covered a stall.
    • _maybe_fill holds filler back while any appended segment is still being fetched.
    • Each round queued about ten segments and took about 45 s to fetch them, so one was nearly always in flight.
    • Filler was listed once in five minutes.
  • Skipped game was silent. Segments that left the source's window before they were fetched only show as an INFO clock: seq N splices line.
  • The trouble count counts recoveries, not stalls. trouble.stall() is called only when a recovery starts, so the card showed 0 stalls.
  • The swap that finally happened changed the picture's shape. That crashes Emby's hardware transcode until #39 lands, so the remedy ended the session.

Proposed change

  • Measure what the viewer gets, on the playout clock that already ticks every second, independent of the puller:
    • the served edge's age: seconds since the served playlist last grew;
    • the served rate: game seconds released per wall second over the last minute;
    • game skipped: segments that left the source's window unfetched, and timestamp jumps within one upstream;
    • how far the channel is behind the source's edge (from #56).
  • Classify on those, on that clock. Rough thresholds:
    • The served edge older than one target duration: list filler now, even with a fetch in flight. A segment that lands later splices back in as today. Count a stall.
    • A served rate under 0.9 for a minute, two stalls in two minutes, or any skipped game: degrading.
    • The served edge older than two target durations, or three stalls in two minutes: failed, and recovery starts.
  • Find out whose fault it is before acting.
    • If the source's edge keeps advancing while the channel falls behind, the fault is the channel's. Catch up by jumping to the edge and fetching in parallel (#56). Don't swap, and don't cool the link.
    • If the source's edge stalls too, recover or swap as today.
  • Make a different-shape swap the last step while #39 is open. First re-read the link, then try a standby or a ladder link of the same picture size.
  • Base the card on the viewer's side.
    • Never say "Nothing to do" while the served edge is stalling.
    • Count every stall the viewer sees in the trouble window, not only recoveries.
    • Show the served rate beside the slow, short and late strips.

Done when

These run through the real loops (_run and _playout), not only supervise():

  • A _pull_once that blocks for 15 s on one download gets filler listed within one target duration plus a second, a stall counted, and a card that does not say Healthy.
  • A source delivered at 0.7× real time, with no single download slower than its own duration, turns the channel degrading within a minute, and the channel acts.
  • Segments that leave the source's window unfetched are counted, and the channel turns degrading.
  • A replay of tonight's timings turns the card from Healthy within the first minute, and no swap to another picture size happens before a same-shape option has been tried.

Related: #56 (tonight's cause), #39 (a splice keeps the stream's shape), #50 (filler before the player runs dry), #53 (the buffer), #40 (health notices dropped frames).

## What happened (2026-10-30, Wizards vs. Hornets, Emby on the Shield) The player froze 18 times in five minutes, for 6–18 s each, and finally quit. Throughout, the Admin card said **Healthy · Nothing to do** and the trouble count showed **0 stalls**. Every remedy the channel has (recovery, the standby, the filler card, the buffer) was in place, and none of them acted until 18:53. When one finally did, Emby crashed. Times are local (UTC−4). - **18:47:31:** tuned on topsurge.st. - **18:47–18:53:** 52 segments served in 323 s, about 208 s of game: **0.64× real time**. - Stalls of 9, 6, 10, 16, 10, 6, 8, 10, 6, 10, 8, 14, 8, 8, 6, 12, 6 and 18 s. - 8–20 s of game skipped five times. - **18:48:49:** after a 16 s stall, Emby killed its direct stream and started a VAAPI→QSV transcode. - **18:53:02:** a degrading rule finally held for the 60 s that a swap to another picture size waits. The channel swapped to givemestreams.cfd (1280×720 at 30 fps, after 1152×648 at 59.94 fps). The transcode died at the splice (`Error reinitializing filters!`, #39), and the Shield left the stream. Tonight's cause was the channel's own downloads (#56): the source kept time. This issue is about why the channel could not see that and act on it. ## Why nothing acted Every trigger judges the source as the downloader (the puller) sees it, and none of them judges what the viewer gets. - **Recovery never started.** It needs FAILED, and the rule that should have caught this is "no new segment for two target durations" (8 s). - `classify` runs only in `tick()`, after `_pull_once()` returns. - `_pull_once()` returns only after the last segment of its round has landed, and `segment_in()` has just reset the clock. - So the rule only looks at the one moment it cannot be true. A 16 s stall the viewer sat through was 0 s old by the time it was checked. - `supervise()`, which classifies on its own, is called only from the tests. The tests check the rules, not when production runs them. - **The degrading rules sat just under their thresholds.** - Downloads took 3–4.5 s for 4 s of video, so the slow rule counted 1–2 of the last 5. - Each stall made one release late, so the late rule counted 2 of 5. - The channel ran at 0.64× real time and still read as Healthy. - **Filler never covered a stall.** - `_maybe_fill` holds filler back while any appended segment is still being fetched. - Each round queued about ten segments and took about 45 s to fetch them, so one was nearly always in flight. - Filler was listed once in five minutes. - **Skipped game was silent.** Segments that left the source's window before they were fetched only show as an INFO `clock: seq N splices` line. - **The trouble count counts recoveries, not stalls.** `trouble.stall()` is called only when a recovery starts, so the card showed 0 stalls. - **The swap that finally happened changed the picture's shape.** That crashes Emby's hardware transcode until #39 lands, so the remedy ended the session. ## Proposed change - **Measure what the viewer gets,** on the playout clock that already ticks every second, independent of the puller: - the served edge's age: seconds since the served playlist last grew; - the served rate: game seconds released per wall second over the last minute; - game skipped: segments that left the source's window unfetched, and timestamp jumps within one upstream; - how far the channel is behind the source's edge (from #56). - **Classify on those, on that clock.** Rough thresholds: - The served edge older than one target duration: list filler now, even with a fetch in flight. A segment that lands later splices back in as today. Count a stall. - A served rate under 0.9 for a minute, two stalls in two minutes, or any skipped game: degrading. - The served edge older than two target durations, or three stalls in two minutes: failed, and recovery starts. - **Find out whose fault it is before acting.** - If the source's edge keeps advancing while the channel falls behind, the fault is the channel's. Catch up by jumping to the edge and fetching in parallel (#56). Don't swap, and don't cool the link. - If the source's edge stalls too, recover or swap as today. - **Make a different-shape swap the last step while #39 is open.** First re-read the link, then try a standby or a ladder link of the same picture size. - **Base the card on the viewer's side.** - Never say "Nothing to do" while the served edge is stalling. - Count every stall the viewer sees in the trouble window, not only recoveries. - Show the served rate beside the slow, short and late strips. ## Done when These run through the real loops (`_run` and `_playout`), not only `supervise()`: - A `_pull_once` that blocks for 15 s on one download gets filler listed within one target duration plus a second, a stall counted, and a card that does not say Healthy. - A source delivered at 0.7× real time, with no single download slower than its own duration, turns the channel degrading within a minute, and the channel acts. - Segments that leave the source's window unfetched are counted, and the channel turns degrading. - A replay of tonight's timings turns the card from Healthy within the first minute, and no swap to another picture size happens before a same-shape option has been tried. Related: #56 (tonight's cause), #39 (a splice keeps the stream's shape), #50 (filler before the player runs dry), #53 (the buffer), #40 (health notices dropped frames).
Author
Owner

Closing: deployed 2026-10-31 (PR #60 and #61).

What changed:

  • The watch: the buffer's one-second clock classifies the stream and acts (recovery, or a swap in its own task). It never waits on a download.
  • Filler: it goes in ahead of a segment still on its way.
  • The card: shows the stalls the viewer sees, the game served per second, the lag behind the source and the game skipped.

Seen live: on Kent State vs. Sacramento State the card read Healthy and Nothing to do while the viewer was served at 1.0× real time with a full buffer. One stream played for about 4.5 min without a swap.

Found on the first live run and fixed in #61: the lag rule ("three behind the source") fired on one-second bursts while the viewer was fine, and with a standby ready the channel swapped about once a minute. Now the lag alone degrades nothing and stays on the card as a chip. Falling behind means game skipped, and Stalling means dry spells or under 0.9× real time.

Not seen live yet: a real stall; none happened during the runs. The tests cover it through the real loops: on virtual time, a 25 s download gets filler within a target duration and a stall counted, and two dry spells in two minutes make the stream Stalling.

Closing: deployed 2026-10-31 (PR #60 and #61). **What changed:** - **The watch:** the buffer's one-second clock classifies the stream and acts (recovery, or a swap in its own task). It never waits on a download. - **Filler:** it goes in ahead of a segment still on its way. - **The card:** shows the stalls the viewer sees, the game served per second, the lag behind the source and the game skipped. **Seen live:** on Kent State vs. Sacramento State the card read Healthy and Nothing to do while the viewer was served at 1.0× real time with a full buffer. One stream played for about 4.5 min without a swap. **Found on the first live run and fixed in #61:** the lag rule ("three behind the source") fired on one-second bursts while the viewer was fine, and with a standby ready the channel swapped about once a minute. Now the lag alone degrades nothing and stays on the card as a chip. Falling behind means game skipped, and Stalling means dry spells or under 0.9× real time. **Not seen live yet:** a real stall; none happened during the runs. The tests cover it through the real loops: on virtual time, a 25 s download gets filler within a target duration and a stall counted, and two dry spells in two minutes make the stream Stalling.
Author
Owner

First real stall seen live, 2026-10-31 17:47 UTC (a test tune of Spurs vs. C Palace on develop 1938e97).

The channel ran dry once: 4 s of filler, served at 0.8× real time. The card read Stalling and "Let it work", and the channel swapped to its ready standby, iohit.st, within 3 s. It was Healthy at 1.0× real time with a 12 s buffer for the next four minutes.

**First real stall seen live, 2026-10-31 17:47 UTC** (a test tune of Spurs vs. C Palace on develop 1938e97). The channel ran dry once: 4 s of filler, served at 0.8× real time. The card read Stalling and "Let it work", and the channel swapped to its ready standby, iohit.st, within 3 s. It was Healthy at 1.0× real time with a 12 s buffer for the next four minutes.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/sports-breaker#57
No description provided.