Admin: how far each standby is from the stream playing, and how far each swap moved the picture #167

Closed
opened 2026-11-14 19:25:46 -05:00 by cmoriarty · 2 comments
Owner

What happens

The owner, 2026-11-14: "Currently, the user experience when watching is the video will abruptly jump to another point in time, and it's annoying when it goes back in time because you have to re-watch something, and confusing when it jumps forward because you missed something."

Every change of stream starts the new one three segments from its end (Timeline.append_upstream, SPLICE_EDGE_SEGMENTS), and nothing compares where the two streams are in the game. Measured that afternoon on four games:

Swap The picture moved
To a faster stream (13 readings on three games) between back 6 s and forward 7 s
To a slower stream (one reading, and a real swap at 15:59 EST) back 23.8 s and 23.6 s
To the same stream on another site (one reading, eight in the log) back 3 to 9 s

Two streams of one game ran 5 to 16 s apart. The app knows none of this today, and Admin cannot say how far a swap moved the picture.

This is step 1 of 3 in the milestone. It changes nothing a viewer sees.

The idea

Hear where each standby is. The watch already downloads each standby's newest segment every 45 s (watch._sample) and throws it away. Decode its sound and find it in the sound of the stream playing, which the channel holds in memory (its buffered and lately served segments). Where it fits gives the relation between the two streams: how many seconds apart their clocks are for the same moment of the game.

What the research found (notes and scripts: /Users/cmoriarty/Library/Application Support/Claude/side-session-notes/local_daf105a8-87c5-4403-af2c-b78517c66b67/stream-swap-research.md; the owner's page: https://claude.ai/artifact/TCY5qaAe54rzrNnmjajbok):

  • Sound decoded to 8 kHz mono and cross-correlated. The right game scored 0.35 to 0.98, against 0.03 to 0.2 at the next best place. A wrong game scored 0.02. All 15 readings of the same game matched; all 4 against a wrong game did not.
  • One 3 s segment is enough. About 0.1 s to decode and 0.1 to 0.25 s to compare, with numpy, which the backend does not depend on yet. The image already has ffmpeg.
  • The relation between two streams' clocks moved 0.001 s in 20 minutes, while the gap between their newest segments wobbled by a segment or two. So the relation is the thing to keep.
  • Sites that carry one stream need no listening: their timestamps agree (stream_identity).

Three things the build has to get right:

  • Keep the relation between the streams' own clocks, not against the channel's clock. The channel's clock moves against the game at every swap that is not aimed.
  • A standby that is ahead shows moments the playing stream has not reached. Keep the sample's sound for half a minute and look again, or it is never found.
  • Keep it per pair of streams (by stream identity), so sites carrying one stream share it.

Rules that keep it honest

The owner, 2026-11-14:

  • "it is fairly common for the streams to just be wrong, and this new feature will need to be robust to that scenario."
  • "sometimes the streams are from different markets and have different commercials, but they are showing the same game. We will need to be robust to this scenario as well."

So:

  • A sample matches only with a high score and a clear margin over the next best place (start: 0.3, and twice the next).
  • A relation is trusted once matches at least 3 minutes apart agree within 0.1 s. Until then it is found, not trusted. Two different games on sister channels can carry the same commercial, so one match, or several inside one break, could be a commercial.
  • A sample that does not match changes nothing. Different commercials, a break on one stream, halftime: the relation stays, with the time it was last confirmed.
  • A match at another place replaces the relation, starts trust over, and is logged with the size of the change.
  • The relation is dropped when either stream's own clock jumps (it restarted) or the link turns to another stream.
  • A link showing another game never shows a figure. It reads "No match". Acting on that (demoting the link, or saying which of two links is the wrong one) is not part of this ticket.

How a relation behaves across a break when the two streams carry different commercials was not measured: no such pair was seen. This step logs it, and step 2 is built on what it shows.

Show it

  • Game page, Links table: each kept link says where it is against the stream playing: "11.6 s ahead", "4.8 s behind", "Same stream", "No match" (after about 5 minutes of samples with none: another game, or another broadcast of this one), and "matched 3 min ago" when the last samples did not confirm it.
  • Every change of stream says how far the picture moved, on its Log line and on the Link lane: "Picture moved back 6.0 s", "forward 7.4 s", or "not known". From the relation, or from the timestamps when both sites carry one stream.

What it costs

  • No new request for a stream in the clear. The segment is already downloaded.
  • An encrypted standby needs its key to be heard: one small request to the key host per comparison. These are the embedsports player's streams; the watch downloads their segments today and cannot read them. Keep it low: compare every sample until the relation is trusted, then every few minutes.
    • Not known: whether the key itself changes when its address does. The address was different on every read on 2026-11-14. If the key stays, one request per session is enough.
    • The key host is the one #136 renews sessions at, so the two should agree on how often it is asked.
  • Only games with an open channel are compared. A kept game with no channel has no playing stream to compare with.

Data it needs

  • On each link in a channel's admin view: offset_seconds (positive when the link is ahead of the stream playing), offset_state (same_stream, trusted, found, stale, none) and offset_confirmed_at.
  • On each change-of-stream event (Channel._swap_fields): moved_seconds (negative when the picture went back, null when not known) and moved_by (timestamps or sound).
  • A log line for every relation found, changed or dropped, with the pair and the seconds.

Done when

In the unit lane, with streams generated for the test (no broadcast sound in the repository): one programme cut into 3 s and 5 s segments at different delays, a second programme, and a pair with different commercials spliced into the same programme.

  • A standby 11.6 s ahead reads "11.6 s ahead" within 0.1 s, and is trusted after matches three minutes apart.
  • A standby 4.8 s behind reads "4.8 s behind" after its first sample.
  • A standby showing another programme never shows a figure, and reads "No match".
  • A pair with different commercials keeps its figure through the break, reads "matched … ago" during it, and is confirmed at the same figure after it.
  • The same commercial at two different times on two different programmes does not become a trusted relation.
  • A standby whose stream restarts loses its figure until it matches again.
  • A swap between sites carrying one stream records the seconds it moved from the timestamps; between two matched streams, from the relation; otherwise not known.
  • Requests are counted in the test: none added for a stream in the clear; for an encrypted one, key requests only, and fewer once trusted.

Live: on a game with two different streams the Links table shows the gap, and a Swap by hand records how far the picture moved.

Specs: stream-supervision, admin-page, event-log.

## What happens The owner, 2026-11-14: "Currently, the user experience when watching is the video will abruptly jump to another point in time, and it's annoying when it goes back in time because you have to re-watch something, and confusing when it jumps forward because you missed something." Every change of stream starts the new one three segments from its end (`Timeline.append_upstream`, `SPLICE_EDGE_SEGMENTS`), and nothing compares where the two streams are in the game. Measured that afternoon on four games: | Swap | The picture moved | | --- | --- | | To a faster stream (13 readings on three games) | between back 6 s and forward 7 s | | To a slower stream (one reading, and a real swap at 15:59 EST) | back 23.8 s and 23.6 s | | To the same stream on another site (one reading, eight in the log) | back 3 to 9 s | Two streams of one game ran 5 to 16 s apart. The app knows none of this today, and Admin cannot say how far a swap moved the picture. This is step 1 of 3 in the milestone. It changes nothing a viewer sees. ## The idea **Hear where each standby is.** The watch already downloads each standby's newest segment every 45 s (`watch._sample`) and throws it away. Decode its sound and find it in the sound of the stream playing, which the channel holds in memory (its buffered and lately served segments). Where it fits gives the relation between the two streams: how many seconds apart their clocks are for the same moment of the game. What the research found (notes and scripts: `/Users/cmoriarty/Library/Application Support/Claude/side-session-notes/local_daf105a8-87c5-4403-af2c-b78517c66b67/stream-swap-research.md`; the owner's page: https://claude.ai/artifact/TCY5qaAe54rzrNnmjajbok): - Sound decoded to 8 kHz mono and cross-correlated. The right game scored 0.35 to 0.98, against 0.03 to 0.2 at the next best place. A wrong game scored 0.02. All 15 readings of the same game matched; all 4 against a wrong game did not. - One 3 s segment is enough. About 0.1 s to decode and 0.1 to 0.25 s to compare, with numpy, which the backend does not depend on yet. The image already has ffmpeg. - The relation between two streams' clocks moved 0.001 s in 20 minutes, while the gap between their newest segments wobbled by a segment or two. So the relation is the thing to keep. - Sites that carry one stream need no listening: their timestamps agree (`stream_identity`). Three things the build has to get right: - **Keep the relation between the streams' own clocks,** not against the channel's clock. The channel's clock moves against the game at every swap that is not aimed. - **A standby that is ahead shows moments the playing stream has not reached.** Keep the sample's sound for half a minute and look again, or it is never found. - **Keep it per pair of streams** (by stream identity), so sites carrying one stream share it. ## Rules that keep it honest The owner, 2026-11-14: - "it is fairly common for the streams to just be wrong, and this new feature will need to be robust to that scenario." - "sometimes the streams are from different markets and have different commercials, but they are showing the same game. We will need to be robust to this scenario as well." So: - A sample **matches** only with a high score and a clear margin over the next best place (start: 0.3, and twice the next). - A relation is **trusted** once matches at least 3 minutes apart agree within 0.1 s. Until then it is found, not trusted. Two different games on sister channels can carry the same commercial, so one match, or several inside one break, could be a commercial. - **A sample that does not match changes nothing.** Different commercials, a break on one stream, halftime: the relation stays, with the time it was last confirmed. - **A match at another place replaces the relation,** starts trust over, and is logged with the size of the change. - **The relation is dropped** when either stream's own clock jumps (it restarted) or the link turns to another stream. - **A link showing another game never shows a figure.** It reads "No match". Acting on that (demoting the link, or saying which of two links is the wrong one) is not part of this ticket. How a relation behaves across a break when the two streams carry different commercials was not measured: no such pair was seen. This step logs it, and step 2 is built on what it shows. ## Show it - **Game page, Links table:** each kept link says where it is against the stream playing: "11.6 s ahead", "4.8 s behind", "Same stream", "No match" (after about 5 minutes of samples with none: another game, or another broadcast of this one), and "matched 3 min ago" when the last samples did not confirm it. - **Every change of stream says how far the picture moved,** on its Log line and on the Link lane: "Picture moved back 6.0 s", "forward 7.4 s", or "not known". From the relation, or from the timestamps when both sites carry one stream. ## What it costs - **No new request for a stream in the clear.** The segment is already downloaded. - **An encrypted standby needs its key to be heard:** one small request to the key host per comparison. These are the embedsports player's streams; the watch downloads their segments today and cannot read them. Keep it low: compare every sample until the relation is trusted, then every few minutes. - Not known: whether the key itself changes when its address does. The address was different on every read on 2026-11-14. If the key stays, one request per session is enough. - The key host is the one #136 renews sessions at, so the two should agree on how often it is asked. - **Only games with an open channel are compared.** A kept game with no channel has no playing stream to compare with. ## Data it needs - On each link in a channel's admin view: `offset_seconds` (positive when the link is ahead of the stream playing), `offset_state` (`same_stream`, `trusted`, `found`, `stale`, `none`) and `offset_confirmed_at`. - On each change-of-stream event (`Channel._swap_fields`): `moved_seconds` (negative when the picture went back, null when not known) and `moved_by` (`timestamps` or `sound`). - A log line for every relation found, changed or dropped, with the pair and the seconds. ## Done when In the unit lane, with streams generated for the test (no broadcast sound in the repository): one programme cut into 3 s and 5 s segments at different delays, a second programme, and a pair with different commercials spliced into the same programme. - A standby 11.6 s ahead reads "11.6 s ahead" within 0.1 s, and is trusted after matches three minutes apart. - A standby 4.8 s behind reads "4.8 s behind" after its first sample. - A standby showing another programme never shows a figure, and reads "No match". - A pair with different commercials keeps its figure through the break, reads "matched … ago" during it, and is confirmed at the same figure after it. - The same commercial at two different times on two different programmes does not become a trusted relation. - A standby whose stream restarts loses its figure until it matches again. - A swap between sites carrying one stream records the seconds it moved from the timestamps; between two matched streams, from the relation; otherwise not known. - Requests are counted in the test: none added for a stream in the clear; for an encrypted one, key requests only, and fewer once trusted. Live: on a game with two different streams the Links table shows the gap, and a Swap by hand records how far the picture moved. Specs: `stream-supervision`, `admin-page`, `event-log`.
Author
Owner

Reference from the research, not production code: the comparison as the scripts did it (analyze.py in the notes folder, with numpy). It is here so the method does not depend on a scratch folder.

SR = 8000

def pcm(path):  # one segment's sound, 8 kHz mono, float32
    raw = run(["ffmpeg", "-v", "error", "-i", path, "-vn", "-ac", "1", "-ar", str(SR), "-f", "f32le", "-"])
    return np.frombuffer(raw, dtype=np.float32).copy()

def preemph(x):  # lift the highs, so speech and whistles count more than rumble
    y = np.empty_like(x); y[0] = 0.0; y[1:] = x[1:] - 0.97 * x[:-1]
    return y

def ncc(ref, q):
    """Normalised cross-correlation of q against every position of ref."""
    n = len(ref) + len(q) - 1
    size = 1 << (n - 1).bit_length()
    q = q - q.mean(); qn = np.linalg.norm(q)
    if qn == 0 or len(ref) < len(q):
        return np.zeros(1)
    f = np.fft.rfft(ref, size) * np.conj(np.fft.rfft(q, size))
    c = np.fft.irfft(f, size)[: len(ref) - len(q) + 1]
    csum = np.concatenate(([0.0], np.cumsum(ref.astype(np.float64) ** 2)))
    energy = csum[len(q):] - csum[: len(ref) - len(q) + 1]
    msum = np.concatenate(([0.0], np.cumsum(ref.astype(np.float64))))
    mean = (msum[len(q):] - msum[: len(ref) - len(q) + 1]) / len(q)
    var = np.maximum(energy - len(q) * mean ** 2, 1e-12)
    return c / (qn * np.sqrt(var))

def peak(c, rate, guard=1.0):
    """The best place, its score, and the best score more than `guard` seconds away."""
    i = int(np.argmax(c)); g = int(guard * rate)
    rest = np.concatenate((c[: max(0, i - g)], c[i + g + 1:]))
    nxt = float(rest.max()) if len(rest) else 0.0
    return {"lag": i / rate, "score": float(c[i]), "next": nxt}

How it was used:

  • ref is the playing stream's sound laid on its own timestamps (each segment's samples placed by its first audio timestamp), q is one segment of the standby. Both go through preemph first.
  • The relation is (the standby segment's first audio timestamp) - (the timestamp of ref at the best place).
  • The research called a match at a score over 0.2 and twice the next best. The 15 right readings scored 0.35 to 0.98 with the next best at 0.03 to 0.2; the 4 wrong-game readings scored 0.016 to 0.024 with the next best about the same. So the ticket starts at 0.3 and twice.
  • 44 of 45 single segments of 3 to 5 s passed "0.3 and twice". Samples of one second still landed on the right place, with thinner margins.
  • A standby that is ahead is found only once the playing stream has reached the moment it showed: the research waited 65 to 75 s after taking the standby's segments before reading the playing stream.
  • The loudness envelope and small video frames found the same places, with thinner margins. Not worth it beside the waveform.
  • EXT-X-PROGRAM-DATE-TIME is the packager's own clock: within 3 s of the fetch on two streams that were 16 s apart in the game. No use here.
Reference from the research, not production code: the comparison as the scripts did it (`analyze.py` in the notes folder, with numpy). It is here so the method does not depend on a scratch folder. ```python SR = 8000 def pcm(path): # one segment's sound, 8 kHz mono, float32 raw = run(["ffmpeg", "-v", "error", "-i", path, "-vn", "-ac", "1", "-ar", str(SR), "-f", "f32le", "-"]) return np.frombuffer(raw, dtype=np.float32).copy() def preemph(x): # lift the highs, so speech and whistles count more than rumble y = np.empty_like(x); y[0] = 0.0; y[1:] = x[1:] - 0.97 * x[:-1] return y def ncc(ref, q): """Normalised cross-correlation of q against every position of ref.""" n = len(ref) + len(q) - 1 size = 1 << (n - 1).bit_length() q = q - q.mean(); qn = np.linalg.norm(q) if qn == 0 or len(ref) < len(q): return np.zeros(1) f = np.fft.rfft(ref, size) * np.conj(np.fft.rfft(q, size)) c = np.fft.irfft(f, size)[: len(ref) - len(q) + 1] csum = np.concatenate(([0.0], np.cumsum(ref.astype(np.float64) ** 2))) energy = csum[len(q):] - csum[: len(ref) - len(q) + 1] msum = np.concatenate(([0.0], np.cumsum(ref.astype(np.float64)))) mean = (msum[len(q):] - msum[: len(ref) - len(q) + 1]) / len(q) var = np.maximum(energy - len(q) * mean ** 2, 1e-12) return c / (qn * np.sqrt(var)) def peak(c, rate, guard=1.0): """The best place, its score, and the best score more than `guard` seconds away.""" i = int(np.argmax(c)); g = int(guard * rate) rest = np.concatenate((c[: max(0, i - g)], c[i + g + 1:])) nxt = float(rest.max()) if len(rest) else 0.0 return {"lag": i / rate, "score": float(c[i]), "next": nxt} ``` How it was used: - `ref` is the playing stream's sound laid on its own timestamps (each segment's samples placed by its first audio timestamp), `q` is one segment of the standby. Both go through `preemph` first. - The relation is `(the standby segment's first audio timestamp) - (the timestamp of ref at the best place)`. - The research called a match at a score over 0.2 and twice the next best. The 15 right readings scored 0.35 to 0.98 with the next best at 0.03 to 0.2; the 4 wrong-game readings scored 0.016 to 0.024 with the next best about the same. So the ticket starts at 0.3 and twice. - 44 of 45 single segments of 3 to 5 s passed "0.3 and twice". Samples of one second still landed on the right place, with thinner margins. - A standby that is ahead is found only once the playing stream has reached the moment it showed: the research waited 65 to 75 s after taking the standby's segments before reading the playing stream. - The loudness envelope and small video frames found the same places, with thinner margins. Not worth it beside the waveform. - `EXT-X-PROGRAM-DATE-TIME` is the packager's own clock: within 3 s of the fetch on two streams that were 16 s apart in the game. No use here.
Author
Owner

Merged and live.

  • Pull request #171 merged into main as 0a9d8e1 (the change itself is f32c5f6). The OpenSpec change was archived in a96b034, pushed two seconds after the merge so that one build and one restart carried both: the merge's own build (run 3973) was cancelled by the newer push, as publish-image.yml is set to do, and the build of a96b034 (run 3974) is success.
  • Deployed 2026-11-15 at 11:46 EST. The app was away for 6 s and came back on a96b034 (/api/version).

Seen live in the first quarter of an hour, on Lions vs. Patriots, from the app's own API and the container's log (nothing pressed, no site read by hand):

Kept link Stream First match Trusted The game page says
topsurge.st, standby B, in the clear 15 s after the link was verified, about 1.5 min after the restart 184 s later 30.0 s behind, then 31.3
s2watch.cyou, backup C, encrypted 60 s after the link was verified 187 s later 26.9 s behind, then 25.3 and 25.8
  • The log since the restart holds the comparison's four lines (two "found", two "held … (trusted)") and no error.
  • The stream playing (reddittrend.sbs, stream A) runs 25 to 31 s ahead of the other two, which are about 5 s from each other. That is wider than anything in the research (5 to 16 s), and it matters for #168: with the head start at its default of 15 s (the live setting), a swap to this standby could not meet the stream playing. It would repeat 15 s or more, where today it would go back 40 to 50 s (the gap plus three segments; worked out, not seen).

Not seen yet:

  • A change of stream: no swap since the deploy, so no "moved" tag on a Log row from the real app.
  • "No match" on a link that shows something else.
  • A relation across a break where the two streams carry different commercials. Both relations held across their first three minutes; what was on air in those minutes was not looked at.
  • An encrypted standby's key requests, counted on the wire. The unit lane counts one per due comparison.

How good the figure is: to a second or two. The relation between two streams' clocks was logged once per link and did not move, but the seconds shown also need each link's clock against the wall. That comes from the moment the watch first saw a segment listed: it reads a kept link's playlist every 1 to 3 s, samples a standby's segment every 45 s at most, and keeps ten minutes of readings. So the figure moved between reads while the relation stood still (the table's last column). #168 joins on the relation and the segments' own timestamps, not on this figure.

Merged and live. - Pull request #171 merged into `main` as 0a9d8e1 (the change itself is f32c5f6). The OpenSpec change was archived in a96b034, pushed two seconds after the merge so that one build and one restart carried both: the merge's own build (run 3973) was cancelled by the newer push, as `publish-image.yml` is set to do, and the build of a96b034 (run 3974) is `success`. - Deployed 2026-11-15 at 11:46 EST. The app was away for 6 s and came back on a96b034 (`/api/version`). **Seen live** in the first quarter of an hour, on Lions vs. Patriots, from the app's own API and the container's log (nothing pressed, no site read by hand): | Kept link | Stream | First match | Trusted | The game page says | | --- | --- | --- | --- | --- | | topsurge.st, standby | B, in the clear | 15 s after the link was verified, about 1.5 min after the restart | 184 s later | 30.0 s behind, then 31.3 | | s2watch.cyou, backup | C, encrypted | 60 s after the link was verified | 187 s later | 26.9 s behind, then 25.3 and 25.8 | - The log since the restart holds the comparison's four lines (two "found", two "held … (trusted)") and no error. - The stream playing (reddittrend.sbs, stream A) runs 25 to 31 s ahead of the other two, which are about 5 s from each other. That is wider than anything in the research (5 to 16 s), and it matters for #168: with the head start at its default of 15 s (the live setting), a swap to this standby could not meet the stream playing. It would repeat 15 s or more, where today it would go back 40 to 50 s (the gap plus three segments; worked out, not seen). **Not seen yet:** - A change of stream: no swap since the deploy, so no "moved" tag on a Log row from the real app. - "No match" on a link that shows something else. - A relation across a break where the two streams carry different commercials. Both relations held across their first three minutes; what was on air in those minutes was not looked at. - An encrypted standby's key requests, counted on the wire. The unit lane counts one per due comparison. **How good the figure is:** to a second or two. The relation between two streams' clocks was logged once per link and did not move, but the seconds shown also need each link's clock against the wall. That comes from the moment the watch first saw a segment listed: it reads a kept link's playlist every 1 to 3 s, samples a standby's segment every 45 s at most, and keeps ten minutes of readings. So the figure moved between reads while the relation stood still (the table's last column). #168 joins on the relation and the segments' own timestamps, not on this figure.
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/sports-breaker#167
No description provided.