A recovery re-reads a link that keeps failing while a ready standby waits #65

Closed
opened 2026-10-31 11:33:55 -04:00 by cmoriarty · 2 comments
Owner

What happens

A failed stream's recovery always starts by re-reading the same link (Channel._recover, step 1). If the page still yields a stream, the channel lands on it again, however often that link has just failed. The ready standby is only tried when the re-read finds nothing.

Seen live, 2026-10-31, Chelsea vs. Man Utd (UTC)

  • 14:17:27, 14:17:56, 14:18:36, 14:19:16: four recoveries on iohit.st in two minutes. Each landed and failed again within 40 s.
  • The standby: shd247.world had been ready since 13:42 and was only taken at 14:21:48, when a re-read finally found nothing.
  • What the viewer got, from the card's trouble record for the 10 minutes to 14:26: 7 stalls, 6 recoveries, 89 s stalled and 116 s of filler.

Decided

Re-read the link first as now (a fresh read often fixes an expired token), unless that link already needed a recovery in the last 5 minutes. Then the ready standby comes first, and the link cools as a swapped-out link does. This was my recommended default; the operator didn't choose otherwise when the batch was set up.

## What happens A failed stream's recovery always starts by re-reading the same link (`Channel._recover`, step 1). If the page still yields a stream, the channel lands on it again, however often that link has just failed. The ready standby is only tried when the re-read finds nothing. ## Seen live, 2026-10-31, Chelsea vs. Man Utd (UTC) - **14:17:27, 14:17:56, 14:18:36, 14:19:16:** four recoveries on iohit.st in two minutes. Each landed and failed again within 40 s. - **The standby:** shd247.world had been ready since 13:42 and was only taken at 14:21:48, when a re-read finally found nothing. - **What the viewer got,** from the card's trouble record for the 10 minutes to 14:26: 7 stalls, 6 recoveries, 89 s stalled and 116 s of filler. ## Decided Re-read the link first as now (a fresh read often fixes an expired token), unless that link already needed a recovery in the last 5 minutes. Then the ready standby comes first, and the link cools as a swapped-out link does. This was my recommended default; the operator didn't choose otherwise when the batch was set up.
Author
Owner

Deployed 2026-10-31 17:44 UTC (PR #67, 1938e97). Left open until a relapse is seen live.

Done when a channel's history shows "recovered on ( failed again within 5 min)". That needs a link that fails twice within five minutes while a standby is ready, so it can't be forced.

**Deployed 2026-10-31 17:44 UTC (PR #67, 1938e97). Left open until a relapse is seen live.** **Done when** a channel's history shows "recovered on <host> (<link> failed again within 5 min)". That needs a link that fails twice within five minutes while a standby is ready, so it can't be forced.
Author
Owner

Closed on the tests at the operator's word (2026-11-01): reopen, or file a new ticket, if production shows a problem.

Closed on the tests at the operator's word (2026-11-01): reopen, or file a new ticket, if production shows a problem.
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/sports-breaker#65
No description provided.