Confirm the dropped-connection retry (PR #134) against a live source #138

Closed
opened 2026-11-09 23:45:02 -05:00 by cmoriarty · 2 comments
Owner

PR #134 sends an upstream request once more when the server closes a kept-alive connection before answering (httpx.RemoteProtocolError). It was written from one event and is covered by its unit test only.

The event (2026-11-10 01:09:26Z): a thetvapp channel failed on "no new segment for 17 s; last error: playlist: upstream unreachable: RemoteProtocolError". The error was the channel's last one, not proven to be why it stalled.

To check

  • The retry leaves no log line. Add one at INFO (host, once per occurrence) so production shows how often it happens and on which hosts.
  • With that in place, look for a channel that keeps playing through a dropped connection, and for any channel still failing with RemoteProtocolError as its last error.
  • If the stall has another cause (thetvapp's listing gaps reached 15 s that night against a 16 s rule), say so here and open a ticket for that.

Done when production logs show the retry carrying a channel through, or show it is not what failed that channel.

PR #134 sends an upstream request once more when the server closes a kept-alive connection before answering (`httpx.RemoteProtocolError`). It was written from one event and is covered by its unit test only. **The event** (2026-11-10 01:09:26Z): a thetvapp channel failed on "no new segment for 17 s; last error: playlist: upstream unreachable: RemoteProtocolError". The error was the channel's last one, not proven to be why it stalled. **To check** - The retry leaves no log line. Add one at INFO (host, once per occurrence) so production shows how often it happens and on which hosts. - With that in place, look for a channel that keeps playing through a dropped connection, and for any channel still failing with `RemoteProtocolError` as its last error. - If the stall has another cause (thetvapp's listing gaps reached 15 s that night against a 16 s rule), say so here and open a ticket for that. **Done when** production logs show the retry carrying a channel through, or show it is not what failed that channel.
Author
Owner

The retry now logs upstream: <host> dropped a kept-alive connection; asked again. Merged in PR #140 (25fa577), deployed 2026-11-10 05:21Z, archived in 4952376. No followed game was live at the deploy, so none of it has been seen in production yet.

Left open for the check itself: look for that line in the container log beside a channel that kept playing, and for any channel that still fails with RemoteProtocolError as its last error.

The retry now logs `upstream: <host> dropped a kept-alive connection; asked again`. Merged in PR #140 (25fa577), deployed 2026-11-10 05:21Z, archived in 4952376. No followed game was live at the deploy, so none of it has been seen in production yet. Left open for the check itself: look for that line in the container log beside a channel that kept playing, and for any channel that still fails with `RemoteProtocolError` as its last error.
Author
Owner

Seen in production on 2026-11-10, so closing. The retry is PR #134 and its log line is PR #140 (25fa577).

  • 23:16:39Z: upstream: a.hatemamere3.space dropped a kept-alive connection; asked again, the only such line that evening.
  • No RemoteProtocolError appeared in the log around it, and no channel failed with it.
  • Florida St vs. Florida's channel, the only one running then, was still playing and graded steady at 23:23Z.

Not proven: the log names only the host, so it does not show whether that request was the channel's or a stream check's. A new ticket can follow if a channel fails with RemoteProtocolError as its last error.

Seen in production on 2026-11-10, so closing. The retry is PR #134 and its log line is PR #140 (25fa577). - 23:16:39Z: `upstream: a.hatemamere3.space dropped a kept-alive connection; asked again`, the only such line that evening. - No `RemoteProtocolError` appeared in the log around it, and no channel failed with it. - Florida St vs. Florida's channel, the only one running then, was still playing and graded steady at 23:23Z. **Not proven:** the log names only the host, so it does not show whether that request was the channel's or a stream check's. A new ticket can follow if a channel fails with `RemoteProtocolError` as its last error.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/sports-breaker#138
No description provided.