Say where a stuck comparison is stuck (diagnostics) #176

Merged
cmoriarty merged 1 commit from diagnose-stuck-compare into main 2026-11-15 22:07:54 -05:00
Owner

What #175's lines showed

The live app measures no standby because comparisons pile up behind one that never ends:

Report (UTC, 2026-11-16) Comparisons running The oldest for
02:49 1 20 s
02:54 26 272 s
02:59 65 573 s

Read from the container at 03:00, read only:

  • one worker thread of the app ran at full speed in user space, from about 02:50:50 on;
  • that thread made no system call and took no new memory;
  • the other worker threads were idle, and no ffmpeg was running.

One comparison runs at a time, so one that never ends stops every one after it. That is how every pair stayed "pending" after #172's deploy: the few comparisons that finished first were too few to reach "No match".

Before it stuck the measuring worked. Two pairs were found, and a recovery at 02:47:26 on Stars vs. Panthers was the first aimed change of stream live: it moved the picture −0.1 s, by the timestamps.

What this adds (logging only)

  • Each comparison and each join's check says what it is doing:
    • waiting for the gate;
    • decoding its sound, with its size in bytes;
    • decoding the sound searched in, with the piece number and its size;
    • locating N s of sound in M s, with the number of pieces and the longest;
    • waiting to look again.
  • The report names the oldest one and the one under way.
  • When the oldest has run past two minutes, the report says every busy thread's Python stack, one line a thread.

Checks

make test: 1,485 passed, 1 skipped. The new test holds a comparison in its search and checks three things: the report says what it was doing, it names the busy thread's stack, and the comparison is no longer listed once it ends.

🤖 Generated with Claude Code

## What #175's lines showed The live app measures no standby because comparisons pile up behind one that never ends: | Report (UTC, 2026-11-16) | Comparisons running | The oldest for | | --- | --- | --- | | 02:49 | 1 | 20 s | | 02:54 | 26 | 272 s | | 02:59 | 65 | 573 s | Read from the container at 03:00, read only: - one worker thread of the app ran at full speed in user space, from about 02:50:50 on; - that thread made no system call and took no new memory; - the other worker threads were idle, and no `ffmpeg` was running. One comparison runs at a time, so one that never ends stops every one after it. That is how every pair stayed "pending" after #172's deploy: the few comparisons that finished first were too few to reach "No match". Before it stuck the measuring worked. Two pairs were found, and a recovery at 02:47:26 on Stars vs. Panthers was the first aimed change of stream live: it moved the picture −0.1 s, by the timestamps. ## What this adds (logging only) - **Each comparison and each join's check says what it is doing:** - waiting for the gate; - decoding its sound, with its size in bytes; - decoding the sound searched in, with the piece number and its size; - locating N s of sound in M s, with the number of pieces and the longest; - waiting to look again. - **The report names the oldest one and the one under way.** - **When the oldest has run past two minutes**, the report says every busy thread's Python stack, one line a thread. ## Checks `make test`: 1,485 passed, 1 skipped. The new test holds a comparison in its search and checks three things: the report says what it was doing, it names the busy thread's stack, and the comparison is no longer listed once it ends. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The first reports (#175) showed why no standby is measured on the live
app: comparisons pile up behind one that never ends. At 02:59 UTC on
2026-11-16, 65 comparisons were waiting, the oldest for 573 s, while one
worker thread ran at full speed in user space, with no system call and
no new memory, from 02:50:50 on. One comparison runs at a time, so one
that never ends stops all of them.

Still logging only:

- Each comparison and each join's check says what it is doing
  (Running.stage): waiting for the gate, decoding its sound (bytes),
  decoding the sound searched in (piece, bytes), locating N s of sound
  in M s (pieces, the longest), waiting to look again.
- The report names the oldest one and the one under way.
- When the oldest has run past two minutes, the report says every busy
  thread's Python stack, one line a thread.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/sports-breaker!176
No description provided.