Measuring stops for good a few minutes after the app starts (an OpenBLAS dot product never returns) #177

Closed
opened 2026-11-15 22:35:07 -05:00 by cmoriarty · 0 comments
Owner

What happens

A few minutes after the app starts, it stops measuring where standbys are against the stream playing, for good. Every pair stays "pending", so no swap is aimed (#168): every change of stream records "not aimed: no match". It is so on main since #167, and was seen after the deploys of #172, #175 and #176 on 2026-11-16.

Why

  • One comparison never ends. The comparisons run one at a time, so all the others wait behind it. The report lines of #175 and #176 showed it:
    • 02:59 UTC: 65 waiting, the oldest for 573 s;
    • 03:04 UTC: 103 waiting, the oldest for 874 s.
  • The one under way was "locating 3.0 s of sound in 74.0 s", stuck in sound.py locate → numpy.linalg.norm → x.dot(x). That dot product runs in OpenBLAS (numpy 2.5.5, scipy-openblas 0.3.34), and its worker thread spun a core at full speed with no system call.
  • A fresh Python in the image starts 17 OpenBLAS helper threads when numpy loads; the app's process had none left. A dot product of 24,000 numbers is split among those helpers and waits for them, forever once they are gone.
  • Reproduced in the image (a separate short Python, 20 s limit): one thread runs that dot product in a loop while another starts child processes with fork().
    • As the image runs OpenBLAS now: 0 dot products a second for 12 seconds; hung on the first.
    • With OPENBLAS_NUM_THREADS=1: about 100,000 a second.

Done when

  • The app and the image run OpenBLAS on one thread.
  • sound.locate makes no call into OpenBLAS.
  • On the live app, the report shows comparisons finishing for an hour and more, and standbys found.
## What happens A few minutes after the app starts, it stops measuring where standbys are against the stream playing, for good. Every pair stays "pending", so no swap is aimed (#168): every change of stream records "not aimed: no match". It is so on `main` since #167, and was seen after the deploys of #172, #175 and #176 on 2026-11-16. ## Why - One comparison never ends. The comparisons run one at a time, so all the others wait behind it. The report lines of #175 and #176 showed it: - 02:59 UTC: 65 waiting, the oldest for 573 s; - 03:04 UTC: 103 waiting, the oldest for 874 s. - The one under way was "locating 3.0 s of sound in 74.0 s", stuck in `sound.py` `locate` → `numpy.linalg.norm` → `x.dot(x)`. That dot product runs in OpenBLAS (numpy 2.5.5, scipy-openblas 0.3.34), and its worker thread spun a core at full speed with no system call. - A fresh Python in the image starts 17 OpenBLAS helper threads when numpy loads; the app's process had none left. A dot product of 24,000 numbers is split among those helpers and waits for them, forever once they are gone. - Reproduced in the image (a separate short Python, 20 s limit): one thread runs that dot product in a loop while another starts child processes with `fork()`. - As the image runs OpenBLAS now: 0 dot products a second for 12 seconds; hung on the first. - With `OPENBLAS_NUM_THREADS=1`: about 100,000 a second. ## Done when - The app and the image run OpenBLAS on one thread. - `sound.locate` makes no call into OpenBLAS. - On the live app, the report shows comparisons finishing for an hour and more, and standbys found.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/sports-breaker#177
No description provided.