Run numpy's OpenBLAS on one thread, and keep it out of locate (#177) #178

Merged
cmoriarty merged 1 commit from blas-single-thread into main 2026-11-16 14:49:03 -05:00
Owner

Fixes #177.

The fault

A few minutes after each start, the live app stops measuring where standbys are against the stream playing, for good. Every pair stays "pending", so no swap is aimed (#168). This has been on main since #167. #172 only made it count, because aiming depends on the measurement.

What the diagnostics of #175 and #176 showed:

  • One comparison never ends, and all the others wait behind it. At 02:59 UTC, 65 were waiting, the oldest for 573 s. At 03:04 UTC, 103 were waiting, the oldest for 874 s.
  • Where it was stuck: "locating 3.0 s of sound in 74.0 s", in sound.py locate → numpy.linalg.norm → x.dot(x). That is a dot product in OpenBLAS (numpy 2.5.5, scipy-openblas 0.3.34). Its thread spun a core with no system call.
  • Why it never ends: a fresh Python in the image starts 17 OpenBLAS helper threads when numpy loads, but the app's process had none left. A dot product of 24,000 numbers is split among those helpers and waits for them, forever.

Reproduced in the image, in a separate short Python with a 20 s limit. One thread runs that dot product in a loop while another starts child processes with fork():

OpenBLAS Dot products finished, each second
As the image runs it now 0 0 0 0 0 0 0 0 0 0 0 0
OPENBLAS_NUM_THREADS=1 about 100,000

The fix

  • One thread for OpenBLAS. backend/app/__init__.py sets OPENBLAS_NUM_THREADS=1 before anything imports numpy, keeping a value already set. The Dockerfile sets it for every Python in the image. The app's arrays are small, so one thread loses nothing.
  • No call into OpenBLAS in sound.locate: it takes the sample's norm with numpy's own sum. That was the only OpenBLAS call in the app.

Checks

  • make test: 1,488 passed, 1 skipped. Three new tests:
    • locate finds its place with linalg.norm unavailable;
    • a fresh Python that imports app has OPENBLAS_NUM_THREADS=1;
    • the image's final stage sets it.
  • The image's own end-to-end lane runs in publish-image before latest is pushed.

After the deploy

  • The report lines (#175, #176) should show comparisons finishing, a handful running at a time and none for minutes, for an hour and more.
  • Standbys should read "found" and then "trusted".
  • Then the live swap tests can start.

🤖 Generated with Claude Code

Fixes #177. ## The fault A few minutes after each start, the live app stops measuring where standbys are against the stream playing, for good. Every pair stays "pending", so no swap is aimed (#168). This has been on `main` since #167. #172 only made it count, because aiming depends on the measurement. What the diagnostics of #175 and #176 showed: - **One comparison never ends, and all the others wait behind it.** At 02:59 UTC, 65 were waiting, the oldest for 573 s. At 03:04 UTC, 103 were waiting, the oldest for 874 s. - **Where it was stuck:** "locating 3.0 s of sound in 74.0 s", in `sound.py` `locate` → `numpy.linalg.norm` → `x.dot(x)`. That is a dot product in OpenBLAS (numpy 2.5.5, scipy-openblas 0.3.34). Its thread spun a core with no system call. - **Why it never ends:** a fresh Python in the image starts 17 OpenBLAS helper threads when numpy loads, but the app's process had none left. A dot product of 24,000 numbers is split among those helpers and waits for them, forever. Reproduced in the image, in a separate short Python with a 20 s limit. One thread runs that dot product in a loop while another starts child processes with `fork()`: | OpenBLAS | Dot products finished, each second | | --- | --- | | As the image runs it now | 0 0 0 0 0 0 0 0 0 0 0 0 | | `OPENBLAS_NUM_THREADS=1` | about 100,000 | ## The fix - **One thread for OpenBLAS.** `backend/app/__init__.py` sets `OPENBLAS_NUM_THREADS=1` before anything imports numpy, keeping a value already set. The `Dockerfile` sets it for every Python in the image. The app's arrays are small, so one thread loses nothing. - **No call into OpenBLAS in `sound.locate`:** it takes the sample's norm with numpy's own sum. That was the only OpenBLAS call in the app. ## Checks - `make test`: 1,488 passed, 1 skipped. Three new tests: - `locate` finds its place with `linalg.norm` unavailable; - a fresh Python that imports `app` has `OPENBLAS_NUM_THREADS=1`; - the image's final stage sets it. - The image's own end-to-end lane runs in publish-image before `latest` is pushed. ## After the deploy - The report lines (#175, #176) should show comparisons finishing, a handful running at a time and none for minutes, for an hour and more. - Standbys should read "found" and then "trusted". - Then the live swap tests can start. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
A few minutes after each start the live app stopped measuring for good:
one comparison never ended and every other waited behind it, so no
standby was found and no swap aimed. It was stuck in sound.locate's
numpy.linalg.norm, a dot product in OpenBLAS split among helper threads
the app's process no longer had (a fresh Python in the image starts 17
when numpy loads; the app had none left), so it waited for them forever,
spinning a core.

Reproduced in the image: a dot product of 24,000 numbers in a loop while
another thread starts child processes with fork() finished none in 12
seconds; with OPENBLAS_NUM_THREADS=1, about 100,000 a second.

- app/__init__.py sets OPENBLAS_NUM_THREADS=1 before anything imports
  numpy (a value already set is kept), and the image sets it for every
  Python in it.
- sound.locate takes the sample's norm with numpy's own sum: no call into
  OpenBLAS is left in the app.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/sports-breaker!178
No description provided.