The slot strip reads 1/6 on a llama.cpp server that serves one slot: the total is a constant, not the server's #184

Closed
opened 2026-10-16 22:50:42 -04:00 by cmoriarty · 1 comment
Owner

Reported by the user on 2026-10-17. While trogdor ran llama.cpp at -np 1 (run 67), the top bar's slot strip showed 1/6, though the server serves one sequence at a time.

Cause

/api/slots and /api/slots/history both report capacity from Config.max_concurrent_upstream. That is MAX_CONCURRENT_UPSTREAM = 6, measured once for vLLM's --max-num-seqs 6 (§0a) and overridable only by the environment, OSF_MAX_UPSTREAM. Nothing asks the server.

  • The strip, the Inference page's rows and the history chart all draw that 6.
  • With one real slot, the strip shows five free slots that do not exist.

Wanted

  • On a llama.cpp backend, the capacity is the server's own: total_slots from its GET /props. Read it at most once a minute, since tabs poll the strip.
  • vLLM does not publish --max-num-seqs, so it keeps the configured value, as does a server that does not answer.
  • /api/slots says where its number came from, so the hover can say "reported by llama.cpp" or "configured".
  • A test with a llama.cpp server reporting total_slots: 1: the strip reads 1/1 and the Inference page draws one row.
Reported by the user on 2026-10-17. While trogdor ran llama.cpp at `-np 1` (run 67), the top bar's slot strip showed **1/6**, though the server serves one sequence at a time. ## Cause `/api/slots` and `/api/slots/history` both report `capacity` from `Config.max_concurrent_upstream`. That is `MAX_CONCURRENT_UPSTREAM = 6`, measured once for vLLM's `--max-num-seqs 6` (§0a) and overridable only by the environment, `OSF_MAX_UPSTREAM`. Nothing asks the server. - The strip, the Inference page's rows and the history chart all draw that 6. - With one real slot, the strip shows five free slots that do not exist. ## Wanted - On a llama.cpp backend, the capacity is the server's own: `total_slots` from its `GET /props`. Read it at most once a minute, since tabs poll the strip. - vLLM does not publish `--max-num-seqs`, so it keeps the configured value, as does a server that does not answer. - `/api/slots` says where its number came from, so the hover can say "reported by llama.cpp" or "configured". - A test with a llama.cpp server reporting `total_slots: 1`: the strip reads 1/1 and the Inference page draws one row.
Author
Owner

Shipped in 0a4d053 (change slots-are-the-servers, commit b3a451d), live since 03:53Z on 2026-10-17, and archived in cfb2a50.

  • On a llama.cpp backend, the slot capacity is now the server's own: total_slots from its GET /props (the -np it was started with), read at most once a minute.
  • vLLM doesn't publish its --max-num-seqs, so it keeps the configured count. So does a server that doesn't answer.
  • /api/slots and /api/slots/history say which (capacity_source), and the strip's sentence says "the count the server reports" or "Braid's configured count".
  • Checked on production: on vLLM, /api/slots reads 6, configured. trogdor runs vLLM now, so the llama.cpp path stands on a unit test with a fake /props answering total_slots: 1 (the strip reads 1/1 and draws one slot). The next llama.cpp profile will show it live.

Closing. Please reopen if a llama.cpp server's strip still reads 6.

Shipped in 0a4d053 (change `slots-are-the-servers`, commit b3a451d), live since 03:53Z on 2026-10-17, and archived in cfb2a50. - On a llama.cpp backend, the slot capacity is now the server's own: `total_slots` from its `GET /props` (the `-np` it was started with), read at most once a minute. - vLLM doesn't publish its `--max-num-seqs`, so it keeps the configured count. So does a server that doesn't answer. - `/api/slots` and `/api/slots/history` say which (`capacity_source`), and the strip's sentence says "the count the server reports" or "Braid's configured count". - Checked on production: on vLLM, `/api/slots` reads 6, `configured`. trogdor runs vLLM now, so the llama.cpp path stands on a unit test with a fake `/props` answering `total_slots: 1` (the strip reads 1/1 and draws one slot). The next llama.cpp profile will show it live. Closing. Please reopen if a llama.cpp server's strip still reads 6.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#184
No description provided.