Inference page: prefill and decode speed go blank while a long request is decoding #351

Closed
opened 2026-10-31 11:02:16 -04:00 by cmoriarty · 1 comment
Owner

What happens

  • During run 163 (vLLM, apply, one request at a time), the Inference page's Prefill speed and Decode speed tiles read "—" and "no recent requests", while Braid's requests read 1 of 6. At the same moment (14:55Z), vLLM's own counters showed one request running and decoding at 31 tok/s.
  • The tiles count only requests that finished in the last five minutes (RECENT_S = 300 in src/osf/throughput.py; the "no recent requests" text in ui/src/components/Slots.tsx). Braid times a request only at its step-finish (RequestClock).
  • At xhigh effort a single request can stream for more than five minutes. So the tiles go blank exactly while one long request is generating, which is most of apply. The history charts behave the same way: a long request shows up as one bar when it ends.

What we want

  • While a request is in flight, the Decode speed tile shows its live rate: tokens streamed since its first token, over the time since then. RequestClock already sees the first message.part.delta and each one after it. The count can come from the server's own counter where one exists (vLLM vllm:generation_tokens_total, llama.cpp /slots n_decoded), and otherwise from the streamed text.
  • With nothing finished in the window, Prefill speed shows the last value and its age, for example "1,200 tok/s, 7 min ago", instead of a dash. Reused work already falls back to "all time" the same way.

Done when

  • Through a single request that runs longer than five minutes, both tiles show a number the whole time, and the decode tile tracks the server's own rate.
  • The speeds Braid serves on /metrics keep their finished-request meaning, or an in-flight gauge is added beside them.

Where to look: src/osf/throughput.py (RECENT_S, RequestClock, aggregate), src/osf/api/inference.py, ui/src/components/Slots.tsx (speedSub), ui/src/components/InferenceCharts.tsx. Related: #344, decode speed that counts a turn's tool time.

Seen by the owner during braid#350's run C (vLLM, Qwen3.8-27B), 2026-10-31.

**What happens** - During run 163 (vLLM, apply, one request at a time), the Inference page's Prefill speed and Decode speed tiles read "—" and "no recent requests", while Braid's requests read 1 of 6. At the same moment (14:55Z), vLLM's own counters showed one request running and decoding at 31 tok/s. - The tiles count only requests that *finished* in the last five minutes (`RECENT_S = 300` in `src/osf/throughput.py`; the "no recent requests" text in `ui/src/components/Slots.tsx`). Braid times a request only at its `step-finish` (`RequestClock`). - At xhigh effort a single request can stream for more than five minutes. So the tiles go blank exactly while one long request is generating, which is most of apply. The history charts behave the same way: a long request shows up as one bar when it ends. **What we want** - While a request is in flight, the Decode speed tile shows its live rate: tokens streamed since its first token, over the time since then. `RequestClock` already sees the first `message.part.delta` and each one after it. The count can come from the server's own counter where one exists (vLLM `vllm:generation_tokens_total`, llama.cpp `/slots` `n_decoded`), and otherwise from the streamed text. - With nothing finished in the window, Prefill speed shows the last value and its age, for example "1,200 tok/s, 7 min ago", instead of a dash. Reused work already falls back to "all time" the same way. **Done when** - Through a single request that runs longer than five minutes, both tiles show a number the whole time, and the decode tile tracks the server's own rate. - The speeds Braid serves on `/metrics` keep their finished-request meaning, or an in-flight gauge is added beside them. **Where to look:** `src/osf/throughput.py` (`RECENT_S`, `RequestClock`, `aggregate`), `src/osf/api/inference.py`, `ui/src/components/Slots.tsx` (`speedSub`), `ui/src/components/InferenceCharts.tsx`. Related: #344, decode speed that counts a turn's tool time. *Seen by the owner during braid#350's run C (vLLM, Qwen3.8-27B), 2026-10-31.*
Author
Owner

Done in batch B, merged as #355 (91f4b85f), and live on production: Braid reports commit c00c62f, which includes it. It was checked with the repo's tests and sims before the merge. A live check on vLLM is still to come: trogdor is off the network. Closed at the owner's request.

Done in batch B, merged as #355 (`91f4b85f`), and live on production: Braid reports commit `c00c62f`, which includes it. It was checked with the repo's tests and sims before the merge. A live check on vLLM is still to come: trogdor is off the network. Closed at the owner's request.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#351
No description provided.