Inference page: prefill and decode speed go blank while a long request is decoding #351
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happens
RECENT_S = 300insrc/osf/throughput.py; the "no recent requests" text inui/src/components/Slots.tsx). Braid times a request only at itsstep-finish(RequestClock).What we want
RequestClockalready sees the firstmessage.part.deltaand each one after it. The count can come from the server's own counter where one exists (vLLMvllm:generation_tokens_total, llama.cpp/slotsn_decoded), and otherwise from the streamed text.Done when
/metricskeep their finished-request meaning, or an in-flight gauge is added beside them.Where to look:
src/osf/throughput.py(RECENT_S,RequestClock,aggregate),src/osf/api/inference.py,ui/src/components/Slots.tsx(speedSub),ui/src/components/InferenceCharts.tsx. Related: #344, decode speed that counts a turn's tool time.Seen by the owner during braid#350's run C (vLLM, Qwen3.8-27B), 2026-10-31.
Done in batch B, merged as #355 (
91f4b85f), and live on production: Braid reports commitc00c62f, which includes it. It was checked with the repo's tests and sims before the merge. A live check on vLLM is still to come: trogdor is off the network. Closed at the owner's request.