Inference page: one view per backend, from what it reports, with readable charts #289
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happens
The Inference page is built for vLLM. On llama.cpp (run 136): "All requests: not answering" in red, KV cache "—", Reused work "—", though the server is fine and reports more than the page shows. The history charts are dots with no axis values, so a dot can't be read. #199 noted the rows; this is the redesign.
Fix
One page per backend kind, with only what that backend reports, in charts with labelled axes and units, and exact values on hover.
/metricsand/slots, both live on trogdor): reused prompt (prompt_tokens_cached_total/prompt_tokens_total: 98% now), draft acceptance (spec_decode_num_accepted_tokens_total/..._num_draft_tokens_total), context fill per slot (n_prompt_tokensofn_ctx), prompt and generation tokens/s, requests processing and deferred./metrics): running and waiting requests, KV cache use, prefix cache hit rate, time to first token, time per output token, preemptions.Done when
On each backend the page shows no row it can't feed, every chart has axis values, and llama.cpp shows its reuse and draft acceptance.
From #279, run 136. Supersedes #199's point 1.
Shipped in PR #297 (
1d98b25). The page draws only what its backend reports. llama.cpp, from/metricsand/slots: requests, each slot's context (slot 0 · 1.3k of 262k), prompt reuse and draft acceptance. tabbyAPI: Braid's rows only. A hosted API: Braid's tokens in, cached and out, with their cost at the profile's prices (Settings, each profile its own); no slots. Every chart has a value axis in its unit.Verified on production after the deploy, on llama.cpp: the page draws its seven rows with labelled axes (
171 tok/s),Context in use 3% · slot 0 · 7.3k of 262k, reused work97% all time, draft tokens94% recently · 61% all time. The hosted view (no slots; tokens and cost at 3 / 15 / 0.3) was seen on a local osfd of the same build, so production's profile stayed on llama.cpp during run 137.