Inference page: one view per backend, from what it reports, with readable charts #289

Closed
opened 2026-10-28 19:37:17 -04:00 by cmoriarty · 1 comment
Owner

What happens

The Inference page is built for vLLM. On llama.cpp (run 136): "All requests: not answering" in red, KV cache "—", Reused work "—", though the server is fine and reports more than the page shows. The history charts are dots with no axis values, so a dot can't be read. #199 noted the rows; this is the redesign.

Fix

One page per backend kind, with only what that backend reports, in charts with labelled axes and units, and exact values on hover.

  • llama.cpp (/metrics and /slots, both live on trogdor): reused prompt (prompt_tokens_cached_total / prompt_tokens_total: 98% now), draft acceptance (spec_decode_num_accepted_tokens_total / ..._num_draft_tokens_total), context fill per slot (n_prompt_tokens of n_ctx), prompt and generation tokens/s, requests processing and deferred.
  • vLLM (/metrics): running and waiting requests, KV cache use, prefix cache hit rate, time to first token, time per output token, preemptions.
  • tabbyAPI (ExLlamaV3) and fastllm: read their docs and endpoints first and list what each reports; show nothing they don't.
  • Anthropic and other hosted backends: tokens in, out and cached per run and per step, and the cost they come to at the model's prices; no slots.
  • Free: as #199 says, quota by tier.

Done when

On each backend the page shows no row it can't feed, every chart has axis values, and llama.cpp shows its reuse and draft acceptance.

From #279, run 136. Supersedes #199's point 1.

**What happens** The Inference page is built for vLLM. On llama.cpp (run 136): "All requests: not answering" in red, KV cache "—", Reused work "—", though the server is fine and reports more than the page shows. The history charts are dots with no axis values, so a dot can't be read. #199 noted the rows; this is the redesign. **Fix** One page per backend kind, with only what that backend reports, in charts with labelled axes and units, and exact values on hover. - **llama.cpp** (`/metrics` and `/slots`, both live on trogdor): reused prompt (`prompt_tokens_cached_total` / `prompt_tokens_total`: 98% now), draft acceptance (`spec_decode_num_accepted_tokens_total` / `..._num_draft_tokens_total`), context fill per slot (`n_prompt_tokens` of `n_ctx`), prompt and generation tokens/s, requests processing and deferred. - **vLLM** (`/metrics`): running and waiting requests, KV cache use, prefix cache hit rate, time to first token, time per output token, preemptions. - **tabbyAPI (ExLlamaV3)** and **fastllm**: read their docs and endpoints first and list what each reports; show nothing they don't. - **Anthropic and other hosted backends**: tokens in, out and cached per run and per step, and the cost they come to at the model's prices; no slots. - **Free**: as #199 says, quota by tier. **Done when** On each backend the page shows no row it can't feed, every chart has axis values, and llama.cpp shows its reuse and draft acceptance. _From #279, run 136. Supersedes #199's point 1._
Author
Owner

Shipped in PR #297 (1d98b25). The page draws only what its backend reports. llama.cpp, from /metrics and /slots: requests, each slot's context (slot 0 · 1.3k of 262k), prompt reuse and draft acceptance. tabbyAPI: Braid's rows only. A hosted API: Braid's tokens in, cached and out, with their cost at the profile's prices (Settings, each profile its own); no slots. Every chart has a value axis in its unit.

Verified on production after the deploy, on llama.cpp: the page draws its seven rows with labelled axes (171 tok/s), Context in use 3% · slot 0 · 7.3k of 262k, reused work 97% all time, draft tokens 94% recently · 61% all time. The hosted view (no slots; tokens and cost at 3 / 15 / 0.3) was seen on a local osfd of the same build, so production's profile stayed on llama.cpp during run 137.

Shipped in PR #297 (1d98b25). The page draws only what its backend reports. llama.cpp, from `/metrics` and `/slots`: requests, each slot's context (`slot 0 · 1.3k of 262k`), prompt reuse and draft acceptance. tabbyAPI: Braid's rows only. A hosted API: Braid's tokens in, cached and out, with their cost at the profile's prices (Settings, each profile its own); no slots. Every chart has a value axis in its unit. Verified on production after the deploy, on llama.cpp: the page draws its seven rows with labelled axes (`171 tok/s`), `Context in use 3% · slot 0 · 7.3k of 262k`, reused work `97% all time`, draft tokens `94% recently · 61% all time`. The hosted view (no slots; tokens and cost at 3 / 15 / 0.3) was seen on a local osfd of the same build, so production's profile stayed on llama.cpp during run 137.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#289
No description provided.