A long prefill on llama.cpp is redone as a stall #295
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happens
Run 136's fix round forked a 77k-token session onto llama.cpp's one slot, which the judge had just used, so nothing was cached. llama.cpp prefills at about 270 tokens/s on this model, so nothing streamed for 8 minutes, and the watchdog redid the step: "stalled: silent for 480s while busy". The second attempt found the prompt cached and went straight through. Judge round 2 waited 385 s for its first token the same way and only just made it.
The watchdog's exemption for a busy server reads vLLM's queue only. On llama.cpp it sees nothing.
Fix
Scrape llama.cpp's
/slots(#289 adds the scrape). A slot that is processing a prompt, withn_prompt_tokens_processedstill rising, is the server working, not the step stalling. Follow the count, not a longer timeout.Done when
A step prefilling a long prompt on llama.cpp is not redone while the count rises, and a step that is truly stuck still is.
Shipped in PR #297 (
1d98b25). The sampler follows the prompt tokens llama.cpp's slots have read (n_prompt_tokens_processed). While that count rose in the last 90 s, the watchdog counts the server as working, for up to three stall windows as for a queue. Once the count stops, the step's silence is judged as before.Evidence:
/api/slotsshowedprompt_readingrising in 2,048-token batches while the slot was processing. That held for each ofagent.code-review's turns, and whenagent.judgestarted on a fresh 11k-token prompt (0 → 2,048 → 3,758, …).tests/api/test_inference_backends.py).