A long prefill on llama.cpp is redone as a stall #295

Closed
opened 2026-10-28 21:13:31 -04:00 by cmoriarty · 1 comment
Owner

What happens
Run 136's fix round forked a 77k-token session onto llama.cpp's one slot, which the judge had just used, so nothing was cached. llama.cpp prefills at about 270 tokens/s on this model, so nothing streamed for 8 minutes, and the watchdog redid the step: "stalled: silent for 480s while busy". The second attempt found the prompt cached and went straight through. Judge round 2 waited 385 s for its first token the same way and only just made it.

The watchdog's exemption for a busy server reads vLLM's queue only. On llama.cpp it sees nothing.

Fix
Scrape llama.cpp's /slots (#289 adds the scrape). A slot that is processing a prompt, with n_prompt_tokens_processed still rising, is the server working, not the step stalling. Follow the count, not a longer timeout.

Done when
A step prefilling a long prompt on llama.cpp is not redone while the count rises, and a step that is truly stuck still is.

**What happens** Run 136's fix round forked a 77k-token session onto llama.cpp's one slot, which the judge had just used, so nothing was cached. llama.cpp prefills at about 270 tokens/s on this model, so nothing streamed for 8 minutes, and the watchdog redid the step: "stalled: silent for 480s while busy". The second attempt found the prompt cached and went straight through. Judge round 2 waited 385 s for its first token the same way and only just made it. The watchdog's exemption for a busy server reads vLLM's queue only. On llama.cpp it sees nothing. **Fix** Scrape llama.cpp's `/slots` (#289 adds the scrape). A slot that is processing a prompt, with `n_prompt_tokens_processed` still rising, is the server working, not the step stalling. Follow the count, not a longer timeout. **Done when** A step prefilling a long prompt on llama.cpp is not redone while the count rises, and a step that is truly stuck still is.
Author
Owner

Shipped in PR #297 (1d98b25). The sampler follows the prompt tokens llama.cpp's slots have read (n_prompt_tokens_processed). While that count rose in the last 90 s, the watchdog counts the server as working, for up to three stall windows as for a queue. Once the count stops, the step's silence is judged as before.

Evidence:

  • The signal is live on production. During run 137, production's /api/slots showed prompt_reading rising in 2,048-token batches while the slot was processing. That held for each of agent.code-review's turns, and when agent.judge started on a fresh 11k-token prompt (0 → 2,048 → 3,758, …).
  • The rule is tested. A rising count holds the watchdog off, and a still one does not, past 90 s (tests/api/test_inference_backends.py).
  • Not seen end to end. Run 137 never had a prefill long enough to reach 480 s of silence: its prompts stayed under 32k. Reopen this if a step is redone as stalled while llama.cpp is still reading its prompt.
Shipped in PR #297 (1d98b25). The sampler follows the prompt tokens llama.cpp's slots have read (`n_prompt_tokens_processed`). While that count rose in the last 90 s, the watchdog counts the server as working, for up to three stall windows as for a queue. Once the count stops, the step's silence is judged as before. Evidence: - **The signal is live on production.** During run 137, production's `/api/slots` showed `prompt_reading` rising in 2,048-token batches while the slot was processing. That held for each of `agent.code-review`'s turns, and when `agent.judge` started on a fresh 11k-token prompt (0 → 2,048 → 3,758, …). - **The rule is tested.** A rising count holds the watchdog off, and a still one does not, past 90 s (`tests/api/test_inference_backends.py`). - **Not seen end to end.** Run 137 never had a prefill long enough to reach 480 s of silence: its prompts stayed under 32k. Reopen this if a step is redone as stalled while llama.cpp is still reading its prompt.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#295
No description provided.