The model server drops a waiting step's context, and sometimes recomputes one with its cache far from full #534

Open
opened 2026-11-13 15:07:32 -05:00 by cmoriarty · 0 comments
Owner

What happens

Two costs on the vLLM server, both measured:

  • A step that waits minutes for a helper comes back to a context the server has dropped. Run 195's group 4 waited 172 s and 180 s (165k and 171k tokens) right after two helpers reported: six minutes of its 67 (braid#505, finding 2).
  • A context is sometimes computed again with the cache far from full: 3 times in 149 turns with one session alone in run 193, and in the first 20 minutes of three side by side (braid#497, finding 17). Not explained. One guess, not checked: the agent server pruning old tool output changes the prompt's start.

The cache holds 496,770 tokens (2026-11-11), so two large contexts fit and three do not.

Fix (to decide)

  • Keeping a waiting step warm: Braid would re-send the step's last request for one token. The agent server talks to the model server directly, so Braid does not have that request; it needs a pass-through in that path. Recommended to leave unless the pass-through earns its place another way (per-request accounting, a real cap on requests). Not tested: whether a one-token request protects the whole context.
  • Explain the recomputes: compare the prompt of a turn that was recomputed with the turn before it.
  • More cache on the server is a server change, not Braid's.

Done when

The recomputes are explained, and the owner has said yes or no to the pass-through.

**What happens** Two costs on the vLLM server, both measured: - A step that waits minutes for a helper comes back to a context the server has dropped. Run 195's group 4 waited 172 s and 180 s (165k and 171k tokens) right after two helpers reported: six minutes of its 67 (braid#505, finding 2). - A context is sometimes computed again with the cache far from full: 3 times in 149 turns with one session alone in run 193, and in the first 20 minutes of three side by side (braid#497, finding 17). Not explained. One guess, not checked: the agent server pruning old tool output changes the prompt's start. The cache holds 496,770 tokens (2026-11-11), so two large contexts fit and three do not. **Fix (to decide)** - Keeping a waiting step warm: Braid would re-send the step's last request for one token. The agent server talks to the model server directly, so Braid does not have that request; it needs a pass-through in that path. Recommended to leave unless the pass-through earns its place another way (per-request accounting, a real cap on requests). Not tested: whether a one-token request protects the whole context. - Explain the recomputes: compare the prompt of a turn that was recomputed with the turn before it. - More cache on the server is a server change, not Braid's. **Done when** The recomputes are explained, and the owner has said yes or no to the pass-through.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#534
No description provided.