The model server drops a waiting step's context, and sometimes recomputes one with its cache far from full #534
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happens
Two costs on the vLLM server, both measured:
The cache holds 496,770 tokens (2026-11-11), so two large contexts fit and three do not.
Fix (to decide)
Done when
The recomputes are explained, and the owner has said yes or no to the pass-through.