A resumed run waits for its old slot while another run holds it #336

Closed
opened 2026-10-30 10:46:00 -04:00 by cmoriarty · 1 comment
Owner

What happens
Run 158 (vLLM) was stopped so a deploy could go through. The Stocks rerun (run 159, Free) was then admitted into slot 0, the slot run 158 had recorded. Once resumed, run 158 logged slot 0 is held by run …; its server waits for the slot every 30 s, and its apply sat at ready with the hold engine_down. Since #313, runs on different backends go at once, so the holder can run for hours.

Fix
When a resumed run's recorded slot is held, reattach gives it a free slot, as admission would. osf.run.server_started records the new slot, so the run's preview port and compose project move with it.

Done when
A test resumes a run whose slot is held: its server starts in a free slot and the run row records it. On production, run 158 resumes and its apply runs alongside the Stocks run.

**What happens** Run 158 (vLLM) was stopped so a deploy could go through. The Stocks rerun (run 159, Free) was then admitted into slot 0, the slot run 158 had recorded. Once resumed, run 158 logged `slot 0 is held by run …; its server waits for the slot` every 30 s, and its apply sat at `ready` with the hold `engine_down`. Since #313, runs on different backends go at once, so the holder can run for hours. **Fix** When a resumed run's recorded slot is held, `reattach` gives it a free slot, as admission would. `osf.run.server_started` records the new slot, so the run's preview port and compose project move with it. **Done when** A test resumes a run whose slot is held: its server starts in a free slot and the run row records it. On production, run 158 resumes and its apply runs alongside the Stocks run.
Author
Owner

Shipped in #337 (bee3ab2). On production, run 158 was resumed while the Stocks run held its recorded slot 0. It started in slot 1 (opencode on :8801) without waiting, and its apply runs as attempt 3 beside the Stocks run. The test tests/engine/test_resume_slot.py fails without the fix.

Shipped in #337 (bee3ab2). On production, run 158 was resumed while the Stocks run held its recorded slot 0. It started in slot 1 (opencode on :8801) without waiting, and its apply runs as attempt 3 beside the Stocks run. The test `tests/engine/test_resume_slot.py` fails without the fix.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#336
No description provided.