Planned projects run one item at a time: plan for the backend's concurrency #306

Closed
opened 2026-10-29 15:32:32 -04:00 by cmoriarty · 4 comments
Owner

What happens
A planned project starts its next item only after the last one merged and replanned. On vLLM, which serves six sessions at little cost, the Breakout project used one: its first item took 203 min (apply 66 min as one step, judge 29 min), and qwen3.8-27b's long thinking at ~60 tok/s is the bottleneck. The parallel review lenses bought little (#279: 1,240 s for four against 1,120 s for one combined reviewer).

Fix (to design)

  • With a backend kept per project, Braid knows its concurrency and speed, and the plan marks Now items that can run side by side, each on its own branch.
  • The replan copes with runs finishing in any order; merges go through the sync-base step.
  • Within a run, prefer side-by-side apply groups over review lenses on such a backend.
  • Measure first whether trogdor's total tokens per second scale with sessions.

Done when
The Breakout brief as a parallel project finishes in less total time than run 139's sequential baseline, with judge results no worse.

The operator's direction, 2026-10-29. Baseline: run 139.

**What happens** A planned project starts its next item only after the last one merged and replanned. On vLLM, which serves six sessions at little cost, the Breakout project used one: its first item took 203 min (apply 66 min as one step, judge 29 min), and qwen3.8-27b's long thinking at ~60 tok/s is the bottleneck. The parallel review lenses bought little (#279: 1,240 s for four against 1,120 s for one combined reviewer). **Fix (to design)** - With a backend kept per project, Braid knows its concurrency and speed, and the plan marks Now items that can run side by side, each on its own branch. - The replan copes with runs finishing in any order; merges go through the sync-base step. - Within a run, prefer side-by-side apply groups over review lenses on such a backend. - Measure first whether trogdor's total tokens per second scale with sessions. **Done when** The Breakout brief as a parallel project finishes in less total time than run 139's sequential baseline, with judge results no worse. _The operator's direction, 2026-10-29. Baseline: run 139._
Author
Owner

Measured 2026-10-29: trogdor (vLLM, qwen3.8-27b) scales with sessions up to its cap of 6. N streams at once, 512 tokens each, short prompts:

sessions total tok/s per session
1 65 67
2 122 63
4 219 56
6 264 45
8 204 the 7th and 8th waited ~12 s for a slot

So four sessions give 3.4x the work at 0.84x the speed each.

Prefill is the limit. Four sessions that each start with ~5,000 words of fresh context got their first token after up to 47 s, against 12 s alone. Agent turns mostly reuse cached context, so steady work should scale like the table. Fresh sessions and compactions contend.

Run 147, the Breakout project's item: its five agent.write-spec-tests steps ran one after another, 1,905 s in all, with 5 of 6 slots free. The longest took 832 s. Side by side, each at ~0.8x speed, that is ~17 min instead of 32.

**Measured 2026-10-29: trogdor (vLLM, qwen3.8-27b) scales with sessions up to its cap of 6.** N streams at once, 512 tokens each, short prompts: | sessions | total tok/s | per session | | --- | --- | --- | | 1 | 65 | 67 | | 2 | 122 | 63 | | 4 | 219 | 56 | | 6 | 264 | 45 | | 8 | 204 | the 7th and 8th waited ~12 s for a slot | So four sessions give 3.4x the work at 0.84x the speed each. **Prefill is the limit.** Four sessions that each start with ~5,000 words of fresh context got their first token after up to 47 s, against 12 s alone. Agent turns mostly reuse cached context, so steady work should scale like the table. Fresh sessions and compactions contend. **Run 147, the Breakout project's item:** its five `agent.write-spec-tests` steps ran one after another, 1,905 s in all, with 5 of 6 slots free. The longest took 832 s. Side by side, each at ~0.8x speed, that is ~17 min instead of 32.
Author
Owner

Run 192 (braid#484) tried this: side by side saved about 35 to 40 of 403 minutes, because only two groups overlapped, each at about 0.8 times its speed alone. More gain needs the planner to make groups independent. Paused by the owner until agents run in isolated environments (braid#201, braid#203).

Run 192 (braid#484) tried this: side by side saved about 35 to 40 of 403 minutes, because only two groups overlapped, each at about 0.8 times its speed alone. More gain needs the planner to make groups independent. Paused by the owner until agents run in isolated environments (braid#201, braid#203).
Author
Owner

A fact that has changed since this was written: the vLLM server keeps two large contexts in its cache, not six (496,770 tokens on 2026-11-11). Braid now starts at most two agent steps at once on a local server (braid#498), and a step's helpers take places too (braid#502, braid#509). Three side by side made no more tokens a minute than one (braid#497, finding 17); two made about one and a half times (braid#505). Any plan for several items at once starts from two. The gaps left in the cap are braid#535.

A fact that has changed since this was written: the vLLM server keeps two large contexts in its cache, not six (496,770 tokens on 2026-11-11). Braid now starts at most two agent steps at once on a local server (braid#498), and a step's helpers take places too (braid#502, braid#509). Three side by side made no more tokens a minute than one (braid#497, finding 17); two made about one and a half times (braid#505). Any plan for several items at once starts from two. The gaps left in the cap are braid#535.
Author
Owner

Closing: the first slice shipped, and the rest is tracked elsewhere.

  • Scenario-test steps run side by side (f63d2ca2, change side-by-side-spec-tests, archived now).
  • Run 192 measured side-by-side groups: about 35 to 40 of 403 minutes saved (braid#484).
  • The premise changed: the server's cache keeps two large contexts, not six. Braid now caps agent steps at two (braid#498), and the cap's gaps are braid#535.
  • Several items or issues at once, on separate branches, is braid#361. You paused this on 11-11 until agents are isolated (braid#203, done).
Closing: the first slice shipped, and the rest is tracked elsewhere. - Scenario-test steps run side by side (`f63d2ca2`, change `side-by-side-spec-tests`, archived now). - Run 192 measured side-by-side groups: about 35 to 40 of 403 minutes saved (braid#484). - The premise changed: the server's cache keeps two large contexts, not six. Braid now caps agent steps at two (braid#498), and the cap's gaps are braid#535. - Several items or issues at once, on separate branches, is braid#361. You paused this on 11-11 until agents are isolated (braid#203, done).
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#306
No description provided.