Most of the GPU's batch capacity goes unused: run independent steps side by side #128

Closed
opened 2026-10-02 15:51:21 -04:00 by cmoriarty · 2 comments
Owner

Split off from #121 (finding 12 of the scratch-run analysis, docs/design/measurements/2026-10-02-scratch-run-efficiency.md).

On trogdor one stream decodes about 48 tok/s. Seven streams decode about 33 tok/s each, roughly 230 tok/s together. Yet 75% of the model turns in the scratch runs had no other stream running: after the spec fan-out and the four review lenses, the pipeline is one step after another, so the run waits on a single stream for most of its wall clock.

Ideas:

  • Steps that could overlap: the code review with the browser check, the unit tests with reconcile, and backend and frontend apply groups when their files are disjoint.

Related: #125 (verification steps that redo each other).

Split off from #121 (finding 12 of the scratch-run analysis, `docs/design/measurements/2026-10-02-scratch-run-efficiency.md`). On trogdor one stream decodes about 48 tok/s. Seven streams decode about 33 tok/s each, roughly 230 tok/s together. Yet 75% of the model turns in the scratch runs had no other stream running: after the spec fan-out and the four review lenses, the pipeline is one step after another, so the run waits on a single stream for most of its wall clock. Ideas: - Steps that could overlap: the code review with the browser check, the unit tests with reconcile, and backend and frontend apply groups when their files are disjoint. Related: #125 (verification steps that redo each other).
cmoriarty changed title from Most of the GPU's batch capacity goes unused: run independent steps side by side, and measure speculative decoding to Most of the GPU's batch capacity goes unused: run independent steps side by side 2026-10-04 21:55:45 -04:00
Author
Owner

Apply groups side by side, one of the ideas above, is now #172: worktree per group, a three-way merge back into the run, conflicts resolved by the group's own session. It follows #171, because two groups at once need a port each and only opencode 2 can give a session its own environment.

Apply groups side by side, one of the ideas above, is now #172: worktree per group, a three-way merge back into the run, conflicts resolved by the group's own session. It follows #171, because two groups at once need a port each and only opencode 2 can give a session its own environment.
Author
Owner

Re-scoped against today's pipelines and #170, and shipped as side-by-side-steps: a guard and documentation, with no new parallelism.

What runs side by side already. The scheduler admits every ready step at once. The built-ins already overlap wherever it is safe: the delta specs beside the design, and the four review lenses. After the review gate, every agent step either edits the one worktree or must see it unchanged, like the judge.

The issue's two ideas:

  • The code review now runs before the loop. Running it beside the judge would cost 0.2–6.7 min per run and change who fixes what.
  • agent.test.unit no longer exists.

Measured. On production's seven thorough runs, two or more agent steps ran at once for only 4–9% of agent time. Apply groups took 376 of run 62's 508 agent minutes.

What shipped. A named-pipelines requirement, held by unit tests over every built-in: no two implementation steps can run at once, and no committing or pushing step can run beside one.

The real speed-up is running independent apply groups side by side, each in a worktree of its own. That is #172, split off from #170. Its dependency half (Depends on: lines in tasks.md) shipped with #170. Closing this in favour of #172.

Re-scoped against today's pipelines and #170, and shipped as `side-by-side-steps`: a guard and documentation, with no new parallelism. **What runs side by side already.** The scheduler admits every ready step at once. The built-ins already overlap wherever it is safe: the delta specs beside the design, and the four review lenses. After the review gate, every agent step either edits the one worktree or must see it unchanged, like the judge. **The issue's two ideas:** - The code review now runs before the loop. Running it beside the judge would cost 0.2–6.7 min per run and change who fixes what. - `agent.test.unit` no longer exists. **Measured.** On production's seven thorough runs, two or more agent steps ran at once for only 4–9% of agent time. Apply groups took 376 of run 62's 508 agent minutes. **What shipped.** A `named-pipelines` requirement, held by unit tests over every built-in: no two implementation steps can run at once, and no committing or pushing step can run beside one. **The real speed-up** is running independent apply groups side by side, each in a worktree of its own. That is #172, split off from #170. Its dependency half (`Depends on:` lines in `tasks.md`) shipped with #170. Closing this in favour of #172.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#128
No description provided.