cmoriarty/braid
1
0
Fork 0
Code Issues 15 Pull requests Projects Releases Packages 1 Wiki Activity Actions

Most of the GPU's batch capacity goes unused: run independent steps side by side #128

New issue
Open
opened 2026-10-02 15:51:21 -04:00 by cmoriarty · 0 comments
cmoriarty commented 2026-10-02 15:51:21 -04:00
Owner
Copy link

Split off from #121 (finding 12 of the scratch-run analysis, docs/design/measurements/2026-10-02-scratch-run-efficiency.md).

On trogdor one stream decodes about 48 tok/s. Seven streams decode about 33 tok/s each, roughly 230 tok/s together. Yet 75% of the model turns in the scratch runs had no other stream running: after the spec fan-out and the four review lenses, the pipeline is one step after another, so the run waits on a single stream for most of its wall clock.

Ideas:

  • Steps that could overlap: the code review with the browser check, the unit tests with reconcile, and backend and frontend apply groups when their files are disjoint.

Related: #125 (verification steps that redo each other).

Split off from #121 (finding 12 of the scratch-run analysis, `docs/design/measurements/2026-10-02-scratch-run-efficiency.md`). On trogdor one stream decodes about 48 tok/s. Seven streams decode about 33 tok/s each, roughly 230 tok/s together. Yet 75% of the model turns in the scratch runs had no other stream running: after the spec fan-out and the four review lenses, the pipeline is one step after another, so the run waits on a single stream for most of its wall clock. Ideas: - Steps that could overlap: the code review with the browser check, the unit tests with reconcile, and backend and frontend apply groups when their files are disjoint. Related: #125 (verification steps that redo each other).
cmoriarty changed title from Most of the GPU's batch capacity goes unused: run independent steps side by side, and measure speculative decoding to Most of the GPU's batch capacity goes unused: run independent steps side by side 2026-10-04 21:55:45 -04:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
archive/prefect-kestra-experiments
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cmoriarty/braid#128
No description provided.