Braid's decode speed counts the turn's tool time: a turn that runs tests or a subagent reads as a slow decode #344

Closed
opened 2026-10-30 13:27:34 -04:00 by cmoriarty · 1 comment
Owner

What happens
Braid times a request from when it was sent to opencode's step-finish, but step-finish arrives only after the turn's tools have run. So test runs and subagents count as decoding. On vLLM runs 139, 147, 152 and 156, 516 min of tool time was counted that way: the decode Braid recorded is 32.6 tok/s, where the model generated at 50.6. On run 160 (fastllm), the turn that started a 5-minute browser subagent reads as 1,485 tokens in 326 s (4.6 tok/s). The same turns also look like concurrent streams: 1,189 of those 1,875 vLLM requests seem to overlap another, but only 130 do. The Inference page's decode tile and /metrics gauges read low whenever a step runs tests or a subagent.

Fix
End a request when its generation ends, not at step-finish: at its last streamed token, or when its last tool call starts running (osf/throughput.py, RequestClock; the firehose already sees both).

Done when
A turn that runs a 60-second command records its decode time without those 60 s, and a test covers a turn that calls a subagent.

**What happens** Braid times a request from when it was sent to opencode's `step-finish`, but `step-finish` arrives only after the turn's tools have run. So test runs and subagents count as decoding. On vLLM runs 139, 147, 152 and 156, 516 min of tool time was counted that way: the decode Braid recorded is 32.6 tok/s, where the model generated at 50.6. On run 160 (fastllm), the turn that started a 5-minute browser subagent reads as 1,485 tokens in 326 s (4.6 tok/s). The same turns also look like concurrent streams: 1,189 of those 1,875 vLLM requests seem to overlap another, but only 130 do. The Inference page's decode tile and `/metrics` gauges read low whenever a step runs tests or a subagent. **Fix** End a request when its generation ends, not at `step-finish`: at its last streamed token, or when its last tool call starts running (`osf/throughput.py`, `RequestClock`; the firehose already sees both). **Done when** A turn that runs a 60-second command records its decode time without those 60 s, and a test covers a turn that calls a subagent.
Author
Owner

Done in batch B, merged as #355 (91f4b85f), and live on production: Braid reports commit c00c62f, which includes it. It was checked with the repo's tests and sims before the merge. A live check on vLLM is still to come: trogdor is off the network. Closed at the owner's request.

Done in batch B, merged as #355 (`91f4b85f`), and live on production: Braid reports commit `c00c62f`, which includes it. It was checked with the repo's tests and sims before the merge. A live check on vLLM is still to come: trogdor is off the network. Closed at the owner's request.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#344
No description provided.