fastllm trial: run 160's findings, and its speed against vLLM #342

Closed
opened 2026-10-30 13:09:48 -04:00 by cmoriarty · 0 comments
Owner

Run 160 on cmoriarty/fastllm-helloworld (thorough, "Press Me" Hello World brief), profile "fastllm qwen38-27b": trogdor's stack 129, the same Qwen3.8-27B AWQ as vLLM, with fastllm's MTP 4. Started 2026-10-30 16:58Z. This issue keeps the findings and the speed against vLLM as the run goes.

Outcome: run 160 succeeded at 17:42Z, 44.6 min after it was created, with no failed attempt, one refused tool call and no person. Its PR (fastllm-helloworld PR 1, 18 files, +684) merged itself, and the operator reviewed the app: excellent. GPU memory did not grow: 15.1 GB per card at the start, 13.8 GB during, 14.1 GB idle after.

Speed against vLLM

Braid's own timing of each request (metrics.sqlite), with the tool time Braid counts as decoding taken out (#344). vLLM: thorough runs 139, 147, 152 and 156 (1,875 requests). fastllm: all of run 160 (140 requests). Matched by prompt size; every request here ran with no other request at once (#341).

One request at a time vLLM fastllm
Decode, prompts under 16k 62.5 tok/s 106.8 tok/s 1.71x
Decode, 16k–32k 59.4 tok/s 107.5 tok/s 1.81x
Decode, 32k–64k 56.4 tok/s 105.7 tok/s 1.87x
Decode, 64k–128k 48.5 tok/s 104.7 tok/s 2.16x
Time to first token, median, 8k–16k 0.5 s 7.9 s screenshots
Time to first token, median, 16k–32k 1.3 s 3.2 s
Time to first token, median, 32k–64k 1.7 s 20.0 s
Prefill, tokens actually computed ~1,000–1,200 tok/s 1,200 falling to 330 as context grows

The whole run's model time, tools excluded: fastllm 42.8 min, 20.0 waiting for first tokens and 22.8 generating. The same requests at vLLM's mean first-token wait and decode: 46.4 min, 4.6 waiting and 41.8 generating. So fastllm was about 8% faster overall, though it generates 1.7–2.2x faster. Its first-token waits ate almost all of the gain. The model was busy for 43 of the run's 44.6 minutes, so model time is the run's time.

fastllm step wall waiting for first token generating decode cache hits
explore 0.7 min 0.2 0.5 110 tok/s 83%
propose 2.2 min 0.5 1.7 102 tok/s 92%
review (combined) 7.1 min 1.3 5.8 109 tok/s 85%
revise 1.8 min 0.9 0.9 120 tok/s 79%
apply 14.1 min 6.0 7.8 105 tok/s 82%
code review 1.8 min 0.5 1.3 107 tok/s 80%
Visual check 12.2 min 7.8 4.1 109 tok/s 15%
summarize 3.5 min 2.7 0.8 92 tok/s 49%

The Visual check reads screenshots, and with an image in context fastllm recomputes the whole prompt on every turn (554k tokens over 47 turns). Its 11.9 min of model time is more than the 8.5 min vLLM would need (estimated: 1.2 waiting + 7.4 generating). Summarize waited 152 s for a cold fork.

If fastllm cached as vLLM does (the three gaps below), its wait would be about vLLM's 4.6 min and the run's model time about 27 min, 1.7x faster than vLLM. Prefix-cache hits: fastllm 69% of the run's 3.69M prompt tokens (1.13M recomputed); vLLM 88–97% by its counters.

Findings

  • #341 The fastllm profile was saved before profiles kept run settings, so the run took Free's: one session at a time, 4 subagents, 179,520 per agent. Its proposal review was one combined reviewer, and fastllm's 6 slots go unused. Since fixed on the fastllm profile itself: 6 parallel sessions, 5 subagents, 200,000 per agent, 3 runs, apply auto.
  • #344 Braid's decode speed counts the turn's tool time: test runs and subagents read as decoding. On the four vLLM runs, 516 min of tool time made Braid's decode read 32.6 tok/s where the model generated at 50.6; the turn that started run 160's browser subagent reads 4.6 tok/s.
  • #346 The Visual check can't see an animation shorter than a model turn. Its browser subagent clicked, waited and took the screenshot in separate turns (about 10 s each on fastllm), so every "mid-animation" screenshot showed the settled text. It judged the brief's "exciting" animation from the app's state hook without seeing it.
  • #345 (console, not fastllm) The Subagents panel's tab reads "Subagent Agent": opencode 2 sends the agent's name as info.agent, which the console doesn't read.
  • Prefill can't be compared on the Inference page. vLLM reports no cached tokens (stack 126 lacks --enable-prompt-tokens-details), so Braid's vLLM prefill (16–20k tok/s) counts prefix-cache hits as computed; fastllm's counts only what it computed. Compare TTFT instead, or turn the flag on.
  • fastllm reports no reasoning tokens: every turn's reasoning is 0 and the thinking is counted in output.
  • The combined reviewer thinks the review through twice (run 160). The prompt says the code is pasted below, but a greenfield plan has none: its first turn spent 12k tokens judging the pasted files, then it re-read all four from disk "in case the paste normalized them" and judged again (80k characters of thinking) before writing. About 3 of the step's 7 minutes.
  • fastllm never caches its own output. Each turn re-prefills what the model wrote on the turn before, thinking included: revise's second turn had 4,351 uncached tokens after 4,132 out. Through revise, 56k such tokens cost about 1.5 min of first-token wait. A turn after a long thinking turn waited 39 s (23k tokens at 595 tok/s); apply's second turn waited 48 s to re-prefill the 21k tokens its first turn wrote (442 tok/s at 69k).
  • A screenshot in context turns fastllm's prefix cache off. In apply's browser subagent, every turn after the first screenshot was a full cold prefill: 8.5k, 8.7k, 9.0k, 10.2k, 13.8k and 16.0k tokens, 7–13 s each, though the 6.9k-token system prompt had hit just before. On vLLM, the screenshot-heavy Visual check steps wait a median of 0.4–2.0 s for the first token, no slower than other steps. The Visual check and browser checks pay this on every turn.
  • A forked step starts cold on fastllm. Revise inherited 34k tokens from propose and got no cache hit (42 s to its first token): 32 snapshot records (--prefix_cache_snapshot_max_records 32) didn't survive the review's 12 turns in between. Summarize, forked from apply 14 min after it ended, prefilled all 85k inherited tokens cold: 152 s to its first token (561 tok/s). Propose and apply, each forked a second after the step before, hit. Every new session also re-prefills the shared system prompt (6–8k tokens, 6–7 s).
  • fastllm's prefill is about half of vLLM's, and slows as the context grows: about 1,100–1,200 tok/s at 8–16k, 820 at 34k, 561 at 85k, and 331–442 tok/s for a turn's new tokens at 69–89k.
  • Small: the browser subagent isn't told where its screenshots go. Playwright reports ../screenshots/…, so it globbed the run folder and was refused. The refusal said "This run's own folder is …/runs/160; check the path you asked for against it" (the very path it asked for) and blamed a cd it never ran. It recovered from the message's last line, which names the screenshots folder. Rare: 1 of 10 run-folder refusals in five days.
  • Small: the browser subagent's mid-animation screenshot came 8 s after its click (cold prefills make each turn slow), so it caught the settled state; it worked around it by restarting and pausing the animation inside one evaluate.
  • Small: the Visual check ran two full browser passes, the page opened from disk and the page served over HTTP, because the README documents both (3.4 and about 2.3 min on fastllm).
  • Small: the browser check flags the browser's own /favicon.ico 404 as a console error, so apply added an inline favicon and ran a second browser subagent to re-check (1 min 40 s). Every new web app meets this; the browser brief could say to ignore it.
  • Small: the first agent step was held engine_down for 1 s while the probe met the fastllm backend.
Run 160 on `cmoriarty/fastllm-helloworld` (thorough, "Press Me" Hello World brief), profile "fastllm qwen38-27b": trogdor's stack 129, the same Qwen3.8-27B AWQ as vLLM, with fastllm's MTP 4. Started 2026-10-30 16:58Z. This issue keeps the findings and the speed against vLLM as the run goes. **Outcome:** run 160 succeeded at 17:42Z, 44.6 min after it was created, with no failed attempt, one refused tool call and no person. Its PR (fastllm-helloworld PR 1, 18 files, +684) merged itself, and the operator reviewed the app: excellent. GPU memory did not grow: 15.1 GB per card at the start, 13.8 GB during, 14.1 GB idle after. ## Speed against vLLM Braid's own timing of each request (`metrics.sqlite`), with the tool time Braid counts as decoding taken out (#344). vLLM: thorough runs 139, 147, 152 and 156 (1,875 requests). fastllm: all of run 160 (140 requests). Matched by prompt size; every request here ran with no other request at once (#341). | One request at a time | vLLM | fastllm | | |---|---|---|---| | Decode, prompts under 16k | 62.5 tok/s | 106.8 tok/s | 1.71x | | Decode, 16k–32k | 59.4 tok/s | 107.5 tok/s | 1.81x | | Decode, 32k–64k | 56.4 tok/s | 105.7 tok/s | 1.87x | | Decode, 64k–128k | 48.5 tok/s | 104.7 tok/s | 2.16x | | Time to first token, median, 8k–16k | 0.5 s | 7.9 s | screenshots | | Time to first token, median, 16k–32k | 1.3 s | 3.2 s | | | Time to first token, median, 32k–64k | 1.7 s | 20.0 s | | | Prefill, tokens actually computed | ~1,000–1,200 tok/s | 1,200 falling to 330 as context grows | | **The whole run's model time, tools excluded:** fastllm **42.8 min**, 20.0 waiting for first tokens and 22.8 generating. The same requests at vLLM's mean first-token wait and decode: **46.4 min**, 4.6 waiting and 41.8 generating. So fastllm was about 8% faster overall, though it generates 1.7–2.2x faster. Its first-token waits ate almost all of the gain. The model was busy for 43 of the run's 44.6 minutes, so model time is the run's time. | fastllm step | wall | waiting for first token | generating | decode | cache hits | |---|---|---|---|---|---| | explore | 0.7 min | 0.2 | 0.5 | 110 tok/s | 83% | | propose | 2.2 min | 0.5 | 1.7 | 102 tok/s | 92% | | review (combined) | 7.1 min | 1.3 | 5.8 | 109 tok/s | 85% | | revise | 1.8 min | 0.9 | 0.9 | 120 tok/s | 79% | | apply | 14.1 min | 6.0 | 7.8 | 105 tok/s | 82% | | code review | 1.8 min | 0.5 | 1.3 | 107 tok/s | 80% | | Visual check | 12.2 min | **7.8** | 4.1 | 109 tok/s | **15%** | | summarize | 3.5 min | **2.7** | 0.8 | 92 tok/s | 49% | The Visual check reads screenshots, and with an image in context fastllm recomputes the whole prompt on every turn (554k tokens over 47 turns). Its 11.9 min of model time is more than the 8.5 min vLLM would need (estimated: 1.2 waiting + 7.4 generating). Summarize waited 152 s for a cold fork. **If fastllm cached as vLLM does** (the three gaps below), its wait would be about vLLM's 4.6 min and the run's model time about 27 min, 1.7x faster than vLLM. Prefix-cache hits: fastllm 69% of the run's 3.69M prompt tokens (1.13M recomputed); vLLM 88–97% by its counters. ## Findings - **#341** The fastllm profile was saved before profiles kept run settings, so the run took Free's: one session at a time, 4 subagents, 179,520 per agent. Its proposal review was one combined reviewer, and fastllm's 6 slots go unused. Since fixed on the fastllm profile itself: 6 parallel sessions, 5 subagents, 200,000 per agent, 3 runs, apply `auto`. - **#344** Braid's decode speed counts the turn's tool time: test runs and subagents read as decoding. On the four vLLM runs, 516 min of tool time made Braid's decode read 32.6 tok/s where the model generated at 50.6; the turn that started run 160's browser subagent reads 4.6 tok/s. - **#346** The Visual check can't see an animation shorter than a model turn. Its browser subagent clicked, waited and took the screenshot in separate turns (about 10 s each on fastllm), so every "mid-animation" screenshot showed the settled text. It judged the brief's "exciting" animation from the app's state hook without seeing it. - **#345** (console, not fastllm) The Subagents panel's tab reads "Subagent Agent": opencode 2 sends the agent's name as `info.agent`, which the console doesn't read. - **Prefill can't be compared on the Inference page.** vLLM reports no cached tokens (stack 126 lacks `--enable-prompt-tokens-details`), so Braid's vLLM prefill (16–20k tok/s) counts prefix-cache hits as computed; fastllm's counts only what it computed. Compare TTFT instead, or turn the flag on. - **fastllm reports no reasoning tokens:** every turn's `reasoning` is 0 and the thinking is counted in `output`. - **The combined reviewer thinks the review through twice** (run 160). The prompt says the code is pasted below, but a greenfield plan has none: its first turn spent 12k tokens judging the pasted files, then it re-read all four from disk "in case the paste normalized them" and judged again (80k characters of thinking) before writing. About 3 of the step's 7 minutes. - **fastllm never caches its own output.** Each turn re-prefills what the model wrote on the turn before, thinking included: revise's second turn had 4,351 uncached tokens after 4,132 out. Through revise, 56k such tokens cost about 1.5 min of first-token wait. A turn after a long thinking turn waited 39 s (23k tokens at 595 tok/s); apply's second turn waited 48 s to re-prefill the 21k tokens its first turn wrote (442 tok/s at 69k). - **A screenshot in context turns fastllm's prefix cache off.** In apply's browser subagent, every turn after the first screenshot was a full cold prefill: 8.5k, 8.7k, 9.0k, 10.2k, 13.8k and 16.0k tokens, 7–13 s each, though the 6.9k-token system prompt had hit just before. On vLLM, the screenshot-heavy Visual check steps wait a median of 0.4–2.0 s for the first token, no slower than other steps. The Visual check and browser checks pay this on every turn. - **A forked step starts cold on fastllm.** Revise inherited 34k tokens from propose and got no cache hit (42 s to its first token): 32 snapshot records (`--prefix_cache_snapshot_max_records 32`) didn't survive the review's 12 turns in between. Summarize, forked from apply 14 min after it ended, prefilled all 85k inherited tokens cold: **152 s** to its first token (561 tok/s). Propose and apply, each forked a second after the step before, hit. Every new session also re-prefills the shared system prompt (6–8k tokens, 6–7 s). - **fastllm's prefill is about half of vLLM's, and slows as the context grows:** about 1,100–1,200 tok/s at 8–16k, 820 at 34k, 561 at 85k, and 331–442 tok/s for a turn's new tokens at 69–89k. - Small: the browser subagent isn't told where its screenshots go. Playwright reports `../screenshots/…`, so it globbed the run folder and was refused. The refusal said "This run's own folder is `…/runs/160`; check the path you asked for against it" (the very path it asked for) and blamed a `cd` it never ran. It recovered from the message's last line, which names the screenshots folder. Rare: 1 of 10 run-folder refusals in five days. - Small: the browser subagent's mid-animation screenshot came 8 s after its click (cold prefills make each turn slow), so it caught the settled state; it worked around it by restarting and pausing the animation inside one `evaluate`. - Small: the Visual check ran two full browser passes, the page opened from disk and the page served over HTTP, because the README documents both (3.4 and about 2.3 min on fastllm). - Small: the browser check flags the browser's own `/favicon.ico` 404 as a console error, so apply added an inline favicon and ran a second browser subagent to re-check (1 min 40 s). Every new web app meets this; the browser brief could say to ignore it. - Small: the first agent step was held `engine_down` for 1 s while the probe met the fastllm backend.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#342
No description provided.