fastllm trial: run 160's findings, and its speed against vLLM #342
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Run 160 on
cmoriarty/fastllm-helloworld(thorough, "Press Me" Hello World brief), profile "fastllm qwen38-27b": trogdor's stack 129, the same Qwen3.8-27B AWQ as vLLM, with fastllm's MTP 4. Started 2026-10-30 16:58Z. This issue keeps the findings and the speed against vLLM as the run goes.Outcome: run 160 succeeded at 17:42Z, 44.6 min after it was created, with no failed attempt, one refused tool call and no person. Its PR (fastllm-helloworld PR 1, 18 files, +684) merged itself, and the operator reviewed the app: excellent. GPU memory did not grow: 15.1 GB per card at the start, 13.8 GB during, 14.1 GB idle after.
Speed against vLLM
Braid's own timing of each request (
metrics.sqlite), with the tool time Braid counts as decoding taken out (#344). vLLM: thorough runs 139, 147, 152 and 156 (1,875 requests). fastllm: all of run 160 (140 requests). Matched by prompt size; every request here ran with no other request at once (#341).The whole run's model time, tools excluded: fastllm 42.8 min, 20.0 waiting for first tokens and 22.8 generating. The same requests at vLLM's mean first-token wait and decode: 46.4 min, 4.6 waiting and 41.8 generating. So fastllm was about 8% faster overall, though it generates 1.7–2.2x faster. Its first-token waits ate almost all of the gain. The model was busy for 43 of the run's 44.6 minutes, so model time is the run's time.
The Visual check reads screenshots, and with an image in context fastllm recomputes the whole prompt on every turn (554k tokens over 47 turns). Its 11.9 min of model time is more than the 8.5 min vLLM would need (estimated: 1.2 waiting + 7.4 generating). Summarize waited 152 s for a cold fork.
If fastllm cached as vLLM does (the three gaps below), its wait would be about vLLM's 4.6 min and the run's model time about 27 min, 1.7x faster than vLLM. Prefix-cache hits: fastllm 69% of the run's 3.69M prompt tokens (1.13M recomputed); vLLM 88–97% by its counters.
Findings
auto.info.agent, which the console doesn't read.--enable-prompt-tokens-details), so Braid's vLLM prefill (16–20k tok/s) counts prefix-cache hits as computed; fastllm's counts only what it computed. Compare TTFT instead, or turn the flag on.reasoningis 0 and the thinking is counted inoutput.--prefix_cache_snapshot_max_records 32) didn't survive the review's 12 turns in between. Summarize, forked from apply 14 min after it ended, prefilled all 85k inherited tokens cold: 152 s to its first token (561 tok/s). Propose and apply, each forked a second after the step before, hit. Every new session also re-prefills the shared system prompt (6–8k tokens, 6–7 s).../screenshots/…, so it globbed the run folder and was refused. The refusal said "This run's own folder is…/runs/160; check the path you asked for against it" (the very path it asked for) and blamed acdit never ran. It recovered from the message's last line, which names the screenshots folder. Rare: 1 of 10 run-folder refusals in five days.evaluate./favicon.ico404 as a console error, so apply added an inline favicon and ran a second browser subagent to re-check (1 min 40 s). Every new web app meets this; the browser brief could say to ignore it.engine_downfor 1 s while the probe met the fastllm backend.