Compare vLLM qwen3.8-27b and llama.cpp qwen3.8-flash-next on one project #279
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What
One small project, built twice on
thoroughwith the exact same brief: once on vLLM (qwen3.8-27b), once on llama.cpp (qwen3.8-flash-next, Parallel sessions 1, so the combined reviewer runs). Compare time, turns, tokens, fix rounds and the quality of what each built.Runs
cmoriarty/mortgage-vllmcmoriarty/mortgage-llamacppBrief (both runs)
Done when
Both runs have finished, and the comparison and the findings from their transcripts are comments here.
Findings so far (updated as the runs go)
Bugs
#280 Run 134's judge asked a one-option "status checkpoint" question while its browser subagent ran; it held the auto-approved run for a person.
#278 again, worse: B7 wants
mortgage.test.jsto import a module namedmortgage, so run 134's build renamed its source fileapp.js→mortgage.js.#280 also loops: the judge asked five such "still waiting" questions; each one-click answer woke it with nothing to do, so it asked again.
#281 A typed answer to an agent's question can't contain a space: the console trims the answer on every keystroke.
#283 Run 135's apply had no way to look at its page ("no playwright, no chromium"): with Parallel sessions 1 there's no
browser, and the build agent is denied Playwright. Run 134's apply could have usedbrowserbut checked withcurl.Working as intended
autochose one step (400k context ≥ 200k), so apply kept the session.Proposals
Watching
agent.write-spec-teststook 8 min for one test; the 3 visual scenarios it left go to the judge.Run 136 (llama.cpp)
Bugs
write-spec-testsover a dynamicimport(). In the fix round it shaped the code again: the agent turnedapp.jsinto a UMD-style script to give the test a relative import..gitignore, which Braid adds itself right after.script.scenarios.Proposals
Working as intended
Run 136 (llama.cpp, qwen3.8-flash-next) against run 134 (vLLM, qwen3.8-27b), same brief
Run 136 was stopped during judge round 2, after the evidence for #282 and #266 was in. The agent time below runs up to that point.
Quality
.cards { display: grid }overrodehidden. vLLM's kept stale figures too, with no mark.So far: the same correctness, vLLM about a third faster, and vLLM ahead on finish (one regression fewer). Not part of the pipeline: run 136 lost time to #294 (a failed step, repaired by hand) and #295.
Closing: the comparison and the findings from both runs' transcripts are the comments above. Run 134 (vLLM) finished. Run 136 (llama.cpp) was stopped in judge round 2 once its evidence was in, and run 137 (llama.cpp) then finished a project end to end. Every finding is filed, and all but #280, which waits on a vLLM run with the judge's browser subagent, is closed.