Compare vLLM qwen3.8-27b and llama.cpp qwen3.8-flash-next on one project #279

Closed
opened 2026-10-28 15:24:12 -04:00 by cmoriarty · 3 comments
Owner

What

One small project, built twice on thorough with the exact same brief: once on vLLM (qwen3.8-27b), once on llama.cpp (qwen3.8-flash-next, Parallel sessions 1, so the combined reviewer runs). Compare time, turns, tokens, fix rounds and the quality of what each built.

Runs

  • vLLM: run 134, cmoriarty/mortgage-vllm
  • llama.cpp: after swapping trogdor's stack, cmoriarty/mortgage-llamacpp

Brief (both runs)

A mortgage calculator that runs in the browser: one page in plain HTML, CSS and JavaScript, with no build step. The user enters the loan amount, the annual interest rate, the term in years and an optional extra monthly payment. The page shows the monthly payment, the total interest, the payoff date and a month-by-month amortization table. With an extra payment it also shows how many months earlier the loan is paid off and how much interest that saves. Out-of-range inputs get a clear message next to the field, and the page works on a phone.

Done when

Both runs have finished, and the comparison and the findings from their transcripts are comments here.

**What** One small project, built twice on `thorough` with the exact same brief: once on vLLM (`qwen3.8-27b`), once on llama.cpp (`qwen3.8-flash-next`, Parallel sessions 1, so the combined reviewer runs). Compare time, turns, tokens, fix rounds and the quality of what each built. **Runs** - vLLM: run 134, `cmoriarty/mortgage-vllm` - llama.cpp: after swapping trogdor's stack, `cmoriarty/mortgage-llamacpp` **Brief (both runs)** > A mortgage calculator that runs in the browser: one page in plain HTML, CSS and JavaScript, with no build step. The user enters the loan amount, the annual interest rate, the term in years and an optional extra monthly payment. The page shows the monthly payment, the total interest, the payoff date and a month-by-month amortization table. With an extra payment it also shows how many months earlier the loan is paid off and how much interest that saves. Out-of-range inputs get a clear message next to the field, and the page works on a phone. **Done when** Both runs have finished, and the comparison and the findings from their transcripts are comments here.
Author
Owner

Findings so far (updated as the runs go)

Bugs

  • #280 Run 134's judge asked a one-option "status checkpoint" question while its browser subagent ran; it held the auto-approved run for a person.

  • #278 again, worse: B7 wants mortgage.test.js to import a module named mortgage, so run 134's build renamed its source file app.js → mortgage.js.

  • #280 also loops: the judge asked five such "still waiting" questions; each one-click answer woke it with nothing to do, so it asked again.

  • #281 A typed answer to an agent's question can't contain a space: the console trims the answer on every keystroke.

  • #283 Run 135's apply had no way to look at its page ("no playwright, no chromium"): with Parallel sessions 1 there's no browser, and the build agent is denied Playwright. Run 134's apply could have used browser but checked with curl.

Working as intended

  • One build session from explore to the spec tests: propose, revise, apply and write-spec-tests each forked the step before (7 → 22 → 27 → 51 messages inherited).
  • Apply mode auto chose one step (400k context ≥ 200k), so apply kept the session.
  • Round 1 of the review caught a real error (a payoff date of June 15, 2035 where the arithmetic gives February 15, 2035); the revision fixed it and round 2 confirmed it.

Proposals

  • #282 Judge rounds start from scratch: round 2 took 903 s and 79 turns to re-check a fix that only changed how the script loads.

Watching

  • agent.write-spec-tests took 8 min for one test; the 3 visual scenarios it left go to the judge.

Run 136 (llama.cpp)

Bugs

  • #294 B7 failed write-spec-tests over a dynamic import(). In the fix round it shaped the code again: the agent turned app.js into a UMD-style script to give the test a relative import.
  • #286 The proposal planned a .gitignore, which Braid adds itself right after.
  • #295 Fix attempt 1 was redone as stalled at 480 s while llama.cpp was still prefilling a cold 77k-token session (482 s lost).
  • #290 A run's tab opens on an empty stage, not its running step.
  • #291 A cut-off step name can't be read: its hover changes every second.
  • #293 The spine splits APPLY in two around script.scenarios.
  • #285 A simulated run that failed in the deploy's self-check left no log of why.

Proposals

  • #287 Review chips: green next to seven things to fix, and "note" means nothing to a reader.
  • #288 Show a review's findings as a panel.
  • #289 An inference page per backend.
  • #292 Show a running step's elapsed time, not each action's.

Working as intended

  • #283 Apply and fix opened the page with their own browser tools. The judge used its own Playwright: no subagent and no questions, so nothing waited.
  • #282 and #266 Judge round 2 forked round 1 (35 messages), closed the browser first, and re-checked what had failed: round 1's defect was fixed, and it found a new one.
  • One combined reviewer ran all four lenses in one session (Parallel sessions 1).
**Findings so far** (updated as the runs go) **Bugs** - #280 Run 134's judge asked a one-option "status checkpoint" question while its browser subagent ran; it held the auto-approved run for a person. - #278 again, worse: B7 wants `mortgage.test.js` to import a module named `mortgage`, so run 134's build renamed its *source* file `app.js` → `mortgage.js`. - #280 also loops: the judge asked five such "still waiting" questions; each one-click answer woke it with nothing to do, so it asked again. - #281 A typed answer to an agent's question can't contain a space: the console trims the answer on every keystroke. - #283 Run 135's apply had no way to look at its page ("no playwright, no chromium"): with Parallel sessions 1 there's no `browser`, and the build agent is denied Playwright. Run 134's apply could have used `browser` but checked with `curl`. **Working as intended** - One build session from explore to the spec tests: propose, revise, apply and write-spec-tests each forked the step before (7 → 22 → 27 → 51 messages inherited). - Apply mode `auto` chose one step (400k context ≥ 200k), so apply kept the session. - Round 1 of the review caught a real error (a payoff date of June 15, 2035 where the arithmetic gives February 15, 2035); the revision fixed it and round 2 confirmed it. **Proposals** - #282 Judge rounds start from scratch: round 2 took 903 s and 79 turns to re-check a fix that only changed how the script loads. **Watching** - `agent.write-spec-tests` took 8 min for one test; the 3 visual scenarios it left go to the judge. **Run 136 (llama.cpp)** Bugs - #294 B7 failed `write-spec-tests` over a dynamic `import()`. In the fix round it shaped the code again: the agent turned `app.js` into a UMD-style script to give the test a relative import. - #286 The proposal planned a `.gitignore`, which Braid adds itself right after. - #295 Fix attempt 1 was redone as stalled at 480 s while llama.cpp was still prefilling a cold 77k-token session (482 s lost). - #290 A run's tab opens on an empty stage, not its running step. - #291 A cut-off step name can't be read: its hover changes every second. - #293 The spine splits APPLY in two around `script.scenarios`. - #285 A simulated run that failed in the deploy's self-check left no log of why. Proposals - #287 Review chips: green next to seven things to fix, and "note" means nothing to a reader. - #288 Show a review's findings as a panel. - #289 An inference page per backend. - #292 Show a running step's elapsed time, not each action's. Working as intended - #283 Apply and fix opened the page with their own browser tools. The judge used its own Playwright: no subagent and no questions, so nothing waited. - #282 and #266 Judge round 2 forked round 1 (35 messages), closed the browser first, and re-checked what had failed: round 1's defect was fixed, and it found a new one. - One combined reviewer ran all four lenses in one session (Parallel sessions 1).
Author
Owner

Run 136 (llama.cpp, qwen3.8-flash-next) against run 134 (vLLM, qwen3.8-27b), same brief

Run 136 was stopped during judge round 2, after the evidence for #282 and #266 was in. The agent time below runs up to that point.

Phase vLLM llama.cpp
explore + propose 338 s 592 s
review + revise 1,240 s (4 lenses side by side, 465 s longest) 1,120 s (one combined reviewer)
apply + spec tests 1,609 s 1,957 s
code review 413 s 928 s
judge r1 + fix + judge r2 2,163 s (317 s waiting on questions) 2,955 s (judge r2 unfinished, plus 482 s lost to #295)
Agent time 5,762 s 7,550 s (+31%)

Quality

  • Both builds compute the right figures to the cent. llama.cpp's: 400k at 6.5% over 30 years gives $2,528.27 a month and $510,179.81 interest; $200 extra pays it off 67 months earlier and saves $111,892.52.
  • Both failed judge round 1 the same way: opened from disk, the ES-module script is blocked.
  • llama.cpp's also showed stale results while an input was invalid: .cards { display: grid } overrode hidden. vLLM's kept stale figures too, with no mark.
  • Fix round: llama.cpp's fixed both, but the fresh page then showed red errors on fields nobody had touched yet. Judge round 2 caught that. vLLM's passed judge round 2.

So far: the same correctness, vLLM about a third faster, and vLLM ahead on finish (one regression fewer). Not part of the pipeline: run 136 lost time to #294 (a failed step, repaired by hand) and #295.

**Run 136 (llama.cpp, qwen3.8-flash-next) against run 134 (vLLM, qwen3.8-27b)**, same brief Run 136 was stopped during judge round 2, after the evidence for #282 and #266 was in. The agent time below runs up to that point. | Phase | vLLM | llama.cpp | |---|---|---| | explore + propose | 338 s | 592 s | | review + revise | 1,240 s (4 lenses side by side, 465 s longest) | 1,120 s (one combined reviewer) | | apply + spec tests | 1,609 s | 1,957 s | | code review | 413 s | 928 s | | judge r1 + fix + judge r2 | 2,163 s (317 s waiting on questions) | 2,955 s (judge r2 unfinished, plus 482 s lost to #295) | | **Agent time** | **5,762 s** | **7,550 s** (+31%) | **Quality** - Both builds compute the right figures to the cent. llama.cpp's: 400k at 6.5% over 30 years gives $2,528.27 a month and $510,179.81 interest; $200 extra pays it off 67 months earlier and saves $111,892.52. - Both failed judge round 1 the same way: opened from disk, the ES-module script is blocked. - llama.cpp's also showed stale results while an input was invalid: `.cards { display: grid }` overrode `hidden`. vLLM's kept stale figures too, with no mark. - Fix round: llama.cpp's fixed both, but the fresh page then showed red errors on fields nobody had touched yet. Judge round 2 caught that. vLLM's passed judge round 2. So far: the same correctness, vLLM about a third faster, and vLLM ahead on finish (one regression fewer). Not part of the pipeline: run 136 lost time to #294 (a failed step, repaired by hand) and #295.
Author
Owner

Closing: the comparison and the findings from both runs' transcripts are the comments above. Run 134 (vLLM) finished. Run 136 (llama.cpp) was stopped in judge round 2 once its evidence was in, and run 137 (llama.cpp) then finished a project end to end. Every finding is filed, and all but #280, which waits on a vLLM run with the judge's browser subagent, is closed.

Closing: the comparison and the findings from both runs' transcripts are the comments above. Run 134 (vLLM) finished. Run 136 (llama.cpp) was stopped in judge round 2 once its evidence was in, and run 137 (llama.cpp) then finished a project end to end. Every finding is filed, and all but #280, which waits on a vLLM run with the judge's browser subagent, is closed.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#279
No description provided.