Backend test: fastllm (Qwen3.8-27B) vs Ornith-1.5-122B on a tiny Split the Bill app #350
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What we're testing
Is Ornith-1.5-122B, at about 36 t/s and one request at a time, worth its slower decode on a Braid build? We compare it with fastllm's Qwen3.8-27B, which runs at about 105 t/s with six slots. Each backend gets one thorough run from the same tiny brief, and the owner judges both apps.
The brief (paste it verbatim into New Run for both runs)
How it runs
ornith-1.5-122bon port 8090 (stack 132), with one slot. It has no effort levels; thinking on is its maximum.Stats (from Braid's records, with tool time taken out as in #344)
npm testandnpm run build.Owner's judging (1-5 each)
Optional round 2 (only if both runs succeed and the owner says go)
A change request on each app's own repo:
Set up 2026-10-31 after the engine refresh. The speed comparison against vLLM is in #342; effort levels are in #349.
Findings from both runs. Both have finished; the side-by-side is in the next comment.
Run A, fastllm: run 161 on
cmoriarty/split-the-bill-fastllm, started 05:23Z.npm test(40 of 40 pass) andnpm run build(clean). One note, left for a person: a scenario says" ana "shows as "Ana", but its own requirement and the code keep the trimmed name, "ana".dist/also works when opened straight from disk, but its module script can't run fromfile://.Run B, Ornith: run 162 on
cmoriarty/split-the-bill-ornith, started 09:45Z.npm run buildbroken and fixed it.npm testpassed whiletscfailed.src/vite-env.d.ts; then the build passed and all 32 tests passed. One note was left, about a Node localStorage warning.Findings
[Decode]lines at alive = 4 split into long unbroken stretches: fast at about 170–215 t/s combined, and slow at about 93–102. The slow stretches are not prefill (pending = 0).max_input_tokens,max_model_lenorcontext_lengthat the top of each model entry. llama.cpp reports it asmeta.n_ctx(262,144 here).npm run buildto pass, not for the build to open from disk. The README promised that, and the browser check holds the app to its README..mapcall on a NodeList.Side-by-side: run 161 (fastllm, Qwen3.8-27B) and run 162 (Ornith-1.5-122B)
Both runs succeeded on the same brief, and both apps pass
npm testandnpm run buildon a clean checkout of main. Run 162 took 85 min; run 161 took 257.Ornith generated at about a third of fastllm's speed. But run 161 spent 180 of its 257 min waiting for first tokens: once browser screenshots were in its sessions, fastllm recomputed most of each prompt. Ornith's llama.cpp kept 97% of prompt tokens cached throughout, and run 162 waited 14 min in all.
The two runs didn't do the same work:
Totals
npm test/npm run buildon mainPer step: wall (first-token wait + generating + no request running), then requests
fastllm decode by sessions decoding at once (run 161)
From fastllm's own
[Decode]lines, one a second over the run.The apps
cmoriarty/split-the-bill-fastllmcmoriarty/split-the-bill-ornithnpm run buildwrites one self-containeddist/index.html, which opens from disknpm install, thennpm run dev. The build needs a serverAfterwards, stack 129 serves again, with Braid's fastllm profile active. The Ornith profile stays saved.
Owner's judging (2026-10-31)
The two apps tied on the scoring sheet. Both work well, and Ornith's looks a little better.
Reset looked broken in both apps, but the cause is the Claude desktop app's browser pane, where the owner judged. The pane answers
window.confirm()with Cancel without showing a dialog (it returned false in 1 ms). In a tab with its own storage, both apps' Reset left the data alone on Cancel and cleared everything once the confirmation was accepted. Reset counts against neither app.Verdict: between these two backends, Ornith-1.5-122B (stack 132) becomes the workhorse. The quality was the same, and its run took 85 min against 257.
Findings from the second round: vLLM (run 163) and Flash-Next (runs 164 and 165). Run 163 has finished. Run D was stopped, run E cancelled and a follow-up check of 8-bit weights on vLLM skipped, all at the owner's call (below).
Run C, vLLM: run 163 on
cmoriarty/split-the-bill-vllm, started 14:20Z.Findings
--enable-prompt-tokens-details, so Braid's cached column is 0.playmissing,finishednot true).--limit-mm-per-promptwith 16 images.find /to locate a browser, which Braid refused. In run 161, the judge settled what was left by reading the code.package.jsonasks for Node 26 or newer. Braid gives every new Node project an.npmrcwithengine-strict=true(#273), sonpm cirefuses Node 22..nvmrcnames Node 26 for every run, so all three apps were checked on Node 26: each installs, passesnpm testand builds.Run D, Flash-Next on ExLlamaV3: first run 164 on
cmoriarty/split-the-bill-flashnext, started 18:27Z and stopped at 19:27Z; then run 165 on a bigger window, 20:00-21:34Z (below).max_batch_size2). Loop detection is off, so a loop ends as "length".cache_mode"6,6"). An 8-bit cache at 262,144 didn't fit on the cards: the loader stopped with "Insufficient VRAM in split for model and cache".cmoriarty/split-the-bill-flashnext-262k, same brief and settings, started 20:00:55Z.openspec.apply[all]), as the 220,000 per agent intended: 38.1 min, all 19 tasks ticked. Braid then wrote spec tests for one scenario group (settlement, 4.3 min).Run E, DeepSeek V4.1 on llama.cpp: cancelled at the owner's call. In the owner's words: "no need for any extensive testing with deepseek, once you prove the stack works, you can stop it."
A short Braid check of 8-bit weights on vLLM: skipped at the owner's call.
Run 166: Qwen3.8-27B on vLLM with 8-bit weights, a 16-bit cache and no context stretch, against run 163
Run 166 succeeded in 93 min on
cmoriarty/split-the-bill-vllm-w8(05:30–07:04Z). It had the same brief and settings as run 163, which took 109 min.npm test(21 tests) and builds. It passes the money check, as run 163's app does.What changed on stack 126 (the owner's call, 2026-11-01)
Braid's settings
Side by side
npm ci/npm test/npm run build, Node 26Money check, run with each app's own money module on Node 26:
Findings
Afterwards, stack 126 still serves this setting. Braid's main profile keeps the 262,144 window and 230,144 per agent, and its effort is back to auto.