Backend test: fastllm (Qwen3.8-27B) vs Ornith-1.5-122B on a tiny Split the Bill app #350

Closed
opened 2026-10-31 01:18:05 -04:00 by cmoriarty · 5 comments
Owner

What we're testing
Is Ornith-1.5-122B, at about 36 t/s and one request at a time, worth its slower decode on a Braid build? We compare it with fastllm's Qwen3.8-27B, which runs at about 105 t/s with six slots. Each backend gets one thorough run from the same tiny brief, and the owner judges both apps.

The brief (paste it verbatim into New Run for both runs)

Build "Split the Bill", a small single-page web app for splitting shared expenses in a group. Stack: Vite + TypeScript, no UI framework, Vitest for tests; data stays in the browser (localStorage).

  1. People: add and remove people. Names are required and unique (ignoring case). A person with expenses can't be removed.
  2. Expenses: add one with a description, an amount, who paid, and who shares it (default: everyone). Amounts are positive with at most two decimals. Expenses can be deleted.
  3. Balances: show each person's net balance.
  4. Settle up: list the payments that clear every balance, as few as practical ("Ana pays Ben $12.50").
  5. Money is exact: work in cents, and give leftover cents to the first people in a split, so every split adds up to its total.
  6. Everything survives a reload. A Reset button clears all data after a confirmation.
  7. It works at phone width and with the keyboard alone.

Keep the money and settle-up logic in a module with no DOM code, with unit tests for uneven splits, rounding and settling. npm test and npm run build pass. The README says how to run, test and build the app, and states the rounding rule.

How it runs

  • Settings: thorough mode, auto-approve on, effort high, and a fresh project for each backend.
  • Isolation: one run at a time, with nothing else on the GPU host.
  • fastllm: profile "fastllm qwen38-27b" on stack 129, which now thinks at xhigh by default. Run 160 is not a baseline, because it thought at medium.
  • Ornith: a new llama.cpp profile, model ornith-1.5-122b on port 8090 (stack 132), with one slot. It has no effort levels; thinking on is its maximum.
  • Known handicaps:
    • fastllm has no prefix cache for requests that carry a screenshot, so it rereads those sessions in full.
    • Ornith's single slot makes Braid's parallel steps queue.

Stats (from Braid's records, with tool time taken out as in #344)

  • Time: wall time, time per step, and model time against tool time.
  • Tokens: requests, prompt tokens computed against cached, and output tokens split into thinking and answer.
  • Speed: decode t/s by prompt size, and time to first token (median and p90).
  • Reliability: failed or retried attempts, fix rounds, review findings, and compactions.
  • Result: npm test and npm run build.

Owner's judging (1-5 each)

  1. All seven features work.
  2. The money is right: three people, $100 split three ways, then settle up.
  3. Look and feel.
  4. Bad input is handled.
  5. Code you'd keep.
  6. Tests that mean something.
  7. The README is accurate.

Optional round 2 (only if both runs succeed and the owner says go)
A change request on each app's own repo:

  • Editing an expense.
  • Splitting by shares, for example 2:1:1.
  • A currency setting (USD, EUR, GBP), with all money formatting moved into one module.
  • Existing tests keep passing, and new tests cover shares.

Set up 2026-10-31 after the engine refresh. The speed comparison against vLLM is in #342; effort levels are in #349.

**What we're testing** Is Ornith-1.5-122B, at about 36 t/s and one request at a time, worth its slower decode on a Braid build? We compare it with fastllm's Qwen3.8-27B, which runs at about 105 t/s with six slots. Each backend gets one thorough run from the same tiny brief, and the owner judges both apps. **The brief** (paste it verbatim into New Run for both runs) > Build "Split the Bill", a small single-page web app for splitting shared expenses in a group. Stack: Vite + TypeScript, no UI framework, Vitest for tests; data stays in the browser (localStorage). > 1. People: add and remove people. Names are required and unique (ignoring case). A person with expenses can't be removed. > 2. Expenses: add one with a description, an amount, who paid, and who shares it (default: everyone). Amounts are positive with at most two decimals. Expenses can be deleted. > 3. Balances: show each person's net balance. > 4. Settle up: list the payments that clear every balance, as few as practical ("Ana pays Ben $12.50"). > 5. Money is exact: work in cents, and give leftover cents to the first people in a split, so every split adds up to its total. > 6. Everything survives a reload. A Reset button clears all data after a confirmation. > 7. It works at phone width and with the keyboard alone. > > Keep the money and settle-up logic in a module with no DOM code, with unit tests for uneven splits, rounding and settling. `npm test` and `npm run build` pass. The README says how to run, test and build the app, and states the rounding rule. **How it runs** - **Settings:** thorough mode, auto-approve on, effort high, and a fresh project for each backend. - **Isolation:** one run at a time, with nothing else on the GPU host. - **fastllm:** profile "fastllm qwen38-27b" on stack 129, which now thinks at xhigh by default. Run 160 is not a baseline, because it thought at medium. - **Ornith:** a new llama.cpp profile, model `ornith-1.5-122b` on port 8090 (stack 132), with one slot. It has no effort levels; thinking on is its maximum. - **Known handicaps:** - fastllm has no prefix cache for requests that carry a screenshot, so it rereads those sessions in full. - Ornith's single slot makes Braid's parallel steps queue. **Stats** (from Braid's records, with tool time taken out as in #344) - **Time:** wall time, time per step, and model time against tool time. - **Tokens:** requests, prompt tokens computed against cached, and output tokens split into thinking and answer. - **Speed:** decode t/s by prompt size, and time to first token (median and p90). - **Reliability:** failed or retried attempts, fix rounds, review findings, and compactions. - **Result:** `npm test` and `npm run build`. **Owner's judging (1-5 each)** 1. All seven features work. 2. The money is right: three people, $100 split three ways, then settle up. 3. Look and feel. 4. Bad input is handled. 5. Code you'd keep. 6. Tests that mean something. 7. The README is accurate. **Optional round 2** (only if both runs succeed and the owner says go) A change request on each app's own repo: - Editing an expense. - Splitting by shares, for example 2:1:1. - A currency setting (USD, EUR, GBP), with all money formatting moved into one module. - Existing tests keep passing, and new tests cover shares. *Set up 2026-10-31 after the engine refresh. The speed comparison against vLLM is in #342; effort levels are in #349.*
Author
Owner

Findings from both runs. Both have finished; the side-by-side is in the next comment.

Run A, fastllm: run 161 on cmoriarty/split-the-bill-fastllm, started 05:23Z.

  • Thorough New Project (one-shot), auto-approve on, effort high.
  • Profile "fastllm qwen38-27b": 6 sessions, 5 subagents, 200k per agent, apply auto. Stack 129 thinks at xhigh by default.
  • Proposal review done 05:41Z: four reviewers in parallel. Spec found nothing; scope found 3 to fix, test 3 to fix and 3 notes, risk 1 to fix and 2 notes. One revise pass marked all of them revised. No failed attempts so far.
  • Apply done 07:06Z: one step, 84 min, 271 turns, all 17 tasks ticked. It wrote tests for 28 of the 43 scenarios and left the other 15 to the browser check, so the extra test-writing step was skipped. Browser checks ran in subagents inside the step. No failed attempts.
  • Code review done 07:12Z: 6 min, 17 turns, no code changes. The reviewer ran npm test (40 of 40 pass) and npm run build (clean). One note, left for a person: a scenario says " ana " shows as "Ana", but its own requirement and the code keep the trimmed name, "ana".
  • Tests passed on the first round, 07:12Z.
  • Browser check round 1 failed, 07:54Z: 42 min, 114 turns. All 18 criteria passed. The fail was one way of opening the app: the README says the built dist/ also works when opened straight from disk, but its module script can't run from file://.
  • Fix round 1 done 08:39Z: 45 min, 59 turns. It made the built page work from disk and checked that in a browser. Tests round 2 passed at 08:40Z.
  • Browser check round 2 passed, 09:36Z: 56 min, 121 turns, peak 155k. All 18 criteria pass, and both ways of opening work, from disk included. This time it measured phone width in a browser at 360px. Its last browser subagent ended at 09:13Z, and the step closed 23 min later.
  • Run A succeeded at 09:40Z, after 4 h 17 min. Auto-approve merged its pull request (cmoriarty/split-the-bill-fastllm#1) and archived the change.

Run B, Ornith: run 162 on cmoriarty/split-the-bill-ornith, started 09:45Z.

  • Same brief and settings: Thorough New Project (one-shot), auto-approve on, effort high.
  • Profile "llama.cpp ornith-1.5-122b". Connect found one slot, so the console's one-session defaults apply: one session, no subagents, one run at a time, apply in a single step.
  • The proposal review is one combined reviewer, where Run A had four in parallel.
  • With no subagents, browser checks run in the step's own session.
  • The server was healthy 20 s after start. A test completion decoded at 37 t/s.
  • Propose done 09:52Z: explore 3 min (8 turns), propose 4.5 min (18 turns, peak 29k), validated strictly.
  • Proposal review done 10:08Z: one combined reviewer, 14 min and 12 turns, found 3 to fix and 1 note. One revise pass (2.5 min) marked all three revised. Run A's four reviewers had found 7 to fix and 5 notes.
  • Apply done 10:51Z: 38.5 min and 98 turns in one step, peak 100k, all 15 tasks ticked. Run A's apply took 85 min and 271 turns. The apply step checked the page in a browser itself, with no subagents.
    • One scenario of the persistence spec had no tagged test, so the extra test-writing step ran: 4 min, 5 turns.
    • No failed attempts, no compaction. The one tool error was a read of a page snapshot that didn't exist.
  • Code review done 10:59Z: 8.6 min, 10 turns. It found npm run build broken and fixed it.
    • The test the extra step wrote for persistence (10:50Z) didn't type-check against the page's typing. Vitest doesn't type-check, so npm test passed while tsc failed.
    • It fixed the typing in src/vite-env.d.ts; then the build passed and all 32 tests passed. One note was left, about a Node localStorage warning.
  • Tests passed on the first round, 10:59Z.
  • Browser check passed on the first round, 11:06Z: 7 min, 36 turns, peak 38k, so no fix round.
    • It judged 4 criteria of its own: what the page shows, validation messages, the look at phone width, and the README. The 22 logic scenarios were left to the unit tests.
    • It took a screenshot at 390 px wide. It pressed no keys: it judged keyboard use from the focus styles.
    • Run A's browser check judged 18 criteria and pressed Tab for real.
  • Run B succeeded at 11:10Z, after 1 h 25 min. Auto-approve merged its pull request (cmoriarty/split-the-bill-ornith#1) and archived the change.
  • llama.cpp kept its prompt cache with screenshots in the session. Through apply, 98% of all prompt tokens were cached. The median wait for the first token was about 1 s (p90 5 s), 9 min in all, against 52 min generating at 31–36 t/s.

Findings

  • fastllm starves some sequences at four sessions. Over 05:28–05:37Z, its [Decode] lines at alive = 4 split into long unbroken stretches: fast at about 170–215 t/s combined, and slow at about 93–102. The slow stretches are not prefill (pending = 0).
    • In the slow stretches, the four reviewers don't slow evenly. Scope and risk kept producing at about 30–50 t/s each.
    • Spec's turn at 05:33:35 streamed 76 characters of reasoning over 61 s (83 tokens). Test's turn wrote 165 tokens in 107 s.
    • Every turn in both modes was a reviewer's, so no single step lines up with the slow mode. The mode changes when a request joins or leaves the batch.
    • Run A never had four sessions alive again after 05:37Z. After that, at most two were alive: in apply (06:00–06:34Z, about 3 min in all) and in code review (07:11Z). The full table by sessions alive comes with the stats.
  • Run A's time went mostly into re-reading prompts once screenshots were in the sessions. Of its 257 min, 180 were spent waiting for the first token and 73 generating. Under 5 min had no request running.
    • In apply, before the first browser check (06:00Z): 52 requests, 99% of prompt tokens cached, median 3.3 s to the first token.
    • After it: 219 requests, 52% cached, median 14.7 s.
    • The steps after that cached even less: browser check 28%, fix 3%, browser check round 2 12%, summary 0%. Those prompts were recomputed at about 1,000–1,100 t/s.
    • This matches what the owner measured on 10-30: fastllm doesn't reuse its prefix cache for requests that carry an image.
  • Connect leaves the context window empty for llama.cpp. It filled the output (20,480) but not the context.
    • Braid looks for the window under max_input_tokens, max_model_len or context_length at the top of each model entry. llama.cpp reports it as meta.n_ctx (262,144 here).
    • I typed in the server's own 262,144, as a person would.
  • A context alarm stopped a browser check, which then ran again from the start. The apply session had continued an earlier step's session. It reached the 170k alarm at 06:33Z, just as a browser subagent began checking balances, Reset and the end-to-end scenario.
    • Braid waited its 300 s for running subagents. The check didn't finish in time, so Braid stopped it and the session at 06:38Z. It went on in a fresh session from its own 9k-character note.
    • The fresh session ran the same check again from the start, 06:40–06:46Z. About 5 min of browser work was lost.
  • The skip note for the test-writing step reads "every scenario has a tagged test (28 of 43)". The other 15 were left to the browser check, which the note doesn't say.
  • A line the run added to its own README cost a 45-min fix round. The brief asks for npm run build to pass, not for the build to open from disk. The README promised that, and the browser check holds the app to its README.
  • The browser check ran out of browser sessions before it was done. Its four subagent checks took 11, 2, 3 and 7 min, and Braid refused a fifth.
    • The judge settled phone width from the stylesheet, without a 360px capture.
    • It judged the from-disk opening from the build output, without opening it.
    • It was right about the build: the fix step's own browser check confirmed the page was broken from disk.
  • Tool errors in apply were the agent's own, and each was fixed on the next turn. A mistyped work path that Braid refused, a bad Playwright selector, and a .map call on a NodeList.
**Findings from both runs.** Both have finished; the side-by-side is in the next comment. **Run A, fastllm:** run 161 on `cmoriarty/split-the-bill-fastllm`, started 05:23Z. - Thorough New Project (one-shot), auto-approve on, effort high. - Profile "fastllm qwen38-27b": 6 sessions, 5 subagents, 200k per agent, apply auto. Stack 129 thinks at xhigh by default. - **Proposal review done 05:41Z:** four reviewers in parallel. Spec found nothing; scope found 3 to fix, test 3 to fix and 3 notes, risk 1 to fix and 2 notes. One revise pass marked all of them revised. No failed attempts so far. - **Apply done 07:06Z:** one step, 84 min, 271 turns, all 17 tasks ticked. It wrote tests for 28 of the 43 scenarios and left the other 15 to the browser check, so the extra test-writing step was skipped. Browser checks ran in subagents inside the step. No failed attempts. - **Code review done 07:12Z:** 6 min, 17 turns, no code changes. The reviewer ran `npm test` (40 of 40 pass) and `npm run build` (clean). One note, left for a person: a scenario says `" ana "` shows as "Ana", but its own requirement and the code keep the trimmed name, "ana". - **Tests passed on the first round, 07:12Z.** - **Browser check round 1 failed, 07:54Z:** 42 min, 114 turns. All 18 criteria passed. The fail was one way of opening the app: the README says the built `dist/` also works when opened straight from disk, but its module script can't run from `file://`. - **Fix round 1 done 08:39Z:** 45 min, 59 turns. It made the built page work from disk and checked that in a browser. Tests round 2 passed at 08:40Z. - **Browser check round 2 passed, 09:36Z:** 56 min, 121 turns, peak 155k. All 18 criteria pass, and both ways of opening work, from disk included. This time it measured phone width in a browser at 360px. Its last browser subagent ended at 09:13Z, and the step closed 23 min later. - **Run A succeeded at 09:40Z, after 4 h 17 min.** Auto-approve merged its pull request (cmoriarty/split-the-bill-fastllm#1) and archived the change. **Run B, Ornith:** run 162 on `cmoriarty/split-the-bill-ornith`, started 09:45Z. - Same brief and settings: Thorough New Project (one-shot), auto-approve on, effort high. - Profile "llama.cpp ornith-1.5-122b". Connect found one slot, so the console's one-session defaults apply: one session, no subagents, one run at a time, apply in a single step. - The proposal review is one combined reviewer, where Run A had four in parallel. - With no subagents, browser checks run in the step's own session. - The server was healthy 20 s after start. A test completion decoded at 37 t/s. - **Propose done 09:52Z:** explore 3 min (8 turns), propose 4.5 min (18 turns, peak 29k), validated strictly. - **Proposal review done 10:08Z:** one combined reviewer, 14 min and 12 turns, found 3 to fix and 1 note. One revise pass (2.5 min) marked all three revised. Run A's four reviewers had found 7 to fix and 5 notes. - **Apply done 10:51Z:** 38.5 min and 98 turns in one step, peak 100k, all 15 tasks ticked. Run A's apply took 85 min and 271 turns. The apply step checked the page in a browser itself, with no subagents. - One scenario of the persistence spec had no tagged test, so the extra test-writing step ran: 4 min, 5 turns. - No failed attempts, no compaction. The one tool error was a read of a page snapshot that didn't exist. - **Code review done 10:59Z:** 8.6 min, 10 turns. It found `npm run build` broken and fixed it. - The test the extra step wrote for persistence (10:50Z) didn't type-check against the page's typing. Vitest doesn't type-check, so `npm test` passed while `tsc` failed. - It fixed the typing in `src/vite-env.d.ts`; then the build passed and all 32 tests passed. One note was left, about a Node localStorage warning. - **Tests passed on the first round, 10:59Z.** - **Browser check passed on the first round, 11:06Z:** 7 min, 36 turns, peak 38k, so no fix round. - It judged 4 criteria of its own: what the page shows, validation messages, the look at phone width, and the README. The 22 logic scenarios were left to the unit tests. - It took a screenshot at 390 px wide. It pressed no keys: it judged keyboard use from the focus styles. - Run A's browser check judged 18 criteria and pressed Tab for real. - **Run B succeeded at 11:10Z, after 1 h 25 min.** Auto-approve merged its pull request (cmoriarty/split-the-bill-ornith#1) and archived the change. - **llama.cpp kept its prompt cache with screenshots in the session.** Through apply, 98% of all prompt tokens were cached. The median wait for the first token was about 1 s (p90 5 s), 9 min in all, against 52 min generating at 31–36 t/s. **Findings** - **fastllm starves some sequences at four sessions.** Over 05:28–05:37Z, its `[Decode]` lines at alive = 4 split into long unbroken stretches: fast at about 170–215 t/s combined, and slow at about 93–102. The slow stretches are not prefill (pending = 0). - In the slow stretches, the four reviewers don't slow evenly. Scope and risk kept producing at about 30–50 t/s each. - Spec's turn at 05:33:35 streamed 76 characters of reasoning over 61 s (83 tokens). Test's turn wrote 165 tokens in 107 s. - Every turn in both modes was a reviewer's, so no single step lines up with the slow mode. The mode changes when a request joins or leaves the batch. - Run A never had four sessions alive again after 05:37Z. After that, at most two were alive: in apply (06:00–06:34Z, about 3 min in all) and in code review (07:11Z). The full table by sessions alive comes with the stats. - **Run A's time went mostly into re-reading prompts once screenshots were in the sessions.** Of its 257 min, 180 were spent waiting for the first token and 73 generating. Under 5 min had no request running. - In apply, before the first browser check (06:00Z): 52 requests, 99% of prompt tokens cached, median 3.3 s to the first token. - After it: 219 requests, 52% cached, median 14.7 s. - The steps after that cached even less: browser check 28%, fix 3%, browser check round 2 12%, summary 0%. Those prompts were recomputed at about 1,000–1,100 t/s. - This matches what the owner measured on 10-30: fastllm doesn't reuse its prefix cache for requests that carry an image. - **Connect leaves the context window empty for llama.cpp.** It filled the output (20,480) but not the context. - Braid looks for the window under `max_input_tokens`, `max_model_len` or `context_length` at the top of each model entry. llama.cpp reports it as `meta.n_ctx` (262,144 here). - I typed in the server's own 262,144, as a person would. - **A context alarm stopped a browser check, which then ran again from the start.** The apply session had continued an earlier step's session. It reached the 170k alarm at 06:33Z, just as a browser subagent began checking balances, Reset and the end-to-end scenario. - Braid waited its 300 s for running subagents. The check didn't finish in time, so Braid stopped it and the session at 06:38Z. It went on in a fresh session from its own 9k-character note. - The fresh session ran the same check again from the start, 06:40–06:46Z. About 5 min of browser work was lost. - **The skip note for the test-writing step reads "every scenario has a tagged test (28 of 43)".** The other 15 were left to the browser check, which the note doesn't say. - **A line the run added to its own README cost a 45-min fix round.** The brief asks for `npm run build` to pass, not for the build to open from disk. The README promised that, and the browser check holds the app to its README. - **The browser check ran out of browser sessions before it was done.** Its four subagent checks took 11, 2, 3 and 7 min, and Braid refused a fifth. - The judge settled phone width from the stylesheet, without a 360px capture. - It judged the from-disk opening from the build output, without opening it. - It was right about the build: the fix step's own browser check confirmed the page was broken from disk. - **Tool errors in apply were the agent's own, and each was fixed on the next turn.** A mistyped work path that Braid refused, a bad Playwright selector, and a `.map` call on a NodeList.
Author
Owner

Side-by-side: run 161 (fastllm, Qwen3.8-27B) and run 162 (Ornith-1.5-122B)

Both runs succeeded on the same brief, and both apps pass npm test and npm run build on a clean checkout of main. Run 162 took 85 min; run 161 took 257.

Ornith generated at about a third of fastllm's speed. But run 161 spent 180 of its 257 min waiting for first tokens: once browser screenshots were in its sessions, fastllm recomputed most of each prompt. Ornith's llama.cpp kept 97% of prompt tokens cached throughout, and run 162 waited 14 min in all.

The two runs didn't do the same work:

  • Sessions: run 161 had six sessions, with subagents. Run 162 had one, so the console's one-session defaults applied: no subagents and one combined reviewer.
  • Browser check: run 161's held the app to a README line saying the build opens from disk, which cost a fix round. Run 162's passed on the first round, on 4 criteria; run 161's judged 18.
  • Thinking: Braid's "high" sends nothing to either backend. Stack 129 thinks at its default, xhigh. Ornith's template has only thinking on or off, and it was on.

Totals

Run 161: fastllm, Qwen3.8-27B Run 162: Ornith-1.5-122B
Result succeeded; pull request merged succeeded; pull request merged
npm test / npm run build on main pass, 40 tests / pass pass, 32 tests / pass
Wall time 257 min 85 min
First-token wait / generating / no request running 179.6 / 73.0 / 4.4 min 14.3 / 64.5 / 6.3 min
Requests 638 197
Prompt tokens: computed / cached 11.6M / 13.4M (54% cached) 0.35M / 10.1M (97% cached)
Output tokens: thinking / answer (estimated) 290k / 137k 83k / 46k
Decode t/s, request decoding alone: under 16k / 16–32k / 32–64k / 64k+ 120 / 108 / 97 / 99 36 / 36 / 34 / 31
First token, median / p90 11.1 s / 32.9 s 1.3 s / 6.3 s
First token at 64k+, median / p90 13.2 s / 112 s 1.4 s / 4.0 s
Failed or retried attempts 0 0
Fix rounds 1 0
Review findings proposal: 7 to fix (all revised), 5 notes; code: 1 note proposal: 3 to fix (all revised), 1 note; code: 1 note, and it fixed a build break
Browser check round 1 failed on opening from disk; round 2 passed 18 of 18 round 1 passed 4 of 4
Compactions 0; one fresh session at the context alarm 0
  • Time is wall clock, and overlapping requests count once. A step's tool time counts as "no request running" only when no other request was running. A step waiting on its own subagent counts as model time.
  • The thinking and answer split is an estimate. Neither backend reports reasoning tokens, so each message's output tokens are split by its reasoning characters against its text and tool-call characters.
  • Decode is a request's output tokens over the time from its first token to the end of its generation. It counts only requests that decoded alone.

Per step: wall (first-token wait + generating + no request running), then requests

Step Run 161 Run 162
Explore 1.2 min (0.3 + 0.9 + 0.1), 7 2.5 min (0.7 + 1.8 + 0.1), 8
Propose 4.2 min (0.8 + 3.2 + 0.1), 18 4.5 min (0.7 + 3.7 + 0.1), 18
Proposal review 4 reviewers in parallel, 7.7–9.1 min each, 23 in all 1 combined reviewer: 13.6 min (0.9 + 12.7 + 0.1), 12
Revise 3.2 min (1.7 + 1.5 + 0.1), 6 2.7 min (1.1 + 1.6 + 0.1), 8
Apply 84.8 min (55.2 + 27.6 + 2.0), 271 38.4 min (4.4 + 29.0 + 5.0), 98
Extra spec tests skipped 4.0 min (0.7 + 3.2 + 0.1), 5
Code review 6.4 min (1.7 + 4.6 + 0.1), 17 8.5 min (1.2 + 7.2 + 0.1), 10
Browser check 41.5 min (28.0 + 12.9 + 0.5), 114 7.0 min (2.0 + 4.8 + 0.2), 36
Fix 45.3 min (39.4 + 5.6 + 0.3), 59 not needed
Browser check, round 2 56.9 min (49.3 + 6.9 + 0.6), 121 not needed
Summary 3.9 min (2.9 + 1.0 + 0.0), 2 3.5 min (2.9 + 0.6 + 0.0), 2
  • The runs were close through proposal review: 18 min for run 161, 24 for run 162. The gap opened in apply and widened after it.
  • Run 161's first-token waits grew once screenshots arrived. Its apply step cached 99% of prompt tokens before the first browser check, then 52% after. The browser checks, the fix and the summary cached 0–28%.

fastllm decode by sessions decoding at once (run 161)

From fastllm's own [Decode] lines, one a second over the run.

Sessions Seconds Combined t/s, median (p10–p90) Per session Owner's benchmark
1 3,562 97.5 (75–125) 97.5 97
2 196 99 (75–145) 50 131
3 67 178 (24–216) 59 182
4 363 102 (82–191) 25 about 180
  • At four, every second was the four proposal reviewers, 05:29–05:36Z. Nothing later reached four.
    • The samples split into two modes: 90 near 182 t/s and 258 near 94 t/s. No prefill was waiting in either (pending 0).
    • Every request decoding in both modes was a reviewer's, so no one step lines up with the slow mode. The mode changed when a request joined or left the batch.
    • In the slow mode the reviewers didn't slow evenly; the findings comment above has the detail.
  • At two, the rate depended on when. In the proposal review (05:37Z) the combined rate was 130–190, as benchmarked. In apply (06:00–06:34Z) it mostly ran at 90–120, about one session's rate. In code review (07:11Z) it ran at 150 for 17 s, then 89.

The apps

Run 161 Run 162
Repo cmoriarty/split-the-bill-fastllm cmoriarty/split-the-bill-ornith
Pull request cmoriarty/split-the-bill-fastllm#1 cmoriarty/split-the-bill-ornith#1
Tools Vite 8, Vitest 5, TypeScript 7 Vite 5, Vitest 2, TypeScript 5; tests run in jsdom
To open it npm run build writes one self-contained dist/index.html, which opens from disk npm install, then npm run dev. The build needs a server

Afterwards, stack 129 serves again, with Braid's fastllm profile active. The Ornith profile stays saved.

**Side-by-side: run 161 (fastllm, Qwen3.8-27B) and run 162 (Ornith-1.5-122B)** Both runs succeeded on the same brief, and both apps pass `npm test` and `npm run build` on a clean checkout of main. Run 162 took 85 min; run 161 took 257. Ornith generated at about a third of fastllm's speed. But run 161 spent 180 of its 257 min waiting for first tokens: once browser screenshots were in its sessions, fastllm recomputed most of each prompt. Ornith's llama.cpp kept 97% of prompt tokens cached throughout, and run 162 waited 14 min in all. The two runs didn't do the same work: - **Sessions:** run 161 had six sessions, with subagents. Run 162 had one, so the console's one-session defaults applied: no subagents and one combined reviewer. - **Browser check:** run 161's held the app to a README line saying the build opens from disk, which cost a fix round. Run 162's passed on the first round, on 4 criteria; run 161's judged 18. - **Thinking:** Braid's "high" sends nothing to either backend. Stack 129 thinks at its default, xhigh. Ornith's template has only thinking on or off, and it was on. **Totals** | | Run 161: fastllm, Qwen3.8-27B | Run 162: Ornith-1.5-122B | |---|---|---| | Result | succeeded; pull request merged | succeeded; pull request merged | | `npm test` / `npm run build` on main | pass, 40 tests / pass | pass, 32 tests / pass | | Wall time | 257 min | 85 min | | First-token wait / generating / no request running | 179.6 / 73.0 / 4.4 min | 14.3 / 64.5 / 6.3 min | | Requests | 638 | 197 | | Prompt tokens: computed / cached | 11.6M / 13.4M (54% cached) | 0.35M / 10.1M (97% cached) | | Output tokens: thinking / answer (estimated) | 290k / 137k | 83k / 46k | | Decode t/s, request decoding alone: under 16k / 16–32k / 32–64k / 64k+ | 120 / 108 / 97 / 99 | 36 / 36 / 34 / 31 | | First token, median / p90 | 11.1 s / 32.9 s | 1.3 s / 6.3 s | | First token at 64k+, median / p90 | 13.2 s / 112 s | 1.4 s / 4.0 s | | Failed or retried attempts | 0 | 0 | | Fix rounds | 1 | 0 | | Review findings | proposal: 7 to fix (all revised), 5 notes; code: 1 note | proposal: 3 to fix (all revised), 1 note; code: 1 note, and it fixed a build break | | Browser check | round 1 failed on opening from disk; round 2 passed 18 of 18 | round 1 passed 4 of 4 | | Compactions | 0; one fresh session at the context alarm | 0 | - **Time is wall clock, and overlapping requests count once.** A step's tool time counts as "no request running" only when no other request was running. A step waiting on its own subagent counts as model time. - **The thinking and answer split is an estimate.** Neither backend reports reasoning tokens, so each message's output tokens are split by its reasoning characters against its text and tool-call characters. - **Decode** is a request's output tokens over the time from its first token to the end of its generation. It counts only requests that decoded alone. **Per step:** wall (first-token wait + generating + no request running), then requests | Step | Run 161 | Run 162 | |---|---|---| | Explore | 1.2 min (0.3 + 0.9 + 0.1), 7 | 2.5 min (0.7 + 1.8 + 0.1), 8 | | Propose | 4.2 min (0.8 + 3.2 + 0.1), 18 | 4.5 min (0.7 + 3.7 + 0.1), 18 | | Proposal review | 4 reviewers in parallel, 7.7–9.1 min each, 23 in all | 1 combined reviewer: 13.6 min (0.9 + 12.7 + 0.1), 12 | | Revise | 3.2 min (1.7 + 1.5 + 0.1), 6 | 2.7 min (1.1 + 1.6 + 0.1), 8 | | Apply | 84.8 min (55.2 + 27.6 + 2.0), 271 | 38.4 min (4.4 + 29.0 + 5.0), 98 | | Extra spec tests | skipped | 4.0 min (0.7 + 3.2 + 0.1), 5 | | Code review | 6.4 min (1.7 + 4.6 + 0.1), 17 | 8.5 min (1.2 + 7.2 + 0.1), 10 | | Browser check | 41.5 min (28.0 + 12.9 + 0.5), 114 | 7.0 min (2.0 + 4.8 + 0.2), 36 | | Fix | 45.3 min (39.4 + 5.6 + 0.3), 59 | not needed | | Browser check, round 2 | 56.9 min (49.3 + 6.9 + 0.6), 121 | not needed | | Summary | 3.9 min (2.9 + 1.0 + 0.0), 2 | 3.5 min (2.9 + 0.6 + 0.0), 2 | - **The runs were close through proposal review: 18 min for run 161, 24 for run 162.** The gap opened in apply and widened after it. - **Run 161's first-token waits grew once screenshots arrived.** Its apply step cached 99% of prompt tokens before the first browser check, then 52% after. The browser checks, the fix and the summary cached 0–28%. **fastllm decode by sessions decoding at once (run 161)** From fastllm's own `[Decode]` lines, one a second over the run. | Sessions | Seconds | Combined t/s, median (p10–p90) | Per session | Owner's benchmark | |---|---|---|---|---| | 1 | 3,562 | 97.5 (75–125) | 97.5 | 97 | | 2 | 196 | 99 (75–145) | 50 | 131 | | 3 | 67 | 178 (24–216) | 59 | 182 | | 4 | 363 | 102 (82–191) | 25 | about 180 | - **At four, every second was the four proposal reviewers, 05:29–05:36Z.** Nothing later reached four. - The samples split into two modes: 90 near 182 t/s and 258 near 94 t/s. No prefill was waiting in either (pending 0). - Every request decoding in both modes was a reviewer's, so no one step lines up with the slow mode. The mode changed when a request joined or left the batch. - In the slow mode the reviewers didn't slow evenly; the findings comment above has the detail. - **At two, the rate depended on when.** In the proposal review (05:37Z) the combined rate was 130–190, as benchmarked. In apply (06:00–06:34Z) it mostly ran at 90–120, about one session's rate. In code review (07:11Z) it ran at 150 for 17 s, then 89. **The apps** | | Run 161 | Run 162 | |---|---|---| | Repo | `cmoriarty/split-the-bill-fastllm` | `cmoriarty/split-the-bill-ornith` | | Pull request | cmoriarty/split-the-bill-fastllm#1 | cmoriarty/split-the-bill-ornith#1 | | Tools | Vite 8, Vitest 5, TypeScript 7 | Vite 5, Vitest 2, TypeScript 5; tests run in jsdom | | To open it | `npm run build` writes one self-contained `dist/index.html`, which opens from disk | `npm install`, then `npm run dev`. The build needs a server | Afterwards, stack 129 serves again, with Braid's fastllm profile active. The Ornith profile stays saved.
Author
Owner

Owner's judging (2026-10-31)

The two apps tied on the scoring sheet. Both work well, and Ornith's looks a little better.

Reset looked broken in both apps, but the cause is the Claude desktop app's browser pane, where the owner judged. The pane answers window.confirm() with Cancel without showing a dialog (it returned false in 1 ms). In a tab with its own storage, both apps' Reset left the data alone on Cancel and cleared everything once the confirmation was accepted. Reset counts against neither app.

Verdict: between these two backends, Ornith-1.5-122B (stack 132) becomes the workhorse. The quality was the same, and its run took 85 min against 257.

**Owner's judging (2026-10-31)** The two apps tied on the scoring sheet. Both work well, and Ornith's looks a little better. Reset looked broken in both apps, but the cause is the Claude desktop app's browser pane, where the owner judged. The pane answers `window.confirm()` with Cancel without showing a dialog (it returned false in 1 ms). In a tab with its own storage, both apps' Reset left the data alone on Cancel and cleared everything once the confirmation was accepted. Reset counts against neither app. **Verdict:** between these two backends, Ornith-1.5-122B (stack 132) becomes the workhorse. The quality was the same, and its run took 85 min against 257.
Author
Owner

Findings from the second round: vLLM (run 163) and Flash-Next (runs 164 and 165). Run 163 has finished. Run D was stopped, run E cancelled and a follow-up check of 8-bit weights on vLLM skipped, all at the owner's call (below).

Run C, vLLM: run 163 on cmoriarty/split-the-bill-vllm, started 14:20Z.

  • Same brief and settings: Thorough New Project (one-shot), auto-approve on, effort high.
  • Braid's main profile, "vLLM · qwen3.8-27b". Its effort was set to high for the run and back to auto after. The stack thinks at xhigh by default, as run 161's did.
  • That profile keeps its own run settings: 400k per agent (alarm at 370k) and up to 8 subagents. Run 161's fastllm profile had 200k per agent and 5 subagents.
  • vLLM v0.32.0, six sequences at once, a 524k window.
  • Propose done 14:30Z: explore 1.7 min (7 turns), propose 7.8 min (19 turns, peak 48k).
  • Proposal review done 14:45Z: four reviewers in parallel.
    • Spec found 1 to fix, scope 6 to fix and 1 note, test 2 to fix and 3 notes. Risk found nothing.
    • One revise pass (3.8 min) marked all nine revised.
  • Apply done 15:29Z: one step, 43.6 min, 196 turns, peak 175k.
    • It tagged tests for 23 of the 42 scenarios and left the other 19 to the browser check, so the extra test-writing step was skipped.
    • Its browser subagents took the run's first screenshots at 15:06Z.
  • Code review done 15:34Z: 5.2 min, 9 turns. It fixed two minor findings and left two notes.
  • Tests passed on the first round, 15:34Z.
  • Browser check passed on the first round, 16:04Z, on a second attempt: 30 min, 157 turns, 24 of 24 criteria, so no fix round.
    • Attempt 1 failed at 15:58Z (below). Braid retried the step, and attempt 2 took 5.7 min.
  • Run C succeeded at 16:09Z, after 1 h 49 min. Auto-approve merged its pull request (cmoriarty/split-the-bill-vllm#1) and archived the change.

Findings

  • vLLM kept its prompt cache with screenshots in the sessions. Over the run, 94% of prompt tokens came from the cache: 94% before the first screenshot and 95% after.
    • Apply cached 98%, the browser check 93%. The median wait for the first token was 0.6 s (p90 4.0 s), 12 min in all.
    • Run 161 on fastllm cached 54% over the run. In its apply step, the rate fell from 99% before the first browser check to 52% after.
    • These numbers come from vLLM's own prefix-cache counters. vLLM reports cached tokens per request only when started with --enable-prompt-tokens-details, so Braid's cached column is 0.
    • Its prompt-token count matches Braid's total to within 0.05%, so nothing else used the server during the run.
  • The first browser check attempt hit vLLM's limit of 16 images per prompt.
    • The judge read the 8 screenshots its four browser subagents had saved.
    • Braid refused a fifth browser check. The judge then wrote a Playwright script of its own, ran it from the shell and read 10 more screenshots.
    • The request carrying the 17th and 18th images got HTTP 400: "At most 16 image(s) may be provided in one prompt". The session errored, and the judge file was left unfinished (play missing, finished not true).
    • Braid retried the step, and attempt 2 passed. Attempt 1 had run for 25 min.
    • The cap is the stack's own setting, --limit-mm-per-prompt with 16 images.
    • The stack has since been redeployed with a cap of 999 images.
  • The judge worked around its cap of four browser checks. Braid counts subagents, so the judge drove the browser itself from a script. It first tried find / to locate a browser, which Braid refused. In run 161, the judge settled what was left by reading the code.
  • The skip note for the test-writing step again reads "every scenario has a tagged test (23 of 42)". As in run 161, it doesn't say the other 19 were left to the browser check.
  • The summary step opens with a prompt no backend has cached. Its first request waited 85 s on fastllm (90k tokens), 170 s on Ornith (93k) and 158 s on vLLM (179k).
  • The app requires Node 26, by design. Its package.json asks for Node 26 or newer. Braid gives every new Node project an .npmrc with engine-strict=true (#273), so npm ci refuses Node 22.
    • The scaffold's .nvmrc names Node 26 for every run, so all three apps were checked on Node 26: each installs, passes npm test and builds.
  • Three Triton kernels compiled during the run's first requests (14:20–14:21Z). The stack's warmup doesn't cover them; it cost a few seconds, once.

Run D, Flash-Next on ExLlamaV3: first run 164 on cmoriarty/split-the-bill-flashnext, started 18:27Z and stopped at 19:27Z; then run 165 on a bigger window, 20:00-21:34Z (below).

  • The stack: ExLlamaV3 1.6.5 with tabbyAPI, the 3.05 bpw quant, a 131,072-token window with an FP16 cache shared by two requests at once (max_batch_size 2). Loop detection is off, so a loop ends as "length".
  • Braid's profile "exl3 qwen38-flash-next", effort high, so the stack's xhigh default applies.
    • The profile had no run settings of its own, so it would have taken whatever profile was in use. I saved the main profile's: up to 8 subagents, mediation 4, apply auto, auto-approve.
    • Connect read two parallel sessions from tabby's slot count.
    • 110,592 tokens per agent (alarm at 82,944): what the window leaves after the 20,480-token output reserve.
  • Propose done 18:36Z: explore and propose took 8.5 min.
  • Proposal review done 19:03Z. With two sessions, Braid ran the four reviewers two at a time. They found 9 to fix (spec 4, scope 1, test 2, risk 2) and 8 notes, and one revise pass followed.
  • Apply split into six parts, run one after another. Under apply auto, Braid applies in one step only when an agent gets 200,000 tokens or more. Parts 1-3 finished 19:03-19:23Z.
  • Paused for a stack change the owner asked for: stopped at 19:27:31Z during apply part 4; stack 128 stopped at 19:28Z. The change is an 8-bit cache and a 262,144 window, so that each agent gets about 220k and apply runs as one session.
    • Braid can't rewind run 164 onto the bigger window. A run keeps the window it started with, which caps its agents at 110,592 whatever the profile says later. Its apply split is decided once, and a redo replays the same six parts.
  • Restarted fresh on the bigger window (the owner: "Start fresh, 220k"). Run 164 stays stopped as a record, with its repo and branch.
    • Stack 128 now has a 262,144-token window with a 6-bit cache (cache_mode "6,6"). An 8-bit cache at 262,144 didn't fit on the cards: the loader stopped with "Insufficient VRAM in split for model and cache".
    • The exl3 profile now has the 262,144 window from Connect and 220,000 tokens per agent (alarm at 190,000), enough for apply auto to take one step.
    • Run 165 on cmoriarty/split-the-bill-flashnext-262k, same brief and settings, started 20:00:55Z.
  • Run 165, propose done 20:13Z: explore (9 turns, peak 12k) and propose (24 turns, peak 39k) took 12 min.
  • Run 165, proposal review done 20:43Z. Again two reviewers at a time. Spec found 5 to fix and 3 notes, scope 1 blocking, 1 to fix and 1 note, test 4 to fix and 4 notes, risk 5 to fix and 2 notes. One revise pass, then a second review round for the blocking finding.
  • Run 165, apply done 21:21Z in one step (openspec.apply[all]), as the 220,000 per agent intended: 38.1 min, all 19 tasks ticked. Braid then wrote spec tests for one scenario group (settlement, 4.3 min).
  • Run 165 stopped at 21:34:45Z, at the owner's call, 9 min into code review. Stack 128 stopped at 21:35Z. In the owner's words: "it seems pretty clear that lowering that some combination of 3-bit quant and lowering kv to 6-bit is causing this version of flash-next to perform worse than usual. everything seems to be taking too long. I keep flash-next around because its a bigger model, and generally can solve harder problems with more reliability than qwen3.8-27b. But right now, it seems like 27b on vLLM is the better solution."
  • Run D's phases, against run 163:
Run 165: Flash-Next, 6-bit cache, 262,144 Run 164: Flash-Next, FP16 cache, 131,072 Run 163: Qwen3.8-27B on vLLM
Explore and propose 11.6 min 8.2 min 9.6 min
Proposal review, with revise 30.6 min, two rounds 26.9 min 15.3 min
Review findings 1 blocking, 15 to fix, 10 notes 0 blocking, 9 to fix, 8 notes 0 blocking, 9 to fix, 4 notes
Apply 38.1 min, one step six parts, 1-3 in 20.6 min 43.6 min, one step
  • Runs 164 and 165 used the same 3-bit file on the same engine. On the stack, the cache went from FP16 to 6-bit and the window from 131,072 to 262,144; in Braid, each agent got 220,000 instead of 110,592. There is one run of each, so how much of the difference comes from the cache isn't measured.
  • Both Flash-Next runs ran the four reviewers two at a time, with two slots. Run 163 ran all four at once, with six.

Run E, DeepSeek V4.1 on llama.cpp: cancelled at the owner's call. In the owner's words: "no need for any extensive testing with deepseek, once you prove the stack works, you can stop it."

  • Its stack got a 221,184-token window and passed a short live check (thinking, an image and a tool call), then was stopped.
  • It decodes about 8 tokens a second (7.9 at 3k and at 20k). The owner won't use it as the workhorse here until it decodes above 20.

A short Braid check of 8-bit weights on vLLM: skipped at the owner's call.

  • The owner had three fidelity levers measured for the 27B on vLLM: 8-bit weights, a 16-bit cache, and dropping the context stretch (YaRN). A Braid run on 8-bit weights, with production's cache and context stretch, was to explore, propose and review the proposal, then stop before apply, for comparison with run 163.
  • The owner's rule: skip the run if that setting also did worse at depth, as the first 8-bit setup had ("Yes, skip it then"). It did, by perplexity (lower is better):
    • On the last 8,192 tokens of a 100,000-token stretch of run 163's own session text: 3.519 with 8-bit weights, against production's 2.395, 47% worse.
    • On 4k-token chunks the two were close: 2.897 against 2.867.
    • All three 8-bit setups did 33-49% worse at depth than the 4-bit setups with the same cache and context settings.
  • Production's setting stays as it is. Braid's vLLM profile is back in use, with effort auto.
  • Since overtaken (2026-11-01): an independent CPU reference scored a 100k-token Braid transcript at a perplexity next to the 8-bit setting's and well above the 4-bit's. The 4-bit's lower perplexity was drift from the original model, not better fidelity. The owner adopted 8-bit weights, a 16-bit cache and no stretch on stack 126; run 166, in the next comment, is the full Braid run on that setting.
**Findings from the second round: vLLM (run 163) and Flash-Next (runs 164 and 165).** Run 163 has finished. Run D was stopped, run E cancelled and a follow-up check of 8-bit weights on vLLM skipped, all at the owner's call (below). **Run C, vLLM:** run 163 on `cmoriarty/split-the-bill-vllm`, started 14:20Z. - Same brief and settings: Thorough New Project (one-shot), auto-approve on, effort high. - Braid's main profile, "vLLM · qwen3.8-27b". Its effort was set to high for the run and back to auto after. The stack thinks at xhigh by default, as run 161's did. - That profile keeps its own run settings: 400k per agent (alarm at 370k) and up to 8 subagents. Run 161's fastllm profile had 200k per agent and 5 subagents. - vLLM v0.32.0, six sequences at once, a 524k window. - **Propose done 14:30Z:** explore 1.7 min (7 turns), propose 7.8 min (19 turns, peak 48k). - **Proposal review done 14:45Z:** four reviewers in parallel. - Spec found 1 to fix, scope 6 to fix and 1 note, test 2 to fix and 3 notes. Risk found nothing. - One revise pass (3.8 min) marked all nine revised. - **Apply done 15:29Z:** one step, 43.6 min, 196 turns, peak 175k. - It tagged tests for 23 of the 42 scenarios and left the other 19 to the browser check, so the extra test-writing step was skipped. - Its browser subagents took the run's first screenshots at 15:06Z. - **Code review done 15:34Z:** 5.2 min, 9 turns. It fixed two minor findings and left two notes. - **Tests passed on the first round, 15:34Z.** - **Browser check passed on the first round, 16:04Z, on a second attempt:** 30 min, 157 turns, 24 of 24 criteria, so no fix round. - Attempt 1 failed at 15:58Z (below). Braid retried the step, and attempt 2 took 5.7 min. - **Run C succeeded at 16:09Z, after 1 h 49 min.** Auto-approve merged its pull request (cmoriarty/split-the-bill-vllm#1) and archived the change. **Findings** - **vLLM kept its prompt cache with screenshots in the sessions.** Over the run, 94% of prompt tokens came from the cache: 94% before the first screenshot and 95% after. - Apply cached 98%, the browser check 93%. The median wait for the first token was 0.6 s (p90 4.0 s), 12 min in all. - Run 161 on fastllm cached 54% over the run. In its apply step, the rate fell from 99% before the first browser check to 52% after. - These numbers come from vLLM's own prefix-cache counters. vLLM reports cached tokens per request only when started with `--enable-prompt-tokens-details`, so Braid's cached column is 0. - Its prompt-token count matches Braid's total to within 0.05%, so nothing else used the server during the run. - **The first browser check attempt hit vLLM's limit of 16 images per prompt.** - The judge read the 8 screenshots its four browser subagents had saved. - Braid refused a fifth browser check. The judge then wrote a Playwright script of its own, ran it from the shell and read 10 more screenshots. - The request carrying the 17th and 18th images got HTTP 400: "At most 16 image(s) may be provided in one prompt". The session errored, and the judge file was left unfinished (`play` missing, `finished` not true). - Braid retried the step, and attempt 2 passed. Attempt 1 had run for 25 min. - The cap is the stack's own setting, `--limit-mm-per-prompt` with 16 images. - The stack has since been redeployed with a cap of 999 images. - **The judge worked around its cap of four browser checks.** Braid counts subagents, so the judge drove the browser itself from a script. It first tried `find /` to locate a browser, which Braid refused. In run 161, the judge settled what was left by reading the code. - **The skip note for the test-writing step again reads "every scenario has a tagged test (23 of 42)".** As in run 161, it doesn't say the other 19 were left to the browser check. - **The summary step opens with a prompt no backend has cached.** Its first request waited 85 s on fastllm (90k tokens), 170 s on Ornith (93k) and 158 s on vLLM (179k). - **The app requires Node 26, by design.** Its `package.json` asks for Node 26 or newer. Braid gives every new Node project an `.npmrc` with `engine-strict=true` (#273), so `npm ci` refuses Node 22. - The scaffold's `.nvmrc` names Node 26 for every run, so all three apps were checked on Node 26: each installs, passes `npm test` and builds. - **Three Triton kernels compiled during the run's first requests (14:20–14:21Z).** The stack's warmup doesn't cover them; it cost a few seconds, once. **Run D, Flash-Next on ExLlamaV3:** first run 164 on `cmoriarty/split-the-bill-flashnext`, started 18:27Z and stopped at 19:27Z; then run 165 on a bigger window, 20:00-21:34Z (below). - The stack: ExLlamaV3 1.6.5 with tabbyAPI, the 3.05 bpw quant, a 131,072-token window with an FP16 cache shared by two requests at once (`max_batch_size` 2). Loop detection is off, so a loop ends as "length". - Braid's profile "exl3 qwen38-flash-next", effort high, so the stack's xhigh default applies. - The profile had no run settings of its own, so it would have taken whatever profile was in use. I saved the main profile's: up to 8 subagents, mediation 4, apply auto, auto-approve. - Connect read two parallel sessions from tabby's slot count. - 110,592 tokens per agent (alarm at 82,944): what the window leaves after the 20,480-token output reserve. - **Propose done 18:36Z:** explore and propose took 8.5 min. - **Proposal review done 19:03Z.** With two sessions, Braid ran the four reviewers two at a time. They found 9 to fix (spec 4, scope 1, test 2, risk 2) and 8 notes, and one revise pass followed. - **Apply split into six parts, run one after another.** Under apply auto, Braid applies in one step only when an agent gets 200,000 tokens or more. Parts 1-3 finished 19:03-19:23Z. - **Paused for a stack change the owner asked for:** stopped at 19:27:31Z during apply part 4; stack 128 stopped at 19:28Z. The change is an 8-bit cache and a 262,144 window, so that each agent gets about 220k and apply runs as one session. - Braid can't rewind run 164 onto the bigger window. A run keeps the window it started with, which caps its agents at 110,592 whatever the profile says later. Its apply split is decided once, and a redo replays the same six parts. - **Restarted fresh on the bigger window** (the owner: "Start fresh, 220k"). Run 164 stays stopped as a record, with its repo and branch. - Stack 128 now has a 262,144-token window with a 6-bit cache (`cache_mode` "6,6"). An 8-bit cache at 262,144 didn't fit on the cards: the loader stopped with "Insufficient VRAM in split for model and cache". - The exl3 profile now has the 262,144 window from Connect and 220,000 tokens per agent (alarm at 190,000), enough for apply auto to take one step. - **Run 165** on `cmoriarty/split-the-bill-flashnext-262k`, same brief and settings, started 20:00:55Z. - **Run 165, propose done 20:13Z:** explore (9 turns, peak 12k) and propose (24 turns, peak 39k) took 12 min. - **Run 165, proposal review done 20:43Z.** Again two reviewers at a time. Spec found 5 to fix and 3 notes, scope 1 blocking, 1 to fix and 1 note, test 4 to fix and 4 notes, risk 5 to fix and 2 notes. One revise pass, then a second review round for the blocking finding. - **Run 165, apply done 21:21Z in one step** (`openspec.apply[all]`), as the 220,000 per agent intended: 38.1 min, all 19 tasks ticked. Braid then wrote spec tests for one scenario group (settlement, 4.3 min). - **Run 165 stopped at 21:34:45Z, at the owner's call, 9 min into code review.** Stack 128 stopped at 21:35Z. In the owner's words: "it seems pretty clear that lowering that some combination of 3-bit quant and lowering kv to 6-bit is causing this version of flash-next to perform worse than usual. everything seems to be taking too long. I keep flash-next around because its a bigger model, and generally can solve harder problems with more reliability than qwen3.8-27b. But right now, it seems like 27b on vLLM is the better solution." - **Run D's phases, against run 163:** | | Run 165: Flash-Next, 6-bit cache, 262,144 | Run 164: Flash-Next, FP16 cache, 131,072 | Run 163: Qwen3.8-27B on vLLM | |---|---|---|---| | Explore and propose | 11.6 min | 8.2 min | 9.6 min | | Proposal review, with revise | 30.6 min, two rounds | 26.9 min | 15.3 min | | Review findings | 1 blocking, 15 to fix, 10 notes | 0 blocking, 9 to fix, 8 notes | 0 blocking, 9 to fix, 4 notes | | Apply | 38.1 min, one step | six parts, 1-3 in 20.6 min | 43.6 min, one step | - Runs 164 and 165 used the same 3-bit file on the same engine. On the stack, the cache went from FP16 to 6-bit and the window from 131,072 to 262,144; in Braid, each agent got 220,000 instead of 110,592. There is one run of each, so how much of the difference comes from the cache isn't measured. - Both Flash-Next runs ran the four reviewers two at a time, with two slots. Run 163 ran all four at once, with six. **Run E, DeepSeek V4.1 on llama.cpp: cancelled at the owner's call.** In the owner's words: "no need for any extensive testing with deepseek, once you prove the stack works, you can stop it." - Its stack got a 221,184-token window and passed a short live check (thinking, an image and a tool call), then was stopped. - It decodes about 8 tokens a second (7.9 at 3k and at 20k). The owner won't use it as the workhorse here until it decodes above 20. **A short Braid check of 8-bit weights on vLLM: skipped at the owner's call.** - The owner had three fidelity levers measured for the 27B on vLLM: 8-bit weights, a 16-bit cache, and dropping the context stretch (YaRN). A Braid run on 8-bit weights, with production's cache and context stretch, was to explore, propose and review the proposal, then stop before apply, for comparison with run 163. - The owner's rule: skip the run if that setting also did worse at depth, as the first 8-bit setup had ("Yes, skip it then"). It did, by perplexity (lower is better): - On the last 8,192 tokens of a 100,000-token stretch of run 163's own session text: 3.519 with 8-bit weights, against production's 2.395, 47% worse. - On 4k-token chunks the two were close: 2.897 against 2.867. - All three 8-bit setups did 33-49% worse at depth than the 4-bit setups with the same cache and context settings. - Production's setting stays as it is. Braid's vLLM profile is back in use, with effort auto. - **Since overtaken (2026-11-01):** an independent CPU reference scored a 100k-token Braid transcript at a perplexity next to the 8-bit setting's and well above the 4-bit's. The 4-bit's lower perplexity was drift from the original model, not better fidelity. The owner adopted 8-bit weights, a 16-bit cache and no stretch on stack 126; run 166, in the next comment, is the full Braid run on that setting.
Author
Owner

Run 166: Qwen3.8-27B on vLLM with 8-bit weights, a 16-bit cache and no context stretch, against run 163

Run 166 succeeded in 93 min on cmoriarty/split-the-bill-vllm-w8 (05:30–07:04Z). It had the same brief and settings as run 163, which took 109 min.

  • No attempt failed, there was no fix round, and the browser check passed on its first attempt.
  • Auto-approve merged its pull request (cmoriarty/split-the-bill-vllm-w8#1).
  • On Node 26 its app installs, passes npm test (21 tests) and builds. It passes the money check, as run 163's app does.
  • Decode was 28% slower than in run 163. Prefill speed, the prompt-cache rate (94%) and preemptions (none) were unchanged.

What changed on stack 126 (the owner's call, 2026-11-01)

  • Weights: 8-bit (INT8 W8A16, group 128). Run 163 had the 4-bit AWQ.
  • Cache: 16-bit. Run 163 had fp8.
  • Window: the native 262,144. Run 163 had 524,288, stretched with YaRN.
  • Unchanged: vLLM v0.32.0, six sequences at once, thinking at xhigh by default.
  • The KV pool is 352,856 tokens, room for 1.35 full windows. Run 163's held 991,688.
    • That is less than the 400k expected. The stack sets the pool 0.46 GiB under vLLM's own fit (about 383k), to leave room for a fresh compile and for images.

Braid's settings

  • Connect read the 262,144 window for the main profile.
  • Each agent gets 230,144 (alarm at 200,144). That is the window less run 163's 32,000-token output reservation.
    • Braid's 20,480 default would have given 241,664. But with thinking at xhigh, two of run 163's turns wrote about 28,000 tokens each, and keeping the reservation keeps the runs comparable.
    • 230,144 still clears the 200,000 apply auto needs to apply in one step, so apply was one step, as in run 163.
  • Effort was high for the run, as in run 163. It is back to auto.

Side by side

Run 166: vLLM, 8-bit weights, 16-bit cache Run 163: vLLM, 4-bit weights, fp8 cache Run 165: Flash-Next, 6-bit cache Run 164: Flash-Next, FP16 cache
Explore and propose 7.9 min, 22 turns 9.6 min, 26 turns 11.6 min, 33 turns 8.2 min, 24 turns
Proposal review, with revise 14.8 min, 31 turns 15.3 min, 31 turns 30.6 min, 52 turns, two rounds 26.9 min, 39 turns
Review findings: blocking / to fix / notes 0 / 6 / 9 0 / 9 / 4 1 / 15 / 10 0 / 9 / 8
Apply 33.1 min, 135 turns, peak 123k 43.7 min, 196 turns, peak 175k 42.4 min, 129 turns, peak 129k stopped in part 4 of 6
Code review 7.0 min, 15 turns: fixed 3, left 0 5.2 min, 9 turns: fixed 2, left 2 notes stopped after 9.3 min -
Tests passed, round 1 passed, round 1 - -
Browser check 26.4 min, 145 turns: passed on attempt 1 30.4 min, 157 turns: passed on attempt 2 - -
Fix rounds 0 0 - -
Summary and merge 3.6 min 4.9 min - -
Whole run 92.9 min, succeeded 109.1 min, succeeded 93.8 min, stopped 59.7 min, stopped
Turns 350 421 233 108
Failed attempts; mediations 0; 0 1 (browser check); 1 0; 0 0; 0
Output tokens 243k 344k 172k 105k
Decode t/s, all / decoding alone 38.5 / 40.6 53.5 / 54.7 29.4 / 38.0 26.8 / 37.1
Prefill t/s 1,235 1,198 396 305
Prompt cache hits 94% 94% 96% 95%
First-token wait 9.2 min 13.3 min 19.1 min 12.1 min
Preemptions 0 0 - -
KV pool, peak use 130k of 352,856 (37%) 188k of 991,688 (19%) - -
npm ci / npm test / npm run build, Node 26 pass / 21 pass / pass pass / 22 pass / pass - -
Money check pass pass - -
  • Runs 164 and 165 show only the phases they reached before they were stopped. The stop cut off one step in each; that step is not counted as a failed attempt.
  • Run 166's phases ended at these times: propose 05:38Z, proposal review 05:53Z, apply 06:26Z, code review and tests 06:33Z, browser check 07:00Z, merge 07:03Z.
  • Turns are model requests.
  • Decode is a request's output tokens over the time from its first token to its end. It counts requests of 50 tokens or more.
  • Cache hits and preemptions for runs 163 and 166 come from vLLM's own counters, sampled every 10 s. For runs 164 and 165, cache hits are as tabby reported them per request.
  • Run 166's prompt and output tokens on vLLM match Braid's totals to the token, so nothing else used the server during the run.

Money check, run with each app's own money module on Node 26:

  • $100 split three ways: $33.34, $33.33, $33.33 in both apps.
  • Ana pays $100 for all three: Ben and Cy each pay Ana $33.33, and every balance ends at zero.
  • Ben pays $100 for all, Cy pays $45.50 for Ben and Cy, Ana pays $10 for all: Ana pays Ben $26.68 and Cy pays Ben $13.91, and every balance ends at zero.
  • Both apps give the same balances and the same payments.

Findings

  • Run 166 was quicker than run 163 despite slower decode, because it wrote 29% fewer output tokens: 243k against 344k.
    • Apply wrote 70k tokens in 135 turns; run 163's apply wrote 117k in 196.
    • Run 166's largest single reply was 18,292 tokens; run 163's was 28,183. No request in either run hit the output limit.
    • There is one run of each, so how much of the difference comes from the weights isn't measured.
  • The browser check passed on its first attempt. Run 163's first attempt lost 25 min to vLLM's 16-image cap, which has since been raised to 999. Run 166's busiest session held 9 images; run 163's held 18.
  • Decode fell 28%, close to the stack's own measurement of about 32% on short prompts. Prefill held at about 1,200 t/s.
  • The pool had room to spare.
    • At most four requests ran at once, the four proposal reviewers. No request waited for space: queue time totalled 0.01 s over the run.
    • Use peaked at 130k tokens, 37% of the pool. The biggest sessions, apply (peak 123k) and the summary (129k), ran alone.
    • Two agents near the 230,144 limit at the same time would need about 460k, more than the pool holds. This run never came close.
  • The judge left 3 of 19 criteria unclear, yet Braid's row reads "pass · 19 of 19". Braid counts an unclear criterion as not failed.
    • The three: trying to remove a person who has expenses (it should be blocked), and Reset with Cancel and with OK.
    • The judge reached Braid's cap of four browser checks before it got to them, and marked them unclear. Unlike run 163's judge, it didn't work around the cap.
    • Apply's own browser checks had already seen all three behave as specified (06:14–06:25Z). The pull request lists them under "Not done", for the reviewer to click through.
  • The code review fixed three minor issues and left none, so its row reads "nothing found". Braid counts only findings left open.
    • Two fixes are to the loader for saved data, which accepted two corruptions: an expense that lists one sharer twice, so its shares add up to more than its total, and two people whose names differ only in case.
    • The third escapes ids where the app puts them in HTML attributes.
    • Correction to the comment above: run 163's code review also changed code. It fixed two of its four findings, saved data with duplicate or dangling person ids and a stylesheet loaded twice, and left two as notes.
  • The test-writing step was skipped again, with "every scenario has a tagged test (19 of 34)". The other 15 scenarios have no tagged test, and the note doesn't say so, as in runs 161 and 163.
  • The summary step's first request waited 97 s, for a 126k prompt that no earlier request had cached. In run 163 it waited 158 s for 179k.

Afterwards, stack 126 still serves this setting. Braid's main profile keeps the 262,144 window and 230,144 per agent, and its effort is back to auto.

**Run 166: Qwen3.8-27B on vLLM with 8-bit weights, a 16-bit cache and no context stretch, against run 163** Run 166 succeeded in 93 min on `cmoriarty/split-the-bill-vllm-w8` (05:30–07:04Z). It had the same brief and settings as run 163, which took 109 min. - No attempt failed, there was no fix round, and the browser check passed on its first attempt. - Auto-approve merged its pull request (cmoriarty/split-the-bill-vllm-w8#1). - On Node 26 its app installs, passes `npm test` (21 tests) and builds. It passes the money check, as run 163's app does. - Decode was 28% slower than in run 163. Prefill speed, the prompt-cache rate (94%) and preemptions (none) were unchanged. **What changed on stack 126** (the owner's call, 2026-11-01) - Weights: 8-bit (INT8 W8A16, group 128). Run 163 had the 4-bit AWQ. - Cache: 16-bit. Run 163 had fp8. - Window: the native 262,144. Run 163 had 524,288, stretched with YaRN. - Unchanged: vLLM v0.32.0, six sequences at once, thinking at xhigh by default. - The KV pool is 352,856 tokens, room for 1.35 full windows. Run 163's held 991,688. - That is less than the 400k expected. The stack sets the pool 0.46 GiB under vLLM's own fit (about 383k), to leave room for a fresh compile and for images. **Braid's settings** - Connect read the 262,144 window for the main profile. - Each agent gets 230,144 (alarm at 200,144). That is the window less run 163's 32,000-token output reservation. - Braid's 20,480 default would have given 241,664. But with thinking at xhigh, two of run 163's turns wrote about 28,000 tokens each, and keeping the reservation keeps the runs comparable. - 230,144 still clears the 200,000 apply auto needs to apply in one step, so apply was one step, as in run 163. - Effort was high for the run, as in run 163. It is back to auto. **Side by side** | | Run 166: vLLM, 8-bit weights, 16-bit cache | Run 163: vLLM, 4-bit weights, fp8 cache | Run 165: Flash-Next, 6-bit cache | Run 164: Flash-Next, FP16 cache | |---|---|---|---|---| | Explore and propose | 7.9 min, 22 turns | 9.6 min, 26 turns | 11.6 min, 33 turns | 8.2 min, 24 turns | | Proposal review, with revise | 14.8 min, 31 turns | 15.3 min, 31 turns | 30.6 min, 52 turns, two rounds | 26.9 min, 39 turns | | Review findings: blocking / to fix / notes | 0 / 6 / 9 | 0 / 9 / 4 | 1 / 15 / 10 | 0 / 9 / 8 | | Apply | 33.1 min, 135 turns, peak 123k | 43.7 min, 196 turns, peak 175k | 42.4 min, 129 turns, peak 129k | stopped in part 4 of 6 | | Code review | 7.0 min, 15 turns: fixed 3, left 0 | 5.2 min, 9 turns: fixed 2, left 2 notes | stopped after 9.3 min | - | | Tests | passed, round 1 | passed, round 1 | - | - | | Browser check | 26.4 min, 145 turns: passed on attempt 1 | 30.4 min, 157 turns: passed on attempt 2 | - | - | | Fix rounds | 0 | 0 | - | - | | Summary and merge | 3.6 min | 4.9 min | - | - | | **Whole run** | **92.9 min, succeeded** | **109.1 min, succeeded** | 93.8 min, stopped | 59.7 min, stopped | | Turns | 350 | 421 | 233 | 108 | | Failed attempts; mediations | 0; 0 | 1 (browser check); 1 | 0; 0 | 0; 0 | | Output tokens | 243k | 344k | 172k | 105k | | Decode t/s, all / decoding alone | 38.5 / 40.6 | 53.5 / 54.7 | 29.4 / 38.0 | 26.8 / 37.1 | | Prefill t/s | 1,235 | 1,198 | 396 | 305 | | Prompt cache hits | 94% | 94% | 96% | 95% | | First-token wait | 9.2 min | 13.3 min | 19.1 min | 12.1 min | | Preemptions | 0 | 0 | - | - | | KV pool, peak use | 130k of 352,856 (37%) | 188k of 991,688 (19%) | - | - | | `npm ci` / `npm test` / `npm run build`, Node 26 | pass / 21 pass / pass | pass / 22 pass / pass | - | - | | Money check | pass | pass | - | - | - Runs 164 and 165 show only the phases they reached before they were stopped. The stop cut off one step in each; that step is not counted as a failed attempt. - Run 166's phases ended at these times: propose 05:38Z, proposal review 05:53Z, apply 06:26Z, code review and tests 06:33Z, browser check 07:00Z, merge 07:03Z. - **Turns** are model requests. - **Decode** is a request's output tokens over the time from its first token to its end. It counts requests of 50 tokens or more. - **Cache hits and preemptions** for runs 163 and 166 come from vLLM's own counters, sampled every 10 s. For runs 164 and 165, cache hits are as tabby reported them per request. - Run 166's prompt and output tokens on vLLM match Braid's totals to the token, so nothing else used the server during the run. **Money check**, run with each app's own money module on Node 26: - **$100 split three ways:** $33.34, $33.33, $33.33 in both apps. - **Ana pays $100 for all three:** Ben and Cy each pay Ana $33.33, and every balance ends at zero. - **Ben pays $100 for all, Cy pays $45.50 for Ben and Cy, Ana pays $10 for all:** Ana pays Ben $26.68 and Cy pays Ben $13.91, and every balance ends at zero. - Both apps give the same balances and the same payments. **Findings** - **Run 166 was quicker than run 163 despite slower decode, because it wrote 29% fewer output tokens:** 243k against 344k. - Apply wrote 70k tokens in 135 turns; run 163's apply wrote 117k in 196. - Run 166's largest single reply was 18,292 tokens; run 163's was 28,183. No request in either run hit the output limit. - There is one run of each, so how much of the difference comes from the weights isn't measured. - **The browser check passed on its first attempt.** Run 163's first attempt lost 25 min to vLLM's 16-image cap, which has since been raised to 999. Run 166's busiest session held 9 images; run 163's held 18. - **Decode fell 28%, close to the stack's own measurement** of about 32% on short prompts. Prefill held at about 1,200 t/s. - **The pool had room to spare.** - At most four requests ran at once, the four proposal reviewers. No request waited for space: queue time totalled 0.01 s over the run. - Use peaked at 130k tokens, 37% of the pool. The biggest sessions, apply (peak 123k) and the summary (129k), ran alone. - Two agents near the 230,144 limit at the same time would need about 460k, more than the pool holds. This run never came close. - **The judge left 3 of 19 criteria unclear, yet Braid's row reads "pass · 19 of 19".** Braid counts an unclear criterion as not failed. - The three: trying to remove a person who has expenses (it should be blocked), and Reset with Cancel and with OK. - The judge reached Braid's cap of four browser checks before it got to them, and marked them unclear. Unlike run 163's judge, it didn't work around the cap. - Apply's own browser checks had already seen all three behave as specified (06:14–06:25Z). The pull request lists them under "Not done", for the reviewer to click through. - **The code review fixed three minor issues and left none, so its row reads "nothing found".** Braid counts only findings left open. - Two fixes are to the loader for saved data, which accepted two corruptions: an expense that lists one sharer twice, so its shares add up to more than its total, and two people whose names differ only in case. - The third escapes ids where the app puts them in HTML attributes. - **Correction to the comment above:** run 163's code review also changed code. It fixed two of its four findings, saved data with duplicate or dangling person ids and a stylesheet loaded twice, and left two as notes. - **The test-writing step was skipped again, with "every scenario has a tagged test (19 of 34)".** The other 15 scenarios have no tagged test, and the note doesn't say so, as in runs 161 and 163. - **The summary step's first request waited 97 s, for a 126k prompt that no earlier request had cached.** In run 163 it waited 158 s for 179k. Afterwards, stack 126 still serves this setting. Braid's main profile keeps the 262,144 window and 230,144 per agent, and its effort is back to auto.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#350
No description provided.