qwen38-flash-next

Qwen3.8-Flash-Next (125B-A6B MoE, Qwen4-preview qwen4_exp arch, released 2026-08-26) on trogdor (4× Quadro RTX 5000 16GB, Turing SM75, 376GB DDR4) via llama.cpp — an experiment stack. The incumbent (qwen38-vllm-flashinfer, Portainer stack 105, Qwen3.8-27B) remains the default daily driver; both use port 8090 and all four cards, so exactly one runs at a time.

Current numbers are at the bottom of this file — after the 2026-09-02 engine update + MTP work, decode is 25.0 t/s @20k (was 16.7) and the g2 gate passes. Sections between here and there are kept as the build log and are superseded where they conflict.

Full research report (2026-08-27, 7-agent deep-research run): https://claude.ai/code/artifact/ca94187b-b7df-46c4-8cb4-64eaf3fb822c

Why llama.cpp and not vLLM

Source-verified, three independent kills — any one is sufficient:

1. vLLM's QSA backend hard-blocks Turing. qwen4_exp/nvidia/qsa.py (vLLM PR #53896): bf16-only dtype, requires FlashAttention (SM80+), forbids KV-cache quantization. No fp16, FlashInfer, or dense fallback exists.
2. Nothing fits 64GB VRAM. Routed experts alone = 120.8B params ≈ 62–64GB at INT4; the best W4A16 checkpoint needs ~66GB GPU-resident after offloading the 51B n-gram table; the smallest GGUF anywhere is 72.5GB (1-bit).
3. Requires vLLM ≥ 0.29.0, which does not exist as a tag.

The one real path: llama.cpp PR #27742 (unmerged, approved — SHA-pinned in the Dockerfile), unsloth UD-IQ4_XS GGUF (93.7GB), routed experts of ~12 layers in CPU RAM (--n-cpu-moe 12), the 51B PLE n-gram table auto-placed in host RAM (~28GB). Expected 15–25 t/s decode vs the incumbent's 46–49 — zero Turing field reports exist for this model; first boot is the experiment.

Contents

File	What
Dockerfile	llama.cpp @ PR #28243 (master + MTP), CUDA arch 75. Local image — build on trogdor, never push.
docker-compose.yml	The stack: port 8090, pooled 262k KV (--kv-unified, 3 slots), f16 KV, reasoning_effort pinned medium. Every flag and trap documented in the header.

Deploy

# 1. Model (~97 GB → /mnt/models) — weights, vision proj, and MTP draft head
ssh trogdor 'hf download unsloth/Qwen3.8-Flash-Next-GGUF \
  --include "UD-IQ4_XS/*" --include "mmproj-F16.gguf" --include "MTP/*" \
  --local-dir /mnt/models/Qwen3.8-Flash-Next-GGUF'

# 2. Image (built on trogdor's daemon; stack keeps PullImage OFF)
git clone ssh://git@forgejo.underthere.xyz:222/cmoriarty/qwen38-flash-next.git
DOCKER_HOST=ssh://cmoriarty@trogdor docker build \
  -t llama-server-qwen4exp:mtp-pr28243 qwen38-flash-next/

# 3. Portainer (endpoint 39 "trogdor"): git stack on this repo,
#    compose path docker-compose.yml, ref refs/heads/main, PullImage OFF.

# 4. Swap (mutually exclusive with stacks 105/103/102/98/97 — all four
#    cards, port 8090): stop stack 105 → start this stack.
#    Revert: stop this → start 105 (its warmup sidecar re-warms itself).

Day-1 gates (before pointing opencode at it)

Gate	Test	Keep	Park
g1	Boots, coherent text (settles unproven SM75 QSA kernels)	clean stream	CUDA abort / garbage
g2	llama-bench -p 20480 -n 256 decode @20k	≥ 25 t/s	< 15 t/s (same day)
g3	pp20480 cold prefill	≤ ~90 s	slower = agent UX dead
g4	5-prompt output diff vs hosted Qwen3.8-Flash API	matches	silent numeric garbage

Incumbent baselines to beat/justify: 46–49 t/s @20k, 13s cold 21k prefill, 0.17–0.5s cached-turn TTFT, AA Intelligence 52 (Flash-Next: 56, with ~1.25× more output tokens).

opencode

The global ~/.config/opencode/opencode.jsonc carries this model beside the untouched 27B entry (exactly one resolves at a time — same port, mutually exclusive stacks):

"qwen3.8-flash-next": {
  "name": "Qwen3.8-Flash-Next · 125B-A6B · vision · 262k pooled",
  "tools": true,
  "reasoning": true,
  "attachment": true,
  "modalities": { "input": ["image", "text"], "output": ["text"] },
  "limit": { "context": 196608, "output": 32768 } // pool 262k, leave ~65k for subagents
}

Notes: llama.cpp ignores the OpenAI reasoning_effort body field — the server pins medium via --chat-template-kwargs (xhigh ruminates on this model). The GGUF-embedded template raises on a system message that is not strictly first — the same trap opencode hit on stock Qwen3.8-27B; if opencode 400s, extract and fix the template, mount via --chat-template-file.

Re-evaluation triggers

• PR #27742 merges into a tagged release → replace the local image with ghcr.io/ggml-org/llama.cpp:server-cuda-bXXXXX, delete the Dockerfile.
• llama.cpp MTP for qwen4exp lands → re-bench (draft is the single small 6B-active MTP block, so the 27B's "MTP inverts at context" failure may not repeat; expected ~1.5–2× decode).
• Any vLLM fp16 QSA path appears (none filed upstream as of 2026-08-27).
• Independent coding benches publish (AA Coding Index, Aider polyglot); local A/B harness: trogdor:~/aider-bench/ pinned-40 run.
• An official smaller/denser Qwen4-family variant ships.

Measured (2026-08-27, first deploy, stack 109)

Metric	Flash-Next	Qwen3.8-27B (stack 105)
Cold 21k prefill	41.1 s (~510 t/s)	~13 s
Decode @20k ctx	16.9 t/s	46–49 t/s
Cached-turn TTFT	2.1 s	0.17–0.5 s
Decode short ctx	20.8 t/s	54 t/s
VRAM (GB/card)	15.4 / 12.5 / 12.5 / 13.7	~15.2 × 4

Gates: g1 pass (clean boot, coherent output — SM75 QSA kernels work), g3 pass (41 s ≤ 90 s), g2 marginal (16.9 in the 15–25 judgment band, below the ≥25 keep bar). Effective agent-turn wall-clock ≈ 3.4× the incumbent after the ~1.25× verbosity tax. Re-bench when llama.cpp MTP lands (expected 1.5–2×).

Boot lessons (already encoded in the compose): --n-cpu-moe N strips the FIRST N layers only → catastrophic imbalance under -sm layer; use the even-spread -ot regex. --mlock is deprecated/broken on this branch; --load-mode mlock works. Quote the -ot regex or compose's command lexer rejects the file.

Retune (2026-08-27 evening): pooled 262k context

-c 262144 -np 3 --kv-unified replaced the 3×64k fixed lanes: one shared 262k KV pool, any single slot may grow to the full 262k — the primary opencode session runs large while worker/explore stay small. blk 1's experts joined the CPU set (17 layers) to make VRAM room on card 0. Re-bench: 16.6 t/s @20k, 20.7 short, cached TTFT 2.3s — no regression vs the 64k-lane config. VRAM 15.5/14.3/14.3/15.5 (~0.5GB headroom on cards 0/3; if a runtime compute-buffer spike OOMs, first lever is -c 229376). opencode limit.context = 196608 so the primary compacts while ~65k pool remains for subagents.

Retune 2 (2026-08-27): 327k pool, native-max primary

-c 327680 --kv-unified — pool 25% over native ON PURPOSE: -c sizes the shared pool, not a sequence; llama-server itself caps each slot at the 262,144 training context. Result: the primary session can reach native max while ~65k of pool stays free for LIVE background worker/explore streams (in-flight sequences cannot be evicted; only idle finished caches can — verified). blk 2 + blk 43 experts joined the CPU set (19 layers) to fund the KV on cards 0/3. Re-bench: 16.7 t/s @20k, 20.5 short, cached TTFT 2.3s — the extra offload cost nothing measurable. VRAM 14.4/15.2/15.2/15.3. opencode limit.context: 262144.

Grafana dashboard

grafana/build_dashboard.py generates the llama.cpp dashboard (uid llamacpp-flash-next) on the metrics host: http://192.168.1.107:3000/d/llamacpp-flash-next — no Prometheus changes were needed: the existing vllm scrape job reads trogdor:8090/metrics whichever engine holds the port. Re-upload after edits:

cd grafana && python3 build_dashboard.py > llamacpp-dashboard.json
security find-generic-password -s grafana-api -w | { read T; \
  curl -s -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \
       -d @llamacpp-dashboard.json http://192.168.1.107:3000/api/dashboards/db; }

llama.cpp exports no latency histograms (no TTFT panels possible — the bench script remains the latency tool) and no KV-occupancy gauge (watch requests_deferred as the pool-pressure proxy). Add the llamacpp:spec_decode_* acceptance panels when MTP lands.

Re-benchmark 2026-09-02: +50% from engine updates + MTP

Three changes, measured cumulatively on the same 21k agent-turn bench:

	orig (PR-head)	b10769	+ lazy-mode off	+ MTP
decode, short ctx	20.5	20.0	21.0	28.6
decode @20k	16.7	17.2	18.4	25.0
decode @60k	—	14.3	14.9	22.7
cold 21k prefill	41-43s	53.9s	42.1s	45.5s
conc-2 aggregate	—	—	25.5	32.0
conc-3 aggregate	—	—	29.7	33.0

1. b10769 — PR #27742 merged hours after the original build; four qwen4exp commits landed since. #28011/#28040 fixed an n-gram lookup that scanned all 256 sequences per KV cell, which was the short-to-long decode falloff. #27941 fixed two correctness bugs this config was exposed to: blocks keyed on position alone could pool another sequence's cells under --kv-unified (silent bad output), and a CUDA abort at n_kv 262144 that -c 327680 sat past.
2. --lazy-mode off — the default auto reads any tensor over 4 GiB from disk on demand, and the PLE n-gram table is ~28 GB, so every prefill was paging it off /mnt/models while 376 GB of RAM sat idle. This also makes PR #28136 (">2x prefill" for the lazy table) irrelevant here — it optimizes the path we stopped using.
3. MTP speculative decoding — draft heads published 2026-09-01; engine from PR #28243. Acceptance 69% (83/69/55% by draft position); n-max 3 beat n-max 4 (61% acceptance, 22.9 t/s). Unlike the Qwen3.8-27B experience, the gain grows with context, because verifying k tokens in one target forward amortizes the DDR4 expert-weight read that bounds decode here.

Against the incumbent vLLM Qwen3.8-27B (51.4 t/s @20k, 9.8s cold prefill): the decode gap narrowed from ~3x to ~2x, and g2 (>=25 t/s @20k) now passes. Prefill remains the weak axis at ~45s vs ~10s.