dsv4-flash

DeepSeek-V4-Flash-0731 (284B total / 13B active MoE, unsloth UD-Q8_K_XL, 162 GB) served by llama.cpp on trogdor, with the routed experts streaming from host RAM and attention / dense / shared-expert weights plus seven pinned expert layers on the four GPUs. An OpenCode web seat rides in the same stack, wired to it. This is Portainer stack dsv4-flash (trogdor, environment 39), git-connected to this repo on refs/heads/main, compose file docker-compose.yml. It is inactive by design — see Bracketing.

Where it came from

It lived in the trog repo as experiments/dsv4-flash/ (Portainer stack 97) from 2026-08-13 and moved here with its full history (git subtree split) on 2026-10-18 when trog was retired. The vLLM daily driver made the same move the day before (qwen38-vllm-flashinfer, stack 126).

What is in the stack

service	image	port	notes
dsv4	ghcr.io/ggml-org/llama.cpp:server-cuda-b11718	:8091	OpenAI-compatible at http://trogdor:8091/v1, model id deepseek-v4-flash, /health, /metrics
opencode	forgejo.underthere.xyz/cmoriarty/trog-opencode:latest	:4096	OpenCode web UI, provider dsv4 configured inline under configs:; waits for dsv4 to be healthy

The OpenCode image was built and pushed by the retired trog repo (opencode 1.18.7) and is now frozen at :latest. Its state lives in the named volumes dsv4-flash_opencode_workspace and dsv4-flash_opencode_state. Portainer deploys with the stack name as the compose project name (it overrides the compose name: key), so the stack must stay named dsv4-flash or the seat loses its workspace.

Weights: /mnt/models/DeepSeek-V4-Flash-0731-GGUF/UD-Q8_K_XL/ (5 shards, block_count 43, 256 experts / 6 used, 1 KV head, 1M context metadata). The DSpark drafter GGUF (11 GB) sits next to it; it was rejected (below). The GGUF carries no nextn (MTP) tensors — checked with gguf-dump on all five shards on 2026-10-18, layers stop at blk.42 — so llama.cpp's native --spec-type draft-mtp is not an option for this quant.

The rig

trogdor: 2x Xeon Silver 4214 (48 threads, two NUMA nodes), 376 GB RAM, 4x Quadro RTX 5000 16 GB (Turing, SM75). The host must have /proc/sys/kernel/numa_balancing = 0.

Measured (b10398, 2026-08-13/14)

configuration	decode t/s	prefill t/s	note
first boot: --cpu-moe, 44 threads	6.5	13.8	GPUs ~8% busy
shipped: last 7 expert layers pinned 2/2/2/1 via -ot, -ub 2048, -t 44, --numa distribute, --no-mmap, f16 KV	7.7 (warm)	133 (3.2k prompt)
shipped + --load-mode mmap	5.4		1.2 s restarts, but file-backed expert pages defeat --numa distribute; reverted
shipped + DSpark drafter (--spec-type draft-dspark, draft-n-max 3)	7.4–7.9		acceptance 0.35–0.40 vs 0.76–0.85 in unsloth's all-GPU runs; a wash, and card 0 lost its expert layers; rejected

Thread sweep, one cold boot per point: t6 4.4, t12 5.3, t24 5.7, t44 5.6 t/s; warm t44 7.4–7.7. Flat past 12 threads: decode is RAM-bandwidth / NUMA bound, not core bound, and the first run after every boot is ~30% cold.

Quant physics: the routed experts in UD-Q8_K_XL are the native MXFP4 DeepSeek shipped, so UD-Q4_K_XL would decode at the same speed (only ~4% of the weights differ). A single socket's bandwidth ceiling for 13B active is ~17 t/s; the gap from 7.7 is NUMA placement and dequant efficiency, not quant size.

Hard-won constraints (from the compose header — read it before changing flags)

• KV cache type. f16 is shipped. q8_0 KV was believed broken on the deepseek4 arch (llama.cpp #26423) when this shipped; that report was retracted (closed 2026-08-05; works since #25202) and the 2026-10-18 retest on b11718 confirms it is coherent at temp 0 with decode within noise. Use --cache-type-k q8_0 --cache-type-v q8_0 when VRAM is tight.
• Per-card -ot pins, catch-all =CPU last. Plain --n-cpu-moe 36 OOMed: layer-split dumped all seven GPU expert layers on one 16 GB card.
• --load-mode none (the old --no-mmap, which b11xxx removed along with --mlock; see the mmap row). ~200 s loads are the price of NUMA-placed experts.
• -fa on needs the CUDA flash-attention fix from ~2026-08-06 (#26509); b10143 and older produce garbage. If output is ever garbage, try -fa off first (~30% slower prefill, known-good path).
• --jinja is required for tool calls: llama-server converts DSML tool calls to OpenAI tool_calls (PR #26269).
• Sampling as DeepSeek's agentic recommendation: temp 1.0, top-p 0.95, min-p 0.01. Thinking defaults to high; flip with --chat-template-kwargs '{"enable_thinking":false}' or --reasoning off.
• -c 131072 is cheap (one KV head); the model's max is 1M.
• DSpark speculative decoding is rejected on this rig (flags kept in the compose for a future engine; never pass -devd).
• Portainer: a git redeploy that looks hung is still loading 162 GB — do not retry. A double-fire SIGKILLs the loading container and orphans a <hash>_dsv4-flash; recover with stack stop/start.
• The model sometimes self-identifies as Claude — a template quirk, ignore.
• Known llama.cpp risks for agent use as of 2026-08: #26965 (SIGSEGV on long uniform-ASCII tool output — cap tool output), #26694 (DSML leakage in long sessions, Metal-reported).

Engine refresh 2026-10-18: b10398 → b11718

Same night, same flags, same host, one boot per arm; bench/bench.py against the native /completion endpoint (the engine's own timings), 256-token decodes at temp 0, a nonce line to force cold prefill, and a 5-prompt coherence set via /v1/chat/completions (thinking off). Raw JSON is in bench/results/; bench/summary.py prints the table.

arm	build	prefill 3k t/s	prefill 20k t/s	decode 3k t/s	decode @20k t/s	warm 3k decode t/s	cached-turn TTFT (re-prefilled tokens)
b10398-f16-control	b10398	123.1	138.6	7.0	6.88	7.29	1.17 s (15)
b11718-f16-mlock	b11718	130.8	141.2	6.94	5.74	4.65	1.38 s (15)
b11718-f16	b11718	130.7	140.6	7.01	7.03	6.94	1.06 s (15)
b11718-q8kv	b11718	129.6	140.6	6.8	6.61	7.01	1.04 s (15)

Load to /health 200: b10398 190 s, b11718 212 s (f16) / 218 s (mlock). The mlock row's 20k and warm-repeat decodes were taken while a root docker build (fastllm, nvcc at ~12 jobs) started on the host at 03:37; its 3k decode (6.94, before the build) equals f16's. Decode here is CPU-bound, so any concurrent compile shows up directly.

• b11718 adopted. Prefill +6% at 3k, flat at 20k; decode unchanged at 7.0 t/s; cached-turn TTFT 1.17 → 1.06 s. The f16 coherence set matches the b10398 control on 2 of 5 prompts and diverges only at near-ties on the rest, with sensible content throughout.
• --no-mmap is gone. b11xxx rejects --no-mmap and --mlock ("invalid argument"); the loader flag is --load-mode none|mmap|mlock|mmap+mlock|dio, and none reproduces the old layout exactly (117 GB RSS, 116 GB shared, both NUMA nodes within 2 GB of each other), so the 4 GiB pinned-buffer cap (#30449) changed nothing measurable here.
• Cached turns were already fine. Both engines re-process only 15 tokens on a 20k cached prefix, so the SWA cache-reuse fix (#30580) was not load-bearing for this model and config.
• --load-mode mlock: nothing to gain. Same prefill, same clean 3k decode, 2 GB of RAM left free with 117 GB locked. Not adopted.
• q8_0 KV is valid on this arch now. -ctk q8_0 -ctv q8_0 on b11718: coherent at temp 0 on all five prompts (fact and format answers identical to f16, math identical to the b10398 control, code and summary diverge at near-ties into other valid answers); prefill 130/141 t/s, decode 6.8 / 6.6 / 7.0 t/s (3k / 20k / warm; the 20k figure overlapped a second root build), cached TTFT 1.04 s, load 221 s. Not a measured win, so the compose keeps f16; it is the switch to flip when VRAM is the constraint, since it halves the KV cache.
• Native MTP is not available. --spec-type draft-mtp exists in b11xxx, but gguf-dump over all five shards shows no nextn tensors (layers stop at blk.42); the quant was converted without the MTP head.
• Deferred arms (recorded so nobody re-plans them from scratch): ik_llama.cpp (ik-llama-server:c446f1e is on trogdor; the Flash-Next session measured it 2–4 t/s slower than mainline with a 14x worse cached-turn TTFT on Flash-Next, and an earlier ik port measured −25% on DeepSeek-4 — if retried, pre-registered kill criterion: adopt only if warm 3k decode ≥ mainline + 0.5 t/s and cached TTFT ≤ 3x mainline); --load-mode mmap+mlock and dio (restart-time trades, not decode levers — plain mmap cost 30% decode in August). --moe-cache-mib is not worth trying here (the ik port measured −25% with it on DeepSeek-4).

Bracketing (the rig is GPU-exclusive)

Stack 126 (qwen38-vllm-flashinfer, :8090) holds all four cards.

1. Stacks → qwen38-vllm-flashinfer → Stop
2. Stacks → dsv4-flash → Start (first load: several minutes; docker logs -f dsv4-flash)
3. Ready when http://trogdor:8091/health is 200; OpenCode at http://trogdor:4096
4. Stacks → dsv4-flash → Stop
5. Stacks → qwen38-vllm-flashinfer → Start

API form: POST https://192.168.1.119:9443/api/stacks/<id>/stop?endpointId=39 and /start, with the X-API-Key header.