qwen38-vllm-flashinfer

Qwen3.8-27B (W4A16) served by vLLM with FlashInfer un-gated on Turing (SM75). This is Portainer stack 126 (formerly 105) on trogdor, the daily-driver LLM endpoint on :8090.

It began life as experiments/qwen38-vllm-flashinfer/ in the trog repo and moved here on 2026-10-17 with its history once trog was deprecated and this became the production deployment. The stock-image fallback (stack 103) still lives in trog at experiments/qwen38-vllm.

Files

• Dockerfile — vllm/vllm-openai with ONE comparison changed: the FlashInfer SM80 floor lowered to SM75. Built on trogdor, never pushed; the stack must keep PullImage OFF.
• docker-compose.yml — the stack. The long comment at the top is the measured history (bench numbers, tuning pass, what was refuted). The warmup sidecar's script is inlined under configs: because Portainer git stacks check out only the compose file.
• warmup.py — the readable copy of that inlined script.

Rebuild the image

Only when the base tag changes:

DOCKER_HOST=ssh://cmoriarty@trogdor docker build -t vllm-openai-turing-flashinfer:v0.31.0 .

Then redeploy stack 126 from Portainer (git-connected to this repo, refs/heads/main).

Tried: DFlash2 speculative decoding (2026-10-18) — REJECTED

Drafter z-lab/Qwen3.8-27B-DFlash2 (1.92B block-diffusion drafter, 7 drafts per round, kept at /mnt/models/Qwen3.8-27B-DFlash2) on this repo's v0.31.0 image with the production flags plus --speculative-config '{"method":"dflash","model":"/models/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}', PIECEWISE cudagraphs, prefix caching OFF (raw acceptance), V2 model runner (DFlash2 is V2-only; on V1 the same checkpoint silently degrades to DFlash1). Same-night baseline on the same image and cache volume, xbench suite (trogdor:~/exp1004).

	baseline	DFlash2 (FlashInfer drafter, k=7)
decode short ctx	67.3 t/s	18.8 t/s
decode @20k ctx	63.8 t/s	18.2 t/s
conc-2 aggregate	103.4 t/s	16.6 t/s
conc-4 aggregate	170.4 t/s	18.8 t/s
cold 18k prefill	10.5 s	10.6 s
code-edit decode	66.1 t/s	15.5 t/s
accepted draft tokens per round	—	0.00 (0 of 28175 drafted, 4025 rounds over the whole xbench)
KV pool	991,688 tok	528,750 tok (pin cut to 7.0e9)
engine	stable	OOM-crashed on a 21k-token tool-call prefill

Every pre-registered kill criterion fired. Mechanism, in order of weight:

1. Memory. The drafter adds 1.44 GiB per card (weights 4.61 → 6.05 GiB). The production --kv-cache-memory 8290319872 OOMs at allocation; at 7.0e9 the pool is 528,750 tokens (−47%: smaller pin, plus the drafter's sliding-window layers turn the 4 KV-cache groups into 15 with padding waste), and the engine still OOM-crashed (80 MiB request inside the drafter's hidden-state projection, fc) during the 21k-token tool-call prefill. ~6.5e9 would be needed for headroom, i.e. a ~55% pool cut.
2. Per-step cost on Turing. FlashInfer reports no uniform-batch cudagraph support on SM75 (can_use_trtllm_attention is false), so the 8-query verify step runs piecewise and the drafter runs eager (DFlash refuses piecewise graphs for the draft). With nothing accepted a step costs ~3.5x a plain decode step (18.2 vs 63.8 t/s at 20k), so break-even needs ≥2.5 accepted draft tokens per round; the model card's own acceptance (3.1-4.5 on an H200, bf16) would make it a modest win at best.
3. Zero acceptance, structural. 0 accepted at every draft position on every request, and still 0 of 35,028 drafted over 5,004 rounds with the drafter moved to TRITON_ATTN + fp16 draft KV — so neither the FlashInfer non-causal path nor fp8 draft KV is the cause. The target itself stays correct (tool calls intact, greedy text diverges from the baseline only at near-ties). Controls: MTP k=2 on the identical V2 verify/rejection path accepts normally (1.3-2.0 per round); FlashInfer top_k, which the DFlash2 candidate selector uses, matches torch.topk on this GPU; the drafter's weights fit fp16 (max |w| 20.9, no overflow). Upstream users who see the same boot warning ("no KV cache group could be identified as the draft model's", vllm #57032 / #61443) on Ampere/Ada/Orin get 20-90% acceptance, so this is Turing-specific. Remaining suspects: fp16 activations in the bf16-trained drafter (SM75 has no bf16), the DFlash2 Triton kernels (grouped conv, selector walk) on SM75, inductor codegen on SM75. Splitting them needs a bf16-capable card (nzxt's A4500) running the same drafter against the same target.
4. Not run, on purpose: the prefix-caching-ON arm and the soak. Acceptance is zero before any cache hit, so the #58894 collapse question is moot here. #55601 (merged 2026-10-07) and #61455 (merged 2026-10-17, explicit draft-layer tags) land after v0.31.0 and fix cache-hit behaviour, not a zero baseline. No vllm/vllm-openai:v0.32.0 image on Docker Hub as of 2026-10-18 (tag exists; nightlies only).

Side results worth keeping: MTP k=2 on the V2 runner (PIECEWISE, prefix caching off, KV 7.0e9 → 780,986 tokens) is no longer the 3x loss measured on V1 in September: 20k prose 60-65 t/s (baseline 62), code edit 78 (66), tool call 95 (72), thinking 64 (62), short prose 52 (66), acceptance 1.3-2.0 per round. Not xbenched (no conc-2/4), so not adopted; it deserves its own measured pass with the production KV pin. ngram/ngram_gpu are not supported on the V2 runner (V1 is removed in 0.32).

Re-test DFlash2 only when a bf16-capable card shows this drafter accepting against this target (then the fault is Turing numerics/kernels and a vLLM fix is needed), on a release carrying #55601 + #61455, with the KV pin lowered to ~6.5e9. Harness and raw numbers: trogdor:~/exp1004 (drun.sh, arm.sh, specprobe.py, results/*1018*, results/spec_dflash-*.json, logs/*.engine.log).