From 41a4e224cf9fcd7f6a791fda6bc73ea9c6eef7c0 Mon Sep 17 00:00:00 2001 From: Michael Stufflebeam Date: Thu, 27 Aug 2026 04:57:01 +0000 Subject: [PATCH 1/2] docs: qwen4exp support, an upstream-sync warning, and a bit-identical caveat Ran the expert cache on Qwen3.8-Flash-Next (qwen4exp, 177B, 512 experts/layer) on a single RTX 3090: +34.5% decode for 9.2 GB, where moving whole layers with --n-cpu-moe instead spent 12.5 GB to buy 10%. Three things a downstream user would want in the guide: 1. After syncing past ggml-org ebb546b7e (#26802), the pack chains trip GGML_ASSERT(ggml_cuda_mul_mat_id_needs_sync) on the first decode. op_params[0]=1 correctly disqualifies both fast paths, but the new predicate predates the flag and answers 'no sync needed' for a small quantized batch. Two-line fix included; not a code PR because this branch has no needs_sync to patch. 2. On qwen4exp the output is NOT bit-identical to baseline -- it diverges at temperature 0, consistent with float non-associativity between one fused chain and two summed pack chains. Quality is unaffected as far as a proper test can see: paired over 64 chunks, dPPL +0.047% (t=+0.88) against that test's own 0.107% detection floor. 3. --moe-cache-slots without --moe-cache-profile is SILENT. Added to the troubleshooting table: tok/s reads as a small regression and VRAM is the only tell. --- README.md | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 53 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 211df1da5c77..0f471ef2770e 100644 --- a/README.md +++ b/README.md @@ -57,9 +57,60 @@ Measured on an RTX 3060 12GB (`-ngl 99 -ncmoe 99 -fa 1`): | GLM-4.7-Flash Q4_K_M (64 experts/layer) | 32.1 | **46.3 (+44%)** @ 40 slots | +64% | | Laguna-S-2.1-118B-A8B IQ4_XS (256 experts/layer) | 11.5 | **12.1 (+5%)** @ 36 slots | +12% | -Supported architectures: `qwen35moe`, `deepseek2`, `laguna` (plain fused-SILU gated expert FFN, +Supported architectures: `qwen35moe`, `deepseek2`, `laguna`, `qwen4exp` (plain fused-SILU gated expert FFN, separate gate/up/down tensors). Other architectures run unchanged. +### Note for a future upstream sync (and one caveat on "bit-identical") + +**After syncing past ggml-org `ebb546b7e` (#26802, 2026-08-11), the pack chains trip a new +assert.** That commit adds `ggml_cuda_mul_mat_id_needs_sync(dst, cc)` and asserts it at the +single `mul_mat_id` caller. The cache sets `op_params[0] = 1` ("ids may contain -1"), which +correctly disqualifies both the MMQ and MMF fast paths — so the op reaches the sync path — but +the new predicate predates the flag and answers *"no sync needed"* for a small quantized batch. +Result, on the first decode, every time: + +``` +ggml-cuda.cu:1948: GGML_ASSERT(ggml_cuda_mul_mat_id_needs_sync(dst, cc)) failed +``` + +Two lines, at the top of `ggml_cuda_mul_mat_id_needs_sync` — the caller has already established +the fast paths are unavailable, so this agrees with it rather than overriding it: + +```c +if (dst->op_params[0] != 0) { + return true; +} +``` + +Not sent as a code PR because this branch does not carry `needs_sync` yet, so the patch has +nothing to apply to. Ready when you take the sync: +[`cpuchip/llama.cpp@threadchip/mul-mat-id-needs-sync-fix`](https://github.com/cpuchip/llama.cpp/tree/threadchip/mul-mat-id-needs-sync-fix). + +**The caveat:** on `qwen4exp` the cached output is **not** bit-identical to baseline. At +temperature 0 it diverges from the uncached build — coherent, on-topic, consistent with a +single argmax flip, which is what float non-associativity between one fused chain and two +summed pack chains would produce. **Quality is unaffected as far as a proper test can tell**: +paired over the same 64 chunks, ΔPPL is **+0.047% (t = +0.88), below that test's own 0.107% +detection floor**, with the cached arm's peak VRAM confirming it engaged rather than silently +no-op'ing. The claim that holds on this architecture is *statistically indistinguishable*, not +*bit-identical* — worth softening the wording above for architectures beyond the three you +measured. + +### Measured on `qwen4exp` (Qwen3.8-Flash-Next, 177B, 512 experts/layer), single RTX 3090 + +| arm | VRAM | decode | +| --- | --- | --- | +| `--n-cpu-moe 48`, no cache | 6,284 MiB | 16.00 tok/s | +| `--n-cpu-moe 48` + `--moe-cache-slots 64` + profile | 15,522 MiB | **21.52 tok/s (+34.5%)** | +| `--n-cpu-moe 40` (whole layers on GPU instead) | 18,784 MiB | 17.63 tok/s (+10%) | + +The comparison that surprised us: moving *whole layers* onto the card spent **12.5 GB to buy +10%**, while the cache spent **9.2 GB to buy 34.5%** — because it picks hot experts across all +48 layers rather than every expert of a few. Oracle hit rates from `llama-moe-trace` + +`simulate.py`: 68.2% at 64 slots/layer, 88.0% at 128 (128 OOMs on 24 GB). We expected the +routing to be too flat for a cache to help — 512 fine-grained experts with load balancing look +designed against one — and it is not. + ### Quick start **1. Capture a routing profile** (one time per model — records which experts the router picks): @@ -137,6 +188,7 @@ A warning instead of this line means the cache fell back to baseline (see Tuning | `pack allocation failed` | Slot count too big — read the fit math in the warning and reduce. | | `no CPU-resident MoE layers` | Experts are already on GPU (no `--n-cpu-moe`) — nothing to cache. | | No init line, no warning | Architecture not wired for the cache — model runs unchanged. | +| `--moe-cache-slots` set but nothing changes | **No `--moe-cache-profile`.** `moe_map_hot` stays null, `use_moe_packs` is false, everything reports clean and tok/s reads as a small *regression*. VRAM is the only tell — it does not move a megabyte. | | Model loads, then context creation OOMs | Pack fits but KV/compute don't — drop a few slots or shrink/compress KV. | ## Recent API changes From 296f53f59a9d0cb8138ba9ecdd8941a4f8745d11 Mon Sep 17 00:00:00 2001 From: Michael Stufflebeam Date: Thu, 27 Aug 2026 06:07:25 +0000 Subject: [PATCH 2/2] docs: strengthen the quality evidence with a second, unmemorised corpus The first draft cited only Pride and Prejudice, and that corpus is a poor discriminator: baseline PPL 1.0594 means the model is near-certain of every token, so there is little room for a defect to show. Re-ran the whole comparison on technical prose (NASA SP-4205 OCR, baseline PPL 8.2438). NASA cache-64 vs baseline -0.047% t -0.85 floor 0.112% Austen cache-64 vs baseline +0.113% t +1.68 floor 0.107% Neither detectable, and the SIGN FLIPS between corpora -- a real quality cost would keep its sign. Both rows shown rather than only the favourable one. --- README.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 0f471ef2770e..5abbaec7681a 100644 --- a/README.md +++ b/README.md @@ -89,10 +89,19 @@ nothing to apply to. Ready when you take the sync: **The caveat:** on `qwen4exp` the cached output is **not** bit-identical to baseline. At temperature 0 it diverges from the uncached build — coherent, on-topic, consistent with a single argmax flip, which is what float non-associativity between one fused chain and two -summed pack chains would produce. **Quality is unaffected as far as a proper test can tell**: -paired over the same 64 chunks, ΔPPL is **+0.047% (t = +0.88), below that test's own 0.107% -detection floor**, with the cached arm's peak VRAM confirming it engaged rather than silently -no-op'ing. The claim that holds on this architecture is *statistically indistinguishable*, not +summed pack chains would produce. **Quality is unaffected, on two corpora with opposite signs**: +paired over 64 chunks each, with the cached arm's peak VRAM (15,544 MiB vs 6,286) confirming +it engaged rather than silently no-op'ing — + +| corpus | baseline PPL | cache-64 ΔPPL | t | detection floor | +| --- | --- | --- | --- | --- | +| technical prose (NASA SP-4205 OCR) | 8.2438 | **−0.047%** | −0.85 | 0.112% | +| *Pride and Prejudice* | 1.0594 | +0.113% | +1.68 | 0.107% | + +Neither is detectable, and **the sign flips between corpora** — a real quality cost would keep +its sign. The Austen row is included only for completeness: at PPL 1.06 the model is +near-certain of every token, which we take as memorisation rather than a usable discriminator, +and it is why the technical corpus was run as a control. The claim that holds on this architecture is *statistically indistinguishable*, not *bit-identical* — worth softening the wording above for architectures beyond the three you measured.