feat: overlap CPU cold-expert and GPU hot-expert computation in the MoE cache

[thecodacus]

What

Runs the MoE expert cache's two chains concurrently: the cold-expert chain executes on a CPU worker thread while the hot-pack chain runs on the GPU, joined at the merge. On by default; disable with --no-sched-async-cpu (llama-server, llama-cli) or bench both sides in one run with llama-bench --sched-async-cpu 0,1 (column sac).

Measured (RTX 3060, Qwen3.6-35B, 88 slots, merged profile)

	off	on
generation + MTP (llama-cli, 256 tok)	70.0	72.7–74.2 (+4–5%)
generation, cache only	59.9	61.0 (+1.8%)
pp256 (after batch gate)	309.1	297.7 (−3.7%)

Outputs byte-identical on vs off, with and without MTP. Serving stress test at 32k ctx / 96 slots: 509 t/s prefill on a 1.9k-token prompt, 69.8 t/s over 400 generated tokens.

How

• ggml sched: ggml_backend_sched_set_async_cpu() — CPU splits execute on a persistent worker; the main loop keeps launching later splits and joins lazily at the first split that is CPU or reads a CPU-resident tensor. Leftover jobs are drained on eval entry, synchronize, and free.
• graph: the cold chain is built before the hot chain so the hot split follows it with no cross-backend inputs (it launches while the worker computes), and the merge add is pinned to CPU for decode-size batches (n_tokens <= 8) so it cannot pull the hot chain into a split that waits on the cold result. Large prefill batches keep the previous graph — the CPU merge and its activation copies cost more than the overlap hides there.
• fix en route: the expert cache pack buffer is now marked GGML_BACKEND_BUFFER_USAGE_WEIGHTS. Without it, the scheduler's assignment pass does not pin pack tensors to their backend, and a CPU-assigned consumer propagates backward and drags the hot matmuls (plus a per-layer weight copy) onto the CPU.
• plumbing: llama_context_params.sched_async_cpu (default true) → cparams → sched setter; common args flag pair; llama-bench test dimension.

Notes

• Plain decode without speculative decoding measures ~1–2% slower with the overlap at bench (46.7 vs 47.5 tg64) — the split overhead is not fully repaid when the chains are short. With --spec-type draft-mtp the verify batches lengthen both chains and the gain is consistent.
• Deployment sizing note: keep ~900 MB VRAM free beyond the pack — runtime CUDA pool growth on large prompts allocates beyond what the load-time check sees.