feat: EP registration + monitoring — subprocess isolation, structured failures, universal op-tracing dispatch, lazy CLI startup

[tezheng]

Summary

Six capability areas landed across 3 commits. Op-tracing is one; the others are load-bearing infrastructure the CLI relies on.

1. EP registration hardening (commands/sys.py)

isolated_ep_register runs each filesystem-backed EP registration in a fresh Python subprocess so Windows' process-wide, base-name-keyed loaded-modules table can't leak the first-loaded plugin_impl.dll's metadata into every subsequent call. ThreadPoolExecutor(max_workers=4) parallelizes the fleet — 14 s → ~9 s on a 7-EP box. Nested _worker() function shipped via inspect.getsource to python -c; subprocess.run(timeout=...) cleanly terminates hung children.

2. Structured EP failure attribution (session/ep_device.py)

WinMLEPRegistrationFailed upgraded from an opaque message wrapper to a structured exception carrying .code, .reason, .dll_path, .fallback_version, all parsed at construction. Reason table covers Win32 error codes 2, 5, 126, 127, 193, 1114; unknown codes get a generic fallback. fallback_version reads the DLL's PE VS_VERSIONINFO via pywin32 so [failed] rows still surface a truthful version even when ORT never loaded the binary.

3. EP registry API surface (session/ep_registry.py)

WinMLEP.arg0 (identifier for unregister_execution_provider_library), WinMLEP.to_dict() serializing plugin_version + per-device device_facts so Architecture/Driver rows survive the subprocess boundary, WinMLEPRegistry.unregister_ep() for the register → snapshot → unregister cadence.

4. Op-tracing monitor extension (session/monitor/openvino_monitor.py NEW + commands/perf.py)

Second op-tracing monitor implementation. Universal _resolve_ep_monitor dispatch: auto-selects by device (NPU/auto → try monitor A, fall back to B; GPU → B), explicit --ep honored, --op-tracing detail rejected when the resolved EP has no detail surface. Enabled via ORT env var + load_config provider option; per-inference CSV flush (not per-session-destroy); env var save/restore in __enter__/__exit__ so we never leak monitor state.

5. CLI cold-start perf (transformers_compat.py NEW, _transformers_compat.py deleted)

Prior state eagerly imported the transformers 5.x compat shim from winml.modelkit.__init__, dragging transformers (~10 s cold) into every invocation — including winml --help. That defeated the PEP 562 lazy dispatch the package already had. Fix: wrap the shim body in install(), register _OptimumImportHook at sys.meta_path[0] that fires install() the first time anything imports optimum.*. Lightweight commands never fire the hook; heavy commands fire it once, transparently. install() sets the guard flag at entry (prevents recursion when install() itself imports optimum) and rolls it back on any exception. winml --help cold path: 277 ms with transformers NOT in sys.modules.

6. Test coverage gap closed

tests/cli/ was outside the CI matrix and never caught the transformers regression that landed with the transformers-5 upgrade. Moved to tests/unit/cli/ per tests/CLAUDE.md layout rule, added cli group to .github/workflows/modelkit-ci.yml matrix, deleted the 460-line duplicate at repo root, added TestWallClock guardrail (winml --help < 8 s in a fresh subprocess).

Commits

• e6be7659 — feat(session): op-tracing monitor for second EP + universal dispatch
• 0e2b5f05 — perf(cli): auto-fire transformers 5.x compat shim via sys.meta_path hook
• 9a8993f4 — feat(session): isolate EP registration in fresh subprocesses for truthful per-DLL metadata

Pre-existing baseline failures (out of scope, reproduce on 15861f75)

• 3× _LAZY_IMPORTS[winml.modelkit.onnx] drift — onnx/__init__.py lacks the PEP 562 dispatch pattern other subpackages already use
• 3× TestSysCommand AttributeError: 'list' object has no attribute 'items' shape drift in sys command test fixtures

Both categories reproduce on 15861f75 before any commit on this branch and are explicitly declared out of scope in commit 0e2b5f05's message.

Test plan

• uv run pytest tests/unit/ tests/unit/cli/ --timeout=60 — expect 1149 passed, 6 pre-existing baseline failures, 8 legit skips
• uv run pytest tests/unit/session/monitor/test_openvino_monitor.py — 27 tests, all green
• uv run pytest tests/unit/commands/test_perf_optracing.py — 45 tests (7 new for op-tracing dispatch), all green
• uv run winml --help — sub-second, 'transformers' not in sys.modules
• $env:WINMLCLI_EP_PATH="...\x64\Release"; uv run winml sys — 5 truthful OpenVINO row versions surface + Architecture/Driver rendered for NPU/GPU + [failed] rows show compact Win32 reason + PE fallback version
• uv run winml perf -m microsoft/resnet-50 — exit 0, latency ~2.5 ms, throughput > 300 samples/sec
• uv run winml perf -m microsoft/resnet-50 --monitor — HW monitor renders, no monitor/EP errors
• uv run winml perf --help — --op-tracing help text mentions "Auto-selects the monitor for the chosen EP" (no vendor-specific dispatch language)

[timenick]

Let’s exclude the .md files from this pr.

[xieofxie]

please

• merge conflicts
• clean up commit-analysis and docs folder
• consider split pr into self-contained features like
  • update transformers to 5
  • speed up cold-start
  • EP related
  • OP tracing related

Thanks

[tezheng]

E2E verification checklist

Runnable verification protocol for this branch — see E2E-VERIFICATION.md at repo root (added in f084ca7f).

Verified on Intel/OpenVINO NPU host (author). Other platforms — please pick up the delta.

Minimum smoke (~5 min with warm cache)

uv run winml sys --format json
uv run winml perf -m microsoft/resnet-50 --device auto --ep auto --monitor
uv run winml perf -m ./convnext/model_opt_qdq.onnx --ep qnn --device npu --op-tracing basic
uv run winml perf -m t5-small --device cpu

Platforms wanted

Host	Priority checks not yet covered by author
Snapdragon (Qualcomm NPU)	perf + analyze with --ep qnn --device npu --op-tracing basic — currently ENV on author's box (Intel AI Boost NPU, not Qualcomm)
AMD (MIGraphX GPU)	sys --list-ep should surface MIGraphX rows; perf against --device gpu --ep migraphx
NVIDIA (TensorRT / TensorRTRTX GPU)	perf against --device gpu --ep nv_tensorrt_rtx
Fresh Windows install (WinML Catalog only)	Verify sys --list-ep renders Catalog source tag; perf with --ep openvino

How to classify failures

• REGRESSION: click parse error, ImportError, NameError, TypeError, or Python traceback originating from src/winml/modelkit/…. Block ship.
• ENVIRONMENTAL: hardware/EP unavailable on your host (No source for QNNExecutionProvider/npu exposed device class 'NPU', CompiledModel was not initialized, etc.). Doc has an ENV triage table. Not a ship blocker.
• DOWNSTREAM: known composite ONNX export issue (onnx_nodes: 0 → tuple index out of range on t5-small). Separate concern, unrelated to this commit's B3/B6 fixes.
• DOC-BUG: command runs but the doc's expected-output snippet doesn't match your platform's actual render. File a follow-up so the doc can be tuned.

Anti-pattern grep

Every regression signature has a grep pattern in the doc's grep table. If you find EP: ( in any output, T-01 tuple leak regressed. If you find TypeError: from_pretrained, B3 regressed. Full list in the doc.

Author's results (two runs, Intel host)

• Run 1 (initial doc): 42 PASS / 0 REG / 10 ENV
• Run 2 (revised doc): 39 PASS / 0 REG / 7 ENV / 1 DOWNSTREAM / 5 DOC-BUG (all format/casing polish; substantive claims verified)

Raw command outputs under commit-analysis/cc7a6650/e2e-results{,-v2}/ (untracked, session artifacts).

If you're picking up a platform

1. Run the minimum smoke first — that's the fastest signal.
2. If clean, run the section of the full matrix specific to your hardware.
3. Post results back here with PASS/REG/ENV counts.

[tezheng]

PR #1019 E2E Verification Plan

Branch: feat/ep-registration-and-monitoring (winml-cli, formerly ModelKit)
PR: #1019 — "feat: EP registration + monitoring — subprocess isolation, structured failures, universal op-tracing dispatch, lazy CLI startup"
Verification target commit: cc7a6650 ("feat(session): unified-source EP refactor + P0 fixes + simplification pass")

Context / why this doc exists

PR #1019's own "Test plan" checklist references a three-commit chain
(9a8993f4, 0e2b5f05, e6be7659) and a baseline of "reproduce on
15861f75". That history no longer exists — the branch was rewritten
upstream into a single squashed commit, cc7a6650, which the local repo
has already been synced to (git reset --hard, no force-push performed).
The PR description was not updated to match, so its stated expected
numbers (1149 passed, 6 pre-existing baseline failures, 8 legit skips)
are not guaranteed to still hold and must be re-derived, not assumed.

A prior manual check on cc7a6650 already found 6 failures in
tests/unit/commands/test_perf_optracing.py, all related to OpenVINO
op-tracing dispatch (RuntimeError: Op-tracing --ep openvino is not currently supported) — a different failure set than the two categories
the PR describes as pre-existing/out-of-scope. Task 1 below must
determine whether these 6 are old-known, newly-introduced, or something
else, before any task can be marked "matches expectation."

High-level goal

Independently re-verify every item in PR #1019's Test plan checklist
against the actual current branch tip (cc7a6650), using measured
results — not the PR body's (now possibly stale) claimed numbers — and
produce a clear per-item verdict: PASS (behavior matches what the
PR claims or a reasonable equivalent), FAIL (behavior contradicts
the claim, needs investigation/fix), or DEVIATION (behavior differs
from the claim but is explainable/documented, e.g. the OpenVINO
failures above).

High-level passing criteria

All of the following must be true to close this verification:

1. Every task below has a recorded PASS/FAIL/DEVIATION verdict with the
   actual command output (or a summary of it) as evidence — no verdict
   without evidence.
2. Any FAIL is either fixed (with a follow-up commit) or explicitly
   escalated to the user with a clear description of the discrepancy —
   never silently ignored or reclassified as DEVIATION to make the plan
   "pass."
3. The final report reconciles the PR body's stale claims (commit list,
   baseline failure count) against what's actually true on cc7a6650,
   so whoever reads the PR next isn't misled by outdated text.

Tasks

Each task is independent and can be picked up in any order, but Task 1
should run first since its finding affects how to interpret Task 2/3's
failure counts.

---

Task 1 — Full unit test suite baseline

Goal: establish the true current pass/fail/skip counts on cc7a6650,
and classify every failure as pre-existing/known vs. new.

Command:

uv run pytest tests/unit/ tests/unit/cli/ --timeout=60

Pass criteria:

• Command completes (no collection errors, no hangs).
• Every failing test is individually classified: matches one of the
  PR's two stated pre-existing categories (_LAZY_IMPORTS[winml.modelkit.onnx]
  drift, TestSysCommand shape drift), OR matches the 6
  already-found OpenVINO op-tracing dispatch failures, OR is genuinely
  new (→ FAIL, needs a bug report or fix).
• Skip count and reasons are sane (hardware-gated skips only, no
  accidental skips of things that should run).

Verdict recorded as: PASS if all failures fall into a known
category; FAIL if any failure is unclassified and looks like a real
regression; DEVIATION if the counts differ from the PR's claim but
every individual failure is still explainable.

---

Task 2 — OpenVINO monitor unit tests

Goal: confirm the second op-tracing monitor implementation
(OpenVINOMonitor) is fully green in isolation, independent of the
dispatch-layer failures found in Task 1/3.

Command:

uv run pytest tests/unit/session/monitor/test_openvino_monitor.py

Pass criteria: all tests pass (PR claims 27). If the count differs,
record the actual count — a differing count alone is not a FAIL as
long as every test present passes.

---

Task 3 — Perf/op-tracing dispatch unit tests

Goal: confirm commands/perf.py's _resolve_ep_monitor dispatch
logic (NPU/auto → try monitor A, fall back to B; GPU → B; explicit
--ep; --op-tracing detail rejection) works as designed.

Command:

uv run pytest tests/unit/commands/test_perf_optracing.py

Pass criteria: all tests pass (PR claims 45, 7 new for dispatch).
Known current state: 6 failures already found here, all
RuntimeError: Op-tracing --ep openvino is not currently supported /
related dispatch errors — these must be root-caused as part of this
task, not just re-confirmed. Determine: is OpenVINO op-tracing support
actually wired up on this branch, or does _resolve_ep_monitor have a
bug that never dispatches to it? The PR's own summary text explicitly
claims OpenVINO dispatch is implemented ("NPU/auto → try monitor A,
fall back to B") — if the tests say otherwise, that's a real
implementation/test mismatch to report, not paper over.

---

Task 4 — CLI cold-start performance

Goal: confirm winml --help no longer eagerly imports transformers
(PEP 562 lazy-dispatch regression fix).

Commands:

uv run winml --help

then verify transformers is absent from a fresh subprocess's
sys.modules (the PR's own TestWallClock guardrail test does this —
running it directly is an acceptable substitute for a manual check):

uv run pytest tests/unit/cli/test_import_time.py tests/unit/cli/test_main.py -k "WallClock or cold"

Pass criteria: winml --help returns sub-second; transformers is
not in sys.modules for that invocation.

---

Task 5 — winml sys with directory-sourced EP override

Goal: confirm EP registration hardening (subprocess isolation,
structured failure attribution, PE fallback-version reads) produces
truthful output when pointed at a real EP DLL directory.

Command (adjust path to an actual available EP build directory —
ask the user for the correct WINMLCLI_EP_PATH target if none is on
hand; do not fabricate a path):

$env:WINMLCLI_EP_PATH="<path>\x64\Release"; uv run winml sys

Pass criteria: 5 truthful OpenVINO row versions surface; Architecture/
Driver columns render for NPU/GPU rows; any [failed] rows show a
compact Win32 reason string and a PE-read fallback version rather than
a blank/opaque failure.

Note: this task requires actual hardware/EP-build access this
session may not have. If no suitable directory is available, record
this task as BLOCKED (no test fixture), not FAIL — do not skip it
silently from the final report.

---

Task 6 — winml perf baseline (no monitor)

Goal: confirm the EP registration/dispatch changes haven't
regressed baseline inference performance or exit codes.

Command:

uv run winml perf -m microsoft/resnet-50

Pass criteria: exit code 0; latency in the ballpark of ~2.5 ms
(flag as DEVIATION, not FAIL, if it's in the same order of magnitude
but not exactly 2.5 ms — hardware/thermal variance is expected);
throughput > 300 samples/sec.

---

Task 7 — winml perf --monitor (hardware monitor)

Goal: confirm the HW monitor path still renders correctly and
doesn't throw monitor/EP errors after the registration changes.

Command:

uv run winml perf -m microsoft/resnet-50 --monitor

Pass criteria: HW monitor output renders (CPU/RAM/NPU utilization
etc.); no monitor- or EP-related errors or tracebacks in output.

---

Task 8 — winml perf --help wording check

Goal: confirm the --op-tracing help text was generalized away
from vendor-specific (QNN-only) language now that OpenVINO is a second
supported monitor.

Command:

uv run winml perf --help

Pass criteria: help text for --op-tracing mentions "Auto-selects
the monitor for the chosen EP" (or equivalent EP-agnostic phrasing) —
no leftover QNN-specific or vendor-specific wording implying only one
EP is supported.

---

Task 9 — Reconcile PR description with actual state

Goal: not a test — a documentation fix. After Tasks 1–8 are done,
summarize the discrepancies between the PR body's stated commit list /
baseline-failure claims and what's actually true on cc7a6650.

Pass criteria: a clear written summary (in the final report, not
necessarily a PR edit unless the user asks for one) of what changed
between what the PR says and what's actually on the branch now,
so a reviewer isn't misled.

Final report format

For each task: task number, verdict (PASS / FAIL / DEVIATION / BLOCKED),
one-line evidence summary, and a pointer to full command output if
retained. Close with the high-level goal's three passing criteria,
explicitly confirmed or not.