feat(perf): add --apply-template flag to control genai prompt templating#1042
Conversation
winml perf --runtime winml-genai always wrapped --prompt in the bundle's chat template before timing. Add --apply-template/--no-apply-template (default on, preserving current behavior) so users can benchmark a prompt that is already formatted -- or a raw completion prompt -- verbatim. - GenaiPerfConfig.apply_template gates GenaiPerfBenchmark._prompt_text; when disabled the prompt is encoded as-is (no apply_chat_template call). - Record apply_template in the JSON report's benchmark_info. - Wire the flag through the perf command. Closes #1030
|
I’m not sure why we need this user scenario, since onnxruntime-genai already includes an apply template API for it. |
…-template is on Detects when --prompt already contains the bundle's chat-template markers while --apply-template (the default) is set, which would double-wrap the prompt, and logs a warning suggesting --no-apply-template. Markers are derived from the bundle's own template (rendered around a unique sentinel), so nothing model-specific is hardcoded -- it works for ChatML, Llama [INST], Phi <|user|>, etc.
Removes the sentinel-based _looks_pretemplated heuristic and its warning. The core --apply-template/--no-apply-template flag is enough: callers who pass a pre-formatted prompt just add --no-apply-template. Simpler, no reverse-engineering of the bundle's template markers.
|
This is the scenario from #1030 (the issue this PR closes): benchmarking a prompt that is already formatted, instead of having The default ( |
benchmark_info now includes the verbatim prompt string, so a report captures what was actually timed -- most important with --no-apply-template, where the caller supplies a pre-formatted prompt. Also groups apply_template next to the prompt/max_new_tokens fields since it's a prompt-shaping setting.
Closes #1030 (follow-up to #1015).
Problem
winml perf --runtime winml-genaialways wraps--promptin the bundle's own chat template before timing. There was no way to benchmark a prompt that is already formatted (or a raw completion prompt), so users couldn't test their own templated input.Change
Add
--apply-template/--no-apply-template(default on, preserving current behavior). With--no-apply-template, the prompt is encoded and timed verbatim.GenaiPerfConfig.apply_templategatesGenaiPerfBenchmark._prompt_text; when disabled, the prompt is used as-is with noapply_chat_templatecall.apply_templateis recorded in the JSON report'sbenchmark_infofor reproducibility.perfcommand and documented in--prompt/--apply-templatehelp text.Usage