Skip to content

feat(perf): add --apply-template flag to control genai prompt templating#1042

Merged
DingmaomaoBJTU merged 4 commits into
mainfrom
dingmaomaobjtu-add-apply-template-flag
Jul 6, 2026
Merged

feat(perf): add --apply-template flag to control genai prompt templating#1042
DingmaomaoBJTU merged 4 commits into
mainfrom
dingmaomaobjtu-add-apply-template-flag

Conversation

@DingmaomaoBJTU

Copy link
Copy Markdown
Collaborator

Closes #1030 (follow-up to #1015).

Problem

winml perf --runtime winml-genai always wraps --prompt in the bundle's own chat template before timing. There was no way to benchmark a prompt that is already formatted (or a raw completion prompt), so users couldn't test their own templated input.

Change

Add --apply-template/--no-apply-template (default on, preserving current behavior). With --no-apply-template, the prompt is encoded and timed verbatim.

  • GenaiPerfConfig.apply_template gates GenaiPerfBenchmark._prompt_text; when disabled, the prompt is used as-is with no apply_chat_template call.
  • apply_template is recorded in the JSON report's benchmark_info for reproducibility.
  • The flag is wired through the perf command and documented in --prompt / --apply-template help text.

Usage

# Default: prompt wrapped in the bundle's chat template
winml perf -m path/to/bundle --runtime winml-genai --prompt "Hello"

# Benchmark a pre-formatted prompt verbatim
winml perf -m path/to/bundle --runtime winml-genai \
  --no-apply-template --prompt "<|user|>Hello<|end|><|assistant|>"

winml perf --runtime winml-genai always wrapped --prompt in the bundle's chat template before timing. Add --apply-template/--no-apply-template (default on, preserving current behavior) so users can benchmark a prompt that is already formatted -- or a raw completion prompt -- verbatim.

- GenaiPerfConfig.apply_template gates GenaiPerfBenchmark._prompt_text; when disabled the prompt is encoded as-is (no apply_chat_template call).
- Record apply_template in the JSON report's benchmark_info.
- Wire the flag through the perf command.

Closes #1030
@DingmaomaoBJTU DingmaomaoBJTU requested a review from a team as a code owner July 3, 2026 08:49
@timenick

timenick commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

I’m not sure why we need this user scenario, since onnxruntime-genai already includes an apply template API for it.

…-template is on

Detects when --prompt already contains the bundle's chat-template markers while --apply-template (the default) is set, which would double-wrap the prompt, and logs a warning suggesting --no-apply-template.

Markers are derived from the bundle's own template (rendered around a unique sentinel), so nothing model-specific is hardcoded -- it works for ChatML, Llama [INST], Phi <|user|>, etc.
Comment thread src/winml/modelkit/commands/_perf_genai.py Outdated
Removes the sentinel-based _looks_pretemplated heuristic and its warning. The core --apply-template/--no-apply-template flag is enough: callers who pass a pre-formatted prompt just add --no-apply-template. Simpler, no reverse-engineering of the bundle's template markers.
@DingmaomaoBJTU

Copy link
Copy Markdown
Collaborator Author

This is the scenario from #1030 (the issue this PR closes): benchmarking a prompt that is already formatted, instead of having perf wrap --prompt in the single-turn chat template on every run.

The default (--apply-template) still goes through ort-genai's apply_chat_template, so that path is unchanged. --no-apply-template is the opposite direction: it feeds the prompt to the tokenizer verbatim so you can time the exact token sequence you care about — a multi-turn conversation, a RAG prompt with a large retrieved context, or a custom/non-default template. Since perf measures prefill (TTFT) and decode throughput, the prompt's real length and shape drive the numbers; the single-turn template wrapper doesn't represent those cases. The genai apply_chat_template API templates a prompt, but doesn't give perf a way to skip templating and benchmark a pre-formatted one, which is what this flag adds.

Comment thread src/winml/modelkit/commands/_perf_genai.py Outdated
Comment thread src/winml/modelkit/commands/_perf_genai.py Outdated
benchmark_info now includes the verbatim prompt string, so a report captures what was actually timed -- most important with --no-apply-template, where the caller supplies a pre-formatted prompt. Also groups apply_template next to the prompt/max_new_tokens fields since it's a prompt-shaping setting.
@DingmaomaoBJTU DingmaomaoBJTU merged commit ffd424c into main Jul 6, 2026
8 checks passed
@DingmaomaoBJTU DingmaomaoBJTU deleted the dingmaomaobjtu-add-apply-template-flag branch July 6, 2026 04:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add a apply_template flag to control whether to apply template

3 participants