Improve GenAI session: smarter max_length, DML skip, flat-decoder support, and response logging#1054
Merged
Conversation
Contributor
Author
|
tested on
|
Move token collection from inside the per-token timing loop to after generation completes, using generator.get_sequence(0) to fetch all output tokens at once. This prevents get_next_tokens() from polluting decode_s/TPOT measurements with host/device synchronization overhead on hardware EPs.
DingmaomaoBJTU
approved these changes
Jul 6, 2026
DingmaomaoBJTU
left a comment
DingmaomaoBJTU
left a comment
Collaborator
There was a problem hiding this comment.
The latest commit (34eab68) resolves my earlier concern: token fetching via get_sequence() is now done after the timing loop, so decode_s/TPOT measurements are no longer polluted by potential host/device sync costs from get_next_tokens(). CI is green. LGTM.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Several improvements to GenAI session benchmarking and hardware EP detection.
Smarter
max_lengthallocation_new_generatornow computesmax_lengthasprompt_len + max_new_tokens, capped at the bundle'scontext_length. This avoids pre-allocating the full KV cache (which can be 128K+ for DML bundles) when only a small generation is requested.Hardware EP detection improvements
_bundle_uses_hardware_epnow returns the EP name (str | None) instead ofbool, enabling better logging.pipelinewrapper), used by e.g. OpenVINO exports.Response text capture and logging
response_textfield toGenerationTimingto capture decoded model output.generate_timednow logs a summary line with input/output token counts and timing metrics.Tests
_bundle_uses_hardware_eptests to match the new return type.test_max_length_is_prompt_plus_max_new_tokensfor the new max_length logic.test_dml_only_returns_none).