Lower opset 24→23 for non-default EPs to eliminate memcpy - #254
Merged
Conversation
ORT's CUDA EP registers kernels up to opset 23 for standard ops (Reshape, RMSNormalization, etc.). When the model declares opset 24, these ops fall to CPUExecutionProvider, creating ~280 MemcpyFromHost and MemcpyToHost nodes that destroy inference performance. This implements the ort_lower_opset_for_ep flag (which was declared in _flags.py but never wired up). When enabled (default), the opset is lowered from 24 to 23 for all non-default EPs after optimization. Result: 282 memcpy → 4 memcpy for Gemma4 CUDA EP model. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
37 tasks
Performance Comparison
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
titaiwangms
reviewed
May 5, 2026
titaiwangms
left a comment
titaiwangms
left a comment
Contributor
There was a problem hiding this comment.
It would be better when this trigger, it sends a message to users to remind us of adding opset to ort.
Address review feedback: log an info message with the original opset, model name, EP, and a link to the ORT tracking PR when opset lowering is triggered. This helps users understand the workaround and track when it can be removed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Member
Author
|
Addressed the feedback — added an Commit: 69fc529 |
justinchuby
commented
May 6, 2026
justinchuby
commented
May 6, 2026
Co-authored-by: Justin Chu <justinchuby@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ORT CUDA EP registers kernels up to opset 23. Opset 24 models have Reshape/Cast fall to CPU → 280 memcpy. Wire up the ort_lower_opset_for_ep flag to lower opset after optimization. Result: 282→4 memcpy. Workaround until ORT PR #28368 adds opset 24 registrations.