Fix CUDA EP provider_options and use float masks for fallback layers - #255
Closed
justinchuby wants to merge 2 commits into
Closed
Fix CUDA EP provider_options and use float masks for fallback layers#255justinchuby wants to merge 2 commits into
justinchuby wants to merge 2 commits into
Conversation
justinchuby
commented
May 5, 2026
justinchuby
commented
Member
- Remove CUDA provider_options from EP defaults — explicit options conflict with GenAI's internal session setup, causing NaN. 2. Use float16 additive masks for all fallback Attention layers instead of bool masks (more robust on CUDA). 3. Return empty provider_options list when no options needed.
Two fixes for Gemma4 CUDA EP inference: 1. Remove CUDA provider_options from EP defaults. Explicit CUDA provider_options in genai_config.json conflict with GenAI's internal session setup (ClearOutput, ReuseEmbeddingsBuffer, etc.), causing NaN or crashes for multimodal CUDA models. GenAI's C++ code handles all CUDA EP configuration internally. 2. Use float16 additive masks for all fallback Attention layers (KV-shared and head_dim>256 layers) instead of bool masks. Bool masks triggered NaN in ORT's CUDA ConvertAttnMaskToBias path. Float16 masks match the working default EP model's behavior. Tested: CUDA EP model (12 GQA + 23 Attention) generates at 13.5 tok/s through GenAI with valid output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
37 tasks
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
Performance Comparison
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Signed-off-by: Justin Chu <justinchu@microsoft.com>
Contributor
Member
Author
|
Closing: this PR is superseded by other changes:
Relevant PRs:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.