Skip to content

[BUG6108338] Update windows documentation for onnxruntime quantization with Cuda13.x #1368

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kevalmorabia97 merged 1 commit into from
Apr 29, 2026
+16 −0
Merged

[BUG6108338] Update windows documentation for onnxruntime quantization with Cuda13.x #1368

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions docs/source/getting_started/windows/_installation_standalone.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,22 @@ If you need to use any other EP for calibration, you can uninstall the existing

By default, ModelOpt-Windows utilizes the `cupy-cuda12x <https://cupy.dev//>`_ tool for GPU acceleration during the INT4 ONNX quantization process. This is compatible with CUDA 12.x.

If you are using CUDA 13.x, update CUDA-dependent packages manually:

For official ONNX Runtime guidance, see `Nightly builds for CUDA 13.x <https://onnxruntime.ai/docs/install/#nightly-for-cuda-13x>`_.

1. Uninstall ``cupy-cuda12x`` and install ``cupy-cuda13x``.
2. Uninstall ``onnxruntime-genai-cuda`` and ``onnxruntime-gpu``.
3. Install ONNX Runtime CUDA 13 nightly and the pre-release ``onnxruntime-genai-cuda`` package.

Comment on lines +71 to +74

@coderabbitai coderabbitai Bot Apr 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Description: Search for any references to onnxruntime-genai in project configuration files

# Search for onnxruntime-genai in dependency files
rg -i 'onnxruntime-genai' --type=toml --type=txt --type=cfg -g '!**/*.rst'

Repository: NVIDIA/Model-Optimizer

Length of output: 94

🏁 Script executed:

#!/bin/bash
# Search for onnxruntime-genai references with corrected syntax
rg -i 'onnxruntime-genai' --type=toml --type=txt

Repository: NVIDIA/Model-Optimizer

Length of output: 156

🏁 Script executed:

#!/bin/bash
# Also search more broadly without type restrictions
rg -i 'onnxruntime-genai'

Repository: NVIDIA/Model-Optimizer

Length of output: 7460

🏁 Script executed:

#!/bin/bash
# Read the full context of the documentation file around lines 67-82
cat -n docs/source/getting_started/windows/_installation_standalone.rst | sed -n '60,90p'

Repository: NVIDIA/Model-Optimizer

Length of output: 1951

🏁 Script executed:

#!/bin/bash
# Check pyproject.toml for any onnxruntime references
rg -i 'onnxruntime' pyproject.toml

Repository: NVIDIA/Model-Optimizer

Length of output: 677

🏁 Script executed:

#!/bin/bash
# Check if there are earlier sections in the same file that mention installing onnxruntime-genai-cuda
head -n 70 docs/source/getting_started/windows/_installation_standalone.rst | cat -n

Repository: NVIDIA/Model-Optimizer

Length of output: 4397

🏁 Script executed:

#!/bin/bash
# Also check if _installation_with_olive.rst (which mentions onnxruntime-genai-cuda) is a prerequisite or related guide
grep -n "onnxruntime-genai" docs/source/getting_started/windows/_installation_with_olive.rst | head -20

Repository: NVIDIA/Model-Optimizer

Length of output: 496

Clarify the context for uninstalling onnxruntime-genai-cuda in Section 5.

Step 2 instructs users to uninstall onnxruntime-genai-cuda, but this package is not installed as part of the standalone ModelOpt-Windows installation guide (Sections 1-4). Users following this guide from the beginning would not have this package, making the uninstall step confusing and potentially causing errors.

While onnxruntime-genai-cuda is used in other installation paths (e.g., the Olive workflow guide), Section 5 lacks context about when this package would be present.

Please clarify:

  • Is Section 5 intended for users who previously installed onnxruntime-genai-cuda through alternative paths?
  • Should the uninstall step be conditional (e.g., "If you previously installed onnxruntime-genai-cuda from another guide, uninstall it")?
  • Or should onnxruntime-genai-cuda be added to the prerequisite installations before Section 5?
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@docs/source/getting_started/windows/_installation_standalone.rst` around
lines 71 - 74, Clarify that Step 2's uninstall of onnxruntime-genai-cuda is
conditional by updating Section 5 to state that this step applies only if the
user previously installed onnxruntime-genai-cuda via another guide (e.g., the
Olive workflow); change the wording to "If you previously installed
onnxruntime-genai-cuda, uninstall it" or alternatively add
onnxruntime-genai-cuda to the prerequisite list before Section 5 if the intent
is that it should have been installed earlier; ensure the text references the
package name onnxruntime-genai-cuda and the other packages mentioned
(onnxruntime-gpu, cupy-cuda12x/cupy-cuda13x) so readers know when the uninstall
is required.

.. code-block:: bash

pip uninstall -y cupy-cuda12x onnxruntime-genai-cuda onnxruntime-gpu
pip install cupy-cuda13x
pip install coloredlogs flatbuffers numpy packaging protobuf sympy
pip install --pre --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ort-cuda-13-nightly/pypi/simple/ onnxruntime-gpu
pip install --pre onnxruntime-genai-cuda

**6. Verify Installation**

Ensure the following steps are verified:
Expand Down
Loading