NVIDIA · kevalmorabia97 · Apr 27, 2026 · Apr 22, 2026 · Apr 22, 2026 · Apr 22, 2026
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -21,12 +21,14 @@ Changelog
 **Backward Breaking Changes**
 
 - The ``quant_cfg`` field in quantization configs is now an **ordered list** of ``QuantizerCfgEntry`` dicts instead of a flat dictionary. Each entry specifies a ``quantizer_name`` wildcard, an optional ``parent_class`` filter, a ``cfg`` dict of quantizer attributes, and/or an ``enable`` flag. Entries are applied in list order with later entries overriding earlier ones. The old dict-based format is still accepted and automatically converted via ``normalize_quant_cfg_list()``, but now emits a ``DeprecationWarning``; new code should use the list format. All built-in configs (e.g. ``FP8_DEFAULT_CFG``, ``INT4_AWQ_CFG``, ``NVFP4_DEFAULT_CFG``), examples, and YAML recipes have been updated. See the :ref:`quant-cfg` documentation for the new format reference and migration guide.
+- Deprecated Mllama (Llama 3.2 Vision) support in the ``llm_ptq`` and ``vlm_ptq`` examples. The ``model_type == "mllama"`` branches and ``MllamaImageProcessor`` usage have been removed from ``hf_ptq.py`` and ``example_utils.py``. For image-text calibration of VLMs, use ``--calib_with_images`` with a supported VLM (see Nemotron VL section in ``examples/llm_ptq/README.md``).
 
 **Bug Fixes**
 
 - Fix Megatron utility functions for generation (with pipeline parallelism) and ~10x speedup in MMLU score evaluation (by batching prefill passes).
 - Fix Minitron pruning (``mcore_minitron``) for MoE models. Importance estimation hooks were incorrectly registered for MoE modules and NAS step was hanging before this.
 - Fix TRT support for remote autotuning in ONNX Autotune from 10.16+ to 10.15+ and fix TRT versioning check to the ``trtexec`` version instead of the TRT Python API when using ``trtexec`` backend.
+- Exclude MatMul/Gemm nodes with K or N < 16 from ONNX INT8 and FP8 quantization. Such small-dimension GEMMs cannot efficiently use INT8/FP8 Tensor Cores and the added Q/DQ layers cause perf regressions in TensorRT. Honors Gemm ``transB`` when deriving K.
 
 **Misc**
 

@@ -1,4 +1,4 @@
-SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 SPDX-License-Identifier: Apache-2.0
 
 Licensed under the Apache License, Version 2.0 (the "License");

@@ -69,6 +69,8 @@ To install stable release packages for Model Optimizer with `pip` from [PyPI](ht
 pip install -U nvidia-modelopt[all]
 ```
 
+Model Optimizer will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.
+
 To install from source in editable mode with all development dependencies or to use the latest features, run:
 
 ```bash
@@ -79,8 +81,14 @@ cd Model-Optimizer
 pip install -e .[dev]
 ```
 
-You can also directly use the [TensorRT-LLM docker images](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/release/tags)
-(e.g., `nvcr.io/nvidia/tensorrt-llm/release:<version>`), which have Model Optimizer pre-installed.
+You can also directly use NVIDIA container images, which have Model Optimizer pre-installed:
+
+- `nvcr.io/nvidia/pytorch:<version>-py3`
+- `nvcr.io/nvidia/nemo:<version>`
+- `nvcr.io/nvidia/tensorrt-llm/release:<version>`
+- `nvcr.io/nvidia/tensorrt:<version>-py3`
+
+Before pulling and using the container images, please review their respective license terms.
 Make sure to upgrade Model Optimizer to the latest version as described above.
 Visit our [installation guide](https://nvidia.github.io/Model-Optimizer/getting_started/2_installation.html) for
 more fine-grained control on installed dependencies or for alternative docker images and environment variables to setup.

diff --git a/docs/source/getting_started/_installation_for_Linux.rst b/docs/source/getting_started/_installation_for_Linux.rst
@@ -32,11 +32,11 @@ Environment setup
 
     To use Model Optimizer with full dependencies (e.g. TensorRT/TensorRT-LLM deployment), we recommend using the
     `TensorRT-LLM docker image <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/release/tags>`_,
-    e.g., ``nvcr.io/nvidia/tensorrt-llm/release:<version>``.
+    e.g., ``nvcr.io/nvidia/tensorrt-llm/release:<version>`` (Model Optimizer pre-installed).
 
     Make sure to upgrade Model Optimizer to the latest version using ``pip`` as described in the next section.
 
-    You would also need to setup appropriate environment variables for the TensorRT binaries as follows:
+    If relevant, you would also need to setup appropriate environment variables for the TensorRT binaries as follows:
 
     .. code-block:: shell
 
@@ -48,11 +48,16 @@ Environment setup
     **Alternative NVIDIA docker images**
 
     For PyTorch, you can also use `NVIDIA NGC PyTorch container <https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch/tags>`_
-    and for NVIDIA Megatron-Bridge or Megatron-LM framework, you can use the `NeMo container <https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo/tags>`_.
-    Both of these containers come with Model Optimizer pre-installed. Make sure to update the Model Optimizer to the latest version if not already.
+    (``nvcr.io/nvidia/pytorch:<version>-py3``, Model Optimizer pre-installed)
+    and for NVIDIA Megatron-Bridge or Megatron-LM framework, you can use the `NeMo container <https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo/tags>`_
+    (``nvcr.io/nvidia/nemo:<version>``, Model Optimizer pre-installed).
+    Make sure to update the Model Optimizer to the latest version if not already.
 
     For ONNX / TensorRT use cases, you can also use the `TensorRT container <https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tensorrt/tags>`_
-    which provides superior performance to the PyTorch container.
+    (``nvcr.io/nvidia/tensorrt:<version>-py3``), which provides superior performance to the PyTorch container.
+
+    .. note::
+        Before pulling and using the container images, please review their respective license terms.
 
 .. tab:: Local environment (PIP / Conda)
 
@@ -82,8 +87,8 @@ Environment setup
 Install Model Optimizer
 =======================
 
-ModelOpt including its dependencies can be installed via ``pip``. Please review the license terms of ModelOpt and any
-dependencies before use.
+Model Optimizer will download and install additional third-party open source software projects. Review the license
+terms of these open source projects before use.
 
 If you build and use ModelOpt's docker image, you can skip this step as the image already contains ModelOpt and all
 optional dependencies pre-installed.

diff --git a/docs/source/getting_started/windows/_installation_for_Windows.rst b/docs/source/getting_started/windows/_installation_for_Windows.rst
@@ -30,6 +30,10 @@ The following system requirements are necessary to install and use Model Optimiz
 
 The Model Optimizer - Windows can be used in following ways:
 
+.. note::
+   Model Optimizer will download and install additional third-party open source software projects.
+   Review the license terms of these open source projects before use.
+
 .. toctree::
    :glob:
    :maxdepth: 1

@@ -38,15 +38,15 @@
 # limitations under the License.
 import contextlib
 import warnings
+from importlib.metadata import version
 
 import datasets
-import lm_eval
 from lm_eval import utils
 from lm_eval.__main__ import cli_evaluate, parse_eval_args, setup_parser
 
-if not lm_eval.__version__.startswith("0.4.8"):
+if not version("lm_eval").startswith("0.4.8"):
     warnings.warn(
-        f"lm_eval_hf.py is tested with lm-eval 0.4.8; found {lm_eval.__version__}. "
+        f"lm_eval_hf.py is tested with lm-eval 0.4.8; found {version('lm_eval')}. "
         "Later versions may have incompatible API changes."
     )
 from lm_eval.api.model import T

@@ -46,8 +46,6 @@
 except ImportError:
     snapshot_download = None
 
-from modelopt.torch.utils.image_processor import BaseImageProcessor, MllamaImageProcessor
-
 logger = logging.getLogger(__name__)
 
 SPECULATIVE_MODEL_LIST = ["Eagle", "Medusa"]
@@ -285,13 +283,10 @@ def get_tokenizer(ckpt_path, trust_remote_code=False, **kwargs) -> PreTrainedTok
 def get_processor(
     ckpt_path,
     model_type,
-    device: torch.device = "auto",
     trust_remote_code=False,
     attn_implementation=None,
-) -> BaseImageProcessor | ProcessorMixin | None:
-    """
-    Returns a :class:`modelopt.torch.utils.image_processor.MllamaImageProcessor` object.
-    """
+) -> ProcessorMixin | None:
+    """Load a processor appropriate for the given model type."""
     model_kwargs = {"trust_remote_code": trust_remote_code}
     if attn_implementation is not None:
         model_kwargs["attn_implementation"] = attn_implementation
@@ -309,19 +304,6 @@ def get_processor(
         )
 
         return processor
-    elif model_type == "mllama":
-        processor = AutoProcessor.from_pretrained(
-            ckpt_path,
-            padding_side="left",
-            **model_kwargs,
-        )
-        if processor.tokenizer.pad_token is None:
-            processor.tokenizer.pad_token = processor.tokenizer.eos_token
-        assert processor.tokenizer.pad_token is not None, (
-            f"Pad token for {ckpt_path} cannot be set!"
-        )
-
-        return MllamaImageProcessor(processor, device)
     else:
         # Try to load AutoProcessor for other VL models (e.g., Nemotron-Parse)
         try:

@@ -77,7 +77,6 @@
     get_max_batch_size,
     get_supported_datasets,
 )
-from modelopt.torch.utils.image_processor import BaseImageProcessor, MllamaImageProcessor
 from modelopt.torch.utils.memory_monitor import launch_memory_monitor
 from modelopt.torch.utils.speech_dataset_utils import get_speech_dataset_dataloader
 from modelopt.torch.utils.vlm_dataset_utils import get_vlm_dataset_dataloader
@@ -202,7 +201,7 @@ def _to_device(value):
 def make_calib_dataloader(
     args: argparse.Namespace,
     language_model: torch.nn.Module,
-    processor: BaseImageProcessor | ProcessorMixin | None,
+    processor: ProcessorMixin | None,
     tokenizer: PreTrainedTokenizerBase | None,
     device: torch.device,
     model_type: str | None,
@@ -250,19 +249,6 @@ def make_calib_dataloader(
             use_media_shards=True,
             max_shards=1,
         )
-    elif model_type == "mllama":
-        assert processor is not None and isinstance(processor, MllamaImageProcessor), (
-            "The MllamaImageProcessor must be set."
-        )
-        assert len(args.calib_size) == 1, (
-            "mllama only supports one dataset for calibration, can extend this in the future"
-        )
-        calib_dataloader = get_vlm_dataset_dataloader(
-            dataset_name=args.dataset[0] if args.dataset else "scienceqa",
-            processor=processor,
-            batch_size=args.batch_size,
-            num_samples=args.calib_size[0],
-        )
     elif model_type == "whisper":
         assert processor is not None and isinstance(processor, WhisperProcessor), (
             "The AutoProcessor must be set."
@@ -292,6 +278,7 @@ def make_calib_dataloader(
             tokenizer=tokenizer,
             batch_size=args.batch_size,
             num_samples=args.calib_size,
+            max_sample_length=args.calib_seq,
             device=device,
             include_labels=include_labels,
         )
@@ -472,23 +459,14 @@ def load_model(args: argparse.Namespace):
         print("Nemotron VL model detected. Enabling image-text calibration by default.")
         args.calib_with_images = True
 
-    if model_type == "mllama":
+    if model_type == "whisper":
         processor = get_processor(
             args.pyt_ckpt_path,
             model_type,
-            device,
             trust_remote_code=args.trust_remote_code,
-            attn_implementation=args.attn_implementation,
         )
-    elif model_type == "whisper":
-        processor = get_processor(
-            args.pyt_ckpt_path,
-            model_type,
-            device,
-            trust_remote_code=args.trust_remote_code,
-        )
-    elif is_nemotron_vl_model and args.calib_with_images:
-        # For Nemotron VL image calibration, we need an AutoProcessor to build multimodal inputs.
+    elif args.calib_with_images:
+        # For VLM image calibration, we need an AutoProcessor to build multimodal inputs.
         processor = AutoProcessor.from_pretrained(
             args.pyt_ckpt_path,
             trust_remote_code=args.trust_remote_code,
@@ -715,13 +693,6 @@ def export_quantized(
                 print(f"Warning: Could not save processor config: {e}")
                 print("This is normal for some VLM architectures that don't use AutoProcessor")
 
-        if model_type == "mllama":
-            full_model_config = full_model.config
-            # TRT-LLM expects both the vision_config and text_config to be set for export.
-            setattr(full_model.config, "vision_config", full_model_config.vision_config)
-            setattr(full_model.config, "text_config", full_model_config.text_config)
-            setattr(full_model.config, "architectures", full_model_config.architectures)
-
         start_time = time.time()
         if (
             model_type in ["t5", "bart", "whisper"]
@@ -858,7 +829,7 @@ def post_quantize(
     language_model: torch.nn.Module,
     model_type: str | None,
     tokenizer: PreTrainedTokenizerBase | None,
-    processor: BaseImageProcessor | ProcessorMixin | None,
+    processor: ProcessorMixin | None,
     preview_input_ids,
     generated_ids_before_ptq,
     is_nemotron_vl_model,
@@ -921,9 +892,7 @@ def post_quantize(
         )
 
     def input_decode(input_ids):
-        if processor is not None and isinstance(processor, MllamaImageProcessor):
-            return processor.tokenizer.batch_decode(input_ids)
-        elif processor is not None and isinstance(processor, WhisperProcessor):
+        if processor is not None and isinstance(processor, WhisperProcessor):
             return first_text_speech_dataset
         elif tokenizer is not None:
             return tokenizer.batch_decode(input_ids)
@@ -936,8 +905,6 @@ def output_decode(generated_ids, input_shape):
                 return processor.tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
             elif tokenizer is not None:
                 return tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
-        elif processor is not None and isinstance(processor, MllamaImageProcessor):
-            return processor.tokenizer.batch_decode(generated_ids[:, input_shape:])
         elif tokenizer is not None:
             return tokenizer.batch_decode(generated_ids[:, input_shape:])
         else:
@@ -982,7 +949,7 @@ def quantize_main(
     language_model: torch.nn.Module,
     model_type: str | None,
     calibration_only: bool,
-    processor: BaseImageProcessor | ProcessorMixin | None,
+    processor: ProcessorMixin | None,
     tokenizer: PreTrainedTokenizerBase | None,
     default_padding_side,
     default_pad_token,

@@ -270,19 +270,59 @@ For a quick smoke test, add `--limit 10`.
 > **Alternative:** For server-based evaluation via an OpenAI-compatible endpoint,
 > see [evaluation/nemo_evaluator_instructions.md](./evaluation/nemo_evaluator_instructions.md).
 
-## Inference Performance Benchmarking
+## Deploy compressed model in vLLM
 
-Now let's evaluate how much speedup we get with the compressed model in terms of throughput and latency.
+To deploy a compressed model in vLLM, install vLLM fork with AnyModel enabled:
+
+```bash
+git clone https://github.com/askliar/vllm.git
+cd vllm
+git checkout feature/add_anymodel_to_vllm
+VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto
+```
+
+See [vLLM documentation](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source) for more details on installation.
+
+**NOTE:** This is a temporary workaround pending official vLLM integration. You can track merge status [here](https://github.com/vllm-project/vllm/pull/36512).
+
+Then, add the following to the model's `config.json` file (here we use Llama as an example):
 
-- Install [vLLM from source](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source).
-- Rearrange the model safetensors to be used for vLLM.
+```json
+{
+  ...
+  "architectures": ["AnyModel"],
+  "base_architecture": "LlamaForCausalLM",
+  ...
+}
+```
+
+For new architectures that are not supported by vLLM, you additionally need to add the following to the `config.json` file (using Llama3 as an example):
+
+```json
+{
+  ...
+  "anymodel_arch_info": {
+    "decoder_layer_module": ".<module_name>",
+    "decoder_layer_class": "<decoder_layer_class_name>",
+    "base_model_module": ".<base_model_module_name>",
+    "layers_path": "<layers_path>",
+    "init_prefix": "model",
+    "Layer_hf_config": "<Layer_hf_config>"
+  }
+  ...
+}
+```
+
+With these changes it is now possible to load the compressed model in vLLM for inference:
 
 ```bash
-cd path/to/model
-mv subblocks_safetensors/* .
-sed -i 's+subblocks_safetensors/++g' model.safetensors.index.json
+vllm serve <model_name_or_path>
 ```
 
+### Inference Performance Benchmarking
+
+Now let's evaluate how much speedup we get with the compressed model in terms of throughput and latency.
+
 - Benchmark latency
 
 ```bash