Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 13 additions & 35 deletions .agents/skills/deployment/references/trtllm.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,46 +42,24 @@ llm = LLM(model="<checkpoint_path>", tensor_parallel_size=4)

## AutoDeploy (for AutoQuant / mixed-precision)

AutoDeploy automates graph transformations for optimized inference. Required for AutoQuant checkpoints.
AutoDeploy automates graph transformations for optimized inference and is useful for
AutoQuant / mixed-precision checkpoints. The standalone `examples/llm_autodeploy` example
was removed in 0.46; use TensorRT-LLM's
[AutoDeploy](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/auto_deploy)
directly together with a ModelOpt-quantized checkpoint.

### End-to-end script
### Workflow

```bash
# Quantize and deploy in one step
./examples/llm_autodeploy/scripts/run_auto_quant_and_deploy.sh \
--hf_ckpt <model_path> \
--save_quantized_ckpt <output_path> \
--quant fp8,nvfp4 \
--effective_bits 4.5
```

Parameters:

- `--hf_ckpt`: Path to unquantized HuggingFace checkpoint
- `--save_quantized_ckpt`: Output path for quantized checkpoint
- `--quant`: Quantization formats (e.g., `fp8,nvfp4`)
- `--effective_bits`: Target precision (higher = more accuracy for sensitive layers)
- `--world_size`: Number of GPUs for tensor parallelism
- `--calib_batch_size`: Calibration batch size (reduce if OOM, default 8)

### AutoDeploy API server

```python
# examples/llm_autodeploy/api_server.py provides a FastAPI server
# with OpenAI-compatible endpoints using AutoDeploy
```

### Test AutoDeploy

```bash
python examples/llm_autodeploy/api_client.py --prompt "What is AI?" "What is golf?"
```
1. Quantize the checkpoint with ModelOpt PTQ (including AutoQuant / mixed precision) via
`examples/llm_ptq` (`hf_ptq.py` / `scripts/huggingface_example.sh`), which produces a
unified HuggingFace checkpoint with `hf_quant_config.json`.
2. Deploy that checkpoint with TensorRT-LLM's AutoDeploy backend (see the upstream
`examples/auto_deploy` docs for the current API and `trtllm-serve` flags).

### Notes

- NVFP4 in AutoDeploy requires Blackwell GPUs
- For Hopper: remove `nvfp4` from `--quant` and set `--effective_bits` above 8.0
- AutoDeploy supports CUDA graphs, torch compile backends, and KV cache optimization
- NVFP4 in AutoDeploy requires Blackwell GPUs; on Hopper use FP8 instead.
- AutoDeploy supports CUDA graphs, torch compile backends, and KV cache optimization.

## Legacy TRT-LLM Checkpoint (deprecated)

Expand Down
10 changes: 4 additions & 6 deletions .agents/skills/deployment/scripts/deploy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -337,12 +337,10 @@ start_trtllm() {

cat <<TRTEOF

# Option 1: AutoDeploy (recommended)
./examples/llm_autodeploy/scripts/run_auto_quant_and_deploy.sh \\
--hf_ckpt "$MODEL" \\
--save_quantized_ckpt <output_path> \\
--quant fp8,nvfp4 \\
--effective_bits 4.5
# Option 1: AutoDeploy (recommended for AutoQuant / mixed-precision)
# Quantize with ModelOpt PTQ (examples/llm_ptq) to produce a unified HF checkpoint,
# then deploy it with TensorRT-LLM's AutoDeploy backend:
# https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/auto_deploy

# Option 2: Python API
python3 -c "
Expand Down
1 change: 0 additions & 1 deletion .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,6 @@ modelopt_recipes @NVIDIA/modelopt-recipes-codeowners
/examples/deepseek @NVIDIA/modelopt-deploy-codeowners
/examples/diffusers @NVIDIA/modelopt-examples-diffusers-codeowners
/examples/gpt-oss @NVIDIA/modelopt-examples-gpt-oss-codeowners
/examples/llm_autodeploy @NVIDIA/modelopt-deploy-codeowners
/examples/llm_distill @NVIDIA/modelopt-torch-distill-codeowners
/examples/llm_eval @NVIDIA/modelopt-examples-llm_ptq-codeowners
/examples/llm_ptq @NVIDIA/modelopt-examples-llm_ptq-codeowners
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/example_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ jobs:
pip_install_extras: "[hf,dev-test]"
runner: ${{ startsWith(github.ref, 'refs/heads/pull-request/') && 'linux-amd64-gpu-rtxpro6000-latest-1' || 'linux-amd64-gpu-rtxpro6000-latest-2' }}

##### TensorRT-LLM Example Tests (pr/non-pr split: non-pr runs extra autodeploy+eval examples) #####
##### TensorRT-LLM Example Tests (pr/non-pr split: non-pr runs extra eval examples) #####
trtllm-pr:
needs: [pr-gate]
if: startsWith(github.ref, 'refs/heads/pull-request/') && needs.pr-gate.outputs.any_changed == 'true'
Expand All @@ -69,7 +69,7 @@ jobs:
strategy:
fail-fast: false
matrix:
example: [llm_autodeploy, llm_eval, llm_ptq]
example: [llm_eval, llm_ptq]
uses: ./.github/workflows/_example_tests_runner.yml
secrets: inherit
with:
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ Changelog
**Backward Breaking Changes**

- Remove the ``examples/diffusers/eval`` image-quality evaluation example (ImageReward / CLIP-IQA / CLIP metrics) and its references in ``examples/diffusers/README.md``. The example was deprecated in 0.45 and is no longer maintained.
- Remove the deprecated ``examples/llm_autodeploy`` example (deprecated in 0.45). Use TensorRT-LLM's `AutoDeploy <https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/auto_deploy>`_ directly together with ModelOpt PTQ in ``examples/llm_ptq``.

**Deprecations**

Expand Down
56 changes: 0 additions & 56 deletions examples/llm_autodeploy/README.md

This file was deleted.

61 changes: 0 additions & 61 deletions examples/llm_autodeploy/api_client.py

This file was deleted.

Loading
Loading