Environment
- TensorRT-LLM version: 1.0.0rc3 and 1.0.0rc4
- GPU: 2x NVIDIA RTX 4090 (24GB VRAM each)
- Model: Qwen3-32B-FP8
- Parallelism: TP=2
Description
After upgrading from v0.21.0rc1 to v1.0.0rc3, loading the Qwen3-32B-FP8 model with TP=2 fails with OOM errors during model initialization.
-
v0.21.0rc1 behavior (expected):
Model load consumes ~18GB VRAM per GPU (total 36GB/48GB available).
-
v1.0.0rc3 and later behavior (regression):
Model load attempts to allocate >24GB VRAM per GPU during initialization, exceeding available memory and causing OOM.
Loading safetensors weights in parallel: 100%|██████████| 7/7 [00:00<00:00, 33.52it/s]
Loading safetensors weights in parallel: 100%|██████████| 7/7 [00:00<00:00, 43.78it/s]
Loading weights concurrently: 47%|████▋ | 667/1417 [00:03<00:05, 139.82it/s][07/23/2025-06:37:06] [TRT-LLM] [RANK 1] [E] Error executing load_single_module with args ('model.layers.29.mlp.gate_up_proj', Linear()): CUDA out of memory. Tried to allocate 126.00 MiB. GPU 0 has a total capacity of 23.53 GiB of which 43.00 MiB is free. Process 32689 has 404.00 MiB memory in use. Process 33095 has 19.44 GiB memory in use. Process 33096 has 3.63 GiB memory in use. Of the allocated memory 2.98 GiB is allocated by PyTorch, and 281.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Loading weights concurrently: 98%|█████████▊| 1391/1417 [00:06<00:00, 205.30it/s][07/23/2025-06:37:08] [TRT-LLM] [RANK 0] [E] Error executing load_single_module with args ('lm_head', LMHead(
(all_reduce): AllReduce()
)): CUDA out of memory. Tried to allocate 742.00 MiB. GPU 0 has a total capacity of 23.53 GiB of which 523.00 MiB is free. Process 32689 has 404.00 MiB memory in use. Process 33095 has 19.01 GiB memory in use. Process 33096 has 3.59 GiB memory in use. Of the allocated memory 17.99 GiB is allocated by PyTorch, and 583.68 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[07/23/2025-06:37:08] [TRT-LLM] [RANK 0] [E] Failed to initialize executor on rank 0: Executor creation failed due to insufficient GPU memory.
This is unexpected since loading Qwen3-32B-FP8 consumes ~16GB VRAM per GPU theoretically and identical configuration worked in the previous version.
Reproduction Steps
# pull docker image
docker pull nvcr.io/nvidia/tensorrt-llm/release:1.0.0rc3
# in docker env
trtllm-serve serve \
Qwen/Qwen3-32B-FP8 \
--host 0.0.0.0 --port 8001 \
--backend pytorch --max_batch_size 16 --max_seq_len 8192 \
--max_num_tokens 4096 --tp_size 2 --pp_size 1 \
--kv_cache_free_gpu_memory_fraction 0.7
Environment
Description
After upgrading from v0.21.0rc1 to v1.0.0rc3, loading the Qwen3-32B-FP8 model with TP=2 fails with OOM errors during model initialization.
v0.21.0rc1 behavior (expected):
Model load consumes ~18GB VRAM per GPU (total 36GB/48GB available).
v1.0.0rc3 and later behavior (regression):
Model load attempts to allocate >24GB VRAM per GPU during initialization, exceeding available memory and causing OOM.
This is unexpected since loading Qwen3-32B-FP8 consumes ~16GB VRAM per GPU theoretically and identical configuration worked in the previous version.
Reproduction Steps