[https://nvbugs/6374873][fix] Allow fp4 KV Cache + non-FP4 Mamba State#16304
Merged
VALLIS-NERIA merged 6 commits intoJul 15, 2026
Merged
Conversation
…ent) Signed-off-by: Xiwen Yu <13230610+VALLIS-NERIA@users.noreply.github.com>
…gent) Signed-off-by: Xiwen Yu <13230610+VALLIS-NERIA@users.noreply.github.com>
…on (By Agent) Signed-off-by: Xiwen Yu <13230610+VALLIS-NERIA@users.noreply.github.com>
…(By Agent) Signed-off-by: Xiwen Yu <13230610+VALLIS-NERIA@users.noreply.github.com>
Signed-off-by: Xiwen Yu <13230610+VALLIS-NERIA@users.noreply.github.com>
Collaborator
Author
|
/bot run |
VALLIS-NERIA
changed the title
Collaborator
|
PR_Github #58947 [ run ] triggered by Bot. Commit: |
Contributor
📝 WalkthroughWalkthroughCppMambaHybridCacheManager now passes explicit per-window pool configurations. KVCacheManager aligns NVFP4 scale pointers with physical pool mappings, with new C++ and Python tests covering mixed dtypes, pool allocation, mappings, and zero-local-Mamba cases. ChangesMixed KV cache pool handling
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant CppMambaHybridCacheManager
participant KVCacheManager
participant BlockManager
CppMambaHybridCacheManager->>KVCacheManager: pass per-window pool_configurations
KVCacheManager->>BlockManager: create typed cache pools
KVCacheManager->>KVCacheManager: align NVFP4 scale pointers with pool mapping
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tensorrt_llm/_torch/pyexecutor/resource_manager.py`:
- Around line 648-655: Update the dtype_by_window construction in the
kv_cache_block_scale_pool_pointers path to handle empty
self.impl.pool_configurations without raising. Reuse the manager-level
dtype/size fallback established by calculateMaxNumBlocks(), while preserving
per-window configuration values when pool_configurations is populated.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: c287418f-6632-4171-874b-ae433143195a
📒 Files selected for processing (5)
cpp/tests/unit_tests/batch_manager/kvCacheManagerTest.cpptensorrt_llm/_torch/pyexecutor/mamba_cache_manager.pytensorrt_llm/_torch/pyexecutor/resource_manager.pytests/unittest/_torch/executor/test_mamba_cache_manager.pytests/unittest/_torch/executor/test_resource_manager.py
Wanli-Jiang
approved these changes
Jul 13, 2026
…Agent) Signed-off-by: Xiwen Yu <13230610+VALLIS-NERIA@users.noreply.github.com>
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #58960 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #58947 [ run ] completed with state |
Collaborator
|
PR_Github #58960 [ run ] completed with state
|
eopXD
approved these changes
Jul 13, 2026
eopXD
left a comment
eopXD
left a comment
Collaborator
There was a problem hiding this comment.
The fix looks good to me.
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #59059 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #59059 [ run ] completed with state |
Tabrizian
approved these changes
Jul 14, 2026
VALLIS-NERIA
enabled auto-merge (squash)
July 15, 2026 01:42
Wanli-Jiang
added a commit
to Wanli-Jiang/TensorRT-LLM
that referenced
this pull request
Jul 15, 2026
Sever the C++ tree's compile- and link-time dependency on TensorRT so the shared core (runtime, batch manager, executor API, KV cache, sampling, kernels, nanobind bridge) builds, links, and runs without the TensorRT library. Follows the Python TensorRT-backend removal (TRTLLM-14022); PyTorch is the sole backend. Internal types (serialization-compatible): - add tensorrt_llm::DataType/Dims/ILogger (common/tllmDataType.h); DataType enumerator values mirror nvinfer1::DataType for byte-compatible serialization, Dims layout mirrors nvinfer1::Dims - migrate nvinfer1::DataType/Dims/ILogger -> tensorrt_llm:: across 206 surviving files; replace NvInfer*.h includes with the internal header; drop 12 dead NvInfer includes from files that never referenced nvinfer1 - add tests/unittest/bindings/test_datatype_parity.py guarding the enumerator values (auto-scheduled via the existing unittest/bindings test-db entries) Remove the TensorRT-engine execution path (unused by the PyTorch backend): - plugins/ (nvinfer_plugin_tensorrt_llm), engine runtime wrappers (tllmRuntime, tllmStreamReaders, layerProfiler, rawEngine, tllmLogger), TRT model adapters (trtGptModel*/trtEncoderModel/trtGptModelFactory), executor.cpp/executorImpl, the C++ Executor + TllmRuntime + LogitsPostProcessor nanobind bindings (0 Python users each), executor_worker, disaggServerUtil, engine I/O buffers and engine-only logits/decoder algos, executor/model.h (0 consumers), the ModelSpec test helper + binding (orphaned) - relocate the retained KVCacheEvent ctor and executor::version() into executor/kvCacheEvent.cpp - decouple shared files (inflightBatchingUtils, medusaBuffers, lookaheadBuffers, dataTransceiver) from the removed engine runtime - delete the engine-driven C++ tests (e2e_tests engine tests, executorTestSmall*, tllmRuntimeTest, encDecBeamSearchTest, tests/utils engine builders), cudaGraphExecutorCacheTest (tests the removed CudaGraphExecutor), unit_tests/utils (tests the deleted tests/utils helpers), tests/unittest/others/test_leak.py (its body calls the removed python graph-building APIs; unscheduled in CI), C++ benchmarks (bertBenchmark, gptManagerBenchmark, disaggServerBenchmark; the prepare_dataset.py tooling used by trtllm-bench stays), examples/cpp executor examples, and the cpp/tests/resources engine-build scripts - fix kept tests: strip vestigial never-consumed TllmLogger members (7 files), repoint ropeTest.cu to kernels/gptKernels.h (+ cudaUtils.h for QuantMode/getSMVersion), inline the engine-free createDecoderBatchInputs helper into gptDecoderBatchedTest - make dependencies the deleted plugin target satisfied transitively explicit: MPI include dirs for cpp/tests, th_utils -> tensorrt_llm shared-lib link - remove the IS_BUILDING build-time env contract end to end (common/opUtils.h isBuilding + attentionOp gate + _common.py half) - remove the llm_args=None engine path from executor/base_worker.py (tllm.Executor no longer exists); llm_args is now required Build/packaging (no TensorRT): - drop find_package(TensorRT)/TRT_LIB/NvInfer include injection and the plugins/executor_worker/benchmarks subdirs from the cpp CMake; delete FindTensorRT.cmake - build_wheel.py: trt_root optional (default None), drop the tensorrt venv check and the nvinfer_plugin/executorWorker/benchmarks targets - setup.py no longer packages libnvinfer_plugin_tensorrt_llm.so / executorWorker; requirements.txt drops tensorrt - jenkins/Build.groovy: drop --benchmarks, the benchmark/libnvinfer-plugin tarball packaging, and the build_cpp_examples.py step - delist the removed tests from test-db/waives.txt/.test_durations and prune tests/integration/defs/cpp to the shared-core gtest wrappers - CI-run fixes: respell the nvinfer1 usages a post-rebase main commit (NVIDIA#16304) added to kvCacheManagerTest; retire the two CI stages emptied by the delisting (A10-CPP-1, H100_PCIe-CPP-Post-Merge-1) and drop the duplicate tensorrt-chunk scheduling of unittest/disaggregated/test_router.py; make test_to_stage_mapping's CLI check sample only stage-mapped tests - point the remaining benchmarks/cpp references (AutoDeploy bench/dist tests, llmc standalone packager + its test, trtllm-bench docs) at benchmarks/ and drop the orphaned get_cpp_benchmark() helper - fix the two OSS-mode gtest compile failures unmasked by dropping the nvinfer_plugin link (its PUBLIC USING_OSS_CUTLASS_*_GEMM defines had leaked into every gtest, keeping the internal-only branches dormant): inline the deleted tests/utils GpuTimer into gemmAllReduceTest.cu, and only add the mixtureOfExpertsInternalTest target when INTERNAL_CUTLASS_KERNELS_PATH provides the internal headers it includes Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
Wanli-Jiang
added a commit
to Wanli-Jiang/TensorRT-LLM
that referenced
this pull request
Jul 15, 2026
Sever the C++ tree's compile- and link-time dependency on TensorRT so the shared core (runtime, batch manager, executor API, KV cache, sampling, kernels, nanobind bridge) builds, links, and runs without the TensorRT library. Follows the Python TensorRT-backend removal (TRTLLM-14022); PyTorch is the sole backend. Internal types (serialization-compatible): - add tensorrt_llm::DataType/Dims/ILogger (common/tllmDataType.h); DataType enumerator values mirror nvinfer1::DataType for byte-compatible serialization, Dims layout mirrors nvinfer1::Dims - migrate nvinfer1::DataType/Dims/ILogger -> tensorrt_llm:: across 206 surviving files; replace NvInfer*.h includes with the internal header; drop 12 dead NvInfer includes from files that never referenced nvinfer1 - add tests/unittest/bindings/test_datatype_parity.py guarding the enumerator values (auto-scheduled via the existing unittest/bindings test-db entries) Remove the TensorRT-engine execution path (unused by the PyTorch backend): - plugins/ (nvinfer_plugin_tensorrt_llm), engine runtime wrappers (tllmRuntime, tllmStreamReaders, layerProfiler, rawEngine, tllmLogger), TRT model adapters (trtGptModel*/trtEncoderModel/trtGptModelFactory), executor.cpp/executorImpl, the C++ Executor + TllmRuntime + LogitsPostProcessor nanobind bindings (0 Python users each), executor_worker, disaggServerUtil, engine I/O buffers and engine-only logits/decoder algos, executor/model.h (0 consumers), the ModelSpec test helper + binding (orphaned) - relocate the retained KVCacheEvent ctor and executor::version() into executor/kvCacheEvent.cpp - decouple shared files (inflightBatchingUtils, medusaBuffers, lookaheadBuffers, dataTransceiver) from the removed engine runtime - delete the engine-driven C++ tests (e2e_tests engine tests, executorTestSmall*, tllmRuntimeTest, encDecBeamSearchTest, tests/utils engine builders), cudaGraphExecutorCacheTest (tests the removed CudaGraphExecutor), unit_tests/utils (tests the deleted tests/utils helpers), tests/unittest/others/test_leak.py (its body calls the removed python graph-building APIs; unscheduled in CI), C++ benchmarks (bertBenchmark, gptManagerBenchmark, disaggServerBenchmark; the prepare_dataset.py tooling used by trtllm-bench stays), examples/cpp executor examples, and the cpp/tests/resources engine-build scripts - fix kept tests: strip vestigial never-consumed TllmLogger members (7 files), repoint ropeTest.cu to kernels/gptKernels.h (+ cudaUtils.h for QuantMode/getSMVersion), inline the engine-free createDecoderBatchInputs helper into gptDecoderBatchedTest - make dependencies the deleted plugin target satisfied transitively explicit: MPI include dirs for cpp/tests, th_utils -> tensorrt_llm shared-lib link - remove the IS_BUILDING build-time env contract end to end (common/opUtils.h isBuilding + attentionOp gate + _common.py half) - remove the llm_args=None engine path from executor/base_worker.py (tllm.Executor no longer exists); llm_args is now required Build/packaging (no TensorRT): - drop find_package(TensorRT)/TRT_LIB/NvInfer include injection and the plugins/executor_worker/benchmarks subdirs from the cpp CMake; delete FindTensorRT.cmake - build_wheel.py: trt_root optional (default None), drop the tensorrt venv check and the nvinfer_plugin/executorWorker/benchmarks targets - setup.py no longer packages libnvinfer_plugin_tensorrt_llm.so / executorWorker; requirements.txt drops tensorrt - jenkins/Build.groovy: drop --benchmarks, the benchmark/libnvinfer-plugin tarball packaging, and the build_cpp_examples.py step - delist the removed tests from test-db/waives.txt/.test_durations and prune tests/integration/defs/cpp to the shared-core gtest wrappers - CI-run fixes: respell the nvinfer1 usages a post-rebase main commit (NVIDIA#16304) added to kvCacheManagerTest; retire the two CI stages emptied by the delisting (A10-CPP-1, H100_PCIe-CPP-Post-Merge-1) and drop the duplicate tensorrt-chunk scheduling of unittest/disaggregated/test_router.py; make test_to_stage_mapping's CLI check sample only stage-mapped tests - point the remaining benchmarks/cpp references (AutoDeploy bench/dist tests, llmc standalone packager + its test, trtllm-bench docs) at benchmarks/ and drop the orphaned get_cpp_benchmark() helper - fix the two OSS-mode gtest compile failures unmasked by dropping the nvinfer_plugin link (its PUBLIC USING_OSS_CUTLASS_*_GEMM defines had leaked into every gtest, keeping the internal-only branches dormant): inline the deleted tests/utils GpuTimer into gemmAllReduceTest.cu, and only add the mixtureOfExpertsInternalTest target when INTERNAL_CUTLASS_KERNELS_PATH provides the internal headers it includes Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
Wanli-Jiang
added a commit
to Wanli-Jiang/TensorRT-LLM
that referenced
this pull request
Jul 15, 2026
Sever the C++ tree's compile- and link-time dependency on TensorRT so the shared core (runtime, batch manager, executor API, KV cache, sampling, kernels, nanobind bridge) builds, links, and runs without the TensorRT library. Follows the Python TensorRT-backend removal (TRTLLM-14022); PyTorch is the sole backend. Internal types (serialization-compatible): - add tensorrt_llm::DataType/Dims/ILogger (common/tllmDataType.h); DataType enumerator values mirror nvinfer1::DataType for byte-compatible serialization, Dims layout mirrors nvinfer1::Dims - migrate nvinfer1::DataType/Dims/ILogger -> tensorrt_llm:: across 206 surviving files; replace NvInfer*.h includes with the internal header; drop 12 dead NvInfer includes from files that never referenced nvinfer1 - add tests/unittest/bindings/test_datatype_parity.py guarding the enumerator values (auto-scheduled via the existing unittest/bindings test-db entries) Remove the TensorRT-engine execution path (unused by the PyTorch backend): - plugins/ (nvinfer_plugin_tensorrt_llm), engine runtime wrappers (tllmRuntime, tllmStreamReaders, layerProfiler, rawEngine, tllmLogger), TRT model adapters (trtGptModel*/trtEncoderModel/trtGptModelFactory), executor.cpp/executorImpl, the C++ Executor + TllmRuntime + LogitsPostProcessor nanobind bindings (0 Python users each), executor_worker, disaggServerUtil, engine I/O buffers and engine-only logits/decoder algos, executor/model.h (0 consumers), the ModelSpec test helper + binding (orphaned) - relocate the retained KVCacheEvent ctor and executor::version() into executor/kvCacheEvent.cpp - decouple shared files (inflightBatchingUtils, medusaBuffers, lookaheadBuffers, dataTransceiver) from the removed engine runtime - delete the engine-driven C++ tests (e2e_tests engine tests, executorTestSmall*, tllmRuntimeTest, encDecBeamSearchTest, tests/utils engine builders), cudaGraphExecutorCacheTest (tests the removed CudaGraphExecutor), unit_tests/utils (tests the deleted tests/utils helpers), tests/unittest/others/test_leak.py (its body calls the removed python graph-building APIs; unscheduled in CI), C++ benchmarks (bertBenchmark, gptManagerBenchmark, disaggServerBenchmark; the prepare_dataset.py tooling used by trtllm-bench stays), examples/cpp executor examples, and the cpp/tests/resources engine-build scripts - fix kept tests: strip vestigial never-consumed TllmLogger members (7 files), repoint ropeTest.cu to kernels/gptKernels.h (+ cudaUtils.h for QuantMode/getSMVersion), inline the engine-free createDecoderBatchInputs helper into gptDecoderBatchedTest - make dependencies the deleted plugin target satisfied transitively explicit: MPI include dirs for cpp/tests, th_utils -> tensorrt_llm shared-lib link - remove the IS_BUILDING build-time env contract end to end (common/opUtils.h isBuilding + attentionOp gate + _common.py half) - remove the llm_args=None engine path from executor/base_worker.py (tllm.Executor no longer exists); llm_args is now required Build/packaging (no TensorRT): - drop find_package(TensorRT)/TRT_LIB/NvInfer include injection and the plugins/executor_worker/benchmarks subdirs from the cpp CMake; delete FindTensorRT.cmake - build_wheel.py: trt_root optional (default None), drop the tensorrt venv check and the nvinfer_plugin/executorWorker/benchmarks targets - setup.py no longer packages libnvinfer_plugin_tensorrt_llm.so / executorWorker; requirements.txt drops tensorrt - jenkins/Build.groovy: drop --benchmarks, the benchmark/libnvinfer-plugin tarball packaging, and the build_cpp_examples.py step - delist the removed tests from test-db/waives.txt/.test_durations and prune tests/integration/defs/cpp to the shared-core gtest wrappers - CI-run fixes: respell the nvinfer1 usages a post-rebase main commit (NVIDIA#16304) added to kvCacheManagerTest; retire the two CI stages emptied by the delisting (A10-CPP-1, H100_PCIe-CPP-Post-Merge-1) and drop the duplicate tensorrt-chunk scheduling of unittest/disaggregated/test_router.py; make test_to_stage_mapping's CLI check sample only stage-mapped tests - point the remaining benchmarks/cpp references (AutoDeploy bench/dist tests, llmc standalone packager + its test, trtllm-bench docs) at benchmarks/ and drop the orphaned get_cpp_benchmark() helper - fix the two OSS-mode gtest compile failures unmasked by dropping the nvinfer_plugin link (its PUBLIC USING_OSS_CUTLASS_*_GEMM defines had leaked into every gtest, keeping the internal-only branches dormant): inline the deleted tests/utils GpuTimer into gemmAllReduceTest.cu, and only add the mixtureOfExpertsInternalTest target when INTERNAL_CUTLASS_KERNELS_PATH provides the internal headers it includes Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
1 task
Wanli-Jiang
added a commit
to Wanli-Jiang/TensorRT-LLM
that referenced
this pull request
Jul 16, 2026
Sever the C++ tree's compile- and link-time dependency on TensorRT so the shared core (runtime, batch manager, executor API, KV cache, sampling, kernels, nanobind bridge) builds, links, and runs without the TensorRT library. Follows the Python TensorRT-backend removal (TRTLLM-14022); PyTorch is the sole backend. Internal types (serialization-compatible): - add tensorrt_llm::DataType/Dims (common/tllmDataType.h); DataType enumerator values mirror nvinfer1::DataType for byte-compatible serialization, Dims layout mirrors nvinfer1::Dims - migrate nvinfer1::DataType/Dims -> tensorrt_llm:: across 206 surviving files; replace NvInfer*.h includes with the internal header; drop 12 dead NvInfer includes from files that never referenced nvinfer1 - add tests/unittest/bindings/test_datatype_parity.py guarding the enumerator values (auto-scheduled via the existing unittest/bindings test-db entries) Remove the TensorRT-engine execution path (unused by the PyTorch backend): - plugins/ (nvinfer_plugin_tensorrt_llm), engine runtime wrappers (tllmRuntime, tllmStreamReaders, layerProfiler, rawEngine, tllmLogger), TRT model adapters (trtGptModel*/trtEncoderModel/trtGptModelFactory), executor.cpp/executorImpl, the C++ Executor + TllmRuntime + LogitsPostProcessor nanobind bindings (0 Python users each), executor_worker, disaggServerUtil, engine I/O buffers and engine-only logits/decoder algos, executor/model.h (0 consumers), the ModelSpec test helper + binding (orphaned) - relocate the retained KVCacheEvent ctor and executor::version() into executor/kvCacheEvent.cpp - decouple shared files (inflightBatchingUtils, medusaBuffers, lookaheadBuffers, dataTransceiver) from the removed engine runtime - delete the engine-driven C++ tests (e2e_tests engine tests, executorTestSmall*, tllmRuntimeTest, encDecBeamSearchTest, tests/utils engine builders), cudaGraphExecutorCacheTest (tests the removed CudaGraphExecutor), unit_tests/utils (tests the deleted tests/utils helpers), tests/unittest/others/test_leak.py (its body calls the removed python graph-building APIs; unscheduled in CI), C++ benchmarks (bertBenchmark, gptManagerBenchmark, disaggServerBenchmark; the prepare_dataset.py tooling used by trtllm-bench stays), examples/cpp executor examples, and the cpp/tests/resources engine-build scripts - fix kept tests: strip vestigial never-consumed TllmLogger members (7 files), repoint ropeTest.cu to kernels/gptKernels.h (+ cudaUtils.h for QuantMode/getSMVersion), inline the engine-free createDecoderBatchInputs helper into gptDecoderBatchedTest - make dependencies the deleted plugin target satisfied transitively explicit: MPI include dirs for cpp/tests, th_utils -> tensorrt_llm shared-lib link - remove the IS_BUILDING build-time env contract end to end (common/opUtils.h isBuilding + attentionOp gate + _common.py half) - remove the llm_args=None engine path from executor/base_worker.py (tllm.Executor no longer exists); llm_args is now required Build/packaging (no TensorRT): - drop find_package(TensorRT)/TRT_LIB/NvInfer include injection and the plugins/executor_worker/benchmarks subdirs from the cpp CMake; delete FindTensorRT.cmake - build_wheel.py: trt_root optional (default None), drop the tensorrt venv check and the nvinfer_plugin/executorWorker/benchmarks targets - setup.py no longer packages libnvinfer_plugin_tensorrt_llm.so / executorWorker; requirements.txt drops tensorrt - jenkins/Build.groovy: drop --benchmarks, the benchmark/libnvinfer-plugin tarball packaging, and the build_cpp_examples.py step - delist the removed tests from test-db/waives.txt/.test_durations and prune tests/integration/defs/cpp to the shared-core gtest wrappers - CI-run fixes: respell the nvinfer1 usages a post-rebase main commit (NVIDIA#16304) added to kvCacheManagerTest; retire the two CI stages emptied by the delisting (A10-CPP-1, H100_PCIe-CPP-Post-Merge-1) and drop the duplicate tensorrt-chunk scheduling of unittest/disaggregated/test_router.py; make test_to_stage_mapping's CLI check sample only stage-mapped tests - point the remaining benchmarks/cpp references (AutoDeploy bench/dist tests, llmc standalone packager + its test, trtllm-bench docs) at benchmarks/ and drop the orphaned get_cpp_benchmark() helper - fix the two OSS-mode gtest compile failures unmasked by dropping the nvinfer_plugin link (its PUBLIC USING_OSS_CUTLASS_*_GEMM defines had leaked into every gtest, keeping the internal-only branches dormant): inline the deleted tests/utils GpuTimer into gemmAllReduceTest.cu, and only add the mixtureOfExpertsInternalTest target when INTERNAL_CUTLASS_KERNELS_PATH provides the internal headers it includes Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
Wanli-Jiang
added a commit
to Wanli-Jiang/TensorRT-LLM
that referenced
this pull request
Jul 16, 2026
Sever the C++ tree's compile- and link-time dependency on TensorRT so the shared core (runtime, batch manager, executor API, KV cache, sampling, kernels, nanobind bridge) builds, links, and runs without the TensorRT library. Follows the Python TensorRT-backend removal (TRTLLM-14022); PyTorch is the sole backend. Internal types (serialization-compatible): - add tensorrt_llm::DataType/Dims (common/tllmDataType.h); DataType enumerator values mirror nvinfer1::DataType for byte-compatible serialization, Dims layout mirrors nvinfer1::Dims - migrate nvinfer1::DataType/Dims -> tensorrt_llm:: across 206 surviving files; replace NvInfer*.h includes with the internal header; drop 12 dead NvInfer includes from files that never referenced nvinfer1 - add tests/unittest/bindings/test_datatype_parity.py guarding the enumerator values (auto-scheduled via the existing unittest/bindings test-db entries) Remove the TensorRT-engine execution path (unused by the PyTorch backend): - plugins/ (nvinfer_plugin_tensorrt_llm), engine runtime wrappers (tllmRuntime, tllmStreamReaders, layerProfiler, rawEngine, tllmLogger), TRT model adapters (trtGptModel*/trtEncoderModel/trtGptModelFactory), executor.cpp/executorImpl, the C++ Executor + TllmRuntime + LogitsPostProcessor nanobind bindings (0 Python users each), executor_worker, disaggServerUtil, engine I/O buffers and engine-only logits/decoder algos, executor/model.h (0 consumers), the ModelSpec test helper + binding (orphaned) - relocate the retained KVCacheEvent ctor and executor::version() into executor/kvCacheEvent.cpp - decouple shared files (inflightBatchingUtils, medusaBuffers, lookaheadBuffers, dataTransceiver) from the removed engine runtime - delete the engine-driven C++ tests (e2e_tests engine tests, executorTestSmall*, tllmRuntimeTest, encDecBeamSearchTest, tests/utils engine builders), cudaGraphExecutorCacheTest (tests the removed CudaGraphExecutor), unit_tests/utils (tests the deleted tests/utils helpers), tests/unittest/others/test_leak.py (its body calls the removed python graph-building APIs; unscheduled in CI), C++ benchmarks (bertBenchmark, gptManagerBenchmark, disaggServerBenchmark; the prepare_dataset.py tooling used by trtllm-bench stays), examples/cpp executor examples, and the cpp/tests/resources engine-build scripts - fix kept tests: strip vestigial never-consumed TllmLogger members (7 files), repoint ropeTest.cu to kernels/gptKernels.h (+ cudaUtils.h for QuantMode/getSMVersion), inline the engine-free createDecoderBatchInputs helper into gptDecoderBatchedTest - make dependencies the deleted plugin target satisfied transitively explicit: MPI include dirs for cpp/tests, th_utils -> tensorrt_llm shared-lib link - remove the IS_BUILDING build-time env contract end to end (common/opUtils.h isBuilding + attentionOp gate + _common.py half) - remove the llm_args=None engine path from executor/base_worker.py (tllm.Executor no longer exists); llm_args is now required Build/packaging (no TensorRT): - drop find_package(TensorRT)/TRT_LIB/NvInfer include injection and the plugins/executor_worker/benchmarks subdirs from the cpp CMake; delete FindTensorRT.cmake - build_wheel.py: trt_root optional (default None), drop the tensorrt venv check and the nvinfer_plugin/executorWorker/benchmarks targets - setup.py no longer packages libnvinfer_plugin_tensorrt_llm.so / executorWorker; requirements.txt drops tensorrt - jenkins/Build.groovy: drop --benchmarks, the benchmark/libnvinfer-plugin tarball packaging, and the build_cpp_examples.py step - delist the removed tests from test-db/waives.txt/.test_durations and prune tests/integration/defs/cpp to the shared-core gtest wrappers - CI-run fixes: respell the nvinfer1 usages a post-rebase main commit (NVIDIA#16304) added to kvCacheManagerTest; retire the two CI stages emptied by the delisting (A10-CPP-1, H100_PCIe-CPP-Post-Merge-1) and drop the duplicate tensorrt-chunk scheduling of unittest/disaggregated/test_router.py; make test_to_stage_mapping's CLI check sample only stage-mapped tests - point the remaining benchmarks/cpp references (AutoDeploy bench/dist tests, llmc standalone packager + its test, trtllm-bench docs) at benchmarks/ and drop the orphaned get_cpp_benchmark() helper - fix the two OSS-mode gtest compile failures unmasked by dropping the nvinfer_plugin link (its PUBLIC USING_OSS_CUTLASS_*_GEMM defines had leaked into every gtest, keeping the internal-only branches dormant): inline the deleted tests/utils GpuTimer into gemmAllReduceTest.cu, and only add the mixtureOfExpertsInternalTest target when INTERNAL_CUTLASS_KERNELS_PATH provides the internal headers it includes Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
Wanli-Jiang
added a commit
to Wanli-Jiang/TensorRT-LLM
that referenced
this pull request
Jul 16, 2026
Sever the C++ tree's compile- and link-time dependency on TensorRT so the shared core (runtime, batch manager, executor API, KV cache, sampling, kernels, nanobind bridge) builds, links, and runs without the TensorRT library. Follows the Python TensorRT-backend removal (TRTLLM-14022); PyTorch is the sole backend. Internal types (serialization-compatible): - add tensorrt_llm::DataType/Dims (common/tllmDataType.h); DataType enumerator values mirror nvinfer1::DataType for byte-compatible serialization, Dims layout mirrors nvinfer1::Dims - migrate nvinfer1::DataType/Dims -> tensorrt_llm:: across 206 surviving files; replace NvInfer*.h includes with the internal header; drop 12 dead NvInfer includes from files that never referenced nvinfer1 - add tests/unittest/bindings/test_datatype_parity.py guarding the enumerator values (auto-scheduled via the existing unittest/bindings test-db entries) Remove the TensorRT-engine execution path (unused by the PyTorch backend): - plugins/ (nvinfer_plugin_tensorrt_llm), engine runtime wrappers (tllmRuntime, tllmStreamReaders, layerProfiler, rawEngine, tllmLogger), TRT model adapters (trtGptModel*/trtEncoderModel/trtGptModelFactory), executor.cpp/executorImpl, the C++ Executor + TllmRuntime + LogitsPostProcessor nanobind bindings (0 Python users each), executor_worker, disaggServerUtil, engine I/O buffers and engine-only logits/decoder algos, executor/model.h (0 consumers), the ModelSpec test helper + binding (orphaned) - relocate the retained KVCacheEvent ctor and executor::version() into executor/kvCacheEvent.cpp - decouple shared files (inflightBatchingUtils, medusaBuffers, lookaheadBuffers, dataTransceiver) from the removed engine runtime - delete the engine-driven C++ tests (e2e_tests engine tests, executorTestSmall*, tllmRuntimeTest, encDecBeamSearchTest, tests/utils engine builders), cudaGraphExecutorCacheTest (tests the removed CudaGraphExecutor), unit_tests/utils (tests the deleted tests/utils helpers), tests/unittest/others/test_leak.py (its body calls the removed python graph-building APIs; unscheduled in CI), C++ benchmarks (bertBenchmark, gptManagerBenchmark, disaggServerBenchmark; the prepare_dataset.py tooling used by trtllm-bench stays), examples/cpp executor examples, and the cpp/tests/resources engine-build scripts - fix kept tests: strip vestigial never-consumed TllmLogger members (7 files), repoint ropeTest.cu to kernels/gptKernels.h (+ cudaUtils.h for QuantMode/getSMVersion), inline the engine-free createDecoderBatchInputs helper into gptDecoderBatchedTest - make dependencies the deleted plugin target satisfied transitively explicit: MPI include dirs for cpp/tests, th_utils -> tensorrt_llm shared-lib link - remove the IS_BUILDING build-time env contract end to end (common/opUtils.h isBuilding + attentionOp gate + _common.py half) - remove the llm_args=None engine path from executor/base_worker.py (tllm.Executor no longer exists); llm_args is now required Build/packaging (no TensorRT): - drop find_package(TensorRT)/TRT_LIB/NvInfer include injection and the plugins/executor_worker/benchmarks subdirs from the cpp CMake; delete FindTensorRT.cmake - build_wheel.py: trt_root optional (default None), drop the tensorrt venv check and the nvinfer_plugin/executorWorker/benchmarks targets - setup.py no longer packages libnvinfer_plugin_tensorrt_llm.so / executorWorker; requirements.txt drops tensorrt - jenkins/Build.groovy: drop --benchmarks, the benchmark/libnvinfer-plugin tarball packaging, and the build_cpp_examples.py step - delist the removed tests from test-db/waives.txt/.test_durations and prune tests/integration/defs/cpp to the shared-core gtest wrappers - CI-run fixes: respell the nvinfer1 usages a post-rebase main commit (NVIDIA#16304) added to kvCacheManagerTest; retire the two CI stages emptied by the delisting (A10-CPP-1, H100_PCIe-CPP-Post-Merge-1) and drop the duplicate tensorrt-chunk scheduling of unittest/disaggregated/test_router.py; make test_to_stage_mapping's CLI check sample only stage-mapped tests - point the remaining benchmarks/cpp references (AutoDeploy bench/dist tests, llmc standalone packager + its test, trtllm-bench docs) at benchmarks/ and drop the orphaned get_cpp_benchmark() helper - fix the two OSS-mode gtest compile failures unmasked by dropping the nvinfer_plugin link (its PUBLIC USING_OSS_CUTLASS_*_GEMM defines had leaked into every gtest, keeping the internal-only branches dormant): inline the deleted tests/utils GpuTimer into gemmAllReduceTest.cu, and only add the mixtureOfExpertsInternalTest target when INTERNAL_CUTLASS_KERNELS_PATH provides the internal headers it includes Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
Wanli-Jiang
added a commit
to Wanli-Jiang/TensorRT-LLM
that referenced
this pull request
Jul 16, 2026
Sever the C++ tree's compile- and link-time dependency on TensorRT so the shared core (runtime, batch manager, executor API, KV cache, sampling, kernels, nanobind bridge) builds, links, and runs without the TensorRT library. Follows the Python TensorRT-backend removal (TRTLLM-14022); PyTorch is the sole backend. Internal types (serialization-compatible): - add tensorrt_llm::DataType/Dims (common/tllmDataType.h); DataType enumerator values mirror nvinfer1::DataType for byte-compatible serialization, Dims layout mirrors nvinfer1::Dims - migrate nvinfer1::DataType/Dims -> tensorrt_llm:: across 206 surviving files; replace NvInfer*.h includes with the internal header; drop 12 dead NvInfer includes from files that never referenced nvinfer1 - add tests/unittest/bindings/test_datatype_parity.py guarding the enumerator values (auto-scheduled via the existing unittest/bindings test-db entries) Remove the TensorRT-engine execution path (unused by the PyTorch backend): - plugins/ (nvinfer_plugin_tensorrt_llm), engine runtime wrappers (tllmRuntime, tllmStreamReaders, layerProfiler, rawEngine, tllmLogger), TRT model adapters (trtGptModel*/trtEncoderModel/trtGptModelFactory), executor.cpp/executorImpl, the C++ Executor + TllmRuntime + LogitsPostProcessor nanobind bindings (0 Python users each), executor_worker, disaggServerUtil, engine I/O buffers and engine-only logits/decoder algos, executor/model.h (0 consumers), the ModelSpec test helper + binding (orphaned) - relocate the retained KVCacheEvent ctor and executor::version() into executor/kvCacheEvent.cpp - decouple shared files (inflightBatchingUtils, medusaBuffers, lookaheadBuffers, dataTransceiver) from the removed engine runtime - delete the engine-driven C++ tests (e2e_tests engine tests, executorTestSmall*, tllmRuntimeTest, encDecBeamSearchTest, tests/utils engine builders), cudaGraphExecutorCacheTest (tests the removed CudaGraphExecutor), unit_tests/utils (tests the deleted tests/utils helpers), tests/unittest/others/test_leak.py (its body calls the removed python graph-building APIs; unscheduled in CI), C++ benchmarks (bertBenchmark, gptManagerBenchmark, disaggServerBenchmark; the prepare_dataset.py tooling used by trtllm-bench stays), examples/cpp executor examples, and the cpp/tests/resources engine-build scripts - fix kept tests: strip vestigial never-consumed TllmLogger members (7 files), repoint ropeTest.cu to kernels/gptKernels.h (+ cudaUtils.h for QuantMode/getSMVersion), inline the engine-free createDecoderBatchInputs helper into gptDecoderBatchedTest - make dependencies the deleted plugin target satisfied transitively explicit: MPI include dirs for cpp/tests, th_utils -> tensorrt_llm shared-lib link - remove the IS_BUILDING build-time env contract end to end (common/opUtils.h isBuilding + attentionOp gate + _common.py half) - remove the llm_args=None engine path from executor/base_worker.py (tllm.Executor no longer exists); llm_args is now required Build/packaging (no TensorRT): - drop find_package(TensorRT)/TRT_LIB/NvInfer include injection and the plugins/executor_worker/benchmarks subdirs from the cpp CMake; delete FindTensorRT.cmake - build_wheel.py: trt_root optional (default None), drop the tensorrt venv check and the nvinfer_plugin/executorWorker/benchmarks targets - setup.py no longer packages libnvinfer_plugin_tensorrt_llm.so / executorWorker; requirements.txt drops tensorrt - jenkins/Build.groovy: drop --benchmarks, the benchmark/libnvinfer-plugin tarball packaging, and the build_cpp_examples.py step - delist the removed tests from test-db/waives.txt/.test_durations and prune tests/integration/defs/cpp to the shared-core gtest wrappers - CI-run fixes: respell the nvinfer1 usages a post-rebase main commit (NVIDIA#16304) added to kvCacheManagerTest; retire the two CI stages emptied by the delisting (A10-CPP-1, H100_PCIe-CPP-Post-Merge-1) and drop the duplicate tensorrt-chunk scheduling of unittest/disaggregated/test_router.py; make test_to_stage_mapping's CLI check sample only stage-mapped tests - point the remaining benchmarks/cpp references (AutoDeploy bench/dist tests, llmc standalone packager + its test, trtllm-bench docs) at benchmarks/ and drop the orphaned get_cpp_benchmark() helper - fix the two OSS-mode gtest compile failures unmasked by dropping the nvinfer_plugin link (its PUBLIC USING_OSS_CUTLASS_*_GEMM defines had leaked into every gtest, keeping the internal-only branches dormant): inline the deleted tests/utils GpuTimer into gemmAllReduceTest.cu, and only add the mixtureOfExpertsInternalTest target when INTERNAL_CUTLASS_KERNELS_PATH provides the internal headers it includes Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
Wanli-Jiang
added a commit
to Wanli-Jiang/TensorRT-LLM
that referenced
this pull request
Jul 16, 2026
Sever the C++ tree's compile- and link-time dependency on TensorRT so the shared core (runtime, batch manager, executor API, KV cache, sampling, kernels, nanobind bridge) builds, links, and runs without the TensorRT library. Follows the Python TensorRT-backend removal (TRTLLM-14022); PyTorch is the sole backend. Internal types (serialization-compatible): - add tensorrt_llm::DataType/Dims (common/tllmDataType.h); DataType enumerator values mirror nvinfer1::DataType for byte-compatible serialization, Dims layout mirrors nvinfer1::Dims - migrate nvinfer1::DataType/Dims -> tensorrt_llm:: across 206 surviving files; replace NvInfer*.h includes with the internal header; drop 12 dead NvInfer includes from files that never referenced nvinfer1 - add tests/unittest/bindings/test_datatype_parity.py guarding the enumerator values (auto-scheduled via the existing unittest/bindings test-db entries) Remove the TensorRT-engine execution path (unused by the PyTorch backend): - plugins/ (nvinfer_plugin_tensorrt_llm), engine runtime wrappers (tllmRuntime, tllmStreamReaders, layerProfiler, rawEngine, tllmLogger), TRT model adapters (trtGptModel*/trtEncoderModel/trtGptModelFactory), executor.cpp/executorImpl, the C++ Executor + TllmRuntime + LogitsPostProcessor nanobind bindings (0 Python users each), executor_worker, disaggServerUtil, engine I/O buffers and engine-only logits/decoder algos, executor/model.h (0 consumers), the ModelSpec test helper + binding (orphaned) - relocate the retained KVCacheEvent ctor and executor::version() into executor/kvCacheEvent.cpp - decouple shared files (inflightBatchingUtils, medusaBuffers, lookaheadBuffers, dataTransceiver) from the removed engine runtime - delete the engine-driven C++ tests (e2e_tests engine tests, executorTestSmall*, tllmRuntimeTest, encDecBeamSearchTest, tests/utils engine builders), cudaGraphExecutorCacheTest (tests the removed CudaGraphExecutor), unit_tests/utils (tests the deleted tests/utils helpers), tests/unittest/others/test_leak.py (its body calls the removed python graph-building APIs; unscheduled in CI), C++ benchmarks (bertBenchmark, gptManagerBenchmark, disaggServerBenchmark; the prepare_dataset.py tooling used by trtllm-bench stays), examples/cpp executor examples, and the cpp/tests/resources engine-build scripts - fix kept tests: strip vestigial never-consumed TllmLogger members (7 files), repoint ropeTest.cu to kernels/gptKernels.h (+ cudaUtils.h for QuantMode/getSMVersion), inline the engine-free createDecoderBatchInputs helper into gptDecoderBatchedTest - make dependencies the deleted plugin target satisfied transitively explicit: MPI include dirs for cpp/tests, th_utils -> tensorrt_llm shared-lib link - remove the IS_BUILDING build-time env contract end to end (common/opUtils.h isBuilding + attentionOp gate + _common.py half) - remove the llm_args=None engine path from executor/base_worker.py (tllm.Executor no longer exists); llm_args is now required Build/packaging (no TensorRT): - drop find_package(TensorRT)/TRT_LIB/NvInfer include injection and the plugins/executor_worker/benchmarks subdirs from the cpp CMake; delete FindTensorRT.cmake - build_wheel.py: trt_root optional (default None), drop the tensorrt venv check and the nvinfer_plugin/executorWorker/benchmarks targets - setup.py no longer packages libnvinfer_plugin_tensorrt_llm.so / executorWorker; requirements.txt drops tensorrt - jenkins/Build.groovy: drop --benchmarks, the benchmark/libnvinfer-plugin tarball packaging, and the build_cpp_examples.py step - delist the removed tests from test-db/waives.txt/.test_durations and prune tests/integration/defs/cpp to the shared-core gtest wrappers - CI-run fixes: respell the nvinfer1 usages a post-rebase main commit (NVIDIA#16304) added to kvCacheManagerTest; retire the two CI stages emptied by the delisting (A10-CPP-1, H100_PCIe-CPP-Post-Merge-1) and drop the duplicate tensorrt-chunk scheduling of unittest/disaggregated/test_router.py; make test_to_stage_mapping's CLI check sample only stage-mapped tests - point the remaining benchmarks/cpp references (AutoDeploy bench/dist tests, llmc standalone packager + its test, trtllm-bench docs) at benchmarks/ and drop the orphaned get_cpp_benchmark() helper - fix the two OSS-mode gtest compile failures unmasked by dropping the nvinfer_plugin link (its PUBLIC USING_OSS_CUTLASS_*_GEMM defines had leaked into every gtest, keeping the internal-only branches dormant): inline the deleted tests/utils GpuTimer into gemmAllReduceTest.cu, and only add the mixtureOfExpertsInternalTest target when INTERNAL_CUTLASS_KERNELS_PATH provides the internal headers it includes Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
Wanli-Jiang
added a commit
to Wanli-Jiang/TensorRT-LLM
that referenced
this pull request
Jul 17, 2026
Sever the C++ tree's compile- and link-time dependency on TensorRT so the shared core (runtime, batch manager, executor API, KV cache, sampling, kernels, nanobind bridge) builds, links, and runs without the TensorRT library. Follows the Python TensorRT-backend removal (TRTLLM-14022); PyTorch is the sole backend. Internal types (serialization-compatible): - add tensorrt_llm::DataType/Dims (common/tllmDataType.h); DataType enumerator values mirror nvinfer1::DataType for byte-compatible serialization, Dims layout mirrors nvinfer1::Dims - migrate nvinfer1::DataType/Dims -> tensorrt_llm:: across 206 surviving files; replace NvInfer*.h includes with the internal header; drop 12 dead NvInfer includes from files that never referenced nvinfer1 - add tests/unittest/bindings/test_datatype_parity.py guarding the enumerator values (auto-scheduled via the existing unittest/bindings test-db entries) Remove the TensorRT-engine execution path (unused by the PyTorch backend): - plugins/ (nvinfer_plugin_tensorrt_llm), engine runtime wrappers (tllmRuntime, tllmStreamReaders, layerProfiler, rawEngine, tllmLogger), TRT model adapters (trtGptModel*/trtEncoderModel/trtGptModelFactory), executor.cpp/executorImpl, the C++ Executor + TllmRuntime + LogitsPostProcessor nanobind bindings (0 Python users each), executor_worker, disaggServerUtil, engine I/O buffers and engine-only logits/decoder algos, executor/model.h (0 consumers), the ModelSpec test helper + binding (orphaned) - keep the retained KVCacheEvent ctor and executor::version() inline in executor.h (they are trivial; no separate .cpp needed) - decouple shared files (inflightBatchingUtils, medusaBuffers, lookaheadBuffers, dataTransceiver) from the removed engine runtime - remove classes orphaned by the engine-path removal: GuidedDecoder (the PyTorch backend has its own Python guided decoder; xgrammar stays for executor::GuidedDecodingConfig), IntervalSet and DynamicBatchTuner (both only consumed by the removed executorImpl), each with their tests; removing guidedDecoderTest empties cpp/tests/e2e_tests entirely - restore the engine-free half of test_executor_bindings.py (config/ request/result/response/stats construction and pickle tests) since the underlying bindings stay and serve the PyTorch backend; only the trtllm.Executor/engine-fixture tests are dropped; the second, duplicated test_speculative_decoding_config (shadowed, never ran) is renamed to test_decoding_config to match what it tests - delete the engine-driven C++ tests (e2e_tests engine tests, executorTestSmall*, tllmRuntimeTest, encDecBeamSearchTest, tests/utils engine builders), cudaGraphExecutorCacheTest (tests the removed CudaGraphExecutor), unit_tests/utils (tests the deleted tests/utils helpers), tests/unittest/others/test_leak.py (its body calls the removed python graph-building APIs; unscheduled in CI), C++ benchmarks (bertBenchmark, gptManagerBenchmark, disaggServerBenchmark; the prepare_dataset.py tooling used by trtllm-bench stays), examples/cpp executor examples, and the cpp/tests/resources engine-build scripts - fix kept tests: strip vestigial never-consumed TllmLogger members (7 files), repoint ropeTest.cu to kernels/gptKernels.h (+ cudaUtils.h for QuantMode/getSMVersion), inline the engine-free createDecoderBatchInputs helper into gptDecoderBatchedTest, drop the empty tensorrt_llm::runtime forward-declaration block left in batch_manager/utils/debugUtils.h, narrow requestTest's using-namespace-common to a TllmException using-declaration (common now also exports DataType, which made the unqualified executor::DataType references ambiguous) - make dependencies the deleted plugin target satisfied transitively explicit: MPI include dirs for cpp/tests, th_utils -> tensorrt_llm shared-lib link - remove the IS_BUILDING build-time env contract end to end (common/opUtils.h isBuilding + attentionOp gate + _common.py half) - remove the llm_args=None engine path from executor/base_worker.py (tllm.Executor no longer exists); llm_args is now required Build/packaging (no TensorRT): - drop find_package(TensorRT)/TRT_LIB/NvInfer include injection and the plugins/executor_worker/benchmarks subdirs from the cpp CMake; delete FindTensorRT.cmake - build_wheel.py: trt_root optional (default None), drop the tensorrt venv check and the nvinfer_plugin/executorWorker/benchmarks targets - setup.py no longer packages libnvinfer_plugin_tensorrt_llm.so / executorWorker; requirements.txt drops tensorrt - jenkins/Build.groovy: drop --benchmarks, the benchmark/libnvinfer-plugin tarball packaging, and the build_cpp_examples.py step - delist the removed tests from test-db/waives.txt/.test_durations and prune tests/integration/defs/cpp to the shared-core gtest wrappers - CI-run fixes: respell the nvinfer1 usages a post-rebase main commit (NVIDIA#16304) added to kvCacheManagerTest; retire the two CI stages emptied by the delisting (A10-CPP-1, H100_PCIe-CPP-Post-Merge-1) and drop the duplicate tensorrt-chunk scheduling of unittest/disaggregated/test_router.py; make test_to_stage_mapping's CLI check sample only stage-mapped tests - point the remaining benchmarks/cpp references (AutoDeploy bench/dist tests, llmc standalone packager + its test, trtllm-bench docs) at benchmarks/ and drop the orphaned get_cpp_benchmark() helper - fix the two OSS-mode gtest compile failures unmasked by dropping the nvinfer_plugin link (its PUBLIC USING_OSS_CUTLASS_*_GEMM defines had leaked into every gtest, keeping the internal-only branches dormant): inline the deleted tests/utils GpuTimer into gemmAllReduceTest.cu, and only add the mixtureOfExpertsInternalTest target when INTERNAL_CUTLASS_KERNELS_PATH provides the internal headers it includes Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
Wanli-Jiang
added a commit
to Wanli-Jiang/TensorRT-LLM
that referenced
this pull request
Jul 18, 2026
Sever the C++ tree's compile- and link-time dependency on TensorRT so the shared core (runtime, batch manager, executor API, KV cache, sampling, kernels, nanobind bridge) builds, links, and runs without the TensorRT library. Follows the Python TensorRT-backend removal (TRTLLM-14022); PyTorch is the sole backend. Internal types (serialization-compatible): - add tensorrt_llm::DataType/Dims (common/tllmDataType.h); DataType enumerator values mirror nvinfer1::DataType for byte-compatible serialization, Dims layout mirrors nvinfer1::Dims - migrate nvinfer1::DataType/Dims -> tensorrt_llm:: across 206 surviving files; replace NvInfer*.h includes with the internal header; drop 12 dead NvInfer includes from files that never referenced nvinfer1 - add tests/unittest/bindings/test_datatype_parity.py guarding the enumerator values (auto-scheduled via the existing unittest/bindings test-db entries) Remove the TensorRT-engine execution path (unused by the PyTorch backend): - plugins/ (nvinfer_plugin_tensorrt_llm), engine runtime wrappers (tllmRuntime, tllmStreamReaders, layerProfiler, rawEngine, tllmLogger), TRT model adapters (trtGptModel*/trtEncoderModel/trtGptModelFactory), executor.cpp/executorImpl, the C++ Executor + TllmRuntime + LogitsPostProcessor nanobind bindings (0 Python users each), executor_worker, disaggServerUtil, engine I/O buffers and engine-only logits/decoder algos, executor/model.h (0 consumers), the ModelSpec test helper + binding (orphaned) - keep the retained KVCacheEvent ctor and executor::version() inline in executor.h (they are trivial; no separate .cpp needed) - decouple shared files (inflightBatchingUtils, medusaBuffers, lookaheadBuffers, dataTransceiver) from the removed engine runtime - remove classes orphaned by the engine-path removal: GuidedDecoder (the PyTorch backend has its own Python guided decoder; xgrammar stays for executor::GuidedDecodingConfig), IntervalSet and DynamicBatchTuner (both only consumed by the removed executorImpl), each with their tests; removing guidedDecoderTest empties cpp/tests/e2e_tests entirely - restore the engine-free half of test_executor_bindings.py (config/ request/result/response/stats construction and pickle tests) since the underlying bindings stay and serve the PyTorch backend; only the trtllm.Executor/engine-fixture tests are dropped; the second, duplicated test_speculative_decoding_config (shadowed, never ran) is renamed to test_decoding_config to match what it tests - delete the engine-driven C++ tests (e2e_tests engine tests, executorTestSmall*, tllmRuntimeTest, encDecBeamSearchTest, tests/utils engine builders), cudaGraphExecutorCacheTest (tests the removed CudaGraphExecutor), unit_tests/utils (tests the deleted tests/utils helpers), tests/unittest/others/test_leak.py (its body calls the removed python graph-building APIs; unscheduled in CI), C++ benchmarks (bertBenchmark, gptManagerBenchmark, disaggServerBenchmark; the prepare_dataset.py tooling used by trtllm-bench stays), examples/cpp executor examples, and the cpp/tests/resources engine-build scripts - fix kept tests: strip vestigial never-consumed TllmLogger members (7 files), repoint ropeTest.cu to kernels/gptKernels.h (+ cudaUtils.h for QuantMode/getSMVersion), inline the engine-free createDecoderBatchInputs helper into gptDecoderBatchedTest, drop the empty tensorrt_llm::runtime forward-declaration block left in batch_manager/utils/debugUtils.h, narrow requestTest's using-namespace-common to a TllmException using-declaration (common now also exports DataType, which made the unqualified executor::DataType references ambiguous) - make dependencies the deleted plugin target satisfied transitively explicit: MPI include dirs for cpp/tests, th_utils -> tensorrt_llm shared-lib link - remove the IS_BUILDING build-time env contract end to end (common/opUtils.h isBuilding + attentionOp gate + _common.py half) - remove the llm_args=None engine path from executor/base_worker.py (tllm.Executor no longer exists); llm_args is now required Build/packaging (no TensorRT): - drop find_package(TensorRT)/TRT_LIB/NvInfer include injection and the plugins/executor_worker/benchmarks subdirs from the cpp CMake; delete FindTensorRT.cmake - build_wheel.py: trt_root optional (default None), drop the tensorrt venv check and the nvinfer_plugin/executorWorker/benchmarks targets - setup.py no longer packages libnvinfer_plugin_tensorrt_llm.so / executorWorker; requirements.txt drops tensorrt - jenkins/Build.groovy: drop --benchmarks, the benchmark/libnvinfer-plugin tarball packaging, and the build_cpp_examples.py step - delist the removed tests from test-db/waives.txt/.test_durations and prune tests/integration/defs/cpp to the shared-core gtest wrappers - CI-run fixes: respell the nvinfer1 usages a post-rebase main commit (NVIDIA#16304) added to kvCacheManagerTest; retire the two CI stages emptied by the delisting (A10-CPP-1, H100_PCIe-CPP-Post-Merge-1) and drop the duplicate tensorrt-chunk scheduling of unittest/disaggregated/test_router.py; make test_to_stage_mapping's CLI check sample only stage-mapped tests - point the remaining benchmarks/cpp references (AutoDeploy bench/dist/ allreduce-strategy tests, llmc standalone packager + its test, trtllm-bench docs) at benchmarks/ and drop the orphaned get_cpp_benchmark() helper - fix the two OSS-mode gtest compile failures unmasked by dropping the nvinfer_plugin link (its PUBLIC USING_OSS_CUTLASS_*_GEMM defines had leaked into every gtest, keeping the internal-only branches dormant): inline the deleted tests/utils GpuTimer into gemmAllReduceTest.cu, and only add the mixtureOfExpertsInternalTest target when INTERNAL_CUTLASS_KERNELS_PATH provides the internal headers it includes Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
Wanli-Jiang
added a commit
to Wanli-Jiang/TensorRT-LLM
that referenced
this pull request
Jul 19, 2026
Sever the C++ tree's compile- and link-time dependency on TensorRT so the shared core (runtime, batch manager, executor API, KV cache, sampling, kernels, nanobind bridge) builds, links, and runs without the TensorRT library. Follows the Python TensorRT-backend removal (TRTLLM-14022); PyTorch is the sole backend. Internal types (serialization-compatible): - add tensorrt_llm::DataType/Dims (common/tllmDataType.h); DataType enumerator values mirror nvinfer1::DataType for byte-compatible serialization, Dims layout mirrors nvinfer1::Dims - migrate nvinfer1::DataType/Dims -> tensorrt_llm:: across 206 surviving files; replace NvInfer*.h includes with the internal header; drop 12 dead NvInfer includes from files that never referenced nvinfer1 - add tests/unittest/bindings/test_datatype_parity.py guarding the enumerator values (auto-scheduled via the existing unittest/bindings test-db entries) Remove the TensorRT-engine execution path (unused by the PyTorch backend): - plugins/ (nvinfer_plugin_tensorrt_llm), engine runtime wrappers (tllmRuntime, tllmStreamReaders, layerProfiler, rawEngine, tllmLogger), TRT model adapters (trtGptModel*/trtEncoderModel/trtGptModelFactory), executor.cpp/executorImpl, the C++ Executor + TllmRuntime + LogitsPostProcessor nanobind bindings (0 Python users each), executor_worker, disaggServerUtil, engine I/O buffers and engine-only logits/decoder algos, executor/model.h (0 consumers), the ModelSpec test helper + binding (orphaned) - keep the retained KVCacheEvent ctor and executor::version() inline in executor.h (they are trivial; no separate .cpp needed) - decouple shared files (inflightBatchingUtils, medusaBuffers, lookaheadBuffers, dataTransceiver) from the removed engine runtime - remove classes orphaned by the engine-path removal: GuidedDecoder (the PyTorch backend has its own Python guided decoder; xgrammar stays for executor::GuidedDecodingConfig), IntervalSet and DynamicBatchTuner (both only consumed by the removed executorImpl), each with their tests; removing guidedDecoderTest empties cpp/tests/e2e_tests entirely - restore the engine-free half of test_executor_bindings.py (config/ request/result/response/stats construction and pickle tests) since the underlying bindings stay and serve the PyTorch backend; only the trtllm.Executor/engine-fixture tests are dropped; the second, duplicated test_speculative_decoding_config (shadowed, never ran) is renamed to test_decoding_config to match what it tests - delete the engine-driven C++ tests (e2e_tests engine tests, executorTestSmall*, tllmRuntimeTest, encDecBeamSearchTest, tests/utils engine builders), cudaGraphExecutorCacheTest (tests the removed CudaGraphExecutor), unit_tests/utils (tests the deleted tests/utils helpers), tests/unittest/others/test_leak.py (its body calls the removed python graph-building APIs; unscheduled in CI), C++ benchmarks (bertBenchmark, gptManagerBenchmark, disaggServerBenchmark; the prepare_dataset.py tooling used by trtllm-bench stays), examples/cpp executor examples, and the cpp/tests/resources engine-build scripts - fix kept tests: strip vestigial never-consumed TllmLogger members (7 files), repoint ropeTest.cu to kernels/gptKernels.h (+ cudaUtils.h for QuantMode/getSMVersion), inline the engine-free createDecoderBatchInputs helper into gptDecoderBatchedTest, drop the empty tensorrt_llm::runtime forward-declaration block left in batch_manager/utils/debugUtils.h, narrow requestTest's using-namespace-common to a TllmException using-declaration (common now also exports DataType, which made the unqualified executor::DataType references ambiguous) - make dependencies the deleted plugin target satisfied transitively explicit: MPI include dirs for cpp/tests, th_utils -> tensorrt_llm shared-lib link - remove the IS_BUILDING build-time env contract end to end (common/opUtils.h isBuilding + attentionOp gate + _common.py half) - remove the llm_args=None engine path from executor/base_worker.py (tllm.Executor no longer exists); llm_args is now required Build/packaging (no TensorRT): - drop find_package(TensorRT)/TRT_LIB/NvInfer include injection and the plugins/executor_worker/benchmarks subdirs from the cpp CMake; delete FindTensorRT.cmake - build_wheel.py: trt_root optional (default None), drop the tensorrt venv check and the nvinfer_plugin/executorWorker/benchmarks targets - setup.py no longer packages libnvinfer_plugin_tensorrt_llm.so / executorWorker; requirements.txt drops tensorrt - jenkins/Build.groovy: drop --benchmarks, the benchmark/libnvinfer-plugin tarball packaging, and the build_cpp_examples.py step - delist the removed tests from test-db/waives.txt/.test_durations and prune tests/integration/defs/cpp to the shared-core gtest wrappers - CI-run fixes: respell the nvinfer1 usages a post-rebase main commit (NVIDIA#16304) added to kvCacheManagerTest; retire the two CI stages emptied by the delisting (A10-CPP-1, H100_PCIe-CPP-Post-Merge-1) and drop the duplicate tensorrt-chunk scheduling of unittest/disaggregated/test_router.py; make test_to_stage_mapping's CLI check sample only stage-mapped tests - point the remaining benchmarks/cpp references (AutoDeploy bench/dist/ allreduce-strategy tests, llmc standalone packager + its test, trtllm-bench docs) at benchmarks/ and drop the orphaned get_cpp_benchmark() helper - fix the two OSS-mode gtest compile failures unmasked by dropping the nvinfer_plugin link (its PUBLIC USING_OSS_CUTLASS_*_GEMM defines had leaked into every gtest, keeping the internal-only branches dormant): inline the deleted tests/utils GpuTimer into gemmAllReduceTest.cu, and only add the mixtureOfExpertsInternalTest target when INTERNAL_CUTLASS_KERNELS_PATH provides the internal headers it includes Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
Bug Fixes
Tests
Description
Fix NVFP4 KV-cache initialization for hybrid attention/Mamba models by forwarding per-window pool configurations and merging compact pointer tables in Python.
Root Cause
CppMambaHybridCacheManagercreated recurrent-state and attention windows but did not supply theirpool_configurationstoKVCacheManager. Consequently, C++ constructed an emptyBlockManager::poolByWindowand fell back to the manager-level NVFP4 attention dtype for the recurrent-state window.Recurrent-state structures do not support FP4. Their backing pool must use the model's actual SSM-state dtype, while the attention pool keeps the requested KV-cache dtype. The Python manager views the backing allocation as bytes when splitting SSM and convolution state sections.
With mixed recurrent-state and NVFP4 attention pools, C++ returns separate compact data-pointer and block-scale-pointer tables. Directly stacking those tensors fails when only some data pools have scales. Python aligns each compact scale row with its NVFP4 data row before stacking, so the final
kv_cache_pool_pointersremains[num_data_pools, 2, 2]without changing the C++ pointer ABI.Changes
CppMambaHybridCacheManager.torch_dtype_to_binding(self.ssm_state_dtype)for the recurrent-state pool and the requested KV-cache dtype for attention.poolByWindowselects each pool's actual dtype.torch.stack.KVCacheManagerproducer contract for pointer shapes, dtypes, devices, mapping consistency, and scale cardinality instead of duplicating defensive checks in Python.Test Coverage
scripts/build_wheel.pyusing SM100/SM103, ccache, benchmarks, and NIXL; no--cleanrebuild was used.tests/unittest/_torch/executor/test_mamba_cache_manager.py: 26 passed, including FP16, FP32, and BF16 recurrent pools.tests/unittest/_torch/executor/test_resource_manager.py: 20 passed, 7 subtests passed.KVCacheManagerTest.FP4*: 2 passed.quant_algo=NVFP4andkv_cache_quant_algo=NVFP4for Nemotron-Nano-3-30B-A3.5B-dev-1024.kv_cache_quant_algo=NVFP4and generated 8 tokens successfully.git diff --checkpassed.PR Checklist
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.