Update AutoDeploy integration with trtllm-bench. Fixes uses of overlap scheduler in trtllm

[suyoggupta]

See copilot generated description

1. Fixes AD <> trtllm-bench integration
   usage:

trtllm-bench --model meta-llama/Llama-3.1-8B  throughput --dataset /tmp/synthetic_128_128.txt --backend _autodeploy

2. Re-enabled overlap scheduler in trtllm runtime. [AutoDeploy] Support overlap scheduler NVIDIA/TensorRT-LLM#4364
   python examples/auto_deploy/build_and_run_ad.py --config '{"model": "meta-llama/Llama-3.1-8B", "attn_backend":"FlashInfer", "runtime":"trtllm", "world_size":1, "compile_backend":"torch-opt"}' produces legible outputs:

[06/11/2025-07:17:25] [TRT-LLM AUTO-DEPLOY] [I] [PROMPT 0] How big is the universe? :  How many stars are there?  How many galaxies?  How many planets?  How many moons?  How many black holes?  How many planets with life?  How many planets with intelligent life?  How many planets with intelligent life that have developed space travel?  How many planets with intelligent life that have developed space travel that have developed the technology to detect other planets with intelligent life that have developed space travel?  How many planets with
[06/11/2025-07:17:25] [TRT-LLM AUTO-DEPLOY] [I] [PROMPT 1] In simple words and in a single sentence, explain the concept of gravity: : 1. The force of attraction between two bodies due to their masses. 2. The force of attraction between two bodies due to their masses and the distance between them. 3. The force of attraction between two bodies due to their masses and the distance between them and the medium in which they are placed. 4. The force of attraction between two bodies due to their masses and the distance between them and the medium in which they are placed and the time taken by them to reach each other.

3. Some perf fixes in prepare_inputs
   AutoDeploy:

===========================================================                                                                               
= PERFORMANCE OVERVIEW                                                                                                                    
===========================================================                                                                               
Request Throughput (req/sec):                     78.9478                                                                                 
Total Output Throughput (tokens/sec):             10105.3245                                                                              
Total Token Throughput (tokens/sec):              20210.6489                                                                              
Total Latency (ms):                               12666.5898                                                                              
Average request latency (ms):                     12188.4318                                                                              
Per User Output Throughput [w/ ctx] (tps/user):   10.5092                                                                                 
Per GPU Output Throughput (tps/gpu):              10105.3245

Pytorch:

===========================================================                                                                               
= PERFORMANCE OVERVIEW                                                                                                                    
===========================================================                                                                               
Request Throughput (req/sec):                     89.7706                                                                                 
Total Output Throughput (tokens/sec):             11490.6339                                                                              
Total Token Throughput (tokens/sec):              22981.2679
Total Latency (ms):                               11139.5072
Average request latency (ms):                     10719.8539
Per User Output Throughput [w/ ctx] (tps/user):   11.9492
Per GPU Output Throughput (tps/gpu):              11490.6339

4. With rms norm pattern matching
   AutoDeploy with triton rms norm kernel

===========================================================                                                                               
= PERFORMANCE OVERVIEW                                                                                                                    
===========================================================                                                                               
Request Throughput (req/sec):                     85.6230                                                                                 
Total Output Throughput (tokens/sec):             10959.7406                                                                              
Total Token Throughput (tokens/sec):              21919.4813                                                                              
Total Latency (ms):                               11679.1085
Average request latency (ms):                     11243.4913
Per User Output Throughput [w/ ctx] (tps/user):   11.3923
Per GPU Output Throughput (tps/gpu):              10959.7406

AutoDeploy with flashinfer norm kernel:

===========================================================                                                                               
= PERFORMANCE OVERVIEW                                                                                                                    
===========================================================                                                                               
Request Throughput (req/sec):                     85.3631                                                                                 
Total Output Throughput (tokens/sec):             10926.4770                                                                              
Total Token Throughput (tokens/sec):              21852.9540                                                                              
Total Latency (ms):                               11714.6634
Average request latency (ms):                     11281.1014
Per User Output Throughput [w/ ctx] (tps/user):   11.3542
Per GPU Output Throughput (tps/gpu):              10926.4770

[Copilot]

Pull Request Overview

This PR updates the AutoDeploy integration for trtllm-bench by adjusting the configuration handling and enhancing logging in the KV cache transformation.

• In tensorrt_llm/bench/dataclasses/configuration.py, the auto deploy configuration is now modified using dictionary indexing rather than instantiating _AutoDeployLlmArgs.
• In tensorrt_llm/_torch/auto_deploy/transformations/library/kvcache.py, additional logging for free memory ratio is added and the free/total memory log message is refined to display values in MB.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
tensorrt_llm/bench/dataclasses/configuration.py	Changed config construction to use dictionary updates, which may affect type consistency.
tensorrt_llm/_torch/auto_deploy/transformations/library/kvcache.py	Added log for free memory ratio and adjusted memory logging for clarity.

Comments suppressed due to low confidence (1)

tensorrt_llm/bench/dataclasses/configuration.py:114

• The original creation of ad_config via _AutoDeployLlmArgs was replaced with direct assignment from self.pytorch_config; please verify that self.pytorch_config is of the expected dictionary type and that downstream usage will handle the updated structure correctly.

ad_config = self.pytorch_config

[lucaslie]

Should we add a unit test? Could look similar to https://github.com/nv-auto-deploy/TensorRT-LLM/blob/743fb0a159695d59c0186702e7433b7594a6048c/tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_build_small_single.py. We test some miniature models and use a patch to check if configs get passed through correctly

[suyoggupta]

Probably a good idea to add a unit test.

[suyoggupta]

test added

[suyoggupta]

will open a PR to trtllm main to test this on CI