[AutoDeploy] more robust handling of attention interface and input nodes by lucaslie · Pull Request #4369 · NVIDIA/TensorRT-LLM

lucaslie · 2025-05-15T22:30:51Z

This PR improves handling of our inputs that are provided by the cache interface:

better switch between original two inputs and full inputs after cache metadata
handling of case when both MLA and MHA layers are present and hence inputs should only be added once
simplification of attention interface

Test Coverage

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--disable-fail-fast --skip-test --stage-list "A10-1, xxx" --gpu-type "A30, H100_PCIe" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-[Post-Merge]-1, xxx"]

Launch build/test pipelines. All previously running jobs will be killed.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests. Will also run L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-[Post-Merge]-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-[Post-Merge]-1, xxx".

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

lucaslie · 2025-05-15T22:31:00Z

/bot run

tensorrt-cicd · 2025-05-15T22:37:19Z

PR_Github #5411 [ run ] triggered by Bot

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

tensorrt-cicd · 2025-05-16T05:42:14Z

PR_Github #5411 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #3948 completed with status: 'FAILURE'

lucaslie · 2025-05-16T15:32:04Z

/bot run

tensorrt-cicd · 2025-05-16T15:38:00Z

PR_Github #5523 [ run ] triggered by Bot

tensorrt-cicd · 2025-05-16T19:23:00Z

PR_Github #5523 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #4025 completed with status: 'FAILURE'

lucaslie · 2025-05-16T19:48:09Z

/bot run

tensorrt-cicd · 2025-05-16T19:54:22Z

PR_Github #5536 [ run ] triggered by Bot

lucaslie · 2025-05-16T22:17:32Z

/bot run

tensorrt-cicd · 2025-05-16T22:23:30Z

PR_Github #5541 [ run ] triggered by Bot

tensorrt-cicd · 2025-05-16T22:23:36Z

PR_Github #5536 [ run ] completed with state ABORTED
/LLM/main/L0_MergeRequest_PR pipeline #4037 completed with status: 'FAILURE'

tensorrt-cicd · 2025-05-17T05:19:28Z

PR_Github #5541 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #4042 completed with status: 'FAILURE'

lucaslie · 2025-05-19T17:46:21Z

/bot run

tensorrt-cicd · 2025-05-19T17:52:26Z

PR_Github #5759 [ run ] triggered by Bot

tensorrt-cicd · 2025-05-20T00:40:30Z

PR_Github #5759 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #4213 completed with status: 'FAILURE'

* [AutoDeploy] HF factory improvements * improve monkey-patches and add unit tests --------- Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

* [AutoDeploy] bmm sharder Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> * a few more subtle MP clean-ups Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> --------- Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

Signed-off-by: Ubuntu <201670829+Fridah-nv@users.noreply.github.com>

* DS support related fixes Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com> * DS related changes Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com> * Quick fix for dashboard runs Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com> * Quick fix for dashboard runs Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com> * Reduce number of layers in integration test Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com> * formatting Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com> --------- Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com>

Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

…ue (#19) Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

Includes #17 , #18, #21 , #23 * move Phi PR for sprint Signed-off-by: Ubuntu <201670829+Fridah-nv@users.noreply.github.com> * remove Phi3 patch and unfuse_weight transformation Signed-off-by: Ubuntu <201670829+Fridah-nv@users.noreply.github.com> * update Phi3/4 patches Signed-off-by: Ubuntu <201670829+Fridah-nv@users.noreply.github.com> * minor Signed-off-by: Ubuntu <201670829+Fridah-nv@users.noreply.github.com> * minor Signed-off-by: Ubuntu <201670829+Fridah-nv@users.noreply.github.com> * Update the position id to int64 Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com> * Add support for handling aliased parameters Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> * update tests Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> * lint Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> * lint Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> * removed load state dict with assign context manager Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com> * formatting fix Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com> * Removed related unit tests Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com> * [AutoDeploy] correctly identifying how to load HF checkpoint weights Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> * [AutoDeploy] correctly load sharded weights Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> * skip sharding if num_users>1 Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> --------- Signed-off-by: Ubuntu <201670829+Fridah-nv@users.noreply.github.com> Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com> Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com> Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> Co-authored-by: Ubuntu <201670829+Fridah-nv@users.noreply.github.com> Co-authored-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com> Co-authored-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com> Co-authored-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

* add phi3.5 patch Signed-off-by: Ubuntu <201670829+Fridah-nv@users.noreply.github.com> * revert gemm fusion change Signed-off-by: Ubuntu <201670829+Fridah-nv@users.noreply.github.com> * update README Signed-off-by: Ubuntu <201670829+Fridah-nv@users.noreply.github.com> --------- Signed-off-by: Ubuntu <201670829+Fridah-nv@users.noreply.github.com>

Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

lucaslie · 2025-05-23T03:23:11Z

will be merged via nv-auto-deploy#31

lucaslie requested review from Fridah-nv, Copilot, sugunav14 and suyoggupta May 15, 2025 22:30

github-project-automation Bot added this to AutoDeploy Board May 15, 2025

github-project-automation Bot moved this to Backlog in AutoDeploy Board May 15, 2025

lucaslie self-assigned this May 15, 2025

lucaslie moved this from Backlog to In review in AutoDeploy Board May 15, 2025

Copilot AI reviewed May 15, 2025

lucaslie added the AutoDeploy <NV> AutoDeploy Backend label May 15, 2025

suyoggupta reviewed May 15, 2025

View reviewed changes

Comment thread tensorrt_llm/_torch/auto_deploy/custom_ops/attention_interface.py Outdated

lucaslie force-pushed the ll/attn_if_updated branch from 6d1c04b to 6518887 Compare May 16, 2025 15:29

Fridah-nv approved these changes May 16, 2025

View reviewed changes

suyoggupta approved these changes May 16, 2025

View reviewed changes

lucaslie enabled auto-merge (squash) May 16, 2025 16:36

lucaslie force-pushed the ll/attn_if_updated branch from 6518887 to 396c9cf Compare May 16, 2025 19:47

lucaslie force-pushed the ll/attn_if_updated branch from 396c9cf to b921aca Compare May 16, 2025 22:17

lucaslie force-pushed the ll/attn_if_updated branch from b921aca to 706021f Compare May 19, 2025 17:45

lucaslie disabled auto-merge May 19, 2025 23:17

lucaslie and others added 13 commits May 19, 2025 21:00

[AutoDeploy] HF factory improvements (NVIDIA#4371) (#11)

319e4de

* [AutoDeploy] HF factory improvements * improve monkey-patches and add unit tests --------- Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

[AutoDeploy] set default cache resize portion to 0.0 (#13)

1fc348a

Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

[AutoDeploy] no need for cleanup in ADExecutor (#15)

799245f

Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

update graph util for nested subgraphs (#14)

c40131d

Signed-off-by: Ubuntu <201670829+Fridah-nv@users.noreply.github.com>

[AutoDeploy] ensure device id is in our device setting (#16)

12ce127

Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

[AutoDeploy] WAR to address max_batch_size,max_seq_len flashinfer iss…

196f516

…ue (#19) Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

skipping benchmarking for trtllm runtime (#20)

964b122

Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

[AutoDeploy] more robust handling of attention interface and input nodes

6908a46

Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

updated description for AttentionInterface

952a715

Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

lucaslie force-pushed the ll/attn_if_updated branch from 706021f to 952a715 Compare May 23, 2025 03:21

lucaslie requested a review from a team as a code owner May 23, 2025 03:21

lucaslie requested review from byshiue and hlu1 May 23, 2025 03:21

lucaslie mentioned this pull request May 23, 2025

[AutoDeploy] more robust handling of attention interface and input nodes nv-auto-deploy/TensorRT-LLM#31

Merged

lucaslie closed this May 23, 2025

github-project-automation Bot moved this from In review to Done in AutoDeploy Board May 23, 2025

Uh oh!

Conversation

lucaslie commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Coverage

GitHub Bot Help

kill

skip

reuse-pipeline

Uh oh!

lucaslie commented May 15, 2025

Uh oh!

tensorrt-cicd commented May 15, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tensorrt-cicd commented May 16, 2025

Uh oh!

lucaslie commented May 16, 2025

Uh oh!

tensorrt-cicd commented May 16, 2025

Uh oh!

tensorrt-cicd commented May 16, 2025

Uh oh!

lucaslie commented May 16, 2025

Uh oh!

tensorrt-cicd commented May 16, 2025

Uh oh!

lucaslie commented May 16, 2025

Uh oh!

tensorrt-cicd commented May 16, 2025

Uh oh!

tensorrt-cicd commented May 16, 2025

Uh oh!

tensorrt-cicd commented May 17, 2025

Uh oh!

lucaslie commented May 19, 2025

Uh oh!

tensorrt-cicd commented May 19, 2025

Uh oh!

tensorrt-cicd commented May 20, 2025

Uh oh!

lucaslie commented May 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

lucaslie commented May 15, 2025 •

edited

Loading