[fix] Fix Llama4 allgather error due to None tensor by jinyangyuan-nvidia · Pull Request #4511 · NVIDIA/TensorRT-LLM

jinyangyuan-nvidia · 2025-05-21T02:59:06Z

Fix a bug of Llama4 when enabling attention DP (introduced by #3439). None tensors are handled in a more general way. The test of attention DP for LLama4 is also added to unittest (waived by now due to an existing issue). Local tests show that the error caused by None tensor has been solved.

jinyangyuan-nvidia · 2025-05-21T03:05:27Z

/bot run

tensorrt-cicd · 2025-05-21T03:10:33Z

PR_Github #5944 [ run ] triggered by Bot

jinyangyuan-nvidia · 2025-05-21T04:13:31Z

/bot kill

tensorrt-cicd · 2025-05-21T04:18:37Z

PR_Github #5952 [ kill ] triggered by Bot

tensorrt-cicd · 2025-05-21T04:18:38Z

PR_Github #5944 [ run ] completed with state ABORTED

tensorrt-cicd · 2025-05-21T04:19:08Z

PR_Github #5952 [ kill ] completed with state SUCCESS
Successfully killed previous jobs for commit f5b1065

jinyangyuan-nvidia · 2025-05-21T06:13:37Z

/bot run

tensorrt-cicd · 2025-05-21T06:22:59Z

PR_Github #5970 [ run ] triggered by Bot

jinyangyuan-nvidia · 2025-05-21T08:48:14Z

/bot run

tensorrt-cicd · 2025-05-21T08:54:18Z

PR_Github #5993 [ run ] triggered by Bot

tensorrt-cicd · 2025-05-21T08:54:22Z

PR_Github #5970 [ run ] completed with state ABORTED

tensorrt-cicd · 2025-05-21T11:28:49Z

PR_Github #5993 [ run ] completed with state SUCCESS
/LLM/release-0.20/L0_MergeRequest_PR pipeline #16 completed with status: 'FAILURE'

jinyangyuan-nvidia · 2025-05-21T13:04:19Z

/bot run

tensorrt-cicd · 2025-05-21T13:10:01Z

PR_Github #6020 [ run ] triggered by Bot

tensorrt-cicd · 2025-05-21T13:26:44Z

PR_Github #6020 [ run ] completed with state FAILURE
/LLM/release-0.20/L0_MergeRequest_PR pipeline #17 completed with status: 'FAILURE'

jinyangyuan-nvidia · 2025-05-21T14:07:03Z

/bot run

tensorrt-cicd · 2025-05-21T14:15:29Z

PR_Github #6030 [ run ] triggered by Bot

djns99

LGTM, one minor nit

jinyangyuan-nvidia · 2025-05-23T05:21:15Z

/bot run

tensorrt-cicd · 2025-05-23T05:26:42Z

PR_Github #6240 [ run ] triggered by Bot

tensorrt-cicd · 2025-05-23T05:26:46Z

PR_Github #6192 [ run ] completed with state ABORTED
/LLM/release-0.20/L0_MergeRequest_PR pipeline #49 completed with status: 'FAILURE'

tensorrt-cicd · 2025-05-23T08:51:23Z

PR_Github #6240 [ run ] completed with state SUCCESS
/LLM/release-0.20/L0_MergeRequest_PR pipeline #52 completed with status: 'FAILURE'

jinyangyuan-nvidia · 2025-05-23T08:54:34Z

/bot run

tensorrt-cicd · 2025-05-23T09:01:07Z

PR_Github #6273 [ run ] triggered by Bot

tensorrt-cicd · 2025-05-23T15:32:55Z

PR_Github #6273 [ run ] completed with state SUCCESS
/LLM/release-0.20/L0_MergeRequest_PR pipeline #56 completed with status: 'FAILURE'

Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>

jinyangyuan-nvidia · 2025-05-24T01:10:53Z

/bot run

tensorrt-cicd · 2025-05-24T01:16:14Z

PR_Github #6342 [ run ] triggered by Bot

tensorrt-cicd · 2025-05-24T04:41:46Z

PR_Github #6342 [ run ] completed with state SUCCESS
/LLM/release-0.20/L0_MergeRequest_PR pipeline #63 completed with status: 'FAILURE'

jinyangyuan-nvidia · 2025-05-24T05:09:18Z

/bot run

tensorrt-cicd · 2025-05-24T05:14:44Z

PR_Github #6353 [ run ] triggered by Bot

tensorrt-cicd · 2025-05-24T05:59:04Z

PR_Github #6353 [ run ] completed with state ABORTED

jinyangyuan-nvidia · 2025-05-24T06:21:05Z

/bot run

tensorrt-cicd · 2025-05-24T06:27:26Z

PR_Github #6354 [ run ] triggered by Bot

tensorrt-cicd · 2025-05-24T11:12:10Z

PR_Github #6354 [ run ] completed with state SUCCESS
/LLM/release-0.20/L0_MergeRequest_PR pipeline #65 completed with status: 'SUCCESS'

* [fix] Fix Llama4 allgather error due to None tensor Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com> * Refactor modifications Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com> * Minor modification Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com> * Minor fix Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com> --------- Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>

jinyangyuan-nvidia requested review from djns99 and zihaok May 21, 2025 02:59

jinyangyuan-nvidia self-assigned this May 21, 2025

jinyangyuan-nvidia force-pushed the dev/fix_llama_adp branch from df23d58 to f5b1065 Compare May 21, 2025 03:05

jinyangyuan-nvidia requested a review from litaotju May 21, 2025 03:05

djns99 reviewed May 21, 2025

View reviewed changes

Comment thread tensorrt_llm/_torch/modules/fused_moe.py Outdated

Comment thread tensorrt_llm/_torch/modules/fused_moe.py Outdated

jinyangyuan-nvidia changed the base branch from main to release/0.20 May 21, 2025 08:24

jinyangyuan-nvidia requested review from a team as code owners May 21, 2025 08:24

jinyangyuan-nvidia force-pushed the dev/fix_llama_adp branch from d2e36fc to 63188d1 Compare May 21, 2025 08:38

jinyangyuan-nvidia removed the request for review from a team May 21, 2025 08:48

djns99 approved these changes May 21, 2025

View reviewed changes

Comment thread tensorrt_llm/_torch/distributed/ops.py Outdated

jinyangyuan-nvidia force-pushed the dev/fix_llama_adp branch from 63188d1 to 6843775 Compare May 21, 2025 21:51

litaotju approved these changes May 23, 2025

View reviewed changes

jinyangyuan-nvidia added 4 commits May 24, 2025 09:10

[fix] Fix Llama4 allgather error due to None tensor

88d1684

Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>

Refactor modifications

efd47bb

Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>

Minor modification

1b09e22

Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>

Minor fix

ee681bd

Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>

jinyangyuan-nvidia force-pushed the dev/fix_llama_adp branch from 9ea151e to ee681bd Compare May 24, 2025 01:10

jinyangyuan-nvidia enabled auto-merge (squash) May 24, 2025 01:12

jinyangyuan-nvidia merged commit f9a9a1a into NVIDIA:release/0.20 May 24, 2025

jinyangyuan-nvidia deleted the dev/fix_llama_adp branch May 24, 2025 11:52

Uh oh!

Conversation

jinyangyuan-nvidia commented May 21, 2025

Uh oh!

jinyangyuan-nvidia commented May 21, 2025

Uh oh!

tensorrt-cicd commented May 21, 2025

Uh oh!

Uh oh!

Uh oh!

jinyangyuan-nvidia commented May 21, 2025

Uh oh!

tensorrt-cicd commented May 21, 2025

Uh oh!

tensorrt-cicd commented May 21, 2025

Uh oh!

tensorrt-cicd commented May 21, 2025

Uh oh!

jinyangyuan-nvidia commented May 21, 2025

Uh oh!

tensorrt-cicd commented May 21, 2025

Uh oh!

jinyangyuan-nvidia commented May 21, 2025

Uh oh!

tensorrt-cicd commented May 21, 2025

Uh oh!

tensorrt-cicd commented May 21, 2025

Uh oh!

tensorrt-cicd commented May 21, 2025

Uh oh!

jinyangyuan-nvidia commented May 21, 2025

Uh oh!

tensorrt-cicd commented May 21, 2025

Uh oh!

tensorrt-cicd commented May 21, 2025

Uh oh!

jinyangyuan-nvidia commented May 21, 2025

Uh oh!

tensorrt-cicd commented May 21, 2025

Uh oh!

djns99 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jinyangyuan-nvidia commented May 23, 2025

Uh oh!

tensorrt-cicd commented May 23, 2025

Uh oh!

tensorrt-cicd commented May 23, 2025

Uh oh!

tensorrt-cicd commented May 23, 2025

Uh oh!

jinyangyuan-nvidia commented May 23, 2025

Uh oh!

tensorrt-cicd commented May 23, 2025

Uh oh!

tensorrt-cicd commented May 23, 2025

Uh oh!

jinyangyuan-nvidia commented May 24, 2025

Uh oh!

tensorrt-cicd commented May 24, 2025

Uh oh!

tensorrt-cicd commented May 24, 2025

Uh oh!

jinyangyuan-nvidia commented May 24, 2025

Uh oh!

tensorrt-cicd commented May 24, 2025

Uh oh!

tensorrt-cicd commented May 24, 2025

Uh oh!

jinyangyuan-nvidia commented May 24, 2025

Uh oh!

tensorrt-cicd commented May 24, 2025

Uh oh!

tensorrt-cicd commented May 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects