Skip to content

fix: refactor and fix mtp vanilla - #4762

Merged
lfr-0531 merged 13 commits into
NVIDIA:mainfrom
lfr-0531:user/fanrongl/simplify_mtp_vanilla
Jun 19, 2025
Merged

fix: refactor and fix mtp vanilla#4762
lfr-0531 merged 13 commits into
NVIDIA:mainfrom
lfr-0531:user/fanrongl/simplify_mtp_vanilla

Conversation

@lfr-0531

@lfr-0531 lfr-0531 commented May 29, 2025 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

  • Refactor the MTP vanilla implementation
  • Fix two bugs in MTP vanilla forward
    • The one is the saved past token
    • Another one is the input hidden states of the draft layers

Now MTP vanilla can achieve comparable performance to MTP Eagle:

model num_samples nextn Total Output Throughput (tokens/sec) acceptance rate
deepseek-ai/DeepSeek-R1-fp4 10 0 151.1382389365772 1.0
deepseek-ai/DeepSeek-R1-fp4 10 1 266.1345253466809 1.9064093282044987
deepseek-ai/DeepSeek-R1-fp4 10 2 317.71522794209443 2.53556285582307
deepseek-ai/DeepSeek-R1-fp4 10 3 320.31630735801565 2.83028864320805
deepseek-ai/DeepSeek-R1-fp4 10 4 306.35980785857925 2.944817683111899
deepseek-ai/DeepSeek-R1-fp4 10 5 291.8027987875048 2.986063372598476
deepseek-ai/DeepSeek-R1-fp4 10 6 269.4846495353019 2.989467810471117
deepseek-ai/DeepSeek-R1-fp4 10 7 253.9811841654886 3.0055965829479336

@lfr-0531
lfr-0531 requested a review from a team as a code owner May 29, 2025 09:35
@lfr-0531
lfr-0531 requested review from lucaslie, mikeiovine and yweng0828 and removed request for lucaslie and mikeiovine May 29, 2025 09:35
@lfr-0531
lfr-0531 force-pushed the user/fanrongl/simplify_mtp_vanilla branch 5 times, most recently from 0d16978 to 57f634a Compare June 4, 2025 09:01
@lfr-0531 lfr-0531 changed the title draft: refactor and fix mtp vanilla fix: refactor and fix mtp vanilla Jun 4, 2025
@lfr-0531

lfr-0531 commented Jun 4, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Jun 4, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7484 [ run ] triggered by Bot

@tensorrt-cicd

tensorrt-cicd commented Jun 4, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7484 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #5429 completed with status: 'ABORTED'

@lfr-0531
lfr-0531 force-pushed the user/fanrongl/simplify_mtp_vanilla branch from 57f634a to c9765d5 Compare June 5, 2025 00:32
@lfr-0531

lfr-0531 commented Jun 5, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jun 5, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7585 [ run ] triggered by Bot

@tensorrt-cicd

tensorrt-cicd commented Jun 5, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7585 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #5506 completed with status: 'FAILURE'

yweng0828
yweng0828 approved these changes Jun 5, 2025
View reviewed changes
Comment thread tensorrt_llm/_torch/models/modeling_deepseekv3.py Outdated
Comment thread tensorrt_llm/llmapi/llm_args.py Outdated
@lfr-0531
lfr-0531 force-pushed the user/fanrongl/simplify_mtp_vanilla branch from c9765d5 to d2f3183 Compare June 5, 2025 17:14
@lfr-0531

lfr-0531 commented Jun 5, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jun 5, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7785 [ run ] triggered by Bot

@tensorrt-cicd

tensorrt-cicd commented Jun 5, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7785 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #5638 completed with status: 'FAILURE'

@lfr-0531
lfr-0531 force-pushed the user/fanrongl/simplify_mtp_vanilla branch from d2f3183 to 5e5bdf8 Compare June 6, 2025 04:52
@lfr-0531

lfr-0531 commented Jun 6, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jun 6, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7837 [ run ] triggered by Bot

@tensorrt-cicd

tensorrt-cicd commented Jun 6, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7837 [ run ] completed with state FAILURE

@lfr-0531
lfr-0531 force-pushed the user/fanrongl/simplify_mtp_vanilla branch from 5e5bdf8 to bfe434d Compare June 6, 2025 16:21
@lfr-0531

lfr-0531 commented Jun 6, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

lfr-0531 added 10 commits June 19, 2025 18:28
@lfr-0531
open mtp vanilla path and add a TRTLLM_FORCE_MTP_VANILLA env to force…
bb355f4 
… using mtp vanilla.

Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
@lfr-0531
fix position id.
78ee1b1 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
@lfr-0531
fix hidden states in mtp vanilla.
d562b64 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
@lfr-0531
update code comments.
45f372c 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
@lfr-0531
fix mtp kernels and tests.
9812190 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
@lfr-0531
Consolidate all MTP unit tests into a single file.
4af1685 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
@lfr-0531
Add use_mtp_vanilla to mtpConfig and add mtp vanilla tests.
d1818db 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
@lfr-0531
fix.
9b0eccc 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
@lfr-0531
fix position id.
4d199cb 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
@lfr-0531
update post merge tests.
8b8aa4b 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
@lfr-0531
lfr-0531 force-pushed the user/fanrongl/simplify_mtp_vanilla branch from 2c85775 to 8b8aa4b Compare June 19, 2025 10:28
@lfr-0531

lfr-0531 commented Jun 19, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Jun 19, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #9505 [ run ] triggered by Bot

@tensorrt-cicd

tensorrt-cicd commented Jun 19, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #9478 [ run ] completed with state ABORTED

@tensorrt-cicd

tensorrt-cicd commented Jun 19, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #9505 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6972 completed with status: 'FAILURE'

@lfr-0531

lfr-0531 commented Jun 19, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Jun 19, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #9527 [ run ] triggered by Bot

@tensorrt-cicd

tensorrt-cicd commented Jun 19, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #9527 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6990 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

@lfr-0531
lfr-0531 merged commit 5d4ab47 into NVIDIA:main Jun 19, 2025
3 checks passed
@lfr-0531 lfr-0531 mentioned this pull request Jun 20, 2025
Merged
k-l-lambda pushed a commit to k-l-lambda/TensorRT-LLM that referenced this pull request Jun 23, 2025
@lfr-0531 @k-l-lambda
fix: refactor and fix mtp vanilla (NVIDIA#4762)
e342b35 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
@lfr-0531
lfr-0531 deleted the user/fanrongl/simplify_mtp_vanilla branch June 27, 2025 12:43
dominicshanshan pushed a commit to dominicshanshan/TensorRT-LLM that referenced this pull request Jul 9, 2025
@lfr-0531 @dominicshanshan
fix: refactor and fix mtp vanilla (NVIDIA#4762)
d3f059c 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
dominicshanshan pushed a commit to dominicshanshan/TensorRT-LLM that referenced this pull request Jul 10, 2025
@lfr-0531 @dominicshanshan
fix: refactor and fix mtp vanilla (NVIDIA#4762)
e907d7b 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
dominicshanshan pushed a commit to dominicshanshan/TensorRT-LLM that referenced this pull request Jul 10, 2025
@lfr-0531 @dominicshanshan
fix: refactor and fix mtp vanilla (NVIDIA#4762)
546703c 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
dominicshanshan pushed a commit to dominicshanshan/TensorRT-LLM that referenced this pull request Jul 10, 2025
@lfr-0531 @dominicshanshan
fix: refactor and fix mtp vanilla (NVIDIA#4762)
dfb52a4 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
dominicshanshan pushed a commit to dominicshanshan/TensorRT-LLM that referenced this pull request Jul 10, 2025
@lfr-0531 @dominicshanshan
fix: refactor and fix mtp vanilla (NVIDIA#4762)
06ec751 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
dominicshanshan pushed a commit to dominicshanshan/TensorRT-LLM that referenced this pull request Jul 11, 2025
@lfr-0531 @dominicshanshan
fix: refactor and fix mtp vanilla (NVIDIA#4762)
d6b327f 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
dominicshanshan pushed a commit to dominicshanshan/TensorRT-LLM that referenced this pull request Jul 11, 2025
@lfr-0531 @dominicshanshan
fix: refactor and fix mtp vanilla (NVIDIA#4762)
acd17c7 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
dominicshanshan pushed a commit to dominicshanshan/TensorRT-LLM that referenced this pull request Jul 11, 2025
@lfr-0531 @dominicshanshan
fix: refactor and fix mtp vanilla (NVIDIA#4762)
fd32058 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@QiJune QiJune QiJune approved these changes
+1 more reviewer
@yweng0828 yweng0828 yweng0828 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@lfr-0531 @tensorrt-cicd @QiJune @yweng0828