Skip to content

[fix] Fix a few issues with EAGLE3 in PyTorch backend - #3686

Merged
mikeiovine merged 1 commit into
NVIDIA:mainfrom
mikeiovine:fix-flashinfer
Apr 28, 2025
Merged

[fix] Fix a few issues with EAGLE3 in PyTorch backend#3686
mikeiovine merged 1 commit into
NVIDIA:mainfrom
mikeiovine:fix-flashinfer

Conversation

@mikeiovine

@mikeiovine mikeiovine commented Apr 18, 2025

Copy link
Copy Markdown
Collaborator
  • Add a few more explanatory comments
  • Clean up draft model hidden state saving logic.
  • Add eagle3 support to flashinfer (includes CUDA graph support). Flashinfer is still useful for debugging; it helped me identify some issues in the warmup phase.
  • Fix an illegal memory access related to maximum sequence length in warmup phase. This currently does not cause crashes on the TRTLLM attention path (but it's still UB!)

@mikeiovine
mikeiovine requested a review from hlu1 April 18, 2025 20:02
@mikeiovine mikeiovine self-assigned this Apr 18, 2025
@mikeiovine

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #2791 [ run ] triggered by Bot

@mikeiovine
mikeiovine requested a review from lfr-0531 April 18, 2025 21:42
Comment thread tensorrt_llm/_torch/attention_backend/flashinfer.py Outdated
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #2791 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #1974 completed with status: 'FAILURE'

@mikeiovine mikeiovine changed the title fix: Fix a few issues with EAGLE3 in PyTorch backend [fix] Fix a few issues with EAGLE3 in PyTorch backend Apr 25, 2025
@mikeiovine
mikeiovine force-pushed the fix-flashinfer branch 3 times, most recently from e978e53 to d85cb81 Compare April 25, 2025 00:28
@mikeiovine

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #3333 [ run ] triggered by Bot

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #3333 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #2325 completed with status: 'FAILURE'

@lfr-0531
lfr-0531 requested a review from yuxianq April 27, 2025 08:44
@lfr-0531

Copy link
Copy Markdown
Collaborator

Adding @yuxianq for vis of this PR. There are some changes about the flashinfer backend.

Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
@mikeiovine

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #3623 [ run ] triggered by Bot

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #3623 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #2559 completed with status: 'SUCCESS'

@mikeiovine

Copy link
Copy Markdown
Collaborator Author

Will followup with @yuxianq offline if more changes are requested. For now I think the functional changes to the flashinfer backend are small enough to just land this (and they're pretty much limited to the speculation path only). The bulk of this PR is bug fixing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants