Skip to content

[TRTLLM-4995][feat] TRTLLM Sampler log probs support - #4836

Merged
dcampora merged 6 commits into
NVIDIA:mainfrom
dcampora:user/dcampora/trtllm_sampler_log_probs
Jun 11, 2025
Merged

[TRTLLM-4995][feat] TRTLLM Sampler log probs support#4836
dcampora merged 6 commits into
NVIDIA:mainfrom
dcampora:user/dcampora/trtllm_sampler_log_probs

Conversation

@dcampora

@dcampora dcampora commented Jun 2, 2025 •
edited
Loading

Copy link
Copy Markdown
Collaborator

feat: TRTLLM Sampler log probs support

This PR adds support for log probs in the TRTLLM Sampler.

@dcampora
dcampora requested review from a team as code owners June 2, 2025 12:15
@dcampora
dcampora requested review from mikeiovine and suyoggupta June 2, 2025 12:15
@dcampora dcampora changed the title Draft: feat: trtllm sampler log probs feat: TRTLLM Sampler log probs support Jun 3, 2025
@dcampora
dcampora requested a review from Funatiq June 3, 2025 07:30
@dcampora

dcampora commented Jun 3, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Jun 3, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7300 [ run ] triggered by Bot

@tensorrt-cicd

tensorrt-cicd commented Jun 3, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7300 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #5290 completed with status: 'FAILURE'

@dcampora
dcampora force-pushed the user/dcampora/trtllm_sampler_log_probs branch from e67f00f to c91c824 Compare June 3, 2025 08:09
@dcampora

dcampora commented Jun 3, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Jun 3, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7308 [ run ] triggered by Bot

@tensorrt-cicd

tensorrt-cicd commented Jun 3, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7308 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #5296 completed with status: 'FAILURE'

Funatiq
Funatiq reviewed Jun 3, 2025
View reviewed changes
Comment thread tensorrt_llm/_torch/pyexecutor/llm_request.py Outdated
Comment thread tensorrt_llm/_torch/pyexecutor/llm_request.py Outdated
Comment thread tensorrt_llm/_torch/pyexecutor/llm_request.py Outdated
Comment thread tensorrt_llm/_torch/pyexecutor/sampler.py Outdated
@dcampora

dcampora commented Jun 4, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Jun 4, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7426 [ run ] triggered by Bot

@tensorrt-cicd

tensorrt-cicd commented Jun 4, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7426 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #5387 completed with status: 'FAILURE'

@dcampora
dcampora force-pushed the user/dcampora/trtllm_sampler_log_probs branch from c8459ae to 48af1b9 Compare June 4, 2025 07:19
@dcampora

dcampora commented Jun 4, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jun 4, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7459 [ run ] triggered by Bot

@tensorrt-cicd

tensorrt-cicd commented Jun 4, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7459 [ run ] completed with state ABORTED

@dcampora
dcampora force-pushed the user/dcampora/trtllm_sampler_log_probs branch from 48af1b9 to aaaccf9 Compare June 6, 2025 07:30
@dcampora

dcampora commented Jun 6, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Jun 6, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7873 [ run ] triggered by Bot

@tensorrt-cicd

tensorrt-cicd commented Jun 6, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7873 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #5689 completed with status: 'FAILURE'

@dcampora

dcampora commented Jun 7, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Jun 7, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7993 [ run ] triggered by Bot

@tensorrt-cicd

tensorrt-cicd commented Jun 7, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #7993 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #5792 completed with status: 'FAILURE'

@dcampora dcampora mentioned this pull request Jun 9, 2025
Merged
@dcampora
Support log probs in trtllm sampler.
2c4a441 
Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>
dcampora added 4 commits June 10, 2025 05:19
@dcampora
Enabled and fixed return_logits in sampler.
30e7ca7 
Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>
@dcampora
Formatting.
4910ce0 
Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>
@dcampora
Fetch from right location in decoder_State.
be0a774 
Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>
@dcampora
Apply comments.
401557e 
Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>
@dcampora
dcampora force-pushed the user/dcampora/trtllm_sampler_log_probs branch from aaaccf9 to 401557e Compare June 10, 2025 05:20
@dcampora

dcampora commented Jun 10, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@dcampora dcampora changed the title feat: TRTLLM Sampler log probs support [TRTLLM-4995][feat] TRTLLM Sampler log probs support Jun 10, 2025
@tensorrt-cicd

tensorrt-cicd commented Jun 10, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #8212 [ run ] triggered by Bot

@dcampora
Fix sampler num_new_tokens access.
dffac7a 
Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>
@tensorrt-cicd

tensorrt-cicd commented Jun 10, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #8212 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #5956 completed with status: 'FAILURE'

@dcampora

dcampora commented Jun 10, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Jun 10, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #8254 [ run ] triggered by Bot

Funatiq
Funatiq approved these changes Jun 10, 2025
View reviewed changes

@Funatiq Funatiq left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tensorrt-cicd

tensorrt-cicd commented Jun 10, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #8254 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #5978 completed with status: 'SUCCESS'

@dcampora
dcampora merged commit fdf1c47 into NVIDIA:main Jun 11, 2025
@amitz-nv amitz-nv mentioned this pull request Jun 11, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Funatiq Funatiq Funatiq approved these changes
@QiJune QiJune QiJune approved these changes
@mikeiovine mikeiovine Awaiting requested review from mikeiovine mikeiovine was automatically assigned from NVIDIA/trt-llm-torch-runtime-devs
@suyoggupta suyoggupta Awaiting requested review from suyoggupta suyoggupta was automatically assigned from NVIDIA/trt-llm-torch-devs

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@dcampora @tensorrt-cicd @Funatiq @QiJune