[nvbug 5004744][fix] rewrite completion API to avoid repetitive tokens

[LinPoly]

Description

This PR is fixing the repetitive token issue reported from different sources. For completion API, there maybe multiple prompts in a same request, so we have multiple RequestOutput, we want to process them ASAP so we launch multiple producer() tasks and put the result into a queue, and then we take the result from the queue with consumer().

The problem here is, there is no coordination between producer and consumer, a producer may put several times before consumer takes, but __anext__() function of RequestOutput returns itself (we call it inside producer and put the result into the queue), which means all the entries refer to the same object, and we basically process the same thing for these entries, then we get repetitive tokens.

The fix is to create a separate task to process each RequestOutput, then merge their result.

Test Coverage

Original test is enough, and completion test passed locally

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--disable-fail-fast --skip-test --stage-list "A10-1, xxx" --gpu-type "A30, H100_PCIe" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-[Post-Merge]-1, xxx"]

Launch build/test pipelines. All previously running jobs will be killed.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests. Will also run L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-[Post-Merge]-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-[Post-Merge]-1, xxx".

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

[LinPoly]

@Superjomn @kaiyux Do you guys have time to review this PR? I am open to chat/call if necessary.

[LinPoly]

/bot run

[tensorrt-cicd]

PR_Github #10015 [ run ] triggered by Bot

[Superjomn]

@LinPoly could we add a test to confirm that the repeated-token issue has been resolved?

[tensorrt-cicd]

PR_Github #10015 [ run ] completed with state FAILURE
/LLM/release-0.21/L0_MergeRequest_PR pipeline #46 completed with status: 'FAILURE'

[LinPoly]

/bot run

[tensorrt-cicd]

PR_Github #10041 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #10041 [ run ] completed with state SUCCESS
/LLM/release-0.21/L0_MergeRequest_PR pipeline #48 completed with status: 'SUCCESS'

[LinPoly]

Tested with DSV3 lite fp8 in H200, there is no tangible performance regression with this PR.

[LinPoly]

/bot run

[LinPoly]

could we add a test to confirm that the repeated-token issue has been resolved?

@Superjomn As per offline discussion with Kaiyu, an accuracy evaluation maybe needed for ensuring the serving quality, will add one in a new PR, we can still merge this PR because the bug can be manually triggered by adding sleep into async generator, and there is no repetitive token after this fix even with sleep.

[tensorrt-cicd]

PR_Github #10605 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #10605 [ run ] completed with state SUCCESS
/LLM/release-0.21/L0_MergeRequest_PR pipeline #110 completed with status: 'FAILURE'

[LinPoly]

/bot run

[tensorrt-cicd]

PR_Github #10775 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #10775 [ run ] completed with state SUCCESS
/LLM/release-0.21/L0_MergeRequest_PR pipeline #144 completed with status: 'FAILURE'

[LinPoly]

/bot run

[tensorrt-cicd]

PR_Github #10941 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #10941 [ run ] completed with state SUCCESS
/LLM/release-0.21/L0_MergeRequest_PR pipeline #162 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

[Superjomn]

LGTM

[LinPoly]

/bot run --only-multi-gpu-test

[tensorrt-cicd]

PR_Github #10987 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #10987 [ run ] completed with state SUCCESS
/LLM/release-0.21/L0_MergeRequest_PR pipeline #165 (Partly Tested) completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

[kaiyux]

/bot skip --comment "pipeline has passed"

[tensorrt-cicd]

PR_Github #11105 [ skip ] triggered by Bot

[tensorrt-cicd]

PR_Github #11105 [ skip ] completed with state SUCCESS
Skipping testing for commit c99d4df