Skip to content

[TRTLLM-4618][feat] Add Nemotron Super 49B FP8 test on RTX6000 Pro (SM120) - #4363

Merged
kaiyux merged 13 commits into
NVIDIA:mainfrom
farazkh80:nemotron-fp8-b40
May 19, 2025
Merged

[TRTLLM-4618][feat] Add Nemotron Super 49B FP8 test on RTX6000 Pro (SM120)#4363
kaiyux merged 13 commits into
NVIDIA:mainfrom
farazkh80:nemotron-fp8-b40

Conversation

@farazkh80

Copy link
Copy Markdown
Collaborator

Description

  • This PR adds tests to end-to-end tests for Nemotron Super 49B FP8(w torch backend) test suite to be run on SM120.
  • The tests will be used by QA as a part of the B40 Bring-up (RTX6000 Pro SM120) effort.

Test Coverage

Single node tests

  • test_e2e.py::test_ptp_quickstart_advanced[Nemotron-Super-49B-v1-FP8-nemotron-nas/Llama-3_3-Nemotron-Super-49B-v1-FP8]

These tests will be included in the SM120 verification plan for QA sign-off.

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--disable-fail-fast --skip-test --stage-list "A10-1, xxx" --gpu-type "A30, H100_PCIe" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-[Post-Merge]-1, xxx"]

Launch build/test pipelines. All previously running jobs will be killed.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests. Will also run L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-[Post-Merge]-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-[Post-Merge]-1, xxx".

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>
@farazkh80

Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "RTXPro6000-PyTorch-[Post-Merge]-1"

@farazkh80
farazkh80 requested a review from pamelap-nvidia May 15, 2025 17:46
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5386 [ run ] triggered by Bot

Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>
@farazkh80

Copy link
Copy Markdown
Collaborator Author

/bot reuse-pipeline

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5395 [ reuse-pipeline ] triggered by Bot

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5386 [ run ] completed with state ABORTED

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5395 [ reuse-pipeline ] completed with state SUCCESS
Can't reuse PR_Github #5386 (Partly Tested) with status: ABORTED

Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>
@farazkh80

Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "RTXPro6000-PyTorch-[Post-Merge]-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5399 [ run ] triggered by Bot

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5399 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #3940 (Partly Tested) completed with status: 'SUCCESS'

@farazkh80

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5417 [ run ] triggered by Bot

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5428 [ run ] triggered by Bot

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5417 [ run ] completed with state ABORTED

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5428 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #3963 completed with status: 'FAILURE'

@farazkh80

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5509 [ reuse-pipeline ] triggered by Bot

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5510 [ run ] triggered by Bot

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5509 [ reuse-pipeline ] completed with state ABORTED
Can't reuse PR_Github #0 with status: UNKNOWN

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5510 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #4015 completed with status: 'FAILURE'

@farazkh80

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5537 [ run ] triggered by Bot

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5537 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #4038 completed with status: 'FAILURE'

@farazkh80

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5575 [ run ] triggered by Bot

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5575 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #4066 completed with status: 'FAILURE'

@farazkh80

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "A10-PyTorch-1, A10-TensorRT-1, A10-TensorRT-2, A10-TensorRT-4, A10-TensorRT-6"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5618 [ run ] triggered by Bot

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5618 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #4103 (Partly Tested) completed with status: 'FAILURE'

@farazkh80

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "A10-PyTorch-1, A10-TensorRT-1, A10-TensorRT-2, A10-TensorRT-4, A10-TensorRT-6"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5629 [ run ] triggered by Bot

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5629 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #4113 (Partly Tested) completed with status: 'SUCCESS'

@farazkh80

Copy link
Copy Markdown
Collaborator Author

/bot --reuse-pipeline

@github-actions

Copy link
Copy Markdown

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--disable-fail-fast --skip-test --stage-list "A10-1, xxx" --gpu-type "A30, H100_PCIe" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-[Post-Merge]-1, xxx"]

Launch build/test pipelines. All previously running jobs will be killed.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests. Will also run L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-[Post-Merge]-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-[Post-Merge]-1, xxx".

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

@farazkh80

Copy link
Copy Markdown
Collaborator Author

/bot reuse-pipeline

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5641 [ reuse-pipeline ] triggered by Bot

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5641 [ reuse-pipeline ] completed with state SUCCESS
Reusing PR_Github #5629 (Partly Tested) for commit c7bcc48

@farazkh80

Copy link
Copy Markdown
Collaborator Author

/bot reuse-pipeline

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5646 [ reuse-pipeline ] triggered by Bot

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #5646 [ reuse-pipeline ] completed with state SUCCESS
Reusing PR_Github #5629 (Partly Tested) for commit e23a255

@kaiyux
kaiyux merged commit 791c209 into NVIDIA:main May 19, 2025
@farazkh80
farazkh80 deleted the nemotron-fp8-b40 branch May 27, 2026 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants