Skip to content

[TRTLLM-4932] Add QA accuracy tests for NIM-prioritized models - #4242

Merged
chzblych merged 15 commits into
NVIDIA:release/0.20from
moraxu:add-remaining-qa-nim-acc-tests
May 24, 2025
Merged

[TRTLLM-4932] Add QA accuracy tests for NIM-prioritized models#4242
chzblych merged 15 commits into
NVIDIA:release/0.20from
moraxu:add-remaining-qa-nim-acc-tests

Conversation

@moraxu

@moraxu moraxu commented May 12, 2025 •
edited
Loading

Copy link
Copy Markdown
Collaborator

New tests added:

  • Llama-3.2-1B: added mmlu benchmark - removed due to low accuracy
  • Llama-3.1-Nemotron-Nano-8B-v1: added GSM8K, GPQADiamond benchmarks
  • Llama-3_1-Nemotron-Ultra-253B-v1: added the entire model (FP8 variant is being added to ftp/llm-models)
  • Phi-4-mini-instruct: added the model to the tests; skipped the test as the model likely has to be added to Torch models first (given the current error)

tijyojwad
tijyojwad reviewed May 12, 2025
View reviewed changes
Comment thread tests/integration/defs/accuracy/references/gpqa_diamond.yaml Outdated
Comment thread tests/integration/defs/accuracy/references/mmlu.yaml Outdated
Comment thread tests/integration/defs/accuracy/test_llm_api_pytorch.py Outdated
@moraxu

moraxu commented May 13, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

moraxu
moraxu commented May 13, 2025
View reviewed changes
Comment thread tests/integration/defs/accuracy/references/mmlu.yaml Outdated
Comment thread tests/integration/defs/accuracy/test_llm_api_pytorch.py Outdated
@moraxu

moraxu commented May 13, 2025

Copy link
Copy Markdown
Collaborator Author

@syuoni @crazydemo @LarryXFly - can you review this PR? Feel free to unassign yourself and tag someone else instead

crazydemo
crazydemo reviewed May 13, 2025
View reviewed changes
Comment thread tests/integration/test_lists/qa/examples_test_list.txt Outdated
@moraxu

moraxu commented May 13, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

1 similar comment
@moraxu

moraxu commented May 14, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented May 14, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #5121 [ run ] triggered by Bot

@tensorrt-cicd

tensorrt-cicd commented May 14, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #5121 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #3735 completed with status: 'FAILURE'

@moraxu
moraxu force-pushed the add-remaining-qa-nim-acc-tests branch from c54f485 to 254494f Compare May 14, 2025 19:13
@moraxu

moraxu commented May 14, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented May 14, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #5205 [ run ] triggered by Bot

@tensorrt-cicd

tensorrt-cicd commented May 14, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #5205 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #3798 completed with status: 'FAILURE'

@moraxu

moraxu commented May 14, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented May 14, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #5211 [ run ] triggered by Bot

@tensorrt-cicd

tensorrt-cicd commented May 15, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #5211 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #3803 completed with status: 'SUCCESS'

@moraxu
moraxu requested review from Tracin, crazydemo and syuoni May 15, 2025 05:28
Tracin
Tracin reviewed May 16, 2025
View reviewed changes
Comment thread tests/integration/defs/accuracy/references/mmlu.yaml Outdated
Tracin
Tracin reviewed May 16, 2025
View reviewed changes
Comment thread tests/integration/test_lists/qa/examples_test_list.txt Outdated
@moraxu
moraxu requested review from Tracin and chang-l May 16, 2025 17:33
@moraxu

moraxu commented May 18, 2025

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented May 18, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #5627 [ run ] triggered by Bot

syuoni
syuoni reviewed May 19, 2025
View reviewed changes
Comment thread tests/integration/defs/accuracy/test_llm_api_pytorch.py Outdated
Comment thread tests/integration/defs/accuracy/references/mmlu.yaml Outdated
@tensorrt-cicd

tensorrt-cicd commented May 19, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #5627 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #4111 completed with status: 'FAILURE'

chang-l
chang-l reviewed May 19, 2025
View reviewed changes
Comment thread tests/integration/defs/accuracy/test_llm_api_pytorch.py Outdated
chang-l
chang-l reviewed May 19, 2025
View reviewed changes
Comment thread tests/integration/test_lists/qa/llm_sanity_test.txt Outdated
@tensorrt-cicd

tensorrt-cicd commented May 23, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #6323 [ run ] triggered by Bot

@tensorrt-cicd

tensorrt-cicd commented May 23, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #6323 [ run ] completed with state SUCCESS
/LLM/release-0.20/L0_MergeRequest_PR pipeline #61 completed with status: 'SUCCESS'

@moraxu
moraxu force-pushed the add-remaining-qa-nim-acc-tests branch from 1777c43 to b1c068f Compare May 23, 2025 22:56
@moraxu

moraxu commented May 23, 2025

Copy link
Copy Markdown
Collaborator Author

/bot reuse-pipeline

moraxu added 15 commits May 23, 2025 23:07
@moraxu
Add tests
0317ae2 
Signed-off-by: moraxu <mguzek@nvidia.com>
@moraxu
Add tests v2
736d085 
Signed-off-by: moraxu <mguzek@nvidia.com>
@moraxu
Add fixes
319ff97 
Signed-off-by: moraxu <mguzek@nvidia.com>
@moraxu
Skip fp8 test for Ultra
3e06973 
Signed-off-by: moraxu <mguzek@nvidia.com>
@moraxu
Add tests for Phi
016bd1b 
Signed-off-by: moraxu <mguzek@nvidia.com>
@moraxu
Skip tests for Phi
d3e70eb 
Signed-off-by: moraxu <mguzek@nvidia.com>
@moraxu
Skip tests for Phi - fix
d2131d1 
Signed-off-by: moraxu <mguzek@nvidia.com>
@moraxu
Skip tests for Phi - comment out acc refs
b2887b2 
Signed-off-by: moraxu <mguzek@nvidia.com>
@moraxu
Add more test granularity
ac350c3 
Signed-off-by: moraxu <mguzek@nvidia.com>
@moraxu
Fix examples_test_list.txt
af562a8 
Signed-off-by: moraxu <mguzek@nvidia.com>
@moraxu
Update test list file
93975a8 
Signed-off-by: moraxu <mguzek@nvidia.com>
@moraxu
Update yaml files
b453529 
Signed-off-by: moraxu <mguzek@nvidia.com>
@moraxu
Address review comments
8de254b 
Signed-off-by: moraxu <mguzek@nvidia.com>
@moraxu
Remove MMLU tests
d05b747 
Signed-off-by: moraxu <mguzek@nvidia.com>
@moraxu
Add remaining models
17bdbcd 
Signed-off-by: moraxu <mguzek@nvidia.com>
@moraxu
moraxu force-pushed the add-remaining-qa-nim-acc-tests branch from b1c068f to 17bdbcd Compare May 23, 2025 23:10
@moraxu

moraxu commented May 23, 2025

Copy link
Copy Markdown
Collaborator Author

/bot reuse-pipeline

@tensorrt-cicd

tensorrt-cicd commented May 23, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #6339 [ reuse-pipeline ] triggered by Bot

@tensorrt-cicd

tensorrt-cicd commented May 23, 2025

Copy link
Copy Markdown
Collaborator

PR_Github #6339 [ reuse-pipeline ] completed with state SUCCESS
Reusing PR_Github #6323 for commit 17bdbcd

@chzblych
chzblych merged commit 24153c0 into NVIDIA:release/0.20 May 24, 2025
amirkl94 pushed a commit to amirkl94/TensorRT-LLM that referenced this pull request May 28, 2025
@moraxu @amirkl94
[TRTLLM-4932] Add QA accuracy tests for NIM-prioritized models (NVIDI…
ff60c2e 
…A#4242)

* Add tests

Signed-off-by: moraxu <mguzek@nvidia.com>

* Add tests v2

Signed-off-by: moraxu <mguzek@nvidia.com>

* Add fixes

Signed-off-by: moraxu <mguzek@nvidia.com>

* Skip fp8 test for Ultra

Signed-off-by: moraxu <mguzek@nvidia.com>

* Add tests for Phi

Signed-off-by: moraxu <mguzek@nvidia.com>

* Skip tests for Phi

Signed-off-by: moraxu <mguzek@nvidia.com>

* Skip tests for Phi - fix

Signed-off-by: moraxu <mguzek@nvidia.com>

* Skip tests for Phi - comment out acc refs

Signed-off-by: moraxu <mguzek@nvidia.com>

* Add more test granularity

Signed-off-by: moraxu <mguzek@nvidia.com>

* Fix examples_test_list.txt

Signed-off-by: moraxu <mguzek@nvidia.com>

* Update test list file

Signed-off-by: moraxu <mguzek@nvidia.com>

* Update yaml files

Signed-off-by: moraxu <mguzek@nvidia.com>

* Address review comments

Signed-off-by: moraxu <mguzek@nvidia.com>

* Remove MMLU tests

Signed-off-by: moraxu <mguzek@nvidia.com>

* Add remaining models

Signed-off-by: moraxu <mguzek@nvidia.com>

---------

Signed-off-by: moraxu <mguzek@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@syuoni syuoni syuoni left review comments
@chang-l chang-l chang-l approved these changes
@crazydemo crazydemo crazydemo approved these changes
@schetlur-nv schetlur-nv schetlur-nv approved these changes
@LarryXFly LarryXFly LarryXFly approved these changes
@Tracin Tracin Awaiting requested review from Tracin
+1 more reviewer
@tijyojwad tijyojwad tijyojwad left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

10 participants

@moraxu @tensorrt-cicd @tijyojwad @chang-l @Tracin @syuoni @crazydemo @schetlur-nv @LarryXFly @chzblych