[None][test] Add tests for all database configs. by fsaady · Pull Request #11653 · NVIDIA/TensorRT-LLM

fsaady · 2026-02-23T14:49:16Z

Description

Configs under examples/configs/curated/ were not tested. Added them to tests/unittest/llmapi/test_config_database.py and added new tests for better coverage:

Testing that the config can be loaded into llm args.
Sanity testing that the config can be used with trtllm-serve.
Checking that the configs do not have default values set.

Test Coverage

Extended tests/unittest/llmapi/test_config_database.py to include the tests described in the description and to include both configs under examples/configs/curated/ and examples/configs/database/.

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md
and the scripts/test_to_stage_mapping.py helper.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

…onfigs Signed-off-by: Venky <23023424+venkywonka@users.noreply.github.com>

Signed-off-by: Fadi Saady <fsaady@nvidia.com>

- Mock serve functions to make sure the curated configs for trtllm-serve. Signed-off-by: Fadi Saady <fsaady@nvidia.com>

…nfigs Signed-off-by: Fadi Saady <fsaady@nvidia.com>

- Instead of converting load_format to Enum after dumping and loading, using the default mode='python' for model dump. Signed-off-by: Fadi Saady <fsaady@nvidia.com>

- Using same tests for curated/ and database/ configs. - Removed tests for hardcoded deprecated keys since it will be covered in test_database_yaml_config_validates_against_llm_args. Signed-off-by: Fadi Saady <fsaady@nvidia.com>

fsaady · 2026-02-23T14:50:38Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-02-23T14:57:37Z

PR_Github #36523 [ run ] triggered by Bot. Commit: fca59e4 Link to invocation

tensorrt-cicd · 2026-02-23T19:10:23Z

PR_Github #36523 [ run ] completed with state SUCCESS. Commit: fca59e4
/LLM/main/L0_MergeRequest_PR pipeline #28259 completed with status: 'FAILURE'

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

- This is cleaner. Signed-off-by: Fadi Saady <fsaady@nvidia.com>

…ase.py - Added lookup.yaml for curated configs. - Added optional arch field in lookup to get model defaults for specific model archs. Signed-off-by: Fadi Saady <fsaady@nvidia.com>

… fsaady/fix-curated-configs Signed-off-by: Fadi Saady <fsaady@nvidia.com>

…ig_database. Signed-off-by: Fadi Saady <fsaady@nvidia.com>

fsaady · 2026-02-24T14:42:45Z

/bot run

tensorrt-cicd · 2026-02-24T14:49:48Z

PR_Github #36656 [ run ] triggered by Bot. Commit: 2ba3870 Link to invocation

tensorrt-cicd · 2026-02-24T18:11:56Z

PR_Github #36656 [ run ] completed with state SUCCESS. Commit: 2ba3870
/LLM/main/L0_MergeRequest_PR pipeline #28378 completed with status: 'FAILURE'

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

fsaady · 2026-02-24T18:37:32Z

/bot run

tensorrt-cicd · 2026-02-24T18:47:58Z

PR_Github #36680 [ run ] triggered by Bot. Commit: 2039bbf Link to invocation

tensorrt-cicd · 2026-02-24T21:48:31Z

PR_Github #36680 [ run ] completed with state SUCCESS. Commit: 2039bbf
/LLM/main/L0_MergeRequest_PR pipeline #28401 completed with status: 'FAILURE'

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

…nfigs Signed-off-by: Fadi Saady <fsaady@nvidia.com>

…aml. - TODO: check that all entries have an arch field in unit tests. Signed-off-by: Fadi Saady <fsaady@nvidia.com>

- Added a unit test in tests/unittest/llmapi/test_config_database.py. - Fixed kimi k2 curated config. Signed-off-by: Fadi Saady <fsaady@nvidia.com>

fsaady · 2026-02-25T11:07:33Z

/bot run

tensorrt-cicd · 2026-02-25T11:13:38Z

PR_Github #36784 [ run ] triggered by Bot. Commit: 058739a Link to invocation

tensorrt-cicd · 2026-02-25T16:45:14Z

PR_Github #36784 [ run ] completed with state SUCCESS. Commit: 058739a
/LLM/main/L0_MergeRequest_PR pipeline #28487 completed with status: 'FAILURE'

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

fsaady · 2026-02-25T16:55:40Z

/bot run

tensorrt-cicd · 2026-02-25T17:03:11Z

PR_Github #36818 [ run ] triggered by Bot. Commit: 058739a Link to invocation

Signed-off-by: Fadi Saady <fsaady@nvidia.com>

… fsaady/fix-curated-configs Signed-off-by: Fadi Saady <fsaady@nvidia.com>

Signed-off-by: Fadi Saady <fsaady@nvidia.com>

tensorrt-cicd · 2026-02-25T20:08:21Z

PR_Github #36818 [ run ] completed with state SUCCESS. Commit: 058739a
/LLM/main/L0_MergeRequest_PR pipeline #28511 completed with status: 'SUCCESS'

Link to invocation

fsaady · 2026-02-25T22:30:52Z

/bot run

tensorrt-cicd · 2026-02-25T22:38:08Z

PR_Github #36835 [ run ] triggered by Bot. Commit: e2460b5 Link to invocation

tensorrt-cicd · 2026-02-26T01:04:48Z

PR_Github #36835 [ run ] completed with state SUCCESS. Commit: e2460b5
/LLM/main/L0_MergeRequest_PR pipeline #28523 completed with status: 'FAILURE'

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

fsaady · 2026-02-26T08:41:47Z

/bot run

tensorrt-cicd · 2026-02-26T08:48:20Z

PR_Github #36895 [ run ] triggered by Bot. Commit: e8315e3 Link to invocation

tensorrt-cicd · 2026-02-26T13:52:29Z

PR_Github #36895 [ run ] completed with state SUCCESS. Commit: e8315e3
/LLM/main/L0_MergeRequest_PR pipeline #28574 completed with status: 'SUCCESS'

Link to invocation

…phaned high-conc configs Align with PR NVIDIA#11653's CI enforcement (test_database_yaml_config_does_not_set_default_leaves): - Add prune_default_leaves() to pull_sa_configs.py: removes config leaves matching TorchLlmArgs defaults (e.g., backend: pytorch, enable_attention_dp: false) - Add load_defaults_snapshot() with cached JSON + auto-refresh when tensorrt_llm is importable - Add HF_MODEL_TO_ARCH mapping; populate arch field in lookup.yaml entries - Remove hardcoded backend: pytorch and conditional enable_attention_dp: false - Delete 30 orphaned configs exceeding max-concurrency 2048 - Re-pull all configs with pruning enabled, regenerate artifacts - All 1299 test_config_database tests pass Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>

Signed-off-by: Venky <23023424+venkywonka@users.noreply.github.com> Signed-off-by: Fadi Saady <fsaady@nvidia.com> Co-authored-by: Venky <23023424+venkywonka@users.noreply.github.com>

venkywonka and others added 6 commits February 19, 2026 08:25

[None][test] unify yaml validation harness for curated and database c…

d6badd3

…onfigs Signed-off-by: Venky <23023424+venkywonka@users.noreply.github.com>

[None][chore] Removing print_iter_log from curated configs.

ac98aac

Signed-off-by: Fadi Saady <fsaady@nvidia.com>

[None][test] extend curated config unit tests.

a15ea4f

- Mock serve functions to make sure the curated configs for trtllm-serve. Signed-off-by: Fadi Saady <fsaady@nvidia.com>

Merge remote-tracking branch 'origin/main' into fsaady/fix-curated-co…

f1e5647

…nfigs Signed-off-by: Fadi Saady <fsaady@nvidia.com>

[None][test] removing load_format hack from yaml_validation_harness.

7ca2544

- Instead of converting load_format to Enum after dumping and loading, using the default mode='python' for model dump. Signed-off-by: Fadi Saady <fsaady@nvidia.com>

[None][test] Moving config database tests into one place.

fca59e4

- Using same tests for curated/ and database/ configs. - Removed tests for hardcoded deprecated keys since it will be covered in test_database_yaml_config_validates_against_llm_args. Signed-off-by: Fadi Saady <fsaady@nvidia.com>

fsaady requested a review from a team as a code owner February 23, 2026 14:49

fsaady requested review from kaiyux and laikhtewari February 23, 2026 14:49

fsaady mentioned this pull request Feb 23, 2026

[None][fix] unify yaml validation harness for curated and database c… #11585

Closed

1 task

anish-shanbhag reviewed Feb 23, 2026

View reviewed changes

Comment thread tests/unittest/llmapi/yaml_validation_harness.py Outdated

Comment thread tests/unittest/llmapi/test_config_database.py Outdated

fsaady added 5 commits February 24, 2026 12:58

Merge branch 'main' into fsaady/fix-curated-configs

ff9c2df

[None][test] Removing assert_custom_policy, calling policy directly.

5a6d7d6

- This is cleaner. Signed-off-by: Fadi Saady <fsaady@nvidia.com>

[None][fix] Handling model-specific default args in test_config_datab…

fae9ea3

…ase.py - Added lookup.yaml for curated configs. - Added optional arch field in lookup to get model defaults for specific model archs. Signed-off-by: Fadi Saady <fsaady@nvidia.com>

Merge remote-tracking branch 'origin/fsaady/fix-curated-configs' into…

ca44e7e

… fsaady/fix-curated-configs Signed-off-by: Fadi Saady <fsaady@nvidia.com>

[None][chore] Check that all configs have a lookup entry in test_conf…

2ba3870

…ig_database. Signed-off-by: Fadi Saady <fsaady@nvidia.com>

Merge branch 'main' into fsaady/fix-curated-configs

2039bbf

anish-shanbhag mentioned this pull request Feb 24, 2026

[None] [fix] Restructure kv cache memory ratio parameters in curated .yaml config files #11511

Merged

Merge remote-tracking branch 'origin/main' into fsaady/fix-curated-co…

8b4fd78

…nfigs Signed-off-by: Fadi Saady <fsaady@nvidia.com>

fsaady mentioned this pull request Feb 25, 2026

[None][test] extend curated config unit tests. venkywonka/TensorRT-LLM#4

Closed

1 task

[None][chore] Adding model arch to examples/configs/database/lookup.y…

3a68691

…aml. - TODO: check that all entries have an arch field in unit tests. Signed-off-by: Fadi Saady <fsaady@nvidia.com>

fsaady added 2 commits February 25, 2026 12:00

[None][test] Forcing entries in db lookup.yaml to specify architecture.

861caab

- Added a unit test in tests/unittest/llmapi/test_config_database.py. - Fixed kimi k2 curated config. Signed-off-by: Fadi Saady <fsaady@nvidia.com>

Merge branch 'main' into fsaady/fix-curated-configs

058739a

fsaady enabled auto-merge (squash) February 25, 2026 11:04

fsaady added 3 commits February 25, 2026 18:29

[None][chore] removing unneeded mock in test_config_database.

28ec9dd

Signed-off-by: Fadi Saady <fsaady@nvidia.com>

Merge remote-tracking branch 'origin/fsaady/fix-curated-configs' into…

b6806ac

… fsaady/fix-curated-configs Signed-off-by: Fadi Saady <fsaady@nvidia.com>

[None][chore] minor fix.

58aeb59

Signed-off-by: Fadi Saady <fsaady@nvidia.com>

anish-shanbhag approved these changes Feb 25, 2026

View reviewed changes

Merge branch 'main' into fsaady/fix-curated-configs

e2460b5

Merge branch 'main' into fsaady/fix-curated-configs

e8315e3

svc-trtllm-gh-bot added the Community want to contribute PRs initiated from Community label Feb 26, 2026

venkywonka approved these changes Feb 26, 2026

View reviewed changes

fsaady merged commit 41dd9e0 into NVIDIA:main Feb 26, 2026
5 checks passed

lfr-0531 mentioned this pull request Jul 3, 2026

[None][feat] Add DeepSeek-V4-Pro curated configs #15919

Merged

1 task

Uh oh!

Conversation

fsaady commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

GitHub Bot Help

kill

skip

reuse-pipeline

Uh oh!

fsaady commented Feb 23, 2026

Uh oh!

tensorrt-cicd commented Feb 23, 2026

Uh oh!

tensorrt-cicd commented Feb 23, 2026

Uh oh!

Uh oh!

Uh oh!

fsaady commented Feb 24, 2026

Uh oh!

tensorrt-cicd commented Feb 24, 2026

Uh oh!

tensorrt-cicd commented Feb 24, 2026

Uh oh!

fsaady commented Feb 24, 2026

Uh oh!

tensorrt-cicd commented Feb 24, 2026

Uh oh!

tensorrt-cicd commented Feb 24, 2026

Uh oh!

fsaady commented Feb 25, 2026

Uh oh!

tensorrt-cicd commented Feb 25, 2026

Uh oh!

tensorrt-cicd commented Feb 25, 2026

Uh oh!

fsaady commented Feb 25, 2026

Uh oh!

tensorrt-cicd commented Feb 25, 2026

Uh oh!

tensorrt-cicd commented Feb 25, 2026

Uh oh!

fsaady commented Feb 25, 2026

Uh oh!

tensorrt-cicd commented Feb 25, 2026

Uh oh!

tensorrt-cicd commented Feb 26, 2026

Uh oh!

fsaady commented Feb 26, 2026

Uh oh!

tensorrt-cicd commented Feb 26, 2026

Uh oh!

tensorrt-cicd commented Feb 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

fsaady commented Feb 23, 2026 •

edited

Loading