[None][test] Adjust perf test cases to avoid OOM and remove outdated test cases

[yufeiwu-nv]

Description

• Restore the obsolete DeepSeek R1 configuration cleanup from [None][refactor] Clean up model paths and remove deprecated configurations in performance tests #17057 after [None][test] add e2e key model perf test #17047 unintentionally reintroduced the stale aliases and overrides.
• Restrict memory-heavy Qwen3.5 cases to GPUs with sufficient capacity.
• Move the DeepSeek V4 Pro DSpark 8192,1024 case to B300 while retaining the shorter shared B200/B300 case.
• Reduce redundant performance coverage that can trigger OOM without removing the representative workloads.

Test Coverage

• python3 scripts/check_test_list.py --validate
• Python syntax and YAML condition validation for the modified performance configuration files
• Targeted pre-commit checks for all modified files

PR Checklist

• The description explains the motivation and changes.
• Coding guidelines were followed.
• Relevant static validation and pre-commit checks passed.
• No API changes, new dependencies, ownership changes, or architecture changes are introduced.

Dev Engineer Review

• Performance coverage changes remove obsolete model configurations and add GPU-specific Qwen3.5 and DeepSeek V4 cases.
• The changes align with the OOM reduction objective.
• The qwen3_0.6b reduction from 256 to 64 requests needs a timeout-related rationale because it creates a new test ID without a baseline.
• Stale declarations for starcoder2_3b, qwen_14b_chat, and qwen2_7b_instruct remain in allowed_configs.py. Remove them or document their purpose.
• Condition renumbering preserves cases without duplicates or omissions.
• Validation includes test-list checks, Python syntax checks, YAML checks, and targeted pre-commit checks.

QA Engineer Review

• Modified QA test list: tests/integration/test_lists/qa/llm_perf_core.yml.
• Removed legacy Nemotron, DeepSeek, Kimi, and MiniMax M2.5 cases.
• Added DeepSeek R1 throughput, DeepSeek V4 Pro DSpark B300, Qwen3.5 27B, Qwen3.5 397B FP8, and Qwen3.5 122B BF16 cases.
• Separated B300 and RTX-6000 Server conditions and renumbered the conditions.
• No test functions were added, modified, or removed.
• Verdict: needs follow-up because CBTS coverage data and baseline impact for the changed workloads are not confirmed.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0e371189-db25-4456-ae92-b283aedcaac7

📥 Commits

Reviewing files that changed from the base of the PR and between 63eb095 and 15f4ccf.

📒 Files selected for processing (4)

• tests/integration/defs/perf/_model_paths.py
• tests/integration/defs/perf/pytorch_model_config.py
• tests/integration/defs/perf/test_perf.py
• tests/integration/test_lists/qa/llm_perf_core.yml

💤 Files with no reviewable changes (2)

• tests/integration/defs/perf/test_perf.py
• tests/integration/defs/perf/_model_paths.py

🚧 Files skipped from review as they are similar to previous changes (1)

• tests/integration/test_lists/qa/llm_perf_core.yml

---

Walkthrough

The performance configuration replaces legacy model mappings and specialized cases with current variants. It updates remote-code handling and reorganizes workload coverage across GPU-specific conditions.

Changes

Performance configuration and test matrix

Layer / File(s)	Summary
Model mappings and runtime patterns tests/integration/defs/perf/_model_paths.py, tests/integration/defs/perf/pytorch_model_config.py, tests/integration/defs/perf/test_perf.py	Model paths now include current Llama, DeepSeek, MiniMax, and DeepSeek V4 variants. Legacy specialized configurations and selected remote-code entries were removed.
GPU condition organization tests/integration/test_lists/qa/llm_perf_core.yml	A B300-only condition was added. Later GPU conditions were renumbered through the RTX-6000 Server section.
GPU-specific workload coverage tests/integration/test_lists/qa/llm_perf_core.yml	Qwen3.5 workloads were added across several GPU conditions. MiniMax M3 replaced MiniMax M2.5 and DeepSeek V3.2 workloads. DeepSeek R1 and DeepSeek V4 Pro DSpark workloads were added or moved.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• NVIDIA/TensorRT-LLM#17310: Restores DeepSeek-V4-Pro DSpark accuracy coverage and adjusts its waiver and threshold.

Suggested reviewers: bowenfu, larryxfly, schetlur-nv

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main performance test cleanup and OOM-avoidance changes.
Description check	✅ Passed	The description includes the required sections and explains the motivation, changes, validation, and checklist status.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[BowenFu]

The list moves look right and I verified the cleanup is safe: none of the 13 aliases dropped from MODEL_PATH_DICT, nor the ~140 lines of deepseek_r1* blocks in pytorch_model_config.py, nor 6 of the 7 sampler labels, is referenced by any live entry in llm_perf_core.yml, llm_perf_disagg.yml, llm_perf_multinode.* or llm_spark_perf.yml. Condition renumbering also checks out — no case dropped or double-listed.

Two things that aren't OOM changes and shouldn't ride along silently:

1. The sampler label fix turns non-greedy sampling ON for a live case. sampler_options_config.py:32-38 rewrites ...-input_output_len:128,128-gpus:4 to -gpus:8. Your comment is right — to_string() always injects maxbs:/maxnt: and drops tp: when tp_size == num_gpus, so the old label was dead. But the new one matches llm_perf_core.yml:59, which is currently running. That case goes from default sampling to top_k=4, top_p=0.5, temperature=0.5, so its perf number will step. Correct fix, but it needs its own line in the description so whoever sees the baseline move knows why.

2. qwen3_0.6b goes from reqs:256 to reqs:64 (llm_perf_core.yml:40) — a 4× workload cut, and it's a new test ID, so it starts with no baseline. The comment right above it says the con:1 + small-model pairing exists to amplify host-side overhead; 64 requests at con:1 is a short sample for that signal. If this is a timeout fix rather than OOM, worth saying so.

Non-blocking: starcoder2_3b, qwen_14b_chat and qwen2_7b_instruct are dropped from MODEL_PATH_DICT but still declared in allowed_configs.py (:357, :1198, :1253). They still pass PerfTestConfig.validate() via the allowed_models fallback, then resolve to "" and get skipped rather than erroring. Nothing runs them today, so this is only a trap for whoever adds them back — dropping their allowed_configs.py entries too would close it.

Holding for now on (1) and (2); everything else looks good to me.

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[yufeiwu-nv]

/bot run

[tensorrt-cicd]

PR_Github #64032 [ run ] triggered by Bot. Commit: b699db0 Link to invocation

[BowenFu]

The head moved a long way since my last pass, and this one is a much bigger change than "avoid OOM and remove outdated test cases". Counting it mechanically at fab1c974f:

• tests/integration/test_lists/qa/ overall: 297 → 218 perf cases.
• llm_perf_core.yml alone: 243 → 164. 79 net removals, 0 net additions — every + line is a case moved between condition blocks, not a new one.

Nine model families lose all perf coverage, and their aliases come out of _model_paths.py with them:

model	main	this head
llama_v3.1_nemotron_ultra_253b (+_fp8)	21	0
llama_v3.3_nemotron_super_49b (+_fp8)	20	0
minimax_m2.5_fp8	13	0
deepseek_v3.2_fp4 / deepseek_v3.2_fp8	9	0
llama_v3.1_nemotron_nano_8b_fp8	3	0
kimi_k2_nvfp4	1	0

I checked every list under qa/ at your head, not just llm_perf_core.yml — none of these land anywhere else. llama_v3.3_70b_instruct{,_fp4,_fp8} goes 14 → 3.

DeepSeek-V3.2, MiniMax-M2.5 and Kimi-K2-Thinking aren't outdated, so I don't think the title covers them. Two readings and the diff doesn't tell me which:

1. this is a deliberate retirement — then it belongs in the description with the perf owners signing off, because losing a model's entire perf signal isn't a test-list cleanup; or
2. they were meant to move and the move was lost across the rebase/merge.

Which one? The OOM-driven part I have no problem with — the condition re-split (new # 8: B300 block, the qwen3.5_27b and qwen3.5_122b_a10b moves) reads correctly, and the new comments recording the memory limits are a real improvement. If you split the retirement out, I'll approve the rest.

Also: thanks for dropping the sampler_options_config.py change. My earlier finding — the -gpus:4 → -gpus:8 relabel matching llm_perf_core.yml:59 and flipping a live case to top_k=4/top_p=0.5/temperature=0.5 — no longer applies at this head.

[tensorrt-cicd]

PR_Github #64032 [ run ] completed with state SUCCESS. Commit: b699db0
/LLM/main/L0_MergeRequest_PR pipeline #51962 completed with status: 'FAILURE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[brnguyen2]

Approving — the comments below are optional touch-ups, not blockers.

Deletions check out — I grepped tests/ for every removed model key and every removed pytorch_model_config pattern (including the deepseek_v3_lite chunked-prefill block) and found no dangling references, and the surviving 'deepseek_r1' pattern still substring-matches deepseek_r1_0528_* so those defaults are unchanged. Condition renumbering is consistent, no duplicated case across overlapping blocks.

The two re-gating moves are each half-applied; details inline. Both are cheap to fix and worth fixing now, since the whole point of the PR is that the gating be trustworthy.

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[yufeiwu-nv]

/bot skip --comment "only test list modify"

[tensorrt-cicd]

PR_Github #64182 [ skip ] triggered by Bot. Commit: 15f4ccf Link to invocation

[tensorrt-cicd]

PR_Github #64182 [ skip ] completed with state SUCCESS. Commit: 15f4ccf
Skipping testing for commit 15f4ccf

Link to invocation

[BowenFu]

Re-counted at 15f4ccf83. The deletion grew since my last comment, and Llama-3.3-70B is now in it:

model	main	this head
llama_v3.3_70b_instruct (+_fp8, _fp4)	11	0
llama_v3.1_8b_instruct_fp8	1	0

llm_perf_core.yml is 243 → 164 cases: 79 removed, 0 added. I pulled all 15 files under qa/ at this head — the only Llama-3.3-70B entries left anywhere are three maxbs:1 ... con:1 single-GPU latency cases in llm_spark_perf.yml. Every multi-GPU 70B perf case (gpus:2/4/8, incl. the 20k/2k TIMEOUT(120) one) is gone, and minimax_m2.5_fp8, deepseek_v3.2_fp4/fp8, kimi_k2_nvfp4 and the Nemotron Ultra/Super families are at zero.

The 70B aliases are still in _model_paths.py, so this isn't a model retirement — the plumbing stays and only the coverage goes.

This is now approved with CI skipped as "only test list modify", so it can land any moment, and my earlier question is still open. Could you either split the coverage removals out, or say per family why each is obsolete/replaced? The OOM half — the new # 8: B300 block, the qwen3.5_27b / qwen3.5_122b_a10b re-splits, the memory-limit comments — I'd approve on its own today.