[https://nvbugs/5844149][fix] Fix issues with DSV3.2 perf tests

[brb-nv]

Description

This MR addresses https://nvbugspro.nvidia.com/bug/5844149.

Test Coverage

Ran problem configs on target hardware.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Summary by CodeRabbit

• Chores
  • Adjusted benchmark configuration parameters for improved resource management and performance testing optimization.
  • Updated performance testing infrastructure scripts to streamline resource allocation and job submission processes.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 775f32e5-c0f6-478e-af4d-af99b2fee7f6

📥 Commits

Reviewing files that changed from the base of the PR and between 2335ef8 and 5d515cc.

📒 Files selected for processing (4)

• jenkins/scripts/perf/local/submit.py
• tests/scripts/perf-sanity/disaggregated/gb200_deepseek-v32-fp4_32k4k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-UCX.yaml
• tests/scripts/perf-sanity/disaggregated/gb200_deepseek-v32-fp4_32k4k_con2048_ctx1_dep4_gen1_dep32_eplb288_mtp1_ccb-UCX.yaml
• tests/scripts/perf-sanity/disaggregated/gb200_deepseek-v32-fp4_32k4k_con256_ctx1_dep4_gen1_dep32_eplb0_mtp3_ccb-UCX.yaml

---

📝 Walkthrough

Walkthrough

This PR modifies a performance benchmark submission script and updates three corresponding test configuration files. The Python script removes Slurm GPU resource flags and adds conditional logic to override multi-round settings for generation-only mode, writing an updated config file. Three YAML test configs are adjusted with modified token limits, sequence lengths, and KV cache memory allocation parameters.

Changes

Cohort / File(s)	Summary
Slurm Batch Script jenkins/scripts/perf/local/submit.py	Removed GPU resource flags (#SBATCH --gpus-per-node and --gres=gpu) from generate_sbatch_params(). Added logic in main() to detect benchmark_mode == "gen_only", force multi_round = 1, write modified config to a new YAML file in work_dir, and update config_yaml reference.
Disaggregated GB200 Benchmark Configs tests/scripts/perf-sanity/disaggregated/gb200_deepseek-v32-fp4_32k4k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-UCX.yaml, gb200_deepseek-v32-fp4_32k4k_con2048_ctx1_dep4_gen1_dep32_eplb288_mtp1_ccb-UCX.yaml, gb200_deepseek-v32-fp4_32k4k_con256_ctx1_dep4_gen1_dep32_eplb0_mtp3_ccb-UCX.yaml	Added or adjusted explicit max_input_len and max_seq_len limits in worker_config.gen and worker_config.ctx sections. Reduced kv_cache_config.free_gpu_memory_fraction values (gen: 0.85/0.9→0.5, ctx: 0.6→0.4). Increased cache_transceiver_config.max_tokens_in_buffer from 32768 to 32832. Modified generation token constraints (max_num_tokens reduced in one config).

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	❓ Inconclusive	The description references the bug but lacks explanation of what/why. It mentions test coverage and includes the checklist but doesn't explain the actual changes or the root causes.	Add a clear explanation of the OOM/IMA issues being fixed and describe the technical changes made (e.g., GPU memory adjustments, Slurm flag removal, multi_round config changes).

✅ Passed checks (1 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main fix addressing OOM and IMA issues in DSV3.2 perf tests with proper NVBugs reference.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[brb-nv]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #43808 [ run ] triggered by Bot. Commit: 5d515cc Link to invocation

[tensorrt-cicd]

PR_Github #43808 [ run ] completed with state SUCCESS. Commit: 5d515cc
/LLM/main/L0_MergeRequest_PR pipeline #34283 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[brb-nv]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #43899 [ run ] triggered by Bot. Commit: e0838e9 Link to invocation

[tensorrt-cicd]

PR_Github #43899 [ run ] completed with state FAILURE. Commit: e0838e9
/LLM/main/L0_MergeRequest_PR pipeline #34348 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation