[https://nvbugs/6510284][fix] Cap gen-only benchmark queue size

[chienchunhung]

What

• Cap TLLM_BENCHMARK_REQ_QUEUES_SIZE to the GEN executor's maximum active-request capacity in both Jenkins CI and local SLURM submitters.
• Use max_batch_size * tensor_parallel_size for attention-DP GEN workers and max_batch_size otherwise.
• Emit a warning when requested concurrency is above the reachable fill target.
• Add regression coverage for all four affected DeepSeek-V4-Pro FP4 8k1k gen-only configurations.

Why

TLLM_BENCHMARK_REQ_QUEUES_SIZE is used here as a gen-only benchmark fill target. The benchmark fill gate withholds the first decode forward until the executor has fetched and admitted at least that many requests.

Fetched requests occupy the executor's active-request capacity: max_batch_size * tensor_parallel_size with attention DP, or max_batch_size otherwise. If the fill target is above that capacity, the executor needs a forward pass to complete requests and free slots, but the gate prohibits that forward until the unreachable target is met. This creates the iter=0 livelock.

This invariant applies only to the disaggregated gen-only benchmark path when TLLM_BENCHMARK_REQ_QUEUES_SIZE enables the fill gate. Regular production serving does not enable this gate; its pending queue may safely exceed active capacity and is drained incrementally as requests complete. The ctx_only and e2e benchmark modes are also unaffected.

For the four NVBUG 6510284 cases, this changes the unreachable targets as follows:

concurrency	reachable fill target
8	1
180	128
666	512
4301	4096

Timeline

• Add sleep function for disagg gen-only benchmarking #5398 introduced the gen-only fill gate. The uncapped concurrency assignment appeared in [None] [chore] Update wide-ep genonly scripts #6995 and entered the Jenkins multi-node disaggregated path in [TRTLLM-8952][feat] Support Multi-Node Disagg Perf Test in CI #9138.
• [None][fix] Cap TLLM_BENCHMARK_REQ_QUEUES_SIZE to avoid fill-loop hang #13065 capped the example submitter only; the Jenkins CI/local copies remained uncapped through the [None][test] Add TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS in Spec Decoding Perf Test #14438 submitter unification.
• [None][test] Add DeepSeek-V4-Pro perf sanity cases on GB300 #16540 added the DeepSeek-V4-Pro configurations with gen_only disabled; [None][test] Enable gen_only + ctx_only DeepSeek-V4-Pro perf-sanity on GB300 #16795 enabled those tests and exposed the latent capacity mismatch.

Recommended general fix

This PR intentionally keeps the implementation scoped to restoring the four blocked NVBUG 6510284 tests. The submitters derive a reachable fill target from the configuration values available before launch.

As a follow-up, the safety clamp should move into PyExecutor, after the runtime has derived its authoritative admission ceiling, including attention-DP and other runtime constraints. Launchers would continue to pass the requested benchmark concurrency unchanged, while the executor would use an effective target such as:

total_max = (
    tp_size * max_num_active_requests
    if enable_attention_dp
    else max_num_active_requests
)
fill_target = min(requested_fill_target, total_max)

The executor should log both requested and effective targets when it clamps. This would provide one authoritative implementation for Jenkins, local SLURM, examples, and custom launchers. A fully general multi-GEN solution may also need an explicit per-worker benchmark-round target or end-of-burst signal, because global concurrency does not guarantee equal request distribution across GEN workers.

That broader cleanup is intentionally deferred so this PR can first close the immediate test gap.

Verification

• Targeted pipeline #50194 completed with SUCCESS on commit b93ef24 (4 passed, 0 failed, 0 skipped); all four selected GB300 stages and Slurm jobs completed successfully.
• The selected Post-Merge-1 shards exercised the four corresponding disagg_upload-e2e cases. Because these stage definitions use two pytest splits, the affected disagg_upload-gen_only cases are assigned to the Post-Merge-2 shards and were not executed by this targeted run.
• Full CI was requested; validation of the four gen_only cases remains pending the shard-2 results.

Dev Engineer Review

• Added executor-capacity clamping for TLLM_BENCHMARK_REQ_QUEUES_SIZE in Jenkins and local SLURM GEN-only submitters.
• Capacity is correctly derived as max_batch_size * tensor_parallel_size for attention-DP workers and max_batch_size otherwise.
• Warnings are emitted when requested concurrency exceeds reachable capacity, preventing the reported iter=0 livelock without affecting production, ctx_only, or e2e paths.
• Added regression coverage for DeepSeek-V4-Pro FP4 gen-only configurations and reachable-concurrency behavior.
• No configuration or test-list changes were identified.

QA Engineer Review

• Added:
  • test_gen_only_queue_size_does_not_exceed_executor_capacity
  • test_gen_only_queue_size_preserves_reachable_concurrency
• Tests cover both submit.py implementations and validate configured disaggregated benchmark scenarios.
• These test functions are not registered in tests/integration/test_lists/ (test-db/ or qa/).
• Verdict: needs follow-up.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN4-NODE2-GPU8-Post-Merge-1,GB300-40_GPUs-10_Nodes-PyTorch-Disagg-PerfSanity-CTX6-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1,GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1,GB300-56_GPUs-14_Nodes-PyTorch-Disagg-PerfSanity-CTX12-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #62008 [ run ] triggered by Bot. Commit: b93ef24 Link to invocation

[coderabbitai]

Walkthrough

Both performance submission scripts now cap GEN-only request queue sizes to GEN executor capacity and use the result in generated environment variables. New pytest coverage validates both implementations against YAML and in-memory configurations.

Changes

GEN-only queue capacity

Layer / File(s)	Summary
Queue capacity calculation jenkins/scripts/perf/submit.py, jenkins/scripts/perf/local/submit.py	Adds a helper that derives GEN executor capacity from worker settings, caps requested concurrency, and warns when capping occurs.
GEN-only environment wiring jenkins/scripts/perf/submit.py, jenkins/scripts/perf/local/submit.py	Uses the computed queue size when constructing GEN-only benchmark and worker environment variables.
Queue capacity tests tests/unittest/scripts/test_perf_submit.py	Loads both submission implementations and tests capped queue sizes from YAML configurations and reachable in-memory concurrency.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: chzblych

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	Matches the PR's main change and follows the required [ticket][type] summary format.
Description check	✅ Passed	Covers the issue, fix, scope, timeline, and verification, though it doesn't use the exact template headings like Description and Test Coverage.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

tests/unittest/scripts/test_perf_submit.py (1)

59-83: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy lift

Cover generated environment wiring and warnings.

These tests validate the helper only; a regression in either submitter’s GEN_WORKER_ENV_VARS wiring or capped-request warning would still pass. Add output-level assertions for both generated launch scripts.

Test coverage summary: added test_gen_only_queue_size_does_not_exceed_executor_capacity and test_gen_only_queue_size_preserves_reachable_concurrency; tests/integration/test_lists/ entries are not applicable to this unit-test module; verdict: insufficient until generated environment output is covered. As per path instructions, test-code changes must include a coverage verdict and test-list applicability review.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unittest/scripts/test_perf_submit.py` around lines 59 - 83, Extend the
tests around both submitters’ generated launch scripts to assert the generated
output includes the expected GEN_WORKER_ENV_VARS wiring and emits the
capped-request warning when applicable. Keep the existing helper tests, add
output-level coverage for each submitter, and include the required coverage
verdict plus test-list applicability review for this unit-test change.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/unittest/scripts/test_perf_submit.py`:
- Around line 59-83: Extend the tests around both submitters’ generated launch
scripts to assert the generated output includes the expected GEN_WORKER_ENV_VARS
wiring and emits the capped-request warning when applicable. Keep the existing
helper tests, add output-level coverage for each submitter, and include the
required coverage verdict plus test-list applicability review for this unit-test
change.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3008e93f-f52c-4fb1-9113-c1ce33c457ce

📥 Commits

Reviewing files that changed from the base of the PR and between cfeca00 and b93ef24.

📒 Files selected for processing (3)

• jenkins/scripts/perf/local/submit.py
• jenkins/scripts/perf/submit.py
• tests/unittest/scripts/test_perf_submit.py

[tensorrt-cicd]

PR_Github #62008 [ run ] completed with state SUCCESS. Commit: b93ef24
/LLM/main/L0_MergeRequest_PR pipeline #50194 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #62048 [ run ] triggered by Bot. Commit: b93ef24 Link to invocation

[tensorrt-cicd]

PR_Github #62048 [ run ] completed with state SUCCESS. Commit: b93ef24
/LLM/main/L0_MergeRequest_PR pipeline #50233 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[tensorrt-cicd]

PR_Github #62156 [ skip ] triggered by Bot. Commit: b93ef24 Link to invocation

[chenfeiz0326]

/bot run --disable-fail-fast --stage-list "GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN4-NODE2-GPU8-Post-Merge-2,GB300-40_GPUs-10_Nodes-PyTorch-Disagg-PerfSanity-CTX6-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2,GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2,GB300-56_GPUs-14_Nodes-PyTorch-Disagg-PerfSanity-CTX12-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2"

[tensorrt-cicd]

PR_Github #62156 [ skip ] completed with state SUCCESS. Commit: b93ef24
Skipping testing for commit b93ef24

Link to invocation

[tensorrt-cicd]

PR_Github #62159 [ ] completed with state FAILURE. Commit: b93ef24
Not allowed on merged PR

Link to invocation