[None][test] Update stress tests

[xinhe-nv]

Summary by CodeRabbit

• Tests
  • Improved disaggregated inference test suite with per-worker log capture, fatal-error scanning, and better triage output.
  • Added many new disaggregated integration configurations to cover additional models and runtime modes (CUDA graph, FP4/FP8, MTP, speculative decoding).
  • Stress tests now support speculative-model runs and optional request cancellation; test lists and waivers updated to reflect new coverage.
  • test report https://prod.blsm.nvidia.com/swqa-tensorrt-qa-test/view/TRT-LLM-Function-Pipelines/job/DEBUG_LLM_FUNCTION_CLUSTER_TEST/1535/testReport/

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR enhances disaggregated integration testing by adding log capture and post-run fatal-pattern scanning, expanding/adding disaggregated YAML configurations, introducing speculative-model support and cancellation/stddev options for stress runs, and updating test lists/waivers and CI QA guidance.

Changes

Disaggregated Stress Testing Infrastructure

Layer / File(s)	Summary
Worker and cluster log saving tests/integration/defs/disaggregated/disagg_test_utils.py, tests/integration/defs/disaggregated/test_disaggregated.py	run_ctx_worker and run_gen_worker signatures now accept save_log and forward it; setup_disagg_cluster propagates save_log to server and worker launchers; run_disaggregated_aiperf signatures extended for input/output token stddev and optional request cancellation; aiperf invocation includes those args and enables save_log=True.
New and updated disaggregated YAML configs tests/integration/defs/disaggregated/test_configs/*	Adds multiple new disaggregated configs (Qwen3.5 4B FP8, GPT-OSS 120B CUTLASS/Eagle variants, DeepSeek-R1 FP4/MTP variants) and flips enable_attention_dp in several existing configs.
Fatal log pattern detection and scanning tests/integration/defs/disaggregated/test_disaggregated.py	Adds _FATAL_LOG_PATTERNS and scan_logs_for_fatal_errors(); scans saved worker/server logs after aiperf and accuracy runs and raises on matches; improves exception triage by printing last 30 non-empty lines from each process log; run_accuracy_test() handles a specific lm_eval UnboundLocalError case.
Speculative decoding in stress tests tests/integration/defs/disaggregated/test_disaggregated.py	TestConfig adds speculative_model_path, cancellation_rate, and cancellation_delay; test harness symlinks and patches speculative_config.speculative_model into context/generation YAMLs at runtime and runs the stress test with configured stddev/cancellation parameters.
Test suite and waiver updates tests/integration/defs/test_e2e.py, tests/integration/test_lists/qa/llm_function_multinode.txt, tests/integration/test_lists/qa/llm_function_stress.txt, tests/integration/test_lists/waives.txt, .coderabbit.yaml	Removes a multinode e2e case, replaces old stress_test parametrizations with test_disaggregated_stress_test entries, updates waivers to reflect new targets, and simplifies CI QA guidance for tests/**.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested reviewers

• chuangz0
• jieli-matrix

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The PR description contains only the template structure with mostly empty sections; the Description and Test Coverage sections are not filled out with actual details about what was changed or why.	Fill in the Description section explaining the changes (e.g., support for speculative models, log scanning) and Test Coverage section listing relevant tests that validate these changes.
Title check	❓ Inconclusive	The title '[None][test] Update stress tests' is vague and does not clearly convey the specific changes made in the PR, using generic phrasing like 'Update' that lacks meaningful detail.	Provide a more specific title that describes the main change, such as '[None][test] Add support for speculative models and fatal log scanning in disaggregated stress tests' or similar.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 84.62% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 4

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)

tests/integration/test_lists/qa/llm_function_stress.txt (1)

1-11: ⚠️ Potential issue | 🟠 Major

Update waives.txt for the stress-test suite migration

• tests/integration/test_lists/waives.txt still has SKIP entries for stress_test/stress_test.py::test_run_stress_test[...] (lines 344-345). If llm_function_stress.txt no longer selects those nodeids, remove or update these waivers.
• No waiver entries exist for disaggregated/test_disaggregated.py::test_disaggregated_stress_test[...], even though the added params (deepseek_r1_v2_fp4_mtp_stress, gpt_oss_120b_eagle_stress, gpt_oss_120b_cutlass_stress, qwen3_5_4b_fp8_stress) are present in tests/integration/defs/disaggregated/test_disaggregated.py. Add corresponding waivers if these disaggregated cases need them.
• The new llm_function_stress.txt nodeids match existing parametrizations in tests/integration/defs/disaggregated/test_disaggregated.py.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/test_lists/qa/llm_function_stress.txt` around lines 1 - 11,
The waivers file still contains SKIP entries for the old stress_test nodeids
(stress_test/stress_test.py::test_run_stress_test[...]) that are no longer
selected by the new list; remove or update those SKIP lines. Add SKIP entries
for the new disaggregated parametrizations — e.g., nodeids matching
disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-deepseek_r1_v2_fp4_mtp_stress],
...-gpt_oss_120b_eagle_stress, ...-gpt_oss_120b_cutlass_stress, and
...-qwen3_5_4b_fp8_stress — if those cases should be waived. Ensure the nodeids
exactly match the parametrized names used by test_disaggregated_stress_test so
the waiver system will match at runtime.

tests/integration/defs/disaggregated/disagg_test_utils.py (1)

1-1: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Update the NVIDIA copyright year on this modified file.

This file now has 2026 changes, but the header still says 2022-2024.

As per coding guidelines, **/*.{cpp,cc,h,hpp,py,cu,cuh}: Include NVIDIA copyright header on ALL new files; update year on modified files.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/defs/disaggregated/disagg_test_utils.py` at line 1, Update
the file header copyright range to include 2026 by changing the year span from
"2022-2024" to "2022-2026" in the top-of-file comment (the
SPDX/FileCopyrightText header) so the modified file reflects the new copyright
year.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/defs/disaggregated/test_disaggregated.py`:
- Around line 1995-1998: The failure message for the accuracy test references
non-existent log filenames; update the AssertionError raised when
accuracy_test_result is falsy (the block using accuracy_test_result) to point
users to the actual artifacts: "worker_{role}_{port}.log" and
"disagg_server.log" (or similar dynamic names used by the harness) instead of
"output_workers.log / output_disagg.log", so that the exception text directs
triage to the correct saved logs.
- Around line 2265-2271: The pytest.param block creating TestConfig is using a
hanging indent that triggers Flake8 E126; reformat the call so the continued
lines are properly aligned with the opening parenthesis or use a consistent
indentation style. Locate the pytest.param(...) that wraps TestConfig
(referencing TestConfig, model_path='gpt_oss/gpt-oss-120b',
speculative_model_path='gpt_oss/gpt-oss-120b-Eagle3') and adjust the indentation
of the inner lines and the closing parentheses so they align (e.g., align
arguments under the first argument after the opening parenthesis or place the
opening parenthesis at line end and indent subsequent lines consistently) to
satisfy E126.
- Around line 2302-2317: When test_config.speculative_model_path is provided we
must fail fast if the patch is a no-op: after building spec_model_dir and
loading patched_config, compare the original config's 'speculative_model'
entries under 'context_servers' and 'generation_servers' to the values in
patched_config (or track whether the loop that sets spec['speculative_model']
actually changed any entries); if no entry was updated, raise an exception or
assert to fail the test (use the existing variables config_file, patched_config,
spec_model_dir, and test_desc to locate context and produce a clear error).
Ensure this check runs before writing patched_path and replacing config_file so
the test fails immediately when the speculative-model patch had no effect.
- Around line 2018-2032: The current exception handler loads entire log files
with f.read().split("\n"), which can OOM; update the block that iterates over
[*ctx_workers, *gen_workers, disagg_server] and uses log_path to instead stream
the file tail without reading whole file (e.g., use a fixed-size ring buffer
like collections.deque(maxlen=30) or seek-from-end logic) before logging the
last 30 non-empty lines via logger.error; keep the same error handling around
open() and OSError.

---

Outside diff comments:
In `@tests/integration/defs/disaggregated/disagg_test_utils.py`:
- Line 1: Update the file header copyright range to include 2026 by changing the
year span from "2022-2024" to "2022-2026" in the top-of-file comment (the
SPDX/FileCopyrightText header) so the modified file reflects the new copyright
year.

In `@tests/integration/test_lists/qa/llm_function_stress.txt`:
- Around line 1-11: The waivers file still contains SKIP entries for the old
stress_test nodeids (stress_test/stress_test.py::test_run_stress_test[...]) that
are no longer selected by the new list; remove or update those SKIP lines. Add
SKIP entries for the new disaggregated parametrizations — e.g., nodeids matching
disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-deepseek_r1_v2_fp4_mtp_stress],
...-gpt_oss_120b_eagle_stress, ...-gpt_oss_120b_cutlass_stress, and
...-qwen3_5_4b_fp8_stress — if those cases should be waived. Ensure the nodeids
exactly match the parametrized names used by test_disaggregated_stress_test so
the waiver system will match at runtime.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6fdc75d6-4dec-4fea-89ec-289e8f5bc504

📥 Commits

Reviewing files that changed from the base of the PR and between 998f418 and 5404a4f.

📒 Files selected for processing (12)

• tests/integration/defs/disaggregated/disagg_test_utils.py
• tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp1_gentp1_qwen3_5_4b_fp8_tllm.yaml
• tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp2_gentp2_gptoss_tllm.yaml
• tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp2_gentp2_gptoss_tllm_cutlass.yaml
• tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp2_gentp2_gptoss_tllm_eagle.yaml
• tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp4_gentp4_deepseek_r1_v2_fp4_tllm.yaml
• tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp4_gentp4_deepseek_r1_v2_fp4_tllm_mtp.yaml
• tests/integration/defs/disaggregated/test_disaggregated.py
• tests/integration/defs/test_e2e.py
• tests/integration/test_lists/qa/llm_function_multinode.txt
• tests/integration/test_lists/qa/llm_function_stress.txt
• tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (3)

• tests/integration/test_lists/qa/llm_function_multinode.txt
• tests/integration/defs/test_e2e.py
• tests/integration/test_lists/waives.txt

[xinhe-nv]

@coderabbitai review my code

[coderabbitai]

@xinhe-nv Sure! I'll start reviewing your PR right away.

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (2)

tests/integration/defs/disaggregated/test_disaggregated.py (1)

97-121: 💤 Low value

Consider adding return type annotation.

The function is well-implemented but lacks a return type hint. Adding it would improve readability and tooling support.

Suggested annotation

-def scan_logs_for_fatal_errors(processes):
+def scan_logs_for_fatal_errors(processes) -> dict[str, dict[str, int]]:

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/defs/disaggregated/test_disaggregated.py` around lines 97 -
121, Add an explicit return type annotation to the function
scan_logs_for_fatal_errors: change its signature to include -> dict[str,
dict[str, int]] so callers and linters know it returns a mapping of log path to
pattern counts (matching the local findings variable). Ensure your project
Python version supports PEP 585 annotations (or switch to typing.Dict if not)
and update any related import/style if required.

.coderabbit.yaml (1)

40-42: 💤 Low value

Verify simplified test review guidance is intentional.

The test path instructions were simplified from a longer checklist (covering test list hygiene, coverage expectations, and performance-test coverage rules) to a concise two-line directive. While this makes the guidance more accessible, it removes specific rules that may have helped catch test-related issues.

Consider whether:

• The removed checklist items are now covered elsewhere (e.g., in CODING_GUIDELINES.md or CI documentation)
• Review quality might be affected by the loss of specific guidance on test list hygiene and performance coverage rules
• This change aligns with the team's desired review depth for test changes

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.coderabbit.yaml around lines 40 - 42, The simplified QA/test-review
directive in .coderabbit.yaml removed the original checklist; restore explicit
guidance by reintroducing key checklist items or linking to their canonical
source: ensure the directive (the "Act as a QA engineer reviewing test changes
and coverage for TensorRT-LLM" block) includes concrete expectations for test
list hygiene, coverage thresholds, and performance-test coverage rules, or add a
clear pointer to CODING_GUIDELINES.md/CI docs and require concrete file lists
(e.g., tests/unit/*.py, tests/integration/*, perf/tests/*) with labels like
"sufficient/insufficient/follow-up" so reviewers know what to check and where to
record results.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp2_gentp2_gptoss_tllm_cutlass.yaml`:
- Around line 43-57: The cuda_graph_config.batch_sizes list contains entries
larger than the server's max_batch_size (128) so remove any batch sizes >128
(256, 512, 768, 1024) or programmatically generate the batch_sizes up to the
max_batch_size used by the generation server; update the batch_sizes in the
cuda_graph_config block so all entries are <= the max_batch_size (128) to avoid
allocating unused CUDA graphs and free GPU memory for KV cache.

In `@tests/integration/test_lists/waives.txt`:
- Around line 329-330: The waivers added for llama-v3-8b-instruct-hf_tp1 only
shorten stress coverage for stress_time_300s_timeout_450s (GUARANTEED_NO_EVICT
and MAX_UTILIZATION); either document where equivalent longer-duration or
higher-parallelism coverage exists (e.g., reference the
disaggregated/test_disaggregated.py::test_disaggregated_stress_test
parametrizations listed in qa/llm_function_stress.txt) or add/extend stress
tests so the llama-v3-8b_tp1 workload still runs an equivalent
long-duration/high-parallelism scenario; update the waives.txt entry with a
comment pointing to the alternative test target if coverage is provided, or add
the missing stress-test parametrization to
stress_test/stress_test.py::test_run_stress_test for llama-v3-8b-instruct-hf_tp1
if no equivalent exists.

---

Nitpick comments:
In @.coderabbit.yaml:
- Around line 40-42: The simplified QA/test-review directive in .coderabbit.yaml
removed the original checklist; restore explicit guidance by reintroducing key
checklist items or linking to their canonical source: ensure the directive (the
"Act as a QA engineer reviewing test changes and coverage for TensorRT-LLM"
block) includes concrete expectations for test list hygiene, coverage
thresholds, and performance-test coverage rules, or add a clear pointer to
CODING_GUIDELINES.md/CI docs and require concrete file lists (e.g.,
tests/unit/*.py, tests/integration/*, perf/tests/*) with labels like
"sufficient/insufficient/follow-up" so reviewers know what to check and where to
record results.

In `@tests/integration/defs/disaggregated/test_disaggregated.py`:
- Around line 97-121: Add an explicit return type annotation to the function
scan_logs_for_fatal_errors: change its signature to include -> dict[str,
dict[str, int]] so callers and linters know it returns a mapping of log path to
pattern counts (matching the local findings variable). Ensure your project
Python version supports PEP 585 annotations (or switch to typing.Dict if not)
and update any related import/style if required.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d1095b1d-e826-49d6-af1b-abf6b652f209

📥 Commits

Reviewing files that changed from the base of the PR and between 5404a4f and c437d35.

📒 Files selected for processing (13)

• .coderabbit.yaml
• tests/integration/defs/disaggregated/disagg_test_utils.py
• tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp1_gentp1_qwen3_5_4b_fp8_tllm.yaml
• tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp2_gentp2_gptoss_tllm.yaml
• tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp2_gentp2_gptoss_tllm_cutlass.yaml
• tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp2_gentp2_gptoss_tllm_eagle.yaml
• tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp4_gentp4_deepseek_r1_v2_fp4_tllm.yaml
• tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp4_gentp4_deepseek_r1_v2_fp4_tllm_mtp.yaml
• tests/integration/defs/disaggregated/test_disaggregated.py
• tests/integration/defs/test_e2e.py
• tests/integration/test_lists/qa/llm_function_multinode.txt
• tests/integration/test_lists/qa/llm_function_stress.txt
• tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (2)

• tests/integration/test_lists/qa/llm_function_multinode.txt
• tests/integration/defs/test_e2e.py

✅ Files skipped from review due to trivial changes (3)

• tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp4_gentp4_deepseek_r1_v2_fp4_tllm_mtp.yaml
• tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp4_gentp4_deepseek_r1_v2_fp4_tllm.yaml
• tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp2_gentp2_gptoss_tllm.yaml

🚧 Files skipped from review as they are similar to previous changes (4)

• tests/integration/test_lists/qa/llm_function_stress.txt
• tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp1_gentp1_qwen3_5_4b_fp8_tllm.yaml
• tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp2_gentp2_gptoss_tllm_eagle.yaml
• tests/integration/defs/disaggregated/disagg_test_utils.py

[xinhe-nv]

/bot run --skip-test

[xinhe-nv]

/bot run

[tensorrt-cicd]

PR_Github #50478 [ run ] triggered by Bot. Commit: 4831147 Link to invocation

[tensorrt-cicd]

PR_Github #50479 [ run ] triggered by Bot. Commit: 4831147 Link to invocation

[tensorrt-cicd]

PR_Github #50478 [ run ] completed with state ABORTED. Commit: 4831147

Link to invocation

[tensorrt-cicd]

PR_Github #50479 [ run ] completed with state SUCCESS. Commit: 4831147
/LLM/main/L0_MergeRequest_PR pipeline #39988 (Partly Tested) completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

[xinhe-nv]

/bot reuse-pipeline

[tensorrt-cicd]

PR_Github #50523 [ reuse-pipeline ] triggered by Bot. Commit: 58cef7c Link to invocation

[tensorrt-cicd]

PR_Github #50523 [ reuse-pipeline ] completed with state SUCCESS. Commit: 58cef7c
Reusing PR_Github #50479 (Partly Tested) for commit 58cef7c

Link to invocation