[TRTLLM-8113][test] Add pytorch workflow e2e tests with pp enabled

[StanleySun639]

Summary by CodeRabbit

• Tests
  • Added new end-to-end scenarios for advanced quickstart with pipeline parallel enabled, covering CUDA Graphs on/off and GPU-count-aware conditions.
  • Expanded QA core and sanity test lists to include Llama 3.3 70B Instruct FP8 variants across multiple tensor/pipeline parallel configurations and Eagle3 options.
  • Improved coverage for memory and configuration presets (e.g., chunked prefill, KV cache fraction), increasing confidence in large-model FP8 and PP-enabled workflows.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md
and the scripts/test_to_stage_mapping.py helper.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds a new parameterized end-to-end test in tests/integration/defs/test_e2e.py to exercise quickstart_advanced.py with PP enabled, varying CUDA graph usage and TP/PP sizes. Updates QA test list files to include new FP8 70B Llama3.3 scenarios and the new quickstart_advanced PP-enabled tests.

Changes

Cohort / File(s)	Summary
E2E test additions tests/integration/defs/test_e2e.py	Added test_ptp_quickstart_advanced_pp_enabled with pytest params: cuda_graph {False, True}; (tp,pp) {(2,2)[skip if <4 GPUs], (2,4)[skip if <8 GPUs]}; model {Llama3.3-70B-FP8 path with skip_pre_hopper}. Builds command for quickstart_advanced.py with PP, TP, FP8, KV cache, and optional CUDA graph flags; runs via llm_venv.run_cmd.
QA test lists tests/integration/test_lists/qa/llm_function_core.txt, tests/integration/test_lists/qa/llm_function_core_sanity.txt	Appended references for Llama3.3-70B Instruct FP8 cases (tp2pp2, tp4, nvfp4_tp4, fp8_eagle3_tp8 with eagle3_one_model variants). Added four PP-enabled quickstart_advanced entries for 70B FP8 (tp2pp2 with cuda_graph False/True; tp2pp4 with cuda_graph False/True).

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Pre-merge checks and finishing touches

❌ Failed checks (2 warnings, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.
Description Check	⚠️ Warning	The pull request description includes the template headers but none of the required sections are populated with actual content. The PR title is still the placeholder, the Description and Test Coverage sections are only HTML comments, and no summary or test details are provided. This makes it impossible to understand the intent, scope, or coverage of the changes. Therefore the description is largely incomplete and does not meet the repository requirements.	Please replace the placeholder text with a valid PR title following the specified template and fill in the Description and Test Coverage sections with a concise summary of the changes, their rationale, and the relevant tests that verify the new functionality. You may also confirm that the PR Checklist items have been addressed if they apply.
Title Check	❓ Inconclusive	The PR objectives describe adding a submission template and guidance for writing PR titles without any code changes, while the raw summary details the addition of new end-to-end tests with pipeline parallelism enabled. The provided title accurately reflects the test additions but does not align with the stated objectives, creating conflicting signals about the PR’s primary purpose. This discrepancy makes it unclear whether the title summarizes the intended change.	Please clarify the intended scope of this PR by reconciling the objectives with the actual code changes. If the PR is meant to add PyTorch e2e tests with pipeline parallelism, update the objectives section to reflect this focus. Alternatively, if the PR is purely for adding submission templates and guidance, revise the title to accurately describe that change.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (1)

tests/integration/defs/test_e2e.py (1)

2323-2355: LGTM! New PP-enabled end-to-end test for advanced quickstart.

The test implementation is clean and follows the existing pattern in the file. It appropriately parameterizes TP/PP sizes and CUDA graph usage to exercise PP-enabled configurations with chunked prefill.

Consider adding a brief docstring describing the test's purpose, e.g.:

def test_ptp_quickstart_advanced_pp_enabled(llm_root, llm_venv, model_name,
                                            model_path, cuda_graph, tp_size,
                                            pp_size):
    """Test quickstart_advanced.py with PP-enabled configurations.
    
    Exercises chunked prefill with varying TP/PP sizes and optional CUDA graph usage.
    """

Note: Unlike some other tests in this file (e.g., test_ptp_quickstart_advanced), this test doesn't call _check_mem_usage to validate memory consumption. If memory profiling is desired for PP-enabled scenarios, consider adding it in a follow-up.

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 66aa887 and 0dc592b.

📒 Files selected for processing (3)

• tests/integration/defs/test_e2e.py (1 hunks)
• tests/integration/test_lists/qa/llm_function_core.txt (1 hunks)
• tests/integration/test_lists/qa/llm_function_core_sanity.txt (2 hunks)

🧰 Additional context used

📓 Path-based instructions (3)

**/*.{h,hpp,hh,hxx,cpp,cxx,cc,cu,cuh,py}

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

Use only spaces, no tabs; indent with 4 spaces.

Files:

• tests/integration/defs/test_e2e.py

**/*.py

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

**/*.py: Python code must target Python 3.8+.
Indent Python code with 4 spaces; do not use tabs.
Maintain module namespace when importing; prefer 'from package.subpackage import foo' then 'foo.SomeClass()' instead of importing the class directly.
Python filenames should be snake_case (e.g., some_file.py).
Python classes use PascalCase names.
Functions and methods use snake_case names.
Local variables use snake_case; prefix 'k' for variables that start with a number (e.g., k_99th_percentile).
Global variables use upper SNAKE_CASE prefixed with 'G' (e.g., G_MY_GLOBAL).
Constants use upper SNAKE_CASE (e.g., MY_CONSTANT).
Avoid shadowing variables from an outer scope.
Initialize all externally visible members of a class in the constructor.
Prefer docstrings for interfaces that may be used outside a file; comments for in-function or file-local interfaces.
Use Google-style docstrings for classes and functions (Sphinx-parsable).
Document attributes and variables inline so they render under the class/function docstring.
Avoid reflection when a simpler, explicit approach suffices (e.g., avoid dict(**locals()) patterns).
In try/except, catch the most specific exceptions possible.
For duck-typing try/except, keep the try body minimal and use else for the main logic.

Files:

• tests/integration/defs/test_e2e.py

**/*.{cpp,cxx,cc,h,hpp,hh,hxx,cu,cuh,py}

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

Prepend the NVIDIA Apache-2.0 copyright header with current year to the top of all source files (e.g., .cpp, .h, .cu, .py).

Files:

• tests/integration/defs/test_e2e.py

🧠 Learnings (2)

📓 Common learnings

Learnt from: venkywonka
PR: NVIDIA/TensorRT-LLM#6029
File: .github/pull_request_template.md:45-53
Timestamp: 2025-08-27T17:50:13.264Z
Learning: For PR templates in TensorRT-LLM, avoid suggesting changes that would increase developer overhead, such as converting plain bullets to mandatory checkboxes. The team prefers guidance-style bullets that don't require explicit interaction to reduce friction in the PR creation process.

📚 Learning: 2025-07-28T17:06:08.621Z

Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

Applied to files:

• tests/integration/test_lists/qa/llm_function_core.txt
• tests/integration/test_lists/qa/llm_function_core_sanity.txt

🧬 Code graph analysis (1)

tests/integration/defs/test_e2e.py (1)

tests/integration/defs/conftest.py (3)

• llm_root (189-190)
• llm_venv (699-716)
• llm_models_root (77-91)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

• GitHub Check: Pre-commit Check

🔇 Additional comments (3)

tests/integration/test_lists/qa/llm_function_core.txt (1)

421-421: LGTM! Expanded test coverage for Llama3.3-70B Instruct.

The addition of the FP8 TP2PP2 test reference aligns with the PR's goal of extending PP-enabled test coverage for the Llama3.3-70B model.

tests/integration/test_lists/qa/llm_function_core_sanity.txt (2)

120-124: LGTM! Comprehensive FP8 test coverage for Llama3.3-70B Instruct.

The addition of FP8 test references (tp2pp2, tp4, nvfp4_tp4, eagle3 variants) provides thorough coverage for various TP/PP configurations and speculative decoding scenarios.

---

223-226: LGTM! PP-enabled quickstart test coverage.

The four new test entries for test_ptp_quickstart_advanced_pp_enabled cover combinations of TP/PP sizes (2-2, 2-4) and CUDA graph usage (False/True), providing good test matrix coverage for PP-enabled configurations.

[StanleySun639]

/bot run

[tensorrt-cicd]

PR_Github #21413 [ run ] triggered by Bot

[StanleySun639]

/bot run

[tensorrt-cicd]

PR_Github #21418 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #21413 [ run ] completed with state ABORTED
LLM/release-1.1/L0_MergeRequest_PR #140 (Blue Ocean) completed with status: ABORTED

[tensorrt-cicd]

PR_Github #21418 [ run ] completed with state SUCCESS
/LLM/release-1.1/L0_MergeRequest_PR pipeline #141 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.