[https://nvbugs/5846166][fix] Update Perf Triage Scripts to Fix gen_only issue - #11802
Conversation
📝 WalkthroughWalkthroughThe pull request migrates the performance testing workflow from regression-focused reporting to sanity test-focused post-processing. It replaces perf_regression.py with perf_sanity_postprocess.py for HTML report generation with embedded SVG charts, updates Jenkins pipeline steps and submit scripts to reference the new workflow, introduces environment-variable-driven configuration for aggregated and disaggregated test modes, and adds nsys profiling support alongside extensive documentation updates. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~30 minutes 🚥 Pre-merge checks | ❌ 3❌ Failed checks (3 warnings)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
jenkins/scripts/perf/local/submit.py (1)
570-580:⚠️ Potential issue | 🟠 MajorDisaggregated path overwrites required worker env defaults.
Line 578 replaces
worker_env_varsentirely, soTLLM_PROFILE_START_STOPandFLASHINFER_JIT_DIRfrom Line 570–573 are lost for disaggregated runs.💡 Proposed fix
- worker_env_vars = ( + worker_env_vars = ( f"TLLM_PROFILE_START_STOP='{tllm_profile_start_stop}' " f"FLASHINFER_JIT_DIR=/tmp/flashinfer_jit_cache " ) @@ if runtime_mode == "disaggregated": # Build worker env vars - worker_env_vars = env_config.get("worker_env_var", "") + worker_env_vars = f"{worker_env_vars}{env_config.get('worker_env_var', '')}" server_env_vars = env_config.get("server_env_var", "") benchmark_env_var = env_config.get("benchmark_env_var", "")🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@jenkins/scripts/perf/local/submit.py` around lines 570 - 580, The disaggregated branch is overwriting the default worker_env_vars (which include TLLM_PROFILE_START_STOP and FLASHINFER_JIT_DIR); update the runtime_mode == "disaggregated" block to merge or append env_config values instead of replacing them — e.g., read env_config.get("worker_env_var", "") and concatenate it to the existing worker_env_vars string (and do similarly for server_env_vars and benchmark_env_var as needed), referencing the variables worker_env_vars, server_env_vars, benchmark_env_var, env_config, and runtime_mode so the default flags remain present for disaggregated runs.tests/integration/defs/perf/open_search_db_utils.py (1)
1-1:⚠️ Potential issue | 🟡 MinorUpdate the copyright year to include 2026.
This file is modified in this PR, but Line 1 still ends at
2024.As per coding guidelines
Include NVIDIA copyright header on ALL new files; update year on modified files.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/integration/defs/perf/open_search_db_utils.py` at line 1, Update the SPDX copyright header in tests/integration/defs/perf/open_search_db_utils.py to include 2026 (change the year range that currently ends with 2024 to 2026); locate the top-of-file header comment (the SPDX/ Copyright line) and edit the year span so the header reflects "2022-2026" or adds 2026 appropriately.tests/integration/defs/perf/test_perf_sanity.py (1)
1-1:⚠️ Potential issue | 🟡 MinorUpdate the copyright year to include 2026.
This file is modified in this PR, but Line 1 still ends at
2025.As per coding guidelines
Include NVIDIA copyright header on ALL new files; update year on modified files.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/integration/defs/perf/test_perf_sanity.py` at line 1, Update the SPDX copyright header string that currently ends with "2025" so it includes 2026 (change "2022-2025" to "2022-2026") for the file containing the SPDX-FileCopyrightText header; ensure the top-of-file header line with the symbol SPDX-FileCopyrightText is updated exactly and preserved otherwise.
🧹 Nitpick comments (2)
tests/integration/defs/perf/open_search_db_utils.py (1)
625-715: Remove the large commented-out legacy implementation.Keeping retired logic as commented code makes maintenance harder; rely on git history instead.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/integration/defs/perf/open_search_db_utils.py` around lines 625 - 715, Delete the large commented-out legacy implementations of _print_regression_data and check_perf_regression (the entire commented blocks shown) so the file no longer contains retired code; remove any now-unused imports (e.g., yaml, os) that were only there for those functions, and ensure there are no remaining references to _print_regression_data or check_perf_regression elsewhere in the module.tests/integration/defs/perf/test_perf_sanity.py (1)
35-45: Prefer module import over importing individual symbols.Line 35-45 continues direct symbol imports (including the new
generate_perf_yamlon Line 38). Please switch to importing the module and referencing members via namespace.As per coding guidelines
Python imports: use from package.subpackage import module (never from module import Class).🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/integration/defs/perf/test_perf_sanity.py` around lines 35 - 45, The test currently imports many symbols directly (SCENARIO_MATCH_FIELDS, add_id, generate_perf_yaml, get_common_values, get_history_data, get_job_info, post_new_perf_data, prepare_baseline_data, prepare_regressive_test_cases); change this to import the module instead (e.g., from . import open_search_db_utils as open_search_db_utils) and update all references in the file to use the module namespace (open_search_db_utils.generate_perf_yaml, open_search_db_utils.get_common_values, etc.) so individual symbol imports are removed and all usages refer to the module members.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@jenkins/scripts/perf/local/README.md`:
- Around line 17-25: The fenced diagram block showing the submit.py ->
slurm_launch.sh -> slurm_install.sh/slurm_run.sh flow lacks a language tag and
triggers markdownlint MD040; update the opening fence for that diagram (the
triple backticks before the diagram text) to include a language identifier such
as "text" (i.e., change ``` to ```text) so the block is explicitly marked and
the lint warning is resolved.
In `@jenkins/scripts/perf/perf_sanity_postprocess.py`:
- Around line 279-283: The merge step only collects entry.get("history_data")
into merged_history, which drops each entry's history_baseline; update the merge
logic in perf_sanity_postprocess.py (the block that builds merged_history from
entries) to also include entry.get("history_baseline") where present (e.g.,
extend merged_history with history_baseline entries after or before
history_data), ensuring you handle None and avoid adding duplicates if needed so
baseline points appear in the combined history for the chart.
- Line 2: Update the file header in perf_sanity_postprocess.py to reflect the
correct latest modification year (replace the current 2025 year in the SPDX
header with 2026); locate the top-of-file copyright/SPDX header line and change
the year value so the NVIDIA copyright matches the latest meaningful
modification year.
In `@jenkins/scripts/perf/README.md`:
- Line 7: Several fenced blocks in README.md lack language identifiers causing
markdownlint MD040 failures; update each triple-backtick block that contains the
directory listing starting with "jenkins/scripts/perf/", the "submit.py"
snippet, the "SLACK BOT SENDS MESSAGE" block, and the SQL lines beginning with
"UPDATE SET b_is_valid=..." to include appropriate language tags (e.g., ```text
for plain listings and ```sql for the SQL statements) so the fenced blocks are
properly annotated; ensure all other reported blocks (lines around the three
snippets and the other occurrences noted) receive the same treatment.
In `@tests/integration/defs/perf/test_perf_sanity.py`:
- Around line 69-70: Rename the misleading constant AGGR_CONFIG_FOLDER to
AGG_CONFIG_FOLDER so it matches the environment variable name used in
os.environ.get() and the project docs; update the declaration in
tests/integration/defs/perf/test_perf_sanity.py (replace AGGR_CONFIG_FOLDER with
AGG_CONFIG_FOLDER), and update all references/usages of that constant (and any
imports) in the related scripts jenkins/scripts/perf/local/submit.py and
jenkins/scripts/perf/disaggregated/submit.py to use AGG_CONFIG_FOLDER instead;
ensure DISAGG_CONFIG_FOLDER remains unchanged and run tests to verify no import
or reference errors.
---
Outside diff comments:
In `@jenkins/scripts/perf/local/submit.py`:
- Around line 570-580: The disaggregated branch is overwriting the default
worker_env_vars (which include TLLM_PROFILE_START_STOP and FLASHINFER_JIT_DIR);
update the runtime_mode == "disaggregated" block to merge or append env_config
values instead of replacing them — e.g., read env_config.get("worker_env_var",
"") and concatenate it to the existing worker_env_vars string (and do similarly
for server_env_vars and benchmark_env_var as needed), referencing the variables
worker_env_vars, server_env_vars, benchmark_env_var, env_config, and
runtime_mode so the default flags remain present for disaggregated runs.
In `@tests/integration/defs/perf/open_search_db_utils.py`:
- Line 1: Update the SPDX copyright header in
tests/integration/defs/perf/open_search_db_utils.py to include 2026 (change the
year range that currently ends with 2024 to 2026); locate the top-of-file header
comment (the SPDX/ Copyright line) and edit the year span so the header reflects
"2022-2026" or adds 2026 appropriately.
In `@tests/integration/defs/perf/test_perf_sanity.py`:
- Line 1: Update the SPDX copyright header string that currently ends with
"2025" so it includes 2026 (change "2022-2025" to "2022-2026") for the file
containing the SPDX-FileCopyrightText header; ensure the top-of-file header line
with the symbol SPDX-FileCopyrightText is updated exactly and preserved
otherwise.
---
Nitpick comments:
In `@tests/integration/defs/perf/open_search_db_utils.py`:
- Around line 625-715: Delete the large commented-out legacy implementations of
_print_regression_data and check_perf_regression (the entire commented blocks
shown) so the file no longer contains retired code; remove any now-unused
imports (e.g., yaml, os) that were only there for those functions, and ensure
there are no remaining references to _print_regression_data or
check_perf_regression elsewhere in the module.
In `@tests/integration/defs/perf/test_perf_sanity.py`:
- Around line 35-45: The test currently imports many symbols directly
(SCENARIO_MATCH_FIELDS, add_id, generate_perf_yaml, get_common_values,
get_history_data, get_job_info, post_new_perf_data, prepare_baseline_data,
prepare_regressive_test_cases); change this to import the module instead (e.g.,
from . import open_search_db_utils as open_search_db_utils) and update all
references in the file to use the module namespace
(open_search_db_utils.generate_perf_yaml,
open_search_db_utils.get_common_values, etc.) so individual symbol imports are
removed and all usages refer to the module members.
ℹ️ Review info
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (63)
jenkins/L0_MergeRequest.groovyjenkins/scripts/perf/README.mdjenkins/scripts/perf/disaggregated/submit.pyjenkins/scripts/perf/local/README.mdjenkins/scripts/perf/local/submit.pyjenkins/scripts/perf/perf_regression.pyjenkins/scripts/perf/perf_sanity_postprocess.pytests/integration/defs/perf/open_search_db_utils.pytests/integration/defs/perf/test_perf_sanity.pytests/scripts/perf-sanity/README.mdtests/scripts/perf-sanity/aggregated/config_database_b200_nvl.yamltests/scripts/perf-sanity/aggregated/config_database_h200_sxm.yamltests/scripts/perf-sanity/aggregated/deepseek_r1_fp4_v2_2_nodes_grace_blackwell.yamltests/scripts/perf-sanity/aggregated/deepseek_r1_fp4_v2_blackwell.yamltests/scripts/perf-sanity/aggregated/deepseek_r1_fp4_v2_grace_blackwell.yamltests/scripts/perf-sanity/aggregated/deepseek_r1_fp8_blackwell.yamltests/scripts/perf-sanity/aggregated/deepseek_v32_fp4_blackwell.yamltests/scripts/perf-sanity/aggregated/deepseek_v32_fp4_grace_blackwell.yamltests/scripts/perf-sanity/aggregated/gpt_oss_120b_fp4_blackwell.yamltests/scripts/perf-sanity/aggregated/gpt_oss_120b_fp4_grace_blackwell.yamltests/scripts/perf-sanity/aggregated/k2_thinking_fp4_2_nodes_grace_blackwell.yamltests/scripts/perf-sanity/aggregated/k2_thinking_fp4_blackwell.yamltests/scripts/perf-sanity/aggregated/k2_thinking_fp4_grace_blackwell.yamltests/scripts/perf-sanity/disaggregated/b200_deepseek-r1-fp4_1k1k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/b200_deepseek-r1-fp4_1k1k_con2048_ctx1_dep4_gen1_dep8_eplb0_mtp1_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/b200_deepseek-r1-fp4_1k1k_con256_ctx1_dep4_gen1_dep8_eplb0_mtp3_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/b200_deepseek-r1-fp4_8k1k_con1536_ctx1_dep4_gen1_dep8_eplb0_mtp1_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/b200_deepseek-r1-fp4_8k1k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/b200_deepseek-r1-fp4_8k1k_con256_ctx1_dep4_gen1_dep8_eplb0_mtp1_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_deepseek-r1-fp4_128k8k_con128_ctx1_pp8_gen1_dep16_eplb0_mtp2_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_deepseek-r1-fp4_128k8k_con1_ctx1_pp8_gen1_tep8_eplb0_mtp3_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_deepseek-r1-fp4_128k8k_con64_ctx1_pp8_gen1_dep32_eplb0_mtp3_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_deepseek-r1-fp4_1k1k_con1024_ctx1_dep4_gen1_dep32_eplb0_mtp3_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_deepseek-r1-fp4_1k1k_con1024_ctx1_dep4_gen1_dep8_eplb0_mtp0_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_deepseek-r1-fp4_1k1k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_deepseek-r1-fp4_1k1k_con3072_ctx1_dep4_gen1_dep4_eplb0_mtp1_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_deepseek-r1-fp4_8k1k_con1024_ctx1_dep4_gen1_dep32_eplb0_mtp3_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_deepseek-r1-fp4_8k1k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_deepseek-r1-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb0_mtp1_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_deepseek-v32-fp4_1k1k_con1024_ctx1_dep4_gen1_dep32_eplb256_mtp3_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_deepseek-v32-fp4_1k1k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_deepseek-v32-fp4_1k1k_con2048_ctx1_dep4_gen1_dep4_eplb0_mtp1_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_deepseek-v32-fp4_32k4k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_deepseek-v32-fp4_32k4k_con2048_ctx1_dep4_gen1_dep32_eplb288_mtp1_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_deepseek-v32-fp4_32k4k_con256_ctx1_dep4_gen1_dep32_eplb0_mtp3_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_deepseek-v32-fp4_8k1k_con1024_ctx1_dep4_gen1_dep32_eplb256_mtp3_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_deepseek-v32-fp4_8k1k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_deepseek-v32-fp4_8k1k_con4096_ctx1_dep4_gen1_dep32_eplb256_mtp0_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_1k1k_con2048_ctx1_tp1_gen1_dep2_eplb0_mtp0_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_1k1k_con512_ctx1_tp1_gen1_dep2_eplb0_mtp0_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_1k1k_con64_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con128_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con4_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con512_ctx1_tp1_gen1_dep2_eplb0_mtp0_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_kimi-k2-thinking-fp4_1k1k_con2048_ctx1_dep4_gen1_dep32_eplb384_mtp0_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_kimi-k2-thinking-fp4_1k1k_con4096_ctx1_dep4_gen1_dep8_eplb0_mtp0_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_kimi-k2-thinking-fp4_1k1k_con4_ctx1_dep4_gen1_tep4_eplb0_mtp0_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_kimi-k2-thinking-fp4_8k1k_con1024_ctx1_dep4_gen1_dep32_eplb416_mtp3_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_kimi-k2-thinking-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb384_mtp0_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_kimi-k2-thinking-fp4_8k1k_con4_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_qwen3-235b-fp4_8k1k_con1024_ctx1_tp1_gen1_dep8_eplb0_mtp0_ccb-UCX.yamltests/scripts/perf-sanity/disaggregated/gb200_qwen3-235b-fp4_8k1k_con64_ctx1_tp1_gen1_tep4_eplb0_mtp0_ccb-UCX.yamltests/test_common/error_utils.py
💤 Files with no reviewable changes (1)
- jenkins/scripts/perf/perf_regression.py
8c9cbcd to
332b81f
Compare
chenfeiz0326
changed the title
chenfeiz0326
changed the title
|
/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-3,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-3,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-4,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-3,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB200-16_GPUs-4_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE2-GPU8-Post-Merge-1" |
|
PR_Github #37440 [ run ] triggered by Bot. Commit: |
39ea237 to
789d754
Compare
|
PR_Github #37440 [ run ] completed with state
|
|
/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-3,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-3,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-4,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-3,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB200-16_GPUs-4_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE2-GPU8-Post-Merge-1" |
|
PR_Github #37495 [ run ] triggered by Bot. Commit: |
|
PR_Github #37495 [ run ] completed with state
|
|
/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-3,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-3,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB200-16_GPUs-4_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE2-GPU8-Post-Merge-1" |
|
PR_Github #37516 [ run ] triggered by Bot. Commit: |
|
/bot run --disable-fail-fast |
|
PR_Github #37804 [ run ] triggered by Bot. Commit: |
|
/bot run --disable-fail-fast |
|
PR_Github #37832 [ run ] triggered by Bot. Commit: |
|
PR_Github #37832 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #37878 [ run ] triggered by Bot. Commit: |
|
PR_Github #37878 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #37944 [ run ] triggered by Bot. Commit: |
4083b62 to
47b7c73
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #37954 [ run ] triggered by Bot. Commit: |
4e1466c to
9b1aeeb
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #37972 [ run ] triggered by Bot. Commit: |
|
PR_Github #37972 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #38046 [ ] completed with state |
|
/bot run --disable-fail-fast |
|
PR_Github #38049 [ run ] triggered by Bot. Commit: |
|
PR_Github #38049 [ run ] completed with state |
…nly issue (NVIDIA#11802) Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com> Signed-off-by: Chenfei Zhang <chenfeiz@oci-hsg-cs-001-login-01.cm.cluster> Co-authored-by: Chenfei Zhang <chenfeiz@oci-hsg-cs-001-login-01.cm.cluster>
…nly issue (NVIDIA#11802) Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com> Signed-off-by: Chenfei Zhang <chenfeiz@oci-hsg-cs-001-login-01.cm.cluster> Co-authored-by: Chenfei Zhang <chenfeiz@oci-hsg-cs-001-login-01.cm.cluster>
…nly issue (NVIDIA#11802) Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com> Signed-off-by: Chenfei Zhang <chenfeiz@oci-hsg-cs-001-login-01.cm.cluster> Co-authored-by: Chenfei Zhang <chenfeiz@oci-hsg-cs-001-login-01.cm.cluster>
6 participants
Summary by CodeRabbit
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.