Skip to content

[https://nvbugs/6487039][fix] Generalize ADP dummy lifecycle - #16921

Open
chienchunhung wants to merge 5 commits into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6487039-adp-dummy-leak
Open

[https://nvbugs/6487039][fix] Generalize ADP dummy lifecycle#16921
chienchunhung wants to merge 5 commits into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6487039-adp-dummy-leak

Conversation

@chienchunhung

@chienchunhung chienchunhung commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes NVBUG 6487039.

This PR makes the non-PP disaggregated attention-DP dummy-request lifecycle model-independent and derives sequence-slot capacity from scheduler behavior rather than model identity. It is the generalized follow-up to the transactional dummy handling introduced in #16279.

The affected GB300 Kimi E2E test remains waived. This PR fixes the dummy-request bookkeeping failure, but it does not address the separate Python-transceiver liveness and performance failures seen by that workload.

Failure mechanism

MicroBatchScheduler schedules requests only within its configured state window. The legacy non-DeepSeek-V4 active-request count instead treated terminal and wait states outside that window as schedulable. A rank holding only those states could therefore omit a required pad dummy while peer attention-DP ranks allocated tentative dummies.

If the TP-wide queue decision then rejected the iteration, the tentative peer dummies were not rolled back. They accumulated across retries until:

assert self.expected_num_active_requests >= len(self.active_requests)

Recovery subsequently hit threads can only be started once, followed by the HangDetector/MPI abort. Those errors are consequences of the leaked dummy requests, not independent root causes.

The generalization also exposed two related lifecycle assumptions:

  • Non-PP overlap scheduling can keep one batch active while preparing the next, so slot-indexed state needs up to 2 * max_batch_size entries. Non-overlap execution needs max_batch_size; PP retains pp_size * max_batch_size.
  • Encoder-decoder scheduling begins at ENCODER_INIT, not CONTEXT_INIT, and decoder-context requests waiting for encoder output are not yet schedulable.

Changes

  • Apply transactional attention-DP dummy handling to every non-PP model.
  • Expose each scheduler's scheduling-state range and use it for both scheduling and active-request counting, including encoder-decoder readiness.
  • Check full dummy KV capacity, tolerate allocation failure, and roll back only the tentative dummy when any attention-DP rank cannot queue.
  • Derive max_num_seq_slots from PP and overlap lifecycle requirements, then propagate the same capacity to the executor, model engine, sampler, guided decoder, speculative metadata, and Qwen2/Qwen3-VL MRoPE caches.
  • Preserve the existing PP dummy fallback.

The affected Kimi configuration has overlap scheduling disabled, so it retains max_batch_size slots while receiving the generalized dummy-lifecycle correction.

Verification

  • Targeted executor CI passed: On commit 31260522c, both DGX_B200-4_GPUs-PyTorch-3 and DGX_B200-8_GPUs-PyTorch-1, including their pytest stages, passed in downstream build #2623. The parent pipeline #51166 was red because its SBSA build failed; the targeted x86 multi-GPU run succeeded.
  • Rebased-head full CI: Full CI with fail-fast disabled was triggered for current commit 8a8286c3f in pipeline #51225. The live pipeline link is the source of truth for its final result.
  • Exact Kimi E2E evidence: The selector ran unwaived for 90 minutes in pipeline #50411. The active-request assertion, executor restart error, and HangDetector/MPI-abort signature did not recur; the run instead failed later with persistent KV-transfer timeouts. This supports the narrow dummy-lifecycle fix but not test unwaiving.

Unit coverage includes scheduler-state eligibility, encoder-decoder readiness, overlap/no-overlap/PP sequence-slot sizing and propagation, mixed-rank padding, peer-empty rollback, allocation failure, full-capacity checks, draft-token reserve, MRoPE capacity, and PP fallback.

Scope and related work

This PR intentionally does not:

  • unwaive the GB300 Kimi E2E test (the rebased PR has no waiver diff against main);
  • change KV-transfer timeouts, transceiver behavior, or transceiver configuration;
  • fix the broader Python-transceiver liveness/performance issue;
  • change performance-test shard selection or GEN log-sentinel handling;
  • generalize the PP dummy path beyond its existing fallback.

The broader transfer and test-harness work is tracked separately in #16918, #16920, #17134, and #17140.

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2"

@chienchunhung chienchunhung changed the title [NVBUG 6487039][fix] Generalize ADP dummy lifecycle [https://nvbugs/6487039][fix] Generalize ADP dummy lifecycle Jul 28, 2026
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62036 [ run ] triggered by Bot. Commit: 153a3df Link to invocation

@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

The change generalizes ADP dummy handling, makes scheduler state ranges explicit, and propagates runtime sequence-slot capacity into model loading, MRoPE caches, speculative metadata, guided decoding, and executor padding behavior.

Changes

Runtime capacity and ADP scheduling

Layer / File(s) Summary
Sequence-slot capacity propagation
tensorrt_llm/_torch/pyexecutor/_util.py, tensorrt_llm/_torch/pyexecutor/model_engine.py, tensorrt_llm/_torch/pyexecutor/model_loader.py, tensorrt_llm/_torch/models/*, tensorrt_llm/_torch/pyexecutor/py_executor_creator.py, tensorrt_llm/_torch/speculative/*, tests/unittest/_torch/executor/test_seq_slot_sizing.py, tests/unittest/_torch/executor/test_model_loader_gms.py, tests/unittest/_torch/executor/test_pytorch_model_engine.py, tests/unittest/_torch/modeling/test_modeling_qwen2_5vl.py, tests/unittest/_torch/speculative/test_rejection_buffers_guard.py
Runtime sequence-slot sizing replaces DeepSeek-V4-specific overlap-headroom wiring and drives model configuration, MRoPE cache allocation, speculative metadata, guided decoding, and dummy slot indexing.
Scheduler state-range contract
tensorrt_llm/_torch/pyexecutor/_util.py, tensorrt_llm/_torch/pyexecutor/scheduler/*, tests/unittest/_torch/executor/test_dual_pool_kv_cache.py, tests/unittest/_torch/executor/test_py_executor.py
Schedulers expose admissible request-state ranges, reject decoder requests waiting for encoder output, and forward configured bounds to the C++ micro-batch scheduler.
Generic ADP dummy handling
tensorrt_llm/_torch/pyexecutor/model_engine.py, tensorrt_llm/_torch/pyexecutor/py_executor.py, tests/unittest/_torch/executor/test_benchmark_disagg.py, tests/unittest/_torch/executor/test_py_executor.py
ADP dummy-fix gating uses a generic mapping-based flag for request counting, dummy allocation finalization, and attention-DP padding, with expanded state-window and mixed-rank coverage.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ModelEngine
  participant ModelLoader
  participant ModelConfig
  participant PyExecutor
  participant RequestScheduler
  ModelEngine->>ModelLoader: pass max_num_seq_slots
  ModelLoader->>ModelConfig: attach runtime slot capacity
  ModelEngine->>PyExecutor: provide generic ADP dummy-fix flag
  PyExecutor->>RequestScheduler: check request schedulability
  RequestScheduler-->>PyExecutor: return state-range eligibility
  PyExecutor->>PyExecutor: allocate or finalize ADP padding dummy
Loading

Possibly related PRs

Suggested labels: api-compatible

Suggested reviewers: bowenfu

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 26.42% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the bug fix and accurately summarizes the generalized ADP dummy lifecycle change.
Description check ✅ Passed The description clearly explains the failure, solution, scope, and verification, including detailed test coverage.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62036 [ run ] completed with state SUCCESS. Commit: 153a3df
/LLM/main/L0_MergeRequest_PR pipeline #50222 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

@BowenFu

BowenFu commented Jul 28, 2026

Copy link
Copy Markdown

Flagging one thing before this goes further: the gate widening isn't inert for non-DeepSeek-V4 models in the disagg + attention-DP case, which is the case it targets.

Comparing against main, dropping the model_type == "deepseek_v4" term changes behavior in three places for any non-PP model that has a kv_cache_transceiver:

  1. _count_schedulable_active_requests — the old non-DSv4 branch counted every active request except generation-init / transmission-in-progress. The new one counts only [CONTEXT_INIT, GENERATION_TO_COMPLETE), which excludes DISAGG_CONTEXT_WAIT_SCHEDULER and terminal GENERATION_TO_COMPLETE. A rank holding only those requests now reports zero schedulable and requests dummy padding where it previously didn't. Since this feeds needs_dummy per rank, ranks can now disagree about whether to pad.

  2. _pad_attention_dp_dummy_request — the old non-DSv4 branch allocated the dummy unconditionally. The new path can return without padding (live-dummy check, _has_adp_dummy_kv_capacity precheck, OutOfPagesError), leaving that rank with an empty scheduled batch.

  3. _finalize_adp_dummy_allocation — on can_queue=False it now removes the dummy and frees KV/spec resources for all non-PP models, where non-DSv4 previously returned immediately.

That combination is what #16279 deliberately scoped to DSv4, and the PR description acknowledges as much.

The test change makes this harder to catch rather than easier: test_dsv4_adp_dummy_fix_gate loses its ("deepseek_v3", 1, False) case, so the one assertion pinning non-DSv4 to the established path is deleted rather than replaced. If the widening is intended, could you add a non-DSv4 disagg + ADP case showing the two paths agree — plus something covering the multi-rank pad/no-pad agreement in (1)?

Minor, unrelated: this removes the same gb300_kimi-k25-thinking-fp4...NIXL e2e waiver line as #16920, so whichever merges second will need a rebase.

@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@Shixiaowei02

Copy link
Copy Markdown
Collaborator

The dummy path is generalized but the paired 2x seq-slot headroom stays DSv4-only, worth generalizing both, or gating on the overlap scheduler. Could we also land the code fix alone and leave the waiver removal to a follow-up?

@chienchunhung

Copy link
Copy Markdown
Collaborator Author

Flagging one thing before this goes further: the gate widening isn't inert for non-DeepSeek-V4 models in the disagg + attention-DP case, which is the case it targets.

Comparing against main, dropping the model_type == "deepseek_v4" term changes behavior in three places for any non-PP model that has a kv_cache_transceiver:

  1. _count_schedulable_active_requests — the old non-DSv4 branch counted every active request except generation-init / transmission-in-progress. The new one counts only [CONTEXT_INIT, GENERATION_TO_COMPLETE), which excludes DISAGG_CONTEXT_WAIT_SCHEDULER and terminal GENERATION_TO_COMPLETE. A rank holding only those requests now reports zero schedulable and requests dummy padding where it previously didn't. Since this feeds needs_dummy per rank, ranks can now disagree about whether to pad.
  2. _pad_attention_dp_dummy_request — the old non-DSv4 branch allocated the dummy unconditionally. The new path can return without padding (live-dummy check, _has_adp_dummy_kv_capacity precheck, OutOfPagesError), leaving that rank with an empty scheduled batch.
  3. _finalize_adp_dummy_allocation — on can_queue=False it now removes the dummy and frees KV/spec resources for all non-PP models, where non-DSv4 previously returned immediately.

That combination is what #16279 deliberately scoped to DSv4, and the PR description acknowledges as much.

The test change makes this harder to catch rather than easier: test_dsv4_adp_dummy_fix_gate loses its ("deepseek_v3", 1, False) case, so the one assertion pinning non-DSv4 to the established path is deleted rather than replaced. If the widening is intended, could you add a non-DSv4 disagg + ADP case showing the two paths agree — plus something covering the multi-rank pad/no-pad agreement in (1)?

Minor, unrelated: this removes the same gb300_kimi-k25-thinking-fp4...NIXL e2e waiver line as #16920, so whichever merges second will need a rebase.

Thanks.

I added test_non_dsv4_disagg_adp_mixed_rank_states_stay_queueable to cover the generic non-PP path with one busy rank and one terminal-only rank. It verifies that only the terminal rank pads, both ranks expose batch size 1 to _can_queue, the TP-wide decision stays queueable, and the dummy is committed.

Existing allocation-failure and rollback tests cover the other two behavior changes.

@tburt-nv tburt-nv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The whole unittest/_torch/executor directory is covered in the test-db.

@chienchunhung

Copy link
Copy Markdown
Collaborator Author

The dummy path is generalized but the paired 2x seq-slot headroom stays DSv4-only, worth generalizing both, or gating on the overlap scheduler. Could we also land the code fix alone and leave the waiver removal to a follow-up?

IIUC these are separate lifecycle conditions rather than paired requirements.

The dummy fix is needed when rank-local padding or allocation disagrees, including this Kimi configuration where overlap is disabled. The 2× slot pool addresses overlap backfill before previous-iteration slot release; this case uses MTP0 with disable_overlap_scheduler: true, so additional headroom would not participate.

Gating dummy handling on overlap would therefore disable the reported fix, while generalizing slot headroom would require broader validation of all py_seq_slot-indexed consumers. I'd keep the independent gates and moved the waiver removal out of this PR because #16920 already owns the same unwaive with its timeout change.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
tests/unittest/_torch/modeling/test_modeling_qwen2_5vl.py (1)

430-434: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Expand the test coverage
test_mrope_delta_cache_size_uses_runtime_seq_slot_capacity covers the runtime override (8 → 9), but not the fallback branch or a non-default pp_size.

Test coverage summary

  • Added: test_mrope_delta_cache_size_uses_runtime_seq_slot_capacity
  • Modified/removed: none
  • Test-list membership: tests/integration/test_lists/test-db/l0_l40s.yml lists unittest/_torch/modeling/test_modeling_qwen2_5vl.py::TestQwen2_5_VL::test_all; no explicit entry for test_mrope_delta_cache_size_uses_runtime_seq_slot_capacity
  • Coverage verdict: needs follow-up
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unittest/_torch/modeling/test_modeling_qwen2_5vl.py` around lines 430 -
434, Add coverage for _get_mrope_position_delta_cache_size covering both the
default fallback when max_num_seq_slots is absent and a non-default pp_size.
Keep the existing runtime override test unchanged and assert the expected cache
sizes for each additional configuration.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/unittest/_torch/modeling/test_modeling_qwen2_5vl.py`:
- Around line 430-434: Add coverage for _get_mrope_position_delta_cache_size
covering both the default fallback when max_num_seq_slots is absent and a
non-default pp_size. Keep the existing runtime override test unchanged and
assert the expected cache sizes for each additional configuration.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d142e035-42c2-4f06-912b-ee1dcac52c72

📥 Commits

Reviewing files that changed from the base of the PR and between 7db0814 and 3126052.

📒 Files selected for processing (18)
  • tensorrt_llm/_torch/models/modeling_qwen2vl.py
  • tensorrt_llm/_torch/models/modeling_qwen3vl.py
  • tensorrt_llm/_torch/pyexecutor/_util.py
  • tensorrt_llm/_torch/pyexecutor/model_engine.py
  • tensorrt_llm/_torch/pyexecutor/model_loader.py
  • tensorrt_llm/_torch/pyexecutor/py_executor.py
  • tensorrt_llm/_torch/pyexecutor/py_executor_creator.py
  • tensorrt_llm/_torch/pyexecutor/scheduler/scheduler.py
  • tensorrt_llm/_torch/pyexecutor/scheduler/scheduler_v2.py
  • tensorrt_llm/_torch/speculative/interface.py
  • tensorrt_llm/_torch/speculative/utils.py
  • tests/unittest/_torch/executor/test_benchmark_disagg.py
  • tests/unittest/_torch/executor/test_dual_pool_kv_cache.py
  • tests/unittest/_torch/executor/test_model_loader_gms.py
  • tests/unittest/_torch/executor/test_py_executor.py
  • tests/unittest/_torch/executor/test_pytorch_model_engine.py
  • tests/unittest/_torch/executor/test_seq_slot_sizing.py
  • tests/unittest/_torch/modeling/test_modeling_qwen2_5vl.py
🚧 Files skipped from review as they are similar to previous changes (16)
  • tensorrt_llm/_torch/speculative/utils.py
  • tensorrt_llm/_torch/pyexecutor/scheduler/scheduler_v2.py
  • tensorrt_llm/_torch/pyexecutor/py_executor_creator.py
  • tensorrt_llm/_torch/speculative/interface.py
  • tests/unittest/_torch/executor/test_model_loader_gms.py
  • tensorrt_llm/_torch/models/modeling_qwen3vl.py
  • tensorrt_llm/_torch/pyexecutor/model_loader.py
  • tensorrt_llm/_torch/pyexecutor/_util.py
  • tests/unittest/_torch/executor/test_benchmark_disagg.py
  • tests/unittest/_torch/executor/test_seq_slot_sizing.py
  • tests/unittest/_torch/executor/test_pytorch_model_engine.py
  • tests/unittest/_torch/executor/test_dual_pool_kv_cache.py
  • tensorrt_llm/_torch/pyexecutor/py_executor.py
  • tensorrt_llm/_torch/pyexecutor/scheduler/scheduler.py
  • tensorrt_llm/_torch/pyexecutor/model_engine.py
  • tests/unittest/_torch/executor/test_py_executor.py

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62608 [ run ] triggered by Bot. Commit: 3126052 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62608 [ run ] completed with state FAILURE. Commit: 3126052
/LLM/main/L0_MergeRequest_PR pipeline #50750 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@BowenFu

BowenFu commented Jul 30, 2026

Copy link
Copy Markdown

On the --disable-fail-fast run for 31260522c (pipeline 50750): 54 of the 55 failures are in a single stage, DGX_H100-PyTorch-4, while 57,810 tests passed elsewhere. The three aggregate selectors that failed are:

  • test_unittests_v2[unittest/_torch/speculative/hw_agnostic]
  • test_unittests_v2[unittest/_torch/executor]
  • test_unittests_v2[unittest/_torch/modeling -k "modeling_llama"]

and underneath them it is essentially the entire content of those directories — test_dflash_*, test_pard*, test_llama_ngram, test_llama_draft_target*, test_llama_sa[*], test_mtp_eagle_one_model_rejection, test_vanilla_mtp_rejection, test_dynamic_draft_len, test_batched_attention_cuda_graph_capture_replay, test_save_state*, test_spec_gate_e2e, plus test_overlap_scheduler_consistency[*] and every test_llama_sanity/test_llama_allclose_to_hf variant. The one non-H100 failure is test_chunked_prefill[...] on DGX_B200-PyTorch-5.

A whole-stage wipeout that clean is not the shape of unrelated flake, and the blast radius is exactly the surface this PR moved: compute_max_num_sequences now returns 1 if disable_overlap_scheduler else 2, so with overlap on by default every config gets a 2× seq-slot pool, and everything keyed off it follows — SeqSlotManager(max_num_sequences) (_util.py:2574), the sampler args (py_executor_creator.py:823), spec metadata (num_seq_slots is now unconditional, previously Nonemax_num_requests fallback), and the guided-decoder bitmask/token_mask which switched from max_batch_size to max_num_seq_slots (py_executor_creator.py:778). Speculative decoding is the heaviest consumer of slot-indexed state, and it is the directory that went red wholesale.

Two things worth separating before the next run: whether this is a slot-indexing correctness break (something still sized or indexed by max_batch_size while its peer moved to max_num_seq_slots), or simply memory — doubling every slot-indexed buffer on a stage that builds many engines back to back. The stage-level concentration is consistent with either.

Separately, and this is the part I'd still want settled independent of the CI result: the previous form was opt-in (2 if enable_overlap_headroom and not disable_overlap_scheduler else 1) and the new one is opt-out. Generalizing the headroom beyond DSv4 is the right call and it does answer the review, but flipping it to on-by-default for every non-PP deployment is a separate decision from fixing 6487039, and there is no memory measurement in the PR for it. Admission capacity is unaffected — get_max_num_sequences() (model_engine.py:2813) still returns mapping.pp_size * batch_size — so this is purely a resident-footprint question, but it applies to every user.

@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Codex] @thorjohnsen Friendly review reminder: this PR is awaiting your review. Thanks!

@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "DGX_B200-PyTorch-5"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62853 [ run ] triggered by Bot. Commit: 3126052 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62853 [ run ] completed with state SUCCESS. Commit: 3126052
/LLM/main/L0_MergeRequest_PR pipeline #50976 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62890 [ run ] triggered by Bot. Commit: 3126052 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62890 [ run ] completed with state SUCCESS. Commit: 3126052
/LLM/main/L0_MergeRequest_PR pipeline #51012 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "DGX_B200-4_GPUs-PyTorch-3, DGX_B200-8_GPUs-PyTorch-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #63068 [ run ] triggered by Bot. Commit: 3126052 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #63068 [ run ] completed with state SUCCESS. Commit: 3126052
/LLM/main/L0_MergeRequest_PR pipeline #51166 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #63096 [ run ] triggered by Bot. Commit: 3126052 Link to invocation

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
@chienchunhung
chienchunhung force-pushed the codex/nvbug-6487039-adp-dummy-leak branch from 3126052 to 8a8286c Compare July 31, 2026 22:07
@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #63137 [ run ] triggered by Bot. Commit: 8a8286c Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #63096 [ run ] completed with state ABORTED. Commit: 3126052

Link to invocation

@2ez4bz 2ez4bz left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving the modeling changes.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #63137 [ run ] completed with state FAILURE. Commit: 8a8286c
/LLM/main/L0_MergeRequest_PR pipeline #51225 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "A10-Build_Docs, B300-PyTorch-2"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #63244 [ run ] triggered by Bot. Commit: 8a8286c Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #63244 [ run ] completed with state DISABLED
Pipeline is freezed and top-1 instance is under maintenance. For urgent request, contact Yiteng Niu

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants