[https://nvbugs/6487039][fix] Generalize ADP dummy lifecycle - #16921
[https://nvbugs/6487039][fix] Generalize ADP dummy lifecycle#16921chienchunhung wants to merge 5 commits into
Conversation
|
/bot run --disable-fail-fast --stage-list "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2" |
|
PR_Github #62036 [ run ] triggered by Bot. Commit: |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe change generalizes ADP dummy handling, makes scheduler state ranges explicit, and propagates runtime sequence-slot capacity into model loading, MRoPE caches, speculative metadata, guided decoding, and executor padding behavior. ChangesRuntime capacity and ADP scheduling
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant ModelEngine
participant ModelLoader
participant ModelConfig
participant PyExecutor
participant RequestScheduler
ModelEngine->>ModelLoader: pass max_num_seq_slots
ModelLoader->>ModelConfig: attach runtime slot capacity
ModelEngine->>PyExecutor: provide generic ADP dummy-fix flag
PyExecutor->>RequestScheduler: check request schedulability
RequestScheduler-->>PyExecutor: return state-range eligibility
PyExecutor->>PyExecutor: allocate or finalize ADP padding dummy
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
PR_Github #62036 [ run ] completed with state |
|
Flagging one thing before this goes further: the gate widening isn't inert for non-DeepSeek-V4 models in the disagg + attention-DP case, which is the case it targets. Comparing against
That combination is what #16279 deliberately scoped to DSv4, and the PR description acknowledges as much. The test change makes this harder to catch rather than easier: Minor, unrelated: this removes the same |
|
/bot run --disable-fail-fast |
|
The dummy path is generalized but the paired 2x seq-slot headroom stays DSv4-only, worth generalizing both, or gating on the overlap scheduler. Could we also land the code fix alone and leave the waiver removal to a follow-up? |
Thanks. I added Existing allocation-failure and rollback tests cover the other two behavior changes. |
tburt-nv
left a comment
There was a problem hiding this comment.
The whole unittest/_torch/executor directory is covered in the test-db.
IIUC these are separate lifecycle conditions rather than paired requirements. The dummy fix is needed when rank-local padding or allocation disagrees, including this Kimi configuration where overlap is disabled. The 2× slot pool addresses overlap backfill before previous-iteration slot release; this case uses MTP0 with Gating dummy handling on overlap would therefore disable the reported fix, while generalizing slot headroom would require broader validation of all |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tests/unittest/_torch/modeling/test_modeling_qwen2_5vl.py (1)
430-434: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winExpand the test coverage
test_mrope_delta_cache_size_uses_runtime_seq_slot_capacitycovers the runtime override (8 → 9), but not the fallback branch or a non-defaultpp_size.Test coverage summary
- Added:
test_mrope_delta_cache_size_uses_runtime_seq_slot_capacity- Modified/removed: none
- Test-list membership:
tests/integration/test_lists/test-db/l0_l40s.ymllistsunittest/_torch/modeling/test_modeling_qwen2_5vl.py::TestQwen2_5_VL::test_all; no explicit entry fortest_mrope_delta_cache_size_uses_runtime_seq_slot_capacity- Coverage verdict: needs follow-up
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/_torch/modeling/test_modeling_qwen2_5vl.py` around lines 430 - 434, Add coverage for _get_mrope_position_delta_cache_size covering both the default fallback when max_num_seq_slots is absent and a non-default pp_size. Keep the existing runtime override test unchanged and assert the expected cache sizes for each additional configuration.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tests/unittest/_torch/modeling/test_modeling_qwen2_5vl.py`:
- Around line 430-434: Add coverage for _get_mrope_position_delta_cache_size
covering both the default fallback when max_num_seq_slots is absent and a
non-default pp_size. Keep the existing runtime override test unchanged and
assert the expected cache sizes for each additional configuration.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: d142e035-42c2-4f06-912b-ee1dcac52c72
📒 Files selected for processing (18)
tensorrt_llm/_torch/models/modeling_qwen2vl.pytensorrt_llm/_torch/models/modeling_qwen3vl.pytensorrt_llm/_torch/pyexecutor/_util.pytensorrt_llm/_torch/pyexecutor/model_engine.pytensorrt_llm/_torch/pyexecutor/model_loader.pytensorrt_llm/_torch/pyexecutor/py_executor.pytensorrt_llm/_torch/pyexecutor/py_executor_creator.pytensorrt_llm/_torch/pyexecutor/scheduler/scheduler.pytensorrt_llm/_torch/pyexecutor/scheduler/scheduler_v2.pytensorrt_llm/_torch/speculative/interface.pytensorrt_llm/_torch/speculative/utils.pytests/unittest/_torch/executor/test_benchmark_disagg.pytests/unittest/_torch/executor/test_dual_pool_kv_cache.pytests/unittest/_torch/executor/test_model_loader_gms.pytests/unittest/_torch/executor/test_py_executor.pytests/unittest/_torch/executor/test_pytorch_model_engine.pytests/unittest/_torch/executor/test_seq_slot_sizing.pytests/unittest/_torch/modeling/test_modeling_qwen2_5vl.py
🚧 Files skipped from review as they are similar to previous changes (16)
- tensorrt_llm/_torch/speculative/utils.py
- tensorrt_llm/_torch/pyexecutor/scheduler/scheduler_v2.py
- tensorrt_llm/_torch/pyexecutor/py_executor_creator.py
- tensorrt_llm/_torch/speculative/interface.py
- tests/unittest/_torch/executor/test_model_loader_gms.py
- tensorrt_llm/_torch/models/modeling_qwen3vl.py
- tensorrt_llm/_torch/pyexecutor/model_loader.py
- tensorrt_llm/_torch/pyexecutor/_util.py
- tests/unittest/_torch/executor/test_benchmark_disagg.py
- tests/unittest/_torch/executor/test_seq_slot_sizing.py
- tests/unittest/_torch/executor/test_pytorch_model_engine.py
- tests/unittest/_torch/executor/test_dual_pool_kv_cache.py
- tensorrt_llm/_torch/pyexecutor/py_executor.py
- tensorrt_llm/_torch/pyexecutor/scheduler/scheduler.py
- tensorrt_llm/_torch/pyexecutor/model_engine.py
- tests/unittest/_torch/executor/test_py_executor.py
|
PR_Github #62608 [ run ] triggered by Bot. Commit: |
|
PR_Github #62608 [ run ] completed with state
|
|
On the
and underneath them it is essentially the entire content of those directories — A whole-stage wipeout that clean is not the shape of unrelated flake, and the blast radius is exactly the surface this PR moved: Two things worth separating before the next run: whether this is a slot-indexing correctness break (something still sized or indexed by Separately, and this is the part I'd still want settled independent of the CI result: the previous form was opt-in ( |
|
[by Codex] @thorjohnsen Friendly review reminder: this PR is awaiting your review. Thanks! |
|
/bot run --disable-fail-fast --stage-list "DGX_B200-PyTorch-5" |
|
PR_Github #62853 [ run ] triggered by Bot. Commit: |
|
PR_Github #62853 [ run ] completed with state |
|
/bot run --disable-fail-fast |
|
PR_Github #62890 [ run ] triggered by Bot. Commit: |
|
PR_Github #62890 [ run ] completed with state
|
|
/bot run --disable-fail-fast --stage-list "DGX_B200-4_GPUs-PyTorch-3, DGX_B200-8_GPUs-PyTorch-1" |
|
PR_Github #63068 [ run ] triggered by Bot. Commit: |
|
PR_Github #63068 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #63096 [ run ] triggered by Bot. Commit: |
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
3126052 to
8a8286c
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #63137 [ run ] triggered by Bot. Commit: |
|
PR_Github #63096 [ run ] completed with state |
2ez4bz
left a comment
There was a problem hiding this comment.
Approving the modeling changes.
|
PR_Github #63137 [ run ] completed with state
|
|
/bot run --disable-fail-fast --stage-list "A10-Build_Docs, B300-PyTorch-2" |
|
PR_Github #63244 [ run ] triggered by Bot. Commit: |
|
PR_Github #63244 [ run ] completed with state |
Summary
Fixes NVBUG 6487039.
This PR makes the non-PP disaggregated attention-DP dummy-request lifecycle model-independent and derives sequence-slot capacity from scheduler behavior rather than model identity. It is the generalized follow-up to the transactional dummy handling introduced in #16279.
The affected GB300 Kimi E2E test remains waived. This PR fixes the dummy-request bookkeeping failure, but it does not address the separate Python-transceiver liveness and performance failures seen by that workload.
Failure mechanism
MicroBatchSchedulerschedules requests only within its configured state window. The legacy non-DeepSeek-V4 active-request count instead treated terminal and wait states outside that window as schedulable. A rank holding only those states could therefore omit a required pad dummy while peer attention-DP ranks allocated tentative dummies.If the TP-wide queue decision then rejected the iteration, the tentative peer dummies were not rolled back. They accumulated across retries until:
Recovery subsequently hit
threads can only be started once, followed by the HangDetector/MPI abort. Those errors are consequences of the leaked dummy requests, not independent root causes.The generalization also exposed two related lifecycle assumptions:
2 * max_batch_sizeentries. Non-overlap execution needsmax_batch_size; PP retainspp_size * max_batch_size.ENCODER_INIT, notCONTEXT_INIT, and decoder-context requests waiting for encoder output are not yet schedulable.Changes
max_num_seq_slotsfrom PP and overlap lifecycle requirements, then propagate the same capacity to the executor, model engine, sampler, guided decoder, speculative metadata, and Qwen2/Qwen3-VL MRoPE caches.The affected Kimi configuration has overlap scheduling disabled, so it retains
max_batch_sizeslots while receiving the generalized dummy-lifecycle correction.Verification
31260522c, bothDGX_B200-4_GPUs-PyTorch-3andDGX_B200-8_GPUs-PyTorch-1, including their pytest stages, passed in downstream build #2623. The parent pipeline #51166 was red because its SBSA build failed; the targeted x86 multi-GPU run succeeded.8a8286c3fin pipeline #51225. The live pipeline link is the source of truth for its final result.Unit coverage includes scheduler-state eligibility, encoder-decoder readiness, overlap/no-overlap/PP sequence-slot sizing and propagation, mixed-rank padding, peer-empty rollback, allocation failure, full-capacity checks, draft-token reserve, MRoPE capacity, and PP fallback.
Scope and related work
This PR intentionally does not:
main);The broader transfer and test-harness work is tracked separately in #16918, #16920, #17134, and #17140.