[None][fix] Cherry pick 11621 - Accept **kwargs in DynamicYamlWithDeepMergeSettingsSource._read_files for pydantic-settings 2.13 compatibility

[tcherckez-nvidia]

Summary by CodeRabbit

• New Features

  • Added support for first-generation logits and log probabilities in disaggregated serving.
  • Implemented request cancellation support for disaggregated generation scenarios.

• Bug Fixes

  • Fixed NCCL deadlock issue by using plain NCCL instead of NCCL_SYMMETRIC fallback.
  • Improved multimodal embedding validation with stricter None checks.

• Performance

  • Optimized memory usage with in-place tensor operations in model computations.

• Chores

  • Bumped version to 1.3.0rc5.post1.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[tcherckez-nvidia]

/bot run

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 48cbe0b0-0926-4a58-8049-4ab639a723e4

📥 Commits

Reviewing files that changed from the base of the PR and between 2ee7dba and 27f33a0.

📒 Files selected for processing (22)

• README.md
• cpp/include/tensorrt_llm/batch_manager/kvCacheManager.h
• cpp/tensorrt_llm/batch_manager/kvCacheManager.cpp
• cpp/tests/unit_tests/batch_manager/kvCacheManagerTest.cpp
• examples/constraints.txt
• tensorrt_llm/_torch/auto_deploy/utils/_config.py
• tensorrt_llm/_torch/custom_ops/torch_custom_ops.py
• tensorrt_llm/_torch/models/modeling_deepseekv3.py
• tensorrt_llm/_torch/models/modeling_qwen2vl.py
• tensorrt_llm/_torch/models/modeling_qwen3vl.py
• tensorrt_llm/_torch/pyexecutor/llm_request.py
• tensorrt_llm/_torch/pyexecutor/py_executor.py
• tensorrt_llm/disaggregated_params.py
• tensorrt_llm/executor/result.py
• tensorrt_llm/serve/openai_protocol.py
• tensorrt_llm/version.py
• tests/integration/defs/disaggregated/test_disaggregated_single_gpu.py
• tests/integration/test_lists/qa/llm_function_core.txt
• tests/integration/test_lists/test-db/l0_h100.yml
• tests/unittest/_torch/executor/test_chunked_logits.py
• tests/unittest/_torch/multimodal/test_mm_encoder_standalone.py
• tests/unittest/disaggregated/test_openai_disagg_service.py

---

📝 Walkthrough

Walkthrough

This PR bumps the package version to 1.3.0rc5.post1 and introduces features for disaggregated inference: propagating first-generation logits and log probabilities from the prefill phase to the decode phase. Additionally, it refines KV cache reuse accounting by filtering allocated blocks, optimizes memory in DeepSeekV3, tightens None-safety checks in QwenVL models, adjusts fallback tactics for allreduce operations, and expands test coverage for disaggregated scenarios.

Changes

Cohort / File(s)	Summary
Version Updates README.md, tensorrt_llm/version.py, examples/constraints.txt	Bump version from 1.3.0rc5 to 1.3.0rc5.post1 across package metadata and documentation.
KV Cache Manager Enhancements cpp/include/tensorrt_llm/batch_manager/kvCacheManager.h, cpp/tensorrt_llm/batch_manager/kvCacheManager.cpp, cpp/tests/unit_tests/batch_manager/kvCacheManagerTest.cpp	Add onlyAllocated parameter to countReusableBlocks across WindowBlockManager, BlockManager, BaseKVCacheManager, and KVCacheManager classes to exclude free reusable blocks from accounting when enabled. Update test expectations to reflect that free reusable blocks no longer reduce needed/remaining block counts.
Memory Optimization tensorrt_llm/_torch/models/modeling_deepseekv3.py	Replace addition operations with in-place addition (.add_()) in MOE output handling to reduce peak memory usage.
Disaggregated Logits/Logprobs Propagation tensorrt_llm/disaggregated_params.py, tensorrt_llm/serve/openai_protocol.py, tensorrt_llm/_torch/pyexecutor/llm_request.py, tensorrt_llm/_torch/pyexecutor/py_executor.py, tensorrt_llm/executor/result.py	Add first_gen_log_probs and first_gen_logits fields to propagate first-generation token data from prefill to decode servers. Implement serialization/deserialization helpers, new getter methods, and prepending logic in PyExecutor and result handlers to integrate these fields into the disaggregated execution pipeline.
Model and Utility Fixes tensorrt_llm/_torch/models/modeling_qwen2vl.py, tensorrt_llm/_torch/models/modeling_qwen3vl.py, tensorrt_llm/_torch/custom_ops/torch_custom_ops.py, tensorrt_llm/_torch/auto_deploy/utils/_config.py	Tighten None-safety checks in Qwen VL multimodal request selection; change allreduce fallback tactic from NCCL_SYMMETRIC to NCCL to avoid deadlock; add **kwargs parameter to _read_files for parent API compatibility.
Disaggregated Test Infrastructure tests/integration/defs/disaggregated/test_disaggregated_single_gpu.py	Add MPI control tags and worker orchestration support for cancellable requests, streaming logits propagation, and lifecycle logging. Introduce new test functions for logprobs, logits, and cancellation scenarios with improved error handling and cleanup paths.
Test Coverage Additions tests/unittest/_torch/executor/test_chunked_logits.py, tests/unittest/_torch/multimodal/test_mm_encoder_standalone.py, tests/unittest/disaggregated/test_openai_disagg_service.py, tests/integration/test_lists/qa/llm_function_core.txt, tests/integration/test_lists/test-db/l0_h100.yml	Add unit and integration test coverage for logits behavior, chunked prefill smoke tests, serialization roundtrips for first-generation data, and new disaggregated test cases for TinyLlama model.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant ContextServer as Context Server<br/>(Prefill)
    participant GenServer as Generation Server<br/>(Decode)
    participant ParamTransfer as Parameter<br/>Transfer

    Client->>ContextServer: Submit request with prompt
    ContextServer->>ContextServer: Execute prefill phase<br/>Generate first tokens
    ContextServer->>ContextServer: Compute first_gen_logits<br/>and first_gen_log_probs
    ContextServer->>ParamTransfer: Package disaggregated_params<br/>with first_gen_* fields
    ParamTransfer->>GenServer: Transfer params + first_gen_*<br/>to generation server
    GenServer->>GenServer: Deserialize first_gen_logits<br/>and first_gen_log_probs
    GenServer->>GenServer: Prepend first_gen_* to<br/>generation results
    GenServer->>GenServer: Continue generation phase<br/>with prepended data
    GenServer->>Client: Return combined response<br/>(prefill + generation)

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• [https://nvbugs/5934461][fix] Propagate logits from prefill to decode in disagg #11767: Overlapping changes to disaggregated logits propagation, adding first_gen_logits to DisaggregatedParams and wiring serialization/deserialization in prefill-to-decode handoff.
• [None][chore] Weekly mass integration of release/1.1 #8918: Related KV cache manager modifications affecting KVCacheManager::getNeededBlocksOneStep and token sequence handling.
• [https://nvbugs/5935104][fix] Cherry-pick Fix overly aggressive capacity scheduler #11834: Identical changes to KV cache reuse accounting with onlyAllocated parameter propagation across WindowBlockManager, BlockManager, and test updates.

Suggested reviewers

• schetlur-nv
• chuangz0
• Superjomn
• hlu1

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

Tip

Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs).
Share your feedback on Discord.

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}