[TRTLLM-12498][feat] Add support for beam search in disaggregated serving

[athena-nv]

Description

Added support for beam search in disaggregated serving for the PyTorch backend. Previously, the disaggregated path effectively assumed beam_width == 1: the Python transfer path did not carry enough beam-aware block metadata, and the generation server did not receive enough first-token beam state to continue scoring from the context server handoff. This change removes those single-beam assumptions while keeping the transceiver block-id contract as a 1-D list.

Key changes:

• Added a packed 1-D block-id layout for beam search KV transfer:
  • beam 0 contributes the context block prefix;
  • the final per-beam tail blocks are appended to that prefix;
  • the transceiver continues to handle only 1-D block-id arrays.
• Made the Python KV transfer path beam-aware without introducing 2-D transceiver metadata:
  • request block tables are read with req.py_beam_width;
  • KVCacheManager packs per-beam cache indices into the 1-D beam-0-plus-tail layout;
  • _create_kv_slice trims sliding-window and cached-prefix blocks against the beam-0 prefix while preserving appended beam tails;
  • the native sender derives token starts from the beam-0 block count before aligning source and destination block ranges.
• Added first-token beam log probabilities to disaggregated metadata:
  • context-side beam search stores first_gen_log_probs;
  • executor result and OpenAI protocol serialization/deserialization carry those log probabilities with first_gen_tokens;
  • generation-side PyExecutor seeds TorchSampler beam cumulative log probabilities before continuing decoding.
• Seeded generation-side beam history from the context-side first generated tokens:
  • initializes original_tokens at prompt_len;
  • initializes cache_indirection so each first generated token is associated with the correct beam;
  • allows the generation-only server to continue beam history reconstruction as if the request had run continuously.
• Updated the disaggregated OpenAI test client to forward beam-search request fields:
  • sends extra_body={"use_beam_search": True};
  • converts all returned choices into CompletionOutput objects instead of only choice 0.
• Added a disaggregated serving accuracy test for Llama 3.1 8B Instruct beam search using CnnDailymail and the existing beam_width=2 reference.
• Enforced claimResult.totalMatchedTokens <= claimResult.numSharedContextBlocks * mTokensPerBlock so that the prepopulated prompt length only includes tokens whose KV blocks were onboarded for reuse.
  • This is needed for beam search because reusable context is rounded down to shared context blocks; totalMatchedTokens and prepopulatedPromptLength must match that block-level reuse boundary.

Test Coverage

• Added test_beam_search_sampling_batch_disagg_handoff to compare continuous beam search against a simulated disaggregated handoff. It verifies that generation-side seeding from first_gen_log_probs produces the same next tokens, cumulative log probabilities, new log probabilities, and predecessor beams as the continuous path, and that leaving the scores unseeded fails to match.

• Added test_beam_search_disagg_e2e for context-only followed by generation-only beam search with the Python NIXL transceiver. The test validates generated beam outputs and cache_indirection shape.

• Added packed-layout unit tests for block packing, py_beam_width propagation, sender alignment, and _create_kv_slice trimming/cache-skip behavior.

• Extended the disaggregated serving accuracy test to run beam search with max_beam_width=2, Python NIXL KV transfer, and CNN/DailyMail extra_acc_spec="beam_width=2".

• Added the beam-search disaggregated serving accuracy test to l0_dgx_b200.yml.

• Please check this after reviewing the above items as appropriate for this PR.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR enables beam-search sampling across disaggregated context and generation phases by lifting beam-width restrictions, supporting flexible block-ID shapes, propagating first-generation cumulative log probabilities, and seeding generation-side beam state from context-side history.

Changes

Beam-search disaggregated generation

Layer / File(s)	Summary
Cache formatter restriction removal and block-ID shape infrastructure cpp/tensorrt_llm/batch_manager/cacheFormatter.cpp, tensorrt_llm/_torch/disaggregation/base/transfer.py, tensorrt_llm/_torch/disaggregation/native/transfer.py	Removes beamWidth == 1 check from cache formatter; introduces _block_ids_debug_summary helper, updates RecvReqInfo serialization to preserve array shapes with backward compatibility, and adds _as_beam_block_ids normalizer for consistent 2-D beam-shaped block-ID arrays.
Per-beam KV alignment and speculative-decoding trimming tensorrt_llm/_torch/disaggregation/native/transfer.py	Refactors _build_kv_write_meta to normalize block-ID arrays to beam shapes, validate beam-count consistency, trim speculative dst blocks per-beam, and compute per-beam token offsets with sliding-window stale-block adjustments using prompt length; receiver now logs debug summaries before returning.
Cache reuse and transceiver beam-width support tensorrt_llm/_torch/disaggregation/resource/cache_reuse.py, tensorrt_llm/_torch/disaggregation/transceiver.py	Cache reuse adapters now accept beam_width and tile block IDs across beams; transceiver reshapes 1-D block IDs to [beam_width, blocks] form, applies sliding-window trimming, and selects final block-ID shape based on beam count.
First-generation cumulative log-probability field plumbing tensorrt_llm/_torch/pyexecutor/llm_request.py, tensorrt_llm/disaggregated_params.py, tensorrt_llm/serve/openai_protocol.py	Adds first_gen_cum_log_probs field to PyResult (with Diff support for rank sync), DisaggregatedParams, LlmResult (via passthrough), and OpenAI DisaggregatedParams protocol; implements setter, property, and bidirectional conversion.
Result extraction and service validation tensorrt_llm/executor/result.py, tensorrt_llm/serve/openai_disagg_service.py	Extracts first_gen_cum_log_probs from context-phase responses into disaggregated_params with fallback to cumulative logprobs; relaxes choice-count validation and logs per-choice disaggregated metadata.
PyExecutor beam-search state seeding from context phase tensorrt_llm/_torch/pyexecutor/py_executor.py	Seeds generation-side sampler with context-side first-generated tokens and cumulative log-probability scores via new _seed_disagg_beam_original_tokens and _seed_disagg_beam_cum_log_probs methods; includes validation and structured logging.
Resource manager beam-width-aware cache indices tensorrt_llm/_torch/pyexecutor/resource_manager.py	Adds compute_block_token_counts helper and extends get_batch_cache_indices with beam_width parameter; reshapes per-beam block-ID results into layouts with per-block token counts and validates beam-count consistency.
TorchSampler beam-search debug logging and observability tensorrt_llm/_torch/pyexecutor/sampler.py	Adds defensive tensor-to-host helpers for safe logging and instruments beam-search paths (_prepare_beam_search, _prepare_beam_history, _finalize_beam, update_requests, _sample_batched_by_strategy, _unbatch_sampling_results) with detailed buffer state and metadata logs.
Integration test harness for beam-search accuracy tests/integration/defs/accuracy/test_disaggregated_serving.py	Extends run_accuracy_test with optional extra_acc_spec and sampling_params; updates request building to support beam search and guided decoding; generalizes result collection to enumerate all choices; adds test_beam_search for CnnDailymail.
Unit test disaggregation validators and e2e test tests/unittest/_torch/sampler/test_beam_search.py	Adds validate_disagg_output and validate_disagg_outputs helpers for two-stage validation; standardizes kv_cache_config in _build_llm; introduces test_beam_search_disagg_e2e for separate context and generation LLM validation with disaggregated parameters.
Unit test regression: continuous vs. disaggregated handoff tests/unittest/_torch/sampler/test_beam_search.py	Adds test_beam_search_sampling_batch_disagg_handoff regression test verifying disaggregated context→generation beam-search produces identical tokens and sampler buffers (cum_log_probs, predecessor_beams, new_log_probs) as continuous baseline when seeded, diverging when unseeded.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• NVIDIA/TensorRT-LLM#8509: This PR depends on the TorchSampler beam-search metadata and storage infrastructure introduced there; the new first_gen_cum_log_probs propagation directly seeds sampler buffers managed by that PR's foundation.

Suggested labels

api-compatible

Suggested reviewers

• dcampora
• danielafrimi
• suyoggupta
• Funatiq
• Tabrizian

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 26.87% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: adding beam search support to disaggregated serving, with proper ticket format and type annotation.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	PR description is comprehensive, clearly explaining the problem (single-beam assumption in disaggregated serving), the solution (removing that assumption while maintaining transceiver contract), and key implementation details across all affected components.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 11

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)

tensorrt_llm/_torch/pyexecutor/resource_manager.py (1)

1196-1201: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Fix return type for beam-aware get_batch_cache_indices
When beam_width > 1, get_batch_cache_indices returns a per-request list of beams (List[List[List[int]]]), but the signature still declares List[List[int]].

♻️ Proposed typing fix

-    ) -> List[List[int]]:
+    ) -> List[List[int]] | List[List[List[int]]]:

Consider using @overload as a follow-up to precisely type the beam_width=1 vs beam_width>1 cases.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tensorrt_llm/_torch/pyexecutor/resource_manager.py` around lines 1196 - 1201,
The return type annotation for get_batch_cache_indices is incorrect for
beam-aware calls: when beam_width > 1 the function returns a nested
per-request-per-beam structure (List[List[List[int]]]) but the signature
declares List[List[int]]; update the function's return type to reflect the
beam-aware shape (or add `@overload` signatures to distinguish beam_width==1 vs
>1) and adjust any related type imports so callers and linters correctly see the
List[List[List[int]]] result for multi-beam cases; reference the
get_batch_cache_indices function and its beam_width parameter when making the
change.

tensorrt_llm/_torch/pyexecutor/py_executor.py (1)

3575-3590: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Validate first_gen_tokens before indexing by beam.

Line 3589 assumes the handoff always carries one token per beam, but this path never checks that context_phase_params.first_gen_tokens is present and len(...) == beam_width. A truncated payload will raise here and abort the executor loop instead of failing just this request.

Suggested guard

                 first_gen_tokens = req.context_phase_params.first_gen_tokens
                 ctx_draft_tokens = req.context_phase_params.draft_tokens
                 req.py_draft_tokens = [] if ctx_draft_tokens is None else ctx_draft_tokens
                 beam_width = req.sampling_config.beam_width
+                if (first_gen_tokens is None
+                        or len(first_gen_tokens) != beam_width):
+                    self._handle_errors(
+                        "Invalid first_gen_tokens length for disagg beam "
+                        f"request {req.py_request_id}: "
+                        f"{0 if first_gen_tokens is None else len(first_gen_tokens)} != {beam_width}",
+                        requests=[req],
+                        charge_budget=False,
+                    )
+                    continue
                 logger.info(

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tensorrt_llm/_torch/pyexecutor/py_executor.py` around lines 3575 - 3590, The
code assumes req.context_phase_params.first_gen_tokens contains exactly
beam_width tokens before looping; add a guard that checks first_gen_tokens is
not None and len(first_gen_tokens) >= beam_width (or == beam_width) before
indexing. If the check fails, log a request-scoped warning via logger including
req.py_request_id and ctx_request_id, mark or fail only this request (do not
raise), and skip the per-beam add_new_token/seed call (or supply safe defaults)
so the executor loop continues; update the block referencing first_gen_tokens,
req.add_new_token, beam_width, and self._seed_disagg_beam_cum_log_probs
accordingly.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tensorrt_llm/_torch/disaggregation/native/transfer.py`:
- Around line 64-69: The current _block_ids_debug_summary eagerly converts large
numpy arrays to Python lists (arr.tolist()) causing CPU/alloc and log bloat on
the hot path; change the logging payload to include only lightweight metadata
(shape and count) by default and only produce the full block id lists when the
logger is in DEBUG mode: update _block_ids_debug_summary to return shapes and
lengths for each layer_group and move the arr.tolist() conversion behind a debug
check (e.g., logger.isEnabledFor(logging.DEBUG)) so the costly conversion is
skipped in normal INFO logs; apply the same pattern where full block-id dumps
are used elsewhere (references: the other debug-summary-like blocks and
functions around the noted areas and any uses that call
_block_ids_debug_summary) so full lists are generated only when debug logging is
enabled.

In `@tensorrt_llm/_torch/disaggregation/transceiver.py`:
- Around line 223-233: The info log inside _create_kv_slice (the
"create_kv_slice final block_ids" message) currently materializes block_ids with
block_ids.tolist() on the hot path; change it to avoid eager full-array
conversion by removing block_ids.tolist() from the info-level message (log only
metadata such as block_ids.shape and count/length) and, if full contents are
ever needed, emit them at debug level guarded by
logger.isEnabledFor(logging.DEBUG) (or similar) so the array is only converted
when debug logging is enabled.

In `@tensorrt_llm/_torch/pyexecutor/py_executor.py`:
- Around line 3606-3619: If seeding the beam-state fails (missing seq_slot,
sampler.store.beam_search_store, or malformed first_gen_cum_log_probs) do not
simply log and return; instead mark the request as failed (set its status to
DISAGG_TRANS_ERROR or invoke the request-failure path) and abort further
processing so partially-initialized beam state cannot be used. Update the
branches that currently log+return (check occurrences around seq_slot,
beam_search_store, and the first_gen_cum_log_probs handling in the
py_executor.py routine) to call the request failure routine or set req.status =
DISAGG_TRANS_ERROR and ensure the function exits immediately after that change.

In `@tensorrt_llm/_torch/pyexecutor/resource_manager.py`:
- Around line 1212-1215: The loop that zips request_ids with the result of
self.impl.get_batch_cache_block_ids() (in resource_manager.py) can silently
truncate if counts differ; add an explicit guard right after calling
get_batch_cache_block_ids() to compare len(result) with len(request_ids) and
raise a clear RuntimeError or AssertionError (including the mismatched lengths
and identifying info like request_ids[:5]) if they differ; apply the same length
check/early-fail before the second zip/loop around the code near the other
occurrence (around line 1247) so any request→block mapping mismatch fails fast
and surfaces the bug.

In `@tensorrt_llm/_torch/pyexecutor/sampler.py`:
- Around line 119-123: The helper function _debug_tensor_slice (and all other
newly introduced helper functions whose names begin with _debug_ in this module)
lack explicit return type annotations; update each function signature to include
an explicit return type (use -> None for helpers that do not return) so they
comply with the repo typing rule—ensure you modify _debug_tensor_slice and the
other _debug_* helpers referenced in the review to add the -> None return
annotation.
- Around line 3324-3336: In TorchSampler._finalize_beam the variable
request_tokens_after is constructed but never logged; update the subsequent
logger.info call (in the _finalize_beam method) to include request_tokens_after
(e.g., as part of the formatted message or an extra kwarg) so the computed
per-beam token info is emitted; ensure you serialize/format request_tokens_after
concisely (repr or json-safe) to avoid huge logs.
- Around line 2898-2908: The verbose tensor-to-host debug dumps in
TorchSampler._prepare_beam_search are causing eager CUDA syncs and must be
gated: change the logger.info calls that call helpers like _debug_tensor_slice
and _debug_cache_indirection_prefix to be executed only when debug logging (or a
sampler debug flag) is enabled — e.g., wrap the entire formatted message
construction in an if logger.isEnabledFor(logging.DEBUG) (or if
self.debug_beam_dumps) and change the call to logger.debug; ensure no
tensor-inspecting helpers are invoked outside that guard so seq_slots_long,
beam_search_store.cum_log_probs, beam_search_store.predecessor_beams,
cache_indirection and cache_indirection_buffer are not materialized on hot
paths. Apply the same guarding pattern to the other affected blocks around the
noted regions (roughly the blocks at ~2920-2930, ~3176-3183, ~3381-3393,
~3981-3996, and ~4133-4142) so info-level logs remain metadata-only and heavy
tensor-to-host operations only occur under the debug flag.

In `@tensorrt_llm/serve/openai_disagg_service.py`:
- Around line 509-515: The code assumes ctx_response.choices has at least one
element and does ctx_response.choices[0] unguarded; add a guard that checks if
ctx_response.choices is empty before indexing (e.g., if not
ctx_response.choices: ...), log or return a controlled validation/error response
to the caller instead of allowing IndexError, and only proceed to set choice =
ctx_response.choices[0] when the list is non-empty; update the surrounding
branch that inspects each choice (the loop over ctx_response.choices and the
subsequent choice = ctx_response.choices[0]) to handle the empty-case
consistently.

In `@tests/unittest/_torch/sampler/test_beam_search.py`:
- Around line 382-385: The helper currently builds ctx_disagg_params with
disagg_request_id=1000 + idx which causes duplicate IDs across separate helper
invocations (e.g., in test_beam_search_disagg_e2e()), so update the helper that
constructs ctx_disagg_params to accept a request_id_base (or similar) and
compute disagg_request_id as 1000 + request_id_base + idx; then call that helper
from the two test sites with different request_id_base values (e.g., 0 and 100
or another offset) so validate_disagg_outputs() sees unique IDs; apply the same
change to the other occurrence referenced around lines 526-531.
- Around line 821-854: The test currently bypasses the real disagg seeding by
directly writing into BeamSearchMetadata.cum_log_probs via make_metadata and
thus misses bugs in the production handoff; update the test to call the
production seeding helper instead of mutating cum_log_probs directly (or add an
invocation of the real handoff function that populates generation-side sampler
state) and then call beam_search_sampling_batch; additionally assert that the
handoff-populated buffers (e.g., BeamSearchMetadata.cache_indirection,
predecessor_beams, cum_log_probs and new_log_probs for seq_slots) match the
expected seeded values (continuous_* variables) to ensure the full seeding path
is exercised and validated rather than only post-seed behavior.
- Around line 803-805: The trailing line for the torch.full call causes a
hanging indent (E128); fix it by reformatting the assignment to use standard
4-space indentation and either keep the call on one line or align continuation
under the opening parenthesis. Update the continuous_metadata.seq_lens =
torch.full((batch_size, ), prompt_len + 1, dtype=torch.int32) expression so the
second line is aligned with the first parenthesis or collapsed to a single line,
referencing the torch.full call and variables batch_size and prompt_len.

---

Outside diff comments:
In `@tensorrt_llm/_torch/pyexecutor/py_executor.py`:
- Around line 3575-3590: The code assumes
req.context_phase_params.first_gen_tokens contains exactly beam_width tokens
before looping; add a guard that checks first_gen_tokens is not None and
len(first_gen_tokens) >= beam_width (or == beam_width) before indexing. If the
check fails, log a request-scoped warning via logger including req.py_request_id
and ctx_request_id, mark or fail only this request (do not raise), and skip the
per-beam add_new_token/seed call (or supply safe defaults) so the executor loop
continues; update the block referencing first_gen_tokens, req.add_new_token,
beam_width, and self._seed_disagg_beam_cum_log_probs accordingly.

In `@tensorrt_llm/_torch/pyexecutor/resource_manager.py`:
- Around line 1196-1201: The return type annotation for get_batch_cache_indices
is incorrect for beam-aware calls: when beam_width > 1 the function returns a
nested per-request-per-beam structure (List[List[List[int]]]) but the signature
declares List[List[int]]; update the function's return type to reflect the
beam-aware shape (or add `@overload` signatures to distinguish beam_width==1 vs
>1) and adjust any related type imports so callers and linters correctly see the
List[List[List[int]]] result for multi-beam cases; reference the
get_batch_cache_indices function and its beam_width parameter when making the
change.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 46aa4d7c-85fd-483b-bcc7-6c418ed7fd2c

📥 Commits

Reviewing files that changed from the base of the PR and between cd38dfb and f8e5746.

📒 Files selected for processing (15)

• cpp/tensorrt_llm/batch_manager/cacheFormatter.cpp
• tensorrt_llm/_torch/disaggregation/base/transfer.py
• tensorrt_llm/_torch/disaggregation/native/transfer.py
• tensorrt_llm/_torch/disaggregation/resource/cache_reuse.py
• tensorrt_llm/_torch/disaggregation/transceiver.py
• tensorrt_llm/_torch/pyexecutor/llm_request.py
• tensorrt_llm/_torch/pyexecutor/py_executor.py
• tensorrt_llm/_torch/pyexecutor/resource_manager.py
• tensorrt_llm/_torch/pyexecutor/sampler.py
• tensorrt_llm/disaggregated_params.py
• tensorrt_llm/executor/result.py
• tensorrt_llm/serve/openai_disagg_service.py
• tensorrt_llm/serve/openai_protocol.py
• tests/integration/defs/accuracy/test_disaggregated_serving.py
• tests/unittest/_torch/sampler/test_beam_search.py

💤 Files with no reviewable changes (1)

• cpp/tensorrt_llm/batch_manager/cacheFormatter.cpp

[athena-nv]

/bot run

[Tabrizian]

@athena-nv Could you please rebase to resolve the conflicts?

[athena-nv]

/bot run

[Tabrizian]

Please register this test to test_beam_search in l0_dgx_b200

[athena-nv]

/bot run

[Tabrizian]

/bot run

[tensorrt-cicd]

PR_Github #52406 [ run ] triggered by Bot. Commit: 69d173c Link to invocation

[athena-nv]

/bot run

[athena-nv]

/bot run

[pcastonguay]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #53187 [ run ] triggered by Bot. Commit: 17071ee Link to invocation

[tensorrt-cicd]

PR_Github #53187 [ run ] completed with state SUCCESS. Commit: 17071ee
/LLM/main/L0_MergeRequest_PR pipeline #42387 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[athena-nv]

/bot run --disable-fail-fast

[athena-nv]

/bot run --disable-fail-fast

[yuanjingx87]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #53349 [ run ] triggered by Bot. Commit: 17071ee Link to invocation

[athena-nv]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #53370 [ run ] triggered by Bot. Commit: 17071ee Link to invocation

[tensorrt-cicd]

PR_Github #53349 [ run ] completed with state ABORTED. Commit: 17071ee

Link to invocation

[tensorrt-cicd]

PR_Github #53370 [ run ] completed with state FAILURE. Commit: 17071ee
/LLM/main/L0_MergeRequest_PR pipeline #42548 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[athena-nv]

/bot run --disable-fail-fast

[athena-nv]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #53716 [ run ] triggered by Bot. Commit: 17071ee Link to invocation

[tensorrt-cicd]

PR_Github #53716 [ run ] completed with state SUCCESS. Commit: 17071ee
/LLM/main/L0_MergeRequest_PR pipeline #42845 completed with status: 'SUCCESS'

CI Report

Link to invocation

[Tabrizian]

LGTM except one comment which can be addressed in a follow up PR.

[SimengLiu-nv]

The kvcache related changes LGTM.

[schetlur-nv]

rubber stamp for pending review groups since main review is done