[None][feat] Add per-iteration cached KV length to iter log

[yizhang-nv]

Description

Adds a cached_kv_len field to the per-iteration log emitted when print_iter_log is enabled, reporting the cached KV length (in tokens) currently held for the sequences scheduled in that iteration.

The value is computed in PyTorchModelEngine as the sum over the batch of the per-sequence already-cached prefix (num_cached_tokens_per_seq). Tokens computed in the current step (sequence_lengths) are being written this iteration and are not counted as cached. Using the logical cached length rather than the number of allocated blocks avoids under-counting under sliding window attention (SWA), where the number of allocated blocks is capped by the window size.

It is also stored in iter_states as cached_kv_tokens so the /metrics serializer can pick it up.

Changes

• tensorrt_llm/_torch/pyexecutor/model_engine.py: populate iter_states['cached_kv_tokens'] with sum(num_cached_tokens_per_seq).
• tensorrt_llm/_torch/pyexecutor/py_executor.py: add cached_kv_len to the per-iteration log line.

Validation

Built and served Llama-3.2-1B-Instruct on A100 with print_iter_log: true and sent a completion request (12 prompt tokens, 30 generated). The field behaves as expected:

iter = 1 ... cached_kv_len = 0   states = {..., 'num_ctx_tokens': 12, 'num_generation_tokens': 0, 'cached_kv_tokens': 0}
iter = 2 ... cached_kv_len = 12  states = {..., 'num_generation_tokens': 1, 'cached_kv_tokens': 12}
iter = 3 ... cached_kv_len = 13
...
iter = 31 ... cached_kv_len = 40

• Context iteration reports 0 — the prompt is being written this step and is not yet cached.
• Each generation iteration reports the accumulated already-cached length (prompt + previously generated tokens), growing by 1 per step.

Test Coverage

Behavioral validation via trtllm-serve (above). No new unit test added (log-only change).

🤖 Generated with Claude Code

[yizhang-nv]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #51002 [ run ] triggered by Bot. Commit: e8f6630 Link to invocation

[tensorrt-cicd]

PR_Github #51002 [ run ] completed with state SUCCESS. Commit: e8f6630
/LLM/main/L0_MergeRequest_PR pipeline #40451 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[yizhang-nv]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #51315 [ run ] triggered by Bot. Commit: 25df6d3 Link to invocation

[yizhang-nv]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #51325 [ run ] triggered by Bot. Commit: 8cdc8db Link to invocation

[tensorrt-cicd]

PR_Github #51315 [ run ] completed with state ABORTED. Commit: 25df6d3

Link to invocation

[tensorrt-cicd]

PR_Github #51325 [ run ] completed with state SUCCESS. Commit: 8cdc8db
/LLM/main/L0_MergeRequest_PR pipeline #40739 completed with status: 'SUCCESS'

CI Report

Link to invocation