Skip to content

[None][feat] improve iteration stats reporting#14897

Merged
lfr-0531 merged 13 commits into
NVIDIA:feat/deepseek_v4from
yizhang-nv:yizhan/iter-stats-unbounded
Jun 5, 2026
Merged

[None][feat] improve iteration stats reporting#14897
lfr-0531 merged 13 commits into
NVIDIA:feat/deepseek_v4from
yizhang-nv:yizhan/iter-stats-unbounded

Conversation

@yizhang-nv

@yizhang-nv yizhang-nv commented Jun 3, 2026

Copy link
Copy Markdown
Member

Summary:

  • Allow -1 as the unbounded sentinel for iter/request stats buffers.
  • Preserve C++ semantics where 0 disables iter/request stats, while Python max_stats_len=0 keeps the legacy bounded-buffer trim behavior.
  • Avoid overlapping iteration-stats drains in the OpenAI server metrics path.
  • Track KV cache peak block stats by cache level and report primary/secondary peak free, used, and evictable blocks.
  • Serialize the new peak free/evictable fields alongside the existing peak used fields, with C++ binding and serializer coverage.

Testing:

  • python3 scripts/build_wheel.py --use_ccache --skip_building_wheel --linking_install_binary
  • pytest -s tests/unittest/bindings/test_executor_bindings.py::test_executor_config
  • pytest -s tests/unittest/executor/test_stats_serializer.py
  • tests/unittest/llmapi/apps/test_disagg_perf_metrics_collector.py
  • tests/unittest/llmapi/apps/test_openai_server_iteration_stats.py

@yizhang-nv
yizhang-nv changed the base branch from main to feat/deepseek_v4 June 3, 2026 07:44
@yizhang-nv yizhang-nv changed the title [None][feat] support unbounded iter stats buffers [None][feat] improve iteration stats reporting Jun 3, 2026
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
@yizhang-nv
yizhang-nv force-pushed the yizhan/iter-stats-unbounded branch from 49cdc3e to 3ad52f0 Compare June 3, 2026 09:00
@yizhang-nv

Copy link
Copy Markdown
Member Author

/bot run --disable-fail-fast

@yizhang-nv
yizhang-nv marked this pull request as ready for review June 3, 2026 14:24
@yizhang-nv
yizhang-nv requested review from a team as code owners June 3, 2026 14:24
@yizhang-nv
yizhang-nv requested review from achartier and zhenhuaw-me and removed request for a team June 3, 2026 14:24
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51858 [ run ] triggered by Bot. Commit: 3ad52f0 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51858 [ run ] completed with state SUCCESS. Commit: 3ad52f0
/LLM/main/L0_MergeRequest_PR pipeline #41216 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
@yizhang-nv

Copy link
Copy Markdown
Member Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51982 [ run ] triggered by Bot. Commit: 1224a72 Link to invocation

@yizhang-nv
yizhang-nv force-pushed the yizhan/iter-stats-unbounded branch from b01fe80 to 25b3852 Compare June 4, 2026 06:25
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
@yizhang-nv
yizhang-nv force-pushed the yizhan/iter-stats-unbounded branch from 25b3852 to 9b55558 Compare June 4, 2026 06:39
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51982 [ run ] completed with state FAILURE. Commit: 1224a72
/LLM/main/L0_MergeRequest_PR pipeline #41328 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@yizhang-nv

Copy link
Copy Markdown
Member Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #52251 [ run ] triggered by Bot. Commit: 9b55558 Link to invocation

Comment thread tensorrt_llm/runtime/kv_cache_manager_v2/_core/_kv_cache_manager.py Outdated
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Comment thread tensorrt_llm/runtime/kv_cache_manager_v2/__init__.pyi Outdated
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #52251 [ run ] completed with state SUCCESS. Commit: 9b55558
/LLM/main/L0_MergeRequest_PR pipeline #41565 completed with status: 'SUCCESS'

CI Report

Link to invocation

Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
@lfr-0531
lfr-0531 merged commit 337b4a5 into NVIDIA:feat/deepseek_v4 Jun 5, 2026
5 checks passed
jiaganc added a commit to jiaganc/TensorRT-LLM that referenced this pull request Jun 26, 2026
Source-Commit: 337b4a5
Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>
jiaganc added a commit to jiaganc/TensorRT-LLM that referenced this pull request Jun 29, 2026
Source-Commit: 337b4a5
Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants