[TRTLLM-14511][feat] BREAKING: refactor per-request perf metrics for multi-process trtllm-server

[reasonsolo]

Doc: TRTLLM-Serving Performance Metrics Design

• Replaces /perf_metrics polling with per-request metrics returned through response headers or an opt-in post-[DONE] SSE event.
• Adds optional asynchronous JSONL dumping using the former /perf_metrics schema plus status and disagg_request_id.
• Supports aggregated and disaggregated serving; ctx/gen metrics are combined by the serving fleet worker, never the coordinator.
• Public metrics require both return_perf_metrics=true and X-TRTLLM-return-metrics: 1; file dumping is independently controlled by perf_metrics_output_dir.
• Adds shared typed schemas, time-breakdown support, and agg/disagg E2E coverage.

Dev Engineer Review

• Perf metrics exposure refactor (API-compatible intent): Removed /perf_metrics polling/endpoint behavior and moved per-request metrics delivery to:
  • Header-based delivery (gated by both return_perf_metrics=true and X-TRTLLM-return-metrics: 1).
  • Streaming delivery via SSE as an opt-in dedicated post-[DONE] metrics event (suppressed from user-visible streaming output).
  • Server-side JSONL output controlled independently via perf_metrics_output_dir.
• Disaggregated propagation & gating correctness:
  • DisaggServerConfig added return_perf_metrics and perf_metrics_output_dir.
  • extract_disagg_cfg(...) now propagates options into context_servers and generation_servers and forces downstream returning when either return_perf_metrics is enabled or perf_metrics_output_dir is set.
  • Updated config inheritance iteration to avoid mutating/iterating over kwargs directly; switched to explicit keyword construction for DisaggServerConfig.
• Typed, schema-driven metrics + disaggregation composition:
  • Added _perf_metrics_schema.py with TypedDict record types supporting aggregated + disaggregated phases and a unified PerfMetricsRecord union.
  • Centralized record building and disagg merging in tensorrt_llm/serve/perf_metrics.py (build_request_metrics_record, combine_disagg_metrics, header formatting, SSE event construction).
• Server pipeline overhaul:
  • openai_server.py removed the legacy in-memory /perf_metrics implementation and deque/lock buffering; metrics are now built into per-request state and emitted at the correct time (explicitly awaited for both non-streaming and streaming completion).
  • openai_disagg_server.py replaced prior collector behavior with queue-latency-aware hooks + conditional collection/persistence, and removed the in-process /perf_metrics route registration.
  • Added PerfMetricsMiddleware and PerfMetricsJsonlWriter, with middleware opting-in header/SSE exposure based on X-TRTLLM-return-metrics: 1 and server config.
• Client behavior changes:
  • OpenAIHttpClient now sets return-metrics headers when request_perf_metrics is enabled, extracts metrics from response headers and/or the SSE post-[DONE] metrics payload, and emits via hooks.
  • Streaming robustness improved: malformed metrics SSE payloads won’t break streaming; retry logic now re-emits disagg_request_id via hooks when regenerated.
  • Removed the old abstract/concrete collect_metrics() plumbing from OpenAIClient/OpenAIHttpClient.
• Tooling & scripts alignment:
  • Benchmarks and time-breakdown tooling were updated to consume server-written JSONL from perf_metrics_output_dir instead of fetching /perf_metrics.
  • [DONE] handling in SSE consumption now exits immediately (return) to prevent extra-chunk processing.

CI failures noted in context: Four reported failures—prioritize validating the new gating/emission/JSONL expectations and any removed /perf_metrics behaviors:

• PR_Github #60916, pipeline #49181, commit 5e0ac04
• PR_Github #61165, pipeline #49414, commit d9ff361
• PR_Github #61270, pipeline #49505, commit dbce2f7
• PR_Github #61508, pipeline #49725, commit dbce2f7

Recommendation for follow-up investigation: focus on test failures around (1) header/SSE gating conditions, (2) disaggregated record merging and disagg_request_id propagation, and (3) JSONL writer output completeness/ordering (including time-breakdown fields), since the /perf_metrics endpoint and its client/scripting usage were removed.

---

QA Engineer Review

Test code/config changes detected (under tests/)

Integration / config updates

• tests/integration/defs/accuracy/test_disaggregated_serving.py
  • Updated perf-metrics enablement to use perf_metrics_output_dir (temp-derived) instead of perf_metrics_max_requests.
  • Removed local /perf_metrics HTTP polling helper and invocation.
• tests/integration/defs/disaggregated/test_disaggregated.py
  • Updated cluster/test harness to accept and inject perf_metrics_output_dir, validate via local JSONL artifacts, and remove prior /perf_metrics fetch/recording logic.
  • Updated perf-metrics assertions to use wait_for_perf_metrics_jsonl/JSONL readers rather than the HTTP endpoint.
• tests/integration/defs/disaggregated/test_configs/disagg_config_conditional_deepseek_v3_v2.yaml
  • Set top-level return_perf_metrics: true.
• tests/integration/defs/disaggregated/test_configs/disagg_config_metrics.yaml
  • Added top-level return_perf_metrics: true.

Test utilities

• tests/test_common/perf_metrics_utils.py
  • Refactored to read perf_metrics-*.jsonl from an output directory.
  • Added read_perf_metrics_jsonl, wait_for_perf_metrics_jsonl.
  • Changed get_timing_metrics(...) to accept output_dir.

Unittest coverage

• tests/unittest/disaggregated/test_disagg_utils.py
  • Added test_extract_disagg_metrics_controls.
• tests/unittest/llmapi/apps/_test_openai_perf_metrics.py
  • Removed: test_metrics_endpoint
  • Added: test_return_perf_metrics_and_jsonl_dump
  • Added: test_streaming_metrics_require_request_opt_in
• tests/unittest/llmapi/apps/_test_openai_chat_harmony_perf_metrics.py
  • Updated to validate via JSONL output (no /perf_metrics drain/count).
  • Signature updates to pass kv_cache_time_output_dir.
• tests/unittest/llmapi/apps/test_disagg_serving_perf_metrics.py
  • Added fixture perf_metrics_output_dir
  • Added test: test_return_perf_metrics_and_jsonl_dump
  • Updated configuration plumbing to thread perf_metrics_output_dir.
• tests/unittest/llmapi/apps/test_request_metrics.py
  • Added new test module covering header formatting, middleware behavior (headers/SSE), JSONL persistence, and record combination/parsing.
• tests/unittest/others/test_time_breakdown.py
  • Added: test_parse_disaggregated_time_breakdown
  • Added: test_parse_jsonl_file
• tests/unittest/disaggregated/test_disagg_openai_client.py
  • Added: test_internal_client_accepts_perf_metrics_header_size
  • Added: test_malformed_streaming_metrics_do_not_fail_request
• Multi-node disaggregated serving tests
  • tests/unittest/llmapi/apps/_test_disagg_serving_multi_nodes.py: removed perf-metrics/timing-metrics plumbing.
  • tests/unittest/llmapi/apps/_test_disagg_serving_multi_nodes_service_discovery.py: removed perf-metrics/timing-metrics plumbing.
• Removed test module
  • tests/unittest/llmapi/apps/test_disagg_perf_metrics_collector.py: entire module removed (collector serialization test deleted).

Coverage mapping vs test lists (CI/manual)

• No updates to tests/integration/test_lists/ are confirmed from the available context (repo diff inspection returned no discoverable file changes in this environment).
• Verdict: needs follow-up — coverage mapping to CBTS test lists wasn’t verifiable here, and the change is substantial (full migration off /perf_metrics to header/SSE/JSONL), so CI failures may require targeted test-list validation and/or additional waivers.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

Replaces the HTTP /perf_metrics endpoint and in-memory request buffering with request-opt-in headers, streaming SSE metrics, and optional server-side JSONL output. Adds typed schemas, configuration propagation, client/server wiring, updated tooling, and validation.

Changes

Server-side JSONL performance metrics pipeline

Layer / File(s)	Summary
Configuration and metrics contracts tensorrt_llm/llmapi/disagg_utils.py, tensorrt_llm/llmapi/llm_args.py, tensorrt_llm/serve/_perf_metrics_schema.py, tensorrt_llm/serve/responses_utils.py	Adds metrics configuration, typed records, disaggregated propagation, request-ID hooks, and API metadata.
Metrics serialization and persistence tensorrt_llm/serve/perf_metrics.py, tests/unittest/llmapi/apps/test_request_metrics.py	Builds, combines, serializes, persists, and exposes metrics through headers and SSE.
OpenAI client metrics propagation tensorrt_llm/serve/openai_client.py, tests/unittest/disaggregated/test_disagg_openai_client.py	Adds request opt-in, response header/SSE parsing, hook delivery, and stream completion handling.
OpenAI and disaggregated server integration tensorrt_llm/serve/openai_server.py, tensorrt_llm/serve/openai_disagg_server.py, tensorrt_llm/serve/openai_disagg_service.py, tests/unittest/disaggregated/test_openai_disagg_service.py	Gates collection, manages writers and middleware, combines disaggregated metrics, and completes context-only request hooks.
Benchmark and time-breakdown tooling tensorrt_llm/serve/scripts/*, examples/disaggregated/slurm/benchmark/run_benchmark_aiperf.sh	Removes HTTP perf-metrics fetching and supports server-side JSONL input.
Integration and application validation tests/integration/*, tests/unittest/llmapi/apps/*, tests/test_common/perf_metrics_utils.py	Configures JSONL output and validates metrics propagation, persistence, joining, headers, SSE, and timing parsing.

Estimated code review effort: 4 (Complex) | ~75 minutes

Sequence Diagram(s)

sequenceDiagram
  participant OpenAIHttpClient
  participant OpenAIDisaggServer
  participant PerfMetricsMiddleware
  participant PerfMetricsJsonlWriter
  OpenAIHttpClient->>OpenAIDisaggServer: send metrics-enabled request
  OpenAIDisaggServer->>PerfMetricsMiddleware: collect completed request records
  PerfMetricsMiddleware->>OpenAIHttpClient: return headers or SSE metrics event
  PerfMetricsMiddleware->>PerfMetricsJsonlWriter: submit JSONL record
  PerfMetricsJsonlWriter->>PerfMetricsJsonlWriter: write per-process JSONL

Loading

Suggested reviewers: frankd412

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 14.81% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	⚠️ Warning	The description has a useful summary, but the required Description, Test Coverage, and checklist sections are mostly left unfilled.	Fill in the Description and Test Coverage sections, and complete the PR checklist with the relevant tests, labels, and documentation updates.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is specific, concise, and accurately summarizes the main breaking refactor of per-request perf metrics.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[tensorrt-cicd]

PR_Github #61892 [ run ] triggered by Bot. Commit: 228a883 Link to invocation

[tensorrt-cicd]

PR_Github #61892 [ run ] completed with state FAILURE. Commit: 228a883
/LLM/main/L0_MergeRequest_PR pipeline #50083 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[reasonsolo]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #61902 [ run ] triggered by Bot. Commit: 228a883 Link to invocation

[tensorrt-cicd]

PR_Github #61902 [ run ] completed with state FAILURE. Commit: 228a883
/LLM/main/L0_MergeRequest_PR pipeline #50093 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[reasonsolo]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #62043 [ run ] triggered by Bot. Commit: 228a883 Link to invocation

[tensorrt-cicd]

PR_Github #62043 [ run ] completed with state SUCCESS. Commit: 228a883
/LLM/main/L0_MergeRequest_PR pipeline #50229 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[fredricz-20070104]

Approved

[reasonsolo]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #62355 [ run ] triggered by Bot. Commit: 6602b50 Link to invocation

[tensorrt-cicd]

PR_Github #62355 [ run ] completed with state FAILURE. Commit: 6602b50
/LLM/main/L0_MergeRequest_PR pipeline #50522 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation