[None][feat] Add per-rank perf time events capture (TRTLLM_PERF_TIME_EVENTS_PATH) - #16900
[None][feat] Add per-rank perf time events capture (TRTLLM_PERF_TIME_EVENTS_PATH)#16900chenfeiz0326 wants to merge 3 commits into
Conversation
…EVENTS_PATH)
Add a single master switch, TRTLLM_PERF_TIME_EVENTS_PATH (an output
directory, symmetric with TRTLLM_KVCACHE_TIME_OUTPUT_PATH), that turns
on per-request perf-timing capture and writes per-rank time-event files
live from inside the executor -- mirroring how the disaggregation
transceiver already writes per-rank KV-transfer CSVs. No HTTP
/perf_metrics scrape, no server-teardown race, no per-worker
return_perf_metrics / --save-request-time-breakdown fiddling.
What the switch does:
* Force-enables capture independent of return_perf_metrics and turns on
an extended capture mode (capture_extended). All new behavior is
gated on capture_extended, so existing return_perf_metrics=True users
with the env unset see NO behavior or output change.
* Adds the previously-missing per-iteration batch context to each
per-iteration metric dict: iter_batch_size, num_ctx_requests,
num_gen_requests, context_token_number, generation_token_number,
per-request req_context_token_number / req_generation_token_number,
plus a per-iteration starvation count (num_capacity_fitting vs
num_scheduled). These ride inside the existing time_breakdown_metrics
payload, so there is no new payload shape and no create_response
change.
* Writes {dir}/time_events_rank{N}_pid{P}.jsonl live, one JSONL line
per finished request.
Off critical path by design (overhead requirement): the executor loop
only builds a dict and does a non-blocking queue.Queue.put_nowait; a
lazily-started daemon thread does the json.dumps + write + flush;
PyExecutor.shutdown() drains and joins it. queue.Full -> warn + drop,
never block.
An OPTIONAL offline aggregator
(python -m tensorrt_llm.serve.scripts.perf_time_events) stitches the
per-rank files together with the KV-transfer CSVs into one combined
JSON/HTML and computes derived signals (inter-step / inter-chunk gaps,
per-iteration starvation). It reuses time_breakdown's RequestTimeBreakdown
for HTML and is stdlib-only on the parse/merge/JSON path (torch-free;
plotly pulled in lazily only for --html), so it stays under the
GPU-free import contract.
Python only -- no C++. The KV sub-events already exist in Python for the
target PYTHON-runtime (disagg NIXL) workload, so they are only ingested.
Flagged Python-only compromises (see docs + PR):
* Category 1b starvation is a per-iteration COUNT, not per-request
attribution -- _schedule() returns only num_fitting_reqs in Python.
* _executor_loop_pp never calls save_timing_to_requests, so the
extended fields are absent under pipeline parallelism (known gap).
* Per-rank files are per-server; disagg ctx<->gen correlation reuses
the request's disagg ctx_request_id when available, else it is an
aggregator-side join enhancement.
Tests (GPU-free):
* tests/unittest/_torch/executor/test_perf_metrics_events.py --
append_step_metrics merge + token counts (and the disabled-shape
guard), _compute_iter_batch_context, and the off-thread per-rank
writer (writes one line, no-ops when metrics absent / disabled).
* tests/unittest/others/test_perf_time_events.py -- aggregator
parse/merge/join/derive, stdlib-only.
Docs: docs/source/developer-guide/perf-analysis.md gains a
"Perf Time Events (per-rank live capture)" section documenting the
switch, the JSONL record shape, the three known limitations, and the
aggregator recipe.
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
WalkthroughAdds opt-in per-rank, router, and client performance event capture, extended executor timing context, server activation through environment variables, and an offline JSONL/KV aggregation CLI with tests and documentation. ChangesPerf time-events workflow
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant Router
participant OpenAIServer
participant PyExecutor
participant PerfTimeEventsMerger
Client->>Router: send request and record client timestamps
Router->>OpenAIServer: dispatch context and generation requests
OpenAIServer->>PyExecutor: enable performance metrics
PyExecutor->>OpenAIServer: return response with timing events
PerfTimeEventsMerger->>PerfTimeEventsMerger: merge worker, router, client, and KV artifacts
PerfTimeEventsMerger->>Client: write combined JSON or HTML timeline
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (10)
tensorrt_llm/serve/scripts/perf_time_events/__init__.py (1)
26-33: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winSort
__all__to satisfy Ruff RUF022.As per static analysis, Ruff flags "`__all__` is not sorted" (RUF022).♻️ Suggested fix
__all__ = [ "PerfTimeEventsMerger", - "parse_event_dir", - "parse_kv_csv_dir", "main", + "parse_event_dir", + "parse_kv_csv_dir", ]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/serve/scripts/perf_time_events/__init__.py` around lines 26 - 33, Sort the entries in __all__ alphabetically to satisfy Ruff RUF022, while preserving the existing exported symbols from the perf_time_events package.Source: Linters/SAST tools
tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.py (7)
1-1: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winCSV header contract with the KV-transfer producer is hand-duplicated, not imported — drift wouldn't be caught. Both the parser and its test independently hard-code
_NATIVE_TASK_HEADER/_GEN_SUMMARY_HEADER, each claiming to mirror_torch/disaggregation/native/perf_logger.py, but neither imports from that module, so a real header change there would silently fall through to the generic/unknown-layout branch instead of failing a test.
tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.py#L108-177: import the header constants from_torch/disaggregation/native/perf_logger.py(or a shared constants module) instead of re-declaring them here.tests/unittest/others/test_perf_time_events.py#L1-213: assert against the same imported constants (or add a dedicated test that importsperf_logger.py's header and compares it toperf_time_events.py's copy) so an actual producer-side header change trips this suite.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.py` at line 1, Replace the locally duplicated _NATIVE_TASK_HEADER and _GEN_SUMMARY_HEADER declarations in perf_time_events.py with imports from _torch/disaggregation/native/perf_logger.py or a shared constants module. Update test_perf_time_events.py to use those same producer-owned constants, ensuring parser expectations and tests detect future header changes.
205-209: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAdd
-> Noneto__init__.As per coding guidelines, "Annotate every function, use
Nonefor non-returning functions."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.py` around lines 205 - 209, Update the __init__ method signature to include an explicit -> None return annotation, leaving its existing initialization logic unchanged.
347-390: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAdd
-> Nonetomain(). Otherwise the CLI wiring, error path (Line 383-384), and merge/write/html sequencing look correct.As per coding guidelines, "Annotate every function, use
Nonefor non-returning functions."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.py` around lines 347 - 390, Annotate the CLI entry-point function main with an explicit -> None return type, preserving its existing argument parsing, validation, merge, and output sequencing.
61-71: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueMissing return type annotation on
_iter_jsonl.Generator function lacks an annotation.
As per coding guidelines, "Annotate every function, use
Nonefor non-returning functions... prefer built-in generic types."♻️ Suggested fix
-def _iter_jsonl(path: str): +def _iter_jsonl(path: str) -> "Iterator[dict[str, Any]]":(add
from collections.abc import Iteratorto imports)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.py` around lines 61 - 71, The generator function _iter_jsonl lacks a return type annotation. Import Iterator from collections.abc and annotate _iter_jsonl’s return type as Iterator using the appropriate parsed JSON value type, while preserving its existing behavior.
30-36: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse PEP 585 generics instead of
typing.Dict/List/Optional.This is a Python 3.10+ codebase;
Dict[str, Any],List[...],Optional[...]are used throughout this file (parse_event_dir,parse_kv_csv_dir,PerfTimeEventsMergerfields/methods) wheredict[str, Any],list[...],X | Noneare preferred.As per coding guidelines, "Annotate every function... prefer built-in generic types and
|."♻️ Example (apply pattern file-wide)
-from typing import Any, Dict, List, Optional +from typing import AnyThen replace
Dict[K, V]→dict[K, V],List[X]→list[X],Optional[X]→X | Nonethroughout the file.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.py` around lines 30 - 36, Update type annotations throughout perf_time_events.py, including parse_event_dir, parse_kv_csv_dir, and PerfTimeEventsMerger, to use PEP 585 built-in generics: replace Dict with dict, List with list, and Optional[T] with T | None; remove the corresponding typing imports while preserving all annotation semantics.
211-316: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
merge()reuses staleself.kvif called again withoutkv_csv_dir.
self.kvis only overwrittenif kv_csv_dir:(line 228-229); a secondmerge()call on the same instance withoutkv_csv_dirwill silently keep joining against the previous run's KV data. Not an issue for the current single-call CLI usage (main()callsmerge()once), but worth a docstring note or an explicit reset if the class is intended to be reused across merges.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.py` around lines 211 - 316, Update merge() so each invocation clears or reinitializes self.kv when kv_csv_dir is not provided, preventing joins from using KV data loaded by a previous call. Preserve loading parsed KV data when kv_csv_dir is supplied and keep the existing task_events, gen_summary, and cpp_events processing unchanged.
329-344: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winFactor out the timing-row build into
RequestTimeBreakdown.write_html()duplicates the parse/duration loop and ties this path toparser/config.metrics; a small public helper would keep the--htmlflow on one contract and reduce drift.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.py` around lines 329 - 344, Move the timing-row construction loop from write_html into a small public helper on RequestTimeBreakdown that parses each record and adds all configured metric durations. Update write_html to call that helper and pass its result to create_timing_diagram, removing direct parser/config.metrics access from write_html.tests/unittest/others/test_perf_time_events.py (1)
1-213: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueTest coverage summary.
Added tests:
test_globs_and_concatenates,test_skips_malformed_lines(parse_event_dir);test_native_task_and_gen_summary,test_cpp_send_recv_keyed_on_request_id(parse_kv_csv_dir);test_join_by_request_and_ctx_id,test_unjoined_kv_events_reported,test_derived_inter_step_gaps_and_starved,test_write_combined_json,test_merge_perf_json_input(PerfTimeEventsMerger).This file lives under
tests/unittest/others/, nottests/integration/, so it is collected directly by the pytest unittest run rather than throughtests/integration/test_lists/{test-db,qa}entries — no test-list additions are needed for this file.Coverage verdict: sufficient for the parse/merge/JSON path exercised here (globbing, malformed-line skipping, native/C++ CSV parsing, request/ctx-id joins, unjoined reporting, derived gaps/starvation, JSON round-trip, perf-json input). Gaps worth noting:
write_html()is untested (reasonable given the GPU-free/torch-free constraint) andmain()'s argument-parsing/error path (Line 383-384 inperf_time_events.py) has no test. Also,_NATIVE_TASK_HEADER/_GEN_SUMMARY_HEADERhere are independently hand-copied from the same source the module's own copies claim to mirror (_torch/disaggregation/native/perf_logger.py) rather than shared/imported, so this test only catches drift between the module and the test, not real drift against the actual producer.As per path instructions, "Always produce a test coverage summary, even if no issues are found."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/others/test_perf_time_events.py` around lines 1 - 213, Strengthen the CSV header tests in TestParseKvCsvDir by deriving or validating _NATIVE_TASK_HEADER and _GEN_SUMMARY_HEADER against the actual producer definitions in perf_logger.py, rather than relying on independently copied literals. Preserve the existing native task and generation summary parsing assertions while ensuring producer-header drift causes a test failure.Source: Path instructions
tests/unittest/_torch/executor/test_perf_metrics_events.py (1)
1-249: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winTest coverage summary.
Added
TestAppendStepMetricsExtended,TestComputeIterBatchContext, andTestPerRankWriter, covering the new extended-capture merge logic,_compute_iter_batch_context, and the JSONL writer's enqueue/no-op gating. This is a new file undertests/unittest/...and does not require entries intests/integration/test_lists/(that path gates integration tests, not this unit suite). Coverage verdict: sufficient for the happy-path behavior exercised here; consider adding a regression test that simulates a writer-thread failure (e.g. patchopen()to raise inside_writer_loop) and assertsclose()returns promptly instead of blocking, covering the hang risk flagged inperf_metrics_manager.py.As per path instructions, "Act as a QA engineer reviewing test changes and coverage for TensorRT-LLM" and "Always produce a test coverage summary, even if no issues are found."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/_torch/executor/test_perf_metrics_events.py` around lines 1 - 249, Add a regression test in TestPerRankWriter that forces open() to fail inside PerfMetricsManager._writer_loop, enqueues a request event, and verifies PerfMetricsManager.close() returns promptly without hanging. Ensure the test exercises the writer-thread failure path and uses a bounded wait or timeout while preserving cleanup of the manager.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tensorrt_llm/_torch/pyexecutor/perf_metrics_manager.py`:
- Around line 415-450: Update close() to enqueue _WRITER_STOP with a bounded or
non-blocking operation so shutdown cannot hang when _writer_loop has exited or
the queue is full. Replace the broad except Exception with handling for the
specific queue exception(s), and log enqueue failures while preserving the
existing thread join and cleanup behavior.
- Around line 322-340: Update the per-request generation metric in the extended
metrics block to use the same draft-token source as _compute_iter_batch_context,
including its getattr(req, "num_draft_tokens", 0) fallback, instead of
get_draft_token_length(request). Ensure req_generation_token_number remains 1
plus that shared draft-token value so batch and per-request totals stay
consistent.
In `@tensorrt_llm/serve/openai_server.py`:
- Around line 590-603: Move the TRTLLM_PERF_TIME_EVENTS_PATH-based
initialization of self.perf_metrics and self.perf_metrics_lock outside the
return_perf_metrics conditional in the surrounding initialization flow. Ensure
the fallback runs whenever the environment variable is set and self.perf_metrics
is None, so /perf_metrics has a live deque even when
generator.args.return_perf_metrics is false.
In `@tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.py`:
- Around line 108-177: The parser duplicates producer CSV header definitions
through _NATIVE_TASK_HEADER and _GEN_SUMMARY_HEADER, allowing header changes to
desynchronize routing. Update parse_kv_csv_dir to import and reuse the
producer’s shared header constants, or move both definitions into a shared
module consumed by the parser and writer, while preserving the existing native
task and generation-summary classification.
---
Nitpick comments:
In `@tensorrt_llm/serve/scripts/perf_time_events/__init__.py`:
- Around line 26-33: Sort the entries in __all__ alphabetically to satisfy Ruff
RUF022, while preserving the existing exported symbols from the perf_time_events
package.
In `@tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.py`:
- Line 1: Replace the locally duplicated _NATIVE_TASK_HEADER and
_GEN_SUMMARY_HEADER declarations in perf_time_events.py with imports from
_torch/disaggregation/native/perf_logger.py or a shared constants module. Update
test_perf_time_events.py to use those same producer-owned constants, ensuring
parser expectations and tests detect future header changes.
- Around line 205-209: Update the __init__ method signature to include an
explicit -> None return annotation, leaving its existing initialization logic
unchanged.
- Around line 347-390: Annotate the CLI entry-point function main with an
explicit -> None return type, preserving its existing argument parsing,
validation, merge, and output sequencing.
- Around line 61-71: The generator function _iter_jsonl lacks a return type
annotation. Import Iterator from collections.abc and annotate _iter_jsonl’s
return type as Iterator using the appropriate parsed JSON value type, while
preserving its existing behavior.
- Around line 30-36: Update type annotations throughout perf_time_events.py,
including parse_event_dir, parse_kv_csv_dir, and PerfTimeEventsMerger, to use
PEP 585 built-in generics: replace Dict with dict, List with list, and
Optional[T] with T | None; remove the corresponding typing imports while
preserving all annotation semantics.
- Around line 211-316: Update merge() so each invocation clears or reinitializes
self.kv when kv_csv_dir is not provided, preventing joins from using KV data
loaded by a previous call. Preserve loading parsed KV data when kv_csv_dir is
supplied and keep the existing task_events, gen_summary, and cpp_events
processing unchanged.
- Around line 329-344: Move the timing-row construction loop from write_html
into a small public helper on RequestTimeBreakdown that parses each record and
adds all configured metric durations. Update write_html to call that helper and
pass its result to create_timing_diagram, removing direct parser/config.metrics
access from write_html.
In `@tests/unittest/_torch/executor/test_perf_metrics_events.py`:
- Around line 1-249: Add a regression test in TestPerRankWriter that forces
open() to fail inside PerfMetricsManager._writer_loop, enqueues a request event,
and verifies PerfMetricsManager.close() returns promptly without hanging. Ensure
the test exercises the writer-thread failure path and uses a bounded wait or
timeout while preserving cleanup of the manager.
In `@tests/unittest/others/test_perf_time_events.py`:
- Around line 1-213: Strengthen the CSV header tests in TestParseKvCsvDir by
deriving or validating _NATIVE_TASK_HEADER and _GEN_SUMMARY_HEADER against the
actual producer definitions in perf_logger.py, rather than relying on
independently copied literals. Preserve the existing native task and generation
summary parsing assertions while ensuring producer-header drift causes a test
failure.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 9fffc183-8f40-4fb5-9cea-64293732be28
📒 Files selected for processing (12)
docs/source/developer-guide/perf-analysis.mdtensorrt_llm/_torch/pyexecutor/llm_request.pytensorrt_llm/_torch/pyexecutor/perf_metrics_manager.pytensorrt_llm/_torch/pyexecutor/py_executor.pytensorrt_llm/serve/openai_server.pytensorrt_llm/serve/responses_utils.pytensorrt_llm/serve/scripts/perf_time_events/README.mdtensorrt_llm/serve/scripts/perf_time_events/__init__.pytensorrt_llm/serve/scripts/perf_time_events/__main__.pytensorrt_llm/serve/scripts/perf_time_events/perf_time_events.pytests/unittest/_torch/executor/test_perf_metrics_events.pytests/unittest/others/test_perf_time_events.py
| # Extended per-iteration batch context (only when capture_extended is | ||
| # on and the executor stashed a context dict for this iteration). | ||
| # The batch-level fields ride inside each per-iteration metric dict, so | ||
| # they flow through create_response's existing .copy() into | ||
| # time_breakdown_metrics with no payload-shape change. | ||
| if self.capture_extended and perf.iter_batch_context is not None: | ||
| metric.update(perf.iter_batch_context) | ||
| if is_ctx: | ||
| # Tokens processed for THIS request in THIS ctx chunk. | ||
| try: | ||
| metric["req_context_token_number"] = request.context_chunk_size | ||
| except RuntimeError: | ||
| last_chunk = getattr(request, "py_last_context_chunk", None) | ||
| if last_chunk is not None and last_chunk[0] is not None: | ||
| metric["req_context_token_number"] = last_chunk[1] - last_chunk[0] | ||
| else: | ||
| # Tokens this request emits this gen step (1 + speculative draft). | ||
| metric["req_generation_token_number"] = 1 + get_draft_token_length(request) | ||
|
|
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Compare definitions/usages of num_draft_tokens vs py_draft_tokens length
ast-grep run --pattern 'num_draft_tokens' --lang python tensorrt_llm/_torch/pyexecutor | head -50
rg -n -C2 'num_draft_tokens\b' tensorrt_llm/_torch/pyexecutor/llm_request.pyRepository: NVIDIA/TensorRT-LLM
Length of output: 3576
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the relevant symbols before reading targeted slices.
ast-grep outline tensorrt_llm/_torch/pyexecutor/perf_metrics_manager.py --view expanded | sed -n '1,220p'
printf '\n--- PY_EXECUTOR ---\n'
ast-grep outline tensorrt_llm/_torch/pyexecutor/py_executor.py --view expanded | sed -n '1,260p'
printf '\n--- LLM_REQUEST ---\n'
ast-grep outline tensorrt_llm/_torch/pyexecutor/llm_request.py --view expanded | sed -n '1,220p'
printf '\n--- SEARCH get_draft_token_length ---\n'
rg -n -C3 'def get_draft_token_length|get_draft_token_length\(' tensorrt_llm/_torch/pyexecutor
printf '\n--- SEARCH py_draft_tokens / num_draft_tokens ---\n'
rg -n -C3 '\bpy_draft_tokens\b|\bnum_draft_tokens\b' tensorrt_llm/_torch/pyexecutor/llm_request.py tensorrt_llm/_torch/pyexecutor/py_executor.py tensorrt_llm/_torch/pyexecutor/model_engine.pyRepository: NVIDIA/TensorRT-LLM
Length of output: 50375
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Read the exact batch-context and draft-token plumbing.
sed -n '1888,1948p' tensorrt_llm/_torch/pyexecutor/py_executor.py
printf '\n---\n'
sed -n '1223,1246p' tensorrt_llm/_torch/pyexecutor/llm_request.py
printf '\n---\n'
sed -n '732,748p' tensorrt_llm/_torch/pyexecutor/llm_request.py
printf '\n---\n'
sed -n '2100,2130p' tensorrt_llm/_torch/pyexecutor/py_executor.py
printf '\n---\n'
sed -n '2278,2292p' tensorrt_llm/_torch/pyexecutor/py_executor.py
printf '\n---\n'
sed -n '6838,6846p' tensorrt_llm/_torch/pyexecutor/py_executor.pyRepository: NVIDIA/TensorRT-LLM
Length of output: 6983
Use the same draft-token source for both generation metrics. _compute_iter_batch_context sums getattr(req, "num_draft_tokens", 0), while the per-request metric uses get_draft_token_length(request) (py_draft_tokens). That can make the batch total drift from the sum of request values; prefer the same helper/fallback in both places.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tensorrt_llm/_torch/pyexecutor/perf_metrics_manager.py` around lines 322 -
340, Update the per-request generation metric in the extended metrics block to
use the same draft-token source as _compute_iter_batch_context, including its
getattr(req, "num_draft_tokens", 0) fallback, instead of
get_draft_token_length(request). Ensure req_generation_token_number remains 1
plus that shared draft-token value so batch and per-request totals stay
consistent.
| def _writer_loop(self, rank: int) -> None: | ||
| """Daemon loop: drain the queue and append one JSON line per record.""" | ||
| path = os.path.join( | ||
| self._events_dir, | ||
| f"time_events_rank{rank}_pid{os.getpid()}.jsonl", | ||
| ) | ||
| try: | ||
| os.makedirs(self._events_dir, exist_ok=True) | ||
| self._events_file = open(path, "a") | ||
| except OSError as e: | ||
| logger.warning("Failed to open perf time-events file %s: %s", path, e) | ||
| return | ||
| while True: | ||
| record = self._writer_queue.get() | ||
| if record is _WRITER_STOP: | ||
| break | ||
| try: | ||
| self._events_file.write(json.dumps(record) + "\n") | ||
| self._events_file.flush() | ||
| except (OSError, TypeError) as e: | ||
| logger.warning("Failed to write perf time-event record: %s", e) | ||
| try: | ||
| self._events_file.close() | ||
| except OSError: | ||
| pass | ||
|
|
||
| def close(self) -> None: | ||
| """Flush and stop the writer thread; safe to call when disabled.""" | ||
| if self._writer_thread is None: | ||
| return | ||
| try: | ||
| self._writer_queue.put(_WRITER_STOP) | ||
| except Exception: # noqa: BLE001 - best-effort on teardown | ||
| pass | ||
| self._writer_thread.join(timeout=30) | ||
| self._writer_thread = None |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
close() can hang shutdown indefinitely if the writer thread dies or the queue fills.
self._writer_queue.put(_WRITER_STOP) at Line 446 is a blocking put() with no timeout. If _writer_loop failed to open the events file (Line 421-426, OSError) or otherwise stopped draining, the thread exits but _writer_thread stays non-None. Once the bounded queue (maxsize=100000) fills from continued put_nowait calls, this blocking put() in close() will wait forever — the surrounding except Exception: pass does not help since blocking is not an exception. This is invoked from PyExecutor.shutdown() (py_executor.py Line 1485), so a stuck writer can hang the entire executor shutdown.
Separately, Line 447-448's except Exception: pass swallows all errors without logging (flagged by static analysis) and is broader than necessary.
As per coding guidelines, "Catch specific exceptions rather than using broad or bare exception handling such as except:."
🔒 Proposed fix: bound the shutdown wait and log failures
def close(self) -> None:
"""Flush and stop the writer thread; safe to call when disabled."""
if self._writer_thread is None:
return
try:
- self._writer_queue.put(_WRITER_STOP)
- except Exception: # noqa: BLE001 - best-effort on teardown
- pass
+ self._writer_queue.put(_WRITER_STOP, timeout=5)
+ except queue.Full:
+ logger.warning(
+ "perf time-events writer queue still full at shutdown; "
+ "abandoning drain and proceeding with teardown")
self._writer_thread.join(timeout=30)
self._writer_thread = None📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def _writer_loop(self, rank: int) -> None: | |
| """Daemon loop: drain the queue and append one JSON line per record.""" | |
| path = os.path.join( | |
| self._events_dir, | |
| f"time_events_rank{rank}_pid{os.getpid()}.jsonl", | |
| ) | |
| try: | |
| os.makedirs(self._events_dir, exist_ok=True) | |
| self._events_file = open(path, "a") | |
| except OSError as e: | |
| logger.warning("Failed to open perf time-events file %s: %s", path, e) | |
| return | |
| while True: | |
| record = self._writer_queue.get() | |
| if record is _WRITER_STOP: | |
| break | |
| try: | |
| self._events_file.write(json.dumps(record) + "\n") | |
| self._events_file.flush() | |
| except (OSError, TypeError) as e: | |
| logger.warning("Failed to write perf time-event record: %s", e) | |
| try: | |
| self._events_file.close() | |
| except OSError: | |
| pass | |
| def close(self) -> None: | |
| """Flush and stop the writer thread; safe to call when disabled.""" | |
| if self._writer_thread is None: | |
| return | |
| try: | |
| self._writer_queue.put(_WRITER_STOP) | |
| except Exception: # noqa: BLE001 - best-effort on teardown | |
| pass | |
| self._writer_thread.join(timeout=30) | |
| self._writer_thread = None | |
| def _writer_loop(self, rank: int) -> None: | |
| """Daemon loop: drain the queue and append one JSON line per record.""" | |
| path = os.path.join( | |
| self._events_dir, | |
| f"time_events_rank{rank}_pid{os.getpid()}.jsonl", | |
| ) | |
| try: | |
| os.makedirs(self._events_dir, exist_ok=True) | |
| self._events_file = open(path, "a") | |
| except OSError as e: | |
| logger.warning("Failed to open perf time-events file %s: %s", path, e) | |
| return | |
| while True: | |
| record = self._writer_queue.get() | |
| if record is _WRITER_STOP: | |
| break | |
| try: | |
| self._events_file.write(json.dumps(record) + "\n") | |
| self._events_file.flush() | |
| except (OSError, TypeError) as e: | |
| logger.warning("Failed to write perf time-event record: %s", e) | |
| try: | |
| self._events_file.close() | |
| except OSError: | |
| pass | |
| def close(self) -> None: | |
| """Flush and stop the writer thread; safe to call when disabled.""" | |
| if self._writer_thread is None: | |
| return | |
| try: | |
| self._writer_queue.put(_WRITER_STOP, timeout=5) | |
| except queue.Full: | |
| logger.warning( | |
| "perf time-events writer queue still full at shutdown; " | |
| "abandoning drain and proceeding with teardown") | |
| self._writer_thread.join(timeout=30) | |
| self._writer_thread = None |
🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 422-422: File path is request-/variable-derived; validate and normalize to prevent path traversal.
Context: open(path, "a")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(open-filename-from-request)
[info] 431-431: use jsonify instead of json.dumps for JSON output
Context: json.dumps(record)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
🪛 Ruff (0.15.21)
[error] 447-448: try-except-pass detected, consider logging the exception
(S110)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tensorrt_llm/_torch/pyexecutor/perf_metrics_manager.py` around lines 415 -
450, Update close() to enqueue _WRITER_STOP with a bounded or non-blocking
operation so shutdown cannot hang when _writer_loop has exited or the queue is
full. Replace the broad except Exception with handling for the specific queue
exception(s), and log enqueue failures while preserving the existing thread join
and cleanup behavior.
Sources: Coding guidelines, Linters/SAST tools
| self.perf_metrics_maxlen = max_perf_metrics | ||
| self.perf_metrics = deque(maxlen=max_perf_metrics) | ||
| self.perf_metrics_lock = asyncio.Lock() | ||
| # TRTLLM_PERF_TIME_EVENTS_PATH also makes the HTTP /perf_metrics | ||
| # endpoint usable for single-server (aggregated) runs, even when | ||
| # return_perf_metrics / perf_metrics_max_requests were not set. The | ||
| # per-rank live files remain the primary artifact; this is | ||
| # convenience. | ||
| if (self.perf_metrics is None | ||
| and len(os.getenv("TRTLLM_PERF_TIME_EVENTS_PATH", "")) > 0): | ||
| self.perf_metrics_maxlen = ( | ||
| self.generator.args.perf_metrics_max_requests or 65536) | ||
| self.perf_metrics = deque(maxlen=self.perf_metrics_maxlen) | ||
| self.perf_metrics_lock = asyncio.Lock() |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
rg -n 'TRTLLM_PERF_TIME_EVENTS_PATH' --type=py -C3
rg -n 'return_perf_metrics\s*=' --type=py -C2 -g '!**/test*'Repository: NVIDIA/TensorRT-LLM
Length of output: 157
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate the relevant server code and nearby argument wiring.
fd -a 'openai_server.py' .
fd -a 'args.py' .
fd -a 'llm_args.py' .
fd -a 'config.py' .
# Find all references to the env var and perf-metrics flags.
rg -n 'TRTLLM_PERF_TIME_EVENTS_PATH|return_perf_metrics|perf_metrics_max_requests|perf_metrics' tensorrt_llm -g '*.py' -C 2
# Show the relevant section of openai_server.py with line numbers.
python3 - <<'PY'
from pathlib import Path
p = Path('tensorrt_llm/serve/openai_server.py')
text = p.read_text().splitlines()
for i in range(540, 620):
if i <= len(text):
print(f"{i:4d}: {text[i-1]}")
PYRepository: NVIDIA/TensorRT-LLM
Length of output: 50375
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== relevant files =="
git ls-files | rg 'openai_server\.py|llm_args|args\.py|config\.py|server' || true
echo
echo "== perf-metrics references =="
rg -n 'TRTLLM_PERF_TIME_EVENTS_PATH|return_perf_metrics|perf_metrics_max_requests|perf_metrics' tensorrt_llm -g '*.py' -C 3 || true
echo
echo "== openai_server.py excerpt =="
python3 - <<'PY'
from pathlib import Path
p = Path('tensorrt_llm/serve/openai_server.py')
text = p.read_text().splitlines()
for i in range(550, 615):
if i <= len(text):
print(f"{i:4d}: {text[i-1]}")
PYRepository: NVIDIA/TensorRT-LLM
Length of output: 50375
Move the env-var fallback out of the return_perf_metrics gate TRTLLM_PERF_TIME_EVENTS_PATH still won’t populate self.perf_metrics unless self.generator.args.return_perf_metrics is already true. The request path can set sampling_params.return_perf_metrics = True, but that only affects per-request metrics; /perf_metrics still returns [] when the deque is never initialized.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tensorrt_llm/serve/openai_server.py` around lines 590 - 603, Move the
TRTLLM_PERF_TIME_EVENTS_PATH-based initialization of self.perf_metrics and
self.perf_metrics_lock outside the return_perf_metrics conditional in the
surrounding initialization flow. Ensure the fallback runs whenever the
environment variable is set and self.perf_metrics is None, so /perf_metrics has
a live deque even when generator.args.return_perf_metrics is false.
| def parse_kv_csv_dir(kv_csv_dir: str) -> Dict[str, Any]: | ||
| """Parse KV-transfer CSVs into join-ready structures. | ||
|
|
||
| Handles two producers: | ||
|
|
||
| * Python-native transceiver (``perf_logger.py``): | ||
| - ``{instance}_{rank}.csv`` -> task rows keyed by ``unique_rid`` | ||
| (KVSendTask / AuxSendTask / KVRecvTask; recv rows leave the middle | ||
| latency fields blank). | ||
| - ``{instance}_{rank}_gen_transfer_summary.csv`` -> gen summary keyed by | ||
| ``RequestID``. | ||
| * C++ transceiver: ``*_send.csv`` / ``*_recv.csv`` keyed by ctx-phase | ||
| ``RequestID`` (parsed for completeness so mixed dirs still aggregate). | ||
|
|
||
| Returns a dict with ``task_events`` (rid -> list[row]), ``gen_summary`` | ||
| (rid -> row) and ``cpp_events`` (rid -> list[row]). | ||
| """ | ||
| task_events: Dict[str, List[Dict[str, Any]]] = defaultdict(list) | ||
| gen_summary: Dict[str, Dict[str, Any]] = {} | ||
| cpp_events: Dict[str, List[Dict[str, Any]]] = defaultdict(list) | ||
|
|
||
| csv_files = sorted(glob.glob(os.path.join(kv_csv_dir, "*.csv"))) | ||
| for path in csv_files: | ||
| base = os.path.basename(path) | ||
| try: | ||
| with open(path, "r", newline="") as f: | ||
| reader = csv.reader(f) | ||
| try: | ||
| header = next(reader) | ||
| except StopIteration: | ||
| continue | ||
| header = [h.strip() for h in header] | ||
| rows = [row for row in reader if row] | ||
| except OSError as e: | ||
| print(f"WARNING: failed to read KV CSV {path}: {e}") | ||
| continue | ||
|
|
||
| if base.endswith("_gen_transfer_summary.csv") or header == _GEN_SUMMARY_HEADER: | ||
| for row in rows: | ||
| rec = dict(zip(header, row)) | ||
| rid = rec.get("RequestID") | ||
| if rid is not None: | ||
| rec["_source_file"] = base | ||
| gen_summary[str(rid)] = rec | ||
| elif header == _NATIVE_TASK_HEADER: | ||
| for row in rows: | ||
| rec = dict(zip(header, row)) | ||
| rid = rec.get("unique_rid") | ||
| if rid is not None: | ||
| rec["_source_file"] = base | ||
| task_events[str(rid)].append(rec) | ||
| else: | ||
| # C++ (or unknown) layout: try to key on a RequestID-like column. | ||
| rid_key = None | ||
| for candidate in ("RequestID", "request_id", "unique_rid"): | ||
| if candidate in header: | ||
| rid_key = candidate | ||
| break | ||
| for row in rows: | ||
| rec = dict(zip(header, row)) | ||
| rec["_source_file"] = base | ||
| rid = rec.get(rid_key) if rid_key else None | ||
| if rid is not None: | ||
| cpp_events[str(rid)].append(rec) | ||
|
|
||
| return { | ||
| "task_events": dict(task_events), | ||
| "gen_summary": gen_summary, | ||
| "cpp_events": dict(cpp_events), | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
rg -n 'timestamp,task_type,unique_rid|_NATIVE_TASK_HEADER|_GEN_SUMMARY_HEADER' -g '*.py' tensorrt_llm/_torch/disaggregationRepository: NVIDIA/TensorRT-LLM
Length of output: 562
🏁 Script executed:
#!/bin/bash
sed -n '1,240p' tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.py
printf '\n--- PRODUCER ---\n'
sed -n '1,460p' tensorrt_llm/_torch/disaggregation/native/perf_logger.pyRepository: NVIDIA/TensorRT-LLM
Length of output: 25791
Share the KV CSV headers with the producer
_NATIVE_TASK_HEADER and _GEN_SUMMARY_HEADER are duplicated literals here, so a producer-side header change will silently route native rows into the generic C++/unknown path. Import the producer constants or define them once in a shared module to keep the parser and writer in sync.
🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 132-132: File path is request-/variable-derived; validate and normalize to prevent path traversal.
Context: open(path, "r", newline="")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(open-filename-from-request)
🪛 Ruff (0.15.21)
[warning] 147-147: zip() without an explicit strict= parameter
Add explicit value for parameter strict=
(B905)
[warning] 154-154: zip() without an explicit strict= parameter
Add explicit value for parameter strict=
(B905)
[warning] 167-167: zip() without an explicit strict= parameter
Add explicit value for parameter strict=
(B905)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.py` around lines
108 - 177, The parser duplicates producer CSV header definitions through
_NATIVE_TASK_HEADER and _GEN_SUMMARY_HEADER, allowing header changes to
desynchronize routing. Update parse_kv_csv_dir to import and reuse the
producer’s shared header constants, or move both definitions into a shared
module consumed by the parser and writer, while preserving the existing native
task and generation-summary classification.
GPU end-to-end verification (OCI-HSG GB200)Posting the promised follow-up. Workload:
Method — single-wheel A/B. ONE instrumented wheel is used for both arms; only H2 — capture works ✅Each instrumented rep wrote 3 per-rank files (12 files / 4 reps, 4108 records,
All 7 extended keys present ( H3 — switch-off is dormant ✅Every baseline rep (env unset) wrote 0 H2b — overhead when the switch is ON: ≈4% (opt-in only)Two independent signals agree, with zero overlap between arms:
So enabling the switch costs ≈4% on this high-step-count decode workload — real Attribution (verified in source) — the ≈4% is the pre-existing per-iteration
(Gen-rank files are ~131 MB each — 256 req × 1023 per-iteration Summary: H2 PASS · H3 PASS · H2b ≈+4% (opt-in capture cost, off-by-default). |
…request lifecycle scalars
Builds on the per-rank live capture (TRTLLM_PERF_TIME_EVENTS_PATH) by adding
the timing sources the executor cannot see, so a perf timeline can span
client-send -> router-dispatch -> worker admission/decode.
Item 4 (executor): enrich each per-rank JSONL record with the C++
RequestPerfMetrics lifecycle scalars (arrival / first_scheduled / first_token /
last_token / kv_cache_transfer start+end / kv_cache_size) via
executor.result.get_metrics_dict. Populated on the serve/disagg path (the
entrypoints force return_perf_metrics under the env); best-effort otherwise
(empty dict -> key omitted, never crashes the writer). These steady-clock
anchors give per-request admission/prefill/decode boundaries the relative
per-iteration time_breakdown_metrics cannot express.
Item 1 (client + router): two new optional capture files, each written by a
stdlib-only, torch-free writer (atexit flush) so the GPU-free import contract
holds:
* TRTLLM_PERF_TIME_EVENTS_ROUTER_PATH -- the disagg router stamps
arrival / ctx_dispatch / gen_dispatch / first_token / resp_done
(steady-clock, same epoch as the workers) keyed by ctx_request_id; gen-only
requests with no ctx id are collected separately.
* TRTLLM_PERF_TIME_EVENTS_CLIENT_PATH -- the benchmark client stamps
send/first_token/completion (perf_counter) + send_wall_time (time.time).
Standalone timeline: response_id is a fresh UUID, not a worker join key.
Aggregator: --router-dir / --client-dir (default to the two new env vars) join
the router record onto each worker record by ctx_request_id (exposed as
router_dispatch) and derive the cross-process spans
router_arrival_to_ctx_dispatch / router_ctx_to_gen_dispatch /
router_to_worker_arrival, plus the worker-side arrival_to_first_schedule /
schedule_to_first_token / decode_duration from request_timing_metrics.
Unmatched router rows (incl. gen-only) surface under unjoined_router_events;
client rows pass through untouched under client_events. Any one input suffices.
Tests (GPU-free): stdlib JsonlEventWriter contract (inert/drain/default=str/
idempotent); router+client parsers; enriched merge (join + all derived spans +
unjoined + standalone client payload); worker request_timing_metrics enrichment
(populated / empty / exception-swallowed).
Docs: document the two env vars, the --router-dir/--client-dir flags, the
lifecycle-scalar enrichment, and the standalone-client / shared-steady-clock
caveats in perf-analysis.md.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
|
Pushed Item 4 — executor per-request lifecycle scalars. Each per-rank JSONL record is enriched with the C++ Item 1 — client + disagg-router capture files (two new optional env vars, each written by a stdlib-only, torch-free writer with
Aggregator — Tests (GPU-free): stdlib writer contract, router+client parsers, enriched merge (join + all derived spans + unjoined + standalone client), and worker enrichment (populated / empty / exception-swallowed). Docs updated in |
There was a problem hiding this comment.
Actionable comments posted: 7
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tensorrt_llm/serve/openai_disagg_service.py (1)
386-412: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
on_gen_dispatchhook is not called on the generation-first schedule path.
_send_disagg_request_gen_firstcallshooks.on_ctx_dispatch(request)but neverhooks.on_gen_dispatch(...)before dispatching toself._gen_client.send_request(...). As a result,RawRequestResponseHooks.gen_dispatch_timestays0(→Nonein the emitted record) for any deployment usingschedule_style: generation_first, silently degrading the new router dispatch-timeline for that mode. Since the ctx-first path calls the hook right before gen-server resolution, consider adding an equivalent call here (e.g. right before line 403'sget_next_server/dispatch) for parity.🛠️ Proposed fix
disagg_request_id = await self._coordinator.get_disagg_request_id() if need_ctx: # arrival->here = pre-ctx wait in the orchestrator/fleet. if hooks: hooks.on_ctx_dispatch(request) ctx_server, ctx_server_info = await self._ctx_router.get_next_server( request, req_id=disagg_request_id ) ctx_req = self._get_ctx_request(request, disagg_request_id) + # Mark gen-dispatch start for the gen-first schedule style too. + if hooks: + hooks.on_gen_dispatch(request) gen_req = self._get_gen_request(🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/serve/openai_disagg_service.py` around lines 386 - 412, Update _send_disagg_request_gen_first to invoke hooks.on_gen_dispatch(request) before generation-server resolution or dispatch, matching the ctx-first path and ensuring generation-first requests record gen dispatch timing. Keep the existing on_ctx_dispatch call for context routing unchanged.
🧹 Nitpick comments (1)
tests/unittest/others/test_perf_time_events.py (1)
283-332: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAdd a negative case where
request_idcollides with an unrelatedctx_request_id.This test uses a string ctx id (
"ctxA") and an intrequest_id(10), so it can never exercise the router join'srequest_idfallback (flagged onperf_time_events.pyLine 333). A record withrequest_id=10and a router row keyed"10"for a different request would currently join and emit bogusrouter_*spans.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/others/test_perf_time_events.py` around lines 283 - 332, Extend test_router_join_on_ctx_id_and_derived_spans with a negative case where request_id 10 conflicts with an unrelated router row keyed by "10" while ctx_request_id remains different. Assert the router record is not joined, no bogus router_* derived spans are emitted, and the unrelated router event remains unjoined, covering the request_id fallback without changing the existing ctx_request_id match assertions.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tensorrt_llm/serve/openai_disagg_server.py`:
- Around line 115-143: Preserve ctx_request_id as None in
_maybe_write_router_event when no ctx round-trip occurred; do not substitute
disagg_request_id for it. Keep the existing disagg_request_id field for router
dispatch identity so parse_router_dir() continues routing gen-only records to
_no_ctx.
In `@tensorrt_llm/serve/perf_time_events_writer.py`:
- Around line 73-106: Ensure the `_writer_loop` OSError path transitions the
writer to an inert state when directory creation or file opening fails. Update
`write` and the writer-state handling so records are no longer enqueued after
that failure, and ensure `_writer_thread` cannot remain represented as an
active/draining writer. Preserve the existing warning while making the failure
immediately prevent further queue growth.
In `@tensorrt_llm/serve/scripts/backend_request_func.py`:
- Around line 71-82: Update async_request_openai_chat_completions to assign
send_time and send_wall_time at the same POST-send point as
async_request_openai_completions, and capture the returned response ID into
response_id. Preserve the existing timestamp sources and response-ID extraction
behavior so openai-chat perf-time-events records contain the same client
timeline data.
In `@tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.py`:
- Around line 333-341: Update the router dispatch join around router_by_ctx to
look up only the worker record’s ctx_request_id, not both values in rids. Remove
the request_id-first iteration and retain matched_router_ctx tracking and
router_dispatch assignment only when the ctx-key lookup succeeds.
- Around line 105-124: Update the relevant function’s docstring to document the
actual "_no_ctx" sentinel used by merge(), and widen its return annotation/value
type to represent both per-context record dictionaries and the no-context record
list. Remove the unnecessary type: ignore on the "_no_ctx" assignment while
preserving the existing grouping behavior.
In `@tests/unittest/others/test_perf_time_events_writer.py`:
- Around line 30-95: Add tests/unittest/others/test_perf_time_events_writer.py
to the appropriate integration test-list entries under
tests/integration/test_lists/test-db/ and qa/ so all six TestJsonlEventWriter
and TestMakeEnvWriter tests are discovered by integration coverage.
In `@tests/unittest/others/test_perf_time_events.py`:
- Around line 364-395: Add tests/unittest/others/test_perf_time_events.py to the
appropriate CI test-list entry under the existing database or QA test-list
configuration, ensuring the listed test functions and helper coverage run
automatically. Do not modify the test implementations.
---
Outside diff comments:
In `@tensorrt_llm/serve/openai_disagg_service.py`:
- Around line 386-412: Update _send_disagg_request_gen_first to invoke
hooks.on_gen_dispatch(request) before generation-server resolution or dispatch,
matching the ctx-first path and ensuring generation-first requests record gen
dispatch timing. Keep the existing on_ctx_dispatch call for context routing
unchanged.
---
Nitpick comments:
In `@tests/unittest/others/test_perf_time_events.py`:
- Around line 283-332: Extend test_router_join_on_ctx_id_and_derived_spans with
a negative case where request_id 10 conflicts with an unrelated router row keyed
by "10" while ctx_request_id remains different. Assert the router record is not
joined, no bogus router_* derived spans are emitted, and the unrelated router
event remains unjoined, covering the request_id fallback without changing the
existing ctx_request_id match assertions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: a7cf282c-c07e-4a9b-acb4-669c1bb75863
📒 Files selected for processing (14)
docs/source/developer-guide/perf-analysis.mdtensorrt_llm/_torch/pyexecutor/perf_metrics_manager.pytensorrt_llm/_torch/pyexecutor/py_executor.pytensorrt_llm/serve/openai_disagg_server.pytensorrt_llm/serve/openai_disagg_service.pytensorrt_llm/serve/perf_time_events_writer.pytensorrt_llm/serve/responses_utils.pytensorrt_llm/serve/scripts/backend_request_func.pytensorrt_llm/serve/scripts/benchmark_serving.pytensorrt_llm/serve/scripts/perf_time_events/__init__.pytensorrt_llm/serve/scripts/perf_time_events/perf_time_events.pytests/unittest/_torch/executor/test_perf_metrics_events.pytests/unittest/others/test_perf_time_events.pytests/unittest/others/test_perf_time_events_writer.py
🚧 Files skipped from review as they are similar to previous changes (4)
- tensorrt_llm/serve/scripts/perf_time_events/init.py
- docs/source/developer-guide/perf-analysis.md
- tensorrt_llm/_torch/pyexecutor/py_executor.py
- tensorrt_llm/_torch/pyexecutor/perf_metrics_manager.py
| self._maybe_write_router_event(gen_server, request) | ||
|
|
||
| def _maybe_write_router_event(self, gen_server: str, request: UCompletionRequest): | ||
| """Emit one dispatch-timeline JSONL record for this request (best-effort). | ||
|
|
||
| No-op unless TRTLLM_PERF_TIME_EVENTS_ROUTER_PATH is set. Keyed on BOTH the | ||
| router's ``disagg_request_id`` (its own dispatch timeline) and the | ||
| worker-assigned ``ctx_request_id`` (cross-process join key with the | ||
| per-rank worker files). ``ctx_request_id`` is only known after the ctx | ||
| round-trip; it is ``None`` on the gen-only / no-ctx path. | ||
| """ | ||
| if not _ROUTER_EVENTS_WRITER.enabled: | ||
| return | ||
| disagg_params = getattr(request, "disaggregated_params", None) | ||
| ctx_request_id = getattr(disagg_params, "ctx_request_id", None) | ||
| disagg_request_id = getattr(disagg_params, "disagg_request_id", None) | ||
| _ROUTER_EVENTS_WRITER.write({ | ||
| "source": "disagg_router", | ||
| "ctx_request_id": ctx_request_id, | ||
| "disagg_request_id": disagg_request_id, | ||
| "ctx_server": self.ctx_server, | ||
| "gen_server": gen_server, | ||
| # steady-clock seconds, shared with the worker per-rank files. | ||
| "arrival_time": self.request_arrival_time, | ||
| "ctx_dispatch_time": self.ctx_dispatch_time or None, | ||
| "gen_dispatch_time": self.gen_dispatch_time or None, | ||
| "first_token_time": self.server_first_token_time or None, | ||
| "resp_done_time": get_steady_clock_now_in_seconds(), | ||
| }) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
rg -n 'ctx_request_id' tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.pyRepository: NVIDIA/TensorRT-LLM
Length of output: 818
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- perf_time_events.py relevant sections ---'
sed -n '90,130p' tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.py
echo
sed -n '250,360p' tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.py
echo
sed -n '390,520p' tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.py
echo
echo '--- openai_disagg_service.py _get_gen_request ---'
rg -n -A40 -B20 'def _get_gen_request|ctx_response|DisaggregatedParams|ctx_request_id=' tensorrt_llm/serve/openai_disagg_service.py
echo
echo '--- openai_disagg_server.py docstring context ---'
sed -n '115,143p' tensorrt_llm/serve/openai_disagg_server.pyRepository: NVIDIA/TensorRT-LLM
Length of output: 35685
Keep ctx_request_id null on the no-ctx path. perf_time_events.py::parse_router_dir() treats missing ctx_request_id as the gen-only sentinel and routes those records into _no_ctx; setting it to disagg_request_id makes no-ctx router rows look ctx-keyed and breaks that join contract. Either preserve None here or update the merger to match this value.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tensorrt_llm/serve/openai_disagg_server.py` around lines 115 - 143, Preserve
ctx_request_id as None in _maybe_write_router_event when no ctx round-trip
occurred; do not substitute disagg_request_id for it. Keep the existing
disagg_request_id field for router dispatch identity so parse_router_dir()
continues routing gen-only records to _no_ctx.
| def __init__( | ||
| self, | ||
| events_dir: Optional[str], | ||
| filename_prefix: str, | ||
| max_queue: int = 100000, | ||
| ): | ||
| self._events_dir = events_dir or None | ||
| self._filename_prefix = filename_prefix | ||
| self._max_queue = max_queue | ||
| self._events_file = None # opened lazily by the writer thread | ||
| self._writer_queue: Optional[queue.Queue] = None | ||
| self._writer_thread: Optional[threading.Thread] = None | ||
| self._writer_lock = threading.Lock() | ||
| self._atexit_registered = False | ||
| self._dropped = 0 | ||
|
|
||
| @property | ||
| def enabled(self) -> bool: | ||
| return self._events_dir is not None | ||
|
|
||
| def write(self, record: dict) -> None: | ||
| """Enqueue one record for the writer thread (non-blocking). | ||
|
|
||
| No-op when the writer is inert (no output dir configured). | ||
| """ | ||
| if self._events_dir is None: | ||
| return | ||
| self._ensure_writer() | ||
| try: | ||
| self._writer_queue.put_nowait(record) | ||
| except queue.Full: | ||
| # Prefer dropping over blocking the router event loop / client | ||
| # request path. Count so close() can surface it once. | ||
| self._dropped += 1 |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Writer keeps accepting records into a queue that no thread is draining once the output file fails to open.
If os.makedirs/open(path, "a") raises OSError in _writer_loop, the thread prints one warning and returns — but _writer_thread stays set, so write() continues to put_nowait records into the now-unattended queue for the rest of the process's life. Records accumulate silently until max_queue (100000) fills, after which they're dropped with only a single aggregate warning printed in close() (which may be much later, at process exit). For a long-running router/client, this can mean the perf-time-events feature silently produces no output for an entire run with no timely diagnostic.
🔧 Proposed fix: mark the writer inert once the file can't be opened
self._writer_lock = threading.Lock()
self._atexit_registered = False
self._dropped = 0
+ self._open_failed = False
def write(self, record: dict) -> None:
"""Enqueue one record for the writer thread (non-blocking).
No-op when the writer is inert (no output dir configured).
"""
if self._events_dir is None:
return
+ if self._open_failed:
+ return
self._ensure_writer()
try:
self._writer_queue.put_nowait(record)
except queue.Full:
self._dropped += 1
@@
try:
os.makedirs(self._events_dir, exist_ok=True)
self._events_file = open(path, "a")
except OSError as e:
print(f"WARNING: failed to open perf time-events file {path}: {e}")
+ self._open_failed = True
returnAlso applies to: 127-153
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tensorrt_llm/serve/perf_time_events_writer.py` around lines 73 - 106, Ensure
the `_writer_loop` OSError path transitions the writer to an inert state when
directory creation or file opening fails. Update `write` and the writer-state
handling so records are no longer enqueued after that failure, and ensure
`_writer_thread` cannot remain represented as an active/draining writer.
Preserve the existing warning while making the failure immediately prevent
further queue growth.
| # --- Perf time-events (client send timeline) --- | ||
| # Absolute POST-send timestamps. ``send_time`` is a process-local monotonic | ||
| # ``time.perf_counter()`` reading (the same epoch ttft/itl/latency are | ||
| # relative to); ``send_wall_time`` is the ``time.time()`` wall-clock anchor a | ||
| # post-script uses to align this client timeline against server-side steady | ||
| # -clock records (there is no shared request id, so alignment is best-effort). | ||
| send_time: float = 0.0 | ||
| send_wall_time: float = 0.0 | ||
| # Server response id (``cmpl-...``/``chatcmpl-...`` uuid). Captured for | ||
| # reference/dedup; it is minted fresh per response and does NOT equal the | ||
| # worker's internal request id, so it is not a worker-join key. | ||
| response_id: str = "" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
rg -n 'def async_request_openai_chat_completions' -A 80 tensorrt_llm/serve/scripts/backend_request_func.py | rg -n 'send_time|send_wall_time|response_id|data.get\("id"\)'Repository: NVIDIA/TensorRT-LLM
Length of output: 157
🏁 Script executed:
ast-grep outline tensorrt_llm/serve/scripts/backend_request_func.py --view expandedRepository: NVIDIA/TensorRT-LLM
Length of output: 1380
🏁 Script executed:
sed -n '177,430p' tensorrt_llm/serve/scripts/backend_request_func.py | cat -nRepository: NVIDIA/TensorRT-LLM
Length of output: 13138
🏁 Script executed:
sed -n '430,540p' tensorrt_llm/serve/scripts/backend_request_func.py | cat -nRepository: NVIDIA/TensorRT-LLM
Length of output: 2629
🏁 Script executed:
sed -n '250,340p' tensorrt_llm/serve/scripts/backend_request_func.py | cat -nRepository: NVIDIA/TensorRT-LLM
Length of output: 4680
Populate the client timeline fields in async_request_openai_chat_completions — send_time, send_wall_time, and response_id are set in async_request_openai_completions but never assigned in the chat-completions path, so openai-chat runs will emit empty timing/ID data for the perf-time-events client file.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tensorrt_llm/serve/scripts/backend_request_func.py` around lines 71 - 82,
Update async_request_openai_chat_completions to assign send_time and
send_wall_time at the same POST-send point as async_request_openai_completions,
and capture the returned response ID into response_id. Preserve the existing
timestamp sources and response-ID extraction behavior so openai-chat
perf-time-events records contain the same client timeline data.
| Returned keyed by ``str(ctx_request_id)`` so worker records join in O(1). | ||
| Records without a ``ctx_request_id`` (gen-only / no-ctx path) are collected | ||
| under the sentinel key ``""`` as a list appended into ``_no_ctx`` so they are | ||
| still reported rather than silently dropped. | ||
| """ | ||
| by_ctx: Dict[str, Dict[str, Any]] = {} | ||
| no_ctx: List[Dict[str, Any]] = [] | ||
| files = sorted(glob.glob(os.path.join(router_dir, "disagg_router_*.jsonl"))) | ||
| for path in files: | ||
| for obj in _iter_jsonl(path): | ||
| if not isinstance(obj, dict): | ||
| continue | ||
| ctx_id = obj.get("ctx_request_id") | ||
| if ctx_id is None: | ||
| no_ctx.append(obj) | ||
| else: | ||
| by_ctx[str(ctx_id)] = obj | ||
| if no_ctx: | ||
| by_ctx["_no_ctx"] = no_ctx # type: ignore[assignment] | ||
| return by_ctx |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Docstring contradicts the sentinel key, and the return annotation doesn't hold.
The docstring says no-ctx records land under the sentinel key "", but Line 123 uses "_no_ctx" — and that's the key merge() filters on, so the doc is actively misleading. The declared Dict[str, Dict[str, Any]] is also wrong for that entry (it's a list), which is why the type: ignore is needed; widening the value type removes the suppression.
📝 Proposed fix
-def parse_router_dir(router_dir: str) -> Dict[str, Dict[str, Any]]:
+def parse_router_dir(router_dir: str) -> Dict[str, Any]:
@@
Returned keyed by ``str(ctx_request_id)`` so worker records join in O(1).
Records without a ``ctx_request_id`` (gen-only / no-ctx path) are collected
- under the sentinel key ``""`` as a list appended into ``_no_ctx`` so they are
- still reported rather than silently dropped.
+ as a list under the sentinel key ``"_no_ctx"`` so they are still reported
+ rather than silently dropped.
"""
- by_ctx: Dict[str, Dict[str, Any]] = {}
+ by_ctx: Dict[str, Any] = {}
@@
- by_ctx["_no_ctx"] = no_ctx # type: ignore[assignment]
+ by_ctx["_no_ctx"] = no_ctxAs per coding guidelines: "Annotate every function ... avoid Any and unnecessary type ignores" and "Prefer docstrings for external interfaces".
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Returned keyed by ``str(ctx_request_id)`` so worker records join in O(1). | |
| Records without a ``ctx_request_id`` (gen-only / no-ctx path) are collected | |
| under the sentinel key ``""`` as a list appended into ``_no_ctx`` so they are | |
| still reported rather than silently dropped. | |
| """ | |
| by_ctx: Dict[str, Dict[str, Any]] = {} | |
| no_ctx: List[Dict[str, Any]] = [] | |
| files = sorted(glob.glob(os.path.join(router_dir, "disagg_router_*.jsonl"))) | |
| for path in files: | |
| for obj in _iter_jsonl(path): | |
| if not isinstance(obj, dict): | |
| continue | |
| ctx_id = obj.get("ctx_request_id") | |
| if ctx_id is None: | |
| no_ctx.append(obj) | |
| else: | |
| by_ctx[str(ctx_id)] = obj | |
| if no_ctx: | |
| by_ctx["_no_ctx"] = no_ctx # type: ignore[assignment] | |
| return by_ctx | |
| def parse_router_dir(router_dir: str) -> Dict[str, Any]: | |
| """ | |
| Returned keyed by ``str(ctx_request_id)`` so worker records join in O(1). | |
| Records without a ``ctx_request_id`` (gen-only / no-ctx path) are collected | |
| as a list under the sentinel key ``"_no_ctx"`` so they are still reported | |
| rather than silently dropped. | |
| """ | |
| by_ctx: Dict[str, Any] = {} | |
| no_ctx: List[Dict[str, Any]] = [] | |
| files = sorted(glob.glob(os.path.join(router_dir, "disagg_router_*.jsonl"))) | |
| for path in files: | |
| for obj in _iter_jsonl(path): | |
| if not isinstance(obj, dict): | |
| continue | |
| ctx_id = obj.get("ctx_request_id") | |
| if ctx_id is None: | |
| no_ctx.append(obj) | |
| else: | |
| by_ctx[str(ctx_id)] = obj | |
| if no_ctx: | |
| by_ctx["_no_ctx"] = no_ctx | |
| return by_ctx |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.py` around lines
105 - 124, Update the relevant function’s docstring to document the actual
"_no_ctx" sentinel used by merge(), and widen its return annotation/value type
to represent both per-context record dictionaries and the no-context record
list. Remove the unnecessary type: ignore on the "_no_ctx" assignment while
preserving the existing grouping behavior.
Source: Coding guidelines
| # Router dispatch join (by ctx_request_id -- the cross-process key). | ||
| router_rec = None | ||
| for rid in rids: | ||
| if rid in router_by_ctx: | ||
| router_rec = router_by_ctx[rid] | ||
| matched_router_ctx.add(rid) | ||
| break | ||
| if router_rec is not None: | ||
| rec["router_dispatch"] = router_rec |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Router join also matches on request_id, not just ctx_request_id.
rids holds [request_id, ctx_request_id] and request_id is tried first, so a worker-local request_id that numerically collides with an unrelated request's ctx_request_id yields a false router_dispatch attachment and bogus router_* derived spans — silently, since the loop breaks on first hit. The comment and docstring both state ctx id is the cross-process key; restrict the lookup to it.
🐛 Proposed fix
# Router dispatch join (by ctx_request_id -- the cross-process key).
router_rec = None
- for rid in rids:
- if rid in router_by_ctx:
- router_rec = router_by_ctx[rid]
- matched_router_ctx.add(rid)
- break
+ ctx_rid = rec.get("ctx_request_id")
+ if ctx_rid is not None and str(ctx_rid) in router_by_ctx:
+ router_rec = router_by_ctx[str(ctx_rid)]
+ matched_router_ctx.add(str(ctx_rid))
if router_rec is not None:
rec["router_dispatch"] = router_rec📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # Router dispatch join (by ctx_request_id -- the cross-process key). | |
| router_rec = None | |
| for rid in rids: | |
| if rid in router_by_ctx: | |
| router_rec = router_by_ctx[rid] | |
| matched_router_ctx.add(rid) | |
| break | |
| if router_rec is not None: | |
| rec["router_dispatch"] = router_rec | |
| # Router dispatch join (by ctx_request_id -- the cross-process key). | |
| router_rec = None | |
| ctx_rid = rec.get("ctx_request_id") | |
| if ctx_rid is not None and str(ctx_rid) in router_by_ctx: | |
| router_rec = router_by_ctx[str(ctx_rid)] | |
| matched_router_ctx.add(str(ctx_rid)) | |
| if router_rec is not None: | |
| rec["router_dispatch"] = router_rec |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tensorrt_llm/serve/scripts/perf_time_events/perf_time_events.py` around lines
333 - 341, Update the router dispatch join around router_by_ctx to look up only
the worker record’s ctx_request_id, not both values in rids. Remove the
request_id-first iteration and retain matched_router_ctx tracking and
router_dispatch assignment only when the ctx-key lookup succeeds.
| class TestJsonlEventWriter(unittest.TestCase): | ||
| def test_inert_when_no_dir(self): | ||
| # Falsy events_dir -> inert: enabled False, write a no-op, close safe. | ||
| writer = JsonlEventWriter("", "disagg_router") | ||
| self.assertFalse(writer.enabled) | ||
| writer.write({"source": "disagg_router", "x": 1}) # must not raise | ||
| writer.close() # safe even though no thread ever started | ||
|
|
||
| def test_writes_one_line_per_record_and_drains_on_close(self): | ||
| with tempfile.TemporaryDirectory() as d: | ||
| writer = JsonlEventWriter(d, "disagg_router") | ||
| self.assertTrue(writer.enabled) | ||
| writer.write({"source": "disagg_router", "ctx_request_id": 1}) | ||
| writer.write({"source": "disagg_router", "ctx_request_id": 2}) | ||
| writer.close() # drains + joins the daemon thread | ||
|
|
||
| files = glob.glob(os.path.join(d, "disagg_router_pid*.jsonl")) | ||
| self.assertEqual(len(files), 1) | ||
| self.assertIn(f"pid{os.getpid()}", os.path.basename(files[0])) | ||
| with open(files[0]) as f: | ||
| lines = [json.loads(ln) for ln in f if ln.strip()] | ||
| self.assertEqual([r["ctx_request_id"] for r in lines], [1, 2]) | ||
|
|
||
| def test_non_serializable_value_falls_back_to_str(self): | ||
| # default=str keeps a non-JSON-native value from killing the writer | ||
| # thread (and losing every subsequent record). | ||
| with tempfile.TemporaryDirectory() as d: | ||
| writer = JsonlEventWriter(d, "client") | ||
| writer.write({"source": "client", "weird": {1, 2, 3}}) | ||
| writer.close() | ||
| files = glob.glob(os.path.join(d, "client_pid*.jsonl")) | ||
| self.assertEqual(len(files), 1) | ||
| with open(files[0]) as f: | ||
| lines = [json.loads(ln) for ln in f if ln.strip()] | ||
| self.assertEqual(len(lines), 1) | ||
| # The set was stringified rather than raising TypeError. | ||
| self.assertIsInstance(lines[0]["weird"], str) | ||
|
|
||
| def test_close_is_idempotent(self): | ||
| with tempfile.TemporaryDirectory() as d: | ||
| writer = JsonlEventWriter(d, "disagg_router") | ||
| writer.write({"source": "disagg_router"}) | ||
| writer.close() | ||
| writer.close() # second call is a no-op, must not raise | ||
|
|
||
|
|
||
| class TestMakeEnvWriter(unittest.TestCase): | ||
| def test_inert_when_env_unset(self): | ||
| with patch.dict(os.environ, {}, clear=False): | ||
| os.environ.pop(ROUTER_EVENTS_PATH_ENV, None) | ||
| writer = make_env_writer(ROUTER_EVENTS_PATH_ENV, "disagg_router") | ||
| self.assertFalse(writer.enabled) | ||
|
|
||
| def test_enabled_when_env_set(self): | ||
| with tempfile.TemporaryDirectory() as d: | ||
| with patch.dict(os.environ, {CLIENT_EVENTS_PATH_ENV: d}): | ||
| writer = make_env_writer(CLIENT_EVENTS_PATH_ENV, "client") | ||
| self.assertTrue(writer.enabled) | ||
| writer.write({"source": "client", "client_index": 0}) | ||
| writer.close() | ||
| files = glob.glob(os.path.join(d, "client_pid*.jsonl")) | ||
| self.assertEqual(len(files), 1) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| unittest.main() |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
rg -n 'test_perf_time_events_writer' tests/integration/test_listsRepository: NVIDIA/TensorRT-LLM
Length of output: 157
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Test-list files:\n'
git ls-files 'tests/integration/test_lists/**' | sed -n '1,200p'
printf '\nOccurrences of test module name in test lists:\n'
rg -n 'test_perf_time_events_writer|test_perf_time|perf_time_events_writer' tests/integration/test_lists || trueRepository: NVIDIA/TensorRT-LLM
Length of output: 4926
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Relevant mentions in integration test lists:\n'
rg -n 'unittest|tests/unittest|others/test_perf_time_events_writer|perf_time_events_writer|test_perf_time|pytest|python -m unittest' tests/integration/test_lists || true
printf '\nA few sample entries from list files:\n'
sed -n '1,120p' tests/integration/test_lists/qa/llm_perf_core.yml
printf '\n---\n'
sed -n '1,120p' tests/integration/test_lists/test-db/l0_perf.ymlRepository: NVIDIA/TensorRT-LLM
Length of output: 50375
Add tests/unittest/others/test_perf_time_events_writer.py to the integration test lists
Changed tests: test_inert_when_no_dir, test_writes_one_line_per_record_and_drains_on_close, test_non_serializable_value_falls_back_to_str, test_close_is_idempotent, test_inert_when_env_unset, test_enabled_when_env_set. None of these appear in tests/integration/test_lists/test-db/ or qa/, so this module won’t be picked up by integration coverage. Coverage verdict: insufficient.
🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 48-48: File path is request-/variable-derived; validate and normalize to prevent path traversal.
Context: open(files[0])
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(open-filename-from-request)
[warning] 61-61: File path is request-/variable-derived; validate and normalize to prevent path traversal.
Context: open(files[0])
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(open-filename-from-request)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/unittest/others/test_perf_time_events_writer.py` around lines 30 - 95,
Add tests/unittest/others/test_perf_time_events_writer.py to the appropriate
integration test-list entries under tests/integration/test_lists/test-db/ and
qa/ so all six TestJsonlEventWriter and TestMakeEnvWriter tests are discovered
by integration coverage.
Source: Path instructions
| def test_client_events_standalone_in_payload(self): | ||
| with tempfile.TemporaryDirectory() as ev, tempfile.TemporaryDirectory() as cl: | ||
| _write( | ||
| os.path.join(ev, "time_events_rank0_pid1.jsonl"), | ||
| json.dumps(_make_record(10)) + "\n", | ||
| ) | ||
| _write( | ||
| os.path.join(cl, "client_pid1.jsonl"), | ||
| json.dumps( | ||
| { | ||
| "source": "client", | ||
| "client_index": 0, | ||
| "send_wall_time": 1.0, | ||
| "ttft": 0.1, | ||
| "latency": 2.0, | ||
| } | ||
| ) | ||
| + "\n", | ||
| ) | ||
| merger = PerfTimeEventsMerger() | ||
| merger.merge(event_dir=ev, client_dir=cl) | ||
| self.assertEqual(len(merger.client_events), 1) | ||
| self.assertEqual(merger.match_stats["num_client_events"], 1) | ||
| # Client records are NOT joined onto request records. | ||
| self.assertNotIn("client", merger.records[0]) | ||
|
|
||
| out = os.path.join(cl, "combined.json") | ||
| merger.write(out) | ||
| with open(out) as f: | ||
| payload = json.load(f) | ||
| self.assertEqual(len(payload["client_events"]), 1) | ||
| self.assertIn("unjoined_router_events", payload) |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
fd -t f 'test_perf_time_events' tests
rg -n 'unittest/others|test_perf_time_events' tests/integration/test_lists -g '*.yml' -g '*.txt' -g '*.yaml' | head -50Repository: NVIDIA/TensorRT-LLM
Length of output: 2491
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== test file outline ==\n'
ast-grep outline tests/unittest/others/test_perf_time_events.py --view expanded || true
printf '\n== test file head/tail around relevant tests ==\n'
wc -l tests/unittest/others/test_perf_time_events.py
sed -n '1,260p' tests/unittest/others/test_perf_time_events.py
printf '\n--- later section ---\n'
sed -n '260,460p' tests/unittest/others/test_perf_time_events.py
printf '\n== integration list mentions ==\n'
rg -n 'test_perf_time_events\.py|unittest/others/test_perf_time_events\.py|unittest/others/' tests/integration/test_lists -g '*.yml' -g '*.yaml' -g '*.txt' || trueRepository: NVIDIA/TensorRT-LLM
Length of output: 20601
🏁 Script executed:
#!/bin/bash
set -euo pipefail
for f in $(git ls-files 'tests/integration/test_lists/**/*.yml' 'tests/integration/test_lists/**/*.yaml' 'tests/integration/test_lists/**/*.txt'); do
if rg -n 'unittest/others/|test_perf_time_events\.py' "$f" >/dev/null; then
echo "### $f"
rg -n 'unittest/others/|test_perf_time_events\.py' "$f"
fi
doneRepository: NVIDIA/TensorRT-LLM
Length of output: 1766
Add tests/unittest/others/test_perf_time_events.py to the CI test lists
Changed test functions (all added; none modified or removed): TestParseRouterDir.test_keyed_by_ctx_id_and_no_ctx_collected, TestParseRouterDir.test_ignores_unrelated_files, TestParseClientDir.test_sorted_by_send_wall_time, TestParseClientDir.test_missing_wall_time_sorts_first, TestRouterAndClientMerge.test_router_join_on_ctx_id_and_derived_spans, TestRouterAndClientMerge.test_unjoined_router_events_include_no_ctx_and_leftovers, TestRouterAndClientMerge.test_client_events_standalone_in_payload (plus helper _router_record).
Test-list registration: tests/unittest/others/test_perf_time_events.py is not listed in any provided tests/integration/test_lists/test-db/ or tests/integration/test_lists/qa/ entry. Add it to the appropriate CI list so these tests run automatically.
Coverage verdict: insufficient.
🧰 Tools
🪛 ast-grep (0.44.1)
[info] 367-367: use jsonify instead of json.dumps for JSON output
Context: json.dumps(_make_record(10))
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 371-379: use jsonify instead of json.dumps for JSON output
Context: json.dumps(
{
"source": "client",
"client_index": 0,
"send_wall_time": 1.0,
"ttft": 0.1,
"latency": 2.0,
}
)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[warning] 391-391: File path is request-/variable-derived; validate and normalize to prevent path traversal.
Context: open(out)
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(open-filename-from-request)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/unittest/others/test_perf_time_events.py` around lines 364 - 395, Add
tests/unittest/others/test_perf_time_events.py to the appropriate CI test-list
entry under the existing database or QA test-list configuration, ensuring the
listed test functions and helper coverage run automatically. Do not modify the
test implementations.
Source: Path instructions
Add the per-request capacity-scheduler admission event (event 1 of the five per-step/per-chunk perf time events). Previously only a per-iteration starvation count existed for the capacity stage; the other four events (micro-batch admit, forward begin, sampler begin, decode/prefill end) already carried per-request timestamps. - SimpleScheduler stamps `_last_capacity_admit_time` via `steady_clock_now().total_seconds()` right after the capacity scheduler returns the fitting set, before micro-batch scheduling. Gated on a new `capture_admit_time` flag (default False) so the base scheduling path pays nothing. - PyExecutor flips the flag on only under `perf_manager.capture_extended`, and only when the scheduler exposes it (hasattr guard leaves PP / non-Simple schedulers untouched). It reads the stamp in `_prepare_and_schedule_batch` and threads it through `_compute_iter_batch_context` as `capacity_scheduled_time`, riding onto every admitted request's per-chunk / per-step entry. Same steady clock as `scheduled_time`, so the gap `scheduled_time - capacity_scheduled_time` isolates the micro-batch scheduling stage. Key is omitted when the scheduler does not expose the stamp. Adds GPU-free unit tests for the emit gating and the scheduler stamp. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/unittest/_torch/executor/test_perf_metrics_events.py`:
- Around line 398-406: Update test_stamp_refreshes_each_round to mock the
module-local steady_clock_now with two distinct sequential values, then assert
_last_capacity_admit_time equals the first value after the initial
schedule_request and the second value after the next call. Replace the
non-deterministic ordering assertion while preserving the existing two-round
setup.
- Around line 205-241: The current tests cover _compute_iter_batch_context() and
SimpleScheduler but not the PyExecutor handoff from _last_capacity_admit_time.
Add an executor-level test in test_perf_metrics_events.py that exercises the
PyExecutor scheduling path, verifies the stored _last_capacity_admit_time is
passed through, and confirms capacity_scheduled_time is emitted; also cover the
absent-value path if needed to preserve omission behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 43ef280c-ebcd-4842-8d3f-0000dcdc422c
📒 Files selected for processing (3)
tensorrt_llm/_torch/pyexecutor/py_executor.pytensorrt_llm/_torch/pyexecutor/scheduler/scheduler.pytests/unittest/_torch/executor/test_perf_metrics_events.py
🚧 Files skipped from review as they are similar to previous changes (1)
- tensorrt_llm/_torch/pyexecutor/py_executor.py
| def test_admission_timestamps_emitted_when_given(self): | ||
| # Both scheduler-stage timestamps ride into the shared dict; the gap | ||
| # scheduled_time - capacity_scheduled_time isolates the micro-batch stage. | ||
| scheduled = SimpleNamespace( | ||
| num_context_requests=0, | ||
| num_generation_requests=1, | ||
| batch_size=1, | ||
| context_requests=[], | ||
| generation_requests=[SimpleNamespace(num_draft_tokens=0)], | ||
| ) | ||
| fake_self = SimpleNamespace(iter_counter=3) | ||
| ctx = PyExecutor._compute_iter_batch_context( | ||
| fake_self, | ||
| scheduled, | ||
| schedule_time=100.5, | ||
| capacity_admit_time=100.2, | ||
| ) | ||
| assert ctx["scheduled_time"] == 100.5 | ||
| assert ctx["capacity_scheduled_time"] == 100.2 | ||
| assert ctx["scheduled_time"] >= ctx["capacity_scheduled_time"] | ||
|
|
||
| def test_capacity_scheduled_time_omitted_when_absent(self): | ||
| # PP local scheduler / non-SimpleScheduler expose no admit time -> the | ||
| # key must simply not appear (no None leaking into the record). | ||
| scheduled = SimpleNamespace( | ||
| num_context_requests=0, | ||
| num_generation_requests=1, | ||
| batch_size=1, | ||
| context_requests=[], | ||
| generation_requests=[SimpleNamespace(num_draft_tokens=0)], | ||
| ) | ||
| fake_self = SimpleNamespace(iter_counter=1) | ||
| ctx = PyExecutor._compute_iter_batch_context( | ||
| fake_self, scheduled, schedule_time=5.0, capacity_admit_time=None | ||
| ) | ||
| assert ctx["scheduled_time"] == 5.0 | ||
| assert "capacity_scheduled_time" not in ctx |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
test_file='tests/unittest/_torch/executor/test_perf_metrics_events.py'
rg -n "test_perf_metrics_events|${test_file}" tests/integration/test_lists/test-db tests/integration/test_lists/qaRepository: NVIDIA/TensorRT-LLM
Length of output: 157
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the touched test file and nearby context
test_file='tests/unittest/_torch/executor/test_perf_metrics_events.py'
wc -l "$test_file"
sed -n '1,260p' "$test_file"
printf '\n--- tail ---\n'
sed -n '260,460p' "$test_file"
# Find the relevant integration list files and whether this test file/test names are listed
printf '\n--- list files ---\n'
find tests/integration/test_lists -maxdepth 2 -type f \( -path '*/test-db/*' -o -path '*/qa/*' \) | sort
printf '\n--- file/test-name matches in test-db and qa lists ---\n'
rg -n --hidden --no-messages \
-e 'test_perf_metrics_events|tests/unittest/_torch/executor/test_perf_metrics_events\.py|test_admission_timestamps_emitted_when_given|test_capacity_scheduled_time_omitted_when_absent' \
tests/integration/test_lists/test-db tests/integration/test_lists/qa || trueRepository: NVIDIA/TensorRT-LLM
Length of output: 22019
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n --hidden \
-e 'def _compute_iter_batch_context|capacity_admit_time|capture_admit_time|_last_capacity_admit_time|scheduled_time' \
tensorrt_llm/_torch/pyexecutor tensorrt_llm/_torch/pyexecutor/scheduler tests/unittest/_torch/executor/test_perf_metrics_events.py
printf '\n--- py_executor slice ---\n'
sed -n '1,260p' tensorrt_llm/_torch/pyexecutor/py_executor.py
printf '\n--- scheduler slice ---\n'
sed -n '1,260p' tensorrt_llm/_torch/pyexecutor/scheduler/scheduler.pyRepository: NVIDIA/TensorRT-LLM
Length of output: 26880
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Look for any tests that exercise the production PyExecutor scheduling path
# with capacity admission timestamps / admit-time capture.
rg -n --hidden --glob '!**/.git/**' \
-e 'capacity_scheduled_time|capacity_admit_time|capture_admit_time|_last_capacity_admit_time|_compute_iter_batch_context|PyExecutor' \
tests/unittest tests/integration
printf '\n--- potential executor tests ---\n'
find tests/unittest -path '*pyexecutor*' -o -path '*executor*' | sortRepository: NVIDIA/TensorRT-LLM
Length of output: 37063
Add an executor-level handoff test. The new cases cover _compute_iter_batch_context() and SimpleScheduler in isolation, but not the PyExecutor path that reads _last_capacity_admit_time and emits capacity_scheduled_time; a wiring regression there would still pass these tests.
Coverage summary: test_admission_timestamps_emitted_when_given, test_capacity_scheduled_time_omitted_when_absent, and the SimpleScheduler admit-time tests are in tests/unittest/_torch/executor/test_perf_metrics_events.py; this file is not listed under tests/integration/test_lists/test-db/* or tests/integration/test_lists/qa/*. Coverage verdict: insufficient.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/unittest/_torch/executor/test_perf_metrics_events.py` around lines 205
- 241, The current tests cover _compute_iter_batch_context() and SimpleScheduler
but not the PyExecutor handoff from _last_capacity_admit_time. Add an
executor-level test in test_perf_metrics_events.py that exercises the PyExecutor
scheduling path, verifies the stored _last_capacity_admit_time is passed
through, and confirms capacity_scheduled_time is emitted; also cover the
absent-value path if needed to preserve omission behavior.
Source: Path instructions
| def test_stamp_refreshes_each_round(self): | ||
| sched = self._make_scheduler() | ||
| sched.capture_admit_time = True | ||
| sched.schedule_request([], set()) | ||
| first = sched._last_capacity_admit_time | ||
| sched.schedule_request([], set()) | ||
| second = sched._last_capacity_admit_time | ||
| # A new reading every round (monotonic non-decreasing). | ||
| assert second >= first |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Make the refresh assertion deterministic.
second >= first also passes when both timestamps are identical, so it would not detect a regression that stops refreshing _last_capacity_admit_time. Mock the module-local steady_clock_now with distinct values and assert both expected timestamps.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/unittest/_torch/executor/test_perf_metrics_events.py` around lines 398
- 406, Update test_stamp_refreshes_each_round to mock the module-local
steady_clock_now with two distinct sequential values, then assert
_last_capacity_admit_time equals the first value after the initial
schedule_request and the second value after the next call. Replace the
non-deterministic ordering assertion while preserving the existing two-round
setup.
1 participant
Description
Adds a single master switch,
TRTLLM_PERF_TIME_EVENTS_PATH(an outputdirectory, symmetric with
TRTLLM_KVCACHE_TIME_OUTPUT_PATH), that turns onper-request perf-timing capture and writes per-rank time-event files live from
inside the executor — exactly mirroring how the disaggregation transceiver
already writes per-rank KV-transfer CSVs (
_torch/disaggregation/native/perf_logger.py).Today, capturing this timing over the HTTP
/perf_metricsendpoint is fragile:the benchmark client must remember
--save-request-time-breakdown, the dequeonly exists when
perf_metrics_max_requests > 0,return_perf_metricsmust beset per worker, and the data must be pulled before the server tears down —
which for a disagg run means racing teardown across separate ctx/gen servers.
This PR replaces that with a live, per-rank, on-disk artifact and no HTTP scrape.
What the switch does
return_perf_metrics, and turns onan extended capture mode (
capture_extended). All new behavior is gatedon
capture_extended, so existingreturn_perf_metrics=Trueusers with theenv unset see no behavior or output change (regression guard: H3 below).
per-iteration metric dict:
iter_batch_size,num_ctx_requests,num_gen_requests,context_token_number,generation_token_number,per-request
req_context_token_number/req_generation_token_number, plus aper-iteration starvation count (
num_capacity_fittingvsnum_scheduled)— the signal that explains gen bubbles / scheduling stalls. These ride
inside the existing
time_breakdown_metricspayload, so there is no newpayload shape and no
create_responsechange.{dir}/time_events_rank{N}_pid{P}.jsonllive, one JSONL line perfinished request (rank+pid in the name avoids ctx-server/gen-server rank-0
collisions on a shared FS).
Off the critical path (overhead requirement)
The executor loop only builds a dict and does a non-blocking
queue.Queue.put_nowait— no file I/O on the loop thread. A lazily-starteddaemon writer thread drains the queue and does the actual
json.dumps + write + flush;PyExecutor.shutdown()drains and joins it.queue.Full→ warn + drop, never block. So the added on-loop cost is a dictbuild + queue append, keeping the A/B overhead delta structurally near-zero.
Optional offline aggregator
python -m tensorrt_llm.serve.scripts.perf_time_eventsstitches the per-rankfiles (+ the KV-transfer CSVs) into one combined JSON/HTML and computes derived
signals (inter-step / inter-chunk gaps, per-iteration starvation). It reuses
time_breakdown'sRequestTimeBreakdownfor HTML and is stdlib-only on theparse/merge/JSON path (torch-free;
plotlypulled in lazily only for--html),so it stays under the GPU-free import contract. It is a convenience over the
per-rank files, not the load-bearing capture path.
Python only — no C++. The KV sub-events already exist in Python for the
target
PYTHON-runtime (disagg NIXL) workload, so they are only ingested.Flagged Python-only compromises
attribution —
_schedule()returns onlynum_fitting_reqsin Python, sowhich specific requests were starved cannot be attributed without a C++ change.
_executor_loop_ppnever callssave_timing_to_requests, so the extendedfields are absent under pipeline parallelism (known gap, documented).
request's disagg
ctx_request_idwhen available, else it is an aggregator-sidejoin enhancement.
Test Coverage
GPU-free unit tests (both run on CPU; the executor-side one
importorskipstorch, the aggregator one is pure stdlib):
tests/unittest/_torch/executor/test_perf_metrics_events.pyappend_step_metricsmerges the batch-context dict + per-request tokencounts for ctx and gen (incl. draft tokens), and the disabled-shape guard
(
capture_extended=False→ none of the new keys, base shape untouched)._compute_iter_batch_contextagainst a fakeScheduledRequests.time_events_rank{N}_pid{P}.jsonlline with the right shape; no-ops whentime_breakdown_metricsis absent (non-final response) and when disabled.tests/unittest/others/test_perf_time_events.py— aggregatorparse_event_dir/parse_kv_csv_dir(native + gen-summary + C++ headers),mergejoin-by-request-and-ctx-id,unjoined_kv_events+ match-rate for anorphan rid, derived
inter_step_gaps/starved, combined-JSON write.Run:
GPU end-to-end verification (
gen_only ... gpt-oss-120b-fp4 ... dep2 ... NIXLon OCI-HSG GB200; no
--capture-nsysdue to the documented nsys-2026 hang onthis exact case) is in progress and will be posted as a follow-up comment:
per-rank files written with the extended keys, an A/B overhead comparison
(env unset vs set), and the regression guard (env unset → no
time_events_*file and unchanged
/perf_metricskeys).PR Checklist
api-compatible/api-breakinglabel needed): the LLM APIis untouched; only an env-gated, additive capture path.
docs/source/developer-guide/perf-analysis.mdgains a"Perf Time Events (per-rank live capture)" section (switch, JSONL record shape,
the three known limitations, aggregator recipe; cross-references
kv-transfer.md).Summary
Adds opt-in live per-rank performance time-event capture controlled by
TRTLLM_PERF_TIME_EVENTS_PATH.iter_batch_context) when enabled, and enriches capture with token-count and capacity/scheduling-related fields when available.time_events_rank{rank}_pid{pid}.jsonl.TRTLLM_PERF_TIME_EVENTS_ROUTER_PATH)TRTLLM_PERF_TIME_EVENTS_CLIENT_PATH)perf_time_events) that merges rank event JSONL with KV-transfer CSVs (and optionally router/client JSONL) into combined JSON and optional HTML timeline output, including derived gap/starved metrics and join/match-rate reporting./perf_metricsandreturn_perf_metricsare enabled when time-event capture is active.Dev Engineer Review
TRTLLM_PERF_TIME_EVENTS_PATH; when set, extended capture is forced and executor iteration context (iter_batch_context) is wired through to per-iteration and per-request outputs.time_breakdown_metricswhen present; extended fields/token counts are conditionally merged only when extended capture is enabled and required context fields exist.close().atexithandling is used in the writer module.QA Engineer Review
Test changes (code, not test-list-only)
New/updated tests under
tests/:tests/unittest/_torch/executor/test_perf_metrics_events.pyTestAppendStepMetricsExtendedtest_ctx_merges_batch_context_and_token_counttest_gen_merges_batch_context_and_token_counttest_gen_token_count_includes_drafttest_disabled_extended_leaves_shape_untouchedTestComputeIterBatchContexttest_counts_and_tokenstest_starvation_omitted_when_no_fitting_counttest_admission_timestamps_emitted_when_giventest_capacity_scheduled_time_omitted_when_absentTestPerRankWritertest_writes_one_jsonl_line_per_requesttest_no_write_when_metrics_absenttest_no_write_when_disabledtest_record_enriched_with_request_timing_metricstest_empty_timing_metrics_omits_keytest_enrichment_failure_is_swallowedTestSimpleSchedulerAdmitStamptest_no_stamp_by_defaulttest_stamp_when_enabled_uses_steady_clocktest_stamp_refreshes_each_roundtests/unittest/others/test_perf_time_events.pyparse_event_dir,parse_kv_csv_dir,parse_router_dir,parse_client_dirPerfTimeEventsMerger.merge/write/ derived metrics (derived.inter_step_gaps,derived.starved)tests/unittest/others/test_perf_time_events_writer.pyJsonlEventWriter:test_inert_when_no_dirtest_writes_one_line_per_record_and_drains_on_closetest_non_serializable_value_falls_back_to_strtest_close_is_idempotentmake_env_writer:test_inert_when_env_unsettest_enabled_when_env_setCoverage mapping to CI/manual test lists
tests/integration/test_lists/,test-db/,qa/(no evidence of test-list registration changes provided).