[TRTLLM-12842][feat] Maximal LLMAPI capture in usage telemetry by venkywonka · Pull Request #14398 · NVIDIA/TensorRT-LLM

venkywonka · 2026-05-21T08:21:16Z

Motivation

Usage telemetry had coarse feature flags, not effective LLM API runtime-shape config. Needed: know how deployed configs are shaped without collecting private or user-controlled data.

Hard rule: no PII, no prompts, no outputs, no model/tokenizer paths, no env vars, no raw YAML, no arbitrary fields, no arbitrary strings.

Design

Full list and related TRTLLM documentation preview: https://venkywonka.github.io/TensorRT-LLM/developer-guide/telemetry.html

Add field-local Field(..., telemetry=...) opt-in metadata for approved LLM args fields.
Walk only declared Pydantic fields from validated effective args. Unmarked fields stay invisible.
Emit deterministic llmApiConfigJson for sanitized config values.
Emit llmApiConfigMetaJson with capture version, policy version, args class, digests, counts, source, and unsafe-exclusion status.
Fail closed in sanitizer: reject paths, raw strings without allowlists, dict/object payloads, Any, callables, unsafe unions, and heterogeneous tuples.
Keep featuresJson for legacy summary consumers. Add TODO to dedupe with llmApiConfigJson later.
Generate telemetry docs during Sphinx build from llm_args.py annotations so exhaustive captured-key docs do not drift manually.
Preserve existing usage opt-out behavior and precompiled editable validation path for Python/schema-only changes.

Testing

python3 -m json.tool tensorrt_llm/usage/schemas/trtllm_usage_event_schema.json
python3 -m py_compile on telemetry usage modules and docs generator
ruff check / ruff format --check on touched Python files
git diff --check
Usage unit coverage: 199 passed, 5 skipped
Forced real-GPU usage capture smoke: 2 passed
Generated-docs unit coverage: 2 passed
Full Sphinx HTML build completed; telemetry page renders README link, schema link, privacy invariant, and exhaustive LLM API config field tables.
Real GPU precomputed-wheel validation path: 54 passed
Real-CUDA smoke payloads passed for single-GPU, TP=2 compile, and scheduler/KV-cache privacy shapes.
TRT-LLM MCP runtime contract check: status=ready

Summary by CodeRabbit

Documentation
- Enhanced README telemetry section with explicit enumeration of excluded data categories (outputs, model paths, PII, raw configurations, persistent identifiers).
- Expanded developer guide with detailed telemetry capture reference and configuration field allowlists.
New Features
- Introduced sanitized, opt-in LLM API configuration telemetry with deterministic capture and validation; schema version bumped to 0.2.
- Added disaggregated launcher telemetry environment propagation for deployment tracking.
Tests
- Comprehensive test coverage for configuration telemetry capture, sanitization, and end-to-end validation.

venkywonka · 2026-05-21T19:32:30Z

/bot kill

venkywonka · 2026-05-21T19:32:40Z

/bot run

tensorrt-cicd · 2026-05-21T19:39:04Z

PR_Github #49757 [ kill ] triggered by Bot. Commit: 2ea9ea6 Link to invocation

tensorrt-cicd · 2026-05-21T19:39:09Z

PR_Github #49757 [ kill ] completed with state SUCCESS. Commit: 2ea9ea6
Successfully killed previous jobs for commit 2ea9ea6

Link to invocation

tensorrt-cicd · 2026-05-21T19:39:31Z

PR_Github #49758 [ run ] triggered by Bot. Commit: 2ea9ea6 Link to invocation

tensorrt-cicd · 2026-05-21T20:33:38Z

PR_Github #49758 [ run ] completed with state SUCCESS. Commit: 2ea9ea6
/LLM/main/L0_MergeRequest_PR pipeline #39358 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

venkywonka · 2026-05-21T20:35:58Z

/bot kill

venkywonka · 2026-05-21T20:36:06Z

/bot run

tensorrt-cicd · 2026-05-21T20:41:46Z

PR_Github #49763 [ kill ] triggered by Bot. Commit: f8773b6 Link to invocation

tensorrt-cicd · 2026-05-21T20:41:50Z

PR_Github #49763 [ kill ] completed with state SUCCESS. Commit: f8773b6
Successfully killed previous jobs for commit f8773b6

Link to invocation

tensorrt-cicd · 2026-05-21T20:42:34Z

PR_Github #49764 [ run ] triggered by Bot. Commit: f8773b6 Link to invocation

venkywonka · 2026-05-21T21:30:58Z

/bot kill

venkywonka · 2026-05-21T21:32:04Z

/bot run

tensorrt-cicd · 2026-05-21T21:37:31Z

PR_Github #49771 [ kill ] triggered by Bot. Commit: 1159335 Link to invocation

tensorrt-cicd · 2026-05-21T21:40:02Z

PR_Github #49772 [ run ] triggered by Bot. Commit: 1159335 Link to invocation

tensorrt-cicd · 2026-05-21T21:40:06Z

PR_Github #49771 [ kill ] completed with state ABORTED. Commit: 1159335

Link to invocation

tensorrt-cicd · 2026-05-21T21:40:35Z

PR_Github #49764 [ run ] completed with state ABORTED. Commit: f8773b6

Link to invocation

venkywonka · 2026-05-21T22:57:03Z

/bot kill

venkywonka · 2026-05-21T22:57:11Z

/bot run

tensorrt-cicd · 2026-05-21T23:03:11Z

PR_Github #49778 [ run ] triggered by Bot. Commit: 5966c67 Link to invocation

tensorrt-cicd · 2026-05-21T23:03:52Z

PR_Github #49779 [ kill ] triggered by Bot. Commit: 5966c67 Link to invocation

tensorrt-cicd · 2026-05-21T23:03:55Z

PR_Github #49778 [ run ] completed with state ABORTED. Commit: 5966c67

Link to invocation

tensorrt-cicd · 2026-05-21T23:04:23Z

PR_Github #49772 [ run ] completed with state ABORTED. Commit: 1159335

Link to invocation

tensorrt-cicd · 2026-05-21T23:04:36Z

PR_Github #49779 [ kill ] completed with state SUCCESS. Commit: 5966c67
Successfully killed previous jobs for commit 5966c67

Link to invocation

venkywonka · 2026-05-21T23:28:01Z

/bot kill

venkywonka · 2026-05-21T23:28:11Z

/bot run

tensorrt-cicd · 2026-05-21T23:34:47Z

PR_Github #49781 [ run ] triggered by Bot. Commit: 5e06861 Link to invocation

tensorrt-cicd · 2026-05-21T23:35:32Z

PR_Github #49782 [ kill ] triggered by Bot. Commit: 5e06861 Link to invocation

tensorrt-cicd · 2026-05-21T23:35:37Z

PR_Github #49781 [ run ] completed with state ABORTED. Commit: 5e06861

Link to invocation

venkywonka · 2026-06-11T23:41:21Z

/bot run

tensorrt-cicd · 2026-06-11T23:47:47Z

PR_Github #53714 [ run ] triggered by Bot. Commit: b9b771d Link to invocation

tensorrt-cicd · 2026-06-12T00:16:25Z

PR_Github #53714 [ run ] completed with state FAILURE. Commit: b9b771d
/LLM/main/L0_MergeRequest_PR pipeline #42843 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

venkywonka · 2026-06-12T15:40:14Z

/bot run

tensorrt-cicd · 2026-06-12T15:46:23Z

PR_Github #53908 [ run ] triggered by Bot. Commit: b9b771d Link to invocation

tensorrt-cicd · 2026-06-12T20:26:16Z

PR_Github #53908 [ run ] completed with state SUCCESS. Commit: b9b771d
/LLM/main/L0_MergeRequest_PR pipeline #43006 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Capture schema-stable LLM API config telemetry from the type-driven manifest. Add the generated manifest golden, docs updates, and runtime/config capture test coverage. Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com>

Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com>

venkywonka · 2026-06-13T04:38:06Z

/bot run

tensorrt-cicd · 2026-06-13T04:43:45Z

PR_Github #54010 [ run ] triggered by Bot. Commit: 010df91 Link to invocation

tensorrt-cicd · 2026-06-13T08:15:47Z

PR_Github #54010 [ run ] completed with state SUCCESS. Commit: 010df91
/LLM/main/L0_MergeRequest_PR pipeline #43094 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

venkywonka · 2026-06-13T08:48:28Z

/bot run

tensorrt-cicd · 2026-06-13T08:55:50Z

PR_Github #54031 [ run ] triggered by Bot. Commit: 010df91 Link to invocation

tensorrt-cicd · 2026-06-13T09:14:12Z

PR_Github #54031 [ run ] completed with state FAILURE. Commit: 010df91
/LLM/main/L0_MergeRequest_PR pipeline #43115 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

venkywonka · 2026-06-13T09:21:16Z

/bot run

tensorrt-cicd · 2026-06-13T09:26:50Z

PR_Github #54034 [ run ] triggered by Bot. Commit: 010df91 Link to invocation

tensorrt-cicd · 2026-06-13T09:44:33Z

PR_Github #54034 [ run ] completed with state FAILURE. Commit: 010df91
/LLM/main/L0_MergeRequest_PR pipeline #43118 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

venkywonka · 2026-06-13T09:54:17Z

/bot run

tensorrt-cicd · 2026-06-13T10:00:56Z

PR_Github #54039 [ run ] triggered by Bot. Commit: 010df91 Link to invocation

tensorrt-cicd · 2026-06-13T10:21:14Z

PR_Github #54039 [ run ] completed with state FAILURE. Commit: 010df91
/LLM/main/L0_MergeRequest_PR pipeline #43123 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

venkywonka · 2026-06-13T10:28:32Z

/bot run

tensorrt-cicd · 2026-06-13T10:34:15Z

PR_Github #54040 [ run ] triggered by Bot. Commit: 010df91 Link to invocation

tensorrt-cicd · 2026-06-13T10:52:21Z

PR_Github #54040 [ run ] completed with state FAILURE. Commit: 010df91
/LLM/main/L0_MergeRequest_PR pipeline #43124 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

venkywonka · 2026-06-13T12:17:44Z

/bot run

tensorrt-cicd · 2026-06-13T12:23:09Z

PR_Github #54044 [ run ] triggered by Bot. Commit: 010df91 Link to invocation

tensorrt-cicd · 2026-06-13T15:18:26Z

PR_Github #54044 [ run ] completed with state SUCCESS. Commit: 010df91
/LLM/main/L0_MergeRequest_PR pipeline #43128 completed with status: 'SUCCESS'

CI Report

Link to invocation

github-actions Bot assigned venkywonka May 21, 2026

venkywonka added 2 commits June 12, 2026 21:37

[TRTLLM-12842][fix] address telemetry review comments

010df91

Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com>

venkywonka force-pushed the venky/telemetry-config-redesign branch from b9b771d to 010df91 Compare June 13, 2026 04:37

venkywonka merged commit 4e1776a into NVIDIA:main Jun 13, 2026
7 checks passed

coderabbitai Bot mentioned this pull request Jun 22, 2026

[None][feat] Add prefix-aware scheduling config flag to support opt-out #15526

Merged

1 task

coderabbitai Bot mentioned this pull request Jul 9, 2026

[None][feat] BREAKING: add per-model transceiver runtime auto selection #16164

Merged

1 task

Uh oh!

Conversation

venkywonka commented May 21, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Design

Testing

Summary by CodeRabbit

Uh oh!

venkywonka commented May 21, 2026

Uh oh!

venkywonka commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

venkywonka commented May 21, 2026

Uh oh!

venkywonka commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

venkywonka commented May 21, 2026

Uh oh!

venkywonka commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

venkywonka commented May 21, 2026

Uh oh!

venkywonka commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

venkywonka commented May 21, 2026

Uh oh!

venkywonka commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

venkywonka commented Jun 11, 2026

Uh oh!

tensorrt-cicd commented Jun 11, 2026

Uh oh!

tensorrt-cicd commented Jun 12, 2026

Uh oh!

venkywonka commented Jun 12, 2026

Uh oh!

tensorrt-cicd commented Jun 12, 2026

Uh oh!

tensorrt-cicd commented Jun 12, 2026

Uh oh!

venkywonka commented Jun 13, 2026

venkywonka commented May 21, 2026 •

edited by coderabbitai Bot

Loading