[None][infra] Automate and gate LLM args telemetry manifest updates

[venkywonka]

Description

The committed LLM-args telemetry manifest is the reviewable contract for fields that build_capture_manifest() may collect. Its drift test was not enrolled in repository-defined premerge CI, and contributors had no supported regeneration command.

This PR makes that contract deterministic and connects it to telemetry/privacy ownership and the existing LLM API compatibility workflow.

Changes

• Adds scripts/generate_llm_args_golden_manifest.py with deterministic write and --check modes, checkout-local source resolution, and atomic replacement.
• Refreshes the committed manifest and generated telemetry reference from the current capturable LLM API surface; it does not add telemetry collection logic.
• Enrolls the manifest drift test in A10 PyTorch premerge. The gate compares against the PR-built package in-process so CI can use its native bindings and reports the exact regeneration command on drift:
  python3 scripts/generate_llm_args_golden_manifest.py
• Adds concise regeneration and review guidance to AGENTS.md, CODING_GUIDELINES.md, the API-change guide, telemetry documentation, and schema README.
• Routes usage code and tests to @NVIDIA/trt-llm-usage-telemetry-devs; the golden manifest also routes to OSS compliance and the noncommitted API review committee.
• Triggers the existing LLM API compatibility-label check for exact changes to tensorrt_llm/llmapi/llm_args.py or tensorrt_llm/usage/llm_args_golden_manifest.json without broadening it to all of llmapi/.
• Keeps manifest generation out of normal pre-commit while testing that contract explicitly.

Test Coverage

• Final blossom-ci pipeline #46956 passed on commit 8e64daa.
• A10-PyTorch-1 passed 72 tests with 2 skips, including unittest/usage/test_llmapi_config_telemetry_docs.py.
• Check LLM API Compatibility Label passed with the api-compatible label.
• python3 scripts/generate_llm_args_golden_manifest.py --check passed without modifying the manifest.
• Focused generator/gate tests and all applicable pre-commit hooks passed.
• Stage mapping resolves the manifest gate to A10-PyTorch-1 and A10-PyTorch-2.

PR Checklist

• The description explains the intent and implementation.
• Tests cover generation, drift detection, source resolution, and CI enrollment.
• The api-compatible label is applied and CODEOWNERS/docs are updated.
• No new dependency or architecture-diagram change is required.
• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, comment /bot help.

[venkywonka]

/bot run

[tensorrt-cicd]

PR_Github #57853 [ run ] triggered by Bot. Commit: dbfe3c9 Link to invocation

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR adds a deterministic CLI script to generate/check the LLM args telemetry golden manifest, extends the committed golden manifest JSON with new schema fields (KV-cache routing, encoder sizing, sparse attention, quantization), adds corresponding test coverage, and updates developer/telemetry documentation.

Changes

Golden manifest generator, schema, tests, and docs

Layer / File(s)	Summary
Manifest generator script core logic scripts/generate_llm_args_golden_manifest.py	New CLI script renders the manifest deterministically, reads the committed file, supports --check mode with unified diff output, and performs atomic writes preserving file permissions.
Golden manifest schema updates tensorrt_llm/usage/llm_args_golden_manifest.json	Extends TorchLlmArgs and TrtLlmArgs with KV-cache routing, disk/reuse config, encoder sizing, MoE/quant allowlists, and expanded sparse-attention fields; window_size becomes required.
Generator test coverage and CI wiring tests/unittest/usage/test_llmapi_config_telemetry_docs.py, tests/integration/test_lists/test-db/l0_a10.yml	Adds unit tests for canonical writes, diff checks, failure handling, subprocess resolution, and precommit-hook exclusion; updates the golden comparison test to use --check via subprocess and registers the test module in CI.
Developer and telemetry documentation updates AGENTS.md, CODING_GUIDELINES.md, docs/source/_ext/llmapi_config_telemetry.py, docs/source/developer-guide/api-change.md, docs/source/developer-guide/telemetry.md, tensorrt_llm/usage/schemas/README.md	Documents the regeneration workflow, privacy/CODEOWNER review requirements, and expanded captured-field tables for both LlmArgs classes.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CLI as generate_llm_args_golden_manifest.main
  participant Renderer as _render_manifest
  participant FileSystem as llm_args_golden_manifest.json
  CLI->>Renderer: _render_manifest()
  Renderer-->>CLI: generated manifest string
  CLI->>FileSystem: _read_manifest()
  FileSystem-->>CLI: committed manifest string
  alt --check flag
    CLI->>CLI: _check_manifest(committed, generated)
    CLI-->>CLI: print unified diff on mismatch
  else write mode
    CLI->>FileSystem: _write_manifest(generated)
    FileSystem-->>CLI: atomic replace via temp file
  end

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise and clearly summarizes the main change: automating and gating LLM args telemetry manifest updates.
Description check	✅ Passed	The description follows the template with Description, Changes, Test Coverage, and PR Checklist sections filled in.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[tensorrt-cicd]

PR_Github #57853 [ run ] completed with state SUCCESS. Commit: dbfe3c9
/LLM/main/L0_MergeRequest_PR pipeline #46549 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[tensorrt-cicd]

PR_Github #58154 [ run ] completed with state ABORTED. Commit: 7d3ac8b

Link to invocation

[venkywonka]

/bot run

[tensorrt-cicd]

PR_Github #58249 [ run ] triggered by Bot. Commit: 7d3ac8b Link to invocation

[tensorrt-cicd]

PR_Github #58249 [ run ] completed with state SUCCESS. Commit: 7d3ac8b
/LLM/main/L0_MergeRequest_PR pipeline #46890 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[venkywonka]

/bot run

[tensorrt-cicd]

PR_Github #58291 [ run ] triggered by Bot. Commit: 7d72913 Link to invocation

[venkywonka]

/bot run

[tensorrt-cicd]

PR_Github #58293 [ run ] triggered by Bot. Commit: af22012 Link to invocation

[tensorrt-cicd]

PR_Github #58291 [ run ] completed with state ABORTED. Commit: 7d72913

Link to invocation

[venkywonka]

/bot run

[tensorrt-cicd]

PR_Github #58307 [ run ] triggered by Bot. Commit: 8e64daa Link to invocation

[tensorrt-cicd]

PR_Github #58293 [ run ] completed with state ABORTED. Commit: af22012

Link to invocation

[tensorrt-cicd]

PR_Github #58307 [ run ] completed with state SUCCESS. Commit: 8e64daa
/LLM/main/L0_MergeRequest_PR pipeline #46939 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[venkywonka]

/bot run

[tensorrt-cicd]

PR_Github #58326 [ run ] triggered by Bot. Commit: 8e64daa Link to invocation

[tensorrt-cicd]

PR_Github #58326 [ run ] completed with state SUCCESS. Commit: 8e64daa
/LLM/main/L0_MergeRequest_PR pipeline #46956 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

[chang-l]

Heads-up: this gate appears to have landed with a stale golden manifest, and test_build_capture_manifest_matches_committed_golden now fails deterministically on main-based PR runs that select the A10 unittest stage (e.g. PR #15730, L0_MergeRequest_PR build 47040, stage A10-PyTorch-2).

Repro in today's tensorrt-llm-staging/release:main-x86_64 container (TRT-LLM from main, Jul 9): python3 scripts/generate_llm_args_golden_manifest.py --check fails with a 44-line diff — the committed tensorrt_llm/usage/llm_args_golden_manifest.json is missing at least cache_transceiver_config.kv_cache_bounce_size_mb (<class 'int'>, presumably from #15618) and NVFP4_ARC quant-algo entries. The diff is identical under two different installed diffusers versions (0.38/0.39), so it is not an environment artifact — the golden just predates those main changes.

Could you regenerate and commit the manifest (with the telemetry/privacy CODEOWNER approval the gate asks for)? Happy to help verify. cc @venkywonka

[venkywonka]

Hi @chang-l , apologies for the bug - yes its my bad the manifest at the time of merge didn't reflect the latest ToT. fortuantely @2ez4bz came to teh rescue with above PR - which should get merged asap and make the ToT unstale

[venkywonka]

this would require all API-touching PRs to rebase to ToT once above is meged. Small price to pay to ensure better API-churn tracking