[https://nvbugs/6215218][fix] Add escape hatch for KV transfer timeout-triggered UAF in disagg

[hyukn]

Background

NVBug 6215218 reports that disaggregated serving of Kimi-K2.5-NVFP4
(ISL=OSL=1024, concurrency=16384) on GB200 (NVL72, aarch64, dep16)
crashes the CTX worker shortly after benchmark start. The CTX worker
dies with SIGSEGV (exit 139); the disagg server then fails all
subsequent requests with aiohttp.ClientConnectorError and the SLURM
job eventually times out. The same configuration runs cleanly on
nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc12.post1 — so this is a
regression introduced between rc12.post1 and main.

Empirical findings (lyris GB200 dep16, c=16384)

Run	Image	Buffer	kv_transfer_timeout_ms	Result
BAD baseline	main c7d609bd (image 57a1b8)	16384	60000	FAILED, cuBLAS cascade at 14724/16384
GOOD baseline	1.3.0rc12.post1	16384	60000	DONE, 16384/16384 in 181 s
Reduced buffer	BAD	4096	60000	FAILED, Python SIGSEGV at 15036/16384
Newer main	9ec3c8 (May 23, 2 days after BAD)	16384	60000	FAILED, SIGSEGV preceded by Terminating ... due to KV cache transfer timeout
This fix	BAD	16384	None	No crash, no errors, 16305/16384 (99.5%) before SLURM wall-clock

Two key observations:

1. The crash is request-count-correlated (~90% of total), regardless of buffer size — it's a resource leak / use-after-free that accumulates per request, not a backpressure issue.
2. The crash is always preceded (in the runs that get there) by [RANK X] Terminating ... request ... due to KV cache transfer timeout in the CTX worker log. The SEGV stack is consistently _PyObject_GenericGetAttrWithDict → PyObject_GetAttr → _PyEval_EvalFrameDefault — classic Python attribute access on a freed object. Under different timing, the same race surfaces as CUBLAS_STATUS_EXECUTION_FAILED from cublasLtMatmul (latched async CUDA error on the stream from the same freed-pointer access).

Summary

This patch adds an environment-variable escape hatch (TRTLLM_DISABLE_KV_TRANSFER_TIMEOUT=1) to _check_kv_transfer_timeout and documents the workaround in CacheTransceiverConfig.kv_transfer_timeout_ms. Users who hit the crash can disable the timeout via either:

• YAML config: cache_transceiver_config.kv_transfer_timeout_ms: null (already supported)
• Env var: TRTLLM_DISABLE_KV_TRANSFER_TIMEOUT=1 (added by this PR)

Why this is an interim fix

This is not a fix for the underlying use-after-free. It is an escape hatch so users hitting NVBug 6215218 can run their workloads while the proper repair lands. The real fix requires auditing the cancellation / cleanup chain that runs when a request is flagged py_kv_transfer_timed_out=True. Suspect code areas (introduced between v1.3.0rc12.post1 and c7d609bd):

• _flush_pending_transfer_responses / _pending_transfer_responses — added in PR [TRTLLM-12015][feat] Introduce KV reuse in transceiver v2 #13115 (TRTLLM-12015, KV reuse in transceiver v2). This is a TP gather collective that requires lockstep across DP ranks; a response queued for a soon-to-be-terminated request can outlive the request itself.
• commit_blocks_for_reuse(req) on KvCacheTransceiver — same PR. Per-request callback during transfer completion; verify pybind/nanobind INCREF discipline.
• cancelled_reqs tuple-result handling in _check_disagg_gen_cache_transfer_status — added in PR [TRTLLM-11635][feat] Introduce cancellation in transceiver v2 #12734 (TRTLLM-11635, cancellation in transceiver v2). Dereferences req.py_request_id / is_child / parent_request_id on objects coming back from C++; verify the list holds strong references.

A git bisect between v1.3.0rc12.post1 and c7d609bd with the dep16 c=16384 repro as the pass/fail check would conclusively identify the introducing PR (~14 bisect steps).

Impact assessment

• Functional: zero behavior change when the env var is unset and kv_transfer_timeout_ms is left at its default. Sets are purely opt-in.
• Performance: zero — one extra os.environ.get per _check_kv_transfer_timeout call (which is itself called from the executor event loop, not the hot path).
• API: schema field default unchanged. Only the description string is updated.

Test plan

• Empirically verified the workaround on lyris GB200 dep16 c=16384 (job 1907412): bench progressed to 99.5% with zero CTX errors before SLURM wall-clock.
• Pre-commit hooks pass on changed files.
• Reviewer to confirm the env-var approach is acceptable as an interim while the UAF repair is being scoped.

Related

• NVBug: https://nvbugspro.nvidia.com/bug/6215218
• Suspect PRs introducing the regression window: [TRTLLM-12015][feat] Introduce KV reuse in transceiver v2 #13115, [TRTLLM-11635][feat] Introduce cancellation in transceiver v2 #12734

Summary by CodeRabbit

Release Notes

• Bug Fixes

  • Added workaround for KV-cache transfer timeout issues that could cause crashes in high-concurrency disaggregated workloads. Set TRTLLM_DISABLE_KV_TRANSFER_TIMEOUT=1 or configure the timeout to None to resolve the issue.

• Documentation

  • Updated KV transfer timeout configuration documentation with known high-concurrency issue details and recommended solutions.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9ebdcdbe-e9dc-49be-8f3f-794e24d4617e

📥 Commits

Reviewing files that changed from the base of the PR and between a9c35f3 and 620fb4c.

📒 Files selected for processing (2)

• tensorrt_llm/_torch/pyexecutor/py_executor.py
• tensorrt_llm/llmapi/llm_args.py

---

📝 Walkthrough

Walkthrough

This PR introduces a workaround for a known use-after-free issue in KV cache transfer timeout detection under high concurrency in disaggregated deployments. It adds an environment-variable escape hatch and updates the configuration documentation to guide users toward the mitigation.

Changes

KV Cache Transfer Timeout Workaround

Layer / File(s)	Summary
Environment variable guard and configuration documentation tensorrt_llm/llmapi/llm_args.py, tensorrt_llm/_torch/pyexecutor/py_executor.py	Field documentation for kv_transfer_timeout_ms is expanded to document the disaggregated-mode high-concurrency use-after-free issue and recommend setting timeout to None or TRTLLM_DISABLE_KV_TRANSFER_TIMEOUT=1. The _check_kv_transfer_timeout method adds an early-return guard when the environment variable is set to "1", bypassing timeout-based termination logic.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Suggested reviewers

• fredricz-20070104
• xinhe-nv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title clearly identifies the main change: adding an escape hatch for a KV transfer timeout-related use-after-free (UAF) issue in disaggregated serving, with a specific NVBugs reference.
Description check	✅ Passed	The PR description is comprehensive and well-structured, including background, empirical findings, summary, rationale, impact assessment, and test plan. All key sections are thoroughly filled out with relevant details.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[hyukn]

Closing as redundant: the proposed env var (TRTLLM_DISABLE_KV_TRANSFER_TIMEOUT=1) duplicates an already-existing knob — cache_transceiver_config.kv_transfer_timeout_ms: null in YAML produces the same effect (verified empirically on lyris GB200 dep16 c=16384: bench reaches 99.5% with zero CTX errors).

This patch was a workaround, not a fix. The proper repair is to address the use-after-free in the cancellation/cleanup chain that runs when a request is flagged py_kv_transfer_timed_out=True. Suspect upstream code (between v1.3.0rc12.post1 and ~c7d609bd):

• [TRTLLM-12015][feat] Introduce KV reuse in transceiver v2 #13115 (TRTLLM-12015) — _flush_pending_transfer_responses / commit_blocks_for_reuse
• [TRTLLM-11635][feat] Introduce cancellation in transceiver v2 #12734 (TRTLLM-11635) — cancelled_reqs tuple handling in disagg cleanup

Full repro + evidence on the NVBug: https://nvbugspro.nvidia.com/bug/6215218