From ab0991001b2d10da38ce80353a9238f7e91b4ef5 Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Tue, 31 Mar 2026 14:34:25 -0700 Subject: [PATCH 1/2] [https://nvbugs/6000658][fix] Fix disagg gen-only hang where 10s sleep in can_forward gate blocks KV transfers and overflows CTX KV cache In disaggregated gen-only benchmarking mode with ADP enabled, the can_forward gate polls with time.sleep(10) while waiting for enough generation requests to accumulate. During this 10s sleep, GEN does not call _prepare_and_schedule_batch(), so it never posts receiveAsync() for incoming UCX KV transfers from CTX servers. Since UCX uses a rendezvous protocol, CTX's sendAsync() futures block until GEN posts a matching receive. With CTX running TP=1 (full model on one GPU), the KV cache is small (~32 GiB on GB200 after loading the 132 GiB model). The stalled transfers fill up CTX's KV cache within ~5 seconds, preventing CTX from prefilling new requests. GEN then never reaches the can_forward threshold, causing a deadlock. This issue does not affect GB300 because its larger GPU memory (~288 GiB vs 184 GiB) allows a much larger CTX KV cache, which can absorb the full 10s backpressure without overflowing. For most benchmark YAML configs in this repo, the CTX KV cache is large enough (e.g., using higher CTX TP, smaller models, or shorter ISL) that the 10s sleep does not trigger the overflow. The issue specifically manifests on GB200 with large models (e.g., Qwen3-235B), long input sequences, and CTX TP=1. Fix: reduce the ADP waiting sleep from 10s to 0.1s. This allows GEN to post receives frequently enough that CTX's KV cache never overflows. The 0.1s interval provides 49x margin over the theoretical CTX KV overflow threshold (~4.9s for Qwen3-235B with ISL=1024 on GB200). Validated on Lyris GB200 with Qwen3-235B-FP4, 4 CTX servers, concurrency=4096: - sleep(10): deadlock (73 CTX timeouts, 56 "not enough kvCache" warnings) - sleep(0.1): success (0 warnings, E2EL=57076ms) The other two time.sleep(10) calls in this function are intentional: - Line 2061: stabilization wait after can_forward becomes True - Line 2076: non-ADP path (no UCX rendezvous issue without ADP) Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- tensorrt_llm/_torch/pyexecutor/py_executor.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tensorrt_llm/_torch/pyexecutor/py_executor.py b/tensorrt_llm/_torch/pyexecutor/py_executor.py index f3c31d03c244..b45869a63323 100644 --- a/tensorrt_llm/_torch/pyexecutor/py_executor.py +++ b/tensorrt_llm/_torch/pyexecutor/py_executor.py @@ -2062,10 +2062,10 @@ def _executor_loop_overlap(self): else: if self.dist.rank == 0: logger.info( - f"sleep 10 seconds, num_fetched_requests: {self.num_fetch_requests}, " + f"sleep 0.1 seconds, num_fetched_requests: {self.num_fetch_requests}, " f"total_gen_count: {total_gen_count}, " f"scheduled_gen_batch: {local_gen_count}") - time.sleep(10) + time.sleep(0.1) continue else: if scheduled_batch.num_generation_requests < self.benchmark_req_queues_size: From feac977fa6000b3442d2c970bc0d9a1e4bccfbed Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Wed, 1 Apr 2026 17:02:11 -0700 Subject: [PATCH 2/2] [https://nvbugs/6000658][fix] Reduce fill loop sleep from 1s to 0.1s to prevent KV transfer backpressure The benchmark disagg fill loop inside _prepare_and_schedule_batch() retries with time.sleep(1) when the request queue is not yet full. During this sleep, GEN does not process KV transfers or post receiveAsync(), creating the same backpressure risk as the can_forward gate's sleep(10) fixed in the previous commit. For tight configs (e.g., ISL=8192 with CTX TP=1), the CTX KV overflow threshold can be as low as ~0.75s, making sleep(1) borderline unsafe. Reduce to sleep(0.1) for consistency with the can_forward gate fix. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- tensorrt_llm/_torch/pyexecutor/py_executor.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tensorrt_llm/_torch/pyexecutor/py_executor.py b/tensorrt_llm/_torch/pyexecutor/py_executor.py index b45869a63323..738c42da17c7 100644 --- a/tensorrt_llm/_torch/pyexecutor/py_executor.py +++ b/tensorrt_llm/_torch/pyexecutor/py_executor.py @@ -1707,7 +1707,7 @@ def _prepare_and_schedule_batch(self): new_requests += iter_requests self.hang_detector.checkpoint() if self.num_fetch_requests < fill_target: - time.sleep(1) + time.sleep(0.1) iter_stats = None if self.enable_iter_perf_stats: