[None][fix] return an explicit error if the requests can't be schedul…

[Tabrizian]

Description

In gen-only disaggregated benchmark mode, when the KV cache is too small to hold all benchmark requests, the executor gets stuck: requests waiting for KV cache allocation can never be scheduled because in-progress generation requests hold the cache indefinitely. Previously this caused the benchmark to hang silently with no error.

This PR detects the condition and returns an explicit error to every client, allowing the benchmark to terminate cleanly instead of hanging.

Root cause

In gen-only mode the benchmark fill loop pre-queues all requests before the first forward pass. When the KV cache is insufficient, some requests remain in DISAGG_GENERATION_INIT state permanently because the scheduler cannot allocate KV blocks for them while existing generation requests hold theirs.

Fix (4 files)

1. py_executor.py – After all benchmark requests have been fetched (num_fetch_requests >= benchmark_req_queues_size), detect stuck init requests and call _handle_errors on all active + waiting requests so every client receives an error response.

2. postproc_worker.py – ErrorResponse objects lack a result attribute which crashed the postproc worker. Forward them directly to the proxy instead.

3. result.py – Set _done = True when GenerationResult receives an ErrorResponse so the async result completes instead of looping forever.

4. openai_server.py – StreamingResponse commits HTTP 200 before the generator yields, so raising an exception breaks the connection (ClientPayloadError). Yield an SSE error event instead so the stream terminates cleanly.

Test Coverage

• tests/integration/defs/disaggregated/test_disaggregated.py::test_disaggregated_benchmark_gen_only_insufficient_kv
  – Launches a gen-only disagg cluster with a tiny KV cache (max_tokens=128)
  and 64 concurrent requests, then asserts the gen worker process terminates
  instead of hanging.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for this PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Summary by CodeRabbit

• Bug Fixes

  • Improved error handling for streaming responses; errors now gracefully terminate streams instead of propagating exceptions
  • Added safeguard to prevent indefinite waiting when KV cache is insufficient in generation-only benchmark mode

• Tests

  • Added test coverage for generation-only mode with insufficient KV cache scenarios

[Tabrizian]

/bot run --disable-fail-fast

[Tabrizian]

/bot kill

[tensorrt-cicd]

PR_Github #38999 [ run ] triggered by Bot. Commit: 9c49136 Link to invocation

[tensorrt-cicd]

PR_Github #39000 [ kill ] triggered by Bot. Commit: 9c49136 Link to invocation

[tensorrt-cicd]

PR_Github #39000 [ kill ] completed with state SUCCESS. Commit: 9c49136
Successfully killed previous jobs for commit 9c49136

Link to invocation

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds error handling for gen-only disaggregated KV-cache mode: when benchmark requests exceed KV capacity, the system now detects insufficient allocation, logs an error, and fails all active and waiting requests to prevent hangs. Error responses propagate through the execution pipeline and emit proper HTTP status via SSE events in streaming responses.

Changes

Cohort / File(s)	Summary
Disaggregated KV-cache Error Detection tensorrt_llm/_torch/pyexecutor/py_executor.py	Adds guard to detect when benchmark requests cannot fit in KV cache during gen-only mode; aggregates active and waiting requests, invokes error handling, and clears queue to prevent indefinite waiting.
Error Response Propagation tensorrt_llm/executor/postproc_worker.py, tensorrt_llm/executor/result.py	Adds handling for ErrorResponse objects: postproc_worker immediately appends error responses and skips normal finalization; result.py sets completion flag before invoking error handler.
HTTP Streaming Error Handling tensorrt_llm/serve/openai_server.py	Wraps streaming completion exceptions with explicit catch-as-Exception, logs traceback, and emits SSE data event with error details instead of propagating exception.
Test Coverage tests/integration/defs/disaggregated/test_configs/disagg_config_gen_only_insufficient_kv.yaml, tests/integration/defs/disaggregated/test_disaggregated.py	New test configuration with minimal KV cache and benchmark mode enabled; new test case validates graceful failure when KV cache is insufficient for gen-only benchmark workload.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant Server as openai_server
    participant Executor as py_executor
    participant Worker as postproc_worker
    participant Result as result.py
    
    Client->>Server: POST /completions (stream=true)
    Server->>Executor: Enqueue benchmark request(s)
    
    Executor->>Executor: Check KV cache capacity
    Executor->>Executor: No space for new init requests<br/>& all requests waiting for allocation
    Executor->>Executor: Generate ErrorResponse<br/>for all active & waiting requests
    Executor->>Worker: Pass ErrorResponse
    
    Worker->>Worker: Detect ErrorResponse type
    Worker->>Result: Append error to batch<br/>Skip finalization
    
    Result->>Result: Set _done = True<br/>(mark completion)
    Result->>Result: Invoke error handler
    
    Result->>Server: Error response data
    Server->>Server: Catch Exception<br/>Log traceback
    Server->>Client: SSE data: error object<br/>(HTTP 200 + error event)
    Client->>Client: Process error event<br/>terminate stream

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 3

❌ Failed checks (1 warning, 2 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 27.27% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check	❓ Inconclusive	The title is partially related to the changeset but incomplete—it mentions returning an error for requests that can't be scheduled, which is accurate, but the title is truncated (ends with ellipsis) and doesn't fully summarize the core change.	Complete the truncated title to clearly convey the full message. Example: '[None][fix] Return explicit error when insufficient KV cache prevents scheduling in gen-only disaggregated mode.'
Linked Issues check	❓ Inconclusive	The PR title format shows '[None]' indicating no linked JIRA, NVBugs, or GitHub issue, which may be acceptable for internal fixes but limits traceability.	If this addresses a known issue or bug report, link it in the title using [TRTLLM-XXXX], [#XXXX], or [https://nvbugs/XXXXXXX] format for better tracking.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Out of Scope Changes check	✅ Passed	All changes are focused on fixing the gen-only disaggregated benchmark hang issue and supporting infrastructure (error handling, streaming response termination), with appropriate test coverage.
Description check	✅ Passed	The PR description follows the template structure with all required sections completed: clear explanation of the issue and solution, detailed test coverage, and a checked PR checklist.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

📝 Coding Plan

• Generate coding plan for human review comments

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Tip

You can make CodeRabbit's review stricter and more nitpicky using the `assertive` profile, if that's what you prefer.

Change the reviews.profile setting to assertive to make CodeRabbit's nitpick more issues in your PRs.

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tensorrt_llm/_torch/pyexecutor/py_executor.py`:
- Around line 1798-1813: The benchmark-failure path must fail queued
RequestQueueItem entries separately and stop normal scheduling: gather requests
from self.active_requests and from the RequestQueueItem objects in
self.waiting_queue (e.g., extract the contained LlmRequest from each queued
item), call self._handle_errors(error_msg, requests=all_requests) with that
combined list, clear self.waiting_queue, and then return immediately (do not
return or reuse scheduled_batch) so the scheduler does not continue preparing or
forwarding requests that were just terminated; update the block around
waiting_queue, _handle_errors, active_requests, RequestQueueItem and
scheduled_batch accordingly.

In `@tests/integration/defs/disaggregated/test_disaggregated.py`:
- Around line 712-715: The variable returned from setup_disagg_cluster is bound
to config but never used; rename that binding to _config (or drop it and only
unpack the values you need) where you call setup_disagg_cluster so the tuple
assignment becomes _config, ctx_workers, gen_workers, disagg_server,
server_port, work_dir = setup_disagg_cluster(...); update the surrounding call
site that refers to setup_disagg_cluster to use the new name (or reduced
unpacking) to silence the unused-variable warning.
- Around line 717-753: Replace the blind try/except in send_request with
submitting and collecting futures, then wait on each future with a bounded
timeout and assert the per-request failure contains the expected insufficient-KV
error; specifically, stop swallowing all exceptions in send_request, instead
submit callables via pool.submit into a list (e.g., futures =
[pool.submit(lambda: client.completions.create(...)) for _ in range(64)]), then
iterate over futures and call future.result(timeout=10) inside a try/except that
only catches the specific client/error class (or Exception if unavoidable) and
asserts the exception message includes the insufficient-KV text; keep the
existing gen_workers/process poll check but add these per-request assertions to
ensure each request fails as expected.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: b05bdb21-3dd5-4d61-bd5b-db81578fe077

📥 Commits

Reviewing files that changed from the base of the PR and between 8766e81 and 9c49136.

📒 Files selected for processing (6)

• tensorrt_llm/_torch/pyexecutor/py_executor.py
• tensorrt_llm/executor/postproc_worker.py
• tensorrt_llm/executor/result.py
• tensorrt_llm/serve/openai_server.py
• tests/integration/defs/disaggregated/test_configs/disagg_config_gen_only_insufficient_kv.yaml
• tests/integration/defs/disaggregated/test_disaggregated.py

[Tabrizian]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #39116 [ run ] triggered by Bot. Commit: 108b266 Link to invocation

[tensorrt-cicd]

PR_Github #39116 [ run ] completed with state FAILURE. Commit: 108b266
/LLM/main/L0_MergeRequest_PR pipeline #30375 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[Tabrizian]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #39208 [ run ] triggered by Bot. Commit: bc782b4 Link to invocation

[tensorrt-cicd]

PR_Github #39208 [ run ] completed with state SUCCESS. Commit: bc782b4
/LLM/main/L0_MergeRequest_PR pipeline #30460 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[Tabrizian]

/bot run --disable-fail-fast

[qiaoxj07]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #39430 [ run ] triggered by Bot. Commit: 85cfbe4 Link to invocation

[tensorrt-cicd]

PR_Github #39430 [ run ] completed with state SUCCESS. Commit: 85cfbe4
/LLM/main/L0_MergeRequest_PR pipeline #30657 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[Tabrizian]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #39487 [ run ] triggered by Bot. Commit: 85cfbe4 Link to invocation

[tensorrt-cicd]

PR_Github #39487 [ run ] completed with state SUCCESS. Commit: 85cfbe4
/LLM/main/L0_MergeRequest_PR pipeline #30713 completed with status: 'SUCCESS'

CI Report

Link to invocation

[Tabrizian]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #39623 [ run ] triggered by Bot. Commit: 6af3215 Link to invocation

[tensorrt-cicd]

PR_Github #39623 [ run ] completed with state SUCCESS. Commit: 6af3215
/LLM/main/L0_MergeRequest_PR pipeline #30829 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[qiaoxj07]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #39661 [ run ] triggered by Bot. Commit: 6af3215 Link to invocation

[tensorrt-cicd]

PR_Github #39661 [ run ] completed with state SUCCESS. Commit: 6af3215
/LLM/main/L0_MergeRequest_PR pipeline #30865 completed with status: 'SUCCESS'

CI Report

Link to invocation