[https://nvbugs/6312828][test] Verify PR #16402 H100 Qwen3-32B stress test by chienchunhung · Pull Request #16641 · NVIDIA/TensorRT-LLM

chienchunhung · 2026-07-20T22:51:13Z

Purpose

Test-only H100 mirror for PR #16402. Do not merge this PR.

This branch points to the exact same code tree as PR #16402 at fca8e33571. The current revision adds strict at-most-once behavior for ambiguous GEN failures, immutable shared transfer IDs for safe pre-connect retries, finalized C++ handshake replay rejection, and a bounded agent/NIXL replay-drain window.

Required QA tests

disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] at 10,000 requests and concurrency 512
disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_5_4b_fp8_stress] at 3,000 requests and concurrency 512

Both tests must run on H100 and must complete bounded terminal accounting, post-load readiness probes, and GSM8K recovery. Their results are validation evidence for PR #16402 only.

Related bugs: NVBug 6312828 and NVBug 6479324.

Current Validation

Required head: fca8e33571
Prior H100 helper #60804 / QA #307 ran on stale head 9f11f1b7ac: the 10,000-request Qwen3-32B load terminalized at HTTP, but all 10 follow-on recovery probes timed out and GSM8K never ran.
Current-head H100 rerun is pending.

chienchunhung · 2026-07-20T23:14:52Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

chienchunhung · 2026-07-20T23:15:00Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_5_4b_fp8_stress]

tensorrt-cicd · 2026-07-20T23:21:51Z

PR_Github #60488 [ run ] triggered by Bot. Commit: 82969b4 Link to invocation

tensorrt-cicd · 2026-07-20T23:22:07Z

PR_Github #60489 [ run ] triggered by Bot. Commit: 82969b4 Link to invocation

tensorrt-cicd · 2026-07-20T23:22:09Z

PR_Github #60488 [ run ] completed with state ABORTED. Commit: 82969b4

Link to invocation

tensorrt-cicd · 2026-07-21T01:56:47Z

PR_Github #60489 [ run ] completed with state FAILURE. Commit: 82969b4
LLM_FUNCTION_AUTO_V2C #289 completed with status: 'UNSTABLE'
QA verify test: disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] (NVBug 6479324, branch codex/pr16402-h100-qa, fork chienchunhung, dry_run_close=true)

Link to invocation

chienchunhung · 2026-07-21T02:32:57Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-21T02:39:45Z

PR_Github #60540 [ run ] triggered by Bot. Commit: a5e2892 Link to invocation

tensorrt-cicd · 2026-07-21T05:09:21Z

PR_Github #60540 [ run ] completed with state FAILURE. Commit: a5e2892
LLM_FUNCTION_AUTO_V2C #294 completed with status: 'UNSTABLE'
QA verify test: disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] (NVBug 6479324, branch codex/pr16402-h100-qa, fork chienchunhung, dry_run_close=true)

Link to invocation

chienchunhung · 2026-07-21T08:19:40Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-21T08:26:21Z

PR_Github #60648 [ run ] triggered by Bot. Commit: 0983e24 Link to invocation

tensorrt-cicd · 2026-07-21T08:26:23Z

PR_Github #60648 [ run ] completed with state FAILURE. Commit: 0983e24

Link to invocation

chienchunhung · 2026-07-21T08:29:38Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-21T08:36:16Z

PR_Github #60651 [ run ] triggered by Bot. Commit: 0983e24 Link to invocation

tensorrt-cicd · 2026-07-21T08:40:48Z

PR_Github #60651 [ run ] completed with state FAILURE. Commit: 0983e24
LLM_FUNCTION_AUTO_V2C #297 completed with status: 'UNSTABLE'
QA verify test: disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] (NVBug 6479324, branch codex/pr16402-h100-qa, fork chienchunhung, dry_run_close=true)

Link to invocation

chienchunhung · 2026-07-21T22:47:21Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-21T22:53:04Z

PR_Github #60804 [ run ] triggered by Bot. Commit: 9f11f1b Link to invocation

tensorrt-cicd · 2026-07-22T02:34:43Z

PR_Github #60804 [ run ] completed with state FAILURE. Commit: 9f11f1b
LLM_FUNCTION_AUTO_V2C #307 completed with status: 'UNSTABLE'
QA verify test: disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] (NVBug 6479324, branch codex/pr16402-h100-qa, fork chienchunhung, dry_run_close=true)

Link to invocation

chienchunhung · 2026-07-22T04:12:24Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-22T04:20:50Z

PR_Github #60878 [ run ] triggered by Bot. Commit: fca8e33 Link to invocation

tensorrt-cicd · 2026-07-22T06:03:24Z

PR_Github #60878 [ run ] completed with state FAILURE. Commit: fca8e33
LLM_FUNCTION_AUTO_V2C #312 completed with status: 'UNSTABLE'
QA verify test: disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] (NVBug 6479324, branch codex/pr16402-h100-qa, fork chienchunhung, dry_run_close=true)

Link to invocation

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung · 2026-07-23T01:08:28Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-23T01:15:07Z

PR_Github #61140 [ run ] triggered by Bot. Commit: 3c7358c Link to invocation

tensorrt-cicd · 2026-07-23T01:25:39Z

PR_Github #61140 [ run ] completed with state FAILURE. Commit: 3c7358c
LLM_FUNCTION_AUTO_V2C #314 completed with status: 'UNSTABLE'
QA verify test: disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] (NVBug 6312828, branch codex/pr16402-h100-qa, fork chienchunhung, dry_run_close=true)

Link to invocation

github-actions Bot assigned chienchunhung Jul 20, 2026

chienchunhung mentioned this pull request Jul 20, 2026

[https://nvbugs/6312828][fix] Recover disaggregated requests after timeouts #16402

Draft

chienchunhung changed the title ~~[https://nvbugs/6479324][test] Verify PR #16402 H100 stress tests~~ [NVBUG 6479324][test] Verify PR #16402 H100 stress tests Jul 21, 2026

chienchunhung changed the title ~~[NVBUG 6479324][test] Verify PR #16402 H100 stress tests~~ [https://nvbugs/6479324][test] Verify PR #16402 H100 stress tests Jul 21, 2026

chienchunhung force-pushed the codex/pr16402-h100-qa branch from 0983e24 to 9f11f1b Compare July 21, 2026 21:04

chienchunhung changed the title ~~[https://nvbugs/6479324][test] Verify PR #16402 H100 stress tests~~ [NVBUG 6479324][test] Verify PR #16402 H100 stress tests Jul 22, 2026

chienchunhung changed the title ~~[NVBUG 6479324][test] Verify PR #16402 H100 stress tests~~ [https://nvbugs/6479324][test] Verify PR #16402 H100 stress tests Jul 22, 2026

chienchunhung added 2 commits July 22, 2026 18:01

[NVBUG 6312828][fix] Terminalize disaggregated request timeouts

41ce7cb

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][test] Bound post-stress recovery concurrency

65dd09f

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung added 7 commits July 22, 2026 18:01

[NVBUG 6312828][test] Gate post-stress recovery readiness

0e79946

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][fix] Reap idle context transfers safely

98b7544

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][fix] Coordinate default-off transfer cleanup

54e5405

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][fix] Reject replayed KV handshakes

3b32095

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][test] Correct transfer cleanup assertions

d0259ed

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][fix] Repair focused CI regressions

31a2435

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][fix] Bound pre-handshake cancellation state

3c7358c

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung force-pushed the codex/pr16402-h100-qa branch from fca8e33 to 3c7358c Compare July 23, 2026 01:07

chienchunhung changed the title ~~[https://nvbugs/6479324][test] Verify PR #16402 H100 stress tests~~ [https://nvbugs/6312828][test] Verify PR #16402 H100 Qwen3-32B stress test Jul 23, 2026

Uh oh!

Conversation

chienchunhung commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Required QA tests

Current Validation

Uh oh!

chienchunhung commented Jul 20, 2026

Uh oh!

chienchunhung commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

chienchunhung commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

chienchunhung commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

chienchunhung commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

chienchunhung commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

chienchunhung commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

chienchunhung commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chienchunhung commented Jul 20, 2026 •

edited

Loading