[TRTLLM-12648][test] enable disagg cancellation stress test

[chienchunhung]

Summary

Enables the disaggregated cancellation stress-test entry as a real regular guardrail while keeping the full cancellation/poison marathon behind an explicit YAML mode switch.

This change ships:

• Real cluster setup: DisaggCancellationStressHarness.setup() now calls the shared setup_disagg_cluster(..., save_log=True), strips the harness-only stress_config block before passing YAML to trtllm-serve, binds the disagg server endpoint, tracks worker/server logs, and tears the cluster down.
• Default regular-protection mode: stress_config.mode: log_only launches the real 3P3D C++/V1 DeepSeek disagg cluster, sends normal non-cancel completion probes, and fails on UAF, broken-promise, and segmentation-fault-class log signatures.
• Opt-in full-stress mode: stress_config.mode: full_cancel_poison remains the explicit switch for cancellation load, fault injection, token-equivalent canaries, metrics, and KV-growth checks once in-flight cancellation plus poison-buffer hardening are available.
• QA stress-list registration: the C++/V1 DeepSeek entry stays registered in tests/integration/test_lists/qa/llm_function_stress.txt, now with TIMEOUT (45) for the 10-minute regular guardrail plus setup margin. This makes the test eligible for the QA/Jenkins job that consumes that list; it does not create or modify that job schedule.
• Run instructions: the README and config README now describe scheduled-run setup, manual runs, bot trigger flow, and the exact mode-switch checklist.

The registered entry is:

stress_test/disagg_cancel/test_disagg_cancel_stress.py::test_disagg_cancellation_marathon[marathon_cpp_v1_deepseek.yaml] TIMEOUT (45)

Implementation chain

Change	Adds	Assertion it enables
skeleton + log scanner	harness lifecycle + log_scanner_thread	hard-zero log patterns
metrics thread	metrics_thread	KV-cache utilization growth bound
injector thread	injector_thread	SIGSTOP / SIGCONT / SIGKILL+respawn; injection_events count
canary thread	canary_thread	canary correctness plus error-rate / recovery-time inputs
load thread	load_thread	sustained steady/burst cancellation load over the configured duration
this change	real setup, log_only regular guardrail, full-mode switch, QA registration, README	scheduled/manual protection against UAF, broken promises, and segfault-class failures today

There are no remaining test-logic or test-enablement steps in this chain. Future runtime support can flip the YAML from log_only to full_cancel_poison and add canary references/poison expectations without another harness-architecture step.

Mode switch

Current regular CI mode:

stress_config:
  mode: log_only
  duration_min: 10

log_only is intentionally the checked-in default. It does not require in-flight cancellation or poison-buffer support. It provides regular protection by:

• launching the real disaggregated cluster,
• sending normal non-cancel completion probes through /v1/completions,
• scanning saved context-worker, generation-worker, and disagg-server logs.

To switch to the full cancellation/poison marathon after runtime support is ready:

1. Set stress_config.mode: full_cancel_poison.
2. Set stress_config.duration_min: 120.
3. Keep or tune base_concurrency, bursts, and injections.
4. Add configs/stress_canary_prompts.json with recorded token references.
5. Update poison-buffer log expectations to match the finalized runtime behavior.
6. Raise the test-list timeout back to a full-marathon budget, for example TIMEOUT (150).

Scheduled run config and setup

The QA stress-list registration is controlled by:

tests/integration/test_lists/qa/llm_function_stress.txt

Once merged, this test runs automatically only when the existing QA/Jenkins schedule or release validation job consumes llm_function_stress.txt. The exact cadence and wall-clock start time are owned by QA CI configuration outside this PR. The in-repo QA README describes QA coverage as daily/release and weekly/release/on-demand, but does not define a file-specific cadence for this stress list.

CI should run that list from tests/integration/defs with the normal TRT-LLM integration container or virtual environment:

pytest --test-list=../test_lists/qa/llm_function_stress.txt \
  --output-dir=<ci-output-dir> \
  -s -v

Required setup:

• GPU integration environment with trtllm-serve on PATH.
• LLM_MODELS_ROOT set so DeepSeek-V3-Lite/bf16 resolves to local model weights.
• Enough GPUs for the configured 3 context + 3 generation worker shape.

Manual run and trigger

Manual regular guardrail run:

cd /path/to/TensorRT-LLM/tests/integration/defs
export LLM_MODELS_ROOT=/path/to/model/root

pytest stress_test/disagg_cancel/test_disagg_cancel_stress.py \
  --test-list=../test_lists/qa/llm_function_stress.txt \
  --output-dir=/tmp/trtllm-disagg-cancel-stress \
  -s -v

Collection-only check from a full integration environment:

pytest stress_test/disagg_cancel/test_disagg_cancel_stress.py \
  --test-list=../test_lists/qa/llm_function_stress.txt \
  --output-dir=/tmp/trtllm-disagg-cancel-stress \
  -s --co -q

Manual CI trigger:

/bot help
/bot run --extra-stage "<QA stress stage that runs llm_function_stress.txt>"

The exact QA stress stage name is owned by CI/Jenkins configuration and is not declared in this directory.

Validation

Syntax:

PYTHONPYCACHEPREFIX=$PWD/.pycache python -m py_compile \
  tests/integration/defs/stress_test/disagg_cancel/harness.py \
  tests/integration/defs/stress_test/disagg_cancel/test_disagg_cancel_stress.py

Result: passed.

No-GPU component suite:

PYTHONPATH=tests/integration/defs:tests/integration/defs/disaggregated \
python -m pytest -c /dev/null -o addopts= \
  --confcutdir=tests/integration/defs/stress_test \
  tests/integration/defs/stress_test/disagg_cancel/ \
  -k "not test_disagg_cancellation_marathon" -q

Result: 92 passed, 1 deselected.

Static test-list validation:

python scripts/check_test_list.py --validate

Result: OK: 2563 unique test entries validated.

Also ran git diff --check.

Pre-commit on touched files passes formatting/lint hooks. The local waive list check and validate-test-lists hooks fail only because their hook interpreter cannot parse str | None in scripts/check_test_list.py; the same validator command above passes under the project Python.

The full pytest --test-list=... --co command was not runnable in this local Python environment because the integration conftest.py imports torch._inductor, which is not present locally.

Out of scope

• Flipping the checked-in mode to full_cancel_poison before runtime support is ready.
• Adding canary reference JSON generation/fixtures.
• Registering additional marathon YAMLs.

Summary by CodeRabbit

• Documentation

  • Expanded disaggregated cancellation stress suite documentation with detailed explanations of test modes, run procedures, and debugging guidance.

• New Features

  • Introduced explicit test modes (log_only and full_cancel_poison) for the stress test harness, including configuration support and mode-specific execution logic.

• Tests

  • Added test coverage for mode validation and log-only thread execution.

[chienchunhung]

/bot run --disable-fail-fast

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d5bf71f3-2f99-48e9-a58e-55ff577e5ecd

📥 Commits

Reviewing files that changed from the base of the PR and between e0a909a and 25347de.

📒 Files selected for processing (7)

• tests/integration/defs/stress_test/disagg_cancel/README.md
• tests/integration/defs/stress_test/disagg_cancel/configs/README.md
• tests/integration/defs/stress_test/disagg_cancel/configs/marathon_cpp_v1_deepseek.yaml
• tests/integration/defs/stress_test/disagg_cancel/configs/marathon_python_v2_qwen.yaml
• tests/integration/defs/stress_test/disagg_cancel/harness.py
• tests/integration/defs/stress_test/disagg_cancel/test_disagg_cancel_stress.py
• tests/integration/test_lists/qa/llm_function_stress.txt

---

📝 Walkthrough

Walkthrough

This PR introduces a bi-modal disaggregated cancellation stress harness that runs either in lightweight log-only probe mode (for guardrail checks) or full cancellation/poison mode (for comprehensive stress testing). The harness is enhanced with real cluster orchestration, mode-driven thread composition, comprehensive documentation, and new test coverage.

Changes

Disaggregated Stress Test Mode System

Layer / File(s)	Summary
Mode Architecture & Config Contract tests/integration/defs/stress_test/disagg_cancel/harness.py	Mode constants (log_only, full_cancel_poison) are introduced with validation rules, mode-coercion in config loading, and Boolean properties (is_log_only, is_full_cancel_poison) for conditional harness behavior.
Harness Cluster Orchestration & Lifecycle tests/integration/defs/stress_test/disagg_cancel/harness.py	Cluster setup is implemented to launch processes, reconstruct per-worker metadata, and bind server endpoints; _teardown_cluster() transitions from no-op to real termination and directory cleanup.
Mode-Driven Thread Spawning tests/integration/defs/stress_test/disagg_cancel/harness.py	Thread orchestration (start()) branches by mode: log-only mode starts a dedicated probe thread plus log scanner; full mode starts load, canary, injector, and metrics threads.
Log-Only Mode Probe Execution tests/integration/defs/stress_test/disagg_cancel/harness.py	Log-only thread body implements probe scheduling with duration override support, sends periodic deterministic requests, appends canary records, and signals failures via mark_failed.
Log Scanning Server Log Integration tests/integration/defs/stress_test/disagg_cancel/harness.py	Log scanner extends to optionally tail disagg server logs in addition to worker logs, with corresponding messaging updates.
Test Validation & Mode Coverage tests/integration/defs/stress_test/disagg_cancel/test_disagg_cancel_stress.py	Parametrized mode validation tests confirm supported modes are accepted and unknown modes rejected; thread-level test exercises _log_only_thread_body with monkeypatching; marathon test updates use computed timeouts and verify log-only probe success.
YAML Configuration & Comprehensive Documentation tests/integration/defs/stress_test/disagg_cancel/{README.md,configs/README.md,configs/marathon_*.yaml}, tests/integration/test_lists/qa/llm_function_stress.txt	Config YAMLs declare explicit run modes and log-only probe parameters; config README documents mode semantics and switching checklists; main README expands with mode-switch workflows, scheduled QA procedures, manual run instructions, and concrete debug guidance; test is registered in QA test list with 45-second timeout.

---

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

---

Possibly related PRs

• NVIDIA/TensorRT-LLM#15124: Both PRs modify tests/integration/defs/stress_test/disagg_cancel/harness.py for disagg cancellation harness; main PR adds mode-driven orchestration and log-only probe execution, while #15124 implements the load-thread duration/shape logic.
• NVIDIA/TensorRT-LLM#15015: Both PRs enhance threading in the disagg cancellation harness; main PR adds log-only probe thread and mode-based thread composition, while #15015 focuses on canary-thread probing and recording.

---

Suggested reviewers

• pcastonguay
• StanleySun639
• xinhe-nv
• jieli-matrix

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title '[TRTLLM-12648][test] enable disagg cancellation stress test' clearly and concisely summarizes the main change: enabling a disaggregated cancellation stress test as a guardrail.
Description check	✅ Passed	The PR description comprehensively covers the purpose, implementation details, mode-switch instructions, scheduled/manual run configs, and validation steps. It fully addresses the template's requirements for what, why, and testing.
Docstring Coverage	✅ Passed	Docstring coverage is 96.67% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[tensorrt-cicd]

PR_Github #53176 [ run ] triggered by Bot. Commit: 25347de Link to invocation

[tensorrt-cicd]

PR_Github #53176 [ run ] completed with state SUCCESS. Commit: 25347de
/LLM/main/L0_MergeRequest_PR pipeline #42378 completed with status: 'SUCCESS'

CI Report

Link to invocation