[TRTLLM-12648][test] implement disagg cancellation canary thread

[chienchunhung]

Summary by CodeRabbit

• New Features

  • Stress testing harness now includes canary workload capability to detect potential runtime issues through deterministic completion testing.
  • Added configurable parameters for request timeouts and inter-request intervals.
  • New method to bind the stress test harness to server endpoints.

• Documentation

  • Updated test documentation to reflect canary and metrics thread implementation.

• Tests

  • Added comprehensive integration test suite for canary functionality.

Summary

Fourth thread body in the disaggregated cancellation stress-test suite — the marathon-style (hours-running) harness that gates regressions of the bug class fixed by PR #13713. The skeleton + log_scanner_thread landed in PR #14375, the metrics_thread in PR #14807, and the injector_thread in PR #14920; this PR adds the canary_thread.

This PR ships:

• canary_thread body — loads canary.prompts_file (resolved relative to the marathon YAML), then sends deterministic greedy completions to the disagg server's /v1/completions at canary.rate_per_min cadence and appends a per-request record to _canary_records: {timestamp, elapsed_s, prompt_index, success, token_equivalent, latency_s, error}. Request failures are recorded — not fail-fast — because error spikes during bursts / injections are expected; only the log scanner is fail-fast. The end-of-marathon gates (canary error rate, recovery time) are computed from these records in a later step.
• _send_canary_request — urllib POST with temperature=0.0 + fixed seed for greedy determinism and detokenize=False so the response carries generated token_ids on choices[0] (see CompletionResponseChoice.token_ids in tensorrt_llm/serve/openai_protocol.py). Every transport/parse failure (HTTP error, connection refused, timeout, malformed body) folds into (None, None, error_string).
• _load_canary_prompts — validates the stress_canary_prompts.json schema ({"prompts": [{"prompt", "reference_token_ids", "reference_text?"}, ...]}) and raises a clear ValueError on malformed input.
• _tokens_equivalent — exact token-ID list equality; a None on either side (no tokens returned, or no reference recorded) is non-equivalent.
• bind_server_endpoint(server_url, model_name) — registers the disagg front-end the canary targets, mirroring bind_tracked_workers. Called by setup() once setup_disagg_cluster returns; settable by tests today.
• canary_request_timeout_s / canary_interval_s — new ctor parameters on DisaggCancellationStressHarness, matching the existing per-thread tunable pattern (log_scanner_poll_interval_s, metrics_scrape_interval_s, injector_poll_interval_s). Tests pass small values; the marathon derives the interval from canary.rate_per_min.
  The thread sits inside the lifecycle wired up by PR [TRTLLM-12648][test] add disagg cancellation stress-test harness skeleton #14375 — start() already spawns it, stop() joins it, collect_results() already returns canary_records. Previous PRs left it as a no-op stub; this PR makes it actually send canaries.

PR chain plan

PR	Adds	Marathon assertion it enables
PR #14375	skeleton + log_scanner	hard-zero log patterns
PR #14807	metrics_thread	KV-cache utilization growth bound
PR #14920	injector_thread	SIGSTOP / SIGCONT / SIGKILL+respawn; injection_events count
this PR	canary_thread	canary correctness (token-equivalence) + error-rate / recovery-time inputs
next	load_thread + real setup() with save_log=True	sustained load over duration_min; marathon entry point becomes a real marathon

The final PR flips setup_disagg_cluster to launch workers with explicit per-worker ports + save_log=True and calls bind_tracked_workers() / bind_server_endpoint() — at which point the canary thread streams against a live server. Today it correctly no-ops when its inputs are absent: with the cluster launch still stubbed, no server endpoint is bound and the thread logs a warning and exits.

Why urllib, not asyncio

The class docstring previously said the canary thread would run its own asyncio loop. At 5 req/min (one request at a time) asyncio buys nothing and complicates testing, so the canary uses blocking urllib like the merged metrics_thread — which makes it unit-testable against an in-process http.server. The class docstring is updated accordingly; the load thread (next PR) still runs asyncio internally via run_cancel_stress_test.

Tests added

All deterministic, GPU-free, second-scale. Under tests/integration/defs/stress_test/disagg_cancel/.

• test_canary.py — 21 unit tests across three layers:
  • Loader (5): valid file; invalid JSON / missing prompts / non-list prompts / entry missing string prompt all raise ValueError.
  • Token-equivalence (3): exact match, mismatch (value + length), None-on-either-side.
  • Send-request (4, in-process HTTP server): success returns token_ids; HTTP 503 → http_error; connection refused → url_error; malformed body → parse_error.
  • Thread-body integration (9): token_equivalent True/False/None (match / mismatch / no-reference / check disabled), error recorded when server down, prompt round-robin, and prompt exit on no endpoint / missing prompts file / failed_event.

Run locally (no GPU, no cluster):

    PYTHONPATH=tests/integration/defs:tests/integration/defs/disaggregated \
    python3 -m pytest -c /dev/null -o addopts= \
      --confcutdir=tests/integration/defs/stress_test \
      tests/integration/defs/stress_test/disagg_cancel/test_canary.py -v

Out of scope (deferred to later PRs in this chain)

• The remaining load_thread body + real setup_disagg_cluster invocation (cluster launch is still a stub, so no server endpoint is bound in the lifecycle smoke; the canary thread logs a warning and exits).
• End-of-marathon gate computation from _canary_records (overall error rate < 1%, per-injection-window < 10%, recovery time < 30 s).
• configs/stress_canary_prompts.json + tools/generate_canary_references.py (Step 6 — one-shot greedy-decode reference generation).
• YAML schema wiring of canary_request_timeout_s / canary_interval_s (constructor-only for now; the marathon derives cadence from canary.rate_per_min).
• Registration in tests/integration/test_lists/qa/llm_function_stress.txt for nightly / weekly CI.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[chienchunhung]

/bot run --disable-fail-fast

[coderabbitai]

Lost in the diff? Review this PR in Change Stack to follow the change map from intent to exact ranges.

📝 Walkthrough

Walkthrough

This PR implements a canary thread for the disaggregated cancellation stress harness that sends deterministic greedy OpenAI-compatible completions and records per-request outcomes with optional token-equivalence validation. The work spans harness helpers, configuration, thread implementation, and comprehensive integration tests.

Changes

Canary Thread Implementation

Layer / File(s)	Summary
Canary helper functions and harness dependencies tests/integration/defs/stress_test/disagg_cancel/harness.py	Introduces _load_canary_prompts for strict JSON schema validation of prompt files, _send_canary_request for OpenAI-style /v1/completions POST requests with deterministic parameters (temperature=0.0, stream=False, detokenize=False), and _tokens_equivalent for exact token-ID matching. Adds import json to support these utilities. Class docstring updated to document threading model.
Harness configuration and server binding tests/integration/defs/stress_test/disagg_cancel/harness.py	Extends DisaggCancellationStressHarness.__init__ with keyword-only parameters canary_request_timeout_s and canary_interval_s to control HTTP timeouts and inter-request intervals. Adds instance fields for canary server URL and model name, and introduces public bind_server_endpoint(server_url, model_name) method to register the disagg server target.
Canary thread loop implementation tests/integration/defs/stress_test/disagg_cancel/harness.py	Implements _canary_thread_body end-to-end: loads and validates the canary prompts file (resolving relative paths against YAML directory), computes request interval from configuration, repeatedly sends deterministic greedy completions to the bound server, optionally checks generated token_ids against reference_token_ids, and appends timestamped records including success, token equivalence, latency, and error (no fail-fast on canary errors).
Comprehensive test infrastructure and validation tests/integration/defs/stress_test/disagg_cancel/test_canary.py	Provides in-process HTTP server fixture for POST /v1/completions simulation with configurable responses. Includes helper utilities for prompt file generation and harness setup. Covers prompt loader validation (success and failure cases), token equivalence logic, request wire-format validation, and integration tests for successful recording, server error handling, prompt round-robin, and self-termination scenarios.
Documentation updates tests/integration/defs/stress_test/disagg_cancel/README.md	Updates to mark metrics_thread and canary_thread as implemented (with load_thread remaining stubbed), expands listed test coverage to include test_canary.py and test_metrics_thread.py, updates file layout tree, and restructures "How to run" instructions with explicit pytest steps and combined directory-level invocation.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested reviewers

• pcastonguay
• StanleySun639
• xinhe-nv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main change: implementing the canary thread for disagg cancellation testing, with appropriate ticket ID and type.
Description check	✅ Passed	The PR description is comprehensive and well-structured, covering the canary thread implementation, helper functions, integration points, rationale for design choices, and test coverage, exceeding template requirements.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

tests/integration/defs/stress_test/disagg_cancel/test_canary.py (1)

22-22: 💤 Low value

Optional: from __future__ import annotations is not needed.

TensorRT-LLM targets Python 3.10+ (as per setup.py), so PEP 585 generics and PEP 604 union syntax (str | None) work natively without the __future__ import. Removing this line would be consistent with the project's Python version requirements. However, the import is harmless and does not cause issues.

Based on learnings, you can use Python 3.10+ features throughout the codebase without adding from __future__ import annotations.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/defs/stress_test/disagg_cancel/test_canary.py` at line 22,
Remove the unnecessary future import line "from __future__ import annotations"
at the top of the test module
(tests/integration/defs/stress_test/disagg_cancel/test_canary.py); since the
project targets Python 3.10+, PEP 585/604 features are available natively, so
delete that import to keep the file consistent with the codebase style.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/integration/defs/stress_test/disagg_cancel/test_canary.py`:
- Line 22: Remove the unnecessary future import line "from __future__ import
annotations" at the top of the test module
(tests/integration/defs/stress_test/disagg_cancel/test_canary.py); since the
project targets Python 3.10+, PEP 585/604 features are available natively, so
delete that import to keep the file consistent with the codebase style.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b3bbf97c-27f0-4226-90e2-0d4f2c8e6538

📥 Commits

Reviewing files that changed from the base of the PR and between 37ece3f and 8de90b2.

📒 Files selected for processing (3)

• tests/integration/defs/stress_test/disagg_cancel/README.md
• tests/integration/defs/stress_test/disagg_cancel/harness.py
• tests/integration/defs/stress_test/disagg_cancel/test_canary.py

[tensorrt-cicd]

PR_Github #52429 [ run ] triggered by Bot. Commit: 8de90b2 Link to invocation

[tensorrt-cicd]

PR_Github #52429 [ run ] completed with state SUCCESS. Commit: 8de90b2
/LLM/main/L0_MergeRequest_PR pipeline #41722 completed with status: 'SUCCESS'

CI Report

Link to invocation