[https://nvbugs/6104831][fix] disagg KV transfer timeout observation

[chienchunhung]

Summary

Carves out the observe-only KV-transfer deadline diagnostics from PR #13713 into a smaller, mergeable change. No in-flight cancellation logic; no action is taken on timeout — the WARN is strictly diagnostic.

This PR originally also contained the Python recv-side dedup sets. That portion (A3 in PR #13713's component taxonomy) was reverted in commit 7589b865ff to keep the scope minimal and to test whether the cross-rank divergence observed during PR #13713's CI is sensitive to A3.

What this PR changes (all ports from PR #13713)

Code name (per PR #13713)	What it is	Files
A4	mTimedOutSenderIds / mTimedOutRequesterIds dedup sets so the timeout WARN fires at most once per stuck request	cacheTransceiver.h
A6	Eager setKvCacheTransferStart(now()) at the entry of requestAndReceiveAsync so the deadline observation sees a valid elapsed time even before the worker thread has stamped it	cacheTransceiver.cpp
A7	Observe-only timeout WARN sites in checkContextTransferStatus and checkGenTransferStatus. Emitted when the configured kv_transfer_timeout_ms is exceeded. No state transition, no eviction, no cancellation triggered. Dedup'd via A4.	cacheTransceiver.cpp

Issue these resolve independent of in-flight cancellation

Today a wedged disagg KV transfer (peer crash, network partition, slow disk) is silent — the request blocks indefinitely with no operator-visible signal identifying which request is stuck or for how long. This PR gives operators a triage anchor.

• The WARN is only emitted when the user has configured kv_transfer_timeout_ms.
• A4's dedup prevents the WARN from flooding the log on every poll iteration.
• A6 ensures the elapsed-time math is correct even on the first iteration after the request enters the futures vector, before the worker thread has had a chance to stamp the start time itself.

What this PR explicitly excludes

Code name (per PR #13713)	Why excluded
A3	Recv-side dedup sets (_disagg_gen_init_prepared_ids / _disagg_gen_kv_recv_started_ids). Reverted in commit 7589b865ff because they exist primarily to make recv idempotent under the cancel-throw retry pattern, which is not in scope here. Without that pattern, the scheduler's state-based filter is sufficient.
G1–G8	All in-flight cancellation logic from PR #13713 (env var, cancel_request, sendHolder.poison(), mInFlightCancelFlags, deferred-cleanup paths, fail-closed checks, promise idempotency, etc.).
A5	Reordering of setState / receiveAsync / emplace in requestAndReceiveAsync. Upstream ordering preserved.

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #51014 [ run ] triggered by Bot. Commit: 7d7e35f Link to invocation

[tensorrt-cicd]

PR_Github #51014 [ run ] completed with state FAILURE. Commit: 7d7e35f
/LLM/main/L0_MergeRequest_PR pipeline #40462 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #51072 [ run ] triggered by Bot. Commit: 3e7f01e Link to invocation

[tensorrt-cicd]

PR_Github #51072 [ run ] completed with state SUCCESS. Commit: 3e7f01e
/LLM/main/L0_MergeRequest_PR pipeline #40514 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #51146 [ run ] triggered by Bot. Commit: 7589b86 Link to invocation

[tensorrt-cicd]

PR_Github #51146 [ run ] completed with state SUCCESS. Commit: 7589b86
/LLM/main/L0_MergeRequest_PR pipeline #40582 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation