Propagate backend list_changed through vMCP

[JAORMX]

Summary

vMCP already consumed backend notifications/progress and notifications/message, but dropped */list_changed and advertised listChanged: false toward clients. As a result, aggregated tool/resource/prompt views went stale until the short-TTL cache happened to expire, and nothing was ever pushed to clients — the core complaint in #5748.

This wires the list_changed path end-to-end so backend capability changes converge to clients promptly:

• Consume backend notifications/{tools,resources,prompts}/list_changed on both the per-call client (pkg/vmcp/client) and a new persistent (idle) listener on the session-backed connector (pkg/vmcp/session/internal/backend), so changes are caught whether or not a call is in flight.
• Coordinator (pkg/vmcp/server/list_changed.go): on a backend change, invalidate only that backend's aggregation-cache entries (scoped, not whole-cache) and re-project each affected session's capabilities. The per-session go-sdk server then auto-emits list_changed downstream, so a subsequent tools/list reflects the new set.
• Flip the advertised listChanged capability to true for tools, resources, and prompts (pkg/vmcp/server/serve.go).
• Storm coalescing: a debounce keyed by backend collapses the per-session×backend notification fan-in into one sweep. Notification handlers are non-blocking (map-write + non-blocking channel send); the coordinator does heavy work off the receive loop under a shutdown-tied context, and untracks sessions on graceful termination.

Closes #5748

Type of change

• New feature

Test plan

• Unit tests (task test)
• Linting (task lint-fix)

task lint-fix reports 0 issues; go build/go vet clean. go test -race (Taskfile ldflags) across ./pkg/vmcp/... + ./pkg/cache/...: 44 packages pass, 0 fail. Coverage added:

• Unit: ListChangedKindForMethod, LateBoundListChangedNotifier, per-call + persistent handler routing (method→sink with backendID), scoped CacheInvalidator.InvalidateBackend (incl. multi-backend entries), coordinator coalescing/kind-routing/terminated-prune/ListTools-error-retains-old-set/start-stop/untrack-on-terminate, setSessionToolsReplace removal.
• Integration (real go-sdk backend, list_changed_realbackend_integration_test.go): mid-call mutation, idle mutation (proves the persistent listener), two concurrent sessions (fan-out), resources/prompts additions, capability-regression assertions, and a regression guard pinning the documented resources/prompts removal gap.

Does this introduce a user-facing change?

Yes (observability/correctness): with vMCP, backend tool/resource/prompt changes now propagate to connected clients as list_changed notifications (and updated */list results) promptly, instead of only after the aggregation cache's TTL expires. vMCP now advertises listChanged: true. Clients that ignore the notifications continue to work via polling/TTL.

Special notes for reviewers

Reviewed by a 5-axis Opus panel (security, MCP-correctness, code-quality, observability/concurrency, test-coverage). The concurrency axis initially flagged three MAJOR lifecycle bugs (a 30s client timeout that capped the idle stream, a session-registry leak on normal termination, and a shutdown hang); all were fixed and independently re-verified.

Known limitations — tracked follow-ups (not blockers):

1. Resources/prompts removal does not propagate (additions do). mcpcompat's per-session syncSessionResources/syncSessionPrompts are add-only (unlike syncSessionTools, which fully reconciles). Advertising listChanged: true is still correct for additions; removal propagation needs a reconciling change in stacklok/toolhive-core (mcpcompat) + a version bump. Documented at the coordinator's resource/prompt apply path and pinned by a regression-guard test. Read-time access is not affected — ReadResource re-derives admission per call, so a removed/deadmitted resource is denied at read time even while it lingers in the advertised list.
2. In-process Stop() does not eagerly close the new persistent backend streams, and the coordinator's session-tracking map is unbounded for ungracefully-dropped sessions never revisited by a sweep. This is bounded to the embed-and-Stop-without-process-exit and client-crash scenarios (the normal serve path reclaims everything at process exit), and closing it eagerly is blocked today by the absence of a public unregister-session hook in mcpcompat/go-sdk (only AddOnRegisterSession exists). A shim AddOnUnregisterSession in toolhive-core is the clean fix — follow-up.

Both follow-ups are in stacklok/toolhive-core (which we own); I can file them.

Size: larger than the usual limit (~33 files; the diff includes substantial docs/tests and generated mocks) — the maintainer approved a single comprehensive PR for this feature.

Implementation plan

Approved implementation plan

Architected in two passes (Opus gap-analysis, then a Fable architect finalized the authoritative single-PR spec). Key design points, all grounded in the current tree:

• Reuse the existing forwarding seams; the backend OnNotification handler and progress/message forwarding already existed — this adds the list_changed arm + a persistent idle listener.
• W0 capability flip; W1 domain types (BackendListChangedNotifier, ListChangedKind, late-bound holder); W2 per-call handler; W3 persistent listener (WithContinuousListening, no client timeout so the standalone GET stream isn't capped); W4 factory threading; W5 per-backend scoped cache invalidation via a bounded scan (a reverse index would deadlock against the LRU evict path); W6 coordinator (debounce, sessionID→ClientSession registry, fan-out) — enumerating the session cache was rejected (wrong object type; would require forbidden new cache API), so the coordinator keeps its own registry populated at registration and pruned via SetOnTerminate + sweep-time Validate; W7 REPLACE tool setter (so removals propagate); W8 wiring.
• No toolhive-core change in this PR (tools propagate fully; resources/prompts removal is the documented follow-up).

🤖 Generated with Claude Code

https://claude.ai/code/session_01MRh394VZgnqbYjJwjrcJWm

[JAORMX]

Closing as a duplicate of #5965, which was merged first and already implements the tools list_changed propagation for #5748.

This PR overlapped because it was developed in parallel; it is a superset — it additionally propagates resources/prompts list_changed (additions) and flips WithResourceCapabilities(true, true) + WithPromptCapabilities(true), which #5965 deliberately deferred as a tracked follow-up. Rather than reconcile the heavy conflicts against the now-merged #5965, that delta will be delivered as a focused follow-up PR built on top of #5965's design (serve_list_changed.go).

No functional loss from closing this — #5965 covers the tools path that closed the issue.