Skip to content

[None][feat] publish native V2 KV cache events - #16869

Draft
alec-flowers wants to merge 1 commit into
NVIDIA:mainfrom
alec-flowers:agent/native-kv-events-v2
Draft

[None][feat] publish native V2 KV cache events#16869
alec-flowers wants to merge 1 commit into
NVIDIA:mainfrom
alec-flowers:agent/native-kv-events-v2

Conversation

@alec-flowers

Copy link
Copy Markdown
Collaborator

Summary

  • add a top-level strict KVEventsConfig with vLLM-compatible enablement, publisher, endpoint, replay, buffering, HWM, queue-size, and topic fields
  • reuse TensorRT-LLM V2's existing mutation hooks and iteration flush, replacing the attention-DP gather callback with local per-rank event conversion and publishing
  • publish vLLM-compatible msgpack batches over the three-frame ZMQ protocol with sequence numbers and base_port + rank endpoint behavior
  • emit complete blocks from the maximum attention-window lifecycle, suppress trailing partial blocks and their removal events, and ignore native created/storage-tier update events for this proof of concept
  • support attention-DP and conventional TP without KV-event object collectives; reject unsupported PP/CP topologies
  • add bounded background queues, null-publisher construction-only mode, counters, and idempotent shutdown
  • update the LLM-args telemetry golden manifest for the new public nested configuration

The paired Dynamo consumer is available in ai-dynamo/dynamo#12167.

Validation

  • two focused native-event tests passed in the TensorRT-LLM 1.3.0rc21 runtime, covering the no-gather callback, vLLM wire decoding, partial-block filtering, sequence numbering, and shutdown/port reuse
  • LLM-args golden-manifest generation check passed
  • Python compilation and git diff --check passed
  • local Qwen3-0.6B TensorRT-LLM smoke test delivered a stored-event batch containing two full blocks to the real Dynamo KV router; shutdown counters reported one published batch, zero dropped batches, and zero KV-event allgathers

The rc21 benchmark implementation remains on a separate branch; this draft is the clean forward-port against current main.

Signed-off-by: Alec Flowers <aflowers@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant