[None][perf] disagg: optionally drop message history from generation request

[lancelly]

Summary

In disaggregated serving the orchestrator forwards each generation_only request to the
generation worker carrying both the full chat messages history and the
prompt_token_ids produced by the context server. The generation worker uses
prompt_token_ids directly (it skips apply_chat_template) and only reads the last message
for postprocessing, so the full conversation in messages is redundant on the gen side.

At high concurrency the gen worker's single OpenAI-server process is GIL/event-loop bound on
JSON-parsing + pydantic-validating that large request body; the messages history is ~40% of
it. Trimming messages to the last turn — when the gen ingress is the binding constraint —
measurably raises throughput and lowers TTFT.

Change

• New disagg config gen_strip_message_history (default off) on DisaggServerConfig,
  wired through extract_disagg_cfg as an explicit parameter (not **kwargs, so it is not
  inadvertently inherited into the per-server configs).
• When enabled, OpenAIDisaggregatedService._get_gen_request trims messages → messages[-1:]
  for generation_only ChatCompletionRequests (only when prompt_token_ids is set). Tools
  are preserved (still used for tool-parser / guided decoding).

Enable via the disagg YAML top level (same level as hostname / schedule_style):

gen_strip_message_history: true

Why config-gated (not unconditional)

A code audit found two GEN-side hazards an unconditional strip would hit on other model families:

1. harmony / gpt_oss — chat_harmony ignores prompt_token_ids and always re-tokenizes
   request.messages (openai_to_harmony_tokens); stripping → only the last turn is rendered →
   token/KV mismatch → corrupt output.
2. multimodal — parse_chat_messages_coroutines extracts mm_data from messages; stripping
   drops history image/audio parts.

Both are per-model properties and a disagg cluster serves a single model, so a default-off config
flag (an operator assertion "text-only + non-harmony deployment") covers them with zero blast
radius. The standard openai_chat path is unaffected: it consumes prompt_token_ids, skips
apply_chat_template, and only reads messages[-1] (get_role() / last_message_content).

Performance (DeepSeek-V4-Pro, 6p1d DEP4×6 + DEP8 GEN, MTP3, GB300, c2048, metrics-off)

metric	baseline	gen_strip_message_history
TTFT p50	4.89 s	4.00 s (−18%)
TTFT p95	8.61 s	8.06 s (−6%)
req/s	104.5	107.0 (+2.4%)

The gain appears at the un-throttled operating point where the gen-serving JSON-parse GIL is the
binding ingress constraint; it is absorbed when the system is decode-bound (closed-loop workload).

Test

• extract_disagg_cfg parses the new top-level key onto DisaggServerConfig.gen_strip_message_history
  and does not leak it into the per-server configs; default (unset) = False.
• No behavior change when the flag is off (the default), so existing deployments are unaffected.

[lancelly]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #54771 [ run ] triggered by Bot. Commit: 130b558 Link to invocation

[tensorrt-cicd]

PR_Github #54771 [ run ] completed with state SUCCESS. Commit: 130b558
/LLM/main/L0_MergeRequest_PR pipeline #43790 completed with status: 'SUCCESS'

CI Report

Link to invocation