Our SSE keepalive can drop the SLOWEST requests from benchmark metrics, flattering our own numbers

[localai-bot]

Found while staging the first clock-pinned parity measurement. This is our defect, in our server, and it biases our own benchmark in our favour.

Mechanism

include/vllm/entrypoints/openai/serving_utils.h:39-41 — VT_SERVER_SSE_PING_S defaults to 15 seconds, <=0 disables.

src/vllm/entrypoints/openai/serving_completion.cpp CompletionSseStream::WaitOutput waits on engine_.get_output_for(request_, ping_s * 1000) and, on timeout, routes through AssignSseWaitResult, which emits kSsePingFrame (":\n\n") as a standalone comment frame instead of a data frame.

So any request that stays silent longer than 15s — precisely the slowest requests — has a comment frame injected into its stream. If the timing client does not treat a bare SSE comment as a no-op, those requests are mishandled or dropped, and the affected population is exactly the tail of the latency distribution.

Removing the tail flatters us: it raises median throughput and lowers p90/p99 TTFT/ITL for our arm only. The oracle arm is unaffected, because vLLM does not emit these frames.

Impact on recorded numbers

• Reported as making c16 unquotable as a parity point in the in-flight clock-pinned series; a ping-disabled c16 re-measure is staged.
• Any historical grid at a concurrency/workload where a request could go >15s silent is suspect for the same reason. Long-prefill and high-concurrency points are the most exposed.

Not yet established

• Whether the pinned vllm bench serve client drops, mis-times, or correctly ignores the comment frame. The root-cause above is from our source; the client-side half is inferred and should be confirmed against vllm bench serve's SSE parsing before quoting a size for the bias.
• The magnitude. The staged ping-disabled re-measure will size it.

Suggested handling

The keepalive itself is legitimate (it exists for long-prefill TTFT). The bug is that a measurement path inherits it silently. Options, cheapest first:

1. Disable pings in the benchmark recipe (VT_SERVER_SSE_PING_S=0) and record that in the manifest, so a gate cannot inherit the flattering behaviour by default.
2. Assert in the harness that the ping interval was disabled, the way the oracle identity and clock state are now asserted — an unrecorded server-side knob that changes measured latency is the same class as the unrecorded SM clock (27B decode per-call attribution is not reproducible across box states: the same binary/arm moved -6.5% on the dominant fp8 shape, twice the deficit it is used to rank #543) and the rollback oracle (The benchmark harness ENFORCES the 0.25.0 rollback oracle and would REJECT the recorded pin — nobody could have measured correctly even deliberately #520).

Related: #543 (clock state unrecorded), #520 (harness enforced the rollback), #414 (--language-model-only withheld from the oracle).

FOLLOWING_AGENTS_PROTOCOL