Agentic Inference updates#314
Open
hvagadia wants to merge 19 commits into
Open
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces comprehensive support for multi-turn conversation benchmarking, including a new MultiTurnDataset class, a MultiTurnStrategy for turn sequencing, and a ConversationManager for state tracking. It also provides extensive documentation, validation schemas, and utility scripts for dataset conversion and analysis. Feedback focuses on optimizing performance and memory efficiency within the MultiTurnDataset class, specifically recommending the use of vectorized pandas operations to avoid memory-intensive dictionary conversions and suggesting an optimization to reduce the algorithmic complexity of building message histories from quadratic to linear.
nvzhihanj
reviewed
May 18, 2026
nvzhihanj
reviewed
May 18, 2026
hvagadia
force-pushed
the
hvagadia/cache-salt-and-correctness
branch
from
66f0a3d to
1bbc7ab
Compare
nvzhihanj
reviewed
May 18, 2026
nvzhihanj
reviewed
May 18, 2026
nvzhihanj
reviewed
May 18, 2026
nvzhihanj
reviewed
May 18, 2026
nvzhihanj
reviewed
May 18, 2026
nvzhihanj
reviewed
May 18, 2026
hvagadia
changed the title
hvagadia
changed the title
nvzhihanj
approved these changes
May 19, 2026
nv-alicheng
reviewed
May 19, 2026
Collaborator
nv-alicheng
left a comment
nv-alicheng
left a comment
There was a problem hiding this comment.
Review Council — Multi-AI Code Review
Reviewed by: Codex + Claude | Depth: thorough
Found 11 issues across 7 files (3 couldn't be posted inline — see summary comment).
Collaborator
Review Council — Multi-AI Code ReviewReviewed by: Codex + Claude | Depth: thorough Found 11 issues across 7 files.
🔴 Must Fix (critical/high)Issues that will cause hangs, data loss, or incorrect behavior in production.
🟡 Should Fix (medium)Real issues that produce incorrect results under reachable conditions.
🔵 Consider (low)Valid improvements, suitable as follow-ups.
🤖 Generated by |
viraatc
reviewed
May 19, 2026
viraatc
reviewed
May 19, 2026
| # Mandatory: with the default warmup behaviour, every request fails with | ||
| # ConnectionResetError because uvicorn closes pre-warmed idle sockets after 5s. | ||
| client: | ||
| warmup_connections: 0 |
Collaborator
There was a problem hiding this comment.
whats the http error code ur seeing on this?
dont remember seeing this in the past with sglang.
(i expect tcp-keepalive is disabled (default))
also max-idle-time should is likely not making a difference here, lets use default if we can.
Collaborator
Author
There was a problem hiding this comment.
Let me get back on the error. I got errors while back, need to test on latest code.
viraatc
reviewed
May 19, 2026
arekay-nv
reviewed
May 19, 2026
Collaborator
arekay-nv
left a comment
arekay-nv
left a comment
There was a problem hiding this comment.
Review Council — Multi-AI Code Review
Claude-only review (Codex unavailable — bwrap/pivot_root not permitted in this environment). Depth: thorough.
arekay-nv
reviewed
May 19, 2026
Collaborator
arekay-nv
left a comment
arekay-nv
left a comment
There was a problem hiding this comment.
Review Council — Multi-AI Code Review
Claude-only review (Codex unavailable — bwrap/pivot_root not permitted in this environment). Depth: thorough.
arekay-nv
reviewed
May 19, 2026
arekay-nv
reviewed
May 19, 2026
arekay-nv
reviewed
May 19, 2026
| "name", | ||
| "tool_calls", | ||
| "tool_results", | ||
| "reasoning_content", |
Collaborator
There was a problem hiding this comment.
[Claude] medium (api-contract): reasoning_content is unconditionally forwarded in prior-assistant-turn history messages. This is a Kimi/SGLang-specific extension, not in the OpenAI Chat Completions spec for request message objects. Servers that validate input strictly (standard vLLM, TRT-LLM serve) will reject with 400/422 when prior assistant turns carry this field. Since ChatMessage.reasoning_content defaults to None and uses omit_defaults=True, this only triggers for datasets recorded from Kimi runs — but there is no api_type gate, config option to strip it, or documentation warning.
Collaborator
Author
There was a problem hiding this comment.
This is intentional for Kimi replay. The dataset has reasoning_content because SGLang/TRT-LLM with the kimi_k2 reasoning parser returns thinking separately from final content.
For replay, Kimi needs prior thinking in history; with chat_template_kwargs.preserve_thinking: true, the chat template renders reasoning_content back into the prompt. We’ve verified this path with SGLang and TRT-LLM.
arekay-nv
reviewed
May 19, 2026
arekay-nv
reviewed
May 19, 2026
Collaborator
Review Council — Multi-AI Code ReviewReviewed by: Claude (Codex unavailable — bwrap/pivot_root not permitted in this environment) | Depth: thorough | PR: #314 "Agentic Inference updates" Found 5 new issues posted as inline comments (3 previously covered by earlier review pass were deduplicated). Commit hygiene: 14 commits, 6 chore/fix — consider squashing before merge. 🔴 Must Fix (critical/high)
🟡 Should Fix (medium)
🔵 Consider (low)
|
arekay-nv
requested changes
May 20, 2026
Collaborator
arekay-nv
left a comment
arekay-nv
left a comment
There was a problem hiding this comment.
Can you refactor to reuse the existing accuracy infrastructure. This will enable us to reuse the same setup across models/datasets.
viraatc
reviewed
May 20, 2026
Collaborator
viraatc
left a comment
viraatc
left a comment
There was a problem hiding this comment.
Review Council — Multi-AI Code Review
Reviewed by: Claude | Depth: thorough
Codex review failed (CLI/cloud-requirements compatibility issue after recent CLI upgrade, then internal bwrap hangs on Ubuntu 24 AppArmor restriction). Falling back to Claude-only.
7 issues posted inline after deduping against the 44 existing comments and re-verifying each finding against current HEAD. Critical Pandas dict-assignment bug verified by repro. Schema gap on delay_seconds verified empirically.
Collaborator
Review Council — SummaryReviewed by: Claude (Codex unavailable: post-upgrade cloud-requirements compatibility + Ubuntu 24 AppArmor Found 7 issues across 4 files. All findings re-verified against HEAD ( 🔴 Must Fix (critical/high)
🟡 Should Fix (medium)
🔵 Consider (low)
Inline thread: #314 (review) |
Collaborator
Author
|
I have removed the accuracy script for now. It is still under discussion in TF, so doesnt make sense to integrate it yet. I will put up a separate draft PR for folks to experiment. |
Several OpenAI-compatible servers (notably SGLang and vLLM with their reasoning/tool parsers) drift from the strict OpenAI spec in ways the existing decoders silently swallowed: * SSEDelta.reasoning was the wrong field name. SGLang/vLLM emit `reasoning_content` on streaming deltas (matching the non-streaming ChatCompletionResponseMessage). The previous code never matched, so every reasoning chunk on a thinking-mode response was dropped on the floor with no error. * SSEDelta fields default to None (servers send `null`, not `""`). Previous defaults of `""` looked truthy when the server actually had no payload for that channel in a given chunk. * ChatCompletionResponseMessage / ChatCompletionChoice / ChatCompletionResponse: many fields (`content`, `refusal`, `finish_reason`, `usage`, `system_fingerprint`) are not always emitted. Without defaults, msgspec rejected non-streaming responses from these servers entirely. * decode_sse_message now catches msgspec.ValidationError, logs the raw chunk preview, and returns None so the stream keeps draining. The prior behaviour bubbled into the worker's outer Exception handler and lost the diagnostic. TextModelOutput now carries `tool_calls` and `finish_reason` as first-class fields (in addition to the existing metadata copy) so multi-turn replay can build history without reaching into metadata. A diagnostic `chunk_stats` dict counts content / reasoning / tool-call chunks per response for replay-determinism debugging. `array_like=False` on TextModelOutput is required to safely add fields without breaking the positional wire layout. PromptData and ErrorData flip too for consistency.
Two correctness changes for multi-turn replay against thinking-mode and tool-using models: 1. reasoning_content propagation. Prior assistant turns must replay their thinking trace as part of the message history; without it, the chat-template-rendered prompt diverges from what the original capture sent and outputs differ from the captured trajectory even at temperature=0. 2. tools propagation across all turns. Every request must carry the same tools array as turn 1; SGLang's tool-call parser is gated on a non-empty `tools` field in the request, and turns that omit it silently bypass the parser, leaking literal tool-call markup into the assistant `content` channel. New optional cache-bursting salt (multi_turn.enable_salt: bool, default False) appends a per-conversation blake2b digest to the end of each trajectory's system prompt. This keeps within-trajectory prefix caching intact while preventing cross-trajectory KV-cache leak during replay of datasets that share a long system prompt across many trajectories. The salt is computed once per conversation_id and reused on every turn of that conversation. apply_salt is idempotent on already-salted prompts so re-runs are stable. See examples/09_MultiTurn/docs for the full methodology and validation; not included in this commit. Mechanical change in dataset.py: load_from_file gains a **dataset_kwargs passthrough so the factory can forward enable_salt without expanding the signature for every future option.
The customer_support example and its two near-duplicate config files predated the agentic datasets and no longer earn their keep: - multi_turn_benchmark.yaml and multi_turn_with_concurrency.yaml differ only in `name`, two inline comments, and `report_dir`. Both already set target_concurrency: 32 — the README's Concurrency Control section documents the same knob inline. Keeping a second YAML to teach a setting that's already in the first is noise. - customer_support_conversations.jsonl was the toy dataset backing those YAMLs. With them gone it has no consumer. The remaining agentic YAMLs are tuned to actually work as written: - model_params.name: "/model" matches SGLang's --model-path mount - temperature: 0 (greedy) per MLPerf-inference reproducibility convention - max_new_tokens sized to the longest observed turn in each capture - target_concurrency reduced from theoretical maxima to values that match real B200-class server capacity on a 1T MoE - client.warmup_connections: 0 / max_idle_time: 0.5 work around uvicorn closing pre-warmed idle sockets after 5s, which otherwise causes every first request to fail with ConnectionResetError README updated: Basic Configuration example uses agentic_coding / agentic_coding_flat.jsonl (the dataset that's actually in scope post- deletion); Using Configuration File invokes agentic_coding_benchmark.yaml; Example Datasets section dropped (the only entry was customer_support).
…alysis utility - score_inline_accuracy.py: single-script scorer for multi-turn benchmark runs. Coding turns score by multiset IoU on a curated whitelist of ~40 canonical bash exes; workflow turns score by exact-match on `intent: IXXX`. Folds in events.jsonl -> model_assistants.jsonl conversion so a benchmark report dir can be scored in one call. - analyze_flat_jsonl.py: produces a single composite summary plot (turns/conv, ISL/OSL distributions, per-turn growth, token-class violins) for any flat multi-turn JSONL.
…ance - MultiTurnConfig: add `enable_salt: bool = False` knob. - MultiTurnDataset: when enabled, append `\n\n[cache_salt: <hex>]` to the system message once per trajectory, where hex is `blake2b(conversation_id, digest_size=8).hexdigest()`. Same salt is reused across all turns of one trajectory so within-trajectory prefix caching is preserved; differs across trajectories so the cross-trajectory cache match terminates at the salt boundary. - MultiTurnDataset: drop rows with no `conversation_id` after load (e.g. the `_type: dataset_metadata` license/source sentinel some upstream snapshots prepend). Methodology + full-dataset salt-vs-no-salt accuracy comparison in examples/09_MultiTurn/docs/EVALUATION.md.
- README: new "Accuracy Evaluation" subsection under "Running Multi-Turn Benchmarks" documenting the score_inline_accuracy.py command and what it writes into report_dir. Completes the convert -> replay -> score pipeline. - score_inline_accuracy.py: ruff-format/lint pass (formatting only, no behavior change).
- Add ModelParams.chat_template_kwargs (forwarded per request to
vLLM/SGLang). Enables Kimi-K2.6 Thinking-mode by sending
{'thinking': True, 'preserve_thinking': True} on every request so
prior assistant reasoning is rendered back into the prompt as
<think>...</think> blocks during multi-turn replays.
- Wire field through ChatCompletionRequest msgspec struct and
openai_msgspec_adapter so it lands on the wire as a top-level
request key.
- Score workflow accuracy from the structured intent_codes set
stamped on each scorable assistant row (no regex fallback): a turn
is a hit iff the model's extracted I### code is a member of the
per-turn ground-truth set. Tool-only assistant turns remain
unscorable by design.
- inject_tool_delay knob on multi_turn dataset config: when set, the
strategy defers the next turn issue by the dataset's embedded
delay_seconds via loop.call_later, modelling the producer-side
tool/human pause that the original capture saw.
- Update agentic_{coding,workflow}_benchmark.yaml to point at
the t1 datasets, T=1.0/top_p=0.95 sampling, salt enabled, and
chat_template_kwargs preset for Kimi-K2.6.
- Update README + score_inline_accuracy.py + JSONL schema to match
the new _t1.jsonl naming and intent_codes/delay_seconds fields.
- Regenerate full templates to expose the new schema field.
Tests cover end-to-end chat_template_kwargs propagation, the
intent_codes scoring rule, and the inject_tool_delay scheduling
path.
hvagadia
force-pushed
the
hvagadia/cache-salt-and-correctness
branch
from
ead4235 to
1781aa1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Type of change
Related issues
Testing
Checklist