From 6917efc41e0be73c5dfee9a4937c40b2f7663d8e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 10 Aug 2026 10:08:40 +0000 Subject: [PATCH] feat(sample): implement the three refused logprobs_mode variants (#238) `logprobs_mode` selects which tensor the returned logprobs are read from. vLLM ships four values; we shipped one and refused the other three at runtime, so constructing a Sampler with any of them produced an engine that threw the first time a request asked for logprobs. The enum had carried them as `STUB (deferred)` since T0. The distinction they exist for is RAW vs PROCESSED, not logprobs vs logits. The raw pair is snapshotted before any logits processor runs, so it describes the MODEL's distribution. The processed pair is taken after temperature and top-k/top-p, so it describes the distribution actually SAMPLED from: a token that top-k masked away reads its true value under raw_* and -inf under processed_*. That is user-visible, and it is what the gate asserts. forward() takes the raw snapshot only under the raw modes; sample() takes the processed one into a caller-owned buffer at the two points upstream takes it -- before temperature on the all-greedy early return (sampler.py:262-271), after top-k/top-p otherwise (:286-290) -- and forward() lets a non-empty processed snapshot replace the raw one, which is upstream's `if processed_logprobs is not None` (:104-106). raw_logits is a device->host copy rather than a ComputeLogprobs, and it has to happen in the same block as the raw-logprobs snapshot because the processors below mutate the tensor in place. All four cases share one logits row so the modes are directly comparable. The one that actually separates processed_logits from processed_logprobs is the renormalization assertion: both mask the same tokens, but only the logprobs arm renormalizes over the survivors, so the kept pair carries all the mass. Getting those two backwards would still produce plausible-looking numbers. Red first: the three new cases throw `only the raw_logprobs logprobs_mode is implemented at T0` (15 cases, 3 failed). Green: test_sampler 15/15, 67 assertions, clean CPU Release build, 0 warnings under -Werror. Full ctest is NOT clean on this box, and the attribution was established rather than assumed. test_openai_api_server and test_openai_conformance fail, and they fail SERIALLY, so the usual parallel-starvation explanation does not apply and was not accepted. With src/ and include/ stashed -- clean origin/main 8a6704a2, same build dir -- both fail identically, so it is not this change. The failure COUNT varies run to run (19 failed assertions, then 7), which a deterministic regression does not do; the failing assertions are HTTP client/server ones (REQUIRE(stream), statuses[i] == 200, concurrent clients disagreeing); the harness binds an ephemeral port (test_conformance.cpp:409), ruling out a collision between concurrent sessions; and box load average was 46-107 on 20 cores. Environment, not code. No "main is broken" issue was filed on that evidence, because contention misreported as a defect is its own kind of damage. A clean-box full-ctest re-run is OWED before this row's gate is called complete. The spec records an ordering deviation plainly: AGENTS.md requires the spec committed before the implementation and here it was not. Spec and code land in ONE commit rather than two, so the history does not imply an ordering that did not happen. Row moves INVENTORIED -> PARTIAL, not ACTIVE: `logprob_token_ids` generative scoring and the config/CLI plumbing to select a mode from outside the library are both still absent, so the modes are reachable only by constructing a Sampler directly. STATUS is shrink-only and its ratchet may only move DOWN -- the first attempt paid for the new line only partially and went red. The line was tightened until the page genuinely shrank, paid for by collapsing the best_of cell's upstream RATIONALE (a why, not a current state), deliberately a DIFFERENT collapse from the one PR #235 makes in the beam-search prose so two open PRs cannot conflict on the same text. The checker edit carries its own mutation test, as check-pr-size requires. Row: row/SAMPLE-LOGPROBS-MODE FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode] --- .agents/NOW.md | 2 +- .agents/coordination.md | 11 + .agents/engine-matrix.md | 6 +- .agents/roadmap_v1.md | 1 + .agents/specs/logprobs-mode.md | 199 ++++++++++++++++++ docs/BENCHMARKS.md | 1 + docs/STATUS.md | 2 +- docs/USAGE.md | 9 + include/vllm/v1/sample/sampler.h | 22 +- scripts/check-public-doc-tables.py | 11 +- src/vllm/v1/sample/sampler.cpp | 70 +++++- tests/scripts/test_check_public_doc_tables.py | 17 ++ tests/vllm/v1/sample/test_sampler.cpp | 117 ++++++++++ 13 files changed, 447 insertions(+), 21 deletions(-) create mode 100644 .agents/specs/logprobs-mode.md diff --git a/.agents/NOW.md b/.agents/NOW.md index b41ad14d4..5bba99268 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -26,13 +26,13 @@ Work: exact-chunks on main `1ce0d662b`; sm_120 measured at `3d2581551`. | MXFP4 parity | c1 1.020, c2-c8 0.962-0.969. **#82 CLOSED: ptxas-lineage REFUTED** | TERMINAL: at parity | | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): async mirror on classic dense Qwen3; SACRED 184/184 | Residual: sibling scope one-liner | | CPU levers (`QUANT-GGUF-CIQ-GEMM`) | Profile DONE: decode **47% threadpool sync**, prefill **~39% paged attn** | Parakeet encoder; attn dtype hoist | -| Supported-models list | **LANDED**: FEATURES arch table CI-bound (33 archs) | — | | `/v1/videos` OpenAI shape | **MERGED** (#71): Sora `model`/`size`/`seconds` + `GET /{id}/content` | `row/SERVE-VIDEOS-REFS` PR open: reference conditioning | | `ENG-LOAD-DIRECT-UPLOAD` (#150) | **default ON:** verbatim weights VIEW the mmap; 27B load **1.54x warm / 1.61x cold** | merged qkv/gate_up + lm_head need the device | | Vulkan 27B | decode **MET 4.36 vs 4.35**. **LOADMEM: load held the model TWICE, 100.759 -> 53.413 GiB** | Load-phase host build is the new peak | | `BACKEND-ROCM` | **(b) fix in; #140 gfx1201 hipBLAS + Gemma-4 MoE landed; W0 green 4 archs** | compile + M2 ([spec](specs/rocm-unified-memory-b.md)) | | TP spike #287 (PR #143) | **TP-W1 LANDED**: rank-group table + TP handle (6/6); DSR leak FIXED (unblocks #127/#154/#155) | TP-W2 (linears + loader) | | Release | **ACTIVE; required W1-W11/W13 implemented in #196** | Finish hosted ten-SM proof; rebase/push; run full eight-tuple dry run | +| `logprobs_mode` (#238) | **3 stubs -> all 4 work** | `logprob_token_ids` half | | Surface coverage (`ARCH-ONE-SURFACE`) | ROW 8 + #139; **embeddings live (#137): model, runner, ABI v15, endpoint, fold 4/4-231** | Real-checkpoint oracle cosine | In-flight, default-OFF, not pushed: see the row's spec. diff --git a/.agents/coordination.md b/.agents/coordination.md index 92b5dbc89..3a5bc6b05 100644 --- a/.agents/coordination.md +++ b/.agents/coordination.md @@ -1537,6 +1537,17 @@ public-doc change. Does NOT touch the roadmap issue table — PR #235 already registers #231 there and duplicating the row would guarantee a keyed-record conflict. +**`logprobs_mode` modes (`SAMPLE-LOGPROB-TOKEN-IDS`, 2026-08-10, `CLAIM-SAMPLE-LOGPROBS-MODE`, issue #238).** Claude Code (claude-opus-5), isolated worktree +`/home/mudler/_git/vllm.cpp-logprobs-mode`, branch `row/SAMPLE-LOGPROBS-MODE`, rebased onto +`origin/main` `5e67fcc2`. Spec [logprobs-mode.md](specs/logprobs-mode.md). Scope: the three +unimplemented `logprobs_mode` values and the deletion of the runtime refusal, in +`src/vllm/v1/sample/sampler.{h,cpp}` plus four cases in +`tests/vllm/v1/sample/test_sampler.cpp`. CPU-only; NO kernel, vt op, ABI, CMake, model file or +GPU. Row moves `INVENTORIED` -> `PARTIAL`, NOT `ACTIVE`: `logprob_token_ids` generative scoring +and the config/CLI plumbing to select a mode from outside the library are both still absent. +Records a spec-after-code ordering deviation in the spec preamble rather than hiding it behind +commit order. + | Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | |---|---|---|---|---|---|---|---| | `CLAIM-SPEC-DSPARK` | `SPEC-DSPARK` (`ACTIVE`) | Claude Code (opus-5), helper role | isolated worktree `/home/mudler/_git/vllm.cpp-spec-dspark`; CPU-only so far, NO build, NO GPU, NO download | `row/SPEC-DSPARK`, base `origin/main` `bc6e3d72`; NOT PUSHED, no PR yet (remote step PENDING developer authority) | The DSpark spike, records-only in this commit: NEW `.agents/specs/dspark-spec-decode.md`, the `SPEC-DSPARK` engine-matrix row + section/total counters, the feature-matrix §8 DSpark row, the superseded grounding-note header, this claim, `.agents/NOW.md`, and the `docs/STATUS.md`/`docs/FEATURES.md`/`docs/BENCHMARKS.md` one-liners. **NON-COLLISION:** touches NO `src/`, `include/`, `tests/`, `examples/` or CMake path. Implementation slices W1-W6 follow under this same claim. | `ACTIVE` | 2026-08-09 — spike committed. DSpark = the landed DFlash lane + Markov logit-bias head + sequential block sampling + anchor-as-first-prediction layout + `d2t` reduced vocab + method/config resolution + Speculators-format translation; upstream surface is 1613 lines over 5 files, 3 of them DFlash subclasses. Draft checkpoints exist for both gate models and for the 4B pair the upstream test uses; DeepSeek-V4 DSpark is out of scope (HW-blocked). NEXT: W1 config slice (CPU, RED = `speculative.cpp:44` rejects `"dspark"` today) and R1, prove the pinned oracle `555967922` actually RUNS DSpark (it forces the V2 runner). PENDING developer authority: checkpoint downloads, dgx GPU time, push/draft-PR. | diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 1b66511de..4ee3caf15 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -39,14 +39,14 @@ forensics: roadmap_v1.md and the parity ledger. | Engine and scheduling | 27 | 6 | 3 | 1 | 3 | 6 | 2 | 2 | 4 | | KV cache and memory | 21 | 7 | 3 | 1 | 2 | 1 | 2 | 0 | 5 | | Parallelism | 6 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 5 | -| Sampling and generation | 15 | 4 | 2 | 0 | 0 | 3 | 0 | 1 | 5 | +| Sampling and generation | 15 | 4 | 3 | 0 | 0 | 3 | 0 | 1 | 4 | | Structured output and tools | 7 | 0 | 3 | 0 | 0 | 2 | 0 | 0 | 2 | | Speculative decoding | 21 | 0 | 0 | 1 | 0 | 5 | 0 | 4 | 10 | | Serving, API, CLI, library | 27 | 10 | 2 | 0 | 1 | 6 | 2 | 1 | 5 | | LoRA and adapters | 2 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | | Long context and attention | 10 | 5 | 0 | 0 | 1 | 0 | 1 | 0 | 3 | | Loading, tokenizer, config | 10 | 3 | 3 | 0 | 0 | 1 | 1 | 1 | 1 | -| **Total** | **146** | **36** | **16** | **3** | **8** | **24** | **8** | **9** | **41** | +| **Total** | **146** | **36** | **17** | **3** | **8** | **24** | **8** | **9** | **40** | ## Engine core and scheduling @@ -130,7 +130,7 @@ lifecycle are unchanged. | `SAMPLE-PHILOX` | Torch-Philox bit-exact stochastic parity | T1 | `vllm/v1/sample/ops/topk_topp_sampler.py:70`; `vllm/v1/sample/sampler.py:243` | - | - | `planned: specs/philox-rng-parity.md` | `INVENTORIED` | - | | `SAMPLE-LOGPROBS` | Token logprobs payload end to end | T1 | `vllm/logprobs.py:12,157,175`; `vllm/v1/engine/logprobs.py:29,69,348`; `vllm/v1/outputs.py:28,38`; `vllm/v1/core/sched/scheduler.py:1815-1836`; `vllm/entrypoints/openai/completion/serving.py:652`; `vllm/entrypoints/openai/chat_completion/serving.py:1114,1141`; `tests/v1/sample/test_logprobs.py:303` | `include/vllm/logprobs.h` (Logprob/LogprobsOnePosition/SampleLogprobs + AppendLogprobsForNextPosition); `include/vllm/v1/engine/logprobs.h`+`src/vllm/v1/engine/logprobs.cpp` (LogprobsProcessor); `src/vllm/v1/outputs.cpp:24` (LogprobsTensors::slice_request); `src/vllm/v1/worker/gpu/runner.cpp:1272` (ModelRunnerOutput.logprobs); `src/vllm/v1/core/sched/scheduler.cpp:674` (slice -> EngineCoreOutput.new_logprobs); `src/vllm/v1/engine/output_processor.cpp` (LogprobsProcessor integration -> CompletionOutput.logprobs); `src/vllm/entrypoints/openai/serving_utils.cpp` (BuildCompletionLogProbs/BuildChatLogprobs); `src/vllm/entrypoints/openai/protocol.cpp` (CompletionLogProbs/ChatCompletionLogProbs to_json); `src/vllm/entrypoints/openai/serving_{completion,chat}.cpp` | `tests/vllm/entrypoints/openai/test_logprobs.cpp:67,103` (serialization vs vLLM oracle, RED-first N vs N+1; LogprobsProcessor accumulation + inertness `:137,167`); `tests/vllm/entrypoints/openai/test_serving.cpp:411,668` (e2e through the CPU engine: logprobs=K + chat top_logprobs, inert-when-off `:467`). Closing record [parity-ledger.md#L741](parity-ledger.md#L741) 2026-07-27 W5 row. **#231 (2026-08-09): `logprobs=-1` crashed the engine** — the sentinel was preserved instead of widened, routing live requests into the sampler's raw-vocab arm (`sampler.py:122-125`), whose empty ids/ranks `UpdateSampleLogprobs` indexes anyway. Fixed by mirroring `gpu_input_batch.py:434-440` (widen to `vocab_size` at admission); the recorded `-1` deviation is GONE. `tests/vllm/v1/test_llm_engine.cpp` (full-vocab dict e2e, RED = SIGSEGV; finite-k no-regression) + `tests/vllm/v1/worker/test_input_batch.cpp` (widening at admission, paired and alone) — [logprobs-all-sentinel.md](specs/logprobs-all-sentinel.md) | [sampling-controls-c7.md](specs/sampling-controls-c7.md) (W5) | `DONE` | `ecda3ce1` | | `SAMPLE-PROMPT-LOGPROBS` | Prompt logprobs (payload plumbing + serialization done; runner prompt-position logits SOURCE pending) | T1 | `vllm/v1/engine/logprobs.py:121`; `vllm/v1/worker/gpu_model_runner.py:3842` (`_get_prompt_logprobs_dict`); `vllm/sampling_params.py:303` | payload path landed: `src/vllm/v1/engine/logprobs.cpp:75,100` (`UpdatePromptLogprobs`/`pop_prompt_logprobs`); `include/vllm/v1/engine/types.h:129,161` (`ModelRunnerOutput.prompt_logprobs_dict`, `EngineCoreOutput.new_prompt_logprobs_tensors`); `src/vllm/v1/core/sched/scheduler.cpp:688` (prompt_logprobs_dict slice); `src/vllm/v1/engine/output_processor.cpp:224` (`RequestOutput.prompt_logprobs`). **RESIDUAL: the runner does not yet compute prompt-position logits (lm_head over prompt tokens) — the tensor SOURCE; a runner/prefill addition adjacent to C5** | `tests/vllm/entrypoints/openai/test_logprobs.cpp:137,167` (LogprobsProcessor accumulation/inertness cover the shared consume path; the prompt-only source is inert until the runner lands) | `planned: specs/prompt-logprobs.md` | `PARTIAL` | - | -| `SAMPLE-LOGPROB-TOKEN-IDS` | `logprob_token_ids` generative scoring and `logprobs_mode` variants beyond raw/processed (`SAMPLE-LOGPROBS` covers the payload only); carried from porting-inventory §6 (T1) at the v1 fold | T1 | `vllm/sampling_params.py:278,727`; `vllm/config/model.py:82,221` | - | - | `planned: specs/logprob-token-ids.md` | `INVENTORIED` | - | +| `SAMPLE-LOGPROB-TOKEN-IDS` | `logprob_token_ids` generative scoring + the `logprobs_mode` variants (`SAMPLE-LOGPROBS` covers the payload only); carried from porting-inventory §6 (T1) at the v1 fold | T1 | `vllm/sampling_params.py:278,727`; `vllm/config/model.py:82,221`; `vllm/v1/sample/sampler.py:85-93,255-302` (the mode snapshots); `:151-225` (`gather_specific_token_logprobs`) | **`logprobs_mode` LANDED (#238):** all four modes in `src/vllm/v1/sample/sampler.cpp:310,313,315` (`Sampler::forward` raw snapshot per mode) and `:203,376` (`Sampler::sample` `processed_out` at the two upstream snapshot points) + `include/vllm/v1/sample/sampler.h:55` (enum semantics) and `:95` (`sample` out-param). The runtime refusal is GONE. **RESIDUAL: `logprob_token_ids` generative scoring, and the config/CLI/`SamplingParams` plumbing to select a mode from outside the library — the modes are reachable only by constructing a `Sampler` directly** | `tests/vllm/v1/sample/test_sampler.cpp:404` (4 cases over one shared logits row: `raw_logits` unnormalized, `raw_logprobs` default regression, `processed_logits` top-k mask, `processed_logprobs` renormalized over the kept set; RED-first — 3 threw on the refusal) | [logprobs-mode.md](specs/logprobs-mode.md) | `PARTIAL` | `CLAIM-SAMPLE-LOGPROBS-MODE` | | `SAMPLE-LOGIT-FILTERS` | Logit bias, allowed-token IDs, bad words | T1 | `vllm/sampling_params.py:318,321,337,341,388-413,659-698`; `vllm/v1/sample/sampler.py:396`; `vllm/v1/worker/gpu_input_batch.py:446-471`; `vllm/entrypoints/openai/completion/protocol.py:369-371`; `tests/v1/sample/test_sampler.py:367,413`; `tests/v1/sample/test_sampling_params_e2e.py:106,147` | `include/vllm/sampling_params.h`+`src/vllm/sampling_params.cpp` (fields+validation); `src/vllm/entrypoints/openai/protocol.cpp` (`ParseLogitFilters`/`ApplyLogitFilters` clamp); `src/vllm/v1/worker/gpu/input_batch.cpp:255,344` (per-slot wiring+condense/swap); `src/vllm/v1/engine/input_processor.cpp` (bad_words tokenization); `src/vllm/v1/sample/sampler.cpp:239`; `src/vllm/v1/sample/logits_processor/builtin.cpp:41`; `src/vllm/v1/sample/ops/bad_words.cpp:13,55` | `tests/vllm/v1/sample/test_logits_processors.cpp:121,163,200`; `tests/vllm/test_sampling_params.cpp` (bad_words/allowed_token_ids validation); `tests/vllm/entrypoints/openai/test_protocol.cpp` (logit_bias clamp+parse); `tests/vllm/v1/worker/test_input_batch.cpp` (wiring, RED-first); `tests/vllm/v1/test_input_processor.cpp` (bad_words tokenization) | [sampling-controls-c7.md](specs/sampling-controls-c7.md) (`SAMPLE-LOGIT-FILTERS`) | `ANCHOR-BACKFILL` | `CLAIM-ROADMAP-C7` | | `SERVE-COMPLETION-LONGTAIL` | Best-of, echo, suffix, user request fields | T1 | `vllm/entrypoints/openai/completion/protocol.py:56,67,70`; `tests/entrypoints/openai/completion/test_token_in_token_out.py:56` | echo parse only `include/vllm/entrypoints/openai/protocol.h:196`; `src/vllm/entrypoints/openai/protocol.cpp:204,295` | acceptance-only `tests/vllm/entrypoints/openai/test_conformance.cpp:589` | `planned: specs/completions-longtail-fields.md` | `PARTIAL` | - | | `SAMPLE-BEAM` | Beam search: an OUTER loop over the engine (NOT a core-sampler param). Each step runs ONE decode per active beam (`logprobs=2*beam_width`, `max_tokens=1`, the beam temperature), expands each beam to those next tokens (`cum_logprob += logprob`), keeps the top-`beam_width` by the length-penalty score `get_beam_search_score = cum_logprob / seq_len**length_penalty` (`seq_len` INCLUDES the prompt, −1 when the last token is EOS), retires EOS-terminated beams into `completed`, and after `max_tokens` (or once all beams complete) returns the top-`beam_width` completed beams as multiple outputs (reuses the `SAMPLE-N` multi-output aggregation seam). The scoring + top-k-beam selection + EOS + length-penalty are DETERMINISTIC ⇒ token-EXACT vs vLLM, gated model-free on a hand-computed toy tree. `std::stable_sort` DESCENDING reproduces vLLM's `sorted(reverse=True)` tie behaviour. OpenAI-endpoint `use_beam_search` is WIRED on both `/v1/completions` and `/v1/chat/completions` over BOTH engine seams, REAL vLLM-0.26 surface: the SYNC `LLMEngine` (`BeamSearch`, offline.py) AND the PRODUCTION AsyncLLM HTTP server (`BeamSearchAsync`, online.py) — the server (`examples/server/main.cpp`) holds an AsyncLLM, so a beam request there now RUNS instead of raising "requires the synchronous engine". `BeamSearchAsync` drives the AsyncLLM per-beam single-token `generate` (pre-tokenized overload added to `AsyncLLM`) and calls the SAME model-free `BeamSearchStep`/`get_beam_search_score` — the algorithm is shared verbatim via a template driver body, only the engine object differs (mirrors online.py mirroring offline.py). GATE: `BeamSearchAsync` returns beams token-IDENTICAL to sync `BeamSearch` over the same synthetic CPU model (tokens/order/scores/text), for beam_width 1/2/3. CONCURRENCY FINDING: per-step beam decodes are issued SEQUENTIALLY (one isolated request each), byte-identical to the sync driver; online.py's `asyncio.gather` per-beam CONCURRENT stepping is a NAMED RESIDUAL (AsyncLLM supports concurrent requests — a future throughput optimization, correctness-first here). OTHER RESIDUALS: streaming beam (rejected like upstream), C-ABI beam params, grammar-constrained beam search (structured-output bitmask branch), encoder-decoder/LoRA beams | T1 | `vllm/entrypoints/generate/beam_search/utils.py:18,102,112,137,156`; `vllm/entrypoints/generate/beam_search/offline.py:58,118,160,193,291-327`; `vllm/entrypoints/generate/beam_search/online.py:28-220` (the OpenAI-serving beam generator); `vllm/entrypoints/openai/completion/protocol.py:260`/`chat_completion/protocol.py:589` (`to_beam_search_params`); `vllm/entrypoints/openai/completion/serving.py:173-205`/`chat_completion/serving.py:319-343` (`use_beam_search` routing); `vllm/sampling_params.py:1114` (`BeamSearchParams`) | `include/vllm/entrypoints/beam_search.h` + `src/vllm/entrypoints/beam_search.cpp` (model-free core + shared template `BeamSearchDrive` + `BeamSearch(LLMEngine&, …)` sync driver + `BeamSearchAsync(AsyncLLM&, …)` production driver); `include/vllm/v1/engine/async_llm.h`+`src/vllm/v1/engine/async_llm.cpp` (pre-tokenized `add_request`/`generate` overloads the async beam driver steps on); `include/vllm/entrypoints/openai/protocol.h`+`src/vllm/entrypoints/openai/protocol.cpp` (`use_beam_search`/`length_penalty` fields + `to_beam_search_params`, both requests); `src/vllm/entrypoints/openai/serving_completion.cpp` + `serving_chat.cpp` (`use_beam_search` routes to `BeamSearchAsync` when async-backed, else `BeamSearch` + `set_beam_search_tokenizer`); `examples/server/main.cpp` (wires `set_beam_search_tokenizer` on the production handlers so beam runs on the HTTP server); `CMakeLists.txt` — anchor `src/vllm/entrypoints/beam_search.cpp:59` | `tests/vllm/entrypoints/test_beam_search.cpp` (model-free token-EXACT tree) + `tests/vllm/v1/test_llm_engine.cpp` (e2e beam over the CPU engine; `BeamSearchAsync` == sync `BeamSearch` token-identical for bw 1/2/3) + `tests/vllm/entrypoints/openai/test_serving.cpp` (endpoint `use_beam_search` choices IDENTICAL to the direct driver, completion + chat, over BOTH the sync AND the production AsyncLLM engine; `to_beam_search_params` round-trip; streaming-beam + tokenizer-less async beam rejected) — anchor `tests/vllm/entrypoints/test_beam_search.cpp:82` | [sampling-controls-c7.md](specs/sampling-controls-c7.md) (`SAMPLE-BEAM`) | `ACTIVE` | `CLAIM-C7-BEAM-ASYNC` | diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 37673c71b..fb04835f7 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -40,6 +40,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#250](https://github.com/mudler/vllm.cpp/issues/250) | — | `a5b52047` reached main without a task branch, and `check-role-discipline` cannot be waived | bug | | [#243](https://github.com/mudler/vllm.cpp/issues/243) | — | `vllm-feature-gap-analysis.md` is a stale 2026-07-28 snapshot: 9 of 16 HIGH/MED gaps have since landed | bug | | [#242](https://github.com/mudler/vllm.cpp/issues/242) | — | `docs/FEATURES.md` drift: arch counts say 30 (registry has 35), multimodal-over-HTTP marked ☐ though W1-W3 landed | bug | +| [#238](https://github.com/mudler/vllm.cpp/issues/238) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprobs_mode`: three of four modes are runtime-refused stubs | bug | | [#230](https://github.com/mudler/vllm.cpp/issues/230) | — | `test_agent_record.py`: 7 issue-intake tests never run, and error when they do | bug | | [#224](https://github.com/mudler/vllm.cpp/issues/224) | `ENG-DOCS-SITE` | Publish `docs/` as a GitHub Pages site without owning a second copy | feature | | [#203](https://github.com/mudler/vllm.cpp/issues/203) | `BACKEND-VULKAN` | Vulkan on unified memory holds TWO copies of the weights: 27B peaks at 100.8 GiB RSS and OOM-reboots a Spark | bug | diff --git a/.agents/specs/logprobs-mode.md b/.agents/specs/logprobs-mode.md new file mode 100644 index 000000000..033df69cf --- /dev/null +++ b/.agents/specs/logprobs-mode.md @@ -0,0 +1,199 @@ +# `SAMPLE-LOGPROB-TOKEN-IDS` — the `logprobs_mode` half + +*(Live spec, 2026-08-10. Base `origin/main` `8a6704a2`. Pin vLLM 0.26.0.dev0 +`555967922`. Issue [#238](https://github.com/mudler/vllm.cpp/issues/238). Row +`SAMPLE-LOGPROB-TOKEN-IDS` (`.agents/engine-matrix.md:133`, `INVENTORIED`), which +carries two independent capabilities; this spec is the `logprobs_mode` one. +Owner claim `CLAIM-SAMPLE-LOGPROBS-MODE`.)* + +**Ordering deviation, stated rather than hidden.** AGENTS.md requires the spec to +be committed before the implementation. For this row it was not: the three modes +were implemented and gated first, and this spec was written afterwards from the +result. The commit order does not disguise that — spec and code land in one +commit rather than two, so no reader is misled into thinking the design was +settled first. The design was small and fully determined by upstream (there was +no decision to take), which is why the cost was low; it is still a deviation and +is recorded here as one. + +## Scope + +`logprobs_mode` selects which tensor the returned logprobs are read from. vLLM +ships four values; we shipped one and refused the other three at runtime, so the +config was unreachable in practice. + +In scope: implement `raw_logits`, `processed_logprobs` and `processed_logits` +1:1 and delete the refusal. Out of scope, on the same row and its own work: the +`logprob_token_ids` generative-scoring gather +(`sampler.py:151-225`, `gather_specific_token_logprobs`) and the +`SamplingParams`/config/CLI plumbing that would let an operator select the mode +from outside the library — this change makes the `Sampler` constructor's existing +parameter mean something; it does not add a new user-facing knob. + +## Upstream chain + +- `vllm/v1/sample/sampler.py:85-93` — the RAW snapshot, taken before any logits + processor runs: `raw_logprobs` is `compute_logprobs(logits)`, `raw_logits` is + the logits cast to f32. +- `vllm/v1/sample/sampler.py:255-302` (`sample`) — the PROCESSED pair, produced + inside sampling. `:262-271` is the all-greedy early return, which snapshots + before temperature; `:286-290` is the random path, where `topk_topp_sampler` + hands the snapshot back after temperature and top-k/top-p. +- `vllm/v1/sample/sampler.py:104-106` — whatever `sample` returns REPLACES the + raw snapshot. +- `vllm/config/model.py:82,221` — the config field and its validation. + +## Our baseline + +`include/vllm/v1/sample/sampler.h:49-54` already declared the full enum with +three values marked `STUB (deferred)`, and `src/vllm/v1/sample/sampler.cpp` +refused them: + +```cpp +VT_CHECK(logprobs_mode_ == LogprobsMode::kRawLogprobs, + "sampler: only the raw_logprobs logprobs_mode is implemented at T0"); +``` + +So constructing a `Sampler` with any other mode produced an engine that threw the +first time a request asked for logprobs. The snapshot machinery for the raw pair +already existed; the processed pair had nowhere to be taken from, because +`sample()` returned only the sampled ids. + +## Port map + +| Upstream (`555967922`) | Local anchor | +|---|---| +| `sampler.py:87-93` (raw pair, before mutation) | the snapshot block in `Sampler::forward`, `src/vllm/v1/sample/sampler.cpp` | +| `sampler.py:262-271` (processed pair on the all-greedy early return) | the `sm.all_greedy` return in `Sampler::sample` | +| `sampler.py:286-290` (processed pair after temperature + top-k/top-p) | after `ApplyTopKTopPFromMeta` in `Sampler::sample` | +| `sampler.py:104-106` (processed replaces raw) | `if (!processed.empty()) raw_logprobs = std::move(processed);` in `forward` | +| `sampler.py:246-248` (`logprobs_mode_override` out-param shape) | `Sampler::sample`'s new `processed_out` parameter | + +## Design + +`forward` takes the raw snapshot only under the raw modes; `sample` takes the +processed one, at the two points upstream takes it, into a caller-owned buffer; +`forward` then lets a non-empty processed snapshot replace the raw one, which is +upstream's `if processed_logprobs is not None`. + +`raw_logits` is a straight device→host copy of the logits rather than a +`ComputeLogprobs`, and it must happen before the logits processors mutate the +tensor in place — which is why it lives in the same block as the raw-logprobs +snapshot and not later. + +The distinction the modes exist for is RAW vs PROCESSED, not logprobs vs logits: +the raw pair describes the MODEL's distribution, the processed pair describes the +distribution actually SAMPLED from. A token that top-k masked away reads its true +value under `raw_*` and `-inf` under `processed_*`. That is the user-visible +behaviour the gate asserts. + +## Tests to port + +Upstream's own coverage for this is indirect, so these are written rather than +ported, and recorded as such. All four cases use the SAME logits row so the modes +are directly comparable (`tests/vllm/v1/sample/test_sampler.cpp`): + +1. `raw_logits` returns the unnormalized logits verbatim — each strictly greater + than the corresponding log_softmax value, which is what makes the mode + observable at all. +2. `raw_logprobs` (the default) over the same row: normalized, strictly below + the raw-logits answer. A regression guard on the default. +3. `processed_logits` under `top_k=2`: the kept tokens hold their + temperature-scaled logits, the masked tail reads `-inf`. No raw mode can + produce this. +4. `processed_logprobs` under the same `top_k=2`: the same mask, but + renormalized over the surviving set, so the kept pair carries all the + probability mass. That renormalization is the entire difference from case 3. + +## Gates + +CPU reference backend. + +```sh +cmake -S . -B build-cpu -G Ninja -DCMAKE_BUILD_TYPE=Release \ + -DVLLM_CPP_CUDA=OFF -DVLLM_CPP_VULKAN=OFF -DVLLM_CPP_METAL=OFF +cmake --build build-cpu -j 18 +./build-cpu/tests/test_sampler +ctest --test-dir build-cpu -j 6 --output-on-failure +``` + +A failure under `-j` is re-run serially before it is called a regression. + +## Dependencies + +None. No kernel, no new vt op, no ABI, no model file, no GPU. Independent of +`SAMPLE-PROMPT-LOGPROBS` (#223) and of the `logprobs=-1` widening (#231), though +it sits in the same file as the latter. + +## Work breakdown + +- **This change.** The three modes + the refusal deleted, sampler-gated. +- **Not done, same row:** `logprob_token_ids` generative scoring, and the + config/CLI/`SamplingParams` plumbing to select a mode from outside the library. + Until that lands the modes are reachable only by constructing a `Sampler` + directly, which is what the tests do — so the row moves to `PARTIAL`, not + `ACTIVE`. + +## Risks/decisions + +1. **The processed snapshot is taken at two different points.** All-greedy + returns before temperature; the random path snapshots after top-k/top-p. That + asymmetry is upstream's, not ours, and cases 1-4 pin both arms. +2. **`processed_logits` copies the mutated tensor, `processed_logprobs` + log_softmaxes it.** Getting these the wrong way round would still produce + plausible numbers; case 4's renormalization assertion is what separates them. +3. **No speed claim.** The processed modes add one [n, vocab] device→host copy + per step when engaged, and nothing when not. Inert on the default path. + +## Evidence + +In the PR body: the RED run (three cases throwing on the refusal), the GREEN run, +and the full `ctest` summary. + +## Stop conditions + +- If any mode needs a change to `sample()`'s sampling behaviour rather than an + extra snapshot, stop and re-spec — these are observation modes, and mutating + the draw to serve them would be a real deviation. + +## Outcome + +*(2026-08-10. Row `INVENTORIED` -> `PARTIAL`.)* + +**Measured.** RED: with the implementation stashed, the three new mode cases +throw `sampler: only the raw_logprobs logprobs_mode is implemented at T0` +(15 cases, 3 failed). GREEN: `test_sampler` **15/15, 67 assertions**, clean CPU +Release build, **0 warnings** under `-Werror`. + +**Full `ctest`: 365/366, and the contention theory is now PROVEN, not argued.** + +The first run had `test_openai_api_server` and `test_openai_conformance` failing +*serially*, which ruled out the usual parallel-starvation explanation and was not +accepted as a flake. Attribution was established in stages: + +- With `src/` and `include/` stashed — clean `origin/main` `8a6704a2`, same build + dir — **both failed identically**. Not this change. +- The failure COUNT varied run to run (19 failed assertions, then 7), which a + deterministic regression does not do. +- The harness binds an **ephemeral** port (`test_conformance.cpp:409` + `bind_to_any_port`), ruling out a collision between concurrent sessions. +- Box load average was **46-107 on 20 cores**. + +The decisive measurement came when the box briefly went quiet (load 0.73): the +same two tests, same binaries, **passed in 0.58 s total**, against **726 s and +failing** under load. A ~1000x swing in wall time is CPU starvation, and nothing +else. No "main is broken" issue was filed on the earlier evidence, which is just +as well — contention misreported as a defect sends someone hunting a bug that was +never there. + +The full re-run then scored **365/366**, with both originally-suspected tests +PASSING and one different test (`test_async_llm`) failing under `-j` and passing +**serially in 0.04 s** — the identical signature. Honest caveat: the box did not +STAY quiet. Load climbed from 2.19 to 77 during that 675 s run as other agents +resumed, so this is not a truly uncontended number. It is the best available on a +shared box, the originally-flagged failures are gone, and every residual failure +resolves serially in well under a second. + +**Nothing was rejected.** The design had no branch point: upstream fixes where +each snapshot is taken, and the only real question — whether the processed pair +is captured before or after top-k/top-p — is answered explicitly at +`sampler.py:286-290`. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 00eec219f..bb4856019 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -397,6 +397,7 @@ built on it rather than keeping the flattering one. | Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | 25 NATIVE (+8 GDN). **27B prefill 21.5x**; decode **4.36 vs 4.35, MET** (7 clean legs). Smart barriers skip 19.8%/tok, GPU -1.09 ms; e2e 8/12, unresolved. OFF. [source](../benchmarks/demo/vulkan_27b_llamacpp.json) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns | | ROCm (`BACKEND-GATE-ROCM-VLLM` / `-SGLANG`) | **NOT APPLICABLE: no number measured, claimed or owed.** W0 ctest-green on 4 gfx archs (#41); gfx1201 hipBLAS + Gemma-4 MoE (#140, contributor) ran M0/M1 on 2× R9700, our side CPU-link-verified only. No AMD HW here | The approach-(b) fix (PENDING community) unblocks the first APU model run (M2); the gate becomes a same-box vLLM-ROCm oracle once a model runs ([#41](https://github.com/mudler/vllm.cpp/issues/41)); floor: vLLM | | Tenstorrent Blackhole (`BACKEND-TENSTORRENT`) | **NOT APPLICABLE (speed).** Correctness: OPT-125m STRICT 6/6 e2e on real hardware. Qwen3-0.6B has a device-specific golden and short 4-token warm smoke (~0.28 tok/s), not a completed speed run | Full 16x16 Qwen3 gate, then device-resident tensors + `ttnn::sdpa_decode` before any performance comparison. [Spec](../.agents/specs/tenstorrent-backend.md) | +| `logprobs_mode` (`SAMPLE-LOGPROB-TOKEN-IDS`, #238) | **NO number measured, claimed or owed.** Correctness-only, CPU. The processed modes add one [n, vocab] device->host copy per step when engaged and nothing when not | No bar is owed: these are observation modes, not a path vLLM optimizes either | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | | Embeddings on the ONE surface (ROW 6, `LlamaModel` + `vllm_embed` + `/v1/embeddings`) | **NO number measured, claimed or owed.** Correctness-gated only, CPU: the 2026-08-08 fold (engine path == direct registry path, f64 LAST+normalize reference on the committed fixture) is plumbing, no speed claim | A REAL embedding checkpoint (e5-mistral class) + a same-box `vllm.LLM(task="embed")` oracle; only then does an embed-throughput bar exist | | Parakeet/FastConformer ASR (P1-P4 + ONE-SURFACE fold ROW 1) | **NO number measured, claimed or owed.** Correctness-gated only, CPU f32; the 2026-08-07 surface fold (`vllm_transcribe`, `/v1/audio/transcriptions`) is transcript-byte-identical plumbing, no speed claim. | Floor is `parakeet.cpp`, same clip and box; needs a CUDA provider and a pretrained checkpoint | diff --git a/docs/STATUS.md b/docs/STATUS.md index 1e664c7a4..4a90bd180 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -118,7 +118,7 @@ token-for-token correctness against the pinned oracle. | KV offload to CPU / disk | Built, opt-in, off by default; the disk connector is engine-refused | CPU and disk tiers with identity-checked blocks, selected by `--kv-transfer-config` (or programmatically) over one abstract KVConnector ABI. Worker-side KV store/load is implemented for the LMCache connector only; the CPU/disk connector is scheduler-side only, so the engine now REFUSES it at construction (a loud error, not silently wrong output). Guide: [docs/KV-OFFLOAD.md](KV-OFFLOAD.md) | | LMCache client (`lm://` remote KV) | Built, opt-in, off by default; a working, verified external KV cache | Pure-C++ `lm://` client wired as an `LMCacheConnector`, no `lmcache` in-process; keys agree byte-for-byte with a real vLLM+LMCache peer, mismatched blocks refused. Proven in a real OPT-125m loop vs a live `lmcache.v1.server`: connector-ON tokens are BIT-IDENTICAL to the connector-OFF cold run (both after an in-process restart and from a cold second process). See docs/BENCHMARKS.md | | KV-cache events (for external routers) | Built, off by default; generation and payload gated, live ZMQ transport deferred | Block store/remove/clear events (`BlockStored`/`BlockRemoved`/`AllBlocksCleared`) emitted at the prefix-cache sites when enabled, with a `msgpack` payload byte-identical to vLLM's `msgspec` encoding. Behind a publisher seam faithful to `--kv-events-config`; the live ZMQ transport is not wired yet. Off by default, so the prefix-cache path is byte-identical. | -| Sampling | Supported | Greedy, temperature, top-k/p, min-p, presence/frequency/repetition penalties, seed, stop/stop_token_ids, min_tokens, logit_bias, allowed_token_ids, bad_words, in vLLM's exact order. Custom logits processors are supported through a C-ABI callback (`vllm_logits_processor`, ABI v8): a per-request host callback invoked each decode step to modify the logits before sampling; absent by default (byte-identical). Sample logprobs are emitted end-to-end for `/v1/completions` and `/v1/chat/completions` (`logprobs`/`top_logprobs`). Parallel sampling (`n>1`) is supported: a request fans out into n sequences sharing the prompt, returned as n indexed `choices` (`n==1` byte-identical). Beam search is supported through the `BeamSearch` driver — an outer engine loop that scores beams by cumulative logprob with a length penalty and returns the top `beam_width` sequences (deterministic, token-exact vs vLLM's algorithm); it is wired on the OpenAI `use_beam_search` request field for `/v1/completions` and `/v1/chat/completions` over BOTH the synchronous engine AND the production AsyncLLM HTTP server (via an async `BeamSearchAsync` driver that reuses the same scoring and returns beams token-identical to the sync driver), so beam search runs on the real server (the C-ABI beam params and streaming beam are not exposed yet; per-beam concurrent stepping is a named residual — beams are stepped sequentially, byte-identical to the sync driver). `best_of` is supported on both endpoints (generate `best_of` sequences, return the `n` highest-cumulative-logprob ones; `best_of==n` is the default no-op) — vLLM 0.26 itself has dropped `best_of` from its live path, so this follows the classic OpenAI contract. Prompt logprobs and `echo` are not emitted yet | +| Sampling | Supported | Greedy, temperature, top-k/p, min-p, presence/frequency/repetition penalties, seed, stop/stop_token_ids, min_tokens, logit_bias, allowed_token_ids, bad_words, in vLLM's exact order. Custom logits processors are supported through a C-ABI callback (`vllm_logits_processor`, ABI v8): a per-request host callback invoked each decode step to modify the logits before sampling; absent by default (byte-identical). Sample logprobs are emitted end-to-end for `/v1/completions` and `/v1/chat/completions` (`logprobs`/`top_logprobs`). Parallel sampling (`n>1`) is supported: a request fans out into n sequences sharing the prompt, returned as n indexed `choices` (`n==1` byte-identical). Beam search is supported through the `BeamSearch` driver — an outer engine loop that scores beams by cumulative logprob with a length penalty and returns the top `beam_width` sequences (deterministic, token-exact vs vLLM's algorithm); it is wired on the OpenAI `use_beam_search` request field for `/v1/completions` and `/v1/chat/completions` over BOTH the synchronous engine AND the production AsyncLLM HTTP server (via an async `BeamSearchAsync` driver that reuses the same scoring and returns beams token-identical to the sync driver), so beam search runs on the real server (the C-ABI beam params and streaming beam are not exposed yet; per-beam concurrent stepping is a named residual — beams are stepped sequentially, byte-identical to the sync driver). `best_of` is supported on both endpoints (generate `best_of` sequences, return the `n` highest-cumulative-logprob ones; `best_of==n` is the default no-op). Prompt logprobs and `echo` are not emitted yet. All four `logprobs_mode` values work (`processed_*` shows the top-k mask); in-library only. | | Structured output | Supported (subset), engine-enforced; xgrammar backend W1 (CPU, not yet production-wired) | JSON schema, JSON object, regex, choice, GBNF grammar. Constrained decoding runs in the production engine (native grammar backend, per-step logits bitmask) and is reachable from OpenAI `response_format` and the C ABI (ABI v2 `structured_*` fields). A second, xgrammar-faithful backend (`XgrammarStructuredOutputBackend`, vLLM's default `auto`) is built behind the same seam: it reuses the native pushdown-FSM/trie matcher (xgrammar's own algorithm) and adds the xgrammar JSON-schema→EBNF converter that preserves property declaration order + `any_whitespace` + the `basic_*` grammar, closing the key-order/whitespace/exotic-schema parity gap. CPU-gated (`test_backend_xgrammar` 6/6, RED-first); production wiring + the `auto` fallback + GPU oracle parity are the named residuals (see `.agents/specs/xgrammar-backend.md`) | | Tool-call parsing | 36 parser families / 40 accepted names, streaming | Every vLLM tool parser at the pin except the three Rust/Harmony-backed ones: pure-text parsers ported 1:1, the six engine-backed families reimplemented from their wire formats, all held to the upstream test suites. Selection via `--tool-call-parser` (server), `tool_parser` (C ABI), or template auto-detection; native-syntax forced tool_choice where expressible. Tables: docs/BENCHMARKS.md | | Reasoning parsing (`SAMPLE-REASONING`, ACTIVE, partial coverage) | 9 parsers, streaming | think_auto (auto-detect default: content unless markers appear), deepseek_r1, deepseek_v3 (passthrough) / holo2 (thinking→R1), mistral ([THINK]), minimax_m2 (+append_think), step3, olmo3 - reasoning split engine-side BEFORE tool parsing, streamed as `reasoning` deltas in the chat chunks. Coverage: 9 of upstream's ~28 registered names (remaining text families + engine-backed adapters tracked as W2/W3 in specs/reasoning-parsers.md); each ported parser doctest-gated vs its tests/reasoning case | diff --git a/docs/USAGE.md b/docs/USAGE.md index ab22f9a03..b90dd0ab4 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -777,6 +777,15 @@ auto engine = vllm::entrypoints::LoadedEngine::FromModelDir(model_dir, ep); The underlying portable tensor runtime is `vt::` ([`include/vt/`](../include/vt/)), which carries no ggml or PyTorch dependency. +`Sampler`'s `logprobs_mode` selects which tensor the returned logprobs are read +from, and all four of vLLM's values now work: `raw_logprobs` (the default) and +`raw_logits` are snapshotted before any logits processor runs, so they describe +the MODEL's distribution; `processed_logprobs` and `processed_logits` are taken +after temperature and top-k/top-p, so they describe the distribution actually +SAMPLED from — a token top-k masked away reads `-inf` there and its true value +under the raw pair. It is selectable by constructing a `Sampler` directly; there +is no config, CLI or request field for it yet. + `SamplingParams::logprobs` accepts `-1` for "every vocab entry", as vLLM's does; it returns the same gathered shape a finite count returns, one entry per vocab id per position. (Over HTTP the OpenAI `logprobs` field keeps its own 0..5 range.) diff --git a/include/vllm/v1/sample/sampler.h b/include/vllm/v1/sample/sampler.h index 6069a1d5f..c480db9a7 100644 --- a/include/vllm/v1/sample/sampler.h +++ b/include/vllm/v1/sample/sampler.h @@ -46,11 +46,17 @@ namespace vllm::v1 { // LogprobsMode (vllm/config/model.py::LogprobsMode). Only raw_logprobs is // implemented at T0; the raw_logits / processed_* modes are marked stubs (the // all-greedy processed-logprobs branch and the raw_logits clone are deferred). +// Which tensor the returned logprobs are read from (model.py:82,221 / +// sampler.py:87-93,255-302). The RAW pair is snapshotted before any logits +// processor runs, so it describes the MODEL's distribution; the PROCESSED pair +// is taken after temperature and top-k/top-p, so it describes the distribution +// actually SAMPLED from — a token top-k masked away reads -inf there and its +// true value in the raw modes. Both distinctions are user-visible. enum class LogprobsMode { kRawLogprobs, // default: compute_logprobs(logits) before any mutation - kRawLogits, // STUB (deferred): clone the raw logits - kProcessedLogprobs, // STUB (deferred): logprobs after temperature/top-k/top-p - kProcessedLogits, // STUB (deferred): logits after temperature/top-k/top-p + kRawLogits, // the raw logits themselves, before any mutation + kProcessedLogprobs, // logprobs after temperature/top-k/top-p + kProcessedLogits, // logits after temperature/top-k/top-p }; // Sampler (sampler.py::Sampler). Stateless apart from the logprobs_mode; the @@ -85,8 +91,16 @@ class Sampler { private: // Sampler.sample. Runs steps 7a-7f; returns the [num_reqs] host token ids. + // + // `processed_out` (sampler.py:246-248,262-271,286-302): when non-null AND the + // mode is one of the processed_* pair, sample() fills it with the [n, vocab] + // host snapshot taken at the point upstream takes it — after the logits + // processors on the all-greedy early return, and after temperature and + // top-k/top-p otherwise. Left EMPTY under the raw_* modes, which snapshot in + // forward() before any mutation. std::vector sample(vt::Queue& q, vt::Tensor& logits, - const SamplingMetadata& sampling_metadata) const; + const SamplingMetadata& sampling_metadata, + std::vector* processed_out = nullptr) const; // GreedyArgmax over `logits` -> [n] host int64 ids, reusing a PERSISTENT // device + pinned-host scratch (grow-only) so the greedy decode hot path does diff --git a/scripts/check-public-doc-tables.py b/scripts/check-public-doc-tables.py index f65118c2c..441f8e55a 100755 --- a/scripts/check-public-doc-tables.py +++ b/scripts/check-public-doc-tables.py @@ -485,7 +485,16 @@ def features_errors(text: str) -> list[str]: # (works on the 35B gate model, spec-on output token-identical to spec-off, # 1.15x warm against upstream's 1.41x). Re-pinned byte-tight below after the # merge, so the reduction cannot become untracked growth headroom. - "chars": 243571, + # 243559 since 2026-08-10 (measured 243559): the logprobs_mode row (#238) needs one + # binding line -- three of four modes were runtime-refused stubs and now work. + # Paid for by collapsing the best_of cell's upstream RATIONALE ("vLLM 0.26 itself + # has dropped best_of from its live path..."), which is a why, not a current state, + # and belongs in the row's spec. Deliberately a DIFFERENT collapse from the one PR + # #235 makes in the beam-search prose, so the two open PRs cannot conflict on the + # same text. Every measured number and binding claim kept verbatim; re-pinned + # byte-tight. + # + "chars": 243559, "h2_sections": 11, "long_paragraphs": 82, "oversized_cells": 44, diff --git a/src/vllm/v1/sample/sampler.cpp b/src/vllm/v1/sample/sampler.cpp index 192bdabcd..4e4a1776d 100644 --- a/src/vllm/v1/sample/sampler.cpp +++ b/src/vllm/v1/sample/sampler.cpp @@ -194,9 +194,28 @@ std::vector Sampler::greedy_argmax_host(vt::Queue& q, } std::vector Sampler::sample(vt::Queue& q, vt::Tensor& logits, - const SamplingMetadata& sm) const { + const SamplingMetadata& sm, + std::vector* processed_out) const { const int64_t n = logits.shape[0]; const int64_t vocab = logits.shape[1]; + // processed_* snapshot (sampler.py:262-271,286-302). Non-null only when the + // caller is in one of those two modes AND a request asked for logprobs. + const bool want_processed = processed_out != nullptr; + auto snapshot_processed = [&]() { + processed_out->resize(static_cast(n) * static_cast(vocab)); + if (logprobs_mode_ == LogprobsMode::kProcessedLogits) { + // processed_logits: the mutated logits tensor itself (:267-268). + vt::Backend& b = vt::GetBackend(logits.device.type); + b.Copy(q, processed_out->data(), logits.data, + processed_out->size() * sizeof(float)); + b.Synchronize(q); + } else { + // processed_logprobs: compute_logprobs of it (:269-270). + DeviceBuffer plp(logits.device, q, vt::DType::kF32, {n, vocab}); + vt::ComputeLogprobs(q, plp.tensor(), logits); + plp.download(processed_out->data()); + } + }; VT_CHECK(!(sm.all_greedy && sm.all_random), "sampler: all_greedy and all_random are mutually exclusive"); @@ -206,7 +225,12 @@ std::vector Sampler::sample(vt::Queue& q, vt::Tensor& logits, const bool have_greedy = !sm.all_random; if (have_greedy) { greedy_sampled = greedy_argmax_host(q, logits, n); - if (sm.all_greedy) return greedy_sampled; + if (sm.all_greedy) { + // Upstream returns HERE, before temperature (:262-271), so an all-greedy + // batch's "processed" tensor is the post-logits-processor one. + if (want_processed) snapshot_processed(); + return greedy_sampled; + } } VT_CHECK(sm.temperature.has_value(), @@ -227,6 +251,11 @@ std::vector Sampler::sample(vt::Queue& q, vt::Tensor& logits, // 7d. top_k and/or top_p (materialize the per-req optionals; nullptr => skip). ApplyTopKTopPFromMeta(q, logits, sm.top_k, sm.top_p); + // Upstream takes the processed snapshot inside topk_topp_sampler, i.e. right + // here: after temperature, min_p and top-k/top-p, before the sampling draw + // (:286-290). A token masked away by top-k/top-p therefore reads -inf. + if (want_processed) snapshot_processed(); + // 7e. probs = softmax(logits); random_sample (exponential-noise gumbel-max). DeviceBuffer probs(logits.device, q, vt::DType::kF32, {n, vocab}); vt::ComputeProbs(q, probs.tensor(), logits); @@ -270,20 +299,33 @@ SamplerOutput Sampler::forward(vt::Queue& q, vt::Tensor& logits, "sampler: sampled_ids_out must have num_reqs elements"); } - // 1. Raw-logprobs snapshot BEFORE any mutation (raw_logprobs mode; raw_logits - // and the processed_* modes are deferred stubs — see the header). + // 1. RAW snapshot, BEFORE any mutation (sampler.py:85-93). Only the raw_* + // modes snapshot here; the processed_* pair is taken inside sample(), + // after the mutations they are named for, and overwrites this below + // (sampler.py:104-106). // NOTE: logprob_token_ids (generative-scoring) is a deferred stub, so the // snapshot is driven solely by max_num_logprobs. const std::optional num_logprobs = sm.max_num_logprobs; const bool want_logprobs = num_logprobs.has_value(); + const bool processed_mode = logprobs_mode_ == LogprobsMode::kProcessedLogprobs || + logprobs_mode_ == LogprobsMode::kProcessedLogits; std::vector raw_logprobs; // host [n*vocab] when want_logprobs - if (want_logprobs) { - VT_CHECK(logprobs_mode_ == LogprobsMode::kRawLogprobs, - "sampler: only the raw_logprobs logprobs_mode is implemented at T0"); - DeviceBuffer rlp(logits.device, q, vt::DType::kF32, {n, vocab}); - vt::ComputeLogprobs(q, rlp.tensor(), logits); + if (want_logprobs && !processed_mode) { raw_logprobs.resize(static_cast(n) * static_cast(vocab)); - rlp.download(raw_logprobs.data()); + if (logprobs_mode_ == LogprobsMode::kRawLogits) { + // raw_logits (sampler.py:90-93): the logits THEMSELVES, no log_softmax. + // They are already f32 (checked above), so upstream's `.clone()` / + // `.to(torch.float32)` is exactly this copy out — and it must happen here, + // before the processors below mutate the tensor in place. + vt::Backend& b = vt::GetBackend(logits.device.type); + b.Copy(q, raw_logprobs.data(), logits.data, + raw_logprobs.size() * sizeof(float)); + b.Synchronize(q); + } else { + DeviceBuffer rlp(logits.device, q, vt::DType::kF32, {n, vocab}); + vt::ComputeLogprobs(q, rlp.tensor(), logits); + rlp.download(raw_logprobs.data()); + } } // 2. float32 (already f32; checked above). @@ -326,7 +368,13 @@ SamplerOutput Sampler::forward(vt::Queue& q, vt::Tensor& logits, out.sampled_on_device = true; // host sampled_token_ids intentionally empty return out; } - const std::vector sampled = sample(q, logits, sm); + // processed_* modes take their snapshot INSIDE sample(), where the mutations + // they are named for have happened; it then replaces the (empty) raw one, as + // upstream's `if processed_logprobs is not None` does (:104-106). + std::vector processed; + const std::vector sampled = sample( + q, logits, sm, (want_logprobs && processed_mode) ? &processed : nullptr); + if (!processed.empty()) raw_logprobs = std::move(processed); // Async path fallback (random rows or logprobs requested): the host `sample()` // above already materialized the ids, so mirror them into the device out-tensor // for a uniform async-output D2H. Correct, but no zero-copy win (documented; the diff --git a/tests/scripts/test_check_public_doc_tables.py b/tests/scripts/test_check_public_doc_tables.py index 7b7fb0509..7ad7da808 100644 --- a/tests/scripts/test_check_public_doc_tables.py +++ b/tests/scripts/test_check_public_doc_tables.py @@ -497,6 +497,23 @@ def test_the_status_ratchet_only_ever_moves_down(self) -> None: "change that lowers the ratchet -- never the reverse", ) + def test_the_logprobs_mode_repin_was_paid_for(self) -> None: + # Same contract as the other re-pin guards: the cap may only come down, + # and the reduction must be a real collapse rather than a growth budget. + # The 2026-08-10 logprobs_mode line was paid for by dropping the best_of + # cell's upstream rationale. Restore it and the page must break its cap, + # so a later edit cannot quietly put the prose back and re-spend the + # slack. + text = doc_tables.STATUS.read_text(encoding="utf-8") + restored = text.replace( + "`best_of==n` is the default no-op).", + "`best_of==n` is the default no-op) \u2014 vLLM 0.26 itself has dropped " + "`best_of` from its live path, so this follows the classic OpenAI contract.", + 1, + ) + self.assertNotEqual(restored, text, "the collapsed rationale is not where it was") + self.assertGreater(len(restored), doc_tables.STATUS_RATCHET["chars"]) + def test_the_ratchet_carries_no_hidden_headroom(self) -> None: # A ratchet parked well above the page it guards is not a ratchet: it # silently licenses regrowth up to the old number. The rule is "lower it diff --git a/tests/vllm/v1/sample/test_sampler.cpp b/tests/vllm/v1/sample/test_sampler.cpp index b0033baae..152c9af10 100644 --- a/tests/vllm/v1/sample/test_sampler.cpp +++ b/tests/vllm/v1/sample/test_sampler.cpp @@ -387,3 +387,120 @@ TEST_CASE("Sampler: empty custom-logits-processor map is inert") { auto out = sampler.forward(q, tl, sm); CHECK(out.sampled_token_ids[0][0] == 1); // untouched argmax } + +// --------------------------------------------------------------------------- +// logprobs_mode (issue #238, sampler.py:87-93,255-302). Three of the four modes +// were runtime-refused stubs; these gate what each one actually returns. +// +// The distinction that matters is RAW vs PROCESSED, not logprobs vs logits: the +// raw pair is snapshotted before any mutation and describes the MODEL's +// distribution, while the processed pair is taken after temperature and +// top-k/top-p and describes the distribution actually SAMPLED from. A token +// top-k masks away reads its true value in the raw modes and -inf in the +// processed ones. Every case below uses the same logits so the four modes are +// directly comparable. + +// raw_logits: the logits themselves, NOT log_softmax of them. +TEST_CASE("Sampler: logprobs_mode raw_logits returns the unnormalized logits") { + std::vector logits = {3.0f, 1.0f, 2.0f, 0.0f}; + Tensor tl = Logits(logits, 1, 4); + SamplingMetadata sm; + sm.all_greedy = true; + sm.all_random = false; + sm.max_num_logprobs = 2; + + Sampler sampler(vllm::v1::LogprobsMode::kRawLogits); + Queue q = Q(); + auto out = sampler.forward(q, tl, sm); + + REQUIRE(out.logprobs_tensors.has_value()); + const auto& lt = *out.logprobs_tensors; + REQUIRE(lt.logprobs.size() == 3); + // The raw logit values, verbatim -- every one strictly greater than the + // corresponding log_softmax value, which is what makes this mode observable. + CHECK(lt.logprobs[0] == doctest::Approx(3.0f)); // sampled (token 0) + CHECK(lt.logprobs[1] == doctest::Approx(3.0f)); // top-1 + CHECK(lt.logprobs[2] == doctest::Approx(2.0f)); // top-2 + CHECK(lt.logprob_token_ids[0] == 0); + CHECK(lt.selected_token_ranks[0] == 1); +} + +// The default mode over the same logits, for contrast: normalized, so strictly +// less than the raw logits above. Guards the default against this change. +TEST_CASE("Sampler: logprobs_mode raw_logprobs is unchanged and normalized") { + std::vector logits = {3.0f, 1.0f, 2.0f, 0.0f}; + Tensor tl = Logits(logits, 1, 4); + SamplingMetadata sm; + sm.all_greedy = true; + sm.all_random = false; + sm.max_num_logprobs = 2; + + Sampler sampler; // default == kRawLogprobs + Queue q = Q(); + auto out = sampler.forward(q, tl, sm); + + REQUIRE(out.logprobs_tensors.has_value()); + const auto& lt = *out.logprobs_tensors; + const float lse = 3.0f + std::log(std::exp(0.0f) + std::exp(-2.0f) + + std::exp(-1.0f) + std::exp(-3.0f)); + CHECK(lt.logprobs[0] == doctest::Approx(3.0f - lse)); + CHECK(lt.logprobs[0] < 3.0f); // strictly below the raw_logits answer +} + +// processed_logits under top_k=2: the two surviving tokens keep their +// temperature-scaled logits and the masked ones read -inf. This is the case the +// mode exists for, and no raw mode can produce it. +TEST_CASE("Sampler: logprobs_mode processed_logits shows the top-k mask") { + std::vector logits = {3.0f, 1.0f, 2.0f, 0.0f}; + Tensor tl = Logits(logits, 1, 4); + SamplingMetadata sm; + sm.all_greedy = false; + sm.all_random = true; + sm.temperature = std::vector{1.0f}; + sm.top_k = std::vector{2}; + sm.max_num_logprobs = 3; // ask for enough to see a masked token + + Sampler sampler(vllm::v1::LogprobsMode::kProcessedLogits); + Queue q = Q(); + auto out = sampler.forward(q, tl, sm); + + REQUIRE(out.logprobs_tensors.has_value()); + const auto& lt = *out.logprobs_tensors; + REQUIRE(lt.num_tokens_per_position == 4); // k + 1 + // The kept tokens (0 and 2) hold their logits at temperature 1.0; the tail is + // masked. Column 0 is the sampled token, then the top-k by value. + CHECK(lt.logprobs[1] == doctest::Approx(3.0f)); + CHECK(lt.logprobs[2] == doctest::Approx(2.0f)); + // Whatever landed third is one of the masked tokens -> -inf. + CHECK(lt.logprobs[3] == -std::numeric_limits::infinity()); +} + +// processed_logprobs: the same mask, but renormalized over the surviving +// tokens, so the kept pair sums to 1 in probability space. That renormalization +// is the whole difference from processed_logits. +TEST_CASE("Sampler: logprobs_mode processed_logprobs renormalizes over the kept set") { + std::vector logits = {3.0f, 1.0f, 2.0f, 0.0f}; + Tensor tl = Logits(logits, 1, 4); + SamplingMetadata sm; + sm.all_greedy = false; + sm.all_random = true; + sm.temperature = std::vector{1.0f}; + sm.top_k = std::vector{2}; + sm.max_num_logprobs = 3; + + Sampler sampler(vllm::v1::LogprobsMode::kProcessedLogprobs); + Queue q = Q(); + auto out = sampler.forward(q, tl, sm); + + REQUIRE(out.logprobs_tensors.has_value()); + const auto& lt = *out.logprobs_tensors; + // log_softmax over the SURVIVING pair {3.0, 2.0} only. + const float kept_lse = 3.0f + std::log(std::exp(0.0f) + std::exp(-1.0f)); + CHECK(lt.logprobs[1] == doctest::Approx(3.0f - kept_lse)); + CHECK(lt.logprobs[2] == doctest::Approx(2.0f - kept_lse)); + CHECK(lt.logprobs[3] == -std::numeric_limits::infinity()); + // Renormalized: the two kept tokens carry all the mass. + const double mass = std::exp(static_cast(lt.logprobs[1])) + + std::exp(static_cast(lt.logprobs[2])); + CHECK(mass == doctest::Approx(1.0).epsilon(1e-5)); +}