logprob_token_ids: generative scoring over an explicit token set is unported

[localai-bot]

SamplingParams.logprob_token_ids asks for logprobs at an EXPLICIT set of token
ids rather than the top-k. vLLM's generative-scoring API uses it; we do not have
it, and the seam is half-built.

Current state

include/vllm/v1/sample/metadata.h:107-109 already declares the field:

// Upstream `logprob_token_ids: dict[int, list[int]] | None` (generative-
// scoring)...
std::optional<std::map<int, std::vector<int32_t>>> logprob_token_ids;

but nothing populates it and nothing consumes it. include/vllm/v1/sample/sampler.h:30
records the gather itself as DEFERRED, and src/vllm/v1/sample/sampler.cpp's
snapshot comment says so explicitly: "logprob_token_ids (generative-scoring) is a
deferred stub, so the snapshot is driven solely by max_num_logprobs."

SamplingParams does not carry the field at all
(include/vllm/sampling_params.h:20 lists it among the unported ones).

What upstream does

• vllm/v1/sample/sampler.py:151-225 — gather_specific_token_logprobs: builds a
  padded [batch, max_num_tokens + 1] id matrix with the sampled token in column
  0, gathers logprobs at those ids, masks padded positions with -inf, and
  computes the sampled token's rank over the FULL vocab (not the requested subset,
  so the rank stays meaningful).
• vllm/v1/sample/sampler.py:86,113-118 — it also drives whether the raw snapshot
  is taken at all (if num_logprobs is not None or sampling_metadata.logprob_token_ids),
  which is the line our snapshot comment says we deviate from.
• vllm/v1/sample/sampler.py:133-136 — the precedence rule: when a request supplies
  BOTH logprob_token_ids and a logprobs count, the explicit ids win.
• vllm/sampling_params.py:279-283 — the field, documented as "more efficient than
  logprobs=-1 when you only need logprobs for a small set of tokens".

Why it is worth doing now

It is the last unstarted piece of ROAD-V1-C7's W-plan in
specs/roadmap-v1-completion.md
§3 item 4 ("finish SAMPLE-LOGPROBS/SAMPLE-LOGIT-FILTERS payloads; add
prompt_logprobs, logprobs_mode") that is not blocked on an unmerged PR.
logprobs_mode landed in #238/#258 and moved the row to PARTIAL; this closes it.

It is also the efficient answer to #249: scoring a handful of known tokens should
not require logprobs=-1 and a full vocab sort.

Scope

Port gather_specific_token_logprobs 1:1, add the SamplingParams field and its
InputBatch plumbing, honour the precedence rule, and make the raw snapshot fire
when only logprob_token_ids is set.

Row: SAMPLE-LOGPROB-TOKEN-IDS (.agents/engine-matrix.md:133, PARTIAL after
the logprobs_mode half).

Gate: CPU reference backend, RED-first — an explicit id set returns exactly those
ids plus the sampled token, padding masked to -inf, and the sampled rank
computed over the full vocab rather than the subset.