diff --git a/.agents/NOW.md b/.agents/NOW.md index bca56be40..1bb14d227 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -30,7 +30,7 @@ Working head: `row/backend-rocm-w0` (#41). Prior: benchmark checkpoint | `BACKEND-ROCM` W0 | Skeleton in; **HIP never compiled** (no AMD HW) | #41 contributors build it; a compile error IS the deliverable | | TP spike #287 (PR #143) | **LANDED** ([spec](specs/tensor-parallelism-spike.md)); DSpark rider grounded | dispatch TP-W1 (CPU-able) | | Release | SPIKE; 30/30 | #129 | -| Surface coverage (`ARCH-ONE-SURFACE`) | **ROW 8 LANDED; #139 repair CPU-GREEN**: ABI v14 stable; registry-resolved named platform; DSR 39→32; execution guard 52/52 | Fresh re-review #139; CUDA A/B residual | +| Surface coverage (`ARCH-ONE-SURFACE`) | ROW 8 + #139 IN; **ROW 6 IN REVIEW (#137): embeddings LIVE — `LlamaModel` arch, PoolingRunner in the step, `vllm_embed` v15, `/v1/embeddings`, fold gate 4/4-231, 9 kills** | Merge #137; real-ckpt oracle cosine residual | In-flight (default-OFF, not pushed): `laguna-fp4proj-prod`, laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. @@ -46,10 +46,8 @@ both gate models, reproduced 2–3x on an idle box. See [gates.md](gates.md) and ## Next actions -1. **Spike the Parakeet encoder row.** Upstream vLLM has `parakeet.py` + - `conformer_encoder.py` as the audio encoder of `nano_nemotron_vl.py`, which we - already carry `MODEL-MM-nano-nemotron-vl-*` rows for, so it is owed mirror work. - The transducer decode half (RNN-T/TDT/CTC) is NOT in vLLM: separate scope call. +1. **Spike the Parakeet encoder row** (vLLM carries it inside + `nano_nemotron_vl.py`; the transducer half is NOT in vLLM: separate call). 2. **Qwen3.5-4B serving follow-up:** bind the default-ON async-serving path against the same oracle before attributing the remaining TPOT gap. 2. **Merge the invocation-parity prevention** (CI guard + AGENTS.md checklist); diff --git a/.agents/coordination.md b/.agents/coordination.md index 61f1d9e46..335539aef 100644 --- a/.agents/coordination.md +++ b/.agents/coordination.md @@ -1619,6 +1619,8 @@ items a-runner/b stay with the async/GDN `runner.cpp` owners. | `CLAIM-POOLING` | `ENG-POOLER-SEQ` (INVENTORIED-implicit→ACTIVE, W1→**W2**), `ENG-POOLING-RUNNER` (**NEW row, ACTIVE, W3**), `SERVE-POOLING-ENDPOINTS` (INVENTORIED→SPIKE) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/claim-pooling-w2w3` (CPU build `build-cpu` `-DVLLM_CPP_CUDA=OFF -DVLLM_CPP_SERVER=ON` Release + CPU run; NO dgx/GPU — the pooling reductions + activations + runner are host arithmetic) | branch `claim-pooling-w2w3`, base `main` `edf68c91` (confirmed via `git rev-parse HEAD`) | Pooling task class HIGH-priority feature-gap #2. W0 spike + W1 CPU pooler OP (prior pass); **W2 pooler HEADS composite + `SequencePooler`/`DispatchPooler` + `PoolerConfig`/`PoolingParams` and W3 pooling RUNNER path (this pass).** Owns ONLY: NEW `include/vllm/model_executor/layers/pooler/{pooling_metadata,methods,activations,common,pooling_params,pooler_config,heads,poolers,dispatch_pooler}.h` + `src/vllm/model_executor/layers/pooler/{methods,activations,heads,poolers,dispatch_pooler}.cpp` + NEW `include/vllm/v1/worker/gpu/pool/pooling_runner.h` + `src/vllm/v1/worker/gpu/pool/pooling_runner.cpp`; NEW `tests/vllm/model_executor/layers/pooler/{test_pooler,test_pooler_heads}.cpp` + `tests/vllm/v1/worker/gpu/pool/test_pooling_runner.cpp`; `CMakeLists.txt` (6 source lines) + `tests/CMakeLists.txt` (3 tests); NEW `.agents/specs/pooling-task-class.md`; the `ENG-POOLER-SEQ` + NEW `ENG-POOLING-RUNNER` engine-matrix rows + `SERVE-POOLING-ENDPOINTS` note + engine Serving/Total rollup (Serving 21→22/ACTIVE 6→7, Total 130→131/ACTIVE 47→48) + `scripts/check-agent-record.py` ENGINE 130→131; the record surfaces (this row, `roadmap_v1.md` gap #2, `docs/STATUS.md`, `docs/BENCHMARKS.md`, `feature-matrix.md` MODEL-POOLING note, `parity-ledger.md`, `state.md`). **NON-COLLISION:** additive NEW files only — the sole edits to existing compiled headers are ADDITIVE (methods.h defaulted virtuals, pooling_metadata.h new fields); ZERO edits to any existing production forward/runner path; NO pooling MODEL row created (concrete embedding model + real-oracle cosine gate is the named W3-model residual), so README/Metal/model-matrix rows untouched. | `ACTIVE` | 2026-07-29 — **W2 + W3 LANDED + CPU-GATED (foreground, NOT pushed).** `test_pooler_heads` 27/27 (240 asserts, Embedding/Classifier heads + SequencePooler + DispatchPooler incl. mixed embed+classify batch + ctor validation) and `test_pooling_runner` 5/5 (14 asserts, runner path + STRUCTURAL cosine-parity gate vs double-precision LAST+normalize ref) — plus W1 `test_pooler` 17/17 unchanged. RED-first proven: disable matryoshka slice + logit_mean → 8 cases/50 asserts fail (heads); CLS-instead-of-LAST drops cosine <0.5 + disable normalize → 2 unit-L2 asserts fail (runner). Clean CPU `-Wall -Wextra -Werror` 0-warn full-library build. **HONEST RESIDUAL:** the cosine gate is STRUCTURAL (synthetic weights) — the real-model `vllm.LLM(task="embed").encode` oracle cosine gate needs a registered concrete embedding model forward (W3-model, no number fabricated). Residuals (spec §Work breakdown): concrete pooling MODEL + real-oracle cosine gate (W3-model), endpoints /v1/embeddings+score+rerank+classify (W4), tokwise AllPool/StepPool (W5). Prior 2026-07-28 — W0 spike + W1 pooler OP LANDED + CPU-GATED: `test_pooler` 17/17 (50 asserts) vs double-precision refs, RED-first proven. | | `CLAIM-DSV4-GGUF-LOADER` | `QUANT-GGUF-IQ2_XXS` (INVENTORIED→ACTIVE), `QUANT-GGUF-Q2_K` (INVENTORIED→ACTIVE); cross-refs `MODEL-TEXT-deepseek-v4-deepseek-v4-for-causal-lm` (stays `SPIKE`, owned by `CLAIM-DEEPSEEK-V4-IMPL`) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/gguf-iquant-dsv4` (CPU-only `build-cpu` `-DVLLM_CPP_CUDA=OFF`; NO GPU, NO 90 GB download — the dequant unit gate uses known packed bytes; the GGUF header was HTTP-range-read, no download) | branch `feat/gguf-iquant-dsv4`, base `main` `4d1be010` (confirmed via `git rev-parse HEAD`) | GGUF IQ2_XXS + Q2_K dequant, the DeepSeek-V4-Flash single-Spark GGUF quant-path brick (W1). Owns ONLY the GGUF/quant PATH (NOT the forward — the forward TUs stay owned by `CLAIM-DEEPSEEK-V4-*`): the two `dequantize_row_*` decoders + grid/sign tables in `src/vt/cpu/cpu_quant_dequant.cpp`; the `kQ2_K`/`kIQ2_XXS` vt block dtype registration in `src/vt/dtype.{h,cpp}` + `src/vt/ops.cpp`; the id-16 reader trait in `gguf_reader.cpp` + ids-10/16 dispatch in `gguf_dequant.cpp`; `tests/vllm/test_gguf_dequant.cpp` + `tests/vt/test_ops_quant_traits.cpp`; the two `QUANT-GGUF-*` rows; NEW `.agents/specs/gguf-iquant-dsv4.md`; the V4 GGUF-loadable note on the model-matrix V4 row (row stays SPIKE); the record surfaces. **NON-COLLISION:** additive within the existing GGUF dequant switch + vt block table — does NOT touch any DeepSeek-V4 forward TU (`deepseek_v4.{cpp,h}`/`_dsa`/`_weights`), README, or Metal; the k-quant/NVFP4 decoders are byte-unchanged. | `ACTIVE` | 2026-07-29 — **W1 LANDED + CPU-GATED (foreground, NOT pushed).** IQ2_XXS (id 16, codebook `iq2xxs_grid`+signs+4-bit scale) + Q2_K (id 10, nibble sub-scale/min) ported 1:1 from llama.cpp `ggml-quants.c` `237ad9b96`; both DEQUANT-ONLY (no vec_dot ⇒ route to expand-bf16). `test_gguf_dequant` **15/15·480** (hand-derived literals: IQ2_XXS grid[1] byte0=0x2b→5.375, ksigns[1] flips j=0,7→±3.0, db 0.125/0.375; Q2_K 5.75/-0.25/2.5/0.25) + `test_ops_quant_traits` **9/9·5643** (dequant-only contract). All 7 changed TUs clean under full `-Werror`; the `voxtral.cpp` GCC-13 `-Werror=array-bounds` FP PROVEN pre-existing (fails at base with this diff's `dtype.h` reverted), neutralized only to link the test binaries. **W2 (V4-GGUF loader) DERIVED not landed:** HTTP-range-read the real `UD-IQ2_XXS` header — `general.architecture=deepseek4`, `general.file_type=19` (=IQ2_XXS), `split.tensors.count=1328`, full `deepseek4.*` config-KV schema; the tensor NAME manifest is beyond the CDN range cap + uncached ⇒ the V4 registry GGUF reject STAYS. Residuals: V4 forward (W3-W8, multi-Spark) + the V4-GGUF name map (W2, manifest-blocked) + a vec_dot perf leaf. | +| `CLAIM-EMBEDDINGS-ONE-SURFACE` | `ENG-POOLING-RUNNER` (live engine-step invocation), `SERVE-POOLING-ENDPOINTS` (SPIKE→ACTIVE, `/v1/embeddings`), `MODEL-EMBED-llama-llama-for-causal-lm` (INVENTORIED→ACTIVE; PARTIAL on merge — only the `LlamaModel` membership registered); cross-refs `ENG-POOLER-SEQ` (stays `CLAIM-POOLING`, ops untouched) | Claude Code (fable-5) helper, task #285 | isolated worktree `/home/mudler/_git/vllm.cpp-embeddings-one-surface` (CPU-only; lean per-target builds under disk pressure) | branch `row/EMBEDDINGS-ONE-SURFACE`, base `main` `b44ad337`, DRAFT PR #137 (the reservation) | ARCH-ONE-SURFACE fold ROW 6: embeddings/pooling through the ONE surface. Owns: NEW `src/vllm/model_executor/models/llama_embedding_registry.cpp` + `LoadLlamaModelEmbeddingWeights` (llama_weights.cpp) + `Qwen3DenseModel::ForwardHidden` (qwen3.{h,cpp} additive tail); the ADDITIVE task-gated pooling plumb (`LoadedModel::pooler()`, runner `pooling_runner_`+`pool_tokens`, `Request/EngineCoreRequest::pooling_params`, `ModelRunnerOutput::pooler_output`, scheduler pooling stop, `EngineCoreOutput/RequestOutput::pooling_output`, `LLMEngine::add_pooling_request/embed`, `ResolveAsyncEnabled(is_pooling_model)`); `vllm_embed`/`vllm_embedding_result_free` ABI v15 (vllm.h + vllm_c.cpp incl. the refuse-both-directions guards + the v13 `vllm_complete_tokens` missing-guard fix); `handle_embeddings` + `set_embedder` + task-conditional route (api_server.{h,cpp}) + server main pooling dispatch; NEW fixture `tests/vllm/models/fixtures/llama_embed_e2e` + `scripts/mm/llama_embed_fixture_gen.py` + `tests/vllm/models/test_llama_embedding_fold.cpp`; test/guard updates (test_capi v15 section + floor pin >= 15, test_dlopen symbols, c_header_compile.c, test_api_server embeddings section, test_model_registry/gguf arch pins, check-supported-models ARCH_TOKEN_RE); allowlist row removal + FEATURES/STATUS/BENCHMARKS rows + matrices + specs. **NON-COLLISION:** every engine hook is task-gated on `is_pooling_model`/`pooling_params` (nullopt/false = byte-identical text path); no SACRED path rewritten; no example added. | `ACTIVE` | 2026-08-08 — CPU-LANDED on the branch: fold gate `test_llama_embedding_fold` 4/4-231 (engine path == direct registry path + f64 LAST+normalize ref + chunked is_valid arm), `test_capi` 48/48-462, `test_dlopen` 30/30, server suite 50/50, registry 24/24-820, engine suites green (scheduler 423, llm_engine 204, engine_core 44, output_processor 77, qwen3_forward 1557, async_llm 342, llama_forward 509); 9 mutation kills (floor pin, refuse both directions, route gating both ways, engine-step invocation, scheduler stop, registry info pin, async-off wire). RESIDUAL: real embedding checkpoint + `LLM(task="embed")` oracle cosine. | + diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 07c09522f..82a97d9b3 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -42,11 +42,11 @@ forensics: roadmap_v1.md and the parity ledger. | Sampling and generation | 15 | 4 | 2 | 0 | 0 | 3 | 0 | 1 | 5 | | Structured output and tools | 7 | 0 | 3 | 0 | 0 | 2 | 0 | 0 | 2 | | Speculative decoding | 21 | 0 | 0 | 1 | 0 | 4 | 0 | 4 | 11 | -| Serving, API, CLI, library | 25 | 10 | 2 | 2 | 0 | 4 | 2 | 1 | 4 | +| Serving, API, CLI, library | 25 | 10 | 2 | 1 | 0 | 5 | 2 | 1 | 4 | | LoRA and adapters | 2 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | | Long context and attention | 10 | 5 | 0 | 0 | 1 | 0 | 1 | 0 | 3 | | Loading, tokenizer, config | 9 | 3 | 3 | 0 | 0 | 0 | 1 | 1 | 1 | -| **Total** | **143** | **36** | **16** | **5** | **7** | **20** | **8** | **9** | **41** | +| **Total** | **143** | **36** | **16** | **4** | **7** | **21** | **8** | **9** | **41** | ## Engine core and scheduling @@ -209,9 +209,9 @@ claims it. | `SERVE-E2E-NIGHTLY` | Server conformance and real-model nightly suites for all release gates | T0 | `tests/entrypoints/openai/`; `tests/v1/e2e/`; `.buildkite/test-pipeline.yaml` | current unit/conformance tests only; no scheduled DGX suite | `tests/vllm/entrypoints/openai/test_conformance.cpp:1`; `tests/parity/test_qwen36_paged_engine.cpp:78`; `tests/parity/test_qwen27_paged_engine.cpp:110` | `planned: specs/server-e2e-nightly.md` | `INVENTORIED` | - | | `ENG-RELEASE-BINARIES` | Downloadable host-ABI-specific `vllm-server` bundles: adaptive CPU and fat CUDA primary artifacts, optional per-SM diagnostics, and literal-static feasibility boundary | T0 | vLLM release lanes `.buildkite/release-pipeline.yaml:1-18,34-170` @ `555967922`; release-image dependency boundary `docker/Dockerfile.cpu:262-290` | server target only `examples/CMakeLists.txt:54-64`; CPU per-TU/runtime-dispatch baseline `CMakeLists.txt:870-890`, `src/vt/cpu/cpu_matmul_elem.cpp:553-612`, `src/vt/cpu/cpu_quant_dot_arm.cpp:39-77`; cross-family CUDA fat/per-source-gencode and multi-SM AOT gaps remain; no install/archive/publish implementation | help smoke only `examples/CMakeLists.txt:59-63`; issue `#117`; user-reviewed fat-CUDA/adaptive-CPU matrix and gates in [release-binary-matrix.md](specs/release-binary-matrix.md) | [release-binary-matrix.md](specs/release-binary-matrix.md) | `SPIKE` | `CLAIM-ENG-RELEASE-BINARIES-SPIKE` | | `SERVE-CLI-CHAT` | Interactive `chat` and `complete` commands against a running OpenAI-compatible server, plus preservation of the existing local-model completion invocation | T1 | registration `vllm/entrypoints/cli/main.py:17-37,73-98`; client/model resolution + stream shaping `vllm/entrypoints/cli/openai.py:30-100`; chat `:155-234`; complete `:237-312` at `5559679229` | current in-process completion only `examples/cli/main.cpp:1-207`; remote command implementation absent | C-ABI stream baseline `tests/capi/test_capi.cpp:567-711`; chat-template baseline `tests/capi/test_chat_prompt.cpp:37-89`; command/fake-server tests absent | [cli-chat-complete.md](specs/cli-chat-complete.md) | `ANCHOR-BACKFILL` | `CLAIM-SERVE-CLI-CHAT-SPIKE` | -| `SERVE-POOLING-ENDPOINTS` | Embeddings, pooling, score, rerank, classify HTTP surface (`/v1/embeddings`, `/pooling`, `/score`, `/rerank`, `/classify`). **SPIKED 2026-07-28 (`CLAIM-POOLING`):** the whole pooling task class is scoped in [pooling-task-class.md](specs/pooling-task-class.md) — endpoints depend on the pooling RUNNER (`ENG-POOLING-RUNNER`, W3 LANDED 2026-07-29 — `PoolingRunner` returns pooled data via the model `Pooler`) and a concrete pooling model. Endpoint protocol + handler port is the W4 brick | T2 | `vllm/entrypoints/pooling/embed/api_router.py:28`; `vllm/entrypoints/pooling/scoring/api_router.py:37,71`; `vllm/entrypoints/pooling/classify/api_router.py:26` | - | - | [pooling-task-class.md](specs/pooling-task-class.md) | `SPIKE` | `CLAIM-POOLING` | +| `SERVE-POOLING-ENDPOINTS` | Embeddings, pooling, score, rerank, classify HTTP surface (`/v1/embeddings`, `/pooling`, `/score`, `/rerank`, `/classify`). **SPIKED 2026-07-28 (`CLAIM-POOLING`):** the whole pooling task class is scoped in [pooling-task-class.md](specs/pooling-task-class.md). **`/v1/embeddings` LIVE 2026-08-08 (ARCH-ONE-SURFACE ROW 6, `CLAIM-EMBEDDINGS-ONE-SURFACE`):** task-conditional registration (embed/api_router.py:22-28 mirror; the route exists ONLY on a pooling-model server, and the generate routes do not — both directions socket-404-pinned), OpenAI request/response shape (string-or-array input; `dimensions`/base64/token-arrays are named-residual 400s), handler drives the ONE engine path (`LoadedEngine -> LLMEngine::embed -> registry forward -> PoolingRunner`) — the same path `vllm_embed` (ABI v15) drives. RESIDUALS: `/pooling`, `/score`, `/rerank`, `/classify` (need a classify arch) | T2 | `vllm/entrypoints/pooling/embed/api_router.py:28`; `vllm/entrypoints/pooling/embed/protocol.py:34,173-185`; `vllm/entrypoints/pooling/scoring/api_router.py:37,71`; `vllm/entrypoints/pooling/classify/api_router.py:26` | `src/vllm/entrypoints/openai/api_server.cpp` `handle_embeddings` + the `if (embedder_)` route gate; `examples/server/main.cpp` pooling task dispatch | `tests/vllm/entrypoints/openai/test_api_server.cpp` embeddings section (dispatch shape + socket smoke + BOTH-direction 404 pins) | [embeddings-one-surface.md](specs/embeddings-one-surface.md) | `ACTIVE` | `CLAIM-EMBEDDINGS-ONE-SURFACE` | | `ENG-POOLER-SEQ` | The non-generative POOLER OP — turn hidden states into a pooled embedding/logit row instead of a sampled token. **W1 LANDED + CPU-GATED 2026-07-28 (`CLAIM-POOLING`, NOT pushed):** the sequence pooling methods `CLSPool`/`LastPool`/`MeanPool` (+ `GetSeqPoolingMethod` factory) over a packed `[num_tokens, hidden]` CPU buffer keyed by a minimal `PoolingCursor` (CLS/MEAN reject partial prefill, LAST allows it, MeanPool upcasts to float32) and the activation heads `PoolerIdentity`/`PoolerNormalize` (L2 `F.normalize`)/`PoolerMultiLabelClassify` (sigmoid)/`PoolerClassify` (sigmoid if `num_labels<2` else `softmax`). Unit-gated vs DOUBLE-PRECISION references, RED-first. **W2 LANDED + CPU-GATED 2026-07-29 (`CLAIM-POOLING`, NOT pushed):** the pooler HEADS composite (`EmbeddingPoolerHead` = projector→matryoshka→normalize; `ClassifierPoolerHead` = classifier→`(logit-mean)/sigma`→activation), the `SequencePooler` (method∩head task intersection) + `PoolerForEmbed`/`PoolerForClassify` factories, the `DispatchPooler` groupby-task routing (`ForEmbedding`/`ForSeqCls` + a mixed embed+classify batch + ctor task-support validation), and the `PoolerConfig`/`PoolingParams`/`PoolingParamsUpdate` structs; `test_pooler_heads` 27/27 (240 asserts) vs double-precision refs, RED-first (disable matryoshka slice + logit_mean calibration → 8 cases / 50 asserts fail). RESIDUALS (named, spec §Work breakdown): the endpoints (W4), tokwise `AllPool`/`StepPool` (W5), a concrete pooling MODEL + real-oracle cosine gate (W3-model — see `ENG-POOLING-RUNNER`) | T2 | `vllm/model_executor/layers/pooler/seqwise/methods.py:35-121`; `vllm/model_executor/layers/pooler/activations.py:106-158`; `vllm/model_executor/layers/pooler/seqwise/heads.py:19-196`; `vllm/model_executor/layers/pooler/seqwise/poolers.py:41-138`; `vllm/model_executor/layers/pooler/special.py:23-140`; `vllm/model_executor/layers/pooler/common.py:12-30`; `vllm/pooling_params.py:35-70`; `vllm/config/pooler.py:16-90`; `vllm/v1/pool/metadata.py:13-71`; `tests/model_executor/layers/test_pooler_methods.py`, `tests/model_executor/layers/test_pooler_activations.py`, `tests/model_executor/layers/test_pooler_heads.py` | `include/vllm/model_executor/layers/pooler/{methods,activations,pooling_metadata,common,pooling_params,pooler_config,heads,poolers,dispatch_pooler}.h` + `src/vllm/model_executor/layers/pooler/{methods,activations,heads,poolers,dispatch_pooler}.cpp` — anchor `src/vllm/model_executor/layers/pooler/dispatch_pooler.cpp:13` | `tests/vllm/model_executor/layers/pooler/test_pooler.cpp` (CLS/LAST/MEAN + factory + activations, 50 asserts) + `test_pooler_heads.cpp` (Embedding/Classifier heads + SequencePooler + DispatchPooler, 240 asserts) — anchor `tests/vllm/model_executor/layers/pooler/test_pooler.cpp:81` | [pooling-task-class.md](specs/pooling-task-class.md) | `ACTIVE` | `CLAIM-POOLING` | -| `ENG-POOLING-RUNNER` | The pooling RUNNER path — where the generation runner SAMPLES a token, the pooling runner applies the model's `Pooler` to the last hidden state and returns the POOLED DATA (embedding vector / classification logit row). **W3 LANDED + CPU-GATED 2026-07-29 (`CLAIM-POOLING`, NOT pushed):** `PoolingRunner` over a packed `[num_tokens, hidden]` last-hidden-state buffer + a `PoolingMetadata` — `Pool()` delegates to the model pooler (`DispatchPooler.ForEmbedding`), `GetSupportedTasks()`, `ComputeValid()` (`seq_lens==prompt_len`). GATE: a STRUCTURAL cosine-parity gate — the runner's embedding vs an independent double-precision LAST+normalize reference is cosine≈1 (5 cases / 14 asserts), RED-first (CLS-instead-of-LAST drops cosine <0.5; disable normalize → 2 unit-L2 asserts fail). GENERALIZATION DEVIATION: upstream `pooling_runner.py` hardcodes LAST+normalize; we route through the model `Pooler` (the general bert.py path), strictly more capable. HONEST RESIDUAL (named): the REAL-model oracle cosine gate (`vllm.LLM(task="embed").encode`) needs a registered concrete embedding model's forward — no such model is registered yet (W3-model), so no cosine-vs-oracle number is fabricated. The InputBatch→PoolingMetadata construction (logits_indices gather) rides the endpoint brick (W4) | T2 | `vllm/v1/worker/gpu/pool/pooling_runner.py:18-46`; `vllm/tasks.py:10`; `tests/models/language/pooling/test_embedding.py` (real-oracle gate, DEFERRED) | `include/vllm/v1/worker/gpu/pool/pooling_runner.h` + `src/vllm/v1/worker/gpu/pool/pooling_runner.cpp` — anchor `src/vllm/v1/worker/gpu/pool/pooling_runner.cpp:11` | `tests/vllm/v1/worker/gpu/pool/test_pooling_runner.cpp` (runner path + structural cosine gate, 14 asserts, RED-first) — anchor `tests/vllm/v1/worker/gpu/pool/test_pooling_runner.cpp:136` | [pooling-task-class.md](specs/pooling-task-class.md) | `ACTIVE` | `CLAIM-POOLING` | +| `ENG-POOLING-RUNNER` | The pooling RUNNER path — where the generation runner SAMPLES a token, the pooling runner applies the model's `Pooler` to the last hidden state and returns the POOLED DATA (embedding vector / classification logit row). **W3 LANDED + CPU-GATED 2026-07-29 (`CLAIM-POOLING`, NOT pushed):** `PoolingRunner` over a packed `[num_tokens, hidden]` last-hidden-state buffer + a `PoolingMetadata` — `Pool()` delegates to the model pooler (`DispatchPooler.ForEmbedding`), `GetSupportedTasks()`, `ComputeValid()` (`seq_lens==prompt_len`). GATE: a STRUCTURAL cosine-parity gate — the runner's embedding vs an independent double-precision LAST+normalize reference is cosine≈1 (5 cases / 14 asserts), RED-first (CLS-instead-of-LAST drops cosine <0.5; disable normalize → 2 unit-L2 asserts fail). GENERALIZATION DEVIATION: upstream `pooling_runner.py` hardcodes LAST+normalize; we route through the model `Pooler` (the general bert.py path), strictly more capable. HONEST RESIDUAL (named): the REAL-model oracle cosine gate (`vllm.LLM(task="embed").encode`) needs a registered concrete embedding model's forward — no such model is registered yet (W3-model), so no cosine-vs-oracle number is fabricated. **LIVE IN THE ENGINE STEP 2026-08-08 (ARCH-ONE-SURFACE ROW 6, `CLAIM-EMBEDDINGS-ONE-SURFACE`):** `GPUModelRunner` builds a `PoolingRunner` iff the loaded model registration declares `is_pooling_model` (gpu/model_runner.py:368-369 mirror) and `sample_tokens` routes to `pool_tokens()` — pooled data instead of sampled tokens (model_runner.py:1586-1607), validity = the discard predicate (`seq_len < num_tokens` == upstream is_valid, pooling_runner.py:40-41); the scheduler finishes a pooling request on pooled output (scheduler.py:1718-1721) and `EngineCoreOutput.pooling_output` carries it out; async scheduling resolves OFF for pooling models (config/vllm.py:1068-1073, the landed ResolveAsyncScheduling arm now WIRED at model_loader.cpp). First registered pooling arch: `LlamaModel` (`MODEL-EMBED-llama-llama-for-causal-lm`). The fold gate re-anchors the lane's cosine gate THROUGH the registry/runner path: engine path == direct `ModelRegistry::Forward`+`PoolingRunner` path, identical vectors + f64 LAST+normalize reference + chunked-prefill arm (`test_llama_embedding_fold` 4/4-231). REMAINING RESIDUAL: the REAL-model `vllm.LLM(task="embed").encode` oracle cosine (synthetic fixture only — no number fabricated) | T2 | `vllm/v1/worker/gpu/pool/pooling_runner.py:18-46`; `vllm/v1/worker/gpu/model_runner.py:368-369,1586-1607`; `vllm/v1/core/sched/scheduler.py:1718-1721,1837`; `vllm/tasks.py:10`; `tests/models/language/pooling/test_embedding.py` (real-oracle gate, DEFERRED) | `include/vllm/v1/worker/gpu/pool/pooling_runner.h` + `src/vllm/v1/worker/gpu/pool/pooling_runner.cpp:11`; live invocation `src/vllm/v1/worker/gpu/runner.cpp` `pool_tokens` + the `pooling_runner_` ctor gate; scheduler stop `src/vllm/v1/core/sched/scheduler.cpp` pooling elif | `tests/vllm/v1/worker/gpu/pool/test_pooling_runner.cpp:136` (structural cosine gate) + `tests/vllm/models/test_llama_embedding_fold.cpp:206` (registry/engine-path arm, 4/4-231, mutation-killed x9) | [pooling-task-class.md](specs/pooling-task-class.md) + [embeddings-one-surface.md](specs/embeddings-one-surface.md) | `ACTIVE` | `CLAIM-EMBEDDINGS-ONE-SURFACE` | | `SERVE-RESPONSES-MESSAGES` | Responses, Anthropic messages, audio | T2 | `vllm/entrypoints/openai/responses/api_router.py:48`; `vllm/entrypoints/anthropic/api_router.py:49`; `vllm/entrypoints/speech_to_text/transcription/api_router.py:1` | - | - | `planned: specs/responses-messages-endpoints.md` | `INVENTORIED` | - | | `SERVE-ADMIN` | Abort-requests, sleep, pause/resume, profiling, RL weight updates. **`/abort_requests` LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-SERVE-ENDPOINTS`, NOT pushed):** `POST /abort_requests` (from the dev/rlhf admin router) parses `{request_ids:[...]}` and aborts exactly those (external) ids via an injected abort callback wired to the engine abort path (`AsyncLLM::abort`); an empty/missing list means "abort all in-flight" (the callback decides). Response `{"status":"aborted","aborted":}`; malformed JSON → 400 `{"detail":"Invalid JSON format"}`; abort failure → 500 `{"error":...}` — all three shapes mirror the upstream router verbatim. ADDITIVE + opt-in (route registered only when the abort callback is attached → 404 otherwise). **PRODUCTION `main.cpp` WIRING LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-SERVE-PROD-WIRING`, NOT pushed):** the shipped `vllm-server` binary now wires `/abort_requests` to the LIVE `AsyncLLM::abort` through the shared `ConfigureUtilityEndpoints` seam, DEV-mode gated behind the new `--enable-server-dev-mode` flag — mirroring vLLM registering the dev/rlhf router only under `if envs.VLLM_SERVER_DEV_MODE` (api_server.py:238; envs.py:157 default 0). Explicit-id abort tears the request down and reports the exact drop in unfinished requests (before−after); empty `request_ids` (abort-ALL) reports 0 — NAMED RESIDUAL (AsyncLLM exposes no active-request-id accessor). RESIDUAL: the abort-ALL enumeration (missing `AsyncLLM::active_request_ids()`); `/sleep`/`/wake_up`/`/is_sleeping`, `/pause`/`/resume`, `/start_profile`/`/stop_profile`, weight-update/EP endpoints still INVENTORIED | T2/T3 | `vllm/entrypoints/serve/dev/rlhf/api_router.py:94-138` (abort_requests); dev-mode gate `vllm/entrypoints/openai/api_server.py:238-240`, `vllm/entrypoints/serve/__init__.py:35`, `vllm/envs.py:157`; `vllm/entrypoints/serve/dev/sleep/api_router.py:21`; `vllm/entrypoints/serve/dev/rlhf/api_router.py:29,74,136`; `vllm/entrypoints/serve/profile/api_router.py:21` | handler `src/vllm/entrypoints/openai/api_server.cpp:488` (`handle_abort_requests`); opt-in setter `include/vllm/entrypoints/openai/api_server.h:156` (`set_abort_requests`); production seam `src/vllm/entrypoints/openai/api_server.cpp` (`ConfigureUtilityEndpoints`, before/after delta-count) + `examples/server/main.cpp` (`--enable-server-dev-mode`); engine abort path `include/vllm/v1/engine/async_llm.h:115` (`abort`) | `tests/vllm/entrypoints/openai/test_api_server.cpp:1104` (shape + callback wiring: explicit ids passthrough, empty→abort-all branch, malformed→400),`:1143` (aborts an in-flight AsyncLLM request → `has_unfinished_requests()` false),`:1250` (opt-in route gate: 404 no-callback → 200 attached, RED-first),`:1319` (**production seam: dev-mode gate 404→200, live abort exact delta-count==1, empty→0**) — in the 32/32 / 420-assertion suite | [admin-endpoints.md](specs/admin-endpoints.md) | `ANCHOR-BACKFILL` | `CLAIM-C8-SERVE-PROD-WIRING` | | `SERVE-VIDEOS-OAI` | `/v1/videos` in OpenAI's Sora WIRE SHAPE, over the vLLM-Omni-derived job endpoints. **CPU-LANDED + GATED 2026-08-06 (`CLAIM-SERVE-VIDEOS-OAI`):** the OpenAI request spellings (`model`, `size` "WxH", `seconds` as a number OR the string enum OpenAI actually types) parse as ALIASES onto the existing native members, NATIVE-wins precedence applied PER-AXIS, both spellings validated either way so a malformed alias is a 400 even when overridden; an unserved `model` is a job `warning` echoed for the job's whole life, never a rejection (a Sora client cannot know the local model's name); and `GET /v1/videos/{id}/content` serves the finished MP4 (404 unknown / 409 unfinished / 500 failed / 500 vanished), without which a caller could start and poll a job but never fetch the result over HTTP. All four routes still register ONLY with a `VideoRunner` attached, now gated over a REAL socket. RESIDUALS (named): OpenAI's status vocabulary/id shape is not mirrored; reference conditioning (`input_reference`, the `metadata` video/audio references) is a stacked follow-up row; the real-weights leg rides the H3 GB10/disk window. | T2 | OpenAI Sora video API (`POST /v1/videos`, `GET /v1/videos/{video_id}/content`); vLLM-Omni `vllm/entrypoints/openai/video/api_router.py` (the async/sync job pair we already mirror) | `include/vllm/entrypoints/openai/video_api.h:31`; `src/vllm/entrypoints/openai/video_api.cpp:98`; `src/vllm/entrypoints/openai/api_server.cpp:279` | `tests/vllm/entrypoints/openai/test_video_api.cpp:64`; `tests/vllm/entrypoints/openai/test_api_server.cpp:1751` | [minimax-h3.md §9](specs/minimax-h3.md) | `ACTIVE` | `CLAIM-SERVE-VIDEOS-OAI` | diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 4bd380985..cc9d8d732 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -43,9 +43,9 @@ Rollup by lifecycle state (must equal the detailed per-state row counts): | State | Rows | |---|---| -| INVENTORIED | 315 | +| INVENTORIED | 314 | | PARTIAL | 19 | -| ACTIVE | 9 | +| ACTIVE | 10 | | SPIKE | 6 | | BLOCKED | 5 | | DONE | 3 | @@ -53,7 +53,7 @@ Rollup by lifecycle state (must equal the detailed per-state row counts): | GATING | 1 | | **Total** | **361** | -Engaged architectures (the 47 non-`INVENTORIED` rows): +Engaged architectures (the 48 non-`INVENTORIED` rows): | Support | Architecture | Family / example | Status | Row | |---|---|---|---|---| @@ -67,6 +67,7 @@ Engaged architectures (the 47 non-`INVENTORIED` rows): | 🚧 | `ParakeetForRNNT` / `ParakeetForTDT` (transducer heads) | Parakeet RNN-T and TDT ASR: the decode half of the same family, likewise an audio COMPONENT and not a registry arch (vLLM has no transducer call site at all) | P6 LANDED 2026-08-07 on CPU: LSTM prediction network, joint network, TDT duration head and the greedy transducer loop, gated against dumped HuggingFace `ParakeetForRNNT` / `ParakeetForTDT` oracles from transformers `main` with the emitted sequence and per-step durations EXACT, plus an independent in-test LSTM reference. Real transcripts verified on `nvidia/parakeet-rnnt-0.6b` and `-tdt-0.6b-v3`, token ids EXACT vs HF `generate()` end to end. **Corrects the P4 record**, which called the transducer unmirrored off the locally installed transformers 5.3.0. No CUDA, no GPU suite, no aarch64, no speed number; `.nemo`-only checkpoints out of reach. **ONE-SURFACE ROW 1 (2026-08-07): `ParakeetForRNNT`/`ParakeetForTDT` registered** (BEYOND-PIN, transcription-only refuse-by-task) and served through the same `vllm_transcribe` / `/v1/audio/transcriptions` / thin-client surface as the CTC head | `MODEL-AUDIO-PARAKEET-TRANSDUCER` | | ✅ | `OPTForCausalLM` | OPT-125m | STRICT token-exact 6/6 vs vLLM 0.25.0; speed pending | `MODEL-TEXT-opt-optfor-causal-lm` | | ✅ | `DeepseekV2ForCausalLM` | DeepSeek-V2-Lite (MLA) | SACRED gate 8/8 token-exact vs vLLM 0.25.0; speed short (attributed, W9) | `MODEL-TEXT-deepseek-v2-deepseek-v2-for-causal-lm` | +| 🚧 | `LlamaModel` (embedding conversion) | The first POOLING arch (ARCH-ONE-SURFACE ROW 6): upstream `_EMBEDDING_MODELS` maps `LlamaModel` onto the Llama backbone via `as_embedding_model` (registry.py:230 + adapters.py:230) | LIVE 2026-08-08 on `row/EMBEDDINGS-ONE-SURFACE` (PR #137): `is_pooling_model=true`, text paths refuse by task; `vllm_embed` (ABI v15) + task-conditional `/v1/embeddings`; fold gate 4/4-231 on the committed synthetic fixture (engine path == direct registry path + f64 LAST+normalize ref). Residual: real embedding checkpoint + `LLM(task="embed")` oracle cosine; the other 7 memberships unregistered | `MODEL-EMBED-llama-llama-for-causal-lm` | | ✅ | `LlamaForCausalLM` | Llama-3.2-1B dense (+ Yi + `InternLM3ForCausalLM` aliases) | STRICT token-exact 16/16 vs vLLM 0.25.0; speed pending. Llama-alias checkpoints gated 2026-07-26: Yi (`01-ai/Yi-Coder-1.5B-Chat`, arch=LlamaForCausalLM, zero delta) 16/16; InternLM3 (`internlm3-8b-instruct`, one alias line, plain-Llama+dynamic rope) 16/16 — CLOSES the recent-dense TEXT tier | `MODEL-TEXT-llama-llama-for-causal-lm` | | ✅ | `MistralForCausalLM` | Mistral-7B-v0.3 dense | full paged-engine SACRED gate 16/16 vs vLLM 0.25.0; speed pending | `MODEL-TEXT-mistral-mistral-for-causal-lm` | | ✅ | `Qwen3_5MTP` | Qwen3.5 MTP draft (spec-decode) | `DONE` 2026-07-26: k=1 MTP spec-decode e2e on the 27B GDN hybrid — three-way token-exact at c1 (our-ON == vLLM `--speculative-config mtp` == our-OFF, acceptance 16/16), c1 above vLLM every-axis + c2-c8 on-par-or-above, mixed-batch concurrency bit-exact, server/CLI/C-ABI `--speculative-config`; spec-OFF byte-identical | `MODEL-SPEC-qwen3-5-mtp-qwen3-5-mtp` | @@ -292,7 +293,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-EMBED-bert-with-rope-gte-new-model` | `GteNewModel` | `registry.py:221`; `vllm/model_executor/models/bert_with_rope.py::GteNewModel` | embedding / text | encoder attention; pooler; FusedMoE/grouped GEMM | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-EMBED-jina-jina-embeddings-v5-model` | `JinaEmbeddingsV5Model` | `registry.py:222`; `vllm/model_executor/models/jina.py::JinaEmbeddingsV5Model` | embedding / text | encoder attention; pooler | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-EMBED-llama-llama-bidirectional-model` | `LlamaBidirectionalModel` | `registry.py:223`; `vllm/model_executor/models/llama.py::LlamaBidirectionalModel` | embedding / text | encoder attention; pooler; sliding-window attention | ☐ required | `INVENTORIED` | none | unassigned | -| `MODEL-EMBED-llama-llama-for-causal-lm` | `LlamaModel`, `CwmForCausalLM`, `InternLM3ForCausalLM`, `IQuestCoderForCausalLM`, `LlamaForCausalLM`, `LLaMAForCausalLM`, `TeleChat3ForCausalLM`, `MistralModel` | `registry.py:224-231`; `vllm/model_executor/models/llama.py::LlamaForCausalLM` | embedding / text | encoder attention; pooler; sliding-window attention | ☐ required | `INVENTORIED` | none | unassigned | +| `MODEL-EMBED-llama-llama-for-causal-lm` | `LlamaModel`, `CwmForCausalLM`, `InternLM3ForCausalLM`, `IQuestCoderForCausalLM`, `LlamaForCausalLM`, `LLaMAForCausalLM`, `TeleChat3ForCausalLM`, `MistralModel` | `registry.py:224-231`; `vllm/model_executor/models/llama.py::LlamaForCausalLM` | embedding / text | encoder attention; pooler; sliding-window attention | [embeddings-one-surface](specs/embeddings-one-surface.md) | `ACTIVE` | ARCH-ONE-SURFACE ROW 6 (2026-08-08, in flight on `row/EMBEDDINGS-ONE-SURFACE` PR #137): the `LlamaModel` membership is REGISTERED + LIVE (`as_embedding_model` mirror, adapters.py:230 — `is_pooling_model=true`, bare-prefix loader, pooling forward `Qwen3DenseModel::ForwardHidden`, engine-step `pool_tokens`, `vllm_embed` ABI v15 + `/v1/embeddings`); registration `src/vllm/model_executor/models/llama_embedding_registry.cpp:132`; fold gate `tests/vllm/models/test_llama_embedding_fold.cpp:206` 4/4-231 on the committed synthetic fixture. RESIDUALS: the other 7 memberships (incl. `MistralModel`) unregistered; REAL-checkpoint (e5-mistral class) + `LLM(task="embed")` oracle cosine gate not run (synthetic-fixture arm only; no cosine-vs-oracle number fabricated) | `CLAIM-EMBEDDINGS-ONE-SURFACE` | | `MODEL-EMBED-modernbert-modern-bert-model` | `ModernBertModel` | `registry.py:232`; `vllm/model_executor/models/modernbert.py::ModernBertModel` | embedding / text | encoder attention; pooler; sliding-window attention | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-EMBED-bert-with-rope-nomic-bert-model` | `NomicBertModel` | `registry.py:233`; `vllm/model_executor/models/bert_with_rope.py::NomicBertModel` | embedding / text | encoder attention; pooler; FusedMoE/grouped GEMM | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-EMBED-phi3-phi3-for-causal-lm` | `Phi3ForCausalLM` | `registry.py:234`; `vllm/model_executor/models/phi3.py::Phi3ForCausalLM` | embedding / text | encoder attention; pooler | ☐ required | `INVENTORIED` | none | unassigned | diff --git a/.agents/specs/embeddings-one-surface.md b/.agents/specs/embeddings-one-surface.md new file mode 100644 index 000000000..03ad5705c --- /dev/null +++ b/.agents/specs/embeddings-one-surface.md @@ -0,0 +1,180 @@ +# Embeddings/pooling through the ONE surface (`ARCH-ONE-SURFACE` fold ROW 6) + +Row: `ARCH-ONE-SURFACE` leaf, branch `row/EMBEDDINGS-ONE-SURFACE` (task #285). +The audit found the engine-side pooler EXISTS (`ENG-POOLER-SEQ` W1/W2 ops + +`ENG-POOLING-RUNNER` W3 `PoolingRunner`) but is never invoked live: every +registered arch has `is_pooling_model=false`, `/v1/embeddings` is an explicit +residual, and the ABI has no embed entry. This row folds it live: +registry arch -> engine-step invocation -> `vllm_embed` (ABI v15) -> +live `/v1/embeddings`. + +## Scope + +Rows: `ENG-POOLING-RUNNER` (live engine-step invocation), +`SERVE-POOLING-ENDPOINTS` (`/v1/embeddings`), +`MODEL-EMBED-llama-llama-for-causal-lm` (the `LlamaModel` membership), +cross-ref `ENG-POOLER-SEQ` (landed ops, untouched). + +- **In.** (W1) Register the smallest upstream-mirrored embedding arch the + existing code supports: **`LlamaModel`** — upstream `_EMBEDDING_MODELS` + maps `"LlamaModel": ("llama", "LlamaForCausalLM")` + (`vllm/model_executor/models/registry.py:230`) and converts via + `as_embedding_model` (`adapters.py:230`): the CAUSAL backbone forward with + the lm_head removed + `DispatchPooler.for_embedding` (LAST pooling default, + `interfaces_base.py:160`; normalize head). NO new model is built — the + registered forward IS the shared `Qwen3DenseModel` machinery Llama already + routes (`llama.h:39-40`), minus lm_head, plus the landed `DispatchPooler`. + (W1) Invoke the landed `PoolingRunner` in the ENGINE STEP for pooling-task + requests: the runner builds a `PoolingRunner` iff the loaded model is a + pooling model (mirror `gpu/model_runner.py:368-369`) and `sample_tokens` + routes to a pooling branch that returns POOLED DATA instead of sampled + tokens (mirror `gpu/model_runner.py:1586-1607` + `pool/pooling_runner.py:29-42`); + the scheduler finishes a pooling request as soon as pooled output exists + (mirror `v1/core/sched/scheduler.py:1718-1721`), the output carries it to the + frontend (`scheduler.py:1837`), and async scheduling resolves OFF for pooling + models (mirror `vllm/config/vllm.py:1068-1073`, our + `SchedulerConfig::ResolveAsyncScheduling` is_pooling_model arm — already + landed, now WIRED). Text-generation on the pooling arch and embed on a text + arch refuse cleanly both directions (the #121 refuse-by-task precedent). + (W2) `vllm_embed` on `include/vllm.h`: engine handle + text(s) in, float + vector(s) out, explicit free, `vllm_last_error`; `VLLM_ABI_VERSION` 14->15, + floor pin advanced. (W3) live `/v1/embeddings` (OpenAI shape), registered + task-conditionally; socket-level 404 pins BOTH directions. (W4) allowlist + row removed, FEATURES row -> reachable, records. +- **Out (named residuals).** A REAL embedding checkpoint (e5-mistral / + Llama-embed class) through the fold on real hardware — not fetchable + CPU-side in this session; the committed synthetic-fixture arm is the gate + (the #121 precedent). `MistralModel`/`Qwen2Model`/`Gemma2Model` etc. + aliases (upstream registry.py:215-260) — additive follow-ups. Matryoshka + `dimensions` + `encoding_format` on the endpoint/ABI; `/pooling`, `/score`, + `/rerank`, `/classify`; tokwise pooling (W5); classify heads live-wiring. + `vllm_embed` batches sequentially through the synchronous engine (no + AsyncLLM fan-out). + +## Upstream chain (pinned `${VLLM_SOURCE}` @ 555967922, 0.26.0.dev0) + +- Task selection: `LLM.embed` -> `pooling_task="embed"` + (`vllm/entrypoints/pooling/offline.py:47-119`); `--runner pooling` + resolution `vllm/config/model.py:1008-1030`; `convert="embed"` default for + pooling runners `model.py:1058-1060`; arch->pooling class + `registry.py:215-260` (`"LlamaModel": ("llama", "LlamaForCausalLM")` :230); + `as_embedding_model` wrap `adapters.py:230-261` (pooler = + `DispatchPooler.for_embedding`, :257), lm_head replaced by a missing-layer + stage (`adapters.py:135-151`), checkpoint loadable from BOTH `*ForCausalLM` + and bare `*Model` prefixes (`adapters.py:178-181` candidate_prefixes + `["", "model."]`), LAST default (`interfaces_base.py:160`). +- Engine step: `PoolingRunner` built iff pooling model + (`v1/worker/gpu/model_runner.py:368-369`); pool-instead-of-sample + (`model_runner.py:1586-1607`); `pool()` = gather at `logits_indices` + + normalize, `is_valid = seq_lens == prompt_len` + (`pool/pooling_runner.py:29-42`). +- Scheduler: pooling stops as soon as there is output + (`v1/core/sched/scheduler.py:1718-1721`), `pooling_output` on the + EngineCoreOutput (:1837), emit condition includes pooler output (:1792); + `check_stop` asserts non-pooling (`sched/utils.py:95`). +- Config: async scheduling disabled for pooling (`config/vllm.py:1068-1073`); + decoder+LAST pooling SUPPORTS chunked prefill and prefix caching + (`config/model.py:1883-1902,1929-1948`) so those defaults stay untouched. +- Endpoint: `POST /v1/embeddings` (`entrypoints/pooling/embed/api_router.py:28-43`), + handler absent => "does not support" (:22-25); response shape + `entrypoints/pooling/embed/protocol.py` (OpenAI list/data/usage). + +## Our baseline + +- Landed, test-only: `layers/pooler/*` (methods/activations/heads/poolers/ + dispatch_pooler), `v1/worker/gpu/pool/pooling_runner.{h,cpp}`, + `tests/vllm/v1/worker/gpu/pool/test_pooling_runner.cpp` (STRUCTURAL + double-precision LAST+normalize cosine gate — NOT recorded-oracle fixtures; + the qwen36_embed goldens are token-embedding LOOKUP goldens, unrelated). +- `ModelInfo.is_pooling_model` exists (all false); + `SchedulerConfig::ResolveAsyncScheduling(is_pooling_model)` arm landed but + passed `false` at the only call site (`model_loader.cpp:552`). +- Engine plumb points already marked: `scheduler.cpp` "DEFERRED: pooling + stop.", `output_processor.cpp:216,433` pooling-deferred markers, + `input_batch.h:66`, `llm_engine.h` deferred list. +- Precedents: #121 (Parakeet: refuse-by-task registry + capi guard + + task-conditional route + socket 404 pins + committed tiny fixtures), #123 + route-table 404s, #136 append-only ABI field growth. + +## Port map + +| Upstream | Local | Notes | +|---|---|---| +| `adapters.py:230-261` + `registry.py:230` | `src/vllm/model_executor/models/llama_embedding_registry.cpp` (NEW TU) | `REGISTER_VLLM_MODEL(.., "LlamaModel", ..)`, `is_pooling_model=true`, `is_text_generation_model=false`; loader accepts both name prefixes, never loads lm_head; LoadedModel owns `DispatchPooler::ForEmbedding(cfg, kLast)` | +| model returns hidden states (no lm_head) | `Qwen3DenseModel::ForwardHidden` (additive in `qwen3.h`/`qwen3.cpp`) | shared ForwardLayers tail gains a return-hidden arm; text callers byte-identical | +| `VllmModelForPooling.pooler` | `LoadedModel::pooler()` virtual (default nullptr) | additive on the type-erased base | +| `model_runner.py:368-369,1586-1607` | `GPUModelRunner::pooling_runner_` + pool branch in `sample_tokens` (`runner.cpp`) | ADDITIVE + model-task-gated; text path untouched | +| `outputs.py ModelRunnerOutput.pooler_output` | `ModelRunnerOutput::pooler_output` (`v1/engine/types.h`) | `vector>>`, empty on generation steps | +| `request.py pooling_params` | `Request::pooling_params` + `EngineCoreRequest::pooling_params` | `optional`, nullopt = generation, byte-identical | +| `scheduler.py:1718-1721,1792,1837` | the marked elif in `Scheduler::update_from_output` | pooling stop + `EngineCoreOutput::pooling_output` | +| `output_processor.py` pooling branch | `RequestState::make_request_output` + `process_outputs` pooling arm | `RequestOutput::pooling_output` (optional vector; deviation: no separate PoolingRequestOutput class, recorded) | +| `offline.py LLM.embed` | `LLMEngine::add_pooling_request` + `embed()` driver | tokens overload + step loop | +| `config/vllm.py:1068-1073` | pass `info.is_pooling_model` at `model_loader.cpp:552` | wires the landed arm | +| no upstream C ABI (llama.h idiom) | `vllm_embed` / `vllm_embedding_result(_free)`, ABI v15 | `src/capi/vllm_c.cpp`; refuse both directions | +| `pooling/embed/api_router.py:28` + protocol | `handle_embeddings` in `api_server.{h,cpp}` + `set_embedding`; server main task dispatch | task-conditional route registration, 404 both ways | + +## Tests to port + +- `tests/models/language/pooling/test_embedding.py` (real-oracle cosine) -> + the registry-anchored fold arm below; the REAL-checkpoint leg stays the named + residual (no cosine-vs-oracle number fabricated). +- `tests/entrypoints/pooling/embed/*` (endpoint shape) -> + `tests/vllm/entrypoints/openai/test_api_server.cpp` embeddings section. + +## Gates + +- Fold gate `tests/vllm/models/test_llama_embedding_fold.cpp`: committed tiny + synthetic `LlamaModel` checkpoint fixture (deterministic generator script, + #121 precedent) driven through `LoadedEngine::FromModelDir` -> + `add_pooling_request` -> `step` (the REGISTRY/RUNNER path), asserting + (a) IDENTICAL vectors vs the direct `ModelRegistry::Forward`+`PoolingRunner` + path, (b) the pooling lane's double-precision LAST+normalize reference + (cosine ~= 1, unit L2) — the lane's cosine gate now RUNS THROUGH the + registry/runner; (c) refuse-both-directions. +- `test_pooling_runner.cpp` continues to pass unchanged (the op-level gate). +- `test_capi`: v15 floor pin (>= 15), `vllm_embed` bad-path contract + the + REAL fixture-checkpoint smoke through `vllm_engine_load`+`vllm_embed`, + refuse-both-directions pins. +- `test_openai_api_server`: `/v1/embeddings` handler shape + socket-level 404 + pins BOTH directions (route absent on text servers; generate routes absent + on embedding servers). +- `vllm_capi_c_check` (c_header_compile.c references the new symbols). +- Full CPU `-Werror` build; `scripts/agent-preflight.sh` EXIT=0; + mutation-verify every pin RED-first. + +## Dependencies + +- Landed pooling ops (`ENG-POOLER-SEQ` W1/W2) + `PoolingRunner` (W3); the + shared dense backbone (`LlamaModel == Qwen3DenseModel`); the registry, capi + and server seams; the #121/#123/#136 fold precedents. + +## Risks / decisions + +- Depends only on landed code (pooler ops, runner, registry, capi, server). +- Risk: the pooling branch must not perturb the SACRED text path — every hook + is task-gated on `is_pooling_model` (registration-time constant) or + `pooling_params.has_value()`; default nullopt/false everywhere. +- Risk: chunked prefill of pooling prompts — handled by the same + discard/is_valid predicate the text path uses (`seq_len < num_tokens`), + mirroring `pooling_runner.py:40-41`; pooled output only when fully + prefilled. +- Recorded deviation: `vllm_embed`/`/v1/embeddings` drive the SYNCHRONOUS + LLMEngine (embed is blocking request/response); AsyncLLM stays + generation-only. + +## Work breakdown + +1. **W1 — registry + runner (landed on this row):** `LlamaModel` arch TU + + bare-prefix loader + `ForwardHidden` pooling forward; `PoolingRunner` + invoked task-gated in the engine step; scheduler pooling stop; refusals + both directions. +2. **W2 — ABI (landed on this row):** `vllm_embed` + `vllm_embedding_result_free`, + VLLM_ABI_VERSION 15, floor pin advanced, strict-C references, dlopen symbols. +3. **W3 — server (landed on this row):** task-conditional `/v1/embeddings` + + both-direction socket 404 pins; server main pooling dispatch. +4. **W4 — guard/records (landed on this row):** allowlist row removed, + FEATURES/STATUS/BENCHMARKS/matrices/specs updated. +5. **W-next (residuals, unclaimed):** real embedding checkpoint + + `LLM(task="embed")` oracle cosine; `MistralModel`/other membership aliases; + `/pooling`+score/rerank/classify; matryoshka `dimensions`/base64/token-array + inputs; tokwise pooling (W5 of the pooling lane). diff --git a/.agents/specs/one-surface-abi.md b/.agents/specs/one-surface-abi.md index 5997426bd..13fdd4a21 100644 --- a/.agents/specs/one-surface-abi.md +++ b/.agents/specs/one-surface-abi.md @@ -1,6 +1,6 @@ # ONE SURFACE — every capability ships through the C ABI -Row: `ARCH-ONE-SURFACE`. Status: **AUDIT DONE; remediation IN PROGRESS — ROW 1 (Parakeet ASR / audio transcription) LANDED 2026-08-07: ABI v11 `vllm_transcribe`, live `/v1/audio/transcriptions`, registry refuse-by-task, example folded, ratchet 12 -> 11. ROW 2 (MiniMax-H3 video+audio generation) LANDED 2026-08-08 (`row/H3-VIDEO-ABI`, task #283): ABI v12 `vllm_video_engine_load`/`vllm_video_generate`/`vllm_video_result_free` + `vllm_video_mux_argv` over the `MiniMaxH3VideoEngine` library seam, `/v1/videos` routed through the SAME seam, both H3 examples rewritten as `vllm.h` clients byte-identical to the pre-fold binary, ratchet 11 -> 9. GB10 real-video re-verification is a NAMED RESIDUAL (box on the Kimi campaign). ROW 8 (explicit device selection) LANDED 2026-08-08 (`row/DEVICE-KNOB`, task #284): ABI v14 `vllm_model_params.device` (0=auto/1=cpu/2=cuda, the vLLM `DeviceConfig.device` names) -> `EngineParams::device` -> `SelectQueue`; explicit cpu forces the CPU queue without probing, an explicitly named ABSENT device fails LOUD before any model I/O (the vllm/config/device.py:61-66 never-substitute mirror), `--device` on server + cli as pure field consumers; PR #139 follow-up removes PR #136's seven shared CUDA literals by resolving the stable public name through the platform registry and propagating its `DeviceType` (DSR 39 -> 32, `kcuda=0`, baseline/allowlist unchanged). CUDA-build A/B (auto->CUDA vs explicit-cpu->CPU on a GPU box) is a NAMED RESIDUAL — the CPU tier pins that half through the pure `ResolveExplicitDeviceType` matrix instead.** +Row: `ARCH-ONE-SURFACE`. Status: **AUDIT DONE; remediation IN PROGRESS — ROW 1 (Parakeet ASR / audio transcription) LANDED 2026-08-07: ABI v11 `vllm_transcribe`, live `/v1/audio/transcriptions`, registry refuse-by-task, example folded, ratchet 12 -> 11. ROW 2 (MiniMax-H3 video+audio generation) LANDED 2026-08-08 (`row/H3-VIDEO-ABI`, task #283): ABI v12 `vllm_video_engine_load`/`vllm_video_generate`/`vllm_video_result_free` + `vllm_video_mux_argv` over the `MiniMaxH3VideoEngine` library seam, `/v1/videos` routed through the SAME seam, both H3 examples rewritten as `vllm.h` clients byte-identical to the pre-fold binary, ratchet 11 -> 9. GB10 real-video re-verification is a NAMED RESIDUAL (box on the Kimi campaign). ROW 8 (explicit device selection) LANDED 2026-08-08 (`row/DEVICE-KNOB`, task #284): ABI v14 `vllm_model_params.device` (0=auto/1=cpu/2=cuda, the vLLM `DeviceConfig.device` names) -> `EngineParams::device` -> `SelectQueue`; explicit cpu forces the CPU queue without probing, an explicitly named ABSENT device fails LOUD before any model I/O (the vllm/config/device.py:61-66 never-substitute mirror), `--device` on server + cli as pure field consumers; PR #139 follow-up removes PR #136's seven shared CUDA literals by resolving the stable public name through the platform registry and propagating its `DeviceType` (DSR 39 -> 32, `kcuda=0`, baseline/allowlist unchanged). CUDA-build A/B (auto->CUDA vs explicit-cpu->CPU on a GPU box) is a NAMED RESIDUAL — the CPU tier pins that half through the pure `ResolveExplicitDeviceType` matrix instead. ROW 6 (embeddings/pooling) LANDED 2026-08-08 (`row/EMBEDDINGS-ONE-SURFACE`, task #285, PR #137): ABI v15 `vllm_embed`/`vllm_embedding_result_free` over the SAME registry-forward + PoolingRunner engine step the live task-conditional `/v1/embeddings` drives — the first registered POOLING arch (`LlamaModel`, the upstream `_EMBEDDING_MODELS` registry.py:230 + `as_embedding_model` adapters.py:230 mirror: bare-prefix loader, NO lm_head, LAST+normalize `DispatchPooler`), `is_pooling_model=true` with refuse-by-task BOTH directions (registry + capi + route-table 404 pins both ways), the landed `PoolingRunner` invoked task-gated where the sampler would run (gpu/model_runner.py:368-369 + 1586-1607 mirror; scheduler stop scheduler.py:1718-1721; async OFF for pooling config/vllm.py:1068-1073), and the pooling lane's cosine gate RE-ANCHORED THROUGH the registry/runner path (`test_llama_embedding_fold` 4/4-231: full-engine path == direct registry path IDENTICAL vectors + f64 LAST+normalize reference + chunked-prefill is_valid arm, on the committed tiny synthetic fixture). abi-capability allowlist 2 -> 1 (mm-input is the last row). NAMED RESIDUALS: real embedding checkpoint (e5-mistral class) + `LLM(task="embed")` oracle cosine; score/rerank/classify; matryoshka/base64/token-array inputs.** ## The defect diff --git a/.agents/specs/surface-coverage-2026-08-07.md b/.agents/specs/surface-coverage-2026-08-07.md index 0d610a588..2adefa17b 100644 --- a/.agents/specs/surface-coverage-2026-08-07.md +++ b/.agents/specs/surface-coverage-2026-08-07.md @@ -29,7 +29,7 @@ The four surfaces, and the public boundary the guard draws: | 3 | **DeepSeek-V4 fast decode** | yes, but forward is a W3 stub | no (stub) | no | `examples/deepseek_v4_gen` (keep-quant GGUF) | | 4 | **Audio transcription** | **CLOSED (ROW 1)**: Parakeet CTC/RNNT/TDT registered (transcription-only; Whisper/Voxtral still off-registry) | **live `/v1/audio/transcriptions`** (task-conditional; the run_batch line stays a residual) | **`vllm_transcribe` (ABI v11)** | library seam `ParakeetTranscriber`; example is a clean ABI client | | 5 | **Kimi-Linear incremental decode** | yes (recompute forward IS shared) | recompute only | no | `examples/kimi_linear_gen` (§18/§19 paged-incremental + resident loader) | -| 6 | **Embeddings / pooling** | NO (all `is_pooling_model=false`) | no (`/v1/embeddings` = residual) | no | engine-side pooler exists (`ENG-POOLER-SEQ`), never invoked live | +| 6 | **Embeddings / pooling** | **CLOSED (ROW 6, 2026-08-08)**: `LlamaModel` registered `is_pooling_model=true`; `PoolingRunner` invoked task-gated in the engine step | **live `/v1/embeddings`** (task-conditional; 404 both directions) | **`vllm_embed` (ABI v15)** | ONE engine path: `LoadedEngine -> LLMEngine::embed -> registry forward -> PoolingRunner` | | 7 | **Multimodal input over HTTP/ABI** | 5 archs `supports_multimodal` | image seam only; tower not run in engine step | no (text-only chat) | `chat_mm.cpp` seam; towers test-only | 21 of 30 registered text archs are fully on-framework (registry + runner + server + ABI): @@ -64,7 +64,7 @@ All three drivers run a PRIVATE host-argmax greedy loop, not the on-GPU sampler. | Parakeet/FastConformer ASR | **YES (ROW 1)**: ParakeetForCTC/RNNT/TDT, `parakeet_registry.cpp` (SupportsTranscription-only; text paths refuse by task) | `parakeet_transcription.cpp` seam composes encoder/transducer/audio-processor; the example's private `ReadWav16BitMono`/`LoadVocab`/`DecodeIds` are DELETED (`vllm::Tokenizer` now decodes Metaspace split=true) | **`/v1/audio/transcriptions`** (task-conditional) | **`vllm_transcribe` (ABI v11)** | `examples/parakeet_transcribe` = thin `vllm.h` client | | Voxtral audio->text | NO (`VoxtralForConditionalGeneration` unregistered) | `voxtral.cpp` (`vllm::multimodal`) | NO (`/v1/audio/transcriptions` = `run_batch.cpp:188` residual) | NO | tests-only reachability | | Whisper audio encoder | NO | `whisper_audio.cpp:174` | NO | NO | tests-only callers | -| Pooling / embeddings | NO (`is_pooling_model=false` in all 27) | `layers/pooler/*.cpp`, `pool/pooling_runner` (`ENG-POOLER-SEQ`) | NO (`/v1/embeddings` = residual) | NO | `PoolingRunner` test-only | +| Pooling / embeddings | **YES (ROW 6)**: `LlamaModel` (`llama_embedding_registry.cpp`, `is_pooling_model=true`; other `_EMBEDDING_MODELS` memberships still off) | `layers/pooler/*.cpp`, `pool/pooling_runner` (`ENG-POOLER-SEQ`) + the live engine-step invocation (`runner.cpp pool_tokens`) | **`/v1/embeddings`** (task-conditional) | **`vllm_embed` (ABI v15)** | fold gate `test_llama_embedding_fold` | | Multimodal INPUT | 5 archs `supports_multimodal` (Gemma4, KimiK3, Qwen3VL, Qwen3.5/-Moe) | towers `qwen3_vl_vision.cpp:374`, `gemma4_vision.cpp:170`; **Gemma-4 AUDIO USM tower is STANDALONE** (Gemma-4 text+image route via `ModelRegistry::Forward`, audio does NOT) | image seam only, raw-RGB, no stream, **tower not run in live step** (`chat_mm.cpp`; runner never consumes `mm_features`) | NO (text-only; `vllm_c.cpp` sets no mm seam) | — | | MTP / DFlash / ngram speculators | NO (sub-config / draft checkpoint; EAGLE unwired) | `spec_decode/{mtp,dflash}/speculator.cpp`, `ngram_proposer.cpp` | via `speculative_config` | via `speculative_config` (`vllm.h:172`) | — | @@ -119,10 +119,11 @@ before it. Bound to `include/vllm.h` by the marked `abi-capability-table` in `docs/FEATURES.md`. The ABI is text-generation-complete (completion, chat, async, structured output, tool + -reasoning parsers, speculative config, custom logits processor — 7 `reachable` rows). 4 -`embedder-unreachable` rows, each tracked in `scripts/abi-capability-allowlist.txt` -against `ARCH-ONE-SURFACE`: embeddings/pooling, audio transcription, video+audio -generation, multimodal input. +reasoning parsers, speculative config, custom logits processor — 7 `reachable` rows). +ROW 1 (audio transcription, v11), ROW 2 (video+audio generation, v12) and ROW 6 +(embeddings/pooling, v15) each flipped their row `reachable`; ONE +`embedder-unreachable` row remains, tracked in `scripts/abi-capability-allowlist.txt` +against `ARCH-ONE-SURFACE`: multimodal input. **Severity note — the ABI happy path is itself untested.** `vllm_engine_load` is never CI-gated on a REAL model load: `tests/capi/test_capi.cpp` covers only the bad-path error @@ -144,7 +145,7 @@ lanes are leaves of `ARCH-ONE-SURFACE` (do not open parallel rows). | 3 | DeepSeek-V4 fast decode | same as (2) for `DeepseekV4ForCausalLM`; real MLA paged KV (retire the W3 stub) | rewrite `deepseek_v4_gen`; delete `DeepseekV4ForwardGguf*` | M | MLA paged-KV topology | | 4 | Audio transcription | **DONE (ROW 1, 2026-08-07)**: `vllm_transcribe` (ABI v11) + live `/v1/audio/transcriptions`; ParakeetForCTC/RNNT/TDT registered (SupportsTranscription mirror, refuse-by-task) | **DONE**: `parakeet_transcribe` rewritten as a `vllm.h` client (byte-identical transcript goldens); route live, task-conditional | M | encoder→text seam (LANDED: `ParakeetTranscriber`) | | 5 | Kimi-Linear incremental | expose the incremental decode path through the runner/engine (the recompute forward already routes) | rewrite `kimi_linear_gen` | S–M | `KimiDecodeCache` on the runner | -| 6 | Embeddings/pooling | `vllm_embed`/pooling entry point + live `/v1/embeddings`; register a pooling arch (`is_pooling_model=true`); invoke `PoolingRunner` in the step | — | M | pooler live-wiring | +| 6 | Embeddings/pooling | **DONE (ROW 6, 2026-08-08)**: `vllm_embed`/`vllm_embedding_result_free` (ABI v15) + live task-conditional `/v1/embeddings`; `LlamaModel` registered `is_pooling_model=true` (as_embedding_model mirror); `PoolingRunner` invoked in the step (pool-instead-of-sample + scheduler pooling stop) | **DONE**: no example existed to rewrite (the capability was test-only); fold gate re-anchors the lane's cosine gate through the registry path | M | pooler live-wiring (LANDED) | | 7 | Multimodal input | multimodal-content entry point on `vllm_chat`; run the vision/audio tower in the engine step (`mm_features`→`ModelForwardInput.mm`) | wire `chat_mm` seam into the ABI | L | `MM-SERVE-E2E` engine mm-forward residual | | 8 | Device-selection knob | **DONE (ROW 8, 2026-08-08, `row/DEVICE-KNOB`; leakage follow-up PR #139)**: `vllm_model_params.device` (ABI v14: 0=auto/1=cpu/2=cuda, the vLLM `DeviceConfig.device` names, device.py:13) → `EngineParams::device` → `SelectQueue`; the stable public name resolves through `FindPlatformByName` and its registered `DeviceType` is propagated without a shared CUDA literal; explicit cpu never probes, explicit ABSENT cuda fails LOUD before model I/O; DSR 32 / `kcuda=0` | **DONE**: both thin clients consume the field; zero value byte-identical (auto probe) | S | mirror vLLM `--device`/`DeviceConfig` | | 9 | Voxtral + audio chat seam | register `VoxtralForConditionalGeneration` + fold `VoxtralGenerateGreedy` into the registry forward; audio-capable chat fn + an engine consumer for `AudioKwargs` mm_features | rewrite tests→clients | M | mirror upstream `voxtral.py:309`, `SupportsTranscription` | diff --git a/.agents/state.md b/.agents/state.md index a890b51e1..8bcf8f45b 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -43085,3 +43085,93 @@ F79-4 remain open on the landed tree; the review's merge-and-fix map is the binding description. Pi concurrency, BF16 GEMM/speed closure (W6) stay open as the lane's own next steps. + +## 2026-08-08 — ARCH-ONE-SURFACE ROW 6: embeddings/pooling LIVE on the ONE surface (`row/EMBEDDINGS-ONE-SURFACE`, task #285, PR #137) + + +**What landed.** The pooling lane's engine-side pooler (`ENG-POOLER-SEQ` ops + +`ENG-POOLING-RUNNER` `PoolingRunner`) is now INVOKED LIVE through one path: +`vllm_engine_load` on a pooling checkpoint -> `LLMEngine::embed` -> +scheduler -> `GPUModelRunner::pool_tokens` -> pooled vector, driven identically +by `vllm_embed` (ABI v15) and the live task-conditional `/v1/embeddings`. + +**W1 registry+runner.** NEW arch `LlamaModel` +(`llama_embedding_registry.cpp:132`, `is_pooling_model=true`, +`is_text_generation_model=false`) — the exact upstream mirror of +`_EMBEDDING_MODELS` `"LlamaModel": ("llama", "LlamaForCausalLM")` +(registry.py:230) + `as_embedding_model` (adapters.py:230): the SHARED dense +backbone run to the post-final-norm hidden with NO lm_head +(`Qwen3DenseModel::ForwardHidden`, an additive tail arm of the shared +ForwardLayers; text callers byte-identical), loader accepts BOTH name layouts +(adapters.py:178-181 candidate_prefixes) and never loads lm_head +(`LoadLlamaModelEmbeddingWeights`). The runner builds a `PoolingRunner` iff the +registration declares pooling (model_runner.py:368-369 mirror) and +`sample_tokens` routes to `pool_tokens()` (model_runner.py:1586-1607): pooled +rows instead of sampled tokens, validity == the existing discard predicate +(pooling_runner.py:40-41), scheduler pooling stop at the marked DEFERRED site +(scheduler.py:1718-1721), `pooling_output` out through +EngineCoreOutput/RequestOutput, async scheduling OFF for pooling models +(config/vllm.py:1068-1073 — the landed `ResolveAsyncScheduling` arm now WIRED). +Every hook is task-gated on `is_pooling_model`/`pooling_params` (default +nullopt/false = byte-identical text path; engine suites re-run green: +scheduler 423, llm_engine 204, engine_core 44, output_processor 77, +qwen3_forward 1557, async_llm 342, llama_forward 509 asserts). + +**W2 ABI.** `vllm_embed` + `vllm_embedding_result_free`, VLLM_ABI_VERSION +14 -> 15, floor pin advanced to `>= 15`; strict-C references in +c_header_compile.c; dlopen symbol resolution; refuse-by-task BOTH directions +(text entry points on a pooling engine name `vllm_embed`; `vllm_embed` on a +text engine names `vllm_complete`); FIXED en route: v13's +`vllm_complete_tokens` shipped without the v11 task guard (null-deref on a +transcription handle) — guard added. + +**W3 server.** `handle_embeddings` (OpenAI shape per embed/protocol.py:34, +173-185; `dimensions`/base64/token-arrays = named-residual 400s) registered +ONLY when an embedder is attached; server main dispatches pooling archs to a +serving-less embedding server. Socket-level 404 pins BOTH directions. + +**W4 guard/records.** `scripts/abi-capability-allowlist.txt` embeddings row +REMOVED (1 row left: mm-input); FEATURES abi-capability row -> reachable + +`LlamaModel` arch row; `check-supported-models.py` ARCH_TOKEN_RE widened to +bare `*Model`; the runner-routing checker gains the explicit POOLING +classification (a pooling registration is a hidden-state producer BY DESIGN, +never the silently-exempt NONE bucket — pinned in its mutation suite); +gate-commands runnable-baseline re-pinned (+2 rows); STATUS/BENCHMARKS keyed +rows; engine/model matrices + coordination claim +`CLAIM-EMBEDDINGS-ONE-SURFACE`; ROW 6 closed in one-surface-abi.md + +surface-coverage-2026-08-07.md. + +**The fold's correctness anchor.** The pooling lane's cosine gate was +STRUCTURAL (synthetic hidden buffer, registry BYPASSED; the brief's "recorded +oracle fixtures" premise was inaccurate — the qwen36_embed goldens are +token-embedding LOOKUP goldens, unrelated). ROW 6 re-anchors it THROUGH the +registry/runner path: `test_llama_embedding_fold` 4/4-231 on the COMMITTED +deterministic fixture (`scripts/mm/llama_embed_fixture_gen.py`, +`tests/vllm/models/fixtures/llama_embed_e2e`, 151 KB) — (a) direct +`ModelRegistry::Forward`+`PoolingRunner` == f64 LAST+normalize reference, +(b) FULL-ENGINE path == direct path, IDENTICAL vectors, (c) chunked-prefill +(max_num_batched_tokens=2) == unchunked (the is_valid arm), plus +`test_capi` 48/48-462 (v15 section incl. the REAL fixture-checkpoint load +through the public ABI), test_dlopen 30/30, server suite 50/50, +registry 24/24-820. + +**Mutation kills (each RED then reverted, 9).** M1 ABI macro left at 14 -> +floor pin RED; M2 embed-on-text refusal dropped -> capi both-directions RED +(3 asserts); M3 generation-on-pooling refusal dropped -> the refusal case +HANGS (SIGTERM, doctest FAILURE); M4 `if (embedder_)` -> `if (true)` -> +text-server 404 pin RED; M5 generate route unconditional -> embedding-server +404 pin RED; M6 engine-step pooling invocation deleted -> fold gate RED +(2 cases); M7 scheduler pooling stop deleted -> fold gate RED; M9 registry +`is_pooling_model=false` -> registry pin + fold RED (819/820 + 6 asserts); +M11 async-off wire dropped -> fold engine arm RED (hang/SIGTERM + +CHECK_FALSE(async) pin). + +**Residuals (honest).** (1) REAL embedding checkpoint (e5-mistral class) +through the fold + the `vllm.LLM(task="embed").encode` oracle cosine — the +committed synthetic-fixture arm is the gate (the #121 precedent); NO +cosine-vs-oracle number fabricated. (2) The other 7 `_EMBEDDING_MODELS` +memberships (incl. `MistralModel`) unregistered. (3) `/pooling`, `/score`, +`/rerank`, `/classify` (need a classify arch). (4) Matryoshka `dimensions`, +`encoding_format: base64`, token-array inputs = 400s naming the residual. +(5) `vllm_embed` batches sequentially through the synchronous engine +(recorded deviation; AsyncLLM stays generation-only). diff --git a/CMakeLists.txt b/CMakeLists.txt index 473e07085..1fd4721f3 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -599,6 +599,7 @@ add_library(vllm STATIC src/vllm/model_executor/models/opt_weights.cpp src/vllm/model_executor/models/opt.cpp src/vllm/model_executor/models/llama_registry.cpp + src/vllm/model_executor/models/llama_embedding_registry.cpp src/vllm/model_executor/models/llama_weights.cpp src/vllm/model_executor/models/mistral_registry.cpp src/vllm/model_executor/models/mistral_weights.cpp diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index bdbbc4ff2..2a399f608 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -348,6 +348,7 @@ built on it rather than keeping the flattering one. | Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | **NOT APPLICABLE: nothing measured, claimed or owed.** 22 NATIVE kernels (+6 GDN glue, CPU-oracle gated, no speed); 65 host-tier. opt-125m e2e token-exact on llvmpipe. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns | | ROCm (`BACKEND-GATE-ROCM-VLLM` / `-SGLANG`) | **NOT APPLICABLE: no number measured, claimed or owed.** The W0 skeleton registers 1 of 106 ops and its HIP sources have never been compiled by anyone; no AMD hardware here | A contributor's first `-DVLLM_CPP_HIP=ON` build ([#41](https://github.com/mudler/vllm.cpp/issues/41)). Only once a model runs does a same-box vLLM-ROCm oracle become the gate; the floor is vLLM, quant-matched | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | +| Embeddings on the ONE surface (ROW 6, `LlamaModel` + `vllm_embed` + `/v1/embeddings`) | **NO number measured, claimed or owed.** Correctness-gated only, CPU: the 2026-08-08 fold (engine path == direct registry path, f64 LAST+normalize reference on the committed fixture) is plumbing, no speed claim | A REAL embedding checkpoint (e5-mistral class) + a same-box `vllm.LLM(task="embed")` oracle; only then does an embed-throughput bar exist | | Parakeet/FastConformer ASR (P1-P4 + ONE-SURFACE fold ROW 1) | **NO number measured, claimed or owed.** Correctness-gated only, CPU f32; the 2026-08-07 surface fold (`vllm_transcribe`, `/v1/audio/transcriptions`) is transcript-byte-identical plumbing, no speed claim. | Floor is `parakeet.cpp`, same clip and box; needs a CUDA provider and a pretrained checkpoint | | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx | | Ampere consumer (`sm_86`, RTX 3090 class) | **No number owed; no such board here.** 2026-08-06 build-verify: 7/7 FA2 TUs 0-warn, real `sm_86` SASS. [Detail](../.agents/benchmark-record.md) | External RTX 3090 report. Floor is llama.cpp on that card (GGUF, not our Blackwell-only NVFP4 grid) | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 2c3810c54..1826b94d4 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -121,6 +121,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `LagunaForCausalLM` | poolside/Laguna-S-2.1-NVFP4, GGUF-Q4_K, Laguna-XS | byte-exact near-tie (distributional vs vLLM) | vLLM parity+ 1.03x, default on, via the `laguna-gen` CLI; the registered engine forward VT_CHECKs non-bf16 (`ARCH-ONE-SURFACE` fold) | | `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | **Folded onto the shared paged runner (ROW 7 §21, #122): engine==CLI 128/128 byte-identical; vs golden 122/128 (the intrinsic near-tie profile); FA2 paged MLA default-ON; SACRED post-fold green** | Served via `vllm_engine_load` + `vllm_complete_tokens` (ABI v13); server 19.0 tok/s wall vs vLLM ~21 (~0.90×), speed residual open | | `KimiK3ForConditionalGeneration` | Kimi-K3 (2.8T MoE) | scaffold: registry+config+enumeration gated, forward refuses | HW-infeasible (~1.56 TB); no run | +| `LlamaModel` | committed tiny synthetic embedding fixture (engine path == direct pooler path, identical vectors; f64 LAST+normalize reference); real checkpoint (e5-mistral class) is a NAMED residual | pooling/embed only, text paths refuse by task; `vllm_embed` + `/v1/embeddings` | n/a (CPU correctness-grade embeddings) | | `ParakeetForCTC`, `ParakeetForRNNT`, `ParakeetForTDT` | nvidia/parakeet-ctc-0.6b/-1.1b, -rnnt-0.6b, -tdt-0.6b-v3 (transcribed, ids exact vs HF `generate()`, P4/P6 2026-08-07; not retained) + committed synthetic fold fixture | ASR transcription-only (`SupportsTranscription` mirror; text paths refuse by task); fold gate byte-identical to the pre-refactor pipeline | n/a (CPU correctness-grade ASR via `vllm_transcribe` + `/v1/audio/transcriptions`) | | `CohereForCausalLM` | Command-R / Cohere (and Cohere2) | scaffold: W0 tiny-random oracle run-verified; real-checkpoint gate blocked | no run | @@ -149,11 +150,12 @@ Enumerated in `.agents/model-matrix.md`, not registered, no runnable GB10 gate: | `GlmMoeDsaForCausalLM` | GLM-5 (DSA) | ~1404 GiB bf16; dep-blocked (GLM-5.x is DeepSeek-V3.2 verbatim) | | `MiniMaxM2ForCausalLM` | MiniMax-M2 | ~230B, ~428 GiB bf16, ~4x over the unified pool | -25 of the 30 registered architectures carry a passing correctness gate today; -the rest are honestly marked scaffold or blocked above. vLLM registers 130+ text -architectures, so this is a curated, gated subset, not a breadth claim. Embedding -and reranking models are not yet registered: the engine-side pooler landed, no -model architecture is wired. +25 of the 30 registered text-generation architectures carry a passing +correctness gate today; the rest are honestly marked scaffold or blocked above. +vLLM registers 130+ text architectures, so this is a curated, gated subset, not +a breadth claim. The first EMBEDDING architecture is registered and live +(`LlamaModel`, task=embed, LAST pooling, the as_embedding_model mirror, gated +on the committed fixture); reranking/classify models are not yet registered. ## Multimodal @@ -229,12 +231,12 @@ Build with `-DVLLM_CPP_VULKAN=ON`; off by default. | Prometheus metrics | ✅ | ✅ | ✅ | ◐ | | Plugin / out-of-tree model registration | ✅ in-tree factory `DONE` + plugin seam | ✅ | ◐ | ☐ | | LoRA adapters | ☐ CPU brick only | ✅ | ✅ | ✅ | -| Embedding / pooling endpoints | ◐ engine only | ✅ | ✅ | ✅ | +| Embedding / pooling endpoints | ◐ `/v1/embeddings` live (task=embed; score/rerank/classify pending) | ✅ | ✅ | ✅ | | OpenAI video generation `/v1/videos` (Sora shape) | ✅ `model`/`size`/`seconds` aliases + `GET /{id}/content`; `input_reference` and the `metadata` video/audio references condition the render | ◐ (vllm-omni, its own request shape) | ☐ | ☐ | | Flat C ABI for embedding in other languages | ✅ versioned | ☐ | ☐ | ✅ | #### C-ABI capability coverage -- Which capabilities an embedder drives through the flat C ABI (`include/vllm.h`, the only installed header), gated by `scripts/check-surface-coverage.py`: a `reachable` row names an entry point that exists; an `embedder-unreachable` row is tracked in `scripts/abi-capability-allowlist.txt` against its fold row (`ARCH-ONE-SURFACE`). The ABI is text-generation-complete; the two `embedder-unreachable` rows are the open capability gaps. +- Which capabilities an embedder drives through the flat C ABI (`include/vllm.h`, the only installed header), gated by `scripts/check-surface-coverage.py`: a `reachable` row names an entry point that exists; an `embedder-unreachable` row is tracked in `scripts/abi-capability-allowlist.txt` against its fold row (`ARCH-ONE-SURFACE`). The ABI is text-generation-complete; the one `embedder-unreachable` row (multimodal input) is the open capability gap. | Capability | C-ABI surface | Embedder-reachable | |---|---|---| @@ -246,7 +248,7 @@ Build with `-DVLLM_CPP_VULKAN=ON`; off by default. | Tool + reasoning parser selection | `tool_parser`, `reasoning_parser` | reachable | | Speculative decoding config | `speculative_config` | reachable | | Custom logits processor | `vllm_logits_processor` | reachable | -| Embeddings / pooling | none | embedder-unreachable | +| Embeddings / pooling (task=embed) | `vllm_embed`, `vllm_embedding_result_free` (ABI v15; pooling checkpoints load via `vllm_engine_load`) | reachable | | Audio transcription (Parakeet ASR) | `vllm_transcribe`, `vllm_transcription_params_default`, `vllm_transcription_free` | reachable | | Video+audio generation (MiniMax-H3) | `vllm_video_engine_load`, `vllm_video_generate`, `vllm_video_result_free`, `vllm_video_mux_argv` | reachable | | Explicit device selection (auto/cpu/cuda) | `device` field on `vllm_model_params` (ABI v14; 0=auto keeps the probe, explicit absent device fails loud) | reachable | @@ -278,7 +280,7 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the | Multi-GPU execution | Hardware-blocked | TP proven equal to tp=1 on CPU; no 2-GPU box to run it | | LoRA end to end | CPU brick landed | Unwired standalone; not usable through the server | | Multimodal over HTTP | Architecturally blocked | Vision tower lives outside the registered engine forward | -| Embedding / reranking models | Engine side only | Pooler and runner path landed, no model architecture registered | +| Reranking / classify models | Engine side only | Embeddings are LIVE (`LlamaModel`, `vllm_embed`, `/v1/embeddings`); the classify/score heads are landed ops with no registered arch | | ROCm | W0 skeleton, unbuilt | Backend + platform + 1 op (RmsNorm); the HIP sources have never been compiled by anyone (no AMD board here). Open: [ROCM.md](ROCM.md), [#41](https://github.com/mudler/vllm.cpp/issues/41) | | XPU, TPU | Not started | CUDA, CPU, Metal and Vulkan are the built backends | | Custom logits processors on CUDA | Open, not root-caused | Segfaults in a CUDA build, 232/232 green on CPU | diff --git a/docs/STATUS.md b/docs/STATUS.md index 57b59f9a2..bac250fb2 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -113,7 +113,7 @@ token-for-token correctness against the pinned oracle. | Reasoning parsing (`SAMPLE-REASONING`, ACTIVE, partial coverage) | 9 parsers, streaming | think_auto (auto-detect default: content unless markers appear), deepseek_r1, deepseek_v3 (passthrough) / holo2 (thinking→R1), mistral ([THINK]), minimax_m2 (+append_think), step3, olmo3 - reasoning split engine-side BEFORE tool parsing, streamed as `reasoning` deltas in the chat chunks. Coverage: 9 of upstream's ~28 registered names (remaining text families + engine-backed adapters tracked as W2/W3 in specs/reasoning-parsers.md); each ported parser doctest-gated vs its tests/reasoning case | | Unified streaming parser engine | Core, assembly, serving-SSE dispatch landed, gated; all 10 engine-backed families ported (family parity closed); JSON-schema tool-arg type coercion landed | The vLLM 0.26 declarative `parser/engine/` (shared state machine plus all 10 configs: qwen3, seed_oss, kimi_k2, minimax_m2, glm47_moe, deepseek_v4/v32, nemotron_v3, gemma4, inkling) and assembly layer, gated field-for-field vs vLLM 0.26. An engine-backed `--tool-call-parser` name drives the live chat SSE chunks, off by default. When a request's tools declare typed parameters, the assembled tool-call arguments are coerced to the declared JSON types (int/number/bool/string/array/null) 1:1 with vLLM `_fix_arg_types`, in both streaming and one-shot; no schema means the arguments pass through as strings unchanged. Details: .agents/specs/parser-assembly-c8.md | | OpenAI server | Supported (subset); #129: SPIKE∅ | `/v1/completions`, `/v1/chat/completions`, streaming SSE, `/v1/models`, `/health`, `/version`, `/ping`, `/metrics`, `/tokenize` (raw-`prompt` and chat-`messages`), `/detokenize`, `/tokenizer_info`, `/server_info`, `/reset_prefix_cache`, `/abort_requests`; `/v1/videos` in OpenAI's Sora shape + `GET /v1/videos/{id}/content`, conditioned on an `input_reference` image or the `metadata` video/audio references. `/tokenizer_info` and `/abort_requests` are flag-gated; `/metrics` and `/reset_prefix_cache` have handlers but no live backing on the async path. Endpoint list and flags: docs/USAGE.md. Depth-2 async serving no longer corrupts the host heap under `ignore_eos` | -| Pooling task class (embeddings / classify / score / rerank) | Spiked; pooler op + heads composite + pooling runner path landed (CPU), not yet servable end-to-end (no concrete model / endpoints) | The non-generative task class. W0 spike over the whole vLLM pooling surface (`.agents/specs/pooling-task-class.md`, `CLAIM-POOLING`). W1 landed the pooler OP (CLS/LAST/MEAN + Identity/Normalize/MultiLabelClassify/Classify activations, double-precision-gated). **W2 landed the pooler HEADS composite** — `EmbeddingPoolerHead` (projector→matryoshka→normalize), `ClassifierPoolerHead` (classifier→`(logit-mean)/sigma`→activation), the `SequencePooler` + `PoolerForEmbed`/`PoolerForClassify` factories, the `DispatchPooler` task routing, and the `PoolerConfig`/`PoolingParams` structs (`test_pooler_heads` 27/27, 240 asserts, RED-first). **W3 landed the pooling RUNNER path** — `PoolingRunner` applies the model's `Pooler` to the last hidden state and returns pooled embeddings instead of sampled tokens, gated by a STRUCTURAL cosine-parity check vs a double-precision LAST+normalize reference (`test_pooling_runner` 5/5, 14 asserts, RED-first). NOT yet servable / honest residuals: a concrete pooling MODEL forward + the REAL-model oracle cosine gate (`vllm.LLM(task="embed").encode`) — no cosine-vs-oracle number is fabricated (W3-model); the `/v1/embeddings` + score/rerank/classify endpoints (W4); tokwise AllPool/StepPool (W5). See docs/BENCHMARKS.md | +| Pooling task class (embeddings / classify / score / rerank) | **EMBEDDINGS LIVE ON THE ONE SURFACE (ROW 6)**: `LlamaModel` registered, `PoolingRunner` in the engine step, `vllm_embed` (ABI v15) + live `/v1/embeddings`; classify/score/rerank engine-side only | The non-generative task class. W0 spike over the whole vLLM pooling surface (`.agents/specs/pooling-task-class.md`, `CLAIM-POOLING`). W1 landed the pooler OP (CLS/LAST/MEAN + Identity/Normalize/MultiLabelClassify/Classify activations, double-precision-gated). **W2 landed the pooler HEADS composite** (`EmbeddingPoolerHead`, `ClassifierPoolerHead`, `SequencePooler` + factories, `DispatchPooler` routing, `PoolerConfig`/`PoolingParams`; `test_pooler_heads` 27/27-240, RED-first). **W3 landed the pooling RUNNER path** (`PoolingRunner`: pooled embeddings instead of sampled tokens, structural cosine gate vs an f64 LAST+normalize reference, `test_pooling_runner` 5/5-14, RED-first). **ROW 6 (2026-08-08): embeddings LIVE** — fold gate `test_llama_embedding_fold` 4/4-231 (engine path == direct registry path, f64 LAST+normalize ref, chunked is_valid arm); residuals: REAL checkpoint + `LLM(task="embed")` oracle cosine (no number fabricated), score/rerank/classify endpoints, matryoshka/base64/token-array inputs, tokwise (W5). Detail: `.agents/specs/embeddings-one-surface.md` | | Plugin system (out-of-core registration) | Spiked; first CPU brick landed, not yet wired into any production path | The extensibility-first discovery layer. W0 spike over vLLM's plugin surface (general / platform / io_processor / endpoint groups, the `register_model` an out-of-tree plugin calls, the invocation seams) is committed (`.agents/specs/plugin-system.md`, `ENG-PLUGIN-SYSTEM` ACTIVE, `CLAIM-PLUGIN-SYSTEM`). W1 landed `vllm::plugins::LoadGeneralPlugins()` + the out-of-core general-plugin registration seam (`RegisterGeneralPlugin` / `REGISTER_VLLM_GENERAL_PLUGIN`) over the existing `REGISTER_VLLM_MODEL`-style registries (the in-tree factory `MODEL-FACTORY-registry` is record-repaired `DONE` 2026-08-05: 28 self-registering TUs, dgx debt paid by the 2026-07-23 seven-gate run): a 1:1 mirror of `load_general_plugins` (load-once idempotence, the `VLLM_PLUGINS` allowlist, per-plugin failure isolation). Proven by an out-of-core toy-model plugin that registers a toy architecture through the public `RegisterModel` seam — unit-gated RED-first (`test_plugin_system` 1 case / 29 assertions: the toy arch resolves ONLY after LoadGeneralPlugins runs it, and not under `VLLM_PLUGINS=""`). Python entry points have no C++20 analogue, so discovery is the project's static-init/`dlopen` registration idiom (recorded porting-inventory §9). NOT yet wired: real shared-object `dlopen` + the C-ABI `vllm_plugin_register` entry (W2), the engine/CLI `--load-plugins` wiring that calls LoadGeneralPlugins from the construction paths (W3), the platform/quant plugin kinds (W4), and the io_processor/stat_logger/endpoint groups (W5) are named residuals. See docs/BENCHMARKS.md | | Offline Batch API (JSONL file runner) | Spiked; first CPU brick landed, not yet exposed as a CLI | The offline OpenAI Batch API: read a JSONL of OpenAI-format requests, run each through the engine, write a JSONL of responses. W0 spike over vLLM's `run_batch.py` (schema, endpoint dispatch, run loop, file I/O) is committed (`.agents/specs/batch-api.md`, `SERVE-BATCH-API` ACTIVE, `CLAIM-BATCH-API`). W1 landed `RunBatch` (`RunLine`/`RunLines`/`Run`) + `RunBatchFile` — a pure orchestrator over the existing `OpenAIServingChat::create_chat_completion` (NO reimplemented generation), 1:1 with vLLM's endpoint_registry url→handler map: `/v1/chat/completions` dispatch, the `BatchResponseData`/`BatchRequestOutput` schema (`vllm-` ids, custom_id echoed), the `run_request` AllResponse/ErrorResponse/stream branches, and the unsupported-endpoint/url error rows. Unit-gated RED-first (`test_openai_run_batch` 7 cases / 80 assertions over the synthetic serving engine: ordered rows + custom_id echo + per-line BatchRequestOutput schema round-trip, a malformed line isolated into an error row so the batch continues, dispatch + 404 error rows; dropping the custom_id echo fails 9 assertions). Recorded deviation: a malformed line is isolated (batch continues) where upstream aborts the job. NOT yet exposed: the `vllm run-batch` CLI + `BatchFrontendArgs` (W2), embeddings/score/rerank dispatch (W3, rides pooling endpoints), audio transcription/translation + media fetch (W4), and http(s)/data-URL file I/O + metrics + overlapped `AsyncLLM` submission (W5) are named residuals. See docs/BENCHMARKS.md | | Tokenizers | Supported | Byte-level BPE (Qwen/Llama-3/OPT/GPT-2/DeepSeek/OLMo-2) and SentencePiece BPE (Mistral/Gemma), plus GGUF vocab; added-token `lstrip`/`rstrip` whitespace semantics (e.g. Phi-4-mini's special tokens); byte-exact vs the vLLM oracle | diff --git a/examples/server/main.cpp b/examples/server/main.cpp index a06d944d4..30b4e2424 100644 --- a/examples/server/main.cpp +++ b/examples/server/main.cpp @@ -435,6 +435,77 @@ int main(int argc, char** argv) { transcription_only = false; // unknown arch: the text path diagnoses } } + // ── POOLING TASK DISPATCH (ARCH-ONE-SURFACE ROW 6): a model dir whose + // architectures resolve to a POOLING registration (is_pooling_model, + // e.g. "LlamaModel" — vLLM _EMBEDDING_MODELS registry.py:230) serves + // /v1/embeddings through the ONE engine path (LoadedEngine -> + // LLMEngine::embed -> registry forward -> PoolingRunner) — the same + // path vllm_embed drives — and registers NO generate routes (vLLM's + // task-conditional registration, api_server.py:255-265). ────────────── + bool pooling_model = false; + if (!archs.empty()) { + try { + pooling_model = + vllm::ModelRegistry::Resolve(std::span(archs)) + .info.is_pooling_model; + } catch (const std::exception&) { + pooling_model = false; // unknown arch: the text path diagnoses + } + } + if (pooling_model) { + std::cerr << "server: pooling (embedding) model (" << archs[0] + << "); serving /v1/embeddings\n"; + vllm::entrypoints::EngineParams embed_params; + embed_params.block_size = args.block_size; + embed_params.num_blocks = args.num_blocks; + embed_params.max_model_len = args.max_model_len; + embed_params.max_num_seqs = args.max_num_seqs; + embed_params.max_num_batched_tokens = args.max_num_batched_tokens; + embed_params.enable_prefix_caching = args.enable_prefix_caching; + auto loaded_embed = std::shared_ptr( + vllm::entrypoints::LoadedEngine::FromModelDir(args.model_dir, + embed_params)); + namespace oai = vllm::entrypoints::openai; + oai::OpenAIServingModels embed_models(served_model_name); + oai::ApiServer embed_server(embed_models, vllm::Version()); + auto embed_mutex = std::make_shared(); + auto embed_counter = std::make_shared>(0); + embed_server.set_embedder( + [loaded_embed, embed_mutex, embed_counter]( + const std::vector& inputs) { + // Serialize batches: the pooling path drives the SYNCHRONOUS + // LLMEngine (async scheduling resolves OFF for pooling models). + std::lock_guard lock(*embed_mutex); + oai::ApiServer::EmbeddingBatch batch; + for (const std::string& text : inputs) { + std::vector ids = + loaded_embed->tokenizer().EncodeWithSpecialTokens(text); + if (ids.empty()) { + throw std::runtime_error( + "input tokenized to an empty prompt"); + } + batch.prompt_tokens += static_cast(ids.size()); + vllm::RequestOutput ro = loaded_embed->engine().embed( + std::move(ids), vllm::PoolingParams{}, + "embd-" + std::to_string(embed_counter->fetch_add(1))); + if (!ro.finished || !ro.pooling_output.has_value()) { + throw std::runtime_error( + "engine produced no pooled output"); + } + batch.embeddings.push_back(std::move(*ro.pooling_output)); + } + return batch; + }); + std::cerr << "server: listening on http://" << args.host << ":" + << args.port << "\n"; + if (!embed_server.listen(args.host, args.port)) { + std::cerr << "server: failed to bind " << args.host << ":" + << args.port << "\n"; + return 1; + } + return 0; + } + if (transcription_only) { std::cerr << "server: transcription-only model (" << archs[0] << "); serving /v1/audio/transcriptions\n"; diff --git a/include/vllm.h b/include/vllm.h index a276381cb..5dc219853 100644 --- a/include/vllm.h +++ b/include/vllm.h @@ -118,8 +118,19 @@ extern "C" { * CUDA platform; when it is absent the load FAILS with VLLM_ERR_MODEL_LOAD — * an explicitly named device is never silently substituted, device.py:61-66). * Appended at the END of vllm_model_params so a zero-initialized struct keeps - * the pre-v14 engine byte-identical. */ -#define VLLM_ABI_VERSION 14 + * the pre-v14 engine byte-identical. + * v15: vllm_embed / vllm_embedding_result(_free) — EMBEDDINGS through the ONE + * surface (ARCH-ONE-SURFACE fold ROW 6). An engine loaded from a POOLING + * (embedding) checkpoint — a directory whose config.json architectures resolve + * to a pooling registration, e.g. "LlamaModel" (the mirror of vLLM's + * _EMBEDDING_MODELS registry.py:230 + as_embedding_model adapters.py:230) — + * embeds text through the SAME registry forward + PoolingRunner engine step + * the server's /v1/embeddings drives. Text and pooling handles refuse each + * other's tasks LOUDLY: vllm_complete/vllm_chat on a pooling engine name + * vllm_embed, and vllm_embed on a text engine names vllm_complete — the + * SupportsTranscription-refusal precedent (v11) applied to the pooling task. + * Purely additive — no struct changed; zero values preserve behaviour. */ +#define VLLM_ABI_VERSION 15 /* ── Export macro ───────────────────────────────────────────────────────────── * Marks the symbols that make up the stable ABI. Default visibility now; Task 3 @@ -583,6 +594,44 @@ VLLM_API vllm_status vllm_transcribe(vllm_engine* engine, VLLM_API void vllm_transcription_free(vllm_transcription* out); +/* ── Embeddings (ABI v15) ───────────────────────────────────────────────────── + * The embeddings/pooling slice of the ONE-SURFACE fold: an engine loaded from + * a POOLING (embedding) checkpoint — config.json architectures resolving to a + * pooling registration such as "LlamaModel" — turns text into L2-normalized + * embedding vectors through the SAME registry forward + PoolingRunner engine + * step the bundled server's /v1/embeddings drives (task=embed, LAST-token + * pooling: the mirror of vLLM's as_embedding_model conversion). Loading such a + * checkpoint uses the ordinary vllm_engine_load; the handle then serves ONLY + * the embedding entry point (the text/chat entry points refuse, naming this + * one, and vice versa on a text handle). */ + +/* One embedding batch result. OWNERSHIP: `values` is library-allocated; free + * via vllm_embedding_result_free(out). Row-major: embedding i occupies + * values[i*dim .. (i+1)*dim). */ +typedef struct vllm_embedding_result { + float* values; /* n_embeddings * dim floats, row-major */ + int32_t n_embeddings; /* == the number of input texts */ + int32_t dim; /* the model's hidden size */ + int32_t prompt_tokens; /* total input tokens (the OpenAI usage mirror) */ +} vllm_embedding_result; + +/* Embed n_texts NUL-terminated UTF-8 strings on a pooling-capable engine + * handle, filling *out (one embedding per text, input order). BLOCKING; the + * texts are tokenized with the checkpoint's tokenizer and each prompt runs one + * engine prefill + pool step. Returns VLLM_OK on success; + * VLLM_ERR_INVALID_ARGUMENT for a text-generation handle (use vllm_complete / + * vllm_chat there), a NULL texts/out, an n_texts <= 0, or a NULL entry in + * texts; VLLM_ERR_RUNTIME when tokenization or the forward fails. On any + * non-OK status *out is zeroed and vllm_last_error() carries the detail. */ +VLLM_API vllm_status vllm_embed(vllm_engine* engine, + const char* const* texts, int32_t n_texts, + vllm_embedding_result* out); + +/* Free the owned members of an embedding result and zero the struct. The + * struct itself is caller storage. NULL is a no-op. */ +VLLM_API void vllm_embedding_result_free(vllm_embedding_result* out); + + /* ── Video+audio generation (ABI v12, MiniMax-H3) ──────────────────────────── * The video slice of the ONE-SURFACE fold: the SAME library pipeline the * bundled server's /v1/videos routes and the minimax-h3-gen example drive diff --git a/include/vllm/entrypoints/model_loader.h b/include/vllm/entrypoints/model_loader.h index a911bf09b..28bcb7e46 100644 --- a/include/vllm/entrypoints/model_loader.h +++ b/include/vllm/entrypoints/model_loader.h @@ -213,6 +213,14 @@ class LoadedEngine { std::optional named_platform_type); vllm::v1::LLMEngine& engine() { return engine_; } + // ARCH-ONE-SURFACE ROW 6: whether the loaded model registration declares the + // POOLING task class (is_pooling_model). The entrypoints dispatch BY TASK on + // this — text-generation refuses on a pooling engine (naming vllm_embed / + // /v1/embeddings) and embed refuses on a text engine — the mirror of vLLM + // validating runner_type against the model class (config/model.py:607-613). + bool is_pooling_model() const { + return model_->registration().info.is_pooling_model; + } // Lazily start W2's EngineCoreProc + output-handler threads. Once created, // online/server callers use this frontend rather than the synchronous // LLMEngine over the same scheduler/executor. @@ -258,8 +266,12 @@ class LoadedEngine { // CPU construction-matrix test can assert it directly over the // runner_supports_async x VT_ASYNC_SCHED matrix without a disk load. Applies // SchedulerConfig::ResolveAsyncScheduling then the VT_ASYNC_SCHED rollback env. + // `is_pooling_model` (ARCH-ONE-SURFACE ROW 6) resolves async OFF for pooling + // models (mirror of vllm/config/vllm.py:1068-1073); default false is the + // byte-identical text path. static bool ResolveAsyncEnabled(const vllm::SchedulerConfig& scheduler_config, - bool runner_supports_async); + bool runner_supports_async, + bool is_pooling_model = false); private: // Type-erased constructor used by FromModelDir and the concrete-weight diff --git a/include/vllm/entrypoints/openai/api_server.h b/include/vllm/entrypoints/openai/api_server.h index 745f18cc9..fd8d73d2e 100644 --- a/include/vllm/entrypoints/openai/api_server.h +++ b/include/vllm/entrypoints/openai/api_server.h @@ -138,6 +138,18 @@ class ApiServer { DispatchResult handle_audio_transcriptions( const std::string& file_bytes, const std::string& response_format) const; + // POST /v1/embeddings (ARCH-ONE-SURFACE ROW 6). Mirror of vLLM's + // pooling/embed/api_router.py:28 `create_embedding` over the + // EmbeddingCompletionRequest shape (embed/protocol.py:34: `model`, `input` + // as ONE string or an ARRAY of strings) and the EmbeddingResponse shape + // (embed/protocol.py:173-185: id "embd-...", object "list", data rows + // {index, object:"embedding", embedding:[...]}, usage prompt/total tokens). + // Token-array inputs, `dimensions` (matryoshka) and `encoding_format: + // "base64"` are NAMED RESIDUALS -> 400. Registered ONLY when an embedder is + // attached (the transcriber precedent), so a text server answers 404 at the + // route table. + DispatchResult handle_embeddings(const std::string& request_body) const; + DispatchResult handle_videos(const std::string& request_body); DispatchResult handle_videos_sync(const std::string& request_body); DispatchResult handle_video_status(const std::string& job_id) const; @@ -207,6 +219,22 @@ class ApiServer { transcriber_ = std::move(transcriber); } + // Attach the embedding seam backing POST /v1/embeddings (ARCH-ONE-SURFACE + // ROW 6). ADDITIVE and OPT-IN like the transcriber above: absent => route + // unregistered => 404, byte-identical to a server without pooling. The + // callback wraps the ONE engine path (LoadedEngine -> LLMEngine::embed -> + // the registry forward + PoolingRunner step) — the SAME path vllm_embed + // drives — so HTTP and FFI cannot drift. Returns one embedding per input + // (input order) + the total prompt token count for the usage block; throws + // to fail the request (-> 500). + struct EmbeddingBatch { + std::vector> embeddings; + int64_t prompt_tokens = 0; + }; + using EmbedFn = + std::function& inputs)>; + void set_embedder(EmbedFn embedder) { embedder_ = std::move(embedder); } + // Attach the tokenizer + max_model_len backing /tokenize and /detokenize // (non-owning; must outlive the server). void set_tokenizer(const vllm::tok::Tokenizer* tokenizer, @@ -269,6 +297,7 @@ class ApiServer { const v1::metrics::PrometheusStatLogger* metrics_ = nullptr; ::vllm::openai::VideoRunner video_runner_; TranscribeFn transcriber_; + EmbedFn embedder_; mutable ::vllm::openai::VideoJobStore video_jobs_; // Background workers for the ASYNC endpoint. Joined in ~ApiServer, which is // why they are joinable threads and not detached: a detached worker would diff --git a/include/vllm/model_executor/models/llama.h b/include/vllm/model_executor/models/llama.h index 20bdbad20..1562cab78 100644 --- a/include/vllm/model_executor/models/llama.h +++ b/include/vllm/model_executor/models/llama.h @@ -50,6 +50,14 @@ using LlamaModel = Qwen3DenseModel; LlamaWeights LoadLlamaForCausalLMWeights(const std::vector& shards, const HfConfig& config); +// `LlamaModel` EMBEDDING checkpoint loader (ARCH-ONE-SURFACE ROW 6): the same +// name map, accepting BOTH the "model."-prefixed and the bare `*Model` tensor +// layouts (vllm/model_executor/models/adapters.py:178-181 candidate_prefixes +// ["", "model."]) and never loading an lm_head (the as_embedding_model +// conversion has no output layer, adapters.py:135-151). +LlamaWeights LoadLlamaModelEmbeddingWeights( + const std::vector& shards, const HfConfig& config); + // Per-family config hook (mirrors ParseQwen3ForCausalLMConfig). LoadHfConfig // already materializes + validates every consumed Llama field, including the // llama3 rope_scaling dictionary; this explicit no-op hook is the family's diff --git a/include/vllm/model_executor/models/model_registry.h b/include/vllm/model_executor/models/model_registry.h index f3afa8f7f..c33beacb3 100644 --- a/include/vllm/model_executor/models/model_registry.h +++ b/include/vllm/model_executor/models/model_registry.h @@ -114,6 +114,15 @@ class LoadedModel { // instances of a W4A4-capable family may contain only BF16 weights. virtual bool uses_nvfp4_w4a4() const { return false; } + // ARCH-ONE-SURFACE ROW 6: the model-owned Pooler of a POOLING model — the + // mirror of upstream `VllmModelForPooling.pooler` (as_embedding_model wires + // `self.pooler = DispatchPooler.for_embedding(...)`, adapters.py:248-257). + // Non-null iff the registration's info.is_pooling_model; the GPU runner + // builds its PoolingRunner over exactly this pooler (the mirror of + // gpu/model_runner.py:368-369 `PoolingRunner(self.model)`). Default null: + // every text-generation model is byte-identical. + virtual const class Pooler* pooler() const { return nullptr; } + // ── SPEC-MTP I5d-pre: typed access to the MTP draft, without breaking the // type-erasure of this base. Only the concrete Qwen3.5 dense/MoE // LoadedModel (which owns the target Qwen3_5DenseWeights/Qwen3_5MoeWeights) diff --git a/include/vllm/model_executor/models/qwen3.h b/include/vllm/model_executor/models/qwen3.h index 1a360f506..0117b4a4c 100644 --- a/include/vllm/model_executor/models/qwen3.h +++ b/include/vllm/model_executor/models/qwen3.h @@ -162,6 +162,22 @@ class Qwen3DenseModel { const std::vector& attn_kv, const Qwen3DenseWeights& weights, const HfConfig& config, vt::Queue& queue, const std::vector& logits_indices = {}); + + // POOLING forward (ARCH-ONE-SURFACE ROW 6): the same embed + layer stack, + // stopping after the final RMSNorm (+ the logits_indices gather) with NO + // lm_head — the forward of an as_embedding_model conversion + // (vllm/model_executor/models/adapters.py:135-151 replaces the output layer + // with a missing-layer stage; the pooler consumes the post-final-norm + // hidden). Returns a HOST ForwardLogits carrier of [n_out, hidden_size] f32 + // rows (`vocab` == hidden_size on this path); the engine's pooling branch + // hands them to the landed PoolingRunner. Additive: no text caller routes + // here, and the lm_head tail above is byte-identical. + static ForwardLogits ForwardHidden( + const std::vector& token_ids, const std::vector& positions, + const v1::CommonAttentionMetadata& attn_meta, + const std::vector& attn_kv, const Qwen3DenseWeights& weights, + const HfConfig& config, vt::Queue& queue, + const std::vector& logits_indices = {}); }; // SHARED pure-dense decode CUDA-graph driver — the sibling of Qwen3MoeDecodeGraph diff --git a/include/vllm/outputs.h b/include/vllm/outputs.h index 2e19a1c28..17598e1bc 100644 --- a/include/vllm/outputs.h +++ b/include/vllm/outputs.h @@ -113,6 +113,13 @@ struct RequestOutput { std::optional prompt_logprobs; // Whether the whole request is finished. bool finished = false; + // pooling_output (ARCH-ONE-SURFACE ROW 6): the pooled vector of a finished + // POOLING-task request (task=embed: the L2-normalized last-token embedding). + // RECORDED DEVIATION: upstream wraps pooled results in a separate + // PoolingRequestOutput/PoolingOutput class pair (vllm/outputs.py); ours rides + // the ONE RequestOutput as an optional field so every existing consumer of + // the generation shape is byte-identical (nullopt there). + std::optional> pooling_output; // Convenience accessor mirroring the `finished` attribute (upstream exposes // the plain attribute; provided here for parity with the *Output helpers). diff --git a/include/vllm/v1/engine/llm_engine.h b/include/vllm/v1/engine/llm_engine.h index b9064f6da..cdc85f984 100644 --- a/include/vllm/v1/engine/llm_engine.h +++ b/include/vllm/v1/engine/llm_engine.h @@ -112,6 +112,21 @@ class LLMEngine { multimodal::MultiModalInputs mm_inputs, SamplingParams params, int priority = 0); + // add_pooling_request (ARCH-ONE-SURFACE ROW 6): the POOLING-task counterpart + // of the tokens add_request — upstream's `params: SamplingParams | + // PoolingParams` union (llm_engine.py add_request) collapses here to an + // explicit pooling entry point. Builds the request via process_inputs_tokens + // with a benign greedy SamplingParams (the sampler is never invoked on a + // pooling model's step) and attaches the PoolingParams; the scheduler + // finishes it as soon as the runner pooled its prompt + // (scheduler.py:1718-1721). Only meaningful on an engine whose model + // registration declares is_pooling_model — on a text model the request would + // never produce pooled data (the entrypoints refuse it before here). + std::string add_pooling_request(const std::string& request_id, + std::vector prompt_token_ids, + PoolingParams pooling_params, + int priority = 0); + // step (llm_engine.py:296): get the EngineCore outputs -> process_outputs -> // abort any reqs the detokenizer stopped -> return the RequestOutputs. std::vector step(); @@ -159,6 +174,15 @@ class LLMEngine { SamplingParams params, const std::string& request_id = "0", int priority = 0); + // embed (ARCH-ONE-SURFACE ROW 6): the single-request pooling driver — the + // mirror of LLM.embed's add-then-run loop (entrypoints/pooling/offline.py: + // 65-119, pooling_task="embed"). Adds the pooling request, then loops step() + // until it finishes; the returned RequestOutput carries the pooled vector in + // pooling_output. + RequestOutput embed(std::vector prompt_token_ids, + PoolingParams pooling_params = {}, + const std::string& request_id = "0", int priority = 0); + // The rolling prefix-cache hit rate (queries/hits in TOKENS over the most // recent 1000 requests). See EngineCore::prefix_cache_metrics. const CachingMetrics& prefix_cache_metrics() const { diff --git a/include/vllm/v1/engine/output_processor.h b/include/vllm/v1/engine/output_processor.h index 4c0c8a7ac..53f29c40a 100644 --- a/include/vllm/v1/engine/output_processor.h +++ b/include/vllm/v1/engine/output_processor.h @@ -129,10 +129,15 @@ class RequestState { // CompletionOutput/RequestOutput honoring output_kind. Returns nullopt when // FINAL_ONLY-and-not-finished or a stream_interval hold-back suppresses this // step's output. kv_transfer_params deferred (see header). + // `pooling_output` (ARCH-ONE-SURFACE ROW 6; upstream make_request_output's + // pooling_output parameter, output_processor.py:272): the finished POOLING + // request's pooled vector, attached to RequestOutput::pooling_output. + // nullopt on every generation output -> byte-identical text path. std::optional make_request_output( const std::vector& new_token_ids, std::optional finish_reason, - std::optional stop_reason); + std::optional stop_reason, + std::optional> pooling_output = std::nullopt); std::string request_id; std::string external_req_id; // == request_id at T0 (see header). diff --git a/include/vllm/v1/engine/types.h b/include/vllm/v1/engine/types.h index 23193e302..fb5c321d2 100644 --- a/include/vllm/v1/engine/types.h +++ b/include/vllm/v1/engine/types.h @@ -13,7 +13,8 @@ // // DEFERRED upstream fields, intentionally omitted — later units slot these in // without reshaping the structs: -// EngineCoreRequest: mm_features (multimodal), pooling_params, lora_request, +// EngineCoreRequest: mm_features (multimodal) and pooling_params are now +// PRESENT (pooling_params since ARCH-ONE-SURFACE ROW 6); lora_request, // cache_salt, data_parallel_rank, prompt_embeds, prompt_is_token_ids, // client_index, current_wave, trace_headers, resumable, // external_req_id, reasoning_ended / reasoning_parser_kwargs, @@ -22,17 +23,17 @@ // SamplerOutput: logprobs_tensors now carries the real LogprobsTensors payload // (vllm/v1/outputs.py, ported at M1.7); the sampler's gather_logprobs fills // it. It stays std::optional (None => no logprobs requested this step). -// ModelRunnerOutput: logprobs (LogprobsLists) and prompt_logprobs_dict are -// now PRESENT (ROAD-V1-C7 SAMPLE-LOGPROBS payload); pooler_output, -// kv_connector_output / ec_connector_output (P/D KV transfer), +// ModelRunnerOutput: logprobs (LogprobsLists), prompt_logprobs_dict (ROAD-V1-C7 +// SAMPLE-LOGPROBS) and pooler_output (ARCH-ONE-SURFACE ROW 6) are now +// PRESENT; kv_connector_output / ec_connector_output (P/D KV transfer), // num_nans_in_logits, cudagraph_stats, routed_experts, and the // with_kv_conn_output_only / EMPTY_MODEL_RUNNER_OUTPUT helpers stay deferred. // EngineCoreOutput: new_logprobs / new_prompt_logprobs_tensors are now PRESENT // (ROAD-V1-C7); events (EngineCoreEvent) is now PRESENT // (SERVE-RESPONSE-METRICS — the per-request QUEUED/SCHEDULED/PREEMPTED -// timing events the scheduler drains onto each output); pooling_output, -// kv_transfer_params, trace_headers, prefill_stats, routed_experts, -// num_nans_in_logits deferred. +// timing events the scheduler drains onto each output); pooling_output is +// now PRESENT (ARCH-ONE-SURFACE ROW 6); kv_transfer_params, trace_headers, +// prefill_stats, routed_experts, num_nans_in_logits deferred. // EngineCoreOutputs: scheduler_stats (SchedulerStats), utility_output, // finished_requests, wave_complete / start_wave (DP wave signalling), and // the __post_init__ monotonic-timestamp default (the frontend stamps it). @@ -57,6 +58,7 @@ #include #include +#include "vllm/model_executor/layers/pooler/pooling_params.h" // PoolingParams (pooling seam) #include "vllm/multimodal/inputs.h" // multimodal::MultiModalFeatureSpec (mm seam) #include "vllm/sampling_params.h" #include "vllm/v1/metrics/stats.h" // vllm::v1::SchedulerStats (per-step stats) @@ -97,6 +99,13 @@ struct EngineCoreRequest { // carried into Request.lora_name for the prefix-cache extra-key path. nullopt // for a base-model request. Full LoRA runtime is LORA-RUNTIME. std::optional lora_name = std::nullopt; + // pooling_params (EngineCoreRequest.pooling_params, ARCH-ONE-SURFACE ROW 6): + // set iff this is a POOLING-task request (upstream `params: SamplingParams | + // PoolingParams` — the union collapses to sampling_params + this optional). + // nullopt on every generation request -> byte-identical text path. Carried + // into Request::pooling_params; the scheduler's pooling stop + // (scheduler.py:1718-1721 mirror) fires only when it is set. + std::optional pooling_params = std::nullopt; }; // SamplerOutput (vllm/v1/outputs.py): the raw sampler result for a step. @@ -139,6 +148,13 @@ struct ModelRunnerOutput { // tensor SOURCE (lm_head over prompt positions) is a runner/prefill addition // (SAMPLE-PROMPT-LOGPROBS); the OUTPUT plumbing below consumes it 1:1. std::map prompt_logprobs_dict; + // pooler_output (ModelRunnerOutput.pooler_output, vllm/v1/outputs.py; + // ARCH-ONE-SURFACE ROW 6): one entry per req in `req_ids` order on a POOLING + // model's step — the pooled vector, or nullopt while the request is still + // consuming prefill chunks (the is_valid=false rows, + // pool/pooling_runner.py:40-41). EMPTY (size 0) on every generation step -> + // byte-identical text path. + std::vector>> pooler_output; }; // DraftTokenIds (vllm/v1/outputs.py:310-315): the drafter's proposal for the @@ -179,6 +195,11 @@ struct EngineCoreOutput { // IterationStats.update_from_events (stats.py:428-450) to fill the request's // queue/prefill/inference timing intervals + the preemption counter. std::optional> events; + // pooling_output (EngineCoreOutput.pooling_output, ARCH-ONE-SURFACE ROW 6): + // the pooled vector of a finished POOLING request (scheduler.py:1837 + // `pooling_output=pooler_output`). nullopt on every generation output -> + // byte-identical text path. + std::optional> pooling_output; // finished (property): a request is finished iff finish_reason is set. bool Finished() const { return finish_reason.has_value(); } diff --git a/include/vllm/v1/request.h b/include/vllm/v1/request.h index 28eafdffe..13196d4a3 100644 --- a/include/vllm/v1/request.h +++ b/include/vllm/v1/request.h @@ -58,6 +58,7 @@ #include #include +#include "vllm/model_executor/layers/pooler/pooling_params.h" // PoolingParams #include "vllm/multimodal/inputs.h" // multimodal::MultiModalFeatureSpec #include "vllm/sampling_params.h" #include "vllm/v1/core/kv_cache_utils.h" // BlockHash, BlockHasher @@ -173,6 +174,14 @@ struct Request { // model's EOS token id (for the stop check) rides on sampling_params, as // sampling_params.eos_token_id — read it there, matching upstream check_stop. SamplingParams sampling_params; + // pooling_params (upstream Request.pooling_params, vllm/v1/request.py; + // ARCH-ONE-SURFACE ROW 6): set iff this is a POOLING-task request. The + // scheduler finishes such a request as soon as the runner produced its + // pooled output (scheduler.py:1718-1721 mirror); nullopt on every generation + // request keeps the text path byte-identical. sampling_params above stays a + // benign greedy default for the pooling case (the InputBatch admit reads it; + // the sampler is never invoked on a pooling model's step). + std::optional pooling_params; // structured_output_request (request.py:87-92): the per-request structured // output state (constraint params + the compiled grammar), or nullopt when the // request has no structured-output constraint. Populated at construction from diff --git a/include/vllm/v1/worker/gpu/runner.h b/include/vllm/v1/worker/gpu/runner.h index 5b79710d9..ed6f9241b 100644 --- a/include/vllm/v1/worker/gpu/runner.h +++ b/include/vllm/v1/worker/gpu/runner.h @@ -79,6 +79,7 @@ #include "vllm/v1/worker/gpu/input_batch.h" #include "vllm/v1/worker/gpu/model_runner_base.h" #include "vllm/v1/worker/gpu/prepare_inputs.h" +#include "vllm/v1/worker/gpu/pool/pooling_runner.h" // PoolingRunner (pooling arch) #include "vt/device.h" #include "vt/tensor.h" @@ -333,6 +334,18 @@ class GPUModelRunner final : public ModelRunnerBase { std::unique_ptr draft_model = nullptr, std::vector draft_kv = {}); + // ARCH-ONE-SURFACE ROW 6: the pooling counterpart of sample_tokens (mirror + // of gpu/model_runner.py:1586-1607 + pool/pooling_runner.py:29-42). Consumes + // the stashed forward result — for the pooling arch those are the + // [rows, hidden] post-final-norm hidden states, NOT vocab logits — applies + // the model's Pooler via pooling_runner_, and returns a ModelRunnerOutput + // whose pooler_output carries one pooled vector per fully-prefilled request + // (nullopt for rows still consuming prefill chunks — the same + // seq_len == prompt_len validity predicate as is_valid, pooling_runner.py: + // 40-41, which our discard mask already computes). sampled_token_ids rows + // stay EMPTY: a pooling step samples nothing. + ModelRunnerOutput pool_tokens(); + // Allocate the per-full-attn-layer paged KV buffers + the per-GDN-layer // persistent mamba ssm/conv buffers from the KVCacheConfig groups. void initialize_kv_cache(const KVCacheConfig& kv_cache_config); @@ -378,6 +391,13 @@ class GPUModelRunner final : public ModelRunnerBase { vt::Queue queue_; InputBatch input_batch_; Sampler sampler_; + // ARCH-ONE-SURFACE ROW 6 (mirror of gpu/model_runner.py:368-369 + // `if self.is_pooling_model ...: self.pooling_runner = PoolingRunner(model)`): + // non-null iff the loaded model's registration declares is_pooling_model and + // the model owns a Pooler. sample_tokens then routes to pool_tokens() — the + // POOLED DATA takes the place of sampled tokens (model_runner.py:1586-1607). + // Null for every text arch: the sampler path below is byte-identical. + std::unique_ptr pooling_runner_; // KV group layout (resolved from the KVCacheConfig). int full_attn_group_id_ = -1; diff --git a/scripts/abi-capability-allowlist.txt b/scripts/abi-capability-allowlist.txt index 98d28c824..bfa4be883 100644 --- a/scripts/abi-capability-allowlist.txt +++ b/scripts/abi-capability-allowlist.txt @@ -19,5 +19,4 @@ # # Format: ` | fold= | `. -embeddings / pooling | fold=ARCH-ONE-SURFACE | Engine-side pooler landed (ENG-POOLER-SEQ: pooler/{dispatch_pooler,methods,heads,poolers}.cpp + pool/pooling_runner) but PoolingRunner is never invoked by the live step, no arch sets is_pooling_model, /v1/embeddings is a run_batch "does not support endpoint" residual, and vllm.h has no embed/pool symbol. Fold: grow ABI pooling entry point, wire the live route, register a pooling arch multimodal input (image/audio/video) | fold=ARCH-ONE-SURFACE | vllm_chat/vllm_chat_stream are text-only (vllm.h:440,451); the C-ABI's EnsureChatServing never sets the mm seam (src/capi/vllm_c.cpp), and even the server's Qwen3-VL image seam does not run the vision tower in the live engine step (MM-SERVE-E2E residual). Fold: grow ABI multimodal-input entry point + the engine mm-forward diff --git a/scripts/check-gate-commands.py b/scripts/check-gate-commands.py index d6c5ac5dc..64e25e6c1 100755 --- a/scripts/check-gate-commands.py +++ b/scripts/check-gate-commands.py @@ -244,6 +244,11 @@ def audit() -> list[dict]: "KERNEL-GEMM-CPU-ELEM", "KV-CHUNKED-LOCAL-SPEC", "KV-SLIDING-LOCAL-SPECS", + # ARCH-ONE-SURFACE ROW 6 (2026-08-08): embeddings-one-surface.md carries a + # runnable Gates section (preflight + the fold/capi/server suites) for the + # two rows it activates. + "MODEL-EMBED-llama-llama-for-causal-lm", + "SERVE-POOLING-ENDPOINTS", "KV-SLIDING-WINDOW-SPEC", "LOAD-SAFETENSORS-DIRECT-DENSE", "MODEL-FACTORY-registry", diff --git a/scripts/check-runner-routing-consistency.py b/scripts/check-runner-routing-consistency.py index 267dacfee..845f165ef 100644 --- a/scripts/check-runner-routing-consistency.py +++ b/scripts/check-runner-routing-consistency.py @@ -303,13 +303,16 @@ def resolve_alias(cls: str, alias: dict[str, str]) -> str: return cls +_IS_POOLING = re.compile(r"\.is_pooling_model\s*=\s*true") + + @dataclass(frozen=True) class ModelRoute: """The decode-routing verdict for one REGISTER_VLLM_MODEL registration.""" name: str # allowlist key (registry stem minus _registry) reg_file: str forward_fn: str - classification: str # DEVICE | HOST | REFUSE | NONE + classification: str # DEVICE | HOST | REFUSE | POOLING | NONE private_generate_loop: bool # invariant (b): ships a *GenerateCore host loop device_source: str = "" # which delegated class supplied the device seam activation: str = "BF16_RESIDENT" # invariant (c): F32_STREAM | BF16_RESIDENT @@ -402,7 +405,17 @@ def scan_registrations( } device_source = "" classification = "NONE" - if classify_with_helpers(body, text) == "DEVICE": + # POOLING registrations (ARCH-ONE-SURFACE ROW 6): a registry TU that + # declares `.is_pooling_model = true` registers a NON-GENERATIVE model - + # its forward is a HIDDEN-STATE producer for the PoolingRunner (the + # engine pools instead of sampling, gpu/model_runner.py:1586-1607), so + # the device-resident-LOGITS seam does not apply BY DESIGN, exactly as + # a refuse-by-name stub decodes nothing. Classified explicitly (never + # the silently-exempt NONE bucket) and excluded from the HOST-drift + # gate; the bf16-activation invariant still applies to its body. + if _IS_POOLING.search(strip_comments(text)): + classification = "POOLING" + elif classify_with_helpers(body, text) == "DEVICE": classification, device_source = "DEVICE", fn for cls in delegated_classes: impl = fd_bodies.get(cls) @@ -411,7 +424,7 @@ def scan_registrations( impl_class = classify_with_helpers(impl[1], file_text.get(impl[0], "")) if impl_class == "DEVICE" and classification != "DEVICE": classification, device_source = "DEVICE", cls - if classification != "DEVICE": + if classification not in ("DEVICE", "POOLING"): # Not device-reachable: rank the delegated ForwardDevice impls, else the # hook body itself, as HOST > REFUSE > NONE. impl_classes = [ @@ -522,6 +535,7 @@ def main() -> int: n_device = sum(1 for r in scanned.values() if r.classification == "DEVICE") n_host = sum(1 for r in scanned.values() if r.classification == "HOST") n_refuse = sum(1 for r in scanned.values() if r.classification == "REFUSE") + n_pooling = sum(1 for r in scanned.values() if r.classification == "POOLING") n_none = sum(1 for r in scanned.values() if r.classification == "NONE") n_f32 = sum(1 for r in scanned.values() if r.activation == "F32_STREAM") n_bf16 = sum( @@ -566,7 +580,9 @@ def main() -> int: f"OK (runner-routing): {len(scanned)} registered model(s); " f"{n_device} return device-resident logits on the runner, " f"{n_host} host-logits off-framework ({len(allowlisted)} allowlisted), " - f"{n_refuse} refuse-by-name stub(s) skipped, {n_none} no-logit-producer." + f"{n_refuse} refuse-by-name stub(s) skipped, " + f"{n_pooling} pooling (hidden-state producer(s) for the PoolingRunner), " + f"{n_none} no-logit-producer." ) # Invariant (c): bf16-resident activations (no hand-rolled f32 host stream). diff --git a/scripts/check-supported-models.py b/scripts/check-supported-models.py index 9fd9c4429..e647a21a5 100644 --- a/scripts/check-supported-models.py +++ b/scripts/check-supported-models.py @@ -58,7 +58,8 @@ # future registered arch stops matching this, the self-check below fails loudly # rather than silently dropping it from the comparison. ARCH_TOKEN_RE = re.compile( - r"`([A-Za-z0-9_]+For(?:CausalLM|ConditionalGeneration|CTC|RNNT|TDT))`" + r"`([A-Za-z0-9_]+(?:For(?:CausalLM|ConditionalGeneration|CTC|RNNT|TDT)" + r"|Model))`" ) diff --git a/scripts/mm/llama_embed_fixture_gen.py b/scripts/mm/llama_embed_fixture_gen.py new file mode 100644 index 000000000..1331b14b7 --- /dev/null +++ b/scripts/mm/llama_embed_fixture_gen.py @@ -0,0 +1,189 @@ +#!/usr/bin/env python3 +"""Generate the committed tiny `LlamaModel` EMBEDDING fixture (ARCH-ONE-SURFACE +ROW 6, the #121 committed-fixture precedent). + +Writes tests/vllm/models/fixtures/llama_embed_e2e/: + config.json — architectures ["LlamaModel"] (the bare *Model arch the + upstream _EMBEDDING_MODELS maps onto the Llama backbone, + vllm/model_executor/models/registry.py:230) + tokenizer.json — minimal Metaspace BPE vocab (tok::Tokenizer-parseable) + model.safetensors — deterministic tiny bf16 backbone in the BARE `*Model` + name layout (embed_tokens.weight / layers.N... / + norm.weight — NO "model." prefix, NO lm_head), the + layout adapters.py:178-181 maps with candidate_prefixes + +Deterministic: fixed seed, no torch. Re-running reproduces byte-identical +files, so the committed fixture is reviewable. + +Usage: python3 scripts/mm/llama_embed_fixture_gen.py +""" + +from __future__ import annotations + +import json +import struct +from pathlib import Path + +OUT = Path(__file__).resolve().parents[2] / "tests/vllm/models/fixtures/llama_embed_e2e" + +HIDDEN = 64 +LAYERS = 2 +HEADS = 4 +KV_HEADS = 2 +HEAD_DIM = 16 +INTERMEDIATE = 128 +VOCAB = 32 + + +def f32_to_bf16_bits(x: float) -> int: + """Round-to-nearest-even f32 -> bf16, matching vt::F32ToBF16.""" + (bits,) = struct.unpack("> 16) & 1 + rounding = 0x7FFF + lsb + return ((bits + rounding) >> 16) & 0xFFFF + + +class Rng: + """Deterministic xorshift32 in [-scale, scale) — no numpy dependency.""" + + def __init__(self, seed: int): + self.state = seed & 0xFFFFFFFF or 1 + + def next_u32(self) -> int: + x = self.state + x ^= (x << 13) & 0xFFFFFFFF + x ^= x >> 17 + x ^= (x << 5) & 0xFFFFFFFF + self.state = x + return x + + def uniform(self, scale: float) -> float: + return (self.next_u32() / 2**32 * 2.0 - 1.0) * scale + + +def bf16_tensor(shape: list[int], seed: int, scale: float = 0.08) -> bytes: + rng = Rng(seed) + numel = 1 + for s in shape: + numel *= s + return b"".join( + struct.pack(" None: + header: dict[str, dict] = {} + offset = 0 + for name, (shape, data) in tensors.items(): + header[name] = { + "dtype": "BF16", + "shape": shape, + "data_offsets": [offset, offset + len(data)], + } + offset += len(data) + hdr = json.dumps(header, sort_keys=True).encode() + with open(path, "wb") as f: + f.write(struct.pack(" None: + OUT.mkdir(parents=True, exist_ok=True) + + config = { + "architectures": ["LlamaModel"], + "model_type": "llama", + "hidden_size": HIDDEN, + "num_hidden_layers": LAYERS, + "num_attention_heads": HEADS, + "num_key_value_heads": KV_HEADS, + "head_dim": HEAD_DIM, + "intermediate_size": INTERMEDIATE, + "rms_norm_eps": 1e-5, + "rope_theta": 500000.0, + "vocab_size": VOCAB, + "max_position_embeddings": 128, + "torch_dtype": "bfloat16", + "tie_word_embeddings": False, # embedding conversion: NO lm_head at all + "attention_bias": False, + } + (OUT / "config.json").write_text(json.dumps(config, indent=1) + "\n") + + # Minimal Metaspace BPE the tok::Tokenizer parses; ids stay < VOCAB. + tokenizer = { + "version": "1.0", + "pre_tokenizer": { + "type": "Metaspace", + "replacement": "▁", + "prepend_scheme": "always", + "split": True, + }, + "decoder": { + "type": "Metaspace", + "replacement": "▁", + "prepend_scheme": "always", + "split": True, + }, + "model": { + "type": "BPE", + "unk_token": None, + # Character-level vocab (BPE with empty merges tokenizes each + # Metaspace pre-token into characters): every lowercase test + # string encodes to in-vocab ids, so EncodeWithSpecialTokens never + # fails and ids stay < vocab_size. + "vocab": {"▁": 0} | {chr(c): i + 1 for i, c in enumerate(range(ord("a"), ord("z") + 1))}, + "merges": [], + }, + "added_tokens": [], + } + (OUT / "tokenizer.json").write_text(json.dumps(tokenizer, indent=1) + "\n") + + qdim = HEADS * HEAD_DIM + kdim = KV_HEADS * HEAD_DIM + tensors: dict[str, tuple[list[int], bytes]] = {} + seed = 1 + # BARE `*Model` layout: no "model." prefix, no lm_head (adapters.py:135-181). + tensors["embed_tokens.weight"] = ([VOCAB, HIDDEN], bf16_tensor([VOCAB, HIDDEN], seed)) + seed += 1 + tensors["norm.weight"] = ([HIDDEN], bf16_tensor([HIDDEN], seed, 0.5)) + seed += 1 + for layer in range(LAYERS): + base = f"layers.{layer}." + tensors[base + "input_layernorm.weight"] = ( + [HIDDEN], bf16_tensor([HIDDEN], seed, 0.5)) + seed += 1 + tensors[base + "post_attention_layernorm.weight"] = ( + [HIDDEN], bf16_tensor([HIDDEN], seed, 0.5)) + seed += 1 + tensors[base + "self_attn.q_proj.weight"] = ( + [qdim, HIDDEN], bf16_tensor([qdim, HIDDEN], seed)) + seed += 1 + tensors[base + "self_attn.k_proj.weight"] = ( + [kdim, HIDDEN], bf16_tensor([kdim, HIDDEN], seed)) + seed += 1 + tensors[base + "self_attn.v_proj.weight"] = ( + [kdim, HIDDEN], bf16_tensor([kdim, HIDDEN], seed)) + seed += 1 + tensors[base + "self_attn.o_proj.weight"] = ( + [HIDDEN, qdim], bf16_tensor([HIDDEN, qdim], seed)) + seed += 1 + tensors[base + "mlp.gate_proj.weight"] = ( + [INTERMEDIATE, HIDDEN], bf16_tensor([INTERMEDIATE, HIDDEN], seed)) + seed += 1 + tensors[base + "mlp.up_proj.weight"] = ( + [INTERMEDIATE, HIDDEN], bf16_tensor([INTERMEDIATE, HIDDEN], seed)) + seed += 1 + tensors[base + "mlp.down_proj.weight"] = ( + [HIDDEN, INTERMEDIATE], bf16_tensor([HIDDEN, INTERMEDIATE], seed)) + seed += 1 + + write_safetensors(OUT / "model.safetensors", tensors) + total = sum(len(d) for _, d in tensors.values()) + print(f"wrote {OUT} (weights {total} bytes, {len(tensors)} tensors)") + + +if __name__ == "__main__": + main() diff --git a/src/capi/vllm_c.cpp b/src/capi/vllm_c.cpp index 00071ea42..b36d61173 100644 --- a/src/capi/vllm_c.cpp +++ b/src/capi/vllm_c.cpp @@ -95,6 +95,9 @@ struct vllm_engine { // the HTTP server's worker pool). Guarded by chat_mutex for the lazy build. std::mutex chat_mutex; std::unique_ptr chat_serving; + // ABI v15: serialize vllm_embed batches per handle (the pooling path drives + // the SYNCHRONOUS LLMEngine, not the AsyncLLM the text entry points share). + std::mutex embed_mutex; }; // One non-blocking callback-delivery request. The AsyncLLM output handler owns @@ -146,9 +149,22 @@ void ClearError() { g_last_error.clear(); } // A transcription-only handle (Parakeet) reports an actionable error instead // of dereferencing the null LoadedEngine — the SupportsTranscription-only // mirror of vLLM excluding "generate" from supported_tasks -// (vllm/model_executor/models/interfaces.py:1118). +// (vllm/model_executor/models/interfaces.py:1118). Since ABI v15 the SAME +// guard also refuses a POOLING (embedding) engine: its model has no +// text-generation path either (is_pooling_model && !is_text_generation_model, +// the mirror of vLLM's runner_type validation, config/model.py:607-613) — +// running generate on it would sample over hidden states. bool RequireTextEngine(const vllm_engine* engine, const char* fn) { - if (engine->loaded != nullptr) return true; + if (engine->loaded != nullptr) { + if (engine->loaded->is_pooling_model()) { + SetError(std::string(fn) + + ": this engine was loaded from a pooling (embedding) " + "checkpoint; it has no text-generation path — use vllm_embed " + "or the server's /v1/embeddings"); + return false; + } + return true; + } SetError(std::string(fn) + ": this engine was loaded from a transcription-only checkpoint " "(Parakeet); it has no text-generation path — use vllm_transcribe"); @@ -784,6 +800,11 @@ VLLM_API vllm_status vllm_complete_tokens( SetError("vllm_complete_tokens: out_tokens is null with max_out_tokens > 0"); return VLLM_ERR_INVALID_ARGUMENT; } + // Refuse-by-task (ABI v15 tightening): v13 shipped this entry point without + // the v11 guard, so a transcription-only handle would deref the null + // LoadedEngine here; the same guard now also refuses pooling engines. + if (!RequireTextEngine(engine, "vllm_complete_tokens")) + return VLLM_ERR_INVALID_ARGUMENT; try { const vllm::SamplingParams sp = ToSamplingParams(*params, vllm::RequestOutputKind::kCumulative); @@ -1227,6 +1248,127 @@ VLLM_API void vllm_transcription_free(vllm_transcription* out) { out->has_text = 0; } +// ── Embeddings (ABI v15, ARCH-ONE-SURFACE ROW 6) ──────────────────────────── +// The pooling slice of the ONE surface: the SAME registry forward + +// PoolingRunner engine step the server's /v1/embeddings drives +// (LLMEngine::embed -> pool_tokens, the mirror of LLM.embed / +// entrypoints/pooling/offline.py:65-119 with pooling_task="embed"). + +VLLM_API vllm_status vllm_embed(vllm_engine* engine, const char* const* texts, + int32_t n_texts, vllm_embedding_result* out) { + if (out == nullptr) { + SetError("vllm_embed: out is null"); + return VLLM_ERR_INVALID_ARGUMENT; + } + out->values = nullptr; + out->n_embeddings = 0; + out->dim = 0; + out->prompt_tokens = 0; + if (engine == nullptr || texts == nullptr) { + SetError("vllm_embed: engine or texts is null"); + return VLLM_ERR_INVALID_ARGUMENT; + } + if (n_texts <= 0) { + SetError("vllm_embed: n_texts must be > 0"); + return VLLM_ERR_INVALID_ARGUMENT; + } + for (int32_t i = 0; i < n_texts; ++i) { + if (texts[i] == nullptr) { + SetError("vllm_embed: texts[" + std::to_string(i) + "] is null"); + return VLLM_ERR_INVALID_ARGUMENT; + } + } + // Refuse-by-task, the other direction of RequireTextEngine: only a POOLING + // (embedding) engine serves this entry point — the mirror of vLLM refusing + // `--runner pooling` on a non-pooling model (config/model.py:612-617). + if (engine->loaded == nullptr) { + SetError( + "vllm_embed: this engine was loaded from a transcription-only " + "checkpoint (Parakeet); use vllm_transcribe"); + return VLLM_ERR_INVALID_ARGUMENT; + } + if (!engine->loaded->is_pooling_model()) { + SetError( + "vllm_embed: this engine was loaded from a text-generation " + "checkpoint; it has no pooling path — use vllm_complete / vllm_chat " + "(embedding checkpoints resolve to a pooling architecture, e.g. " + "LlamaModel)"); + return VLLM_ERR_INVALID_ARGUMENT; + } + try { + // Serialize embed batches per handle: the pooling path drives the + // SYNCHRONOUS LLMEngine step loop (async scheduling resolves OFF for + // pooling models, config/vllm.py:1068-1073 mirror). + std::lock_guard lock(engine->embed_mutex); + const vllm::tok::Tokenizer& tokenizer = engine->loaded->tokenizer(); + vllm::v1::LLMEngine& e = engine->loaded->engine(); + + std::vector> vectors; + vectors.reserve(static_cast(n_texts)); + int64_t total_prompt_tokens = 0; + for (int32_t i = 0; i < n_texts; ++i) { + // The serving tokenization applies the template's special tokens + // (add_special_tokens=True on the OpenAI embedding path). + std::vector ids = tokenizer.EncodeWithSpecialTokens(texts[i]); + if (ids.empty()) { + SetError("vllm_embed: texts[" + std::to_string(i) + + "] tokenized to an empty prompt"); + return VLLM_ERR_INVALID_ARGUMENT; + } + total_prompt_tokens += static_cast(ids.size()); + const std::string request_id = + "embed-" + std::to_string(engine->next_request_id.fetch_add(1)); + vllm::RequestOutput ro = + e.embed(std::move(ids), vllm::PoolingParams{}, request_id); + if (!ro.finished || !ro.pooling_output.has_value()) { + SetError("vllm_embed: engine produced no pooled output"); + return VLLM_ERR_RUNTIME; + } + vectors.push_back(std::move(*ro.pooling_output)); + } + + const size_t dim = vectors.empty() ? 0 : vectors[0].size(); + for (const std::vector& v : vectors) { + if (v.size() != dim || dim == 0) { + SetError("vllm_embed: inconsistent embedding dimensions"); + return VLLM_ERR_RUNTIME; + } + } + float* values = static_cast( + std::malloc(static_cast(n_texts) * dim * sizeof(float))); + if (values == nullptr) { + SetError("vllm_embed: out-of-memory copying embeddings"); + return VLLM_ERR_RUNTIME; + } + for (int32_t i = 0; i < n_texts; ++i) { + std::memcpy(values + static_cast(i) * dim, + vectors[static_cast(i)].data(), + dim * sizeof(float)); + } + out->values = values; + out->n_embeddings = n_texts; + out->dim = static_cast(dim); + out->prompt_tokens = static_cast(total_prompt_tokens); + ClearError(); + return VLLM_OK; + } catch (const std::exception& e) { + SetError(std::string("vllm_embed: ") + e.what()); + return VLLM_ERR_RUNTIME; + } catch (...) { + SetError("vllm_embed: unknown error"); + return VLLM_ERR_UNKNOWN; + } +} + +VLLM_API void vllm_embedding_result_free(vllm_embedding_result* out) { + if (out == nullptr) return; + std::free(out->values); + out->values = nullptr; + out->n_embeddings = 0; + out->dim = 0; + out->prompt_tokens = 0; +} + // ── Video+audio generation (ABI v12, MiniMax-H3) ──────────────────────────── // Thin C wrappers over the ONE library seam // (vllm::multimodal::MiniMaxH3VideoEngine) the server's /v1/videos routes and diff --git a/src/vllm/entrypoints/model_loader.cpp b/src/vllm/entrypoints/model_loader.cpp index 846d77eb6..bbe5249f1 100644 --- a/src/vllm/entrypoints/model_loader.cpp +++ b/src/vllm/entrypoints/model_loader.cpp @@ -553,9 +553,14 @@ vllm::SchedulerConfig LoadedEngine::MakeSchedulerConfig( // ResolveAsyncScheduling(runner_supports_async) yields runner_supports_async // (when otherwise compatible). bool LoadedEngine::ResolveAsyncEnabled( - const vllm::SchedulerConfig& scheduler_config, bool runner_supports_async) { - return vllm::AsyncSchedulingEnabled( - scheduler_config.ResolveAsyncScheduling(runner_supports_async)); + const vllm::SchedulerConfig& scheduler_config, bool runner_supports_async, + bool is_pooling_model) { + // Pooling models resolve async scheduling OFF (the mirror of vLLM disabling + // it by default for pooling models, vllm/config/vllm.py:1068-1073) — the + // landed is_pooling_model arm of ResolveAsyncScheduling, wired here since + // ARCH-ONE-SURFACE ROW 6. false (every text arch) is byte-identical. + return vllm::AsyncSchedulingEnabled(scheduler_config.ResolveAsyncScheduling( + runner_supports_async, is_pooling_model)); } std::unique_ptr LoadedEngine::MakeScheduler( @@ -734,7 +739,8 @@ LoadedEngine::LoadedEngine(HfConfig config, max_model_len_, params.max_num_seqs > 0 ? params.max_num_seqs : 8, max_num_batched_tokens_, params.policy), - runner_.runner_supports_async())), + runner_.runner_supports_async(), + model_->registration().info.is_pooling_model)), max_concurrent_batches_(MakeSchedulerConfig( max_model_len_, params.max_num_seqs > 0 ? params.max_num_seqs diff --git a/src/vllm/entrypoints/openai/api_server.cpp b/src/vllm/entrypoints/openai/api_server.cpp index 624f46b3d..1f692dbe2 100644 --- a/src/vllm/entrypoints/openai/api_server.cpp +++ b/src/vllm/entrypoints/openai/api_server.cpp @@ -3,6 +3,8 @@ // dependency deviation. #include "vllm/entrypoints/openai/api_server.h" +#include +#include #include #include #include @@ -365,6 +367,113 @@ ApiServer::DispatchResult ApiServer::handle_audio_transcriptions( } } +ApiServer::DispatchResult ApiServer::handle_embeddings( + const std::string& request_body) const { + // Mirror of vLLM pooling/embed/api_router.py:28 `create_embedding` over the + // EmbeddingCompletionRequest shape (embed/protocol.py:34: `model` + `input` + // as ONE string or an ARRAY of strings) and the EmbeddingResponse shape + // (embed/protocol.py:173-185). The embedding itself runs through the ONE + // engine path (LLMEngine::embed -> registry forward -> PoolingRunner) — the + // same code path vllm_embed drives, so HTTP and FFI cannot drift. + if (!embedder_) { + // The api_router `if handler is None` mirror (embed/api_router.py:22-25); + // the socket layer never registers the route without an embedder. + return MakeError(500, "InternalServerError", + "The model does not support Embeddings API"); + } + nlohmann::json body; + try { + body = nlohmann::json::parse(request_body); + } catch (const std::exception& e) { + return MakeError(400, "BadRequestError", + std::string("invalid JSON body: ") + e.what()); + } + if (!body.is_object()) { + return MakeError(400, "BadRequestError", "request body must be an object"); + } + // model: honoured like every other serving handler — an unknown name is 404. + if (body.contains("model") && body["model"].is_string() && + !models_.is_base_model(body["model"].get())) { + return MakeError(404, "NotFoundError", + "The model `" + body["model"].get() + + "` does not exist."); + } + // encoding_format: float (the default) only; base64 is a NAMED residual. + if (body.contains("encoding_format") && body["encoding_format"].is_string() && + body["encoding_format"].get() != "float") { + return MakeError(400, "BadRequestError", + "encoding_format '" + + body["encoding_format"].get() + + "' is not supported (supported: float; base64 is a " + "named residual)"); + } + if (body.contains("dimensions") && !body["dimensions"].is_null()) { + // Matryoshka truncation is a NAMED residual of this fold (the pooler op + // supports it; the request plumb does not yet). + return MakeError(400, "BadRequestError", + "dimensions is not supported yet (named residual)"); + } + // input: ONE string or an ARRAY of strings (embed/protocol.py:34 + // EmbeddingCompletionRequest via CompletionRequestMixin). Token-array + // inputs are a NAMED residual. + std::vector inputs; + if (!body.contains("input")) { + return MakeError(400, "BadRequestError", "input is required"); + } + if (body["input"].is_string()) { + inputs.push_back(body["input"].get()); + } else if (body["input"].is_array()) { + for (const nlohmann::json& item : body["input"]) { + if (!item.is_string()) { + return MakeError(400, "BadRequestError", + "input must be a string or an array of strings " + "(token-array inputs are a named residual)"); + } + inputs.push_back(item.get()); + } + if (inputs.empty()) { + return MakeError(400, "BadRequestError", + "input must contain at least one string"); + } + } else { + return MakeError(400, "BadRequestError", + "input must be a string or an array of strings"); + } + + try { + const EmbeddingBatch batch = embedder_(inputs); + if (batch.embeddings.size() != inputs.size()) { + return MakeError(500, "InternalServerError", + "embedder returned a mismatched batch"); + } + nlohmann::json data = nlohmann::json::array(); + for (size_t i = 0; i < batch.embeddings.size(); ++i) { + data.push_back(nlohmann::json{ + {"index", static_cast(i)}, + {"object", "embedding"}, + {"embedding", batch.embeddings[i]}, + }); + } + // id: "embd-" (upstream f"embd-{random_uuid()}", + // embed/protocol.py:180 — the serving_completion.h counter stand-in). + static std::atomic embd_counter{0}; + DispatchResult r; + r.body = nlohmann::json{ + {"id", "embd-" + std::to_string(embd_counter.fetch_add(1))}, + {"object", "list"}, + {"created", static_cast(std::time(nullptr))}, + {"model", models_.model_name()}, + {"data", std::move(data)}, + {"usage", + nlohmann::json{{"prompt_tokens", batch.prompt_tokens}, + {"total_tokens", batch.prompt_tokens}}}, + }.dump(); + return r; + } catch (const std::exception& e) { + return MakeError(500, "InternalServerError", e.what()); + } +} + ApiServer::DispatchResult ApiServer::handle_videos( const std::string& request_body) { // vLLM-Omni's ASYNC video endpoint: validate, enqueue, and return the job id @@ -905,6 +1014,19 @@ void ApiServer::register_routes() { write(handle_server_info(), res); }); + if (embedder_) { + // Embeddings (ARCH-ONE-SURFACE ROW 6). Registered ONLY when an embedder is + // attached (task-conditional, the api_server.py:255-265 supported_tasks + // mirror), so a text server answers 404 at the route table — and an + // embedding server, having no completion_/chat_ handlers, answers 404 on + // the generate routes the same way. + server.Post("/v1/embeddings", + [this, write](const httplib::Request& req, + httplib::Response& res) { + write(handle_embeddings(req.body), res); + }); + } + if (transcriber_) { // Parakeet ASR (ARCH-ONE-SURFACE ROW 1). Registered ONLY when a // transcriber is attached, so a text server answers 404 exactly as before. diff --git a/src/vllm/model_executor/models/llama_embedding_registry.cpp b/src/vllm/model_executor/models/llama_embedding_registry.cpp new file mode 100644 index 000000000..cf1e50b7c --- /dev/null +++ b/src/vllm/model_executor/models/llama_embedding_registry.cpp @@ -0,0 +1,135 @@ +// `LlamaModel` EMBEDDING registry TU — ARCH-ONE-SURFACE fold ROW 6, the first +// live pooling architecture. ADDITIVE self-registration (new TU + one +// REGISTER_VLLM_MODEL line, zero shared-array edits), the parakeet_registry / +// llama_registry precedent. +// +// UPSTREAM MIRROR, exactly: the pinned vLLM's `_EMBEDDING_MODELS` maps +// "LlamaModel": ("llama", "LlamaForCausalLM") +// (vllm/model_executor/models/registry.py:230) and, because LlamaForCausalLM is +// not itself a pooling model, `--runner pooling` resolves `--convert embed` +// (vllm/config/model.py:1058-1060) and wraps the class with `as_embedding_model` +// (adapters.py:230-261): +// * the output layer (lm_head / logits processor) is replaced by a +// missing-layer stage (adapters.py:135-151) — the model FORWARD returns +// hidden states, not logits; +// * `self.pooler = DispatchPooler.for_embedding(pooler_config)` +// (adapters.py:248-257), LAST sequence pooling by default for a +// decoder-only conversion (interfaces_base.py:160 +// `default_seq_pooling_type: ClassVar = "LAST"`); +// * checkpoint weights load from BOTH the `*ForCausalLM` and bare `*Model` +// name layouts (adapters.py:178-181 candidate_prefixes ["", "model."]). +// The registered forward here is therefore the SHARED dense backbone +// (LlamaModel == Qwen3DenseModel, llama.h:39-40) run to the post-final-norm +// hidden (Qwen3DenseModel::ForwardHidden) — no new model was built. +// +// TASK ROUTING (the #121 refuse-by-task precedent, both directions): +// info.is_pooling_model=true + is_text_generation_model=false is the registry +// truth the entrypoints dispatch on — the C ABI refuses vllm_complete/vllm_chat +// on this arch (pointing at vllm_embed / /v1/embeddings) and refuses vllm_embed +// on a text arch; the server registers /v1/embeddings INSTEAD OF the generate +// routes. The engine step routes this model's batches through the landed +// PoolingRunner instead of the sampler (runner.cpp pooling branch, the mirror +// of gpu/model_runner.py:368-369 + 1586-1607). +#include "vllm/model_executor/models/model_registry.h" + +#include +#include +#include +#include + +#include "vllm/model_executor/layers/pooler/dispatch_pooler.h" +#include "vllm/model_executor/layers/pooler/pooler_config.h" +#include "vllm/model_executor/models/llama.h" +#include "vllm/model_executor/models/qwen3_5.h" // ForwardLogits carrier + +namespace vllm { +namespace { + +// registry.py _ModelInfo for the embedding conversion: a POOLING model with NO +// text-generation path (the wrapped class serves task "embed" only — +// pooling_runner.py:22-27 admits exactly ["embed"]). +inline constexpr ModelInfo kLlamaEmbeddingInfo{ + .is_text_generation_model = false, + .is_pooling_model = true, + .is_hybrid = false, + .has_inner_state = false, + .supports_multimodal = false, + .supports_transcription = false, + .supports_transcription_only = false, + .score_type = "bi-encoder", +}; + +// Opaque owned model: the shared dense weight container + the model-owned +// DispatchPooler (the VllmModelForPooling.pooler mirror the PoolingRunner is +// built over, adapters.py:248-257). +class LlamaEmbeddingLoadedModel final : public LoadedModel { + public: + LlamaEmbeddingLoadedModel(const ModelRegistration& registration, + LlamaWeights weights) + : LoadedModel(registration), + weights_(std::move(weights)), + pooler_(DispatchPooler::ForEmbedding(PoolerConfig{}, + SequencePoolingType::kLast)) {} + + const LlamaWeights& weights() const { return weights_; } + const Pooler* pooler() const override { return pooler_.get(); } + + private: + LlamaWeights weights_; + std::unique_ptr pooler_; +}; + +std::unique_ptr LoadLlamaModelEmbedding( + const ModelRegistration& registration, const HfConfig& config, + const ModelSource& source) { + if (source.kind != ModelSource::Kind::kSafetensors) { + throw std::runtime_error( + "Model architecture LlamaModel does not support GGUF weights"); + } + if (source.safetensors == nullptr) { + throw std::runtime_error("safetensors model source is empty"); + } + return std::make_unique( + registration, LoadLlamaModelEmbeddingWeights(*source.safetensors, config)); +} + +void PrepareLlamaModelEmbedding(LoadedModel& model, const HfConfig& config, + vt::Queue& queue) { + (void)model; + (void)config; + (void)queue; +} + +ForwardLogits ForwardLlamaModelEmbedding(LoadedModel& model, + const ModelForwardInput& input) { + // The POOLING forward: shared dense backbone to the post-final-norm hidden, + // NO lm_head (the as_embedding_model missing-layer stage). The returned + // carrier holds [n_out, hidden_size] f32 host rows; the runner's pooling + // branch (never the sampler) consumes them. logits_indices gathers the + // per-request last-token rows exactly as the text path would — which for + // LAST pooling IS upstream's `hidden_states[input_batch.logits_indices]` + // (pooling_runner.py:36). + // (The runner passes empty logits_indices when the gather toggle is off; the + // pooling branch then host-gathers, mirroring the text host path.) + auto& emb = static_cast(model); + return LlamaModel::ForwardHidden(input.token_ids, input.positions, + input.attn_meta, input.attn_kv, + emb.weights(), input.config, input.queue, + input.logits_indices); +} + +const ModelFactory kLlamaEmbeddingFactory{ + .parse_config = &ParseLlamaForCausalLMConfig, + .load_weights = &LoadLlamaModelEmbedding, + .prepare = &PrepareLlamaModelEmbedding, + .forward = &ForwardLlamaModelEmbedding, + .make_kv_cache = &MakeLlamaForCausalLMKVCache, + .is_dense_model = true, +}; + +} // namespace + +REGISTER_VLLM_MODEL(llama_model_embedding, "LlamaModel", kLlamaEmbeddingFactory, + kLlamaEmbeddingInfo) + +} // namespace vllm diff --git a/src/vllm/model_executor/models/llama_weights.cpp b/src/vllm/model_executor/models/llama_weights.cpp index 39e936c53..1c23a63d2 100644 --- a/src/vllm/model_executor/models/llama_weights.cpp +++ b/src/vllm/model_executor/models/llama_weights.cpp @@ -118,4 +118,50 @@ LlamaWeights LoadLlamaForCausalLMWeights( return w; } +LlamaWeights LoadLlamaModelEmbeddingWeights( + const std::vector& shards, const HfConfig& config) { + // ARCH-ONE-SURFACE ROW 6: the `LlamaModel` EMBEDDING checkpoint loader — + // the SAME name map as LoadLlamaForCausalLMWeights (LoadLlamaLayer above is + // the single source of it), with the two as_embedding_model deltas: + // 1. BOTH name layouts load (adapters.py:178-181 candidate_prefixes + // ["", "model."]): a bare `LlamaModel` checkpoint names its tensors + // `embed_tokens.weight` / `layers.N...` / `norm.weight` (no "model." + // prefix); a `*ForCausalLM`-layout export keeps the prefix. The + // resolver maps the canonical "model."-prefixed ask onto whichever + // layout the shards actually carry. + // 2. NO lm_head, ever (adapters.py:135-151 replaces the output layer with + // a missing-layer stage): tie_word_embeddings is forced true so the + // pooling forward — which never multiplies by an output layer — has a + // well-formed container, and a checkpoint lm_head.weight is ignored. + std::unordered_map where; + for (const SafetensorsFile& shard : shards) + for (const std::string& name : shard.Names()) where[name] = &shard; + const TensorResolver get = + [&where](const std::string& name) -> const StTensor& { + std::string key = name; + auto it = where.find(key); + if (it == where.end() && key.rfind("model.", 0) == 0) { + key = key.substr(6); // the bare `*Model` layout + it = where.find(key); + } + VT_CHECK(it != where.end(), "llama embedding: tensor not found: " + name); + return it->second->Get(key); + }; + + VT_CHECK(config.num_hidden_layers > 0, + "llama embedding: num_hidden_layers must be positive"); + + LlamaWeights w; + w.tie_word_embeddings = true; // no output layer on the pooling forward + w.attention_bias = RawBool(config.raw, "attention_bias", false); + + w.embed_tokens = LoadBf16Direct(get, "model.embed_tokens.weight"); + w.final_norm = LoadBf16Direct(get, "model.norm.weight"); + + w.layers.reserve(static_cast(config.num_hidden_layers)); + for (int64_t l = 0; l < config.num_hidden_layers; ++l) + w.layers.push_back(LoadLlamaLayer(get, l, w.attention_bias)); + return w; +} + } // namespace vllm diff --git a/src/vllm/model_executor/models/qwen3.cpp b/src/vllm/model_executor/models/qwen3.cpp index 014d212c3..169ebc340 100644 --- a/src/vllm/model_executor/models/qwen3.cpp +++ b/src/vllm/model_executor/models/qwen3.cpp @@ -215,12 +215,20 @@ void EmbedInto(Dev d, DBuf& hidden, const std::vector& token_ids, // `hidden` DBuf reassignment (RunLayer's `hidden = MlpBlock(...)`) never disturbs // the persistent embedding — the copy is a pure device->device data move, so the // layer sequence and its output are BYTE-IDENTICAL to the pre-split forward. +// `return_hidden` (ARCH-ONE-SURFACE ROW 6, default false = byte-identical +// text path): when true, STOP after the final RMSNorm (+ the logits_indices +// gather) and return the [n_out, H] hidden rows upcast to f32 — the pooling +// forward of an embedding conversion, whose model has NO lm_head at all +// (adapters.py:135-151: as_embedding_model replaces the output layer with a +// missing-layer stage; the pooler consumes the post-final-norm hidden). Every +// existing caller leaves the default, so the lm_head tail is untouched. DBuf ForwardLayers(Dev d, const Tensor& hidden_in, const std::vector& positions, const CommonAttentionMetadata& attn_meta, const std::vector& attn_kv, const Qwen3DenseWeights& weights, const HfConfig& config, - const std::vector& logits_indices) { + const std::vector& logits_indices, + bool return_hidden = false) { const int64_t T = hidden_in.shape[0]; const int64_t H = config.hidden_size; const int64_t vocab = config.vocab_size; @@ -254,13 +262,6 @@ DBuf ForwardLayers(Dev d, const Tensor& hidden_in, vt::RmsNorm(d.q, dnorm.t(), hidden.t(), w_fn, vt::RmsNormArgs{eps, false}, &res.t()); } - // lm_head. Tied (Qwen3-0.6B): logits = hidden @ embed_tokens^T via MatmulBT - // over the [vocab,H] embed table (== [N=vocab,K=H]). Untied: the loaded - // Matmul-B [H,vocab] lm_head via vt::Matmul. - const bool tied = weights.tie_word_embeddings || weights.lm_head.Empty(); - Tensor lm = tied ? ResidentWeight(d, weights.embed_tokens, {vocab, H}) - : ResidentWeight(d, weights.lm_head); - const bool do_gather = !logits_indices.empty() && static_cast(logits_indices.size()) < T; Tensor src = dnorm.t(); @@ -272,6 +273,23 @@ DBuf ForwardLayers(Dev d, const Tensor& hidden_in, src = dgather.t(); } const int64_t n_out = src.shape[0]; + + // ARCH-ONE-SURFACE ROW 6 pooling tail: the post-final-norm hidden rows, + // upcast bf16 -> f32 (vt::CastF32), with NO lm_head — an embedding-converted + // checkpoint has no output layer to multiply by. Never taken by any text + // caller (return_hidden defaults false). + if (return_hidden) { + DBuf dhid(d, DType::kF32, {n_out, H}); + vt::CastF32(d.q, dhid.t(), src); + return dhid; + } + + // lm_head. Tied (Qwen3-0.6B): logits = hidden @ embed_tokens^T via MatmulBT + // over the [vocab,H] embed table (== [N=vocab,K=H]). Untied: the loaded + // Matmul-B [H,vocab] lm_head via vt::Matmul. + const bool tied = weights.tie_word_embeddings || weights.lm_head.Empty(); + Tensor lm = tied ? ResidentWeight(d, weights.embed_tokens, {vocab, H}) + : ResidentWeight(d, weights.lm_head); DBuf logits(d, DType::kF32, {n_out, vocab}); if (tied) vt::MatmulBT(d.q, logits.t(), src, lm); @@ -289,12 +307,13 @@ DBuf ForwardBody(Dev d, const std::vector& token_ids, const CommonAttentionMetadata& attn_meta, const std::vector& attn_kv, const Qwen3DenseWeights& weights, const HfConfig& config, - const std::vector& logits_indices) { + const std::vector& logits_indices, + bool return_hidden = false) { const int64_t T = static_cast(token_ids.size()); DBuf hidden(d, DType::kBF16, {T, config.hidden_size}); EmbedInto(d, hidden, token_ids, weights, config); return ForwardLayers(d, hidden.t(), positions, attn_meta, attn_kv, weights, config, - logits_indices); + logits_indices, return_hidden); } ForwardLogits WrapDeviceLogits(Dev d, DBuf&& dlogits, int64_t rows, int64_t vocab) { @@ -412,6 +431,30 @@ ForwardLogits Qwen3DenseModel::ForwardDevice( return WrapDeviceLogits(d, std::move(dlogits), n_out, config.vocab_size); } +ForwardLogits Qwen3DenseModel::ForwardHidden( + const std::vector& token_ids, const std::vector& positions, + const CommonAttentionMetadata& attn_meta, const std::vector& attn_kv, + const Qwen3DenseWeights& weights, const HfConfig& config, vt::Queue& queue, + const std::vector& logits_indices) { + // ARCH-ONE-SURFACE ROW 6: the POOLING forward — the same embed + layer stack + // as Forward/ForwardDevice, stopping after the final RMSNorm (+ gather) with + // NO lm_head, mirroring an as_embedding_model conversion whose output layer + // is a missing-layer stage (adapters.py:135-151). The [n_out, H] f32 rows are + // downloaded to the host carrier: the landed pooler ops are host-side, and an + // embedding batch is one prefill (no per-step decode loop to keep resident). + Dev d{vt::GetBackend(queue.device.type), queue}; + DBuf dhidden = ForwardBody(d, token_ids, positions, attn_meta, attn_kv, weights, + config, logits_indices, /*return_hidden=*/true); + const int64_t n_out = dhidden.t().shape[0]; + const int64_t H = config.hidden_size; + ForwardLogits fl; + fl.rows = n_out; + fl.vocab = H; // the carrier's row width IS the hidden size on this path + fl.host.resize(static_cast(n_out) * static_cast(H)); + dhidden.Download(d, fl.host.data()); + return fl; +} + // ─── Qwen3DenseDecodeGraph (shared pure-dense decode CUDA-graph driver) ─────── // The pure-dense sibling of Qwen3MoeDecodeGraph (qwen3_moe.cpp) — SAME cold -> // warm -> capture -> replay state machine, SAME padded-batch capture set diff --git a/src/vllm/v1/core/sched/scheduler.cpp b/src/vllm/v1/core/sched/scheduler.cpp index d4db8b6f0..dce84ecfe 100644 --- a/src/vllm/v1/core/sched/scheduler.cpp +++ b/src/vllm/v1/core/sched/scheduler.cpp @@ -775,7 +775,24 @@ EngineCoreOutputs Scheduler::update_from_output( new_token_ids = std::move(result.first); stopped = result.second; } - // DEFERRED: pooling stop. + // Pooling stop (ARCH-ONE-SURFACE ROW 6; scheduler.py:1718-1721 `elif + // request.pooling_params and pooler_output is not None`): a POOLING request + // finishes as soon as the runner produced its pooled output. The runner + // reports nullopt for a row still consuming prefill chunks (the + // is_valid=false rows, pooling_runner.py:40-41), so such a request keeps + // running. pooler_output is EMPTY on every generation step -> the text path + // above is byte-identical. + std::optional> pooler_output; + if (!model_runner_output.pooler_output.empty() && + req_index < static_cast(model_runner_output.pooler_output.size())) { + pooler_output = + model_runner_output.pooler_output[static_cast(req_index)]; + } + if (new_token_ids.empty() && request->pooling_params.has_value() && + pooler_output.has_value()) { + request->status = RequestStatus::kFinishedStopped; + stopped = true; + } // scheduler.py:1636-1651: advance the structured-output FSM by the sampled // tokens. Only when the request produced tokens and the manager says the FSM @@ -838,11 +855,14 @@ EngineCoreOutputs Scheduler::update_from_output( // (upstream's `if new_token_ids or ... or stopped`). A partial-prefill // request that produced neither is skipped: "EngineCore returns no partial // prefill outputs". - if (!new_token_ids.empty() || stopped) { + if (!new_token_ids.empty() || pooler_output.has_value() || stopped) { EngineCoreOutput out; out.request_id = req_id; out.new_token_ids = new_token_ids; out.finish_reason = finish_reason; + // Pooled data rides the output to the frontend (scheduler.py:1837 + // `pooling_output=pooler_output`); nullopt on every generation output. + out.pooling_output = std::move(pooler_output); out.new_logprobs = std::move(new_logprobs); out.new_prompt_logprobs_tensors = std::move(new_prompt_logprobs_tensors); // stop_reason is int|str|None upstream; our EngineCoreOutput carries an diff --git a/src/vllm/v1/engine/llm_engine.cpp b/src/vllm/v1/engine/llm_engine.cpp index 80667ba6e..1d3e43a28 100644 --- a/src/vllm/v1/engine/llm_engine.cpp +++ b/src/vllm/v1/engine/llm_engine.cpp @@ -113,6 +113,41 @@ std::string LLMEngine::add_request(const std::string& request_id, return req_id; } +std::string LLMEngine::add_pooling_request(const std::string& request_id, + std::vector prompt_token_ids, + PoolingParams pooling_params, + int priority) { + // ARCH-ONE-SURFACE ROW 6 — the POOLING-task add. Mirrors the tokens + // add_request step-for-step; the SamplingParams are a benign greedy default + // (temperature 0, max_tokens 1) because the InputBatch admit reads them but + // the sampler is NEVER invoked on a pooling model's step (the runner routes + // to pool_tokens, model_runner.py:1586-1607 mirror). The PoolingParams ride + // the EngineCoreRequest into Request::pooling_params, which arms the + // scheduler's pooling stop (scheduler.py:1718-1721). + SamplingParams greedy; + greedy.temperature = 0.0; + greedy.max_tokens = 1; + EngineCoreRequest request = input_processor_.process_inputs_tokens( + request_id, std::move(prompt_token_ids), std::move(greedy), + /*arrival_time=*/std::nullopt, priority); + if (!pooling_params.task.has_value()) { + pooling_params.task = PoolingTask::kEmbed; + } + if (!pooling_params.use_activation.has_value()) { + pooling_params.use_activation = true; // pooling_runner.py:38 F.normalize + } + request.pooling_params = std::move(pooling_params); + const std::string req_id = request.request_id; + + output_processor_.add_request(request, /*prompt=*/std::nullopt, + /*request_index=*/0); + + auto req = std::make_unique( + Request::FromEngineCoreRequest(request, block_hasher_)); + engine_core_.add_request(std::move(req)); + return req_id; +} + void LLMEngine::FanOutParallelSampling(const EngineCoreRequest& request, std::optional prompt) { // llm_engine.py:280-291. Build the shared ParentRequest, then register n child @@ -227,6 +262,26 @@ RequestOutput LLMEngine::generate(std::vector prompt_token_ids, return result; } +RequestOutput LLMEngine::embed(std::vector prompt_token_ids, + PoolingParams pooling_params, + const std::string& request_id, int priority) { + // The single-request pooling driver (LLM.embed / offline.py:65-119 mirror): + // add the pooling request, then loop step() until it finishes. The finished + // RequestOutput carries the pooled vector in pooling_output. + add_pooling_request(request_id, std::move(prompt_token_ids), + std::move(pooling_params), priority); + RequestOutput result; + while (has_unfinished_requests()) { + std::vector step_outputs = step(); + for (RequestOutput& out : step_outputs) { + if (out.finished) { + result = std::move(out); + } + } + } + return result; +} + RequestOutput LLMEngine::generate(multimodal::MultiModalInputs mm_inputs, SamplingParams params, const std::string& request_id, int priority) { diff --git a/src/vllm/v1/engine/output_processor.cpp b/src/vllm/v1/engine/output_processor.cpp index 3d2ba3b95..f9b845632 100644 --- a/src/vllm/v1/engine/output_processor.cpp +++ b/src/vllm/v1/engine/output_processor.cpp @@ -182,8 +182,10 @@ RequestState RequestState::FromNewRequest(const tok::Tokenizer* tokenizer, std::optional RequestState::make_request_output( const std::vector& new_token_ids, std::optional finish_reason, - std::optional stop_reason) { - // output_processor.py:272-331 (text path; pooling / parent_req deferred). + std::optional stop_reason, + std::optional> pooling_output) { + // output_processor.py:272-331 (text + pooling; parent_req deferred for + // pooling — a pooling request is always n==1). const bool finished = finish_reason.has_value(); const bool final_only = output_kind == RequestOutputKind::kFinalOnly; @@ -240,7 +242,13 @@ std::optional RequestState::make_request_output( out_external_req_id = parent_req->external_req_id(); } - return NewRequestOutput(out_external_req_id, std::move(outputs), out_finished); + RequestOutput ro = + NewRequestOutput(out_external_req_id, std::move(outputs), out_finished); + // ARCH-ONE-SURFACE ROW 6 (output_processor.py:319 pooling branch; recorded + // deviation: upstream returns a separate PoolingRequestOutput class — ours + // carries the pooled vector as an optional field on the ONE RequestOutput). + ro.pooling_output = std::move(pooling_output); + return ro; } RequestOutput RequestState::NewRequestOutput( @@ -449,9 +457,10 @@ OutputProcessorOutput OutputProcessor::process_outputs( req_state.logprobs_processor->update_from_output(eco); } - // 4) Create and handle the RequestOutput (:650-666). + // 4) Create and handle the RequestOutput (:650-666). The pooled vector of + // a finished pooling request rides through (ARCH-ONE-SURFACE ROW 6). std::optional request_output = req_state.make_request_output( - new_token_ids, finish_reason, stop_reason); + new_token_ids, finish_reason, stop_reason, eco.pooling_output); if (request_output.has_value()) { // streaming_input deferred (false) -> no finished=false override. if (req_state.queue != nullptr) { diff --git a/src/vllm/v1/request.cpp b/src/vllm/v1/request.cpp index a725695f4..1a700d157 100644 --- a/src/vllm/v1/request.cpp +++ b/src/vllm/v1/request.cpp @@ -97,6 +97,9 @@ Request Request::FromEngineCoreRequest(const EngineCoreRequest& request, // ordinary text path -> byte-identical hashes. req.cache_salt = request.cache_salt; req.lora_name = request.lora_name; + // Pooling-task marker (ARCH-ONE-SURFACE ROW 6; upstream Request.pooling_params, + // vllm/v1/request.py). nullopt on every generation request -> byte-identical. + req.pooling_params = request.pooling_params; // Now install the hasher and compute the initial block hashes over the fully // populated request. req.block_hasher_ = std::move(block_hasher); diff --git a/src/vllm/v1/worker/gpu/runner.cpp b/src/vllm/v1/worker/gpu/runner.cpp index a62024b74..dd3ea42bd 100644 --- a/src/vllm/v1/worker/gpu/runner.cpp +++ b/src/vllm/v1/worker/gpu/runner.cpp @@ -320,6 +320,13 @@ GPUModelRunner::GPUModelRunner( // so force the sync host input path here. Byte-identical for non-spec // (spec_config_ is nullopt there, so this is AsyncRunnerEnvDefault()). async_input_combine_ = AsyncRunnerEnvDefault() && !spec_config_.has_value(); + // ARCH-ONE-SURFACE ROW 6 (mirror gpu/model_runner.py:368-369): a POOLING + // model's runner pools instead of sampling — build the PoolingRunner over + // the model-owned Pooler. Null for every text arch (byte-identical). + if (model_->registration().info.is_pooling_model && + model_->pooler() != nullptr) { + pooling_runner_ = std::make_unique(*model_->pooler()); + } initialize_kv_cache(kv_cache_config); ModelRegistry::Prepare(*model_, config_, queue_); } @@ -353,6 +360,13 @@ GPUModelRunner::GPUModelRunner( // so force the sync host input path here. Byte-identical for non-spec // (spec_config_ is nullopt there, so this is AsyncRunnerEnvDefault()). async_input_combine_ = AsyncRunnerEnvDefault() && !spec_config_.has_value(); + // ARCH-ONE-SURFACE ROW 6 (mirror gpu/model_runner.py:368-369): a POOLING + // model's runner pools instead of sampling — build the PoolingRunner over + // the model-owned Pooler. Null for every text arch (byte-identical). + if (model_->registration().info.is_pooling_model && + model_->pooler() != nullptr) { + pooling_runner_ = std::make_unique(*model_->pooler()); + } initialize_kv_cache(kv_cache_config); ModelRegistry::Prepare(*model_, config_, queue_); } @@ -1470,6 +1484,87 @@ ModelRunnerOutput GPUModelRunner::sample_tokens_with_rejection(vt::Tensor& logit return out; } +// ARCH-ONE-SURFACE ROW 6: the pooling counterpart of sample_tokens. Mirror of +// gpu/model_runner.py:1586-1607 (pool instead of sample) over the landed +// PoolingRunner (pool/pooling_runner.py:29-42). The stashed forward result of +// the pooling arch is the [rows, hidden] POST-FINAL-NORM HIDDEN (the model has +// no lm_head — adapters.py:135-151), already gathered at logits_indices on the +// default path, which for LAST pooling IS upstream's +// `hidden_states[input_batch.logits_indices]` (pooling_runner.py:36). +ModelRunnerOutput GPUModelRunner::pool_tokens() { + const int num_reqs = exec_state_.num_reqs; + const int64_t hidden = exec_state_.logits.vocab; // == hidden_size here + ForwardLogits& fl = exec_state_.logits; + VT_CHECK(!fl.on_device(), + "pool_tokens: the pooling forward returns a HOST hidden carrier"); + + // One hidden row per request. Default (gather ON): the forward already + // gathered the per-request last-token rows. VT_LOGITS_GATHER=0: re-gather on + // host from the full [num_actual_tokens, hidden] rows, exactly as the text + // host path re-gathers logits. + std::vector gathered; + const float* rows = nullptr; + if (fl.rows == num_reqs) { + rows = fl.host.data(); + } else { + gathered.resize(static_cast(num_reqs) * static_cast(hidden)); + for (int i = 0; i < num_reqs; ++i) { + const int row = exec_state_.step.logits_indices[static_cast(i)]; + std::memcpy(gathered.data() + static_cast(i) * + static_cast(hidden), + fl.host.data() + static_cast(row) * + static_cast(hidden), + static_cast(hidden) * sizeof(float)); + } + rows = gathered.data(); + } + vt::Tensor hidden_rows = vt::Tensor::Contiguous( + const_cast(rows), vt::DType::kF32, vt::Device{vt::DeviceType::kCPU, 0}, + {static_cast(num_reqs), hidden}); + + // PoolingMetadata over the GATHERED buffer: one row per sequence (first == + // last == i), task embed, activation ON — the unconditional L2 normalize of + // pooling_runner.py:38 (a per-request use_activation knob is the matryoshka/ + // dimensions residual, named in the row spec). + vllm::PoolingMetadata md; + for (int i = 0; i < num_reqs; ++i) { + md.pooling_cursor.first_token_indices.push_back(i); + md.pooling_cursor.last_token_indices.push_back(i); + md.pooling_cursor.prompt_lens.push_back(1); + md.pooling_cursor.seq_lens.push_back(1); + md.pooling_cursor.num_scheduled_tokens.push_back(1); + vllm::PoolingParams pp; + pp.task = vllm::PoolingTask::kEmbed; + pp.use_activation = true; + md.pooling_params.push_back(pp); + md.tasks.push_back(vllm::PoolingTask::kEmbed); + } + vllm::PoolerOutput pooled = pooling_runner_->Pool(hidden_rows, md); + VT_CHECK(static_cast(pooled.size()) == num_reqs, + "pool_tokens: pooler must return one vector per request"); + + // Validity = the request's whole prompt has been seen (seq_lens == prompt_len, + // pooling_runner.py:40-41). Our discard mask is the SAME predicate + // (step.seq_lens[i] < num_tokens_no_spec[i] == still consuming prefill), so a + // chunked-prefill row reports nullopt and the request keeps running. + ModelRunnerOutput out; + out.req_ids.reserve(static_cast(num_reqs)); + out.sampled_token_ids.reserve(static_cast(num_reqs)); + out.pooler_output.reserve(static_cast(num_reqs)); + for (int i = 0; i < num_reqs; ++i) { + const std::string& req_id = exec_state_.req_ids[static_cast(i)]; + out.req_ids.push_back(req_id); + out.req_id_to_index[req_id] = i; + out.sampled_token_ids.push_back({}); // a pooling step samples NOTHING + if (exec_state_.discard[static_cast(i)] != 0) { + out.pooler_output.push_back(std::nullopt); + } else { + out.pooler_output.push_back(std::move(pooled[static_cast(i)])); + } + } + return out; +} + ModelRunnerOutput GPUModelRunner::sample_tokens( const std::optional& grammar_output) { ModelRunnerOutput out; @@ -1478,6 +1573,15 @@ ModelRunnerOutput GPUModelRunner::sample_tokens( return out; // 0-token flush step (nothing sampled). } + // POOLING ROUTING (ARCH-ONE-SURFACE ROW 6), mirroring the model-level task + // split of gpu/model_runner.py:1586-1607: on a POOLING model the step's + // output is the POOLED DATA, never a sampled token. pooling_runner_ is set + // iff the registration declares is_pooling_model (ctor), so every text arch + // takes the sampler path below byte-identically. + if (pooling_runner_ != nullptr) { + return pool_tokens(); + } + std::vector sampled_logits; // host buffer; outlives the sampler when used vt::Tensor logits = assemble_sample_logits(grammar_output, sampled_logits); @@ -2165,6 +2269,13 @@ void GPUModelRunner::replay_last_sampled_ops(AsyncDeviceInputs& dev) { std::unique_ptr GPUModelRunner::sample_tokens_async( const std::optional& grammar_output) { + // ARCH-ONE-SURFACE ROW 6: pooling models resolve async scheduling OFF + // (config/vllm.py:1068-1073 mirror in LoadedEngine::ResolveAsyncEnabled), so + // the depth-2 async sampler must never see one — refuse loudly rather than + // run the device sampler over hidden states. + VT_CHECK(pooling_runner_ == nullptr, + "sample_tokens_async: pooling models use the synchronous scheduler " + "(async scheduling is disabled for pooling, config/vllm.py:1068)"); // When async is NOT engaged (production default), degenerate to the byte- // identical synchronous path wrapped as a ready output — so a caller in the // depth-2 loop can always call sample_tokens_async without branching, yet the diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 7b075be42..939b14356 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -80,6 +80,12 @@ vllm_cpp_add_test(test_qwen3_dflash_forward vllm/models/test_qwen3_dflash_forwar target_include_directories(test_qwen3_dflash_forward PRIVATE ${CMAKE_SOURCE_DIR}/src) vllm_cpp_add_test(test_llama_forward vllm/models/test_llama_forward.cpp) target_include_directories(test_llama_forward PRIVATE ${CMAKE_SOURCE_DIR}/src) +# ARCH-ONE-SURFACE fold ROW 6: embeddings through the registry/runner path on +# the committed tiny LlamaModel fixture (see scripts/mm/llama_embed_fixture_gen.py). +vllm_cpp_add_test(test_llama_embedding_fold + vllm/models/test_llama_embedding_fold.cpp) +target_compile_definitions(test_llama_embedding_fold PRIVATE + LLAMA_EMBED_FIXTURE_DIR="${CMAKE_SOURCE_DIR}/tests/vllm/models/fixtures/llama_embed_e2e") vllm_cpp_add_test(test_mistral_forward vllm/models/test_mistral_forward.cpp) target_include_directories(test_mistral_forward PRIVATE ${CMAKE_SOURCE_DIR}/src) vllm_cpp_add_test(test_gemma3_load vllm/models/test_gemma3_load.cpp) @@ -647,7 +653,11 @@ if(VLLM_CPP_SERVER) # library transcription seam on the committed parakeet_e2e fixture # (ARCH-ONE-SURFACE ROW 1). target_compile_definitions(test_openai_api_server PRIVATE - PARAKEET_E2E_FIXTURE_DIR="${CMAKE_SOURCE_DIR}/tests/vllm/models/fixtures/parakeet_e2e") + PARAKEET_E2E_FIXTURE_DIR="${CMAKE_SOURCE_DIR}/tests/vllm/models/fixtures/parakeet_e2e" + # /v1/embeddings dispatch + socket smoke run against the REAL engine path + # (LoadedEngine -> PoolingRunner) on the committed llama_embed_e2e fixture + # (ARCH-ONE-SURFACE ROW 6). + LLAMA_EMBED_FIXTURE_DIR="${CMAKE_SOURCE_DIR}/tests/vllm/models/fixtures/llama_embed_e2e") # M3.6: the OpenAI server CONFORMANCE suite — the full API contract exercised # end to end over the REAL cpp-httplib server on an ephemeral port. vllm_cpp_add_test(test_openai_conformance vllm/entrypoints/openai/test_conformance.cpp) @@ -677,7 +687,8 @@ vllm_cpp_add_test(test_capi capi/test_capi.cpp) target_include_directories(test_capi PRIVATE ${CMAKE_SOURCE_DIR}/src) target_compile_definitions(test_capi PRIVATE PARAKEET_E2E_FIXTURE_DIR="${CMAKE_SOURCE_DIR}/tests/vllm/models/fixtures/parakeet_e2e" - MINIMAX_H3_VIDEO_FOLD_FIXTURE_DIR="${CMAKE_SOURCE_DIR}/tests/vllm/models/fixtures/minimax_h3_video_fold") + MINIMAX_H3_VIDEO_FOLD_FIXTURE_DIR="${CMAKE_SOURCE_DIR}/tests/vllm/models/fixtures/minimax_h3_video_fold" + LLAMA_EMBED_FIXTURE_DIR="${CMAKE_SOURCE_DIR}/tests/vllm/models/fixtures/llama_embed_e2e") vllm_cpp_add_test(test_chat_prompt capi/test_chat_prompt.cpp) target_include_directories(test_chat_prompt PRIVATE ${CMAKE_SOURCE_DIR}/src) diff --git a/tests/capi/c_header_compile.c b/tests/capi/c_header_compile.c index 669e4f013..d02acc6e8 100644 --- a/tests/capi/c_header_compile.c +++ b/tests/capi/c_header_compile.c @@ -70,6 +70,14 @@ int vllm_capi_c_header_check(vllm_engine* eng, const char* prompt) { vllm_transcription_free(&transcript); } + /* Embeddings (ABI v15). */ + { + const char* texts[1] = {"strict-C embed reference"}; + vllm_embedding_result emb; + st = vllm_embed(eng, texts, 1, &emb); + vllm_embedding_result_free(&emb); + } + vllm_engine_free(eng); } diff --git a/tests/capi/test_capi.cpp b/tests/capi/test_capi.cpp index a241c933b..c677cf21d 100644 --- a/tests/capi/test_capi.cpp +++ b/tests/capi/test_capi.cpp @@ -12,6 +12,7 @@ #include +#include #include #include #include @@ -1296,11 +1297,12 @@ TEST_CASE("capi: version and abi-version are exposed") { // transcription slice (vllm_transcribe) is ABI v11; the video-generation // slice (vllm_video_*) is ABI v12; the pre-tokenized completion entry // point (vllm_complete_tokens) is ABI v13; the device-selection field - // (vllm_model_params.device) is ABI v14. The >= pin is the one check that + // (vllm_model_params.device) is ABI v14; the embeddings slice (vllm_embed / + // vllm_embedding_result_free) is ABI v15. The >= pin is the one check that // can catch a WRONG bump: the == VLLM_ABI_VERSION assertions here and in // test_dlopen compare against the same macro and move with it (the #121 // lesson: an == floor moves with the macro and proves nothing). - CHECK(vllm_abi_version() >= 14); + CHECK(vllm_abi_version() >= 15); } // ─── ABI v11: audio transcription (ARCH-ONE-SURFACE ROW 1) ─────────────────── @@ -1839,3 +1841,123 @@ TEST_CASE("capi v14: explicit cpu forces the CPU queue at the EngineParams seam" std::runtime_error); } } + +// ─── ABI v15: embeddings (ARCH-ONE-SURFACE ROW 6) ──────────────────────────── +// The embeddings slice gated THROUGH the public ABI on the committed tiny +// LlamaModel fixture (tests/vllm/models/fixtures/llama_embed_e2e): a REAL +// checkpoint-directory load through vllm_engine_load, then vllm_embed through +// the SAME registry forward + PoolingRunner engine step the fold gate +// (test_llama_embedding_fold) anchors. Plus the argument contract and the +// refuse-by-task pins in BOTH directions (the v11 precedent applied to the +// pooling task). + +namespace { +std::string LlamaEmbedFixture() { return std::string(LLAMA_EMBED_FIXTURE_DIR); } +} // namespace + +TEST_CASE("capi v15: vllm_embed embeds through the public ABI (fixture load)") { + vllm_model_params mp = vllm_model_params_default(); + const std::string dir = LlamaEmbedFixture(); + mp.model_path = dir.c_str(); + vllm_engine* eng = nullptr; + REQUIRE(vllm_engine_load(&mp, &eng) == VLLM_OK); + REQUIRE(eng != nullptr); + + const char* texts[2] = {"the quick brown fox", "the lazy dog"}; + vllm_embedding_result out; + REQUIRE(vllm_embed(eng, texts, 2, &out) == VLLM_OK); + REQUIRE(out.values != nullptr); + CHECK(out.n_embeddings == 2); + CHECK(out.dim == 64); // the fixture's hidden_size + CHECK(out.prompt_tokens > 0); + // Each embedding is unit-L2 (the pooling normalize ran) and the two DIFFER + // (different prompts pool different last-token hiddens). + double delta = 0.0; + for (int32_t r = 0; r < out.n_embeddings; ++r) { + double l2 = 0.0; + for (int32_t c = 0; c < out.dim; ++c) { + const double v = out.values[r * out.dim + c]; + l2 += v * v; + } + CHECK(std::sqrt(l2) == doctest::Approx(1.0).epsilon(1e-5)); + } + for (int32_t c = 0; c < out.dim; ++c) { + delta += std::abs(static_cast(out.values[c]) - + static_cast(out.values[out.dim + c])); + } + CHECK(delta > 1e-3); + + vllm_embedding_result_free(&out); + CHECK(out.values == nullptr); // zeroed after free + CHECK(out.n_embeddings == 0); + vllm_embedding_result_free(&out); // double-free is a safe no-op + vllm_embedding_result_free(nullptr); + vllm_engine_free(eng); +} + +TEST_CASE("capi v15: refuse-by-task in both directions (pooling vs text)") { + // Pooling handle: every text entry point refuses with the actionable message + // instead of driving generation over hidden states. + vllm_model_params mp = vllm_model_params_default(); + const std::string dir = LlamaEmbedFixture(); + mp.model_path = dir.c_str(); + vllm_engine* emb = nullptr; + REQUIRE(vllm_engine_load(&mp, &emb) == VLLM_OK); + + vllm_sampling_params sp = vllm_sampling_params_default(); + vllm_completion comp; + CHECK(vllm_complete(emb, "hello", &sp, &comp) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(std::string(vllm_last_error()).find("pooling (embedding)") != + std::string::npos); + CHECK(std::string(vllm_last_error()).find("vllm_embed") != std::string::npos); + char* chat_out = nullptr; + CHECK(vllm_chat(emb, "{\"messages\":[]}", &chat_out) == + VLLM_ERR_INVALID_ARGUMENT); + CHECK(std::string(vllm_last_error()).find("vllm_embed") != std::string::npos); + { + const int32_t prompt_ids[1] = {0}; + int32_t out_ids[4]; + int32_t n_out = 0; + CHECK(vllm_complete_tokens(emb, prompt_ids, 1, &sp, out_ids, 4, &n_out, + nullptr) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(std::string(vllm_last_error()).find("vllm_embed") != + std::string::npos); + } + vllm_engine_free(emb); + + // Text handle: vllm_embed refuses symmetrically, naming the text entry + // points. + vllm_engine* text = MakeSyntheticEngine(); + REQUIRE(text != nullptr); + const char* texts[1] = {"hello"}; + vllm_embedding_result out; + CHECK(vllm_embed(text, texts, 1, &out) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(std::string(vllm_last_error()).find("text-generation") != + std::string::npos); + CHECK(std::string(vllm_last_error()).find("vllm_complete") != + std::string::npos); + vllm_engine_free(text); +} + +TEST_CASE("capi v15: vllm_embed argument contract") { + vllm_model_params mp = vllm_model_params_default(); + const std::string dir = LlamaEmbedFixture(); + mp.model_path = dir.c_str(); + vllm_engine* eng = nullptr; + REQUIRE(vllm_engine_load(&mp, &eng) == VLLM_OK); + + const char* texts[2] = {"the fox", nullptr}; + vllm_embedding_result out; + // Null engine / texts / out; non-positive n_texts; a NULL texts entry. + CHECK(vllm_embed(nullptr, texts, 1, &out) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(vllm_embed(eng, nullptr, 1, &out) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(vllm_embed(eng, texts, 1, nullptr) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(vllm_embed(eng, texts, 0, &out) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(vllm_embed(eng, texts, -3, &out) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(vllm_embed(eng, texts, 2, &out) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(std::string(vllm_last_error()).find("texts[1]") != std::string::npos); + // On every refused call *out stays zeroed. + CHECK(out.values == nullptr); + CHECK(out.n_embeddings == 0); + vllm_engine_free(eng); +} diff --git a/tests/capi/test_dlopen.cpp b/tests/capi/test_dlopen.cpp index b50e4dccf..584f3563d 100644 --- a/tests/capi/test_dlopen.cpp +++ b/tests/capi/test_dlopen.cpp @@ -30,6 +30,9 @@ namespace { // declarations in vllm.h; a header-less consumer would type them by hand. using fn_version = const char* (*)(void); using fn_abi_version = int32_t (*)(void); +using fn_embed = vllm_status (*)(vllm_engine*, const char* const*, int32_t, + vllm_embedding_result*); +using fn_embed_free = void (*)(vllm_embedding_result*); using fn_model_params_default = vllm_model_params (*)(void); using fn_sampling_params_default = vllm_sampling_params (*)(void); using fn_engine_load = vllm_status (*)(const vllm_model_params*, vllm_engine**); @@ -94,6 +97,12 @@ TEST_CASE("dlopen: libvllm.so resolves the whole C ABI by name and drives it") { auto p_request_error = Sym(lib, "vllm_request_error"); auto p_request_free = Sym(lib, "vllm_request_free"); auto p_string_free = Sym(lib, "vllm_string_free"); + // ABI v15 (ARCH-ONE-SURFACE ROW 6): the embeddings entry points resolve. + auto p_embed = Sym(lib, "vllm_embed"); + auto p_embed_free = Sym(lib, "vllm_embedding_result_free"); + (void)p_embed; + // A NULL free is the documented no-op — drivable with no model loaded. + p_embed_free(nullptr); auto p_completion_free = Sym(lib, "vllm_completion_free"); auto p_last_error = Sym(lib, "vllm_last_error"); diff --git a/tests/scripts/test_check_runner_routing_consistency.py b/tests/scripts/test_check_runner_routing_consistency.py index 91a3b7102..1b88a9e51 100644 --- a/tests/scripts/test_check_runner_routing_consistency.py +++ b/tests/scripts/test_check_runner_routing_consistency.py @@ -139,6 +139,12 @@ def test_private_device_wrapper_classifies_device(self) -> None: # tree's live case. scanned = mod.scan_registrations(mod.MODELS_DIR, mod.INCLUDE_DIR) self.assertEqual(scanned["deepseek_v4"].classification, "DEVICE") + # ARCH-ONE-SURFACE ROW 6: a registry TU declaring `.is_pooling_model = + # true` is a NON-GENERATIVE registration — a hidden-state producer for + # the PoolingRunner, classified POOLING explicitly (deleting the + # checker's pooling arm would drop it into NONE and red the bucket pin + # below). + self.assertEqual(scanned["llama_embedding"].classification, "POOLING") # No registered model may sit in the silently-exempt NONE bucket at all. self.assertEqual( sorted(n for n, r in scanned.items() if r.classification == "NONE"), [] diff --git a/tests/scripts/test_check_supported_models.py b/tests/scripts/test_check_supported_models.py index f1f4a1feb..19267d914 100644 --- a/tests/scripts/test_check_supported_models.py +++ b/tests/scripts/test_check_supported_models.py @@ -106,10 +106,14 @@ def test_empty_registry_fails(self) -> None: def test_unrepresentable_registered_arch_fails_the_self_check(self) -> None: # A registered string the FEATURES arch-token pattern cannot express must # surface as an error, never be silently excluded from the comparison. - registered = REGISTERED | {"WeirdModel"} + # (ARCH-ONE-SURFACE ROW 6 widened the pattern to bare `*Model` archs — + # the upstream _EMBEDDING_MODELS naming, e.g. `LlamaModel` — so the + # unrepresentable example is now a suffix the pattern still cannot + # express, not a `*Model` name.) + registered = REGISTERED | {"WeirdArchitecture"} errors = chk.supported_models_errors(registered, _features(TWO)) self.assertTrue(any("do not match the FEATURES arch-token pattern" in e for e in errors), errors) - self.assertTrue(any("WeirdModel" in e for e in errors), errors) + self.assertTrue(any("WeirdArchitecture" in e for e in errors), errors) class ScopingTests(unittest.TestCase): diff --git a/tests/vllm/entrypoints/openai/test_api_server.cpp b/tests/vllm/entrypoints/openai/test_api_server.cpp index 7a02c86ad..e5548ba79 100644 --- a/tests/vllm/entrypoints/openai/test_api_server.cpp +++ b/tests/vllm/entrypoints/openai/test_api_server.cpp @@ -18,6 +18,8 @@ #include #include +#include +#include #include #include #include @@ -2353,3 +2355,184 @@ TEST_CASE("api_server: an explicit-cpu device-selected engine serves /v1/complet CHECK(j.at("choices").at(0).at("finish_reason") == "length"); CHECK(j.at("usage").at("completion_tokens") == 5); } + +// ─── /v1/embeddings (ARCH-ONE-SURFACE ROW 6) ───────────────────────────────── +// Task-conditional like /v1/audio/transcriptions: a TEXT server never +// registers the route; a pooling (embedding) server registers it and NOT the +// generate routes — vLLM's supported_tasks-conditional registration +// (api_server.py:255-265) + pooling/embed/api_router.py:28 semantics. The +// embedder wraps the REAL engine path (LoadedEngine::FromModelDir on the +// committed llama_embed_e2e fixture -> LLMEngine::embed -> the registry +// forward + PoolingRunner step), the SAME path vllm_embed drives. + +namespace { + +struct EmbedHarness { + vllm::entrypoints::openai::OpenAIServingModels models{"llama-embed-fixture"}; + ApiServer server{models, "test-version"}; + std::shared_ptr loaded; + std::shared_ptr mutex = std::make_shared(); + + EmbedHarness() { + vllm::entrypoints::EngineParams params; + params.max_model_len = 64; + loaded = std::shared_ptr( + vllm::entrypoints::LoadedEngine::FromModelDir( + std::string(LLAMA_EMBED_FIXTURE_DIR), params)); + auto engine = loaded; + auto mu = mutex; + auto counter = std::make_shared>(0); + server.set_embedder( + [engine, mu, counter](const std::vector& inputs) { + std::lock_guard lock(*mu); + ApiServer::EmbeddingBatch batch; + for (const std::string& text : inputs) { + std::vector ids = + engine->tokenizer().EncodeWithSpecialTokens(text); + REQUIRE(!ids.empty()); + batch.prompt_tokens += static_cast(ids.size()); + vllm::RequestOutput ro = engine->engine().embed( + std::move(ids), vllm::PoolingParams{}, + "embd-" + std::to_string(counter->fetch_add(1))); + REQUIRE(ro.finished); + REQUIRE(ro.pooling_output.has_value()); + batch.embeddings.push_back(std::move(*ro.pooling_output)); + } + return batch; + }); + } +}; + +} // namespace + +TEST_CASE("api_server: embeddings dispatch — OpenAI shape over the engine path") { + EmbedHarness h; + + // ONE string input. + ApiServer::DispatchResult r = h.server.handle_embeddings( + R"({"model":"llama-embed-fixture","input":"the quick brown fox"})"); + CHECK(r.status == 200); + json j = json::parse(r.body); + CHECK(j.at("object") == "list"); + CHECK(j.at("model") == "llama-embed-fixture"); + CHECK(std::string(j.at("id")).rfind("embd-", 0) == 0); + REQUIRE(j.at("data").size() == 1); + CHECK(j.at("data").at(0).at("object") == "embedding"); + CHECK(j.at("data").at(0).at("index") == 0); + REQUIRE(j.at("data").at(0).at("embedding").is_array()); + CHECK(j.at("data").at(0).at("embedding").size() == 64); // hidden_size + // Unit L2: the pooling normalize ran. + double l2 = 0.0; + for (const auto& v : j.at("data").at(0).at("embedding")) + l2 += v.get() * v.get(); + CHECK(std::sqrt(l2) == doctest::Approx(1.0).epsilon(1e-5)); + CHECK(j.at("usage").at("prompt_tokens").get() > 0); + CHECK(j.at("usage").at("total_tokens") == j.at("usage").at("prompt_tokens")); + + // ARRAY input: one embedding per string, input order. + r = h.server.handle_embeddings( + R"({"input":["the quick brown fox","the lazy dog"]})"); + CHECK(r.status == 200); + j = json::parse(r.body); + REQUIRE(j.at("data").size() == 2); + CHECK(j.at("data").at(1).at("index") == 1); + + // Malformed / unsupported requests. + CHECK(h.server.handle_embeddings("not json").status == 400); + CHECK(h.server.handle_embeddings(R"({"model":"x"})").status == 404); + CHECK(h.server.handle_embeddings(R"({"input":42})").status == 400); + CHECK(h.server.handle_embeddings(R"({"input":[]})").status == 400); + CHECK(h.server.handle_embeddings(R"({"input":[[1,2]]})").status == 400); + CHECK(h.server + .handle_embeddings( + R"({"input":"x","encoding_format":"base64"})") + .status == 400); + CHECK(h.server.handle_embeddings(R"({"input":"x","dimensions":16})").status == + 400); +} + +TEST_CASE("api_server: embeddings without an embedder is a 500, not a crash") { + vllm::entrypoints::openai::OpenAIServingModels models{"m"}; + ApiServer server{models, "test-version"}; + ApiServer::DispatchResult r = server.handle_embeddings(R"({"input":"x"})"); + CHECK(r.status == 500); + CHECK(json::parse(r.body).at("error").at("message") == + "The model does not support Embeddings API"); +} + +TEST_CASE("api_server: embeddings socket smoke; generate routes 404 on the " + "embedding server") { + EmbedHarness h; + const int port = h.server.bind_to_any_port("127.0.0.1"); + REQUIRE(port > 0); + std::thread server_thread([&h]() { h.server.serve(); }); + for (int i = 0; i < 500 && !h.server.is_running(); ++i) + std::this_thread::sleep_for(std::chrono::milliseconds(2)); + REQUIRE(h.server.is_running()); + + { + httplib::Client client("127.0.0.1", port); + client.set_connection_timeout(5, 0); + client.set_read_timeout(15, 0); + + auto res = client.Post("/v1/embeddings", + R"({"input":"the quick brown fox"})", + "application/json"); + REQUIRE(res); + CHECK(res->status == 200); + CHECK(json::parse(res->body).at("data").at(0).at("embedding").size() == 64); + + // The generate routes are NOT registered on an embedding server (the + // task-conditional registration, both directions). + auto completions = client.Post("/v1/completions", "{}", "application/json"); + REQUIRE(completions); + CHECK(completions->status == 404); + auto chat = client.Post("/v1/chat/completions", "{}", "application/json"); + REQUIRE(chat); + CHECK(chat->status == 404); + + // Liveness + discovery still serve. + auto health = client.Get("/health"); + REQUIRE(health); + CHECK(health->status == 200); + auto models_res = client.Get("/v1/models"); + REQUIRE(models_res); + CHECK(json::parse(models_res->body).at("data").at(0).at("id") == + "llama-embed-fixture"); + } + + h.server.stop(); + server_thread.join(); +} + +TEST_CASE("api_server: /v1/embeddings does not exist on a TEXT server") { + // The reverse pin, the exact twin of "the audio routes do not exist on a + // TEXT server": task-conditional registration means a TEXT-engine server (no + // embedder attached) must answer 404 from the ROUTE TABLE — a well-formed + // request that the handler WOULD accept proves the route was never + // registered (an `if (true)` registration mutation answers 200/400 from the + // handler instead and only THIS test reds). + const HfConfig c = MakeConfig(); + const Qwen3_5MoeWeights w = MakeWeights(c); + ServerHarness h(c, w, Fixture()); + + const int port = h.server.bind_to_any_port("127.0.0.1"); + REQUIRE(port > 0); + std::thread server_thread([&h]() { h.server.serve(); }); + for (int i = 0; i < 500 && !h.server.is_running(); ++i) + std::this_thread::sleep_for(std::chrono::milliseconds(2)); + REQUIRE(h.server.is_running()); + + { + httplib::Client client("127.0.0.1", port); + client.set_connection_timeout(5, 0); + client.set_read_timeout(15, 0); + auto res = client.Post("/v1/embeddings", R"({"input":"hello"})", + "application/json"); + REQUIRE(res); + CHECK(res->status == 404); + } + + h.server.stop(); + server_thread.join(); +} diff --git a/tests/vllm/models/fixtures/llama_embed_e2e/config.json b/tests/vllm/models/fixtures/llama_embed_e2e/config.json new file mode 100644 index 000000000..b3c9a8af9 --- /dev/null +++ b/tests/vllm/models/fixtures/llama_embed_e2e/config.json @@ -0,0 +1,19 @@ +{ + "architectures": [ + "LlamaModel" + ], + "model_type": "llama", + "hidden_size": 64, + "num_hidden_layers": 2, + "num_attention_heads": 4, + "num_key_value_heads": 2, + "head_dim": 16, + "intermediate_size": 128, + "rms_norm_eps": 1e-05, + "rope_theta": 500000.0, + "vocab_size": 32, + "max_position_embeddings": 128, + "torch_dtype": "bfloat16", + "tie_word_embeddings": false, + "attention_bias": false +} diff --git a/tests/vllm/models/fixtures/llama_embed_e2e/model.safetensors b/tests/vllm/models/fixtures/llama_embed_e2e/model.safetensors new file mode 100644 index 000000000..c7766b2a2 Binary files /dev/null and b/tests/vllm/models/fixtures/llama_embed_e2e/model.safetensors differ diff --git a/tests/vllm/models/fixtures/llama_embed_e2e/tokenizer.json b/tests/vllm/models/fixtures/llama_embed_e2e/tokenizer.json new file mode 100644 index 000000000..23e097628 --- /dev/null +++ b/tests/vllm/models/fixtures/llama_embed_e2e/tokenizer.json @@ -0,0 +1,50 @@ +{ + "version": "1.0", + "pre_tokenizer": { + "type": "Metaspace", + "replacement": "\u2581", + "prepend_scheme": "always", + "split": true + }, + "decoder": { + "type": "Metaspace", + "replacement": "\u2581", + "prepend_scheme": "always", + "split": true + }, + "model": { + "type": "BPE", + "unk_token": null, + "vocab": { + "\u2581": 0, + "a": 1, + "b": 2, + "c": 3, + "d": 4, + "e": 5, + "f": 6, + "g": 7, + "h": 8, + "i": 9, + "j": 10, + "k": 11, + "l": 12, + "m": 13, + "n": 14, + "o": 15, + "p": 16, + "q": 17, + "r": 18, + "s": 19, + "t": 20, + "u": 21, + "v": 22, + "w": 23, + "x": 24, + "y": 25, + "z": 26 + }, + "merges": [] + }, + "added_tokens": [] +} diff --git a/tests/vllm/models/test_llama_embedding_fold.cpp b/tests/vllm/models/test_llama_embedding_fold.cpp new file mode 100644 index 000000000..8751597dd --- /dev/null +++ b/tests/vllm/models/test_llama_embedding_fold.cpp @@ -0,0 +1,272 @@ +// ARCH-ONE-SURFACE fold ROW 6 gate — embeddings through the ONE surface. +// +// The pooling lane's cosine gate (tests/vllm/v1/worker/gpu/pool/ +// test_pooling_runner.cpp) is STRUCTURAL: it drives the PoolingRunner over a +// synthetic hidden buffer, BYPASSING the registry. This test is the fold's +// correctness anchor: the SAME correctness statement (LAST-token pool + L2 +// normalize, checked against an independent DOUBLE-PRECISION reference) now +// runs THROUGH the registry/runner path, on the committed tiny `LlamaModel` +// fixture (scripts/mm/llama_embed_fixture_gen.py, the #121 committed-fixture +// precedent), with a REGISTRY-PATH IDENTITY arm: the full-engine embedding +// (LoadedEngine -> scheduler -> GPUModelRunner::pool_tokens) must be +// IDENTICAL to the direct ModelRegistry::Forward + PoolingRunner path. +// +// Upstream mirror grounding: +// registry.py:230 "LlamaModel": ("llama", "LlamaForCausalLM") in +// _EMBEDDING_MODELS +// adapters.py:230 as_embedding_model — backbone forward, NO lm_head, +// DispatchPooler.for_embedding (LAST, interfaces_base.py:160) +// model_runner.py:368-369, 1586-1607 — PoolingRunner built iff pooling model; +// pool replaces sample +// pooling_runner.py:29-42 — gather at logits_indices + normalize; is_valid +// scheduler.py:1718-1721 — pooling stops as soon as there is output +// config/vllm.py:1068-1073 — async scheduling OFF for pooling models +// +// HONEST RESIDUAL: the fixture weights are synthetic (deterministic seed). A +// REAL embedding checkpoint (the e5-mistral class) through this fold — the +// `vllm.LLM(task="embed").encode` oracle cosine — is the NAMED residual; no +// cosine-vs-oracle number is fabricated here. +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/entrypoints/model_loader.h" +#include "vllm/model_executor/layers/pooler/dispatch_pooler.h" +#include "vllm/model_executor/layers/pooler/pooler_config.h" +#include "vllm/model_executor/layers/pooler/pooling_metadata.h" +#include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/model_executor/models/qwen3_5.h" // ForwardLogits carrier +#include "vllm/transformers_utils/hf_config.h" +#include "vllm/v1/attention/backends/gdn_attn.h" +#include "vllm/v1/worker/gpu/pool/pooling_runner.h" +#include "vt/backend.h" +#include "vt/dtype.h" + +namespace { + +using vllm::HfConfig; +using vllm::ModelForwardInput; +using vllm::ModelRegistry; +using vllm::PagedKvCache; +using vllm::v1::CommonAttentionMetadata; +using vllm::v1::GDNAttentionMetadata; + +std::string FixtureDir() { return std::string(LLAMA_EMBED_FIXTURE_DIR); } + +// The fixture's prompt: 5 in-vocab token ids (the tokenizer.json maps +// "the quick brown fox sat" onto these exact ids). +const std::vector kPromptIds = {0, 1, 2, 3, 9}; + +vt::Queue Q() { return vt::Queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; } + +CommonAttentionMetadata PrefillMeta(int64_t T, int64_t block_size) { + CommonAttentionMetadata m; + m.num_reqs = 1; + m.num_actual_tokens = static_cast(T); + m.query_start_loc = {0, static_cast(T)}; + m.query_start_loc_cpu = m.query_start_loc; + m.seq_lens = {static_cast(T)}; + m.seq_lens_cpu = m.seq_lens; + m.max_query_len = static_cast(T); + m.max_seq_len = static_cast(T); + m.block_table_num_cols = 1; + m.block_table_tensor = {0}; + for (int64_t t = 0; t < T; ++t) m.slot_mapping.push_back(t % block_size); + m.causal = true; + return m; +} + +struct CachePool { + std::vector> buf; + std::vector attn_kv; + CachePool(const HfConfig& c, int64_t num_blocks, int64_t block_size) { + const int64_t Hkv = c.num_key_value_heads, Dh = c.head_dim; + for (int64_t l = 0; l < c.num_hidden_layers; ++l) + buf.emplace_back( + static_cast(num_blocks * 2 * block_size * Hkv * Dh), 0.0f); + for (auto& b : buf) { + PagedKvCache kv; + kv.data = b.data(); + kv.dtype = vt::DType::kF32; + kv.num_blocks = num_blocks; + kv.block_size = block_size; + kv.num_kv_heads = Hkv; + kv.head_size = Dh; + attn_kv.push_back(kv); + } + } +}; + +double L2(const std::vector& v) { + double s = 0.0; + for (float x : v) s += static_cast(x) * x; + return std::sqrt(s); +} + +double Cosine(const std::vector& a, const std::vector& b) { + double dot = 0.0, na = 0.0, nb = 0.0; + for (size_t i = 0; i < a.size(); ++i) { + dot += static_cast(a[i]) * b[i]; + na += static_cast(a[i]) * a[i]; + nb += b[i] * b[i]; + } + return dot / (std::sqrt(na) * std::sqrt(nb) + 1e-30); +} + +// The DIRECT registry path: ModelRegistry::Load + ModelRegistry::Forward (the +// registered pooling forward -> [1, H] last-token hidden) + the landed +// PoolingRunner over the model-owned pooler. Returns {pooled, raw_hidden}. +std::pair, std::vector> DirectRegistryEmbed() { + const std::string dir = FixtureDir(); + HfConfig config = vllm::LoadHfConfig(dir + "/config.json"); + std::vector shards; + shards.push_back(vllm::SafetensorsFile::Open(dir + "/model.safetensors")); + std::unique_ptr model = + ModelRegistry::Load(config, vllm::ModelSource::FromSafetensors(shards)); + REQUIRE(model != nullptr); + REQUIRE(model->registration().info.is_pooling_model); + REQUIRE(model->pooler() != nullptr); + + const int64_t T = static_cast(kPromptIds.size()); + std::vector positions; + for (int64_t t = 0; t < T; ++t) positions.push_back(static_cast(t)); + CachePool pool(config, /*num_blocks=*/4, /*block_size=*/8); + const CommonAttentionMetadata am = PrefillMeta(T, 8); + const GDNAttentionMetadata gm{}; // dense: unused + std::vector gdn_state; + vt::Queue q = Q(); + // gather at the LAST token — upstream pooling_runner.py:36 + // `hidden_states[input_batch.logits_indices]`. + const std::vector logits_indices = {static_cast(T - 1)}; + ModelForwardInput in{kPromptIds, positions, am, gm, + pool.attn_kv, gdn_state, config, q, + logits_indices}; + in.num_reqs = 1; + in.gather_logits = true; + vllm::ForwardLogits fl = ModelRegistry::Forward(*model, in); + + // The pooling forward returns a HOST [1, H] hidden carrier (vocab == H). + REQUIRE(!fl.on_device()); + REQUIRE(fl.rows == 1); + REQUIRE(fl.vocab == config.hidden_size); + std::vector hidden = fl.host; + + // The landed PoolingRunner over the model's own DispatchPooler. + vllm::PoolingRunner runner(*model->pooler()); + vllm::PoolingMetadata md; + md.pooling_cursor.first_token_indices = {0}; + md.pooling_cursor.last_token_indices = {0}; + md.pooling_cursor.prompt_lens = {1}; + md.pooling_cursor.seq_lens = {1}; + md.pooling_cursor.num_scheduled_tokens = {1}; + vllm::PoolingParams pp; + pp.task = vllm::PoolingTask::kEmbed; + pp.use_activation = true; + md.pooling_params = {pp}; + md.tasks = {vllm::PoolingTask::kEmbed}; + vt::Tensor rows = vt::Tensor::Contiguous( + hidden.data(), vt::DType::kF32, vt::Device{vt::DeviceType::kCPU, 0}, + {1, config.hidden_size}); + vllm::PoolerOutput out = runner.Pool(rows, md); + REQUIRE(out.size() == 1u); + return {out[0], hidden}; +} + +// The FULL ENGINE path: LoadedEngine::FromModelDir -> LLMEngine::embed -> +// scheduler -> GPUModelRunner::pool_tokens (the ONE path vllm_embed and +// /v1/embeddings drive). +std::vector EngineEmbed(int max_num_batched_tokens = 0) { + vllm::entrypoints::EngineParams params; + params.max_model_len = 64; + if (max_num_batched_tokens > 0) + params.max_num_batched_tokens = max_num_batched_tokens; + std::unique_ptr loaded = + vllm::entrypoints::LoadedEngine::FromModelDir(FixtureDir(), params); + REQUIRE(loaded != nullptr); + CHECK(loaded->is_pooling_model()); + // config/vllm.py:1068-1073 mirror: async scheduling resolves OFF for a + // pooling model (the landed ResolveAsyncScheduling arm, now WIRED). + CHECK_FALSE(loaded->async_scheduling_enabled()); + + vllm::RequestOutput ro = loaded->engine().embed(kPromptIds); + REQUIRE(ro.finished); + REQUIRE(ro.pooling_output.has_value()); + return *ro.pooling_output; +} + +} // namespace + +TEST_CASE( + "llama embedding fold: registry resolves LlamaModel as a pooling model") { + const std::vector archs{"LlamaModel"}; + const vllm::ModelRegistration& reg = + ModelRegistry::Resolve(std::span(archs)); + CHECK(reg.architecture == "LlamaModel"); + CHECK(reg.info.is_pooling_model); + CHECK_FALSE(reg.info.is_text_generation_model); +} + +TEST_CASE( + "llama embedding fold: direct registry path matches the double-precision " + "LAST+normalize reference (the lane's cosine gate, registry-anchored)") { + auto [pooled, hidden] = DirectRegistryEmbed(); + REQUIRE(pooled.size() == hidden.size()); + + // Independent double-precision reference over the SAME hidden row: L2 + // normalize (pooling_runner.py:38 F.normalize; the pooling lane's reference). + std::vector ref(hidden.size()); + double n = 0.0; + for (size_t i = 0; i < hidden.size(); ++i) { + ref[i] = static_cast(hidden[i]); + n += ref[i] * ref[i]; + } + n = std::sqrt(n); + REQUIRE(n > 0.0); + for (double& x : ref) x /= n; + + CHECK(L2(pooled) == doctest::Approx(1.0).epsilon(1e-6)); + CHECK(Cosine(pooled, ref) == doctest::Approx(1.0).epsilon(1e-6)); + for (size_t i = 0; i < pooled.size(); ++i) { + CAPTURE(i); + CHECK(pooled[i] == doctest::Approx(ref[i]).epsilon(1e-5)); + } +} + +TEST_CASE( + "llama embedding fold: the FULL ENGINE path is identical to the direct " + "registry path (the registry-path identity arm)") { + auto [direct, hidden] = DirectRegistryEmbed(); + (void)hidden; + const std::vector engine = EngineEmbed(); + REQUIRE(engine.size() == direct.size()); + CHECK(L2(engine) == doctest::Approx(1.0).epsilon(1e-6)); + for (size_t i = 0; i < engine.size(); ++i) { + CAPTURE(i); + // Same op sequence (registry forward, gather-at-last, normalize) on both + // paths — identical to fp32 round-off. + CHECK(engine[i] == doctest::Approx(direct[i]).epsilon(1e-5)); + } +} + +TEST_CASE( + "llama embedding fold: chunked prefill pools only the FULLY prefilled " + "prompt (is_valid gating) and matches the unchunked vector") { + // max_num_batched_tokens=2 forces the 5-token prompt through 3 prefill + // chunks; the runner reports nullopt for the partial chunks (the + // seq_lens == prompt_len validity predicate, pooling_runner.py:40-41) and + // the scheduler keeps the request running until the LAST chunk pools it. + const std::vector chunked = EngineEmbed(/*max_num_batched_tokens=*/2); + const std::vector whole = EngineEmbed(); + REQUIRE(chunked.size() == whole.size()); + for (size_t i = 0; i < chunked.size(); ++i) { + CAPTURE(i); + CHECK(chunked[i] == doctest::Approx(whole[i]).epsilon(1e-5)); + } +} diff --git a/tests/vllm/models/test_model_registry.cpp b/tests/vllm/models/test_model_registry.cpp index e21d7fef7..9290fe679 100644 --- a/tests/vllm/models/test_model_registry.cpp +++ b/tests/vllm/models/test_model_registry.cpp @@ -45,8 +45,9 @@ HfConfig Config(std::vector architectures) { TEST_CASE("registry_imports: every registered architecture has a complete factory") { const auto registrations = ModelRegistry::Registrations(); // 30 text archs + the 3 Parakeet transcription-only archs (ARCH-ONE-SURFACE - // ROW 1: ParakeetForCTC/ForRNNT/ForTDT, SupportsTranscription mirror). - REQUIRE(registrations.size() == 33); + // ROW 1: ParakeetForCTC/ForRNNT/ForTDT, SupportsTranscription mirror) + the + // LlamaModel embedding arch (ARCH-ONE-SURFACE ROW 6, is_pooling_model). + REQUIRE(registrations.size() == 34); for (const ModelRegistration& registration : registrations) { CAPTURE(registration.architecture); @@ -138,7 +139,7 @@ TEST_CASE("self_registration: every arch self-registers from its own TU") { // with the kExampleConfigArchitectures ledger; adding a model appends its two // entries here. const std::vector supported = ModelRegistry::SupportedArchs(); - REQUIRE(supported.size() == 33); + REQUIRE(supported.size() == 34); CHECK(std::is_sorted(supported.begin(), supported.end())); // The full byte-order sequence. Note "MiniCPM3" < "MiniCPMF" and "Phi3" < // "PhiF" ('3' 0x33 < 'F' 0x46); "OPT" < "Olmo" ('P' 0x50 < 'l' 0x6C); and among @@ -161,6 +162,7 @@ TEST_CASE("self_registration: every arch self-registers from its own TU") { "KimiLinearForCausalLM", "LagunaForCausalLM", "LlamaForCausalLM", + "LlamaModel", "MiniCPM3ForCausalLM", "MiniCPMForCausalLM", "MistralForCausalLM", @@ -206,6 +208,20 @@ TEST_CASE("registry_model_property: Qwen registrations match pinned _ModelInfo") CHECK_FALSE(registration.info.supports_multimodal); continue; } + if (registration.architecture == "LlamaModel") { + // ARCH-ONE-SURFACE ROW 6: the EMBEDDING conversion of the Llama backbone + // (_EMBEDDING_MODELS "LlamaModel" -> ("llama", "LlamaForCausalLM"), + // registry.py:230, wrapped by as_embedding_model adapters.py:230) — a + // POOLING model with NO text-generation path; the text entrypoints + // refuse it by task and vllm_embed / /v1/embeddings serve it. + CHECK(registration.info.is_pooling_model); + CHECK_FALSE(registration.info.is_text_generation_model); + CHECK_FALSE(registration.info.supports_transcription); + CHECK_FALSE(registration.info.supports_transcription_only); + CHECK_FALSE(registration.info.is_hybrid); + CHECK_FALSE(registration.info.supports_multimodal); + continue; + } CHECK(registration.info.is_text_generation_model); // No text arch is transcription-capable (SupportsTranscription default). CHECK_FALSE(registration.info.supports_transcription); @@ -552,7 +568,7 @@ TEST_CASE("Qwen3.5 SSM cache dtype accepts upstream torch aliases exactly") { TEST_CASE("hf_registry_coverage: every registration has an example config fixture") { // C++ fixture registry for the currently implemented subset. Keep this list // alias-for-alias with the central ordered table, mirroring HF_EXAMPLE_MODELS. - constexpr std::array kExampleConfigArchitectures{ + constexpr std::array kExampleConfigArchitectures{ "CohereForCausalLM", "DeepseekV2ForCausalLM", "DeepseekV4ForCausalLM", @@ -569,6 +585,7 @@ TEST_CASE("hf_registry_coverage: every registration has an example config fixtur "KimiLinearForCausalLM", "LagunaForCausalLM", "LlamaForCausalLM", + "LlamaModel", "MiniCPM3ForCausalLM", "MiniCPMForCausalLM", "MistralForCausalLM", @@ -663,7 +680,7 @@ TEST_CASE("raise_for_unsupported: subset default message and order match oracle" "'InternLM2ForCausalLM', 'InternLM3ForCausalLM', " "'KimiK3ForConditionalGeneration', 'KimiLinearForCausalLM', " "'LagunaForCausalLM', " - "'LlamaForCausalLM', " + "'LlamaForCausalLM', 'LlamaModel', " "'MiniCPM3ForCausalLM', 'MiniCPMForCausalLM', 'MistralForCausalLM', " "'OPTForCausalLM', 'Olmo2ForCausalLM', 'Olmo3ForCausalLM', " "'ParakeetForCTC', 'ParakeetForRNNT', 'ParakeetForTDT', " @@ -685,7 +702,7 @@ TEST_CASE("raise_for_unsupported: subset default message and order match oracle" "'InternLM2ForCausalLM', 'InternLM3ForCausalLM', " "'KimiK3ForConditionalGeneration', 'KimiLinearForCausalLM', " "'LagunaForCausalLM', " - "'LlamaForCausalLM', " + "'LlamaForCausalLM', 'LlamaModel', " "'MiniCPM3ForCausalLM', 'MiniCPMForCausalLM', 'MistralForCausalLM', " "'OPTForCausalLM', 'Olmo2ForCausalLM', 'Olmo3ForCausalLM', " "'ParakeetForCTC', 'ParakeetForRNNT', 'ParakeetForTDT', " diff --git a/tests/vllm/test_model_loader_gguf.cpp b/tests/vllm/test_model_loader_gguf.cpp index fe6cf0e97..27fd5934c 100644 --- a/tests/vllm/test_model_loader_gguf.cpp +++ b/tests/vllm/test_model_loader_gguf.cpp @@ -82,7 +82,7 @@ TEST_CASE("FromModelDir rejects an unknown dense architecture before loading") { "'InternLM2ForCausalLM', 'InternLM3ForCausalLM', " "'KimiK3ForConditionalGeneration', 'KimiLinearForCausalLM', " "'LagunaForCausalLM', " - "'LlamaForCausalLM', " + "'LlamaForCausalLM', 'LlamaModel', " "'MiniCPM3ForCausalLM', 'MiniCPMForCausalLM', 'MistralForCausalLM', " "'OPTForCausalLM', 'Olmo2ForCausalLM', 'Olmo3ForCausalLM', " "'ParakeetForCTC', 'ParakeetForRNNT', 'ParakeetForTDT', "