fix(vllm-cpp): mirror the engine's ABI v10 so the backend loads again - #11386
Merged
Conversation
The Go bindings mirror vllm.h by hand and refuse a library whose vllm_abi_version differs from what they were written against. Two automated pin bumps (#11174, #11352) moved VLLM_CPP_VERSION onto engines declaring ABI v10 while govllmcpp.go still mirrored v5, so every vllm-cpp image built since then panics at startup on every platform: panic: vllm-cpp: ABI mismatch: library reports v10, backend built against v5 Grow both PODs to the v10 layout: vllm_model_params gains speculative_config, enable_prefix_caching, max_num_batched_tokens, scheduling_policy, kv_transfer_config and enable_jump_forward (88 bytes), vllm_sampling_params gains the v8 logits-processor pair (136 bytes). The offsets in the specs come from offsetof() against the pinned header. All of the new fields are inert when zeroed, so the engine behaves exactly as it did under v5; the backend sets none of them. Nothing cross-checked the two files, which is why a blind pin bump could ship a backend that cannot load. The library build now runs abi-check first: it compares VLLM_ABI_VERSION in the fetched header against abiVersion in govllmcpp.go and fails the build naming both, instead of leaving the mismatch for a user's runtime. Fixes #11379 Assisted-by: Claude:claude-fable-5 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
mudler
added a commit
that referenced
this pull request
Aug 5, 2026
Both branches moved the mirrors to ABI v10 independently: this one as part of exposing the v6-v10 config surface, #11386 as the minimal unbreak for issue #11379. The layouts agreed field for field, so the reconcile keeps this branch's tri-state constants and padding comments, plus #11386's Makefile abi-check target and its abiVersion spec. Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
mudler
approved these changes
Aug 6, 2026
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…da12-fallback Two conflicts, both from master moving under a stale branch: - backend/index.yaml: master rewrote the vllm-cpp gallery description (#11363). Keep that rewrite and re-append this branch's paragraph on the CUDA 13 / Blackwell-only constraint and the CPU fallback for CUDA 12 hosts. - backend/go/vllm-cpp/README.md: master gained the ABI pin/mirror note from #11386. Keep both sections; they cover different things. Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
The backend could configure four of the engine's knobs (block size, KV block
count, max sequence length, max concurrent sequences) out of a config surface
that is considerably larger. Speculative decoding, prefix caching, the
chunked-prefill token budget, the scheduling policy and the external KV
connector were reachable from vllm.cpp's own HTTP server and from nothing
LocalAI could write in a model config.
Config now goes through `engine_args:`, the same map the vLLM and SGLang
backends take, with keys spelled as vLLM's own CLI flags so a speculative_config
or kv_transfer_config block written for vLLM works verbatim. The legacy
`options:` list keeps working and reads every key too; engine_args wins where
both set one. Unknown keys are logged and ignored rather than fatal: the field
is shared with the other engines, so a config carrying their knobs must not take
the model down.
Two details worth knowing:
`enable_prefix_caching: false` maps to the ABI tri-state force-OFF (2), not 0.
0 means "let the model capability decide" and dense architectures default the
cache on, so collapsing the two would silently enable it against an explicit
false. enable_jump_forward (ABI v10) shares the encoding, deferring to
VT_ENABLE_JUMP_FORWARD instead of to the model.
The importer probes config.json on a vllm-cpp import and writes
speculative_config: {method: mtp} when the checkpoint declares an MTP head, the
safetensors analogue of the llama-cpp importer's GGUF probe. DFlash draft repos
are refused with a warning instead, since a drafter cannot serve alone and the
pairing is not derivable from either repo. The draft path is resolved against
LocalAI's model directory, because the engine only looks in a directory holding
config.json or in the HF cache and never downloads: the repo-id spelling the
vLLM docs teach used to die deep in the load with "draft checkpoint not found".
docs/content/features/text-generation.md gains a vllm.cpp section covering the
engine_args table, all three speculative methods, LMCache and the legacy list.
The backend had no documentation page before.
This replaces a branch that had gone stale behind master and carried its own
route to ABI v10, which #11386 has since landed in minimal form. Rebased onto
that as a single commit rather than replaying the intermediate steps, whose
ABI v9 mirrors no longer make sense against master's pin. The Darwin build
fixes for Apple Clang's gnu-folding-constant diagnostic on C++, Objective-C and
Objective-C++, originally authored by localai-org-maint-bot, are folded in here.
Verified: `make abi-check` agrees at v10; unit specs, core/config and
core/gallery/importers green; and the full e2e passes in 1330s against a CPU
libvllm.so reporting ABI v10 with Qwen_Qwen3.5-0.8B-Q4_K_M.gguf (load, blocking
completion, streaming, chat and tool calls).
Assisted-by: Claude:claude-fable-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…gs (#11159) The backend could configure four of the engine's knobs (block size, KV block count, max sequence length, max concurrent sequences) out of a config surface that is considerably larger. Speculative decoding, prefix caching, the chunked-prefill token budget, the scheduling policy and the external KV connector were reachable from vllm.cpp's own HTTP server and from nothing LocalAI could write in a model config. Config now goes through `engine_args:`, the same map the vLLM and SGLang backends take, with keys spelled as vLLM's own CLI flags so a speculative_config or kv_transfer_config block written for vLLM works verbatim. The legacy `options:` list keeps working and reads every key too; engine_args wins where both set one. Unknown keys are logged and ignored rather than fatal: the field is shared with the other engines, so a config carrying their knobs must not take the model down. Two details worth knowing: `enable_prefix_caching: false` maps to the ABI tri-state force-OFF (2), not 0. 0 means "let the model capability decide" and dense architectures default the cache on, so collapsing the two would silently enable it against an explicit false. enable_jump_forward (ABI v10) shares the encoding, deferring to VT_ENABLE_JUMP_FORWARD instead of to the model. The importer probes config.json on a vllm-cpp import and writes speculative_config: {method: mtp} when the checkpoint declares an MTP head, the safetensors analogue of the llama-cpp importer's GGUF probe. DFlash draft repos are refused with a warning instead, since a drafter cannot serve alone and the pairing is not derivable from either repo. The draft path is resolved against LocalAI's model directory, because the engine only looks in a directory holding config.json or in the HF cache and never downloads: the repo-id spelling the vLLM docs teach used to die deep in the load with "draft checkpoint not found". docs/content/features/text-generation.md gains a vllm.cpp section covering the engine_args table, all three speculative methods, LMCache and the legacy list. The backend had no documentation page before. This replaces a branch that had gone stale behind master and carried its own route to ABI v10, which #11386 has since landed in minimal form. Rebased onto that as a single commit rather than replaying the intermediate steps, whose ABI v9 mirrors no longer make sense against master's pin. The Darwin build fixes for Apple Clang's gnu-folding-constant diagnostic on C++, Objective-C and Objective-C++, originally authored by localai-org-maint-bot, are folded in here. Verified: `make abi-check` agrees at v10; unit specs, core/config and core/gallery/importers green; and the full e2e passes in 1330s against a CPU libvllm.so reporting ABI v10 with Qwen_Qwen3.5-0.8B-Q4_K_M.gguf (load, blocking completion, streaming, chat and tool calls). Assisted-by: Claude:claude-fable-5 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #11379.
The bug is not L4T-specific
Every
vllm-cppimage built since #11174 refuses to start, on every platform:govllmcpp.gohand-mirrorsvllm.handregisterLibrejects any library whosevllm_abi_versiondiffers from what the mirrors were written against. Twoautomated pin bumps (#11174, then #11352 into v4.8.0) moved
VLLM_CPP_VERSIONonto engines declaring ABI v10 while the bindings still declared v5. The
bindings had not been touched since #11100. The reporter happened to hit it on
an L4T image first.
The fix
v6 through v10 grew both PODs by appending only, so the mirrors just have to
grow with them:
vllm_model_params(48 -> 88 bytes):speculative_config(v6),enable_prefix_caching(v7),max_num_batched_tokens/scheduling_policy/kv_transfer_config(v9),enable_jump_forward(v10).vllm_sampling_params(120 -> 136 bytes): the v8logits_processor/logits_processor_user_datapair.Every one of them is inert when zeroed and the backend sets none of them, so the
engine behaves exactly as it did under v5. This restores the backend, it does not
change inference. Exposing the new knobs as backend options is a separate
follow-up.
The offsets asserted in
vllmcpp_test.goare not eyeballed: they come fromcompiling the pinned header and printing
offsetoffor each field.Keeping it from happening again
Nothing cross-checked the pin against the bindings. The bump PRs did build the
backend image (
backend_pr.ymlruns the path-filtered matrix on pull requests),but a stale mirror compiles perfectly well: the only thing that ever compared the
two was
registerLib, at runtime, on the user's machine. A green build meantnothing here.
The library build now runs
make abi-checkfirst, comparingVLLM_ABI_VERSIONin the fetched header against
abiVersioningovllmcpp.goand failing with bothversions named. That moves the comparison to where the image is produced, so a
future blind bump turns its own PR red.
The other backends using this hand-mirrored purego pattern were checked too. Only
trellis2cppandmagpie-tts-cppenforce a hard ABI equality, and both currentlymatch their pins (11 vs 11, 1 vs 1); the rest probe symbols and degrade
gracefully. Wiring the same guard into those two is worth a follow-up.
Verification
of the suite green) and pass after.
make abi-checkpasses on the matching pin, fails on a simulated v11 header.make lint LINT_NEW_FROM=origin/master: 0 issues.libvllm.so(CPU build) andQwen_Qwen3.5-0.8B-Q4_K_M.gguf: load, blocking completion, streaming, and thechat / tool-call paths. That same run died at
registerLibbefore the fix.abi-checkhas since run inside a real CI backend image build (on feat(vllm-cpp): wire the full engine config surface through engine_args #11159,which carries this commit):
vllm-cpp: ABI v10 matches the pinned engine.The
tests-linuxfailure on the first run of this PR is unrelated: HuggingFaceanswered the runner with HTTP 429, taking down the
pkg/huggingface-apiintegration specs and the importer specs that call HF. This PR touches only
backend/go/vllm-cpp. Re-run requested.Related
#11159 exposes these engine knobs through
engine_args:and already contains thisfix (the two branches moved to v10 independently and have been reconciled). Land
this one first so it can be cherry-picked onto 4.8.x; #11159's diff then narrows to
the config surface.