feat(vllm-cpp): wire the full engine config surface through engine_args

[localai-bot]

Rebased 2026-08-06. This branch had gone 144 commits stale. It is now a single
commit on current master. Three corrections to what is written below:

• The pin is master's 0757cac2, not the eec09bed / 6d4fb5e7 this PR
  originally carried. It is newer, contains them, and is the pin the MLX-on-metal
  default is gated against, so moving back would have quietly un-gated that.
• The mirrors are at ABI v10, not v9, so enable_jump_forward is exposed too.
• fix(vllm-cpp): mirror the engine's ABI v10 so the backend loads again #11386 has since landed the minimal ABI v10 unbreak for vllm.cpp not working (L4T build) #11379, so that part of
  this branch is no longer novel. What remains is the config surface. This keeps
  fix(vllm-cpp): mirror the engine's ABI v10 so the backend loads again #11386's make abi-check build guard alongside the env-gated handshake spec,
  and this branch's tri-state constants over fix(vllm-cpp): mirror the engine's ABI v10 so the backend loads again #11386's plainer field comments.

The intermediate commits were dropped rather than replayed: they stepped through
ABI v9, whose mirrors no longer make sense against master's v10 pin, so every one
of them would have needed conflict surgery to produce a tree nobody wants to bisect
to. The Darwin gnu-folding-constant fixes originally authored by
localai-org-maint-bot are folded into the single commit, which also clears the DCO
failure they were causing.

What

The vllm-cpp backend could configure four of the engine's knobs — block size, KV block count, max sequence length, max concurrent sequences — out of a config surface that is considerably larger. Speculative decoding, prefix caching, the chunked-prefill token budget, the scheduling policy and the external KV connector (LMCache) were reachable from vllm.cpp's own HTTP server and from nothing LocalAI could write in a model config.

Part of that gap was the C ABI itself, which carried strictly less than EngineParams does. That is fixed upstream in vllm.cpp ABI v9 (merged to main as eec09bed, which this PR pins). The rest was here: the backend parsed a flat options: list with five recognised keys and had no way to express a nested JSON document at all.

Configuration

Config now goes through engine_args: — the same map the vLLM and SGLang backends already take — with keys spelled as vLLM's own CLI flags, so a speculative_config or kv_transfer_config block written for vLLM works verbatim:

name: qwen36-dflash
backend: vllm-cpp
context_size: 8192
engine_args:
  max_num_seqs: 32
  max_num_batched_tokens: 8192
  enable_prefix_caching: true
  scheduling_policy: lpm
  speculative_config:
    method: dflash
    model: z-lab/Qwen3.6-27B-DFlash
    num_speculative_tokens: 4
  kv_transfer_config:
    kv_connector: LMCacheConnector
    kv_role: kv_both
    kv_connector_extra_config: {host: 127.0.0.1, port: 65432}

The options: list keeps working and now reads every key too, so no existing config breaks; engine_args wins where both set the same key.

All three speculative methods the engine supports are reachable: mtp (draft head inside the target checkpoint, safetensors only), dflash (separate draft checkpoint, model: required), and ngram (draft-free).

Two details worth reviewer attention

enable_prefix_caching: false maps to the ABI tri-state force-OFF (2), not 0. 0 means "let the model capability decide", and dense architectures default the cache on — collapsing the two would silently enable it against an explicit false.

cSamplingParams grows the ABI v8 logits-processor tail. LocalAI installs no processor, but the C side reads those fields off the pointer we hand it, so a Go struct that stopped at StructuredJSONObject (120 bytes vs C's 136) would have had the engine read past our allocation and call whatever sat there. Latent only because the old v5 ABI gate refused to load a v8 library.

Importer

A vllm-cpp import of a HuggingFace repo now probes config.json and writes speculative_config: {method: mtp} into the generated engine_args when the checkpoint declares an MTP head — the safetensors analogue of the llama-cpp importer's GGUF header probe. An explicit speculative_config is never overwritten, and every probe failure is non-fatal.

Two asymmetries versus the llama.cpp hook, both deliberate:

• DFlash draft repositories are refused with a warning rather than auto-configured. A drafter cannot serve alone, and the target/draft pairing is not derivable from either repo in isolation.
• The llama-cpp importer stops applying spec_type:draft-mtp when the chosen backend is vllm-cpp. Those are llama.cpp option keys vllm-cpp does not read, and vllm.cpp rejects MTP over a GGUF source outright because the mtp.* draft tensors do not survive GGUF conversion. This was a pre-existing bug — a GGUF import with the vllm-cpp preference emitted dead llama.cpp options.

Docs

docs/content/features/text-generation.md gains a vllm.cpp section covering the engine_args table, all three speculative methods, LMCache, and the legacy options: list. The backend previously had no documentation page at all.

Testing

• make lint clean.
• core/config, core/gallery/importers, backend/go/vllm-cpp all green. 9 new importer specs, 11 new engine_args specs, plus updated struct-offset assertions for the v9 / v8 layouts.
• Built a CPU-only libvllm.so at the pinned commit and confirmed purego binds all 19 symbols with vllm_abi_version() returning 9. That check is now a spec: set VLLM_CPP_LIBRARY and it runs without needing model weights, so a future pin bump with a stale struct mirror fails in CI rather than at a user's first load.
• Upstream: 31/31 capi test cases on the merged vllm.cpp tree.

Re-run after the merge to master (2026-08-05), now at ABI v10 and pin 0757cac2:

• make lint LINT_NEW_FROM=origin/master: 0 issues.
• backend/go/vllm-cpp, core/config, core/gallery/importers: green.
• make abi-check (from fix(vllm-cpp): mirror the engine's ABI v10 so the backend loads again #11386) reports v10 on both sides, and fails as intended
  against a header doctored to v11.
• End-to-end against a live engine, which the original body listed as not
  verified: full suite green in 1330s against a CPU libvllm.so reporting ABI
  v10 and Qwen_Qwen3.5-0.8B-Q4_K_M.gguf - load, blocking completion, streaming,
  and the chat / tool-call paths. Weights loaded, tokens generated.

Still not covered: a live generation with the new knobs actually engaged
(speculative decoding, LMCache, a non-fcfs scheduling policy). Those are verified
at the mapping and ABI-handshake level; the e2e above exercises the default path.

🤖 Generated with Claude Code

[localai-bot]

Refreshed and reconciled: merged up to master (144 commits), then merged #11386 in so the two independent ABI v10 moves became one. Kept this branch's tri-state constants plus #11386's make abi-check build guard. The remote copy of this branch had been rebased and carried three Darwin gnu-folding-constant fixes the local worktree lacked, so those were merged in rather than force-pushed over.

Pin is now master's 0757cac2 rather than the older 6d4fb5e7 this branch carried: it is newer, contains it, and the MLX-on-metal default is gated on being at or past 89c46aeb, so moving the pin backwards would have quietly un-gated that.

Post-merge verification: lint clean, backend/go/vllm-cpp + core/config + core/gallery/importers green, and the full e2e now passes in 1330s against a live CPU libvllm.so at ABI v10 with Qwen_Qwen3.5-0.8B-Q4_K_M.gguf (load, completion, streaming, chat/tool-calls) - that closes the "not verified end-to-end" caveat the original description carried. The CI backend build for -cpu-vllm-cpp also exercised the new guard: vllm-cpp: ABI v10 matches the pinned engine.

Still not covered: a live generation with the new knobs actually engaged (speculative decoding, LMCache, non-fcfs scheduling). Those are verified at the mapping and handshake level only.

[localai-bot]

On the red DCO check: it flags three commits, all pre-dating today's refresh and all authored by localai-org-maint-bot (20e537b, c7c6edf, a8fadc5, the Apple Clang folding-diagnostic fixes). Merge commits are ignored by the app, so nothing from the merge-forward is implicated.

Deliberately not fixing it by rebasing: that would rewrite commits already pushed to this shared branch. The plan is to squash-merge, which collapses the branch into a single commit authored and signed off by the merger and satisfies the DCO in substance. Repo settings already disallow merge commits, so squash is the path anyway.

One wart to ignore: commit 578e784 is an empty DCO remediation commit I added before the app told me which commits it was actually complaining about. It is inert under a squash merge. If anyone does decide to rebase this branch instead, drop that commit first, since DCO's own guidance says an empty commit breaks the rebase remedy.

[localai-bot]

Rebased onto current master as a single commit (4b4f9adf), replacing the merge-forward. Mergeable again and DCO now passes, since the three unsigned bot-authored Darwin commits are folded into the signed commit rather than sitting in history, which also makes the earlier squash-merge workaround unnecessary.

The intermediate commits were deliberately not replayed. They stepped through ABI v9 and master is now at v10 via #11386, so each replay conflicted and would have produced intermediate trees that do not build. The final tree is byte-identical to the merge-forward that was verified before the rebase: git diff against it is empty.

Re-verified on the rebased commit: make lint LINT_NEW_FROM=origin/master reports 0 issues, and backend/go/vllm-cpp, core/config and core/gallery/importers are green. The e2e result quoted in the description (1330s, live CPU libvllm.so at ABI v10) was run against this exact tree.

The previous tip is kept locally as backup/engine-args-premerge-20260806 if anyone needs the pre-rebase history.