From 257a28206f964dfd867dd23dbb0baf90e82b3b26 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 4 Aug 2026 22:29:54 +0000 Subject: [PATCH 1/6] docs(blog): final figures for the 4.8 post, and the MLX provider The cycle closed at 374 PRs over twenty-one days, not the 321 over eighteen the post was written against. Corrects the summary, the opening line, the contributor count and the gallery total, and moves the date to the day the release is cut. Adds the MLX GEMM provider (#11137), which merged after the post was written and is the one number an Apple Silicon reader wants: 1.54x to 2.19x on an M4 with time to first token roughly halving, both arms toggled on one binary. The +/-10% caveat travels with the table rather than being left in the PR. Two lines edited against the no-ai-slop skill while I was in the file, the same pass #11324 ran over the engines post: - The opener balanced two clauses across a colon and closed on "without lying to you", which is the built-to-be-quoted shape readers picked out of the HN thread. It is a flat statement now. - "This is a new modality rather than a new backend under an existing one" is a binary contrast that says nothing the next clause does not. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] --- .../blog/what-landed-in-localai-4-8.md | 25 +++++++++++++++---- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/website/content/blog/what-landed-in-localai-4-8.md b/website/content/blog/what-landed-in-localai-4-8.md index 86169ca41405..a1c8f74d657e 100644 --- a/website/content/blog/what-landed-in-localai-4-8.md +++ b/website/content/blog/what-landed-in-localai-4-8.md @@ -1,14 +1,14 @@ --- title: "What landed in LocalAI 4.8" -date: 2026-08-01 +date: 2026-08-04 author: "Ettore Di Giacinto" category: "Release" tags: ["release", "vllm.cpp", "audio.cpp", "3d", "gallery", "distributed", "performance"] -summary: "A new inference engine, 3D generation, one backend that serves six audio endpoints, and a web interface 3.48x lighter. 321 pull requests in eighteen days." +summary: "A new inference engine, 3D generation, one backend that serves six audio endpoints, and a web interface 3.48x lighter. 374 pull requests in twenty-one days." extracss: ["blog.css"] --- -LocalAI 4.8.0 is out. It took eighteen days and 321 merged pull requests, and it pulls in two directions at once: three new things LocalAI can do that it could not do before, and a long list of places where it now does the old things without lying to you. +LocalAI 4.8.0 is out, after twenty-one days and 374 merged pull requests. There are three new things LocalAI can do, and a lot of repair work on things it already did. The full notes list everything. This post covers the parts that change what you do day to day, with the pull request numbers so you can read the diffs. @@ -77,6 +77,21 @@ The CPU path is verified end to end against `Qwen3.5-2B-UD-Q8_K_XL.gguf` with th > The GPU images build and ship, but their runtime behavior has not been through the same e2e gate yet. This is a first release of a young engine: no throughput comparison against upstream vLLM is claimed here, and `llama-cpp` remains the default recommendation for general use. Try it, and please report what breaks. +On Apple Silicon the image now ships vllm.cpp's MLX GEMM provider ([#11137](https://github.com/mudler/LocalAI/pull/11137)). Upstream keeps it off by default because it adds about 124 MB, so we measured before turning it on. Qwen3-1.7B-bf16 on an M4, p=512 g=128, both arms toggled on one binary so a build difference cannot explain the gap: + +
+ + + + + + + +
BatchMLX tok/snative tok/sspeedupMLX TTFTnative TTFT
15.793.081.88x3.32 s7.68 s
415.7510.241.54x9.63 s18.77 s
1638.6517.692.19x18.33 s54.48 s
+
+ +Two reps, with rep spread reaching 9.4%, so treat the multipliers as +/-10%. Time to first token roughly halves across the range. +
vllm.cpp serving a GGUF checkpoint with no Python in the process.
@@ -84,7 +99,7 @@ The CPU path is verified end to end against `Qwen3.5-2B-UD-Q8_K_XL.gguf` with th ## LocalAI generates 3D models now -This is a new modality rather than a new backend under an existing one, so it goes through the whole stack: a `Generate3D` RPC in `backend.proto`, a `FLAG_3D` capability so the loader knows which backends can serve it, and `POST /v1/3d/generations`. +3D generation is a new modality, so it had to be wired through the whole stack: a `Generate3D` RPC in `backend.proto`, a `FLAG_3D` capability so the loader knows which backends can serve it, and `POST /v1/3d/generations`. The first engine behind it is `trellis2cpp`, an image-to-3D backend over TRELLIS.2. You give it an image, you get a GLB back. The web UI has a page for it with a native GLB viewer, so you can turn the result around in the browser instead of downloading it to find out whether it worked, history kept in IndexedDB so a reload does not lose your generations, and previewable print remeshing for output you actually intend to send to a printer ([#10979](https://github.com/mudler/LocalAI/pull/10979)). @@ -179,6 +194,6 @@ Valkey Search joins the vector store options as the `valkey-store` backend ([#11 This is also the release where localai.io split in two: the project site at the root, and the documentation under `/docs/`. Every URL that was published before still resolves, through 214 generated redirect stubs, because GitHub Pages has no server-side rewrites to do it properly ([#11243](https://github.com/mudler/LocalAI/pull/11243)). -Twenty-four people contributed to this release, eleven of them for the first time. The gallery went from 1,221 entries to 1,505. +Twenty-five people contributed to this release, eleven of them for the first time. The gallery went from 1,221 entries to 1,515. To upgrade, pull `localai/localai:latest` or re-run the install script. The [full changelog](https://github.com/mudler/LocalAI/compare/v4.7.1...v4.8.0) has everything this post left out. From cb522a2b310c86662830af0130e6b410c7a1d1b7 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 4 Aug 2026 22:48:44 +0000 Subject: [PATCH 2/6] docs(blog): call vllm.cpp alpha, and finish the no-ai-slop pass vllm.cpp is not a released backend and the post read like it was. The old wording buried the caveat in a block quote at the end of the section and still said "first release of a young engine". It now says plainly, before the caveat can be skipped, that these are alpha development builds, that shipping them in 4.8 is about letting people try the thing rather than recommending it, and that llama-cpp stays the default. Also completes the no-ai-slop pass I had only half run. Counting the lines built to be quoted, headings and section endings included, the post is in reasonable shape: long flat informational stretches, tables followed by a plain finding, headings that are labels rather than epigram-verdicts. Three patterns survived, each one an item in eval.md: - "and inverts that:" set the usual shape against ours across a colon. The sentence works without the frame. - "Two things were conflated there: a signal, which needs one line, and the detail, which needs somewhere to put it" is a role-assignment pair. Says what happens instead. - "The maturity statement from the release notes is worth repeating in full" is throat-clearing in front of a quote, and the quote is gone. Left the rest alone. Minimum effective edit, not a rewrite. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] --- website/content/blog/what-landed-in-localai-4-8.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/content/blog/what-landed-in-localai-4-8.md b/website/content/blog/what-landed-in-localai-4-8.md index a1c8f74d657e..1182eb1e0bfa 100644 --- a/website/content/blog/what-landed-in-localai-4-8.md +++ b/website/content/blog/what-landed-in-localai-4-8.md @@ -73,9 +73,9 @@ options: - max_num_seqs:16 # also: block_size:, num_blocks: ``` -The CPU path is verified end to end against `Qwen3.5-2B-UD-Q8_K_XL.gguf` with the full Ginkgo suite, covering blocking and streaming byte-parity, greedy determinism, stop words, GBNF-constrained generation, concurrent streams, reasoning split and both `required` and `auto` tool calls. The maturity statement from the release notes is worth repeating in full: +**Treat these as alpha development builds, not a released backend.** vllm.cpp is early, and shipping it in 4.8 is about getting it in front of people who want to try it, not about recommending it for anything you care about. `llama-cpp` stays the default for real use. -> The GPU images build and ship, but their runtime behavior has not been through the same e2e gate yet. This is a first release of a young engine: no throughput comparison against upstream vLLM is claimed here, and `llama-cpp` remains the default recommendation for general use. Try it, and please report what breaks. +The CPU path is verified end to end against `Qwen3.5-2B-UD-Q8_K_XL.gguf` with the full Ginkgo suite, covering blocking and streaming byte-parity, greedy determinism, stop words, GBNF-constrained generation, concurrent streams, reasoning split and both `required` and `auto` tool calls. The GPU images build and ship, but their runtime behavior has not been through that gate. No throughput comparison against upstream vLLM is claimed. Expect rough edges, and please report what breaks. On Apple Silicon the image now ships vllm.cpp's MLX GEMM provider ([#11137](https://github.com/mudler/LocalAI/pull/11137)). Upstream keeps it off by default because it adds about 124 MB, so we measured before turning it on. Qwen3-1.7B-bf16 on an M4, p=512 g=128, both arms toggled on one binary so a build difference cannot explain the gap: @@ -110,7 +110,7 @@ The first engine behind it is `trellis2cpp`, an image-to-3D backend over TRELLIS ## One backend, six audio endpoints -The usual shape for audio is one backend per model family, which means a process per capability and a config file for each. `audio-cpp` wraps [audio.cpp](https://github.com/0xShug0/audio.cpp), a multi-family ggml audio engine, and inverts that: one backend process serves several unrelated families through a single runtime vocabulary, and works out which family a checkpoint belongs to from the GGUF's own `audiocpp.model_spec.family` metadata key. There is nothing backend-specific to write in the model config. +The usual shape for audio is one backend per model family, which means a process per capability and a config file for each. `audio-cpp` wraps [audio.cpp](https://github.com/0xShug0/audio.cpp), a multi-family ggml audio engine. One backend process serves several unrelated families through a single runtime vocabulary, and works out which family a checkpoint belongs to from the GGUF's own `audiocpp.model_spec.family` metadata key. There is nothing backend-specific to write in the model config.
@@ -145,7 +145,7 @@ The `bonsai` backend serves the 1-bit (Q1_0) and ternary (Q2_0) Bonsai quantizat ## The operations bar became a page -The old operations bar rendered one row per in-flight operation above every page. Queue four model installs and a backend and it took most of the viewport, on every route, until the last one finished. Two things were conflated there: a global "something is happening" signal, which needs one line, and the detail of what is happening, which needs somewhere to put it. +The old operations bar rendered one row per in-flight operation above every page. Queue four model installs and a backend and it took most of the viewport, on every route, until the last one finished. It was doing two jobs at once. A global "something is happening" signal only needs one line, and the detail of what is happening needs a page of its own. The strip is now one line, permanently, showing a failure first and otherwise the least-advanced running operation, with a `+N more` pill. Its `✕` hides the strip and no longer cancels anything. That is a deliberate behavior change worth knowing about before you click it out of habit: the same glyph used to cancel a 17 GB download in one row and dismiss a message in the next. Cancelling moved to the new page, behind a button that says so. From 82807a1cc4479bed3d339fa5900075be03c77f55 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 4 Aug 2026 22:53:08 +0000 Subject: [PATCH 3/6] docs(blog): present vllm.cpp as a community project, with its own numbers The post described vllm.cpp as "a from-scratch port of vLLM, written and maintained by the LocalAI team". Two things wrong with that. It is a community project, and it has stopped being only a port: it loads GGUF, runs on CPU, Metal and Vulkan, ships speculative decoding and KV offload, and its benchmark page measures against llama.cpp, MLX-LM and DwarfStar as well as vLLM, because those are the engines it competes with on that hardware. vLLM's role is now stated for what it is, the reference implementation. Correctness is checked against it and the scoreboard is kept against it. Also flags that the name will probably change, since it is drifting far enough that vllm.cpp will eventually mislead. Adds real numbers from the project's own docs/BENCHMARKS.md rather than adjectives: 1.045x vLLM at concurrency 1 on Qwen3.6-27B NVFP4 with token-for-token identical output, 1.010x and 1.013x at c16 and c32 on the 35B MoE and behind below that, prefill 1.18x over llama.cpp on CPU aarch64, 97.6% of MLX-LM warm total on an M4. Upstream's own caution travels with them: it treats c2 through c32 as ties because its noise band is 0.5% and those margins are 0.7% to 1.7%. Every figure was checked against ~/_git/vllm.cpp/docs/BENCHMARKS.md rather than restated from memory. The heading is marked alpha to match the section body. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] --- .../blog/what-landed-in-localai-4-8.md | 22 +++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/website/content/blog/what-landed-in-localai-4-8.md b/website/content/blog/what-landed-in-localai-4-8.md index 1182eb1e0bfa..f5ab13e4a97e 100644 --- a/website/content/blog/what-landed-in-localai-4-8.md +++ b/website/content/blog/what-landed-in-localai-4-8.md @@ -55,12 +55,30 @@ Every surface can override the choice: `variant` on `POST /models/apply`, `local One gap worth knowing about: in distributed mode `InstallModel` resolves against the frontend rather than the worker that will serve the model, so a cluster with a small frontend and large workers selects conservatively. PRs [#10943](https://github.com/mudler/LocalAI/pull/10943), [#10983](https://github.com/mudler/LocalAI/pull/10983), [#10992](https://github.com/mudler/LocalAI/pull/10992), [#11027](https://github.com/mudler/LocalAI/pull/11027) and [#11139](https://github.com/mudler/LocalAI/pull/11139). -## A new engine: vllm.cpp +## A new engine: vllm.cpp (alpha) -[vllm.cpp](https://github.com/mudler/vllm.cpp) is a from-scratch C++20 port of vLLM, written and maintained by the LocalAI team under Apache-2.0, and it ships here as the `vllm-cpp` backend ([#11100](https://github.com/mudler/LocalAI/pull/11100)). It mirrors vLLM's V1 architecture, so paged KV cache, continuous batching, prefix caching, scheduler and sampler, on a portable tensor runtime with no Python, no PyTorch and no ggml at inference. It loads Hugging Face safetensors and GGUF, enforces structured output inside the engine (JSON schema, regex, choice, GBNF), and builds for CPU amd64 and arm64, CUDA 12 and 13 including Blackwell, L4T for GB10, Vulkan and Darwin Metal. +[vllm.cpp](https://github.com/mudler/vllm.cpp) is a community project, Apache-2.0, that began as a C++20 port of vLLM. It ships here as the `vllm-cpp` backend ([#11100](https://github.com/mudler/LocalAI/pull/11100)). It implements vLLM's V1 architecture, so paged KV cache, continuous batching, prefix caching, scheduler and sampler, on a portable tensor runtime with no Python, no PyTorch and no ggml at inference. vLLM stays its reference implementation: correctness is checked by comparing output against it, and the benchmark scoreboard is kept against it. + +It has grown features vLLM does not have, which is most of the reason the port exists. It loads GGUF as well as safetensors, runs on CPU, Apple Metal and Vulkan alongside CUDA 12 and 13 and L4T for GB10, and ships speculative decoding and KV offload. Its benchmark page now measures against llama.cpp, MLX-LM and DwarfStar as well as vLLM, because on that hardware those are the engines it competes with. The name will probably change at some point: it is drifting far enough that vllm.cpp will eventually mislead. Tool calling is at llama.cpp parity by construction, because chat deliberately reuses the same autoparser path: full minja chat templates, `tool_choice: auto` lowered to a lazy structural-tag decode constraint, 30 tool dialects, 7 reasoning parsers, and streamed `ChatDelta` and `ToolCallDelta`. +Numbers from the project's own [scoreboard](https://github.com/mudler/vllm.cpp/blob/master/docs/BENCHMARKS.md), which calls ties ties and losses losses. Above 1.0 means vllm.cpp is ahead: + +
+
+ + + + + + + +
ReferenceWorkloadResult
vLLMQwen3.6-27B NVFP4, GB101.045x at concurrency 1, 1.007x to 1.017x from c2 to c32, output token-for-token identical
vLLMQwen3.6-35B-A3B NVFP4, GB101.010x at c16 and 1.013x at c32, behind from c1 to c8 (0.817x at c1)
llama.cppQwen3.5-2B GGUF, CPU aarch64prefill 1.18x, decode a tie, memory parity
MLX-LMQwen3-0.6B, Apple M497.6% of warm total, prefill ahead
+
+ +The upstream page is careful about its own noise: on the 27B grid the run-to-run spread is 0.5% and c2 through c32 land between 0.7% and 1.7%, so it calls those five ties rather than wins. The concurrency-1 result is the one it stands behind. + Configuration is a normal backend install: ```yaml From 340b2b53c61ba3e4f91234e1e962d70c8ea71560 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 4 Aug 2026 23:03:34 +0000 Subject: [PATCH 4/6] docs(blog): say who maintains vllm.cpp, and add the DeepSeek Flash result Two corrections to the previous commit. "A community project" says nothing and was not quite true either. The LocalAI team maintains vllm.cpp. Community-first is the intent, not a description, so it now says that and says what backs it: its own repository, its own docs, benchmark record and issue tracker, and it runs without LocalAI anywhere in the picture. Adds the DeepSeek-V4-Flash result, which makes the divergence point better than any of the prose around it. That model does not run on vLLM on a single GB10: every vLLM-loadable checkpoint is 156 GB or more against a 119 GiB unified pool, and the only quant that fits is an extreme-low-bit GGUF that vLLM cannot load. vllm.cpp reads GGUF and runs it at 16.28 tok/s against ds4's 16.33, a parity result. Also notes MTP speculative decoding, token-identical to vLLM's and about 4% faster at concurrency 1. Both figures checked against ~/_git/vllm.cpp/docs/BENCHMARKS.md. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] --- website/content/blog/what-landed-in-localai-4-8.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/website/content/blog/what-landed-in-localai-4-8.md b/website/content/blog/what-landed-in-localai-4-8.md index f5ab13e4a97e..1c08d8c3ecf1 100644 --- a/website/content/blog/what-landed-in-localai-4-8.md +++ b/website/content/blog/what-landed-in-localai-4-8.md @@ -57,7 +57,7 @@ One gap worth knowing about: in distributed mode `InstallModel` resolves against ## A new engine: vllm.cpp (alpha) -[vllm.cpp](https://github.com/mudler/vllm.cpp) is a community project, Apache-2.0, that began as a C++20 port of vLLM. It ships here as the `vllm-cpp` backend ([#11100](https://github.com/mudler/LocalAI/pull/11100)). It implements vLLM's V1 architecture, so paged KV cache, continuous batching, prefix caching, scheduler and sampler, on a portable tensor runtime with no Python, no PyTorch and no ggml at inference. vLLM stays its reference implementation: correctness is checked by comparing output against it, and the benchmark scoreboard is kept against it. +[vllm.cpp](https://github.com/mudler/vllm.cpp) is Apache-2.0 and maintained by the LocalAI team. We want it community-first rather than a LocalAI-only engine, so it lives in its own repository with its own docs, benchmark record and issue tracker, and it runs without LocalAI anywhere in the picture. It began as a C++20 port of vLLM. It ships here as the `vllm-cpp` backend ([#11100](https://github.com/mudler/LocalAI/pull/11100)). It implements vLLM's V1 architecture, so paged KV cache, continuous batching, prefix caching, scheduler and sampler, on a portable tensor runtime with no Python, no PyTorch and no ggml at inference. vLLM stays its reference implementation: correctness is checked by comparing output against it, and the benchmark scoreboard is kept against it. It has grown features vLLM does not have, which is most of the reason the port exists. It loads GGUF as well as safetensors, runs on CPU, Apple Metal and Vulkan alongside CUDA 12 and 13 and L4T for GB10, and ships speculative decoding and KV offload. Its benchmark page now measures against llama.cpp, MLX-LM and DwarfStar as well as vLLM, because on that hardware those are the engines it competes with. The name will probably change at some point: it is drifting far enough that vllm.cpp will eventually mislead. @@ -73,12 +73,15 @@ Numbers from the project's own [scoreboard](https://github.com/mudler/vllm.cpp/b vLLMQwen3.6-35B-A3B NVFP4, GB101.010x at c16 and 1.013x at c32, behind from c1 to c8 (0.817x at c1) llama.cppQwen3.5-2B GGUF, CPU aarch64prefill 1.18x, decode a tie, memory parity MLX-LMQwen3-0.6B, Apple M497.6% of warm total, prefill ahead +DwarfStar (ds4)DeepSeek-V4-Flash GGUF, GB1016.28 vs 16.33 tok/s decode, 0.997x, a parity result The upstream page is careful about its own noise: on the 27B grid the run-to-run spread is 0.5% and c2 through c32 land between 0.7% and 1.7%, so it calls those five ties rather than wins. The concurrency-1 result is the one it stands behind. +The DeepSeek-V4-Flash row is the one that shows how far this has moved from being a vLLM port. That model does not run on vLLM on a single GB10 at all: every vLLM-loadable checkpoint is 156 GB or more against a 119 GiB unified pool, and the only quant that fits is an extreme-low-bit GGUF, which vLLM cannot load. vllm.cpp reads GGUF, so it runs there anyway, at 16.28 tok/s against ds4's 16.33. Speculative decoding is in similar shape: MTP on Qwen3.6-27B NVFP4 is token-identical to vLLM's MTP and about 4% faster at concurrency 1. + Configuration is a normal backend install: ```yaml From 35cd4c9476b86d4fc0108071715dc41af52ffb6d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 4 Aug 2026 23:08:16 +0000 Subject: [PATCH 5/6] docs(blog): lead the DeepSeek result with what we run, not with what vLLM cannot The previous version opened on "that model does not run on vLLM on a single GB10 at all". Wrong emphasis twice over: it makes a strong negative claim about another project the headline, and it buries the actual result, which is that vllm.cpp runs DeepSeek-V4-Flash at roughly 2-bit (IQ2_XXS mixed, about 80 GB) on a single DGX Spark and decodes at 16.28 tok/s against DwarfStar's 16.33. The size constraint is still there, stated as the reason the quant is what it is rather than as a point about vLLM: at 300B+ total parameters even a 4-bit checkpoint is 156 GB or more, so a 2-bit GGUF is what fits the Spark's 119 GiB unified pool. The table row now names the quant and the box (IQ2_XXS, one DGX Spark) instead of just "GGUF, GB10", since that is the part a reader with a Spark wants. Figures unchanged and still from ~/_git/vllm.cpp/docs/BENCHMARKS.md. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] --- website/content/blog/what-landed-in-localai-4-8.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/website/content/blog/what-landed-in-localai-4-8.md b/website/content/blog/what-landed-in-localai-4-8.md index 1c08d8c3ecf1..ba1a3f86229f 100644 --- a/website/content/blog/what-landed-in-localai-4-8.md +++ b/website/content/blog/what-landed-in-localai-4-8.md @@ -73,14 +73,16 @@ Numbers from the project's own [scoreboard](https://github.com/mudler/vllm.cpp/b vLLMQwen3.6-35B-A3B NVFP4, GB101.010x at c16 and 1.013x at c32, behind from c1 to c8 (0.817x at c1) llama.cppQwen3.5-2B GGUF, CPU aarch64prefill 1.18x, decode a tie, memory parity MLX-LMQwen3-0.6B, Apple M497.6% of warm total, prefill ahead -DwarfStar (ds4)DeepSeek-V4-Flash GGUF, GB1016.28 vs 16.33 tok/s decode, 0.997x, a parity result +DwarfStar (ds4)DeepSeek-V4-Flash IQ2_XXS, one DGX Spark16.28 vs 16.33 tok/s decode, 0.997x, a parity result The upstream page is careful about its own noise: on the 27B grid the run-to-run spread is 0.5% and c2 through c32 land between 0.7% and 1.7%, so it calls those five ties rather than wins. The concurrency-1 result is the one it stands behind. -The DeepSeek-V4-Flash row is the one that shows how far this has moved from being a vLLM port. That model does not run on vLLM on a single GB10 at all: every vLLM-loadable checkpoint is 156 GB or more against a 119 GiB unified pool, and the only quant that fits is an extreme-low-bit GGUF, which vLLM cannot load. vllm.cpp reads GGUF, so it runs there anyway, at 16.28 tok/s against ds4's 16.33. Speculative decoding is in similar shape: MTP on Qwen3.6-27B NVFP4 is token-identical to vLLM's MTP and about 4% faster at concurrency 1. +The DeepSeek-V4-Flash row is the one that shows how far this has moved from being a vLLM port. It runs DeepSeek-V4-Flash at roughly 2-bit (IQ2_XXS mixed, about 80 GB) on a single DGX Spark, decoding at 16.28 tok/s against DwarfStar's 16.33. At 300B+ total parameters even a 4-bit checkpoint is 156 GB or more, so a 2-bit GGUF is what fits inside the Spark's 119 GiB unified pool, and reading GGUF is what makes that possible. + +Speculative decoding is in similar shape: MTP on Qwen3.6-27B NVFP4 is token-identical to vLLM's MTP and about 4% faster at concurrency 1. Configuration is a normal backend install: From a243ccff6a7367c89a028f2c41d2948c9502ea96 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 4 Aug 2026 23:11:06 +0000 Subject: [PATCH 6/6] docs(blog): say the new name is undecided "The name will probably change at some point" invited the obvious question. It now says the rename is expected and the name is still to be decided, which is the actual state. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] --- website/content/blog/what-landed-in-localai-4-8.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/content/blog/what-landed-in-localai-4-8.md b/website/content/blog/what-landed-in-localai-4-8.md index ba1a3f86229f..9a2a5fe3522a 100644 --- a/website/content/blog/what-landed-in-localai-4-8.md +++ b/website/content/blog/what-landed-in-localai-4-8.md @@ -59,7 +59,7 @@ One gap worth knowing about: in distributed mode `InstallModel` resolves against [vllm.cpp](https://github.com/mudler/vllm.cpp) is Apache-2.0 and maintained by the LocalAI team. We want it community-first rather than a LocalAI-only engine, so it lives in its own repository with its own docs, benchmark record and issue tracker, and it runs without LocalAI anywhere in the picture. It began as a C++20 port of vLLM. It ships here as the `vllm-cpp` backend ([#11100](https://github.com/mudler/LocalAI/pull/11100)). It implements vLLM's V1 architecture, so paged KV cache, continuous batching, prefix caching, scheduler and sampler, on a portable tensor runtime with no Python, no PyTorch and no ggml at inference. vLLM stays its reference implementation: correctness is checked by comparing output against it, and the benchmark scoreboard is kept against it. -It has grown features vLLM does not have, which is most of the reason the port exists. It loads GGUF as well as safetensors, runs on CPU, Apple Metal and Vulkan alongside CUDA 12 and 13 and L4T for GB10, and ships speculative decoding and KV offload. Its benchmark page now measures against llama.cpp, MLX-LM and DwarfStar as well as vLLM, because on that hardware those are the engines it competes with. The name will probably change at some point: it is drifting far enough that vllm.cpp will eventually mislead. +It has grown features vLLM does not have, which is most of the reason the port exists. It loads GGUF as well as safetensors, runs on CPU, Apple Metal and Vulkan alongside CUDA 12 and 13 and L4T for GB10, and ships speculative decoding and KV offload. Its benchmark page now measures against llama.cpp, MLX-LM and DwarfStar as well as vLLM, because on that hardware those are the engines it competes with. The project is expected to be renamed, with the new name still to be decided; it is drifting far enough that vllm.cpp will eventually mislead. Tool calling is at llama.cpp parity by construction, because chat deliberately reuses the same autoparser path: full minja chat templates, `tool_choice: auto` lowered to a lazy structural-tag decode constraint, 30 tool dialects, 7 reasoning parsers, and streamed `ChatDelta` and `ToolCallDelta`.