diff --git a/docs/customize/model-providers/more/qvac.mdx b/docs/customize/model-providers/more/qvac.mdx new file mode 100644 index 00000000000..0af5670bb0d --- /dev/null +++ b/docs/customize/model-providers/more/qvac.mdx @@ -0,0 +1,87 @@ +--- +title: "QVAC" +description: "Configure QVAC's local-first, OpenAI-compatible server with Continue for chat, edit, and agent workflows that run entirely on your machine" +--- + + + Get started with [QVAC](https://qvac.tether.io) - an open-source runtime for local-first, peer-to-peer AI + + +## Overview + +QVAC runs models locally and exposes an OpenAI-compatible HTTP server, so you can point Continue at it using the built-in `openai` provider. Everything stays on your machine. + +## Installation + +Install the CLI and start the OpenAI-compatible server: + +```shell +npm i -g @qvac/cli +qvac serve openai +``` + +The server listens on port `11434` by default. Use `--port` to change it. + +Model aliases are defined in a `qvac.config.json` file. The `model` you reference from Continue must match a serve alias: + +```json title="qvac.config.json" +{ + "serve": { + "models": { + "gpt-oss-20b": { + "model": "GPT_OSS_20B_INST_Q4_K_M", + "preload": true, + "config": { + "ctx_size": 32768 + } + } + } + } +} +``` + +## Configuration + +Use the `openai` provider and point `apiBase` at the QVAC server. The `model` is the alias from `qvac.config.json`. QVAC does not validate the API key, but the field is typically required, so any non-empty string works. + + + + ```yaml title="config.yaml" + name: My Config + version: 0.0.1 + schema: v1 + + models: + - name: QVAC gpt-oss-20b + provider: openai + model: gpt-oss-20b + apiKey: qvac + apiBase: http://127.0.0.1:11434/v1 + roles: + - chat + - edit + - apply + ``` + + + ```json title="config.json" + { + "models": [ + { + "title": "QVAC gpt-oss-20b", + "provider": "openai", + "model": "gpt-oss-20b", + "apiKey": "qvac", + "apiBase": "http://127.0.0.1:11434/v1" + } + ] + } + ``` + + + +## Recommendations + +- **Set `ctx_size` explicitly.** QVAC's LLM context size defaults to `1024` tokens, which is too small for most chat and agent prompts. Set it in `qvac.config.json` (e.g. `32768`) as shown above. +- **Disable reasoning for reasoning-tuned models.** For models such as Qwen3.5, set `reasoning_budget: 0` in the model `config` to keep responses suited to Continue's chat and edit flows. +- **Pick a capable model for tools and autocomplete.** Local tool-calling and autocomplete quality is bounded by the model you run. A capable agent-tuned model such as `gpt-oss-20b` is a good default for chat and agent use. diff --git a/docs/customize/model-providers/overview.mdx b/docs/customize/model-providers/overview.mdx index 7ba030dcb4d..388b9517242 100644 --- a/docs/customize/model-providers/overview.mdx +++ b/docs/customize/model-providers/overview.mdx @@ -49,6 +49,7 @@ Beyond the top-level providers, Continue supports many other options: | [llama.cpp](/customize/model-providers/more/llamacpp) | Optimized C++ implementation for running LLMs | | [LlamaStack](/customize/model-providers/more/llamastack) | Stack for running Llama models locally | | [llamafile](/customize/model-providers/more/llamafile) | Self-contained executable model files | +| [QVAC](/customize/model-providers/more/qvac) | Local-first, OpenAI-compatible runtime | ### Enterprise Solutions diff --git a/docs/docs.json b/docs/docs.json index b7a1d83f13a..e95e9c21445 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -117,6 +117,7 @@ "customize/model-providers/more/moonshot", "customize/model-providers/more/nous", "customize/model-providers/more/nvidia", + "customize/model-providers/more/qvac", "customize/model-providers/more/tensorix", "customize/model-providers/more/together", "customize/model-providers/more/xAI",