From b739e357ccab283c247188101f4cd2b66504002c Mon Sep 17 00:00:00 2001 From: Tom Beckenham <34339192+tombeckenham@users.noreply.github.com> Date: Thu, 6 Aug 2026 14:14:48 +1000 Subject: [PATCH 1/2] docs: rewrite all hand-written docs for action-first clarity Cut ~12k lines of preamble and marketing across 141 pages. Lead with "if you need X, do Y", numbered steps, and code-first guidance. Leave auto-generated reference docs alone. --- docs/adapters/acp-compatible.md | 159 +- docs/adapters/anthropic.md | 205 +-- docs/adapters/bedrock.md | 197 +-- docs/adapters/byteplus.md | 243 +-- docs/adapters/claude-code.md | 95 +- docs/adapters/codex.md | 97 +- docs/adapters/elevenlabs.md | 201 +-- docs/adapters/fal.md | 320 +--- docs/adapters/gemini.md | 484 ++---- docs/adapters/grok-build.md | 119 +- docs/adapters/grok.md | 225 +-- docs/adapters/groq.md | 154 +- docs/adapters/mistral.md | 221 +-- docs/adapters/ollama.md | 237 +-- docs/adapters/openai-compatible.md | 117 +- docs/adapters/openai.md | 395 ++--- docs/adapters/opencode.md | 90 +- docs/adapters/openrouter.md | 172 +-- docs/advanced/built-in-middleware.md | 134 +- docs/advanced/debug-logging.md | 88 +- docs/advanced/extend-adapter.md | 147 +- docs/advanced/locks.md | 101 +- docs/advanced/middleware.md | 492 +++--- docs/advanced/multimodal-content.md | 325 ++-- docs/advanced/otel.md | 115 +- docs/advanced/per-model-type-safety.md | 44 +- docs/advanced/runtime-adapter-switching.md | 98 +- docs/advanced/runtime-context.md | 76 +- docs/advanced/tree-shaking.md | 269 +--- docs/advanced/typed-options.md | 80 +- docs/api/ai-angular.md | 179 +-- docs/api/ai-client.md | 434 ++---- docs/api/ai-preact.md | 104 +- docs/api/ai-react.md | 141 +- docs/api/ai-solid.md | 109 +- docs/api/ai-svelte.md | 163 +- docs/api/ai-vue.md | 140 +- docs/api/ai.md | 344 ++--- docs/architecture/approval-flow-processing.md | 118 +- docs/chat/agentic-cycle.md | 75 +- docs/chat/connection-adapters.md | 283 ++-- docs/chat/streaming.md | 226 ++- docs/chat/structured-outputs.md | 16 +- docs/chat/thinking-content.md | 61 +- docs/code-mode/client-integration.md | 62 +- docs/code-mode/code-mode-isolates.md | 135 +- docs/code-mode/code-mode-with-skills.md | 307 ++-- docs/code-mode/code-mode.md | 199 +-- docs/code-mode/lazy-tools.md | 98 +- docs/community-adapters/cencori.md | 148 +- docs/community-adapters/cloudflare.md | 231 +-- docs/community-adapters/decart.md | 251 +-- docs/community-adapters/guide.md | 184 +-- docs/community-adapters/mynth.md | 204 +-- docs/community-adapters/soniox.md | 189 +-- docs/comparison/vercel-ai-sdk.md | 361 ++--- docs/config.json | 358 +++-- docs/getting-started/agent-skills.md | 95 +- docs/getting-started/devtools.md | 147 +- docs/getting-started/overview.md | 117 +- docs/getting-started/quick-start-angular.md | 53 +- .../quick-start-react-native.md | 170 +-- docs/getting-started/quick-start-server.md | 54 +- docs/getting-started/quick-start-svelte.md | 49 +- docs/getting-started/quick-start-vue.md | 49 +- docs/getting-started/quick-start.md | 50 +- docs/interrupts/generic.md | 61 +- docs/interrupts/migration.md | 99 +- docs/interrupts/multiple.md | 65 +- docs/interrupts/overview.md | 93 +- docs/interrupts/tool-approval.md | 99 +- docs/mcp/apps.md | 181 +-- docs/media/audio-generation.md | 98 +- docs/media/audio-recording.md | 154 +- docs/media/generation-hooks.md | 230 ++- docs/media/generations.md | 167 +- docs/media/image-generation.md | 477 ++---- docs/media/realtime-chat.md | 350 ++--- docs/media/text-to-speech.md | 321 +--- docs/media/transcription.md | 406 +---- docs/media/video-generation.md | 675 ++------- docs/memory/adapters.md | 173 +-- docs/memory/custom-adapter.md | 92 +- docs/memory/operating.md | 73 +- docs/memory/overview.md | 99 +- docs/memory/quickstart.md | 77 +- docs/migration/ag-ui-compliance.md | 244 +-- docs/migration/migration-from-vercel-ai.md | 1350 ++--------------- docs/migration/migration.md | 333 +--- .../sampling-options-to-model-options.md | 119 +- docs/persistence/build-a-sandbox-adapter.md | 159 +- docs/persistence/build-your-own-adapter.md | 95 +- .../build-your-own-chat-adapter.md | 141 +- .../build-your-own-generation-adapter.md | 122 +- docs/persistence/chat-persistence.md | 156 +- docs/persistence/client-persistence.md | 151 +- docs/persistence/controls.md | 80 +- docs/persistence/generation-persistence.md | 189 +-- docs/persistence/id-map.md | 173 +-- docs/persistence/internals.md | 299 ++-- docs/persistence/keep-generated-files.md | 292 +--- docs/persistence/migrations.md | 47 +- docs/persistence/overview.md | 110 +- docs/persistence/store-reference.md | 353 ++--- docs/protocol/custom-events.md | 118 +- docs/resumable-streams/advanced.md | 252 +-- docs/resumable-streams/custom-adapter.md | 163 +- docs/resumable-streams/overview.md | 75 +- docs/sandbox/cloudflare.md | 255 +--- docs/sandbox/durability.md | 88 +- docs/sandbox/durable-runs.md | 200 +-- docs/sandbox/events.md | 261 ++-- docs/sandbox/harnesses.md | 70 +- docs/sandbox/journal.md | 369 +---- docs/sandbox/lifecycle.md | 103 +- docs/sandbox/observability.md | 159 +- docs/sandbox/overview.md | 111 +- docs/sandbox/policy.md | 131 +- docs/sandbox/providers.md | 238 +-- docs/sandbox/provisioning.md | 158 +- docs/sandbox/quick-start.md | 88 +- docs/sandbox/reaping.md | 547 +------ docs/sandbox/takeover.md | 673 +------- docs/sandbox/tools.md | 89 +- docs/sandbox/workspace.md | 111 +- docs/structured-outputs/multi-turn.md | 92 +- docs/structured-outputs/one-shot.md | 145 +- docs/structured-outputs/overview.md | 128 +- docs/structured-outputs/streaming.md | 93 +- docs/structured-outputs/with-tools.md | 72 +- docs/tools/client-tools.md | 218 +-- docs/tools/lazy-tool-discovery.md | 118 +- docs/tools/mcp-codegen.md | 26 +- docs/tools/mcp-managed.md | 93 +- docs/tools/mcp-manual.md | 76 +- docs/tools/mcp.md | 256 +--- docs/tools/provider-skills.md | 106 +- docs/tools/provider-tools.md | 91 +- docs/tools/server-tools.md | 128 +- docs/tools/tool-approval.md | 130 +- docs/tools/tool-architecture.md | 226 +-- docs/tools/tools.md | 193 +-- 142 files changed, 7279 insertions(+), 19178 deletions(-) diff --git a/docs/adapters/acp-compatible.md b/docs/adapters/acp-compatible.md index 54c03fd4e..e748c6395 100644 --- a/docs/adapters/acp-compatible.md +++ b/docs/adapters/acp-compatible.md @@ -1,7 +1,7 @@ --- title: ACP-Compatible Harness id: acp-compatible-harness -description: "Plug any Agent Client Protocol (ACP) coding agent into a TanStack AI sandbox with one generic harness adapter — no dedicated package required." +description: "Plug any ACP coding agent into a TanStack AI sandbox — no dedicated package required." keywords: - tanstack ai - acp @@ -12,21 +12,24 @@ keywords: - custom adapter --- -Coding-agent CLIs that speak the [Agent Client Protocol](https://agentclientprotocol.com) (ACP) — `grok`, `gemini --acp`, and others — expose a long-lived JSON-RPC session you can drive from a sandbox. Instead of a dedicated package per agent, `acpCompatible` builds a `chat()` adapter for **any** ACP-compliant CLI: configure how to launch it once, select a model per call, and pass it into a sandbox. +If you need a coding agent that speaks [ACP](https://agentclientprotocol.com) but has no `@tanstack/ai-*` package → use `acpCompatible`. Prefer a dedicated harness ([Grok Build](./grok-build), etc.) when one exists. -It is the harness equivalent of the [OpenAI-Compatible adapter](./openai-compatible). Use it when your agent speaks ACP but has no `@tanstack/ai-*` package. If a dedicated harness adapter exists ([Grok Build](./grok-build), and others), prefer it — those carry curated per-model metadata and vendor-specific behavior. +## When do I need this? -## Installation +- Agent CLI speaks ACP (`grok`, `gemini --acp`, …) and has no first-class adapter +- You already run agents inside a [sandbox](../sandbox/overview) -`acpCompatible` ships in `@tanstack/ai-acp`. You drive it inside a sandbox, so install the sandbox package and a provider too: +## Install ```bash npm install @tanstack/ai-acp @tanstack/ai @tanstack/ai-sandbox @tanstack/ai-sandbox-docker ``` -## Basic Usage +## Do this -Configure the harness once with `acpCompatible({ name, command })`, then select a model per call. `command` builds the shell command that launches the agent's ACP server over **stdio** inside the sandbox: +1. Configure the harness once with `acpCompatible({ name, command })`. +2. Select a model per call. +3. Pass `withSandbox(...)` middleware. ```ts import { chat } from '@tanstack/ai' @@ -41,11 +44,10 @@ import { import { dockerSandbox } from '@tanstack/ai-sandbox-docker' import { messages } from './chat-context' -// Configure the "pi" agent harness once: const pi = acpCompatible({ name: 'pi', command: ({ model, harnessCwd }) => `pi --acp -m ${model} --cwd ${harnessCwd}`, - authMethodId: 'pi-api-key', // when the harness advertises an ACP auth method + authMethodId: 'pi-api-key', refusalMessage: 'Pi refused the request.', }) @@ -54,7 +56,7 @@ const sandbox = defineSandbox({ provider: dockerSandbox({ image: 'node:22' }), workspace: defineWorkspace({ source: githubRepo({ repo: 'owner/app' }), - setup: ['npm install -g pi-cli'], // install the agent CLI into the image + setup: ['npm install -g pi-cli'], secrets: createSecrets({ PI_API_KEY: process.env.PI_API_KEY ?? '' }), }), }) @@ -66,11 +68,9 @@ const stream = chat({ }) ``` -You get the full ACP flow for free: sandbox resolution, `chat()`-tool → MCP bridging, session resume, permission handling, abort, and AG-UI event translation. +You get sandbox resolution, `chat()`-tool → MCP bridging, session resume, permission handling, abort, and AG-UI event translation. -## One-Shot Usage - -For a single model, skip the harness-factory and build the adapter inline with `acpCompatibleText`: +### One-shot (single model) ```ts import { chat } from '@tanstack/ai' @@ -89,14 +89,9 @@ const stream = chat({ }) ``` -## Typed models & options +### Typed models and options -Like `openaiCompatible`, you can declare the harness's **models** and its -per-call **options** so the whole thing is type-checked. `models` constrains the -factory's argument; `modelOptions` is a type-only brand (`{} as { … }`, unused at -runtime) describing what `chat({ modelOptions })` accepts. Declared options are -merged with the base ACP options and handed to `command` / `openTransport` as -`ctx.modelOptions`, so you can turn them into CLI flags: +Declare `models` and a type-only `modelOptions` brand so calls type-check. Options merge with base ACP options and reach `command` / `openTransport` as `ctx.modelOptions`: ```ts import { acpCompatible } from '@tanstack/ai-acp' @@ -111,7 +106,7 @@ const pi = acpCompatible({ }) pi('pi-pro') // ok -// pi('pi-ultra') // type error — not in `models` +// pi('pi-ultra') // type error ``` ```ts @@ -123,40 +118,39 @@ import { messages } from './chat-context' const stream = chat({ adapter: pi('pi-pro'), - modelOptions: { reasoningEffort: 'high' }, // typed against the declared options + modelOptions: { reasoningEffort: 'high' }, messages, middleware: [withSandbox(sandbox)], }) ``` -The base options are always available on `modelOptions` regardless of what you -declare: `sessionId` (resume), `cwd`, `authMethodId`, and `permissionMode`. +Base options always available on `modelOptions`: `sessionId`, `cwd`, `authMethodId`, `permissionMode`. ## Configuration | Field | Purpose | | --- | --- | -| `name` (required) | Harness label, log prefix, and the `.session-id` CUSTOM event name. | -| `models` | The model ids this harness accepts — declaring them makes `harness('id')` type-safe (unknown ids are rejected). Omit to accept any string. | -| `modelOptions` | Type-only brand for the per-call options accepted via `chat({ modelOptions })`. Declare with `{} as { … }`; merged with the base options and exposed on `ctx.modelOptions` in `command` / `openTransport`. | -| `command` | Build the **stdio** launch command from `{ model, cwd, harnessCwd, sandbox, env, modelOptions, signal }`. Required unless `openTransport` is given. | -| `skillsDir` | The harness's skills directory (relative to the workspace root, e.g. `'.pi/skills'`) — its native convention, like Claude Code's `.claude/skills`. `withSandbox` workspace `gitSkill`s are linked here. Omit and gitSkills are left unlinked (warned). | -| `openTransport` | Open any `AcpSessionTransport` yourself (e.g. boot a `serve` process and connect over WebSocket). Overrides `command`. | -| `cwd` | Working directory inside the sandbox (default `/workspace`). | -| `env` | Extra environment variables for the harness process. | -| `authMethodId` | ACP auth method to select before the session starts. | -| `permissionMode` | `'default'` \| `'acceptEdits'` \| `'bypassPermissions'` (default). | -| `permissions` | `'headless'` (auto-resolve, default) or `'interactive'` (emit approval-requested events for `ask` prompts). | -| `onPermissionRequest` | Custom permission handler; overrides `permissions`/`permissionMode`. | -| `refusalMessage` | `RUN_ERROR` message when the harness refuses a request. | -| `planEventName` | Emit ACP `plan` updates as a CUSTOM event under this name. | -| `emitDiff` | Emit the post-run `git diff` of `cwd` as a `file.changed` CUSTOM event (off by default). | -| `onExtNotification` | Handle vendor `_x/…` JSON-RPC notifications. | -| `buildPrompt` | Override how chat history maps to the harness prompt. | - -## WebSocket and Custom Transports - -Some harnesses run an ACP server you reach over WebSocket rather than stdio (the `grok agent serve` pattern). Open the transport yourself with `openTransport` — it receives the same context and returns an `AcpSessionTransport`. Put all teardown in the returned transport's `dispose`: +| `name` (required) | Label, log prefix, and `.session-id` CUSTOM event name | +| `models` | Accepted model ids — type-safe factory args. Omit → any string | +| `modelOptions` | Type-only brand for `chat({ modelOptions })` (`{} as { … }`) | +| `command` | Build stdio launch cmd from `{ model, cwd, harnessCwd, sandbox, env, modelOptions, signal }`. Required unless `openTransport` | +| `skillsDir` | Skills dir relative to workspace (e.g. `'.pi/skills'`) for `gitSkill` links | +| `openTransport` | Custom `AcpSessionTransport` (e.g. WebSocket). Overrides `command` | +| `cwd` | Working dir in sandbox (default `/workspace`) | +| `env` | Extra env for the harness process | +| `authMethodId` | ACP auth method before session start | +| `permissionMode` | `'default'` \| `'acceptEdits'` \| `'bypassPermissions'` (default) | +| `permissions` | `'headless'` (default) or `'interactive'` | +| `onPermissionRequest` | Custom permission handler | +| `refusalMessage` | `RUN_ERROR` message on refuse | +| `planEventName` | Emit ACP `plan` as CUSTOM under this name | +| `emitDiff` | Post-run `git diff` as `file.changed` CUSTOM (off by default) | +| `onExtNotification` | Vendor `_x/…` JSON-RPC notifications | +| `buildPrompt` | Override history → harness prompt | + +## WebSocket / custom transport + +Some harnesses serve ACP over WebSocket. Open the transport yourself; put teardown in `dispose`: ```ts import { acpCompatible, startAcpServerInSandbox } from '@tanstack/ai-acp' @@ -188,7 +182,7 @@ const myAgent = acpCompatible({ ## Permissions -Inside a sandbox the sandbox itself is the security boundary, so the default `'headless'` strategy with `permissionMode: 'bypassPermissions'` lets the agent edit files and run commands without prompting. To surface tool approvals to a client instead, switch to `'interactive'`: +Sandbox is the security boundary. Default: `'headless'` + `permissionMode: 'bypassPermissions'`. For client approvals: ```ts import { acpCompatible } from '@tanstack/ai-acp' @@ -196,21 +190,23 @@ import { acpCompatible } from '@tanstack/ai-acp' const pi = acpCompatible({ name: 'pi', command: ({ model }) => `pi --acp -m ${model}`, - permissions: 'interactive', // emit approval-requested events for `ask` prompts - permissionMode: 'acceptEdits', // still auto-approve file edits + permissions: 'interactive', + permissionMode: 'acceptEdits', }) ``` -`chat()`-provided tools bridged into the agent are always auto-approved, regardless of mode. +`chat()`-bridged tools are always auto-approved. -## Session Resume +## Session resume -On every run the adapter emits the harness session id as a CUSTOM event named `.session-id` (e.g. `pi.session-id`). Thread that id back through `modelOptions.sessionId` on the next call and the harness resumes the session — only the trailing user message is sent, since the agent already holds the prior context: +1. Capture `.session-id` CUSTOM event (e.g. `pi.session-id`). +2. Pass it back as `modelOptions.sessionId`. +3. Send only the latest user message — harness holds prior context. ```ts import { chat, chatParamsFromRequest, toServerSentEventsResponse } from '@tanstack/ai' import { withSandbox } from '@tanstack/ai-sandbox' -import { pi } from './pi-harness' // the configured `acpCompatible(...)` factory +import { pi } from './pi-harness' import { sandbox } from './sandbox' export async function POST(request: Request) { @@ -233,54 +229,27 @@ export async function POST(request: Request) { ## Workspace skills -When you provision a [workspace](../sandbox/workspace) via `withSandbox`, -`acpCompatible` projects its skills into the harness — each kind lands where that -harness expects it: - -| Workspace input | How `acpCompatible` projects it | +| Workspace input | Projection | | --- | --- | -| `mcpSkill(name, config)` | Passed to the agent over **ACP natively** via `newSession`'s `mcpServers` (secrets/bearer headers resolved). No config file — that's the ACP advantage over file-based harnesses. | -| `gitSkill({ repo })` | Cloned during bootstrap, then linked into your declared [`skillsDir`](#configuration) (e.g. `.pi/skills`). Omit `skillsDir` and it's left unlinked (with a warning). | -| `fileSkill({ path, content })` | Written into the workspace root during bootstrap (provider-agnostic). | -| `instructions` | Written to `AGENTS.md` (and symlinks) during bootstrap. | -| `agentSkill(name)`, `plugins` | No generic ACP primitive — warned and skipped. Provide a `gitSkill` or MCP server instead. | +| `mcpSkill(name, config)` | ACP `newSession` `mcpServers` (secrets resolved) | +| `gitSkill({ repo })` | Cloned, linked into `skillsDir` | +| `fileSkill({ path, content })` | Written into workspace root | +| `instructions` | Written to `AGENTS.md` | +| `agentSkill` / `plugins` | No ACP primitive — warned and skipped | -`secrets` declared on the workspace are injected into the agent's environment at -create/resume (never persisted to snapshots), so the harness CLI picks them up -like any env var. +Workspace `secrets` inject into the agent env at create/resume (never snapshotted). ## Protocol coverage -`acpCompatible` implements the **client / orchestration** side of ACP — enough to -drive an agent through a full prompt turn, not the entire protocol surface. It is -a compliant *minimal* client: everything it doesn't implement is either -capability-gated (so advertising non-support is the spec-defined behavior) or a -rendering choice, never a violation. - -**Covered:** - -- `initialize` handshake — sends `clientInfo` + the protocol version, negotiates the version, advertises capabilities. -- `authenticate` (when the agent advertises auth methods), `session/new`, `session/load` (resume), `session/prompt`, `session/cancel`. -- `session/request_permission` with all four option kinds, mapped by [permission mode](#permissions). -- All streamed `session/update`s that carry turn output: `agent_message_chunk`, `agent_thought_chunk` (→ reasoning), `tool_call` / `tool_call_update`, `plan`. -- All five stop reasons (`end_turn`, `max_tokens`, `max_turn_requests`, `refusal`, `cancelled`). - -**Surfaced as `CUSTOM` stream events** (the AG-UI chat-event protocol has no -first-class event for non-text assistant output, so these ride on `CUSTOM`): - -- `.session-id` — the harness session id, for [resume](#session-resume). -- `.message-content` — non-text agent content (`image` / `audio` / `resource` / `resource_link` blocks). Its `value` is `{ content: }`. Non-text **tool** content (diffs, terminal, images) is preserved inside the `TOOL_CALL_RESULT` payload. -- the plan event, when you set `planEventName`. +**Covered:** `initialize`, `authenticate`, `session/new` / `load` / `prompt` / `cancel`, `session/request_permission`, streamed updates (`agent_message_chunk`, `agent_thought_chunk`, tool calls, `plan`), all five stop reasons. -**Not implemented (by design):** +**CUSTOM events:** `.session-id`, `.message-content` (non-text blocks), plan (if `planEventName`). -- `fs/read_text_file`, `fs/write_text_file`, `terminal/*` — advertised as unsupported. The agent runs inside the sandbox with direct filesystem and shell access, so it never delegates these back to the client. -- Sending **multimodal prompts** — prompts are sent as text. (Agent multimodal *output* is surfaced via `message-content` above.) -- Incremental `usage_update` (final turn usage is reported instead), `available_commands_update`, `current_mode_update`, and experimental features (elicitation, NES, providers, session modes/config). +**Not implemented (by design):** `fs/*`, `terminal/*` (agent has sandbox access), multimodal prompts (text only), incremental `usage_update`, experimental ACP features. -## Next Steps +## Next steps -- [Sandbox Overview](../sandbox/overview) — how harnesses run inside a sandbox -- [Grok Build Adapter](./grok-build) — a first-class ACP harness adapter -- [Sandbox Tools](../sandbox/tools) — bridge your app's tools into the agent -- [OpenAI-Compatible Adapter](./openai-compatible) — the same idea for model providers +- [Sandbox Overview](../sandbox/overview) +- [Grok Build Adapter](./grok-build) +- [Sandbox Tools](../sandbox/tools) +- [OpenAI-Compatible Adapter](./openai-compatible) diff --git a/docs/adapters/anthropic.md b/docs/adapters/anthropic.md index 17f154447..eeb1f0b05 100644 --- a/docs/adapters/anthropic.md +++ b/docs/adapters/anthropic.md @@ -2,7 +2,7 @@ title: Anthropic id: anthropic-adapter order: 2 -description: "Use Anthropic Claude models with TanStack AI — Claude Fable 5, Claude Sonnet 5, Claude Opus, and more via the @tanstack/ai-anthropic adapter." +description: "Claude models via @tanstack/ai-anthropic — chat, tools, thinking, summarization, provider tools." keywords: - tanstack ai - anthropic @@ -14,15 +14,19 @@ keywords: - llm --- -The Anthropic adapter provides access to Claude models, including Claude Fable 5, Claude Sonnet 5, Claude Opus 4.8, and more. +If you need Claude chat → install, set `ANTHROPIC_API_KEY`, call `anthropicText(model)`. -## Installation +## Install ```bash npm install @tanstack/ai-anthropic ``` -## Basic Usage +```bash +ANTHROPIC_API_KEY=sk-ant-... +``` + +## Do this ```typescript import { chat } from "@tanstack/ai"; @@ -34,14 +38,14 @@ const stream = chat({ }); ``` -## Basic Usage - Custom API Key +### Explicit API key ```typescript import { chat } from "@tanstack/ai"; import { createAnthropicChat } from "@tanstack/ai-anthropic"; const adapter = createAnthropicChat("claude-sonnet-4-6", process.env.ANTHROPIC_API_KEY!, { - // ... your config options + // baseURL, ... }); const stream = chat({ @@ -50,20 +54,7 @@ const stream = chat({ }); ``` -## Configuration - -```typescript -import { createAnthropicChat, type AnthropicTextConfig } from "@tanstack/ai-anthropic"; - -const config: Omit = { - baseURL: "https://api.anthropic.com", // Optional, for custom endpoints -}; - -const adapter = createAnthropicChat("claude-sonnet-4-6", process.env.ANTHROPIC_API_KEY!, config); -``` - - -## Example: Chat Completion +### Server endpoint ```typescript import { chat, toServerSentEventsResponse } from "@tanstack/ai"; @@ -81,7 +72,7 @@ export async function POST(request: Request) { } ``` -## Example: With Tools +### With tools ```typescript import { chat, toServerSentEventsResponse, toolDefinition } from "@tanstack/ai"; @@ -97,7 +88,6 @@ const searchDatabaseDef = toolDefinition({ }); const searchDatabase = searchDatabaseDef.server(async ({ query }) => { - // Search database return { results: [] }; }); @@ -114,9 +104,9 @@ export async function POST(request: Request) { } ``` -## Model Options +## Model options -Anthropic supports various provider-specific options. Sampling parameters live here too — `temperature`, `top_p`, and `max_tokens` — rather than as root-level props on `chat()`: +Sampling lives in `modelOptions` — not root props on `chat()`: ```typescript import { chat } from "@tanstack/ai"; @@ -135,34 +125,28 @@ const stream = chat({ }); ``` -> If you previously passed `temperature` / `topP` / `maxTokens` at the root of `chat()`, see [Moving Sampling Options into modelOptions](../migration/sampling-options-to-model-options). +> Migrating root-level sampling? See [Moving Sampling Options into modelOptions](../migration/sampling-options-to-model-options). -#### `max_tokens` default +### `max_tokens` default -Anthropic's Messages API _requires_ `max_tokens` on every request, so the adapter always sends a value. When you don't set `modelOptions.max_tokens`, it defaults to the selected model's full output ceiling (`max_output_tokens` from the model metadata — e.g. 64K for Sonnet, 128K for Opus), falling back to a safe constant for unrecognized models. `max_tokens` is a ceiling, not a reservation — billing is on tokens actually generated — so this default costs nothing extra and avoids the silent mid-response truncation (`stop_reason: "max_tokens"`) that a low default would cause. Set `max_tokens` explicitly only when you want to _cap_ output below the model ceiling. If a response is truncated while using the default cap, the adapter logs a warning (visible with [debug logging](../advanced/debug-logging) enabled). +Anthropic requires `max_tokens`. If unset, the adapter sends the model's full output ceiling (`max_output_tokens` from metadata — e.g. 64K Sonnet, 128K Opus). Billing is on tokens generated, so this costs nothing extra and avoids mid-response truncation. Set it only to *cap* below the ceiling. -One exception: structured output (`chat({ outputSchema })`) on models that use the non-streaming finalization path clamps this default to ~21K tokens. The Anthropic SDK rejects a non-streaming request whose `max_tokens` could exceed its 10-minute timeout, so the full ceiling can't be used there. Streaming chat is unaffected. To raise the structured-output ceiling toward a model's true max, stream the response. +**Exception:** structured output (`outputSchema`) on non-streaming finalization clamps to ~21K (SDK timeout). Stream if you need higher. Truncation under the default logs a warning when [debug logging](../advanced/debug-logging) is on. -### Thinking (Extended Thinking) - -Enable extended thinking with a token budget. This allows Claude to show its reasoning process, which is streamed as `thinking` chunks: +### Thinking (extended) ```typescript ignore modelOptions: { thinking: { type: "enabled", - budget_tokens: 2048, // Maximum tokens for thinking + budget_tokens: 2048, }, } ``` -**Note:** `budget_tokens` must be less than `modelOptions.max_tokens` — set `max_tokens` high enough to leave room for the visible response alongside the thinking budget, or the request is rejected. - -### Adaptive Thinking (Claude 4.6+, Sonnet 5, Fable 5) +`budget_tokens` must be less than `modelOptions.max_tokens`. -Newer Claude models use adaptive thinking — the model decides when and how -much to think, and depth is tuned with `output_config.effort` instead of a -token budget: +### Adaptive thinking (Claude 4.6+, Sonnet 5, Fable 5) ```typescript import { chat } from "@tanstack/ai"; @@ -179,30 +163,15 @@ const stream = chat({ }); ``` -Per-model rules (enforced by the adapter's types): - -- **`claude-sonnet-5`, `claude-opus-4-8`, `claude-opus-4-7`** — adaptive - thinking with an explicit `{ type: "disabled" }` opt-out. The manual - `{ type: "enabled", budget_tokens }` shape is rejected with a 400, and - the sampling parameters (`temperature`, `top_p`, `top_k`) are not - accepted (on Sonnet 5 the API rejects non-default values; on Opus - 4.7/4.8 the parameters are removed entirely). -- **`claude-fable-5`** — thinking is always on. The only accepted explicit - config is `{ type: "adaptive" }` (both `disabled` and `budget_tokens` - return a 400), and sampling parameters are rejected. -- **`claude-opus-4-6` / `claude-sonnet-4-6`** — accept - `{ type: "adaptive" }` alongside the deprecated - `{ type: "enabled", budget_tokens }` shape, and still accept sampling - parameters. -- **`display`** defaults to `"omitted"` on Opus 4.7+ and the 5-generation - models — set `"summarized"` to stream the reasoning text. -- **`effort`** accepts `"low" | "medium" | "high" | "xhigh" | "max"`; - `"xhigh"` is available on Claude Opus 4.7+, Claude Sonnet 5, and - Claude Fable 5. - -### Prompt Caching - -Cache prompts for better performance and reduced costs: +| Models | Rules | +| --- | --- | +| `claude-sonnet-5`, `claude-opus-4-8`, `claude-opus-4-7` | Adaptive only; `{ type: "enabled", budget_tokens }` → 400. Sampling params rejected/removed | +| `claude-fable-5` | Thinking always on; only `{ type: "adaptive" }`. Sampling rejected | +| `claude-opus-4-6` / `claude-sonnet-4-6` | Adaptive + deprecated budget shape; sampling still OK | +| `display` | Default `"omitted"` on Opus 4.7+ and 5-gen — set `"summarized"` for reasoning text | +| `effort` | `"low"` \| `"medium"` \| `"high"` \| `"xhigh"` \| `"max"`; `"xhigh"` on Opus 4.7+, Sonnet 5, Fable 5 | + +### Prompt caching ```typescript import { chat } from "@tanstack/ai"; @@ -231,8 +200,6 @@ const stream = chat({ ## Summarization -Anthropic supports text summarization: - ```typescript import { summarize } from "@tanstack/ai"; import { anthropicSummarize } from "@tanstack/ai-anthropic"; @@ -247,56 +214,28 @@ const result = await summarize({ console.log(result.summary); ``` -## Environment Variables - -Set your API key in environment variables: - -```bash -ANTHROPIC_API_KEY=sk-ant-... -``` +## API reference -## API Reference +Short factories read `ANTHROPIC_API_KEY`; `create*` takes an explicit key. Model is always first arg. -Every factory pair follows the same shape: the short factory (`anthropicText`, `anthropicSummarize`) reads `ANTHROPIC_API_KEY` from the environment, while `createAnthropicChat` / `createAnthropicSummarize` take an explicit API key. Both take `model` as the first argument. +| Factory | Purpose | +| --- | --- | +| `anthropicText` / `createAnthropicChat` | Chat | +| `anthropicSummarize` / `createAnthropicSummarize` | Summarization | -### `anthropicText(model, config?)` / `createAnthropicChat(model, apiKey, config?)` +- `model` — e.g. `"claude-sonnet-5"`, `"claude-fable-5"`, `"claude-opus-4-8"` +- `config?.baseURL` — optional -Creates an Anthropic chat adapter. +## Notes -**Parameters:** +- **No image generation** — use OpenAI or Gemini. -- `model` - Claude model id (e.g. `"claude-sonnet-5"`, `"claude-fable-5"`, `"claude-opus-4-8"`) -- `config?.baseURL` - Custom base URL (optional) +## Provider tools -### `anthropicSummarize(model, config?)` / `createAnthropicSummarize(model, apiKey, config?)` - -Creates an Anthropic summarization adapter. - -## Limitations - -- **Image Generation**: Anthropic does not support image generation. Use OpenAI or Gemini for image generation. - -## Next Steps - -- [Getting Started](../getting-started/quick-start) - Learn the basics -- [Tools Guide](../tools/tools) - Learn about tools -- [Other Adapters](./openai) - Explore other providers - -## Provider Tools - -Anthropic exposes several native tools beyond user-defined function calls. -Import them from `@tanstack/ai-anthropic/tools` and pass them into -`chat({ tools: [...] })`. - -> For the full concept, a comparison matrix, and type-gating details, see -> [Provider Tools](../tools/provider-tools.md). +Import from `@tanstack/ai-anthropic/tools`. Full matrix: [Provider Tools](../tools/provider-tools.md). ### `webSearchTool` -Enables Claude to run Anthropic's native web search with inline citations. -Scope the search with `allowed_domains` or `blocked_domains` (mutually -exclusive); set `max_uses` to cap per-turn cost. - ```typescript import { chat } from "@tanstack/ai"; import { anthropicText } from "@tanstack/ai-anthropic"; @@ -315,14 +254,10 @@ const stream = chat({ }); ``` -**Supported models:** every registered Claude model. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). +All registered Claude models. Scope with `allowed_domains` or `blocked_domains` (mutually exclusive). ### `webFetchTool` -Lets Claude fetch the contents of a URL directly, useful when you want the -model to read a specific page rather than run a search. Takes no required -arguments — pass an optional config object to override defaults. - ```typescript import { chat } from "@tanstack/ai"; import { anthropicText } from "@tanstack/ai-anthropic"; @@ -335,14 +270,10 @@ const stream = chat({ }); ``` -**Supported models:** Claude Sonnet 4.x and above. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). +Sonnet 4.x+. ### `codeExecutionTool` -Gives Claude a sandboxed code-execution environment so it can run Python -snippets, analyse data, and return results inline. Choose the version string -that matches your desired API revision. - ```typescript import { chat } from "@tanstack/ai"; import { anthropicText } from "@tanstack/ai-anthropic"; @@ -357,13 +288,7 @@ const stream = chat({ }); ``` -**Supported models:** Claude Sonnet 4.x and above. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). - -#### Attaching hosted skills - -Pass a `skills` array as the second argument to load provider-managed skill -bundles into the sandbox. The adapter auto-lifts them into the API's -`container.skills` param and adds the required beta headers for you. +Sonnet 4.x+. Hosted skills as second arg: ```typescript import { chat, toServerSentEventsResponse } from "@tanstack/ai"; @@ -390,15 +315,10 @@ export async function POST(request: Request) { } ``` -For the full reference — skill shape, constraints, scope, and the OpenAI -equivalent — see [Provider Skills](../tools/provider-skills.md). +See [Provider Skills](../tools/provider-skills.md). ### `computerUseTool` -Allows Claude to observe a virtual desktop (screenshots) and interact with it -via keyboard and mouse events. Provide the screen resolution so Claude can -calculate accurate coordinates. - ```typescript import { chat } from "@tanstack/ai"; import { anthropicText } from "@tanstack/ai-anthropic"; @@ -418,14 +338,10 @@ const stream = chat({ }); ``` -**Supported models:** Claude Sonnet 3.5 and above. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). +Sonnet 3.5+. ### `bashTool` -Provides Claude with a persistent bash shell session, letting it run arbitrary -commands, install packages, or manipulate files on the host. Choose the type -string that matches your API revision. - ```typescript import { chat } from "@tanstack/ai"; import { anthropicText } from "@tanstack/ai-anthropic"; @@ -438,14 +354,10 @@ const stream = chat({ }); ``` -**Supported models:** Claude Sonnet 3.5 and above. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). +Sonnet 3.5+. ### `textEditorTool` -Gives Claude a structured text-editor interface for viewing and modifying files -using `str_replace`, `create`, `view`, and `undo_edit` commands. Choose the -type string for the API revision you target. - ```typescript import { chat } from "@tanstack/ai"; import { anthropicText } from "@tanstack/ai-anthropic"; @@ -460,14 +372,10 @@ const stream = chat({ }); ``` -**Supported models:** Claude Sonnet 3.5 and above. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). +Sonnet 3.5+. ### `memoryTool` -Enables Claude to store and retrieve information across conversation turns -using Anthropic's managed memory service. Call with no arguments to use -default configuration. - ```typescript import { chat } from "@tanstack/ai"; import { anthropicText } from "@tanstack/ai-anthropic"; @@ -480,14 +388,11 @@ const stream = chat({ }); ``` -**Supported models:** Claude Sonnet 4.x and above. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). +Sonnet 4.x+. ### `customTool` -Creates a tool with an inline JSON Schema input definition instead of going -through `toolDefinition()`. Useful when you need fine-grained control over the -schema shape or want to add `cache_control`. Unlike branded provider tools, -`customTool` returns a plain `Tool` and is accepted by any chat model. +Inline JSON Schema tool (plain `Tool`, any chat model): ```typescript import { chat } from "@tanstack/ai"; @@ -508,4 +413,8 @@ const stream = chat({ }); ``` -**Supported models:** all current Claude models. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). +## Next steps + +- [Getting Started](../getting-started/quick-start) +- [Tools Guide](../tools/tools) +- [Other Adapters](./openai) diff --git a/docs/adapters/bedrock.md b/docs/adapters/bedrock.md index 8fec9f6ff..53988fcde 100644 --- a/docs/adapters/bedrock.md +++ b/docs/adapters/bedrock.md @@ -2,7 +2,7 @@ title: Amazon Bedrock id: bedrock-adapter order: 7 -description: "Use Amazon Bedrock with TanStack AI — the Converse API is the default, reaching Claude, Nova, Llama, Mistral, DeepSeek, and more. Opt into OpenAI-compatible Chat Completions or Responses for open-weight and gpt-oss models. Supports streaming, tools, reasoning, and API-key or SigV4 auth." +description: "Amazon Bedrock via Converse (default), Chat Completions, or Responses — Claude, Nova, Llama, gpt-oss, and more." keywords: - tanstack ai - amazon bedrock @@ -19,128 +19,93 @@ keywords: - adapter --- -The Bedrock adapter connects TanStack AI to [Amazon Bedrock](https://aws.amazon.com/bedrock/) with three API paths: +If you need AWS Bedrock models → install, auth, call `bedrockText(model, { region })`. -- **Converse** (default) — Bedrock's model-agnostic API built on `@aws-sdk/client-bedrock-runtime`. Reaches the broad chat catalog including Anthropic Claude, Amazon Nova, Meta Llama, Mistral, DeepSeek, Cohere, AI21, and OpenAI gpt-oss models. -- **Chat Completions** (`api: 'chat'`) — Bedrock's OpenAI-compatible Chat Completions endpoint. Reaches open-weight models only (gpt-oss, DeepSeek V3.x, Gemma, Qwen, Mistral open models, GLM, etc.). Does NOT reach Claude, Nova, or Llama. -- **Responses** (`api: 'responses'`) — Bedrock's OpenAI-compatible Responses API, mantle-only. Currently the OpenAI gpt-oss family. +## When do I need which API? -All paths support streaming and client-side tool calling. Reasoning output is -surfaced when the model emits it (e.g. DeepSeek R1, gpt-oss); on the Converse -path, request-side enablement of extended thinking (e.g. a Claude thinking -budget) is not yet wired, so only models that reason by default surface it. +| Path | Use for | +| --- | --- | +| **Converse** (default) | Claude, Nova, Llama, Mistral, DeepSeek, Cohere, AI21, gpt-oss — broad catalog | +| **Chat Completions** (`api: 'chat'`) | Open-weight only (gpt-oss, DeepSeek, Gemma, Qwen, …). **Not** Claude/Nova/Llama | +| **Responses** (`api: 'responses'`) | gpt-oss on mantle; stateful via `previous_response_id` / `store` | -## Installation +Streaming + client tools on all paths. Reasoning surfaces when the model emits it; request-side Claude thinking budget is not wired on Converse yet. + +## Install ```bash pnpm add @tanstack/ai-bedrock ``` -No additional packages are required. SigV4 authentication is handled by `@aws-sdk/client-bedrock-runtime`, which is a direct dependency. +SigV4 uses `@aws-sdk/client-bedrock-runtime` (direct dependency). -## Quick Start (Converse — default) +## Auth -The default `bedrockText` call uses the Converse API and reaches the broad model catalog: +### API key -```typescript ignore -// ignore: iterating a chat() stream and reading chunk.type/chunk.delta needs the -// AG-UI base event fields, which come from @ag-ui/core. It's a transitive dep of -// @tanstack/ai, so kiira (resolving @tanstack/ai from source under the dist->src -// heuristic) can't follow it and those base fields drop off StreamChunk. The code -// is correct (the same pattern is used throughout ai-client); see -// getting-started/quick-start-server for the type-checked consumption shape. -import { bedrockText } from '@tanstack/ai-bedrock' -import { chat } from '@tanstack/ai' - -const adapter = bedrockText('us.anthropic.claude-haiku-4-5-20251001-v1:0', { - region: 'us-east-1', -}) - -for await (const chunk of chat({ - adapter, - messages: [{ role: 'user', content: 'What is the capital of France?' }], -})) { - if (chunk.type === 'TEXT_MESSAGE_CONTENT') process.stdout.write(chunk.delta ?? '') -} -``` - -Equivalent to passing `{ api: 'converse' }` explicitly. Returns a `bedrock-converse` adapter. - -## Authentication - -Bedrock supports two authentication modes. - -### API Key - -Bedrock issues API keys from the AWS Console. See the [Bedrock API keys guide](https://docs.aws.amazon.com/bedrock/latest/userguide/api-keys.html) for instructions. - -Set one of the following environment variables and the adapter picks it up automatically: +From [AWS Console](https://docs.aws.amazon.com/bedrock/latest/userguide/api-keys.html): ```bash BEDROCK_API_KEY=your-bedrock-api-key -# or the legacy name: +# or legacy: AWS_BEARER_TOKEN_BEDROCK=your-bedrock-api-key ``` -### SigV4 (AWS credential chain) - -For workloads using IAM roles, instance profiles, or `~/.aws/credentials`, set `auth: 'sigv4'` (or leave it as `'auto'` with no API key in the environment). SigV4 works out of the box via `@aws-sdk/client-bedrock-runtime` — no additional packages required. +### SigV4 ```bash AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... -AWS_SESSION_TOKEN=... # optional, for temporary credentials +AWS_SESSION_TOKEN=... # optional ``` -### Auth resolution order (`auth: 'auto'`, the default) +Set `auth: 'sigv4'`, or leave `auth: 'auto'` with no API key. -1. Explicit `apiKey` passed to the factory -2. `BEDROCK_API_KEY` environment variable -3. `AWS_BEARER_TOKEN_BEDROCK` environment variable -4. SigV4 via the standard AWS credential chain +### Resolution (`auth: 'auto'`) -## Configuration +1. Explicit `apiKey` on factory +2. `BEDROCK_API_KEY` +3. `AWS_BEARER_TOKEN_BEDROCK` +4. SigV4 credential chain -`BedrockClientConfig` accepts the following options: +## Do this (Converse) -| Option | Type | Default | Description | -|--------|------|---------|-------------| -| `api` | `'converse' \| 'chat' \| 'responses'` | `'converse'` | Bedrock API to use | -| `region` | `string` | `'us-east-1'` | AWS region string (e.g. `'us-west-2'`) | -| `auth` | `'apikey' \| 'sigv4' \| 'auto'` | `'auto'` | Authentication mode | -| `apiKey` | `string` | — | Explicit API key (overrides env vars) | -| `baseURL` | `string` | — | Override the computed base URL entirely | -| `endpoint` | `'runtime' \| 'mantle'` | `'runtime'` | Bedrock endpoint to target (Chat Completions path only) | - -The `endpoint` option only applies when `api: 'chat'`. The `runtime` endpoint (`bedrock-runtime`) hosts the broad open-weight catalog; `mantle` is an alternative. The Responses API always targets mantle. - -## Converse API (default) +```typescript ignore +// ignore: iterating chat() stream needs @ag-ui/core base fields; see +// getting-started/quick-start-server for the type-checked consumption shape. +import { bedrockText } from '@tanstack/ai-bedrock' +import { chat } from '@tanstack/ai' -`bedrockText(model)` or `bedrockText(model, { api: 'converse' })` returns a `bedrock-converse` adapter backed by `@aws-sdk/client-bedrock-runtime`. This is Bedrock's model-agnostic conversational API and is the recommended path for most use cases. +const adapter = bedrockText('us.anthropic.claude-haiku-4-5-20251001-v1:0', { + region: 'us-east-1', +}) -**Model scope:** Anthropic Claude, Amazon Nova, Meta Llama, Mistral, DeepSeek, Cohere, AI21, OpenAI gpt-oss, and other models accessible in your account. See [Model availability](#model-availability) below. +for await (const chunk of chat({ + adapter, + messages: [{ role: 'user', content: 'What is the capital of France?' }], +})) { + if (chunk.type === 'TEXT_MESSAGE_CONTENT') process.stdout.write(chunk.delta ?? '') +} +``` ```typescript import { bedrockText } from '@tanstack/ai-bedrock' import { chat } from '@tanstack/ai' -// Claude via Converse const claudeAdapter = bedrockText('us.anthropic.claude-haiku-4-5-20251001-v1:0', { region: 'us-east-1', }) -// Amazon Nova via Converse const novaAdapter = bedrockText('us.amazon.nova-pro-v1:0', { region: 'us-east-1', }) -// Meta Llama via Converse const llamaAdapter = bedrockText('us.meta.llama4-maverick-17b-instruct-v1:0', { region: 'us-east-1', }) ``` -### Explicit API key (Converse) +### Explicit API key ```typescript import { createBedrockText } from '@tanstack/ai-bedrock' @@ -152,16 +117,9 @@ const adapter = createBedrockText( ) ``` -## Chat Completions API (`api: 'chat'`) - -Set `api: 'chat'` to use Bedrock's OpenAI-compatible Chat Completions endpoint. Returns a `bedrock` adapter. - -**Model scope:** Open-weight models only — gpt-oss, DeepSeek V3.x, Gemma, Qwen, Mistral open models, GLM, and similar. Claude, Nova, and Llama are NOT available on this endpoint. See the [AWS API compatibility matrix](https://docs.aws.amazon.com/bedrock/latest/userguide/models-api-compatibility.html) for the current list. +### Chat Completions ```typescript ignore -// ignore: see the Converse quick-start above — iterating a chat() stream and -// reading chunk.type/chunk.delta needs @ag-ui/core base fields kiira can't -// resolve transitively through @tanstack/ai source. import { bedrockText } from '@tanstack/ai-bedrock' import { chat } from '@tanstack/ai' @@ -178,16 +136,9 @@ for await (const chunk of chat({ } ``` -## Responses API (`api: 'responses'`) - -Set `api: 'responses'` to use Bedrock's OpenAI-compatible Responses API. Returns a `bedrock-responses` adapter. This API is mantle-only. - -**Model scope:** Currently the OpenAI gpt-oss family. The Responses API is stateful — pass `previous_response_id` and `store` through `modelOptions` to continue a conversation server-side. +### Responses ```typescript ignore -// ignore: see the Converse quick-start above — iterating a chat() stream and -// reading chunk.type/chunk.delta needs @ag-ui/core base fields kiira can't -// resolve transitively through @tanstack/ai source. import { bedrockText } from '@tanstack/ai-bedrock' import { chat } from '@tanstack/ai' @@ -204,53 +155,47 @@ for await (const chunk of chat({ } ``` -## Model Availability +## Configuration -The adapter ships with a hand-seeded snapshot catalog (`src/model-catalog.generated.ts`) of confirmed model IDs. This catalog can be refreshed by the maintainer script `scripts/fetch-bedrock-models.ts`, which calls `ListFoundationModels` with AWS credentials. +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `api` | `'converse' \| 'chat' \| 'responses'` | `'converse'` | API path | +| `region` | `string` | `'us-east-1'` | AWS region | +| `auth` | `'apikey' \| 'sigv4' \| 'auto'` | `'auto'` | Auth mode | +| `apiKey` | `string` | — | Explicit key | +| `baseURL` | `string` | — | Override base URL | +| `endpoint` | `'runtime' \| 'mantle'` | `'runtime'` | Chat Completions only; Responses always mantle | -**Actual model availability depends on your AWS account's model access configuration and the region you are targeting.** Enable model access in the [Amazon Bedrock console](https://console.aws.amazon.com/bedrock/home#/modelaccess) before use. +## Model availability -For the full list of models and which API endpoints they support, see the [AWS API compatibility matrix](https://docs.aws.amazon.com/bedrock/latest/userguide/models-api-compatibility.html). +Catalog ships in `src/model-catalog.generated.ts` (refresh via `scripts/fetch-bedrock-models.ts`). **Enable models in your account/region** in the [console](https://console.aws.amazon.com/bedrock/home#/modelaccess). Compatibility: [AWS matrix](https://docs.aws.amazon.com/bedrock/latest/userguide/models-api-compatibility.html). -## Supported Capabilities +## Supported -- Streaming chat completions -- Client-side tool calling -- Reasoning output (extended thinking surfaced when the model emits it; request-side enablement on Converse is not yet wired) -- Multimodal input (text, images, documents — model-dependent) +- Streaming chat +- Client-side tools +- Reasoning when the model emits it +- Multimodal (model-dependent) - JSON schema / structured output -## API Reference +## API reference ### `bedrockText(model, config?)` -Creates a Bedrock adapter using environment-variable auth. - -- `model` — Model ID (e.g. `'us.anthropic.claude-haiku-4-5-20251001-v1:0'`) -- `config.api` — `'converse'` (default), `'chat'`, or `'responses'` -- `config.region` — AWS region string (default `'us-east-1'`) -- `config.auth` — `'auto'` (default), `'apikey'`, or `'sigv4'` -- `config.apiKey` — Explicit API key (overrides env vars) -- `config.baseURL` — Override base URL -- `config.endpoint` — `'runtime'` (default) or `'mantle'` (Chat Completions path only) - -Returns a chat adapter for use with `chat()` or `generate()`. - -| `api` value | Adapter name | Underlying SDK | +| `api` | Adapter name | SDK | |---|---|---| -| `'converse'` (default) | `bedrock-converse` | `@aws-sdk/client-bedrock-runtime` | -| `'chat'` | `bedrock` | `openai` (OpenAI-compatible) | -| `'responses'` | `bedrock-responses` | `openai` (OpenAI-compatible) | +| `'converse'` | `bedrock-converse` | `@aws-sdk/client-bedrock-runtime` | +| `'chat'` | `bedrock` | `openai` (compatible) | +| `'responses'` | `bedrock-responses` | `openai` (compatible) | ### `createBedrockText(model, apiKey, config?)` -Creates a Bedrock adapter with an explicit API key, bypassing the environment-variable lookup. +Explicit API key; skips env lookup. -## Next Steps +## Next steps -- [Amazon Bedrock API keys](https://docs.aws.amazon.com/bedrock/latest/userguide/api-keys.html) — Create and manage API keys -- [Amazon Bedrock model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html) — Enable models in your account -- [AWS API compatibility matrix](https://docs.aws.amazon.com/bedrock/latest/userguide/models-api-compatibility.html) — Which models work with which APIs -- [Converse API reference](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html) — Native Converse API docs -- [Streaming Guide](../chat/streaming) — Learn about streaming responses -- [Tools Guide](../tools/tools) — Learn about tool calling +- [Bedrock API keys](https://docs.aws.amazon.com/bedrock/latest/userguide/api-keys.html) +- [Model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html) +- [API compatibility matrix](https://docs.aws.amazon.com/bedrock/latest/userguide/models-api-compatibility.html) +- [Converse API](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html) +- [Streaming](../chat/streaming) · [Tools](../tools/tools) diff --git a/docs/adapters/byteplus.md b/docs/adapters/byteplus.md index daa293af3..9f66af75d 100644 --- a/docs/adapters/byteplus.md +++ b/docs/adapters/byteplus.md @@ -2,7 +2,7 @@ title: BytePlus id: byteplus-adapter order: 8 -description: "Use BytePlus ModelArk with TanStack AI — Seed chat models with reasoning, Seedance video generation, Seedream image generation, and Seed Speech text-to-speech and transcription via @tanstack/ai-byteplus." +description: "BytePlus ModelArk — Seed chat, Seedance video, Seedream image, Seed Speech TTS/ASR via @tanstack/ai-byteplus." keywords: - tanstack ai - byteplus @@ -20,43 +20,29 @@ keywords: - adapter --- -The BytePlus adapter connects TanStack AI to [BytePlus](https://www.byteplus.com/), ByteDance's international model platform, across four generation kinds: +If you need BytePlus models → install, set the right key, call the matching factory. -- **Chat** — the Seed model family (plus GLM, DeepSeek and gpt-oss) on ModelArk's OpenAI-compatible `/chat/completions` endpoint, with reasoning on by default. -- **Video** — Seedance, an asynchronous task API. -- **Image** — Seedream. -- **Speech** — Seed Speech text-to-speech and transcription, which live on a **different host with a different API key** (see [Two products, two keys](#two-products-two-keys)). - -## Installation +## Install ```bash npm install @tanstack/ai-byteplus -# or -pnpm add @tanstack/ai-byteplus -# or -yarn add @tanstack/ai-byteplus ``` ## Two products, two keys -BytePlus splits these models across two products, and they do not share credentials: - -| Adapters | Product | Env var | Auth header | +| Adapters | Product | Env var | Auth | | --- | --- | --- | --- | -| `byteplusText`, `byteplusVideo`, `byteplusImage` | ModelArk (Ark) | `ARK_API_KEY` (falls back to `BYTEPLUS_API_KEY`) | `Authorization: Bearer` | +| `byteplusText`, `byteplusVideo`, `byteplusImage` | ModelArk | `ARK_API_KEY` (fallback `BYTEPLUS_API_KEY`) | `Authorization: Bearer` | | `byteplusSpeech`, `byteplusTranscription` | Seed Speech | `BYTEPLUS_VOICE_API_KEY` | `X-Api-Key` | ```bash -# ModelArk: chat, Seedance video, Seedream image ARK_API_KEY=... - -# Seed Speech: TTS and transcription — a separate product key BYTEPLUS_VOICE_API_KEY=... ``` -Passing an Ark key to the speech adapters fails with `45000010 Invalid X-Api-Key`. +Passing an Ark key to speech → `45000010 Invalid X-Api-Key`. -Ark keys are also **region-isolated**: the default base URL is the Asia-Pacific south-east endpoint (`https://ark.ap-southeast.bytepluses.com/api/v3`), and a key issued for one region will not authenticate against another. Point the adapter elsewhere with `baseURL`: +Ark keys are **region-isolated**. Default base: `https://ark.ap-southeast.bytepluses.com/api/v3`. Override with `baseURL`: ```typescript import { createBytePlusText } from '@tanstack/ai-byteplus' @@ -67,13 +53,11 @@ const adapter = createBytePlusText('dola-seed-2-1-turbo-260628', arkApiKey, { }) ``` -Per the BytePlus docs the EU endpoint serves chat and image only — Seedance video is Asia-Pacific only. +EU serves chat + image only — Seedance is Asia-Pacific only. ## Chat -The adapter carries the model, so there is no separate `model` option. - -**Server** — an endpoint that streams the reply back over SSE: +**Server:** ```typescript import { chat, toServerSentEventsResponse } from '@tanstack/ai' @@ -91,7 +75,7 @@ export async function POST(request: Request) { } ``` -**Client** — the same `useChat` hook as every other provider: +**Client:** ```tsx import { useState } from 'react' @@ -135,7 +119,7 @@ export function Chat() { ### Model options -Ark's chat endpoint is OpenAI-compatible, so sampling parameters keep their OpenAI snake_case names and live in `modelOptions`. `thinking`, `reasoning_effort`, `repetition_penalty` and `service_tier` are the Ark-only additions: +OpenAI-compatible snake_case in `modelOptions`, plus Ark: `thinking`, `reasoning_effort`, `repetition_penalty`, `service_tier`. ```typescript import { chat, toServerSentEventsResponse } from '@tanstack/ai' @@ -160,48 +144,28 @@ export async function POST(request: Request) { } ``` -Two constraints the type system can't express, both live-verified as `400`s: +**Must not combine:** -- `max_tokens` and `max_completion_tokens` are mutually exclusive. -- `reasoning_effort` cannot be combined with `thinking: { type: 'disabled' }`. +- `max_tokens` + `max_completion_tokens` +- `reasoning_effort` + `thinking: { type: 'disabled' }` -`service_tier: 'flex'` routes the request to the cheaper offline batch queue with no latency guarantee. +`service_tier: 'flex'` → cheaper offline batch queue. -## Reasoning and `encrypted_content` +### Reasoning / `encrypted_content` -Seed models reason by default. Reasoning arrives as its own stream of `reasoning_content` deltas and is surfaced as reasoning content rather than answer text, so `useChat` renders it separately from the reply. Turn it off per request: +Seed reasons by default (`reasoning_content` deltas). Disable: ```typescript -import { chat, toServerSentEventsResponse } from '@tanstack/ai' -import { byteplusText } from '@tanstack/ai-byteplus' - -export async function POST(request: Request) { - const { messages } = await request.json() - - const stream = chat({ - adapter: byteplusText('dola-seed-2-1-turbo-260628'), - messages, - modelOptions: { thinking: { type: 'disabled' } }, - }) - - return toServerSentEventsResponse(stream) -} +modelOptions: { thinking: { type: 'disabled' } } ``` -`disabled` works everywhere; `auto` is accepted only by `gpt-oss-120b-250805`. `deepseek-v3-2-251201` is the one model that defaults to reasoning *off*. - -The four "thinking summary" models — `dola-seed-2-1-turbo-260628`, `seed-2-0-lite-260428`, `seed-2-0-mini-260428` and `seed-2-0-pro-260328` — also emit an opaque `encrypted_content` blob alongside the reasoning trace. It is a signature over that trace, and BytePlus's docs ask for it back verbatim on the assistant message in the next turn. - -**The adapter round-trips it for you**, over the same seam Anthropic's thinking signatures use: the blob is captured off the stream and attached to the reasoning step as its `signature`, which lands on the thinking part of the assistant message and is echoed back on the next request. Two consequences worth knowing: - -- If you persist and replay conversation history yourself, keep the thinking parts' `signature` — dropping it costs you the reasoning-cache hit. It is not fatal: a live probe confirmed Ark accepts a turn whose assistant message omits `encrypted_content`. -- A structured-output turn doesn't capture the blob, for the same reason — again a lost cache hit, not a failed request. +`disabled` everywhere; `auto` only on `gpt-oss-120b-250805`. `deepseek-v3-2-251201` defaults reasoning *off*. -## Structured output +Thinking-summary models (`dola-seed-2-1-turbo-260628`, `seed-2-0-lite-260428`, `seed-2-0-mini-260428`, `seed-2-0-pro-260328`) emit `encrypted_content`. **Adapter round-trips it** as reasoning `signature`. If you persist history, keep thinking parts' `signature` for cache hits (omitting is non-fatal). Structured-output turns skip the blob (lost cache hit only). -Ten of the eighteen chat models accept `response_format: { type: 'json_schema' }`: +### Structured output -`dola-seed-2-1-turbo-260628`, `seed-2-0-pro-260328`, `seed-2-0-lite-260228`, `seed-2-0-mini-260215`, `seed-1-8-251228`, `seed-1-6-250915`, `seed-1-6-250615`, `seed-1-6-flash-250715`, `seed-1-6-flash-250615`, `glm-5-2-260617`. +Ten models accept `json_schema`. Use `dola-seed-2-1-turbo-260628` or `seed-2-0-lite-260228` — **not** `seed-2-0-lite-260428` (rejects schema). `glm-4-7-251222` ignores schemas (excluded). Unsupported models throw / emit `RUN_ERROR`. Export: `BYTEPLUS_STRUCTURED_OUTPUT_CHAT_MODELS`. ```typescript import { chat } from '@tanstack/ai' @@ -223,20 +187,9 @@ const recipe = await chat({ console.log(recipe.name, recipe.minutes) ``` -On the other eight models the adapter **fails loudly** rather than degrading: `chat({ outputSchema })` throws, and the streaming form emits a `RUN_ERROR` naming the models that do work. There is no JSON-mode fallback to fall back *to* — Ark rejects `response_format: { type: 'json_object' }` on every one of them too. +## Video (Seedance) -Two traps worth knowing, both found by probing the live API rather than reading the tables: - -- **The published capability tables are wrong in both directions.** `seed-2-0-lite-260428` — the obvious default model — is one of the eight that reject a JSON schema. Reach for `seed-2-0-lite-260228` or `dola-seed-2-1-turbo-260628` when you need typed output. -- **`glm-4-7-251222` accepts a schema and then ignores it**, answering in prose with a `200`. It is deliberately excluded from the supported list, so the adapter rejects it up front instead of handing you unparseable output. - -`BYTEPLUS_STRUCTURED_OUTPUT_CHAT_MODELS` is exported if you want to gate a model picker on it. - -## Video generation (Seedance) - -> Video generation is an [experimental feature](../media/video-generation). - -Seedance is an asynchronous task API: `generateVideo()` opens the job and returns a `jobId`, and the video URL arrives with the terminal status. +> [Experimental](../media/video-generation). URLs expire 24h after complete — [keep files](../persistence/keep-generated-files). ```typescript import { generateVideo, getVideoJobStatus } from '@tanstack/ai' @@ -260,13 +213,11 @@ while (status.status === 'pending' || status.status === 'processing') { console.log(status.status === 'completed' ? status.url : status.error) ``` -**Generated video URLs expire 24 hours after the task completes** (the task record itself is kept for seven days), so download anything you intend to keep — see [Keeping generated files](../persistence/keep-generated-files). - -`size` is a `ratio` or `ratio_resolution` template (`'16:9'`, `'16:9_720p'`). Ratios are `16:9`, `9:16`, `4:3`, `3:4`, `1:1`, `21:9` and `adaptive`. +`size`: ratio or `ratio_resolution` (`'16:9'`, `'16:9_720p'`). Ratios: `16:9`, `9:16`, `4:3`, `3:4`, `1:1`, `21:9`, `adaptive`. -### Driving the whole lifecycle from the client +### Client lifecycle -**Server** — hand the polling to the core with `stream: true` and pipe the chunks out: +**Server** (`stream: true`): ```typescript import { generateVideo, toServerSentEventsResponse } from '@tanstack/ai' @@ -288,7 +239,7 @@ export async function POST(request: Request) { } ``` -**Client** — `useGenerateVideo` tracks the job for you: +**Client:** ```tsx import { fetchServerSentEvents, useGenerateVideo } from '@tanstack/ai-react' @@ -317,35 +268,27 @@ export function SeedanceGenerator() { ### Per-model options -Seedance options are model-specific, and **Ark rejects an inapplicable field with a `400` rather than ignoring it** — "the specified parameter `draft` is not supported for model seedance-1-0-pro in t2v, must be empty". The applicable set: +Ark **400s** on inapplicable fields: -| Option | Models that accept it | +| Option | Models | | --- | --- | -| `service_tier` (`'default'` \| `'flex'`) | Seedance 1.x only — the 2.0 family rejects it | -| `camera_fixed` | Seedance 1.x only | -| `frames` (fractional-second output, `25 + 4n` in `[29, 289]`) | `seedance-1-0-pro-250528`, `seedance-1-0-pro-fast-251015` | -| `draft` (cheap low-fidelity preview) | `seedance-1-5-pro-251215` only | -| `priority` (`0`–`9`) | the `dreamina-seedance-2-0-*` family only | -| `duration: -1` (model picks the length) | Seedance 2.0 and `seedance-1-5-pro-251215` | -| `seed`, `watermark`, `generate_audio`, `return_last_frame`, `callback_url` | every model | +| `service_tier`, `camera_fixed` | Seedance 1.x only | +| `frames` (`25 + 4n` in `[29, 289]`) | `seedance-1-0-pro-250528`, `seedance-1-0-pro-fast-251015` | +| `draft` | `seedance-1-5-pro-251215` only | +| `priority` (`0`–`9`) | `dreamina-seedance-2-0-*` | +| `duration: -1` | Seedance 2.0 + `seedance-1-5-pro-251215` | +| `seed`, `watermark`, `generate_audio`, `return_last_frame`, `callback_url` | all | -`watermark` defaults to `false` for video (the opposite of Seedream images). `generate_audio` is accepted everywhere but only Seedance 2.0 and 1.5-pro actually produce an audio track. - -Resolutions are per model too, and the shipped table comes from live probes rather than the published docs: there is **no 2K tier on any Seedance model**, `4k` exists only on `dreamina-seedance-2-0-260128`, and `seedance-1-0-pro-fast-251015` does accept `1080p` despite being documented as 480p/720p only. An unsupported combination is caught locally with a clear error before the request goes out. - -Reference media follows the shared [role hints](../media/video-generation#role-hints) — `start_frame`, `end_frame` and `reference`. The Seedance 2.0 family takes full multimodal references (reference images, video and audio); the 1.x models take start/end frames only, and `seedance-1-0-pro-fast-251015` takes a start frame only. +`watermark` defaults `false` for video. No 2K tier; `4k` only on `dreamina-seedance-2-0-260128`. Reference roles: [video-generation](../media/video-generation#role-hints). ### Seedance 2.5 -Seedance 2.5 was announced on 2026-07-31, initially on BytePlus's consumer surfaces. Its Ark id — `dreamina-seedance-2-5-260628` — is real and reachable, but it is **activation-gated per account**: until the model is switched on in the Ark Console, Ark answers `404 ModelNotOpen`. - -Because its capabilities could not be probed from a non-activated account, 2.5 is **deliberately absent from the typed model tables** above. It is still usable today — `byteplusVideo()`'s model parameter accepts any string, so an id BytePlus publishes after this release works without upgrading the package: +Id `dreamina-seedance-2-5-260628` is activation-gated (`404 ModelNotOpen` until enabled). Untyped but usable as any string — local guards relax. Typed tables lag until capabilities are probeable. ```typescript import { generateVideo } from '@tanstack/ai' import { byteplusVideo } from '@tanstack/ai-byteplus' -// Activate Seedance 2.5 in the Ark Console first, or this 404s. const { jobId } = await generateVideo({ adapter: byteplusVideo('dreamina-seedance-2-5-260628'), prompt: 'a guitar being played in a store', @@ -354,15 +297,9 @@ const { jobId } = await generateVideo({ }) ``` -An unknown id relaxes both halves of the adapter: `size` widens to any string, provider options are ungated, and the local runtime guards that encode per-model capabilities — resolution tiers, closing-frame and reference-media support, frame cardinality and mode exclusivity, duration snapping — stand down so Ark validates the request instead. Known ids keep their probe-verified narrowing. Typed narrowing for 2.5 follows in a package update once its capabilities can be verified. - -The quickest way to try a newly-released id is the [Seedance Studio example](https://github.com/TanStack/ai/tree/main/examples/ts-react-media): its **Advanced: custom model id** field (placeholder `dreamina-seedance-2-5-260628`) takes an arbitrary id, switches the studio into unknown-model mode with every option enabled, and spells the activation caveat out in the UI — so a `ModelNotOpen` response reads as expected rather than broken. +Also on [fal](./fal) — this adapter is direct BytePlus billing + full Seedance fields. -### Seedance here vs. Seedance via fal - -Seedance is also reachable through [`@tanstack/ai-fal`](./fal), which proxies it alongside hundreds of other hosted models. This adapter talks to BytePlus directly, which means BytePlus billing and rate limits, model ids in BytePlus's own naming, and the first-class Seedance request fields above (`camera_fixed`, `draft`, `priority`, reference-media roles) rather than fal's normalized subset. Pick whichever matches the account you already have; there is no reason to install both just for Seedance. - -## Image generation (Seedream) +## Image (Seedream) ```typescript import { generateImage } from '@tanstack/ai' @@ -378,7 +315,7 @@ const result = await generateImage({ console.log(result.images[0]?.url) ``` -`size` takes either a token (`1K`, `2K`, `4K`) or explicit pixels (`2048x2048`) — never a mix of the two. Pass image parts in the `prompt` array to edit or condition on existing images (up to 14 references, 10 on `dola-seedream-5-0-pro-260628`): +`size`: token (`1K`/`2K`/`4K`) **or** pixels (`2048x2048`) — never mix. Edit with image parts (up to 14 refs; 10 on pro): ```typescript import { generateImage } from '@tanstack/ai' @@ -394,20 +331,13 @@ const result = await generateImage({ }, ], }) - -console.log(result.images.length) ``` -Two behaviours surprise people: +**Footguns:** `watermark` defaults **true**. `numberOfImages` is an upper bound (group mode), not a guarantee. URLs expire 24h; use `response_format: 'b64_json'` for bytes. -- **`watermark` defaults to `true`.** BytePlus stamps "AI generated" into the bottom-right corner unless you pass `watermark: false`. The adapter never sets the field implicitly, so the provider default applies. -- **`numberOfImages` is an upper bound, not a count.** Seedream has no `n` parameter; asking for more than one image maps onto its group-image mode (`sequential_image_generation: 'auto'` with `max_images`), where the model decides how many images the prompt actually warrants. A request for four can come back with two, and the adapter logs a warning when it does. +## Text-to-speech -Result URLs expire after 24 hours. Pass `response_format: 'b64_json'` in `modelOptions` to get the bytes inline instead. - -## Text-to-speech (Seed Speech) - -Seed Speech uses `BYTEPLUS_VOICE_API_KEY`, not the Ark key. +Uses `BYTEPLUS_VOICE_API_KEY`. ```typescript import { generateSpeech } from '@tanstack/ai' @@ -423,23 +353,9 @@ const result = await generateSpeech({ console.log(result.contentType, result.audio.length) ``` -Seed Speech has **no top-level `speaker` field** — the voice travels inside a -`references` entry, so the adapter turns `voice` into -`references: [{ speaker: 'en_female_stokie_uranus_bigtts' }]` for you. The -consequence: `modelOptions.references` **replaces** that entry rather than -merging with it, so a request that passes `references` for voice cloning -silently drops `voice`. Include a `speaker` member yourself if you still want a -stock voice alongside your clips. - -The suffix on a voice id tells you which generation you are asking for: +`voice` maps to `references: [{ speaker }]`. Passing `modelOptions.references` **replaces** that — include `speaker` yourself if cloning + stock voice. -- `_uranus_bigtts` — TTS 2.0 voices, the current generation. -- `_mars_bigtts` / `_moon_bigtts` — TTS 1.0 voices. -- `*_emo_v2_*` — TTS 1.0 voices that additionally accept emotion tags. - -The full roster lives in the [BytePlus voice list](https://docs.byteplus.com/en/docs/byteplusvoice/voicelist) and changes far more often than this package ships, so any string is accepted. - -Output formats are `wav`, `mp3`, `pcm` and `ogg_opus`. Reach for `modelOptions` for `ogg_opus`, for an explicit `sample_rate`, for `references` (voice cloning — up to three 30-second audio clips, addressed from the text as `@Audio1`–`@Audio3`), for `watermark`, or for word-level timings: +Voice suffixes: `_uranus_bigtts` (TTS 2.0), `_mars_bigtts`/`_moon_bigtts` (1.0), `*_emo_v2_*` (emotion). Formats: `wav`, `mp3`, `pcm`, `ogg_opus`. Cap: 120s `originalDuration`. Subtitle times in ms; duration in seconds. Result `url` ~2h. ```typescript import { generateSpeech } from '@tanstack/ai' @@ -461,17 +377,7 @@ for (const sentence of result.subtitle?.sentences ?? []) { } ``` -Three things to plan around: - -- **Synthesis is capped at 120 seconds of output.** The cap applies to `originalDuration` (the length before `speech_rate` is applied), which is also what BytePlus bills on. -- **Subtitle timings are milliseconds** while `duration` and `originalDuration` are seconds. The units genuinely differ. -- The `url` on the result expires roughly two hours after generation — persist the base64 `audio`, not the link. - -The client half is the standard [`useGenerateSpeech` hook](../media/text-to-speech#streaming-mode-server-route--client-hook); nothing about it is BytePlus-specific. - -## Transcription (Seed Speech ASR) - -Also on the voice key. The endpoint is synchronous — audio in, transcript out, no polling: +## Transcription ```typescript import { generateTranscription } from '@tanstack/ai' @@ -490,48 +396,35 @@ for (const segment of result.segments ?? []) { } ``` -Audio can be a `File`, base64, a data URL or a public URL, up to two hours and 100 MB. `enable_itn` renders spoken numbers and dates as digits, `enable_punc` inserts punctuation, `enable_ddc` strips fillers, and `enable_speaker_info` attaches speaker labels that surface as `segment.speaker`. - -## Supported models - -- **Chat** — `dola-seed-2-1-turbo-260628`, `seed-2-0-lite-260428`, `seed-2-0-mini-260428`, `seed-2-0-pro-260328`, `seed-2-0-lite-260228`, `seed-2-0-mini-260215`, `seed-2-0-code-preview-260328`, `seed-1-8-251228`, `seed-1-6-250915`, `seed-1-6-250615`, `seed-1-6-flash-250715`, `seed-1-6-flash-250615`, `glm-5-2-260617`, `glm-4-7-251222`, `deepseek-v4-pro-260425`, `deepseek-v4-flash-260425`, `deepseek-v3-2-251201`, `gpt-oss-120b-250805`. -- **Video** — `dreamina-seedance-2-0-260128`, `dreamina-seedance-2-0-fast-260128`, `dreamina-seedance-2-0-mini-260615`, `seedance-1-5-pro-251215`, `seedance-1-0-pro-250528`, `seedance-1-0-pro-fast-251015`. (Seedance 2.5, `dreamina-seedance-2-5-260628`, is untyped-but-usable pending account activation — see [Seedance 2.5](#seedance-25).) -- **Image** — `dola-seedream-5-0-pro-260628`, `seedream-5-0-260128`, `seedream-5-0-lite-260128`, `seedream-4-5-251128`, `seedream-4-0-250828`. -- **Speech** — `seed-audio-1.0` (TTS) and `seed-asr` (transcription). - -BytePlus retires model ids aggressively and its published lists include ids that no longer resolve, so this package ships only dated ids that answered a live request. `BYTEPLUS_CHAT_MODELS`, `BYTEPLUS_VIDEO_MODELS`, `BYTEPLUS_IMAGE_MODELS`, `BYTEPLUS_TTS_MODELS` and `BYTEPLUS_TRANSCRIPTION_MODELS` are the authoritative lists and are exported for building model pickers. +Audio: `File` / base64 / data URL / public URL; up to 2h, 100 MB. Flags: `enable_itn`, `enable_punc`, `enable_ddc`, `enable_speaker_info`. -## API Reference +## Models -Every factory has an environment-variable form and an explicit-key form. +Exports: `BYTEPLUS_CHAT_MODELS`, `BYTEPLUS_VIDEO_MODELS`, `BYTEPLUS_IMAGE_MODELS`, `BYTEPLUS_TTS_MODELS`, `BYTEPLUS_TRANSCRIPTION_MODELS`. -### `byteplusText(model, config?)` / `createBytePlusText(model, apiKey, config?)` +**Must (chat):** `dola-seed-2-1-turbo-260628`, `seed-2-0-lite-260428`, `seed-2-0-mini-260428`, `seed-2-0-pro-260328`, `seed-2-0-lite-260228`, `seed-2-0-mini-260215`, `seed-2-0-code-preview-260328`, `seed-1-8-251228`, `seed-1-6-250915`, `seed-1-6-250615`, `seed-1-6-flash-250715`, `seed-1-6-flash-250615`, `glm-5-2-260617`, `glm-4-7-251222`, `deepseek-v4-pro-260425`, `deepseek-v4-flash-260425`, `deepseek-v3-2-251201`, `gpt-oss-120b-250805`. -Chat adapter for the Seed, GLM, DeepSeek and gpt-oss models. Reads `ARK_API_KEY` (or `BYTEPLUS_API_KEY`). `config.baseURL` overrides the region endpoint. +**Video:** `dreamina-seedance-2-0-260128`, `dreamina-seedance-2-0-fast-260128`, `dreamina-seedance-2-0-mini-260615`, `seedance-1-5-pro-251215`, `seedance-1-0-pro-250528`, `seedance-1-0-pro-fast-251015` (+ untyped 2.5). -### `byteplusVideo(model, config?)` / `createBytePlusVideo(model, apiKey, config?)` +**Image:** `dola-seedream-5-0-pro-260628`, `seedream-5-0-260128`, `seedream-5-0-lite-260128`, `seedream-4-5-251128`, `seedream-4-0-250828`. -Seedance video adapter (experimental). Reads `ARK_API_KEY`. +**Speech:** `seed-audio-1.0`, `seed-asr`. -### `byteplusImage(model, config?)` / `createBytePlusImage(model, apiKey, config?)` +## API reference -Seedream image adapter. Reads `ARK_API_KEY`. - -### `byteplusSpeech(model, config?)` / `createBytePlusSpeech(model, apiKey, config?)` - -Seed Speech TTS adapter. Reads **`BYTEPLUS_VOICE_API_KEY`**. - -### `byteplusTranscription(model, config?)` / `createBytePlusTranscription(model, apiKey, config?)` - -Seed Speech transcription adapter. Reads **`BYTEPLUS_VOICE_API_KEY`**. - -## Provider Tools +| Factory | Env key | +| --- | --- | +| `byteplusText` / `createBytePlusText` | `ARK_API_KEY` | +| `byteplusVideo` / `createBytePlusVideo` | `ARK_API_KEY` | +| `byteplusImage` / `createBytePlusImage` | `ARK_API_KEY` | +| `byteplusSpeech` / `createBytePlusSpeech` | `BYTEPLUS_VOICE_API_KEY` | +| `byteplusTranscription` / `createBytePlusTranscription` | `BYTEPLUS_VOICE_API_KEY` | -BytePlus does not expose provider-specific tool factories. Define your own with `toolDefinition()` from `@tanstack/ai` — Ark's tool calling is the standard OpenAI shape and works with the usual [tools flow](../tools/tools.md). +No provider-tool factories — use `toolDefinition()` ([tools](../tools/tools.md)). -## Next Steps +## Next steps -- [Video Generation](../media/video-generation) — the full jobs/polling flow and the `useGenerateVideo` hook -- [Image Generation](../media/image-generation) — image-conditioned generation and role hints -- [Structured Outputs](../structured-outputs/one-shot) — schemas, validation and streaming -- [Other Adapters](./openai) — explore other providers +- [Video Generation](../media/video-generation) +- [Image Generation](../media/image-generation) +- [Structured Outputs](../structured-outputs/one-shot) +- [Other Adapters](./openai) diff --git a/docs/adapters/claude-code.md b/docs/adapters/claude-code.md index 01bfaeb30..0b52f0cf8 100644 --- a/docs/adapters/claude-code.md +++ b/docs/adapters/claude-code.md @@ -2,7 +2,7 @@ title: Claude Code id: claude-code-adapter order: 11 -description: "Use Claude Code as a chat backend in TanStack AI — agent harness with local tool execution, stateful coding sessions, and tool bridging via @tanstack/ai-claude-code." +description: "Claude Code harness adapter — local agent loop, tools, and sessions via @tanstack/ai-claude-code." keywords: - tanstack ai - claude code @@ -14,26 +14,24 @@ keywords: - adapter --- -The Claude Code adapter runs [Claude Code](https://docs.anthropic.com/en/docs/claude-code) (via the `@anthropic-ai/claude-agent-sdk`) as a chat backend. Unlike HTTP provider adapters, this is a **harness adapter**: Claude Code runs its own agent loop and executes its own tools — bash, file reads and edits, glob/grep search, web search — locally on your server. Each `chat()` call runs one full harness turn; the harness's tool activity streams back as already-resolved tool-call events your UI can render. +If you need Claude Code as a chat backend → **server-only** harness: install, auth, call `claudeCodeText(model, { cwd, permissionMode })`. -> **Server-only.** The harness spawns the Claude Code runtime as a subprocess, so this adapter only works in a Node.js server environment — never in the browser. Treat it like giving Claude a shell on the machine it runs on, and configure permissions accordingly. +> Spawns a subprocess. Never run in the browser. Treat it as shell access on that machine. -## Installation +Demos: [`examples/sandbox-cloudflare`](https://github.com/TanStack/ai/tree/main/examples/sandbox-cloudflare), [`examples/sandbox-web`](https://github.com/TanStack/ai/tree/main/examples/sandbox-web). + +## Install ```bash npm install @tanstack/ai-claude-code ``` -A runnable demo lives at [`examples/sandbox-cloudflare`](https://github.com/TanStack/ai/tree/main/examples/sandbox-cloudflare) — pick Claude Code, Codex, or Grok Build in the UI, with session resume, the harness tool timeline, and tool bridging, wired into a TanStack Start app on Workers. For the same wiring on plain Node with durable, refresh-surviving runs (this adapter on Docker), see [`examples/sandbox-web`](https://github.com/TanStack/ai/tree/main/examples/sandbox-web). - -## Authentication - -The harness resolves credentials the same way Claude Code does: +## Auth -- `ANTHROPIC_API_KEY` in the server's environment (or the `apiKey` config option), or -- an existing Claude subscription login on the machine (`claude login`). +1. `ANTHROPIC_API_KEY` (or `apiKey` config), or +2. Existing `claude login` on the machine -## Basic Usage +## Do this ```typescript import { chat } from "@tanstack/ai"; @@ -50,29 +48,31 @@ const stream = chat({ ## Configuration -| Option | Description | -| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | -| `cwd` | Working directory for the harness session. Defaults to `process.cwd()`. | -| `permissionMode` | Claude Code permission mode (`'default'`, `'acceptEdits'`, `'bypassPermissions'`, `'plan'`, `'dontAsk'`, `'auto'`). See the permissions note below. | -| `allowedTools` | Built-in tools the harness may use without prompting (e.g. `['Read', 'Grep', 'Bash(npm test:*)']`). | -| `disallowedTools` | Built-in tools removed from the harness entirely. | -| `maxTurns` | Maximum harness-internal turns per run. | -| `systemPromptMode` | `'append'` (default) keeps Claude Code's preset system prompt and appends your `systemPrompts`; `'replace'` sends yours as the entire prompt. | -| `mcpServers` | Extra MCP servers passed through to the harness untouched. | -| `apiKey` | Anthropic API key for the harness subprocess. | -| `env` | Extra environment variables for the harness subprocess. | -| `pathToClaudeCodeExecutable` | Use a specific Claude Code executable instead of the SDK's bundled one. | -| `streamPartials` | Emit true token-level text deltas (default `true`). | -| `canUseTool` | Custom permission handler; replaces the adapter's default handler. | -| `settingSources` | Claude Code settings tiers to load. Default `['project']`: the `cwd`'s CLAUDE.md and project settings apply, but user-level config on the host (`~/.claude` plugins, hooks, skills) is ignored. Pass `['user', 'project', 'local']` for CLI-equivalent behavior, or `[]` for full isolation. | +| Option | Description | +| --- | --- | +| `cwd` | Working dir (default `process.cwd()`) | +| `permissionMode` | `'default'` \| `'acceptEdits'` \| `'bypassPermissions'` \| `'plan'` \| `'dontAsk'` \| `'auto'` | +| `allowedTools` | Tools that run without prompt (e.g. `['Read', 'Grep', 'Bash(npm test:*)']`) | +| `disallowedTools` | Tools removed entirely | +| `maxTurns` | Max harness-internal turns | +| `systemPromptMode` | `'append'` (default) or `'replace'` | +| `mcpServers` | Extra MCP servers | +| `apiKey` | Anthropic key for subprocess | +| `env` | Extra env for subprocess | +| `pathToClaudeCodeExecutable` | Override bundled executable | +| `streamPartials` | Token-level deltas (default `true`) | +| `canUseTool` | Custom permission handler | +| `settingSources` | Default `['project']`. Pass `['user', 'project', 'local']` for CLI-equivalent, `[]` for isolation | -**Permissions on headless servers.** Without an explicit `permissionMode` or `canUseTool`, the adapter installs a safe default handler: bridged TanStack tools always run, and any built-in tool call that would normally prompt a human is denied with guidance instead of hanging the request. To let the harness edit files or run commands, set `permissionMode: 'acceptEdits'` / `'bypassPermissions'`, or enumerate `allowedTools`. +**Headless permissions:** without `permissionMode` / `canUseTool`, bridged TanStack tools run; built-in tools that would prompt humans are denied (no hang). Set `permissionMode: 'acceptEdits'` or `'bypassPermissions'`, or list `allowedTools`, to allow edits/commands. -## Stateful Sessions +## Stateful sessions -Claude Code sessions are stateful — the harness keeps the full working context (files read, commands run, conclusions reached) between turns. The adapter surfaces the session id of every run as a custom stream event named `claude-code.session-id`; thread it back via `modelOptions.sessionId` to resume the session. When resuming, only the latest user message is sent — the harness already holds the prior context. +1. Capture `claude-code.session-id` CUSTOM event. +2. Pass `modelOptions.sessionId` on the next turn. +3. Send only the latest user message. -Server endpoint: +**Server:** ```typescript import { @@ -85,7 +85,6 @@ import { claudeCodeText } from "@tanstack/ai-claude-code"; export async function POST(request: Request) { const params = await chatParamsFromRequest(request); - // Extra fields the client puts in the connection `body` arrive here. const sessionId = typeof params.forwardedProps.sessionId === "string" ? params.forwardedProps.sessionId @@ -104,7 +103,7 @@ export async function POST(request: Request) { } ``` -Client (React) — capture the session id from the custom event and send it back on subsequent requests: +**Client:** ```typescript import { useState } from "react"; @@ -131,20 +130,16 @@ function CodingAssistant() { }, }); - // ... render messages; harness tool activity (Bash, Edit, Read, ...) - // arrives as regular tool-call parts with their results attached. + // harness tools (Bash, Edit, Read, ...) arrive as tool-call parts } ``` -Sessions are stored on the machine that ran them (`~/.claude/projects/`), so resuming only works on the same server instance. Pass `modelOptions: { forkSession: true }` alongside `sessionId` to branch a session instead of continuing it. +Sessions live under `~/.claude/projects/` — same server instance only. Fork with `modelOptions: { forkSession: true }` + `sessionId`. ## Tools -Two kinds of tools flow through this adapter: - -1. **Built-in harness tools** (`Bash`, `Read`, `Write`, `Edit`, `Glob`, `Grep`, `WebSearch`, ...) are executed by Claude Code itself. Their activity streams back as tool-call events with results already attached, so `useChat` UIs render them with no extra wiring — but your code never executes them. - -2. **Your TanStack tools** are bridged *into* the harness as an in-process MCP server. Define them as usual with `toolDefinition().server()`; the model sees them as `mcp__tanstack__` and the adapter strips the prefix on the way back out, so events match the names you registered. +1. **Harness tools** (`Bash`, `Read`, `Write`, `Edit`, `Glob`, `Grep`, `WebSearch`, …) — Claude Code runs them; events already have results. +2. **Your tools** — bridge via in-process MCP as `mcp__tanstack__` (prefix stripped on the way out). ```typescript import { z } from "zod"; @@ -166,16 +161,16 @@ const stream = chat({ }); ``` -**Client-side and approval-gated tools are not supported.** The harness executes tools inside a live subprocess, which cannot pause across HTTP requests to wait for a browser round-trip or a human approval. Passing a tool without a server `execute()` implementation — or one marked `needsApproval` — fails fast with a descriptive error. Run those tools outside the harness with a regular provider adapter. +**No client-side or `needsApproval` tools** — fails fast. Use a regular provider adapter for those. -## Structured Output +## Structured output -`structuredOutput()` uses the harness's native JSON-schema output format in a one-shot run (single turn, no tools). It works for finalization after a chat, but a plain provider adapter (e.g. `@tanstack/ai-anthropic`) is the better choice when structured extraction is the primary job — it's faster and doesn't spawn a subprocess. +`structuredOutput()` uses native JSON schema in a one-shot run. Prefer `@tanstack/ai-anthropic` when extraction is the main job. -## Limitations +## Notes -- **Server-only (Node).** The harness spawns a subprocess; Windows support is untested. -- **The harness owns the agent loop.** TanStack's agent-loop strategies and per-iteration middleware don't apply inside a harness turn; `maxTurns` is the equivalent control. -- **No sampling controls.** `temperature`-style options don't exist here. -- **Sessions are machine-local.** Resume requires hitting the same server instance. -- **Cold starts.** Each call spawns a harness turn; expect higher first-token latency than HTTP adapters. +- Server-only (Node); Windows untested +- Harness owns the agent loop — no TanStack loop strategies / per-iteration middleware inside a turn +- No sampling controls +- Sessions machine-local +- Higher cold-start latency than HTTP adapters diff --git a/docs/adapters/codex.md b/docs/adapters/codex.md index 409211423..cd1c914d7 100644 --- a/docs/adapters/codex.md +++ b/docs/adapters/codex.md @@ -2,7 +2,7 @@ title: Codex id: codex-adapter order: 12 -description: "Use OpenAI Codex as a chat backend in TanStack AI — agent harness with local tool execution, stateful coding sessions, and tool bridging via @tanstack/ai-codex." +description: "OpenAI Codex harness adapter — local agent loop, tools, and sessions via @tanstack/ai-codex." keywords: - tanstack ai - codex @@ -14,26 +14,24 @@ keywords: - adapter --- -The Codex adapter runs [OpenAI Codex](https://developers.openai.com/codex) (via the `@openai/codex-sdk`) as a chat backend. Unlike HTTP provider adapters, this is a **harness adapter**: Codex runs its own agent loop and executes its own tools — shell commands, file changes, web search — locally on your server, inside its sandbox. Each `chat()` call runs one full harness turn; the harness's tool activity streams back as already-resolved tool-call events your UI can render. +If you need OpenAI Codex as a chat backend → **server-only** harness: install, auth, call `codexText(model, { cwd, sandboxMode })`. -> **Server-only.** The harness spawns the Codex runtime (bundled with the SDK) as a subprocess, so this adapter only works in a Node.js server environment — never in the browser. The sandbox mode is the safety boundary; configure it deliberately. +> Spawns Codex runtime as a subprocess. Never in the browser. Configure sandbox deliberately. -## Installation +Demos: [`examples/sandbox-cloudflare`](https://github.com/TanStack/ai/tree/main/examples/sandbox-cloudflare), [`examples/sandbox-web`](https://github.com/TanStack/ai/tree/main/examples/sandbox-web) (swap adapter in `src/sandbox-agent.ts`). + +## Install ```bash npm install @tanstack/ai-codex ``` -A runnable demo lives at [`examples/sandbox-cloudflare`](https://github.com/TanStack/ai/tree/main/examples/sandbox-cloudflare) — pick Claude Code, Codex, or Grok Build in the UI, with session resume, the harness tool timeline, and tool bridging, wired into a TanStack Start app on Workers. For the same wiring on plain Node with durable, refresh-surviving runs (Claude Code on Docker), see [`examples/sandbox-web`](https://github.com/TanStack/ai/tree/main/examples/sandbox-web) — swapping in this adapter is a one-line change (`src/sandbox-agent.ts`). - -## Authentication - -The harness resolves credentials the same way the Codex CLI does: +## Auth -- the `apiKey` config option (exported to the subprocess as `CODEX_API_KEY`; usage-based billing), or -- an existing ChatGPT login on the machine (`codex login`). +1. `apiKey` config → `CODEX_API_KEY` on subprocess (usage billing), or +2. Existing `codex login` on the machine -## Basic Usage +## Do this ```typescript import { chat } from "@tanstack/ai"; @@ -50,29 +48,31 @@ const stream = chat({ ## Configuration -| Option | Description | -| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | -| `cwd` | Working directory for the harness session. Defaults to `process.cwd()`. | -| `sandboxMode` | Codex sandbox: `'read-only'` (harness default), `'workspace-write'`, or `'danger-full-access'`. This is the safety boundary on a server. | -| `approvalPolicy` | Codex approval policy. Defaults to `'never'` — headless runs have no approval UI, so anything else can stall a turn. | -| `modelReasoningEffort` | `'minimal'` \| `'low'` \| `'medium'` \| `'high'` \| `'xhigh'`. | -| `skipGitRepoCheck` | Skip the harness's git-repo safety check. Defaults to `true` (server adapters routinely point at scratch directories). | -| `networkAccessEnabled` | Allow network access inside the `workspace-write` sandbox. | -| `webSearchMode` | `'disabled'` \| `'cached'` \| `'live'`. | -| `additionalDirectories`| Extra writable directories beyond `cwd`. | -| `apiKey` | OpenAI API key for the harness subprocess. | -| `baseUrl` | Override the Codex backend base URL. | -| `codexPathOverride` | Use a specific codex executable instead of the SDK's bundled binary. | -| `env` | Environment variables for the subprocess. When set, `process.env` is **not** inherited (Codex SDK semantics). | -| `config` | Extra `--config key=value` overrides passed to the Codex CLI (e.g. additional `mcp_servers` entries). | +| Option | Description | +| --- | --- | +| `cwd` | Working dir (default `process.cwd()`) | +| `sandboxMode` | `'read-only'` (default) \| `'workspace-write'` \| `'danger-full-access'` | +| `approvalPolicy` | Default `'never'` — anything else can stall headless turns | +| `modelReasoningEffort` | `'minimal'` \| `'low'` \| `'medium'` \| `'high'` \| `'xhigh'` | +| `skipGitRepoCheck` | Default `true` | +| `networkAccessEnabled` | Network inside `workspace-write` sandbox | +| `webSearchMode` | `'disabled'` \| `'cached'` \| `'live'` | +| `additionalDirectories` | Extra writable dirs | +| `apiKey` | OpenAI key for subprocess | +| `baseUrl` | Override Codex backend URL | +| `codexPathOverride` | Specific codex binary | +| `env` | Subprocess env — when set, `process.env` is **not** inherited | +| `config` | Extra `--config key=value` (e.g. `mcp_servers`) | -Per-call overrides — `sessionId`, `sandboxMode`, `approvalPolicy`, `modelReasoningEffort`, `workingDirectory`, `skipGitRepoCheck` — go through `modelOptions`. +Per-call: `sessionId`, `sandboxMode`, `approvalPolicy`, `modelReasoningEffort`, `workingDirectory`, `skipGitRepoCheck` via `modelOptions`. -## Stateful Sessions +## Stateful sessions -Codex threads are stateful — the harness keeps the full working context (files read, commands run, conclusions reached) between turns. The adapter surfaces the thread id of every fresh run as a custom stream event named `codex.session-id`; thread it back via `modelOptions.sessionId` to resume. When resuming, only the latest user message is sent — the harness already holds the prior context. +1. Capture `codex.session-id` CUSTOM event. +2. Pass `modelOptions.sessionId`. +3. Send only the latest user message. -Server endpoint: +**Server:** ```typescript import { @@ -85,7 +85,6 @@ import { codexText } from "@tanstack/ai-codex"; export async function POST(request: Request) { const params = await chatParamsFromRequest(request); - // Extra fields the client puts in the connection `body` arrive here. const sessionId = typeof params.forwardedProps.sessionId === "string" ? params.forwardedProps.sessionId @@ -104,7 +103,7 @@ export async function POST(request: Request) { } ``` -Client (React) — capture the session id from the custom event and send it back on subsequent requests: +**Client:** ```typescript import { useState } from "react"; @@ -131,20 +130,16 @@ function CodingAssistant() { }, }); - // ... render messages; harness tool activity (command_execution, - // file_change, ...) arrives as regular tool-call parts with results. + // harness tools stream as tool-call parts with results } ``` -Sessions are stored on the machine that ran them (`~/.codex/sessions/`), so resuming only works on the same server instance. +Sessions under `~/.codex/sessions/` — same server instance only. ## Tools -Two kinds of tools flow through this adapter: - -1. **Built-in harness tools** are executed by Codex itself and stream back as tool-call events with results already attached: `command_execution` (shell), `file_change` (patches), `web_search`, and `todo_list` (the agent's running plan). Your code never executes them. - -2. **Your TanStack tools** are bridged *into* the harness: the adapter starts a short-lived Streamable-HTTP MCP server on `127.0.0.1` for the duration of the turn and points Codex at it. Define tools as usual with `toolDefinition().server()`; tool-call events come back under the names you registered. +1. **Harness tools** — `command_execution`, `file_change`, `web_search`, `todo_list` (results attached). +2. **Your tools** — short-lived Streamable-HTTP MCP on `127.0.0.1` for the turn. ```typescript import { z } from "zod"; @@ -166,17 +161,17 @@ const stream = chat({ }); ``` -**Client-side and approval-gated tools are not supported.** The harness executes tools inside a live subprocess, which cannot pause across HTTP requests to wait for a browser round-trip or a human approval. Passing a tool without a server `execute()` implementation — or one marked `needsApproval` — fails fast with a descriptive error. Run those tools outside the harness with a regular provider adapter. +**No client-side or `needsApproval` tools** — fails fast. -## Structured Output +## Structured output -`structuredOutput()` uses Codex's native `outputSchema` support in a fresh, read-only, one-shot thread whose final message is a JSON string conforming to your schema. It works for finalization after a chat, but a plain provider adapter (e.g. `@tanstack/ai-openai`) is the better choice when structured extraction is the primary job — it's faster and doesn't spawn a subprocess. +`structuredOutput()` uses native `outputSchema` in a fresh read-only thread. Prefer `@tanstack/ai-openai` for primary extraction work. -## Limitations +## Notes -- **No token-level text streaming.** The Codex SDK reports assistant text and reasoning only as completed items, so text arrives message-at-a-time. Tool activity (commands starting/finishing) still streams live, which keeps the UI feeling alive during long turns. -- **Server-only (Node).** The harness spawns a subprocess. -- **The harness owns the agent loop.** TanStack's agent-loop strategies and per-iteration middleware don't apply inside a harness turn. -- **No sampling controls.** `temperature`-style options don't exist here. -- **Sessions are machine-local.** Resume requires hitting the same server instance. -- **Cold starts.** Each call spawns a harness turn; expect higher first-token latency than HTTP adapters. +- No token-level text streaming (message-at-a-time); tool activity still streams live +- Server-only (Node) +- Harness owns the agent loop +- No sampling controls +- Sessions machine-local +- Higher cold-start latency diff --git a/docs/adapters/elevenlabs.md b/docs/adapters/elevenlabs.md index b1c066dc6..2dc72637c 100644 --- a/docs/adapters/elevenlabs.md +++ b/docs/adapters/elevenlabs.md @@ -2,7 +2,7 @@ title: ElevenLabs id: elevenlabs-adapter order: 9 -description: "Build realtime voice-to-voice conversational AI with ElevenLabs agents in TanStack AI via the @tanstack/ai-elevenlabs adapter." +description: "ElevenLabs voice agents, TTS, music/SFX, and transcription via @tanstack/ai-elevenlabs." keywords: - tanstack ai - elevenlabs @@ -13,38 +13,32 @@ keywords: - adapter --- -The ElevenLabs adapter is **voice-focused**. It exposes four capabilities: +If you need ElevenLabs voice → install, configure an agent, issue a token on the server, connect on the client. -- **Realtime voice agents** (`elevenlabsRealtime` / `elevenlabsRealtimeToken`) — full-duplex voice-to-voice conversations powered by ElevenLabs Conversational AI agents. -- **Text-to-speech** (`elevenlabsSpeech`) — one-shot speech generation via `generateSpeech()`. -- **Music & sound effects** (`elevenlabsAudio`) — one-shot audio generation via `generateAudio()`. -- **Transcription** (`elevenlabsTranscription`) — speech-to-text via `generateTranscription()`. +**Has:** realtime agents, TTS, music/SFX, transcription. +**Does not:** `chat()`, `summarize()` — use a text adapter. -It does not support text `chat()` or `summarize()` — use OpenAI, Anthropic, or Gemini for those. - -The realtime adapter uses an **agent-based architecture** where you configure your conversational AI agent in the [ElevenLabs dashboard](https://elevenlabs.io/) (voice, personality, knowledge base, tools) and then connect to it at runtime. The adapter wraps the `@elevenlabs/client` SDK for seamless integration with `useRealtimeChat` and `RealtimeClient`. - -## Installation +## Install ```bash npm install @tanstack/ai-elevenlabs ``` -Peer dependencies: - ```bash -npm install @tanstack/ai @tanstack/ai-client +ELEVENLABS_API_KEY=your-elevenlabs-api-key +ELEVENLABS_AGENT_ID=your-agent-id # optional if passed to token factory ``` -## Server Setup +Create agents in the [ElevenLabs dashboard](https://elevenlabs.io/) (Conversational AI). + +## Realtime: do this -The server generates a **signed WebSocket URL** so your API key never reaches the client. The signed URL is valid for 30 minutes. +### 1. Server — signed URL (30 min) ```typescript group=elevenlabs-1 import { realtimeToken } from '@tanstack/ai' import { elevenlabsRealtimeToken } from '@tanstack/ai-elevenlabs' -// In your API route (Express, Hono, TanStack Start, etc.) export async function POST() { const token = await realtimeToken({ adapter: elevenlabsRealtimeToken({ @@ -56,9 +50,7 @@ export async function POST() { } ``` -### With Overrides - -You can override agent settings at token generation time without changing your dashboard configuration: +Optional overrides at token time: ```typescript group=elevenlabs-1 const token = await realtimeToken({ @@ -74,9 +66,7 @@ const token = await realtimeToken({ }) ``` -## Client Setup - -### React (useRealtimeChat) +### 2. Client — React ```tsx import { useRealtimeChat } from '@tanstack/ai-react' @@ -126,7 +116,7 @@ function VoiceChat() { } ``` -### Non-React (RealtimeClient) +### Non-React ```typescript import { RealtimeClient } from '@tanstack/ai-client' @@ -150,9 +140,7 @@ const client = new RealtimeClient({ await client.connect() ``` -## Client Tools - -ElevenLabs supports client-side tools that execute in the browser. Define tools using the standard `toolDefinition()` API: +### Client tools ```typescript import { toolDefinition } from '@tanstack/ai' @@ -177,7 +165,6 @@ const getWeather = getWeatherDef.client(async ({ location }) => { return res.json() }) -// Pass tools to the hook const chat = useRealtimeChat({ getToken: () => fetch('/api/realtime-token', { method: 'POST' }).then((r) => r.json()), @@ -186,58 +173,43 @@ const chat = useRealtimeChat({ }) ``` -Tool results are automatically serialized to strings and returned to the ElevenLabs agent. The adapter converts TanStack tool definitions into the `@elevenlabs/client` clientTools format internally. - ## Configuration -### `elevenlabsRealtimeToken` Options - -Used on the **server** to generate a signed WebSocket URL. - -| Option | Type | Required | Description | -|--------|------|----------|-------------| -| `agentId` | `string` | No\* | Agent ID configured in the ElevenLabs dashboard. \*Falls back to `ELEVENLABS_AGENT_ID`; required only if that env var is unset | -| `overrides.voiceId` | `string` | No | Custom voice ID to override the agent's default voice | -| `overrides.systemPrompt` | `string` | No | Custom system prompt to override the agent's default | -| `overrides.firstMessage` | `string` | No | First message the agent speaks when the session starts | -| `overrides.language` | `string` | No | Language code (e.g., `'en'`, `'es'`, `'fr'`) | +### `elevenlabsRealtimeToken` (server) -### `elevenlabsRealtime` Options +| Option | Required | Description | +|--------|----------|-------------| +| `agentId` | if env unset | Dashboard agent id (`ELEVENLABS_AGENT_ID` fallback) | +| `overrides.voiceId` | no | Override voice | +| `overrides.systemPrompt` | no | Override system prompt | +| `overrides.firstMessage` | no | First spoken line | +| `overrides.language` | no | e.g. `'en'`, `'es'` | -Used on the **client** to establish the connection. +### `elevenlabsRealtime` (client) -| Option | Type | Default | Description | -|--------|------|---------|-------------| -| `connectionMode` | `'websocket' \| 'webrtc'` | auto-detect | Transport protocol for the connection | -| `debug` | `boolean \| DebugConfig` | `false` | Enable debug logging — pass `true` for all categories, or a `DebugConfig` to select categories/sink | +| Option | Default | Description | +|--------|---------|-------------| +| `connectionMode` | auto | `'websocket'` \| `'webrtc'` | +| `debug` | `false` | Logging | -## Differences from OpenAI Realtime - -ElevenLabs and OpenAI take different approaches to realtime voice: +## vs OpenAI Realtime | | ElevenLabs | OpenAI | |---|---|---| -| **Configuration** | Agent-based. Configure voice, personality, and knowledge in the ElevenLabs dashboard or via `overrides` at token time. | Session-based. Configure `instructions`, `voice`, `temperature`, etc. per session via `useRealtimeChat` options. | -| **Token type** | Signed WebSocket URL (valid 30 minutes) | Ephemeral API token (valid ~10 minutes) | -| **Transport** | WebSocket (default) or WebRTC | WebRTC | -| **Audio handling** | `@elevenlabs/client` SDK manages audio capture and playback automatically | TanStack AI manages WebRTC peer connection and audio tracks | -| **VAD** | Handled by ElevenLabs server-side | Supports `server`, `semantic`, and `manual` modes | -| **Runtime updates** | Session config is set at creation time and cannot be changed mid-session | Supports `updateSession()` for mid-session config changes | -| **Image input** | Not supported | Supported via `sendImage()` | -| **Time domain data** | Not available from the SDK | Available for waveform visualizations | - -## Audio Visualization +| Config | Dashboard agent + token overrides | Per-session options | +| Token | Signed WS URL (30 min) | Ephemeral (~10 min) | +| Transport | WS (default) or WebRTC | WebRTC | +| Mid-session updates | No | `updateSession()` | +| Image input | No | `sendImage()` | +| Time-domain audio | No (freq + volume only) | Yes | -The ElevenLabs adapter provides audio visualization data through the same interface as other realtime adapters: +## Audio visualization ```typescript -import { useRealtimeChat } from '@tanstack/ai-react' -import { elevenlabsRealtime } from '@tanstack/ai-elevenlabs' - const { - inputLevel, // 0-1 normalized microphone volume - outputLevel, // 0-1 normalized speaker volume - getInputFrequencyData, // Uint8Array frequency spectrum + inputLevel, + outputLevel, + getInputFrequencyData, getOutputFrequencyData, } = useRealtimeChat({ getToken: () => @@ -246,27 +218,9 @@ const { }) ``` -**Note:** ElevenLabs provides volume levels and frequency data but does not expose time-domain data. The `getInputTimeDomainData()` and `getOutputTimeDomainData()` methods return static placeholder arrays. The default audio sample rate is 16kHz. - -## Environment Variables - -Set these in your server environment: - -```bash -ELEVENLABS_API_KEY=your-elevenlabs-api-key -ELEVENLABS_AGENT_ID=your-agent-id -``` - -| Variable | Required | Description | -|----------|----------|-------------| -| `ELEVENLABS_API_KEY` | Yes | Your ElevenLabs API key, used server-side for generating signed URLs | -| `ELEVENLABS_AGENT_ID` | No | Default agent ID. Can also be passed directly to `elevenlabsRealtimeToken()` | - -Get your API key from the [ElevenLabs dashboard](https://elevenlabs.io/). Create and configure agents in the **Conversational AI** section of the dashboard. +`getInputTimeDomainData()` / `getOutputTimeDomainData()` return placeholders. Sample rate default 16kHz. -## Text-to-Speech - -For one-shot speech generation (not realtime), use `elevenlabsSpeech` with `generateSpeech()`: +## Text-to-speech ```typescript import { generateSpeech } from "@tanstack/ai"; @@ -279,24 +233,20 @@ const result = await generateSpeech({ format: "mp3", }); -console.log(result.audio); // Base64-encoded audio +console.log(result.audio); ``` -## Music & Sound Effects - -`elevenlabsAudio` covers both music generation and sound effects depending on the model: +## Music & sound effects ```typescript import { generateAudio } from "@tanstack/ai"; import { elevenlabsAudio } from "@tanstack/ai-elevenlabs"; -// Music generation const music = await generateAudio({ adapter: elevenlabsAudio("music_v1"), prompt: "An upbeat synthwave track for a product launch", }); -// Sound effects const sfx = await generateAudio({ adapter: elevenlabsAudio("eleven_text_to_sound_v2"), prompt: "A glass shattering on concrete", @@ -305,8 +255,6 @@ const sfx = await generateAudio({ ## Transcription -Transcribe audio with `elevenlabsTranscription`: - ```typescript import { generateTranscription } from "@tanstack/ai"; import { elevenlabsTranscription } from "@tanstack/ai-elevenlabs"; @@ -320,56 +268,23 @@ const result = await generateTranscription({ console.log(result.text); ``` -## API Reference - -### `elevenlabsRealtimeToken(options)` - -Creates an ElevenLabs realtime token adapter for server-side use with `realtimeToken()`. - -**Parameters:** - -- `options.agentId` - Agent ID from the ElevenLabs dashboard -- `options.overrides?.voiceId` - Custom voice ID -- `options.overrides?.systemPrompt` - Custom system prompt -- `options.overrides?.firstMessage` - First message the agent speaks -- `options.overrides?.language` - Language code - -**Returns:** A `RealtimeTokenAdapter` for use with `realtimeToken()`. - -### `elevenlabsRealtime(options?)` - -Creates an ElevenLabs realtime client adapter for use with `useRealtimeChat` or `RealtimeClient`. - -**Parameters:** - -- `options.connectionMode?` - `'websocket'` or `'webrtc'` (default: auto-detect) -- `options.debug?` - Enable debug logging - -**Returns:** A `RealtimeAdapter` for use with `useRealtimeChat()` or `RealtimeClient`. - -### `elevenlabsSpeech(model, config?)` / `createElevenLabsSpeech(model, apiKey, config?)` - -Creates an ElevenLabs text-to-speech adapter for use with `generateSpeech()`. - -### `elevenlabsAudio(model, config?)` / `createElevenLabsAudio(model, apiKey, config?)` - -Creates an ElevenLabs audio adapter that covers both music generation and sound effects (selected via the model id) for use with `generateAudio()`. - -### `elevenlabsTranscription(model, config?)` / `createElevenLabsTranscription(model, apiKey, config?)` +## API reference -Creates an ElevenLabs transcription adapter for use with `generateTranscription()`. +| Factory | Use with | +| --- | --- | +| `elevenlabsRealtimeToken` | `realtimeToken()` | +| `elevenlabsRealtime` | `useRealtimeChat` / `RealtimeClient` | +| `elevenlabsSpeech` / `createElevenLabsSpeech` | `generateSpeech()` | +| `elevenlabsAudio` / `createElevenLabsAudio` | `generateAudio()` | +| `elevenlabsTranscription` / `createElevenLabsTranscription` | `generateTranscription()` | -## Limitations +## Notes -- **No text chat support** -- Use OpenAI, Anthropic, Gemini, or another text adapter for `chat()`. -- **No summarization** -- Use a text adapter for `summarize()`. -- **No image input** (realtime) -- ElevenLabs realtime does not support sending images during a conversation. -- **No runtime session updates** (realtime) -- Session configuration is fixed at connection time. -- **No time-domain audio data** (realtime) -- Frequency data and volume levels are available, but waveform data is not. -- **Agent required** (realtime) -- You must create and configure an agent in the ElevenLabs dashboard before using the realtime adapter. +- No text chat / summarization +- Realtime: agent required; no image input; no mid-session config; no time-domain audio -## Next Steps +## Next steps -- [Realtime Voice Chat Guide](../media/realtime-chat) - Complete guide to building realtime voice applications -- [OpenAI Adapter](./openai) - Alternative realtime voice provider with WebRTC -- [Tools Guide](../tools/tools) - Learn about the isomorphic tool system +- [Realtime Voice Chat](../media/realtime-chat) +- [OpenAI Adapter](./openai) +- [Tools](../tools/tools) diff --git a/docs/adapters/fal.md b/docs/adapters/fal.md index 46ccc246c..e95922f46 100644 --- a/docs/adapters/fal.md +++ b/docs/adapters/fal.md @@ -1,7 +1,7 @@ --- title: fal.ai id: fal-adapter -description: "Generate images and videos with 600+ models on fal.ai using TanStack AI — Nano Banana Pro, FLUX, and more via the @tanstack/ai-fal adapter." +description: "Images, video, audio, TTS, and transcription on fal.ai via @tanstack/ai-fal." keywords: - tanstack ai - fal.ai @@ -13,38 +13,34 @@ keywords: - adapter --- -The fal.ai adapter provides access to 600+ models on the fal.ai platform for image, video, audio, speech, and transcription. Unlike text-focused adapters, the fal adapter is **media-focused** — it supports `generateImage()`, `generateVideo()`, `generateAudio()`, `generateSpeech()`, and `generateTranscription()` but does not support `chat()` or tools. +If you need fal media → install, set `FAL_KEY`, pass a **string-literal** model id to the matching factory. -For a full working example, see the [fal.ai example app](https://github.com/TanStack/ai/tree/main/examples/ts-react-media). +**Has:** `generateImage`, `generateVideo`, `generateAudio`, `generateSpeech`, `generateTranscription`. +**Does not:** `chat()`, tools, `summarize()`. -## Installation +Example app: [ts-react-media](https://github.com/TanStack/ai/tree/main/examples/ts-react-media). + +## Install ```bash npm install @tanstack/ai-fal ``` -## Type Safety with String Literals - -The fal adapter provides full type safety when you pass the model ID as a **string literal**. This gives you autocomplete for `size` and `modelOptions` specific to that model. Always use string literals — not variables — when creating adapters: +```bash +FAL_KEY=your-fal-api-key +``` -```typescript -import { falImage } from "@tanstack/ai-fal"; +## Type safety -// Good — full type safety and autocomplete -const adapter = falImage("fal-ai/z-image/turbo"); -``` +Pass model IDs as **string literals** for autocomplete on `size` / `modelOptions`. Variables lose inference; unknown new ids still work without types. ```typescript import { falImage } from "@tanstack/ai-fal"; -// Bad — no type inference for model-specific options -const modelId = "fal-ai/z-image/turbo"; -const adapter = falImage(modelId); +const adapter = falImage("fal-ai/z-image/turbo"); // good ``` -You can also pass any string for new models that fal.ai hasn't provided types for yet — you just won't get type safety on those endpoints. - -## Basic Usage +## Image ```typescript import { generateImage } from "@tanstack/ai"; @@ -59,7 +55,7 @@ const result = await generateImage({ console.log(result.images); ``` -## Basic Usage - Custom API Key +### Explicit key / proxy ```typescript import { generateImage } from "@tanstack/ai"; @@ -67,6 +63,7 @@ import { falImage } from "@tanstack/ai-fal"; const adapter = falImage("fal-ai/flux/dev", { apiKey: process.env.FAL_KEY!, + // proxyUrl: "https://your-server.com/api/fal/proxy", }); const result = await generateImage({ @@ -76,32 +73,12 @@ const result = await generateImage({ }); ``` -## Configuration - -```typescript -import { falImage, type FalClientConfig } from "@tanstack/ai-fal"; - -// Direct API key -const adapter = falImage("fal-ai/flux/dev", { - apiKey: "your-api-key", -}); - -// Using a proxy URL (for client-side usage) -const proxiedAdapter = falImage("fal-ai/flux/dev", { - apiKey: "your-api-key", - proxyUrl: "https://your-server.com/api/fal/proxy", -}); -``` - -## Example: Image Generation - -From the [fal.ai example app](https://github.com/TanStack/ai/tree/main/examples/ts-react-media): +### With size + model options ```typescript import { generateImage } from "@tanstack/ai"; import { falImage } from "@tanstack/ai-fal"; -// Use string literals for the model to get full type safety const result = await generateImage({ adapter: falImage("fal-ai/nano-banana-pro"), prompt: "A futuristic cityscape at sunset", @@ -113,17 +90,10 @@ const result = await generateImage({ }); ``` -## Example: Image with Model Options - -Each fal.ai model has its own type-safe options. The adapter uses fal.ai's types to provide autocomplete and type checking for model-specific parameters. There are 1000s of combinations. If you provide the proper model endpoint id as as string literal, you will then only be able to provide `size` and `modelOptions` that the model supports. - -_IMPORTANT_: It is possible to pass strings and new endpoint ids that Fal has not provided types for. You just won't get type safety on those endpoints. You'll find this happens for very new models. - ```typescript import { generateImage } from "@tanstack/ai"; import { falImage } from "@tanstack/ai-fal"; -// Model-specific options are type-safe const result = await generateImage({ adapter: falImage("fal-ai/z-image/turbo"), prompt: "A serene mountain landscape", @@ -136,48 +106,23 @@ const result = await generateImage({ }); ``` -## Image Size Options - -The fal adapter supports a flexible `size` paramater that maps either to `image_size` or to `aspect_ratio` and `resolution` parameters: - -| | `size` | Maps To | -|--------|---------|---------| -| named | `"landscape_16_9"` | `image_size: "landscape_16_9"` | -| width x height (OpenAI) | `"1536x1024"` | | `image_size: "1536x1024"` | -| aspect ratio & resolution | `"16:9_4K"` | `aspect_ratio: "16:9"`, `resolution: "4K"` | -| aspect ratio only | `"16:9"` | `aspect_ratio: "16:9"` | - -```typescript ignore -// Aspect ratio only -size: "16:9" - -// Aspect ratio with resolution -size: "16:9_4K" - -// Named size (model-specific) -size: "landscape_16_9" -``` - -## Video Generation (Experimental) - -> **Note:** Video generation is an experimental feature and may change in future releases. In particular, this version of the adapter does not map the duration paramater +### Size mapping -Video generation uses a queue-based workflow: submit a job, poll for status, then retrieve the video URL when complete. +| Form | Example | Maps to | +|------|---------|---------| +| named | `"landscape_16_9"` | `image_size` | +| width×height | `"1536x1024"` | `image_size` | +| ratio + res | `"16:9_4K"` | `aspect_ratio` + `resolution` | +| ratio only | `"16:9"` | `aspect_ratio` | -```typescript -import { generateVideo, getVideoJobStatus } from "@tanstack/ai"; -import { falVideo } from "@tanstack/ai-fal"; -``` +## Video (experimental) -## Example: Text-to-Video - -From the [fal.ai example app](https://github.com/TanStack/ai/tree/main/examples/ts-react-media): +Queue: submit → poll → URL. Duration not mapped by adapter top-level (use `modelOptions`). ```typescript import { generateVideo, getVideoJobStatus } from "@tanstack/ai"; import { falVideo } from "@tanstack/ai-fal"; -// 1. Submit the video generation job const adapter = falVideo("fal-ai/kling-video/v2.6/pro/text-to-video"); const job = await generateVideo({ @@ -189,16 +134,15 @@ const job = await generateVideo({ }, }); -// 2. Poll for status const status = await getVideoJobStatus({ adapter, jobId: job.jobId, }); -console.log(status.status); // "pending" | "processing" | "completed" +console.log(status.status); ``` -## Example: Image-to-Video +### Image-to-video ```typescript import { generateVideo } from "@tanstack/ai"; @@ -215,9 +159,9 @@ const job = await generateVideo({ }); ``` -## Text-to-Speech +## Text-to-speech -Text-to-speech uses `falSpeech()` with the `generateSpeech()` activity. The adapter fetches the generated audio from fal's CDN and returns it as base64-encoded data to match the `TTSResult` contract. +Audio returned as base64 (`result.audio`). ```typescript import { generateSpeech } from "@tanstack/ai"; @@ -230,14 +174,12 @@ const result = await generateSpeech({ speed: 1.0, }); -// result.audio is a base64-encoded string -console.log(result.format); // e.g. "wav" -console.log(result.contentType); // e.g. "audio/wav" +console.log(result.format, result.contentType); ``` -### Google Gemini 3.1 Flash TTS +### Gemini 3.1 Flash TTS -Google's newest TTS model (`fal-ai/gemini-3.1-flash-tts`) supports 80+ languages and introduces **granular audio tags** for expressive control — you can embed speaker tags and style cues directly in the text. +Embed style tags in text. Newer than `@fal-ai/client` types — works, no `modelOptions` autocomplete. ```typescript import { generateSpeech } from "@tanstack/ai"; @@ -250,8 +192,6 @@ const result = await generateSpeech({ }); ``` -> **Note:** This model is newer than `@fal-ai/client@1.9.1`'s type map, so `modelOptions` won't autocomplete. The call still works — the fal adapter accepts any model ID as a string. Type-safe autocomplete will land when fal's SDK types catch up. - ### ElevenLabs v3 ```typescript @@ -270,7 +210,7 @@ const result = await generateSpeech({ ## Transcription -Speech-to-text uses `falTranscription()` with the `generateTranscription()` activity. The `audio` input accepts a URL string, `Blob`, `File`, or `ArrayBuffer` — `ArrayBuffer` is automatically wrapped in a `Blob` for upload. +`audio`: URL, `Blob`, `File`, or `ArrayBuffer`. ```typescript import { generateTranscription } from "@tanstack/ai"; @@ -283,26 +223,22 @@ const result = await generateTranscription({ }); console.log(result.text); -console.log(result.language); - -// When the model returns word/segment timestamps, they're mapped to result.segments for (const segment of result.segments ?? []) { console.log(`[${segment.start}s → ${segment.end}s] ${segment.text}`); } ``` -## Audio Generation (Music & Sound Effects) +## Audio (music / SFX) -Music and sound-effect generation uses `falAudio()` with the `generateAudio()` activity. Unlike TTS, the result is returned as a URL in `result.audio.url` (you can fetch it yourself if you need raw bytes). +Result URL at `result.audio.url`. ```typescript import { generateAudio } from "@tanstack/ai"; import { falAudio } from "@tanstack/ai-fal"; -// Music generation with MiniMax Music 2.6 (latest) const music = await generateAudio({ adapter: falAudio("fal-ai/minimax-music/v2.6"), - prompt: "City Pop, 80s retro, groovy synth bass, warm female vocal, 104 BPM, nostalgic urban night", + prompt: "City Pop, 80s retro, groovy synth bass, warm female vocal, 104 BPM", }); console.log(music.audio.url); @@ -312,7 +248,6 @@ console.log(music.audio.url); import { generateAudio } from "@tanstack/ai"; import { falAudio } from "@tanstack/ai-fal"; -// DiffRhythm with explicit lyrics const lyrical = await generateAudio({ adapter: falAudio("fal-ai/diffrhythm"), prompt: "An upbeat electronic track with synths", @@ -326,7 +261,6 @@ const lyrical = await generateAudio({ import { generateAudio } from "@tanstack/ai"; import { falAudio } from "@tanstack/ai-fal"; -// Sound effects const sfx = await generateAudio({ adapter: falAudio("fal-ai/elevenlabs/sound-effects/v2"), prompt: "Thunderclap with rain", @@ -334,161 +268,57 @@ const sfx = await generateAudio({ }); ``` -## Popular Models - -### Image Models - -| Model | Description | -|-------|-------------| -| `fal-ai/nano-banana-pro` | Fast, high-quality image generation (4K) | -| `fal-ai/flux-2/klein/9b` | Enhanced realism, crisp text generation | -| `fal-ai/z-image/turbo` | Super fast 6B parameter model | -| `xai/grok-imagine-image` | xAI highly aesthetic images with prompt enhancement | - -### Video Models - -| Model | Mode | Description | -|-------|------|-------------| -| `fal-ai/kling-video/v2.6/pro/text-to-video` | Text-to-Video | High-quality text-to-video | -| `fal-ai/kling-video/v2.6/pro/image-to-video` | Image-to-Video | Animate images with Kling | -| `fal-ai/veo3.1` | Text-to-Video | Google Veo text-to-video | -| `fal-ai/veo3.1/image-to-video` | Image-to-Video | Google Veo image-to-video | -| `xai/grok-imagine-video/text-to-video` | Text-to-Video | xAI video from text | -| `xai/grok-imagine-video/image-to-video` | Image-to-Video | xAI animate images to video | -| `fal-ai/ltx-2/text-to-video/fast` | Text-to-Video | Fast text-to-video | -| `fal-ai/ltx-2/image-to-video/fast` | Image-to-Video | Fast image-to-video animation | - -### Text-to-Speech Models - -| Model | Description | -|-------|-------------| -| `fal-ai/gemini-3.1-flash-tts` | **New** — Google's flagship TTS with 80+ languages and expressive audio tags | -| `fal-ai/elevenlabs/tts/eleven-v3` | ElevenLabs v3 expressive multi-voice TTS | -| `fal-ai/elevenlabs/tts/turbo-v2.5` | Low-latency ElevenLabs TTS | -| `fal-ai/minimax/speech-2.6-hd` | MiniMax HD speech synthesis | -| `fal-ai/minimax/speech-2.6-turbo` | MiniMax low-latency variant | -| `fal-ai/kokoro/american-english` | Kokoro multilingual TTS — also `british-english`, `french`, `spanish`, `italian`, `japanese`, `mandarin-chinese`, `hindi`, `brazilian-portuguese` | -| `fal-ai/inworld-tts` | Inworld TTS-1.5 Max | -| `fal-ai/chatterbox/text-to-speech/multilingual` | Chatterbox multilingual TTS | -| `fal-ai/dia-tts` | Dia expressive dialogue TTS | -| `fal-ai/orpheus-tts` | Orpheus open-source TTS | -| `fal-ai/f5-tts` | F5-TTS voice cloning | -| `fal-ai/vibevoice/7b` | VibeVoice 7B conversational TTS | - -### Transcription Models - -| Model | Description | -|-------|-------------| -| `fal-ai/whisper` | OpenAI Whisper on fal infra | -| `fal-ai/wizper` | Faster-whisper variant with word-level timestamps | -| `fal-ai/speech-to-text/turbo` | Turbo STT with diarization | -| `fal-ai/elevenlabs/speech-to-text` | ElevenLabs STT | - -### Audio / Music Models - -| Model | Mode | Description | -|-------|------|-------------| -| `fal-ai/minimax-music/v2.6` | Music | **New** — MiniMax Music 2.6, full vocal + instrumental compositions from a prompt | -| `fal-ai/minimax-music/v2.5` | Music | MiniMax Music 2.5 | -| `fal-ai/minimax-music/v2` | Music | MiniMax Music v2 — supports `lyrics_prompt` | -| `fal-ai/diffrhythm` | Music | DiffRhythm — prompt + lyrics | -| `fal-ai/lyria2` | Music | Google Lyria 2 high-fidelity music | -| `fal-ai/stable-audio-25/text-to-audio` | Music / Audio | Stability AI Stable Audio 2.5 | -| `fal-ai/mmaudio-v2/text-to-audio` | Audio | MMAudio v2 text-to-audio | -| `fal-ai/elevenlabs/sound-effects/v2` | SFX | ElevenLabs sound-effect generation | -| `fal-ai/beatoven/sound-effect-generation` | SFX | Beatoven professional sound effects | -| `fal-ai/thinksound` | Audio | Thinksound reasoning-based audio generation | - -> **Very new models** (e.g. `gemini-3.1-flash-tts`, `minimax-music/v2.6`, `beatoven/sound-effect-generation`) may not yet appear in `@fal-ai/client`'s type map — they still work as plain string model IDs, you just won't get autocomplete for their `modelOptions`. - -## Environment Variables - -Create an API key at [fal.ai](https://fal.ai) and set it in your environment: - -```bash -FAL_KEY=your-fal-api-key -``` - -## API Reference - -### `falImage(model, config?)` - -Creates a fal.ai image adapter using the `FAL_KEY` environment variable or an explicit config. - -**Parameters:** - -- `model` - The fal.ai model ID (e.g., `"fal-ai/flux/dev"`) -- `config.apiKey?` - Your fal.ai API key (falls back to `FAL_KEY` env var) -- `config.proxyUrl?` - Proxy URL for client-side usage - -**Returns:** A `FalImageAdapter` instance for use with `generateImage()`. - -### `falVideo(model, config?)` - -Creates a fal.ai video adapter using the `FAL_KEY` environment variable or an explicit config. - -**Parameters:** - -- `model` - The fal.ai model ID (e.g., `"fal-ai/kling-video/v2.6/pro/text-to-video"`) -- `config.apiKey?` - Your fal.ai API key (falls back to `FAL_KEY` env var) -- `config.proxyUrl?` - Proxy URL for client-side usage - -**Returns:** A `FalVideoAdapter` instance for use with `generateVideo()` and `getVideoJobStatus()`. - -### `falSpeech(model, config?)` - -Creates a fal.ai text-to-speech adapter. - -**Parameters:** - -- `model` - The fal.ai TTS model ID (e.g., `"fal-ai/kokoro/american-english"`) -- `config.apiKey?` - Your fal.ai API key (falls back to `FAL_KEY` env var) -- `config.proxyUrl?` - Proxy URL for client-side usage - -**Returns:** A `FalSpeechAdapter` instance for use with `generateSpeech()`. The adapter fetches the generated audio URL from fal and returns it as base64 in `result.audio`. - -### `falTranscription(model, config?)` - -Creates a fal.ai transcription (speech-to-text) adapter. - -**Parameters:** +## Popular models -- `model` - The fal.ai STT model ID (e.g., `"fal-ai/whisper"`) -- `config.apiKey?` - Your fal.ai API key (falls back to `FAL_KEY` env var) -- `config.proxyUrl?` - Proxy URL for client-side usage +### Image -**Returns:** A `FalTranscriptionAdapter` instance for use with `generateTranscription()`. +| Model | Notes | +|-------|--------| +| `fal-ai/nano-banana-pro` | Fast 4K | +| `fal-ai/flux-2/klein/9b` | Realism, text | +| `fal-ai/z-image/turbo` | Fast 6B | +| `xai/grok-imagine-image` | Aesthetic | -### `falAudio(model, config?)` +### Video -Creates a fal.ai audio generation adapter (music and sound effects). +| Model | Mode | +|-------|------| +| `fal-ai/kling-video/v2.6/pro/text-to-video` | T2V | +| `fal-ai/kling-video/v2.6/pro/image-to-video` | I2V | +| `fal-ai/veo3.1` / `.../image-to-video` | Veo | +| `xai/grok-imagine-video/text-to-video` / `.../image-to-video` | xAI | +| `fal-ai/ltx-2/text-to-video/fast` / `.../image-to-video/fast` | Fast | -**Parameters:** +### TTS / STT / audio -- `model` - The fal.ai audio model ID (e.g., `"fal-ai/diffrhythm"`, `"fal-ai/minimax-music/v2"`) -- `config.apiKey?` - Your fal.ai API key (falls back to `FAL_KEY` env var) -- `config.proxyUrl?` - Proxy URL for client-side usage +TTS: `fal-ai/gemini-3.1-flash-tts`, `fal-ai/elevenlabs/tts/eleven-v3`, `fal-ai/elevenlabs/tts/turbo-v2.5`, `fal-ai/minimax/speech-2.6-hd`, `fal-ai/kokoro/*`, and more. -**Returns:** A `FalAudioAdapter` instance for use with `generateAudio()`. The result contains a URL at `result.audio.url`. +STT: `fal-ai/whisper`, `fal-ai/wizper`, `fal-ai/speech-to-text/turbo`, `fal-ai/elevenlabs/speech-to-text`. -### `getFalApiKeyFromEnv()` +Music/SFX: `fal-ai/minimax-music/v2.6`, `fal-ai/diffrhythm`, `fal-ai/lyria2`, `fal-ai/elevenlabs/sound-effects/v2`, … -Reads the `FAL_KEY` environment variable. Throws if not set. +Very new models may lack `@fal-ai/client` types — string ids still work. -**Returns:** The API key string. +## API reference -### `configureFalClient(config?)` +| Factory | Activity | +| --- | --- | +| `falImage(model, config?)` | `generateImage()` | +| `falVideo(model, config?)` | `generateVideo()` / `getVideoJobStatus()` | +| `falSpeech(model, config?)` | `generateSpeech()` → base64 audio | +| `falTranscription(model, config?)` | `generateTranscription()` | +| `falAudio(model, config?)` | `generateAudio()` → URL | +| `getFalApiKeyFromEnv()` | Reads `FAL_KEY` | +| `configureFalClient(config?)` | Underlying client setup | -Configures the underlying `@fal-ai/client`. Called automatically by adapter constructors. Uses `proxyUrl` if provided, otherwise sets credentials from the API key. +`config`: `apiKey?`, `proxyUrl?`. -## Limitations +## Notes -- **No text/chat support** — Use OpenAI, Anthropic, Gemini, or another text adapter for `chat()` -- **No tools support** — Tool definitions are not applicable to media generation -- **No summarization** — Use a text adapter for `summarize()` -- **Video is experimental** — The video generation API may change in future releases +- No chat / tools / summarization +- Video API may change -## Next Steps +## Next steps -- [Getting Started](../getting-started/quick-start) - Learn the basics -- [Other Adapters](./openai) - Explore other providers +- [Getting Started](../getting-started/quick-start) +- [Other Adapters](./openai) diff --git a/docs/adapters/gemini.md b/docs/adapters/gemini.md index 0e6286d29..2278a2dab 100644 --- a/docs/adapters/gemini.md +++ b/docs/adapters/gemini.md @@ -2,7 +2,7 @@ title: Google Gemini id: gemini-adapter order: 3 -description: "Use Google Gemini with TanStack AI — text, image generation via Imagen and Gemini native (NanoBanana), and experimental TTS via @tanstack/ai-gemini." +description: "Gemini text, Imagen/NanoBanana images, experimental TTS, and provider tools via @tanstack/ai-gemini." keywords: - tanstack ai - gemini @@ -14,17 +14,24 @@ keywords: - google ai --- -The Google Gemini adapter provides access to Google's Gemini models, including text generation, image generation with both Imagen and Gemini native image models (NanoBanana), and experimental text-to-speech. +If you need Gemini → install, set `GEMINI_API_KEY` or `GOOGLE_API_KEY`, call `geminiText(model)`. -For a full working example with image generation, see the [media generation example app](https://github.com/TanStack/ai/tree/main/examples/ts-react-media). +Media example: [ts-react-media](https://github.com/TanStack/ai/tree/main/examples/ts-react-media). -## Installation +## Install ```bash npm install @tanstack/ai-gemini ``` -## Basic Usage +```bash +GEMINI_API_KEY=your-api-key-here +# or GOOGLE_API_KEY=... +``` + +Key: [Google AI Studio](https://aistudio.google.com/apikey). + +## Do this ```typescript import { chat } from "@tanstack/ai"; @@ -36,14 +43,14 @@ const stream = chat({ }); ``` -## Basic Usage - Custom API Key +### Explicit API key ```typescript import { chat } from "@tanstack/ai"; import { createGeminiChat } from "@tanstack/ai-gemini"; const adapter = createGeminiChat("gemini-3.1-pro-preview", process.env.GEMINI_API_KEY!, { - // ... your config options + // httpOptions.baseUrl, ... }); const stream = chat({ @@ -52,40 +59,7 @@ const stream = chat({ }); ``` -## Configuration - -```typescript -import { createGeminiChat, type GeminiTextConfig } from "@tanstack/ai-gemini"; - -const config: Omit = { - httpOptions: { - baseUrl: "https://generativelanguage.googleapis.com/v1beta", // Optional - }, -}; - -const adapter = createGeminiChat("gemini-3.1-pro-preview", process.env.GEMINI_API_KEY!, config); -``` - - -## Example: Chat Completion - -```typescript -import { chat, toServerSentEventsResponse } from "@tanstack/ai"; -import { geminiText } from "@tanstack/ai-gemini"; - -export async function POST(request: Request) { - const { messages } = await request.json(); - - const stream = chat({ - adapter: geminiText("gemini-3.1-pro-preview"), - messages, - }); - - return toServerSentEventsResponse(stream); -} -``` - -## Example: With Tools +### Server + tools ```typescript import { chat, toServerSentEventsResponse, toolDefinition } from "@tanstack/ai"; @@ -101,7 +75,6 @@ const getCalendarEventsDef = toolDefinition({ }); const getCalendarEvents = getCalendarEventsDef.server(async ({ date }) => { - // Fetch calendar events return { events: [] }; }); @@ -118,21 +91,47 @@ export async function POST(request: Request) { } ``` -## Stateful Conversations — Interactions API (Experimental) +## Model options -Gemini's [Interactions API](https://ai.google.dev/gemini-api/docs/interactions) (currently in Beta) offers server-side conversation state — the Gemini equivalent of OpenAI's Responses API. Instead of replaying the full message history on every turn, you pass a `previous_interaction_id` and the server retains the transcript. This also improves cache hit rates for repeated prefixes. +Sampling in `modelOptions` (camelCase for `geminiText`): -The `geminiTextInteractions` adapter routes through `client.interactions.create` and surfaces the server-assigned interaction id via an AG-UI `CUSTOM` event (`name: 'gemini.interactionId'`) emitted just before `RUN_FINISHED`, so you can chain turns. +```typescript +import { chat } from "@tanstack/ai"; +import { geminiText } from "@tanstack/ai-gemini"; -> **⚠️ Experimental.** Google marks the Interactions API as Beta and explicitly flags possible breaking changes until it reaches general availability. The adapter is exported from the `@tanstack/ai-gemini/experimental` subpath so the experimental status is load-bearing in your editor and bundle. Text output, function tools, and the built-in tools `google_search`, `code_execution`, `url_context`, `file_search`, and `computer_use` are supported. `google_search_retrieval`, `google_maps`, and `mcp_server` still throw on this adapter — use `geminiText()` for those or wait for follow-up work. +const stream = chat({ + adapter: geminiText("gemini-3.1-pro-preview"), + messages: [{ role: "user", content: "Hello!" }], + modelOptions: { + maxOutputTokens: 2048, + temperature: 0.7, + topP: 0.9, + topK: 40, + stopSequences: ["END"], + }, +}); +``` -### Basic Usage +> Root-level sampling migration: [modelOptions](../migration/sampling-options-to-model-options). + +```typescript ignore +modelOptions: { + thinking: { includeThoughts: true }, + // or + responseMimeType: "application/json", +} +``` + +## Interactions API (experimental) + +Stateful conversations via `previous_interaction_id`. Export: `@tanstack/ai-gemini/experimental`. Google Beta — may break. + +**Must:** multi-turn needs `previous_interaction_id` + `store: true`. Fresh interaction with multi-message history throws. Re-pass tools / `system_instruction` / `generation_config` every turn (not inherited). Capture id from CUSTOM `gemini.interactionId`. ```typescript ignore import { chat } from "@tanstack/ai"; import { geminiTextInteractions } from "@tanstack/ai-gemini/experimental"; -// Turn 1: introduce yourself, capture the interaction id. let interactionId: string | undefined; for await (const chunk of chat({ @@ -150,7 +149,6 @@ for await (const chunk of chat({ } } -// Turn 2: only send the new turn's content — the server has the history. for await (const chunk of chat({ adapter: geminiTextInteractions("gemini-3.5-flash"), messages: [{ role: "user", content: "What is my name?" }], @@ -158,15 +156,13 @@ for await (const chunk of chat({ previous_interaction_id: interactionId, }, })) { - // ...stream "Your name is Amir." back to the client. + // stream reply } ``` -### Wiring with `useChat` (React) - -The Interactions API is stateful and **does not accept multi-turn history without a `previous_interaction_id`** — if a chat client sends `[user, assistant, user]` to a fresh interaction the adapter throws `cannot send prior conversation history on a fresh interaction`. To make `useChat` work, persist the server-assigned id and send it back on the next turn: +### useChat wiring -**Server route** (e.g. TanStack Start handler): +**Server:** ```typescript import { @@ -179,8 +175,6 @@ import { geminiTextInteractions } from "@tanstack/ai-gemini/experimental"; export async function POST({ request }: { request: Request }) { const params = await chatParamsFromRequestBody(await request.json()); - // The client sends body.previousInteractionId; AG-UI maps `body` into - // `forwardedProps` on the wire. const previousInteractionId = typeof params.forwardedProps.previousInteractionId === "string" ? params.forwardedProps.previousInteractionId @@ -191,7 +185,7 @@ export async function POST({ request }: { request: Request }) { messages: params.messages, modelOptions: { previous_interaction_id: previousInteractionId, - store: true, // required for chaining on the next turn + store: true, }, }); @@ -199,7 +193,7 @@ export async function POST({ request }: { request: Request }) { } ``` -**React client**: +**Client:** ```tsx import { useEffect, useMemo, useState } from "react"; @@ -228,145 +222,43 @@ function GeminiChat() { }, }); - // Switching provider/model resets the server-side chain — drop the id - // AND the local message history together, otherwise the next turn - // ships multi-turn messages with no previous_interaction_id and the - // adapter errors out. const [provider, setProvider] = useState("gemini-interactions"); useEffect(() => { setInteractionId(undefined); setMessages([]); }, [provider]); - // ...render messages, call sendMessage(input) + // render + sendMessage } ``` -The full working example is in [`examples/ts-react-chat`](https://github.com/TanStack/ai/tree/main/examples/ts-react-chat) — see `src/routes/index.tsx` for the client and `src/routes/api.tanchat.ts` for the route. +Full example: [`examples/ts-react-chat`](https://github.com/TanStack/ai/tree/main/examples/ts-react-chat). -### How it differs from `geminiText` - -| Concern | `geminiText` | `geminiTextInteractions` | +| | `geminiText` | `geminiTextInteractions` | | --- | --- | --- | -| Underlying endpoint | `models:generateContent` | `interactions:create` | -| Conversation state | Stateless — send full history each turn | Stateful — server retains transcript via `previous_interaction_id` | -| Provider options shape | camelCase (`stopSequences`, `responseModalities`, `safetySettings`) | snake_case (`generation_config`, `response_modalities`, `previous_interaction_id`) | -| Built-in tools | `google_search`, `code_execution`, `url_context`, `file_search`, `google_maps`, `google_search_retrieval`, `computer_use` | `google_search`, `code_execution`, `url_context`, `file_search`, `computer_use` (only the first four stream `CUSTOM` event activity; `computer_use` is accepted in the request but does not currently emit per-delta events) | -| Stability | GA | Experimental (Google Beta) | - -### Provider Options - -The adapter exposes Interactions-specific options on `modelOptions`: - -```typescript -import { chat } from "@tanstack/ai"; -import { geminiTextInteractions } from "@tanstack/ai-gemini/experimental"; - -const stream = chat({ - adapter: geminiTextInteractions("gemini-3.5-flash"), - messages: [{ role: "user", content: "Hello!" }], - modelOptions: { - // Stateful chaining — passed only on turn 2+. - previous_interaction_id: "int_abc123", - - // Persist the interaction server-side (default true). Must be true for - // previous_interaction_id to work on the *next* turn. - store: true, - - // Per-request system instruction (interaction-scoped — re-specify each turn). - system_instruction: "You are a helpful assistant.", - - // snake_case generation config distinct from geminiText's camelCase one. - generation_config: { - thinking_level: "low", - thinking_summaries: "auto", - stop_sequences: [""], - }, - - response_modalities: ["text"], - }, -}); -``` - -### Reading the interaction id - -The server's interaction id arrives as an AG-UI `CUSTOM` event emitted just before `RUN_FINISHED`: - -```typescript ignore -for await (const chunk of stream) { - if ( - chunk.type === "CUSTOM" && - chunk.name === "gemini.interactionId" && - typeof chunk.value === "object" && - chunk.value !== null && - "interactionId" in chunk.value - ) { - const id = String(chunk.value.interactionId); - // Persist `id` wherever you store per-user conversation pointers — - // pass it back on the next turn as `previous_interaction_id`. - } -} -``` - -### Caveats - -- **Multi-turn history requires `previous_interaction_id`.** The Interactions API has no stateless replay path — sending more than one message in `messages` without a `previous_interaction_id` throws. Chat UIs that maintain local history must capture the server-assigned id and chain (see [Wiring with `useChat`](#wiring-with-usechat-react)). On provider/model switch, also clear the local message buffer. -- **Tools, `system_instruction`, and `generation_config` are interaction-scoped.** Per Google's docs these are NOT inherited from a prior interaction via `previous_interaction_id` — pass them again on every turn you need them. -- `store: false` is incompatible with `previous_interaction_id` (no state to recall) and with `background: true`. -- Retention (as of the time of writing): **55 days on the Paid Tier, 1 day on the Free Tier.** See [Google's Interactions API docs](https://ai.google.dev/gemini-api/docs/interactions) for current retention policy. -- Built-in tools in scope (`google_search`, `code_execution`, `url_context`, `file_search`, `computer_use`) are wired through as request tools. Per-delta activity for the four search/exec tools streams back as AG-UI `CUSTOM` events — `gemini.googleSearchCall` / `gemini.googleSearchResult` (and the matching `codeExecutionCall`/`Result`, `urlContextCall`/`Result`, `fileSearchCall`/`Result`) — carrying the raw Interactions delta. `computer_use` is accepted in the request but the Interactions API does not currently emit per-delta `CUSTOM` events for it. Function-tool `TOOL_CALL_*` events are unchanged, and `finishReason` stays `stop` when only built-in tools ran. -- `google_search_retrieval`, `google_maps`, and `mcp_server` still throw a targeted error on this adapter. Use `geminiText()` for the first two, or wait for a dedicated follow-up for `mcp_server`. -- Image and audio output via Interactions aren't routed through this adapter yet — it's text-only. Use `geminiImage` / `geminiSpeech` for non-text generation for now. - -## Model Options - -Gemini supports various model-specific options. Sampling parameters live here too — `temperature`, `topP`, and `maxOutputTokens` — rather than as root-level props on `chat()`: +| Endpoint | `generateContent` | `interactions:create` | +| State | Full history each turn | Server via `previous_interaction_id` | +| Options shape | camelCase | snake_case | +| Stability | GA | Experimental | ```typescript -import { chat } from "@tanstack/ai"; -import { geminiText } from "@tanstack/ai-gemini"; - -const stream = chat({ - adapter: geminiText("gemini-3.1-pro-preview"), - messages: [{ role: "user", content: "Hello!" }], - modelOptions: { - maxOutputTokens: 2048, - temperature: 0.7, - topP: 0.9, - topK: 40, - stopSequences: ["END"], - }, -}); -``` - -> If you previously passed `temperature` / `topP` / `maxTokens` at the root of `chat()`, see [Moving Sampling Options into modelOptions](../migration/sampling-options-to-model-options). - -### Thinking - -Enable thinking for models that support it: - -```typescript ignore modelOptions: { - thinking: { - includeThoughts: true, + previous_interaction_id: "int_abc123", + store: true, + system_instruction: "You are a helpful assistant.", + generation_config: { + thinking_level: "low", + thinking_summaries: "auto", + stop_sequences: [""], }, + response_modalities: ["text"], } ``` -### Structured Output - -Configure structured output format: - -```typescript ignore -modelOptions: { - responseMimeType: "application/json", -} -``` +**Notes:** Retention ~55 days paid / 1 day free. `google_search_retrieval`, `google_maps`, `mcp_server` throw here — use `geminiText()`. Text-only for now. ## Summarization -Summarize long text content: - ```typescript import { summarize } from "@tanstack/ai"; import { geminiSummarize } from "@tanstack/ai-gemini"; @@ -375,24 +267,17 @@ const result = await summarize({ adapter: geminiSummarize("gemini-3.1-pro-preview"), text: "Your long text to summarize...", maxLength: 100, - style: "concise", // "concise" | "bullet-points" | "paragraph" + style: "concise", }); console.log(result.summary); ``` -## Image Generation - -The Gemini adapter supports two types of image generation: - -- **Gemini native image models** (NanoBanana) — Use the `generateContent` API with models like `gemini-3.1-flash-image-preview`. These support extended resolution tiers (1K, 2K, 4K) and aspect ratio control. -- **Imagen models** — Use the `generateImages` API with models like `imagen-4.0-generate-001`. These are dedicated image generation models with WIDTHxHEIGHT sizing. - -The adapter automatically routes to the correct API based on the model name — models starting with `gemini-` use `generateContent`, while `imagen-` models use `generateImages`. +## Image generation -### Example: Gemini Native Image Generation (NanoBanana) +Routes by model: `gemini-*` → `generateContent` (NanoBanana); `imagen-*` → `generateImages`. -From the [media generation example app](https://github.com/TanStack/ai/tree/main/examples/ts-react-media): +### NanoBanana ```typescript import { generateImage } from "@tanstack/ai"; @@ -408,7 +293,9 @@ const result = await generateImage({ console.log(result.images); ``` -### Example: Imagen +Size: `"aspectRatio_resolution"` — ratios `1:1`–`21:9`; res `1K`/`2K`/`4K`. + +### Imagen ```typescript import { generateImage } from "@tanstack/ai"; @@ -418,76 +305,26 @@ const result = await generateImage({ adapter: geminiImage("imagen-4.0-generate-001"), prompt: "A futuristic cityscape at sunset", numberOfImages: 1, -}); - -console.log(result.images); -``` - -### Image Size Options - -#### Gemini Native Models (NanoBanana) - -Gemini native image models use a template literal size format combining aspect ratio and resolution tier: - -```typescript ignore -// Format: "aspectRatio_resolution" -size: "16:9_4K" -size: "1:1_2K" -size: "9:16_1K" -``` - -| Component | Values | -|-----------|--------| -| Aspect Ratio | `1:1`, `2:3`, `3:2`, `3:4`, `4:3`, `9:16`, `16:9`, `21:9` | -| Resolution | `1K`, `2K`, `4K` | - -#### Imagen Models - -Imagen models use WIDTHxHEIGHT format, which maps to aspect ratios internally: - -| Size | Aspect Ratio | -|------|-------------| -| `1024x1024` | 1:1 | -| `1920x1080` | 16:9 | -| `1080x1920` | 9:16 | - -Alternatively, you can specify the aspect ratio directly in Model Options: - -```typescript -import { generateImage } from "@tanstack/ai"; -import { geminiImage } from "@tanstack/ai-gemini"; - -const result = await generateImage({ - adapter: geminiImage("imagen-4.0-generate-001"), - prompt: "A landscape photo", modelOptions: { aspectRatio: "16:9", + personGeneration: "DONT_ALLOW", + safetyFilterLevel: "BLOCK_SOME", }, }); ``` -### Image Model Options +Sizes: `1024x1024` (1:1), `1920x1080` (16:9), `1080x1920` (9:16). -```typescript ignore -import { generateImage } from "@tanstack/ai"; -import { geminiImage } from "@tanstack/ai-gemini"; +### Models -const result = await generateImage({ - adapter: geminiImage("imagen-4.0-generate-001"), - prompt: "...", - modelOptions: { - aspectRatio: "16:9", // "1:1" | "3:4" | "4:3" | "9:16" | "16:9" - personGeneration: "DONT_ALLOW", // Control person generation - safetyFilterLevel: "BLOCK_SOME", // Safety filtering - }, -}); -``` - -## Text-to-Speech (Experimental) +| Gemini native | Imagen | +| --- | --- | +| `gemini-3.1-flash-image-preview` | `imagen-4.0-ultra-generate-001` | +| `gemini-3.1-flash-lite-image` | `imagen-4.0-generate-001` | +| `gemini-3-pro-image-preview` | `imagen-4.0-fast-generate-001` | +| `gemini-2.5-flash-image` | | -> **Note:** Gemini TTS is experimental and may require the Live API for full functionality. - -Generate speech from text: +## Text-to-speech (experimental) ```typescript import { generateSpeech } from "@tanstack/ai"; @@ -498,109 +335,28 @@ const result = await generateSpeech({ text: "Hello from Gemini TTS!", }); -console.log(result.audio); // Base64 encoded audio +console.log(result.audio); ``` -## Environment Variables - -Set your API key in environment variables: - -```bash -GEMINI_API_KEY=your-api-key-here -# or -GOOGLE_API_KEY=your-api-key-here -``` - -## Getting an API Key - -1. Go to [Google AI Studio](https://aistudio.google.com/apikey) -2. Create a new API key -3. Add it to your environment variables - -## Popular Image Models - -### Gemini Native Image Models (NanoBanana) - -These models use the `generateContent` API and support resolution tiers (1K, 2K, 4K). - -| Model | Description | -|-------|-------------| -| `gemini-3.1-flash-image-preview` | Latest and fastest Gemini native image generation | -| `gemini-3.1-flash-lite-image` | Nano Banana 2 Lite — ultra-low-latency, low-cost image generation | -| `gemini-3-pro-image-preview` | Higher quality Gemini native image generation | -| `gemini-2.5-flash-image` | Gemini 2.5 Flash with image generation | - -### Imagen Models - -These models use the dedicated `generateImages` API. - -| Model | Description | -|-------|-------------| -| `imagen-4.0-ultra-generate-001` | Best quality Imagen image generation | -| `imagen-4.0-generate-001` | High quality Imagen image generation | -| `imagen-4.0-fast-generate-001` | Fast Imagen image generation | - -## API Reference - -Every factory pair follows the same shape: the short factory (`geminiText`, `geminiImage`, …) reads `GEMINI_API_KEY` (or `GOOGLE_API_KEY`) from the environment, while the `create*` variant takes an explicit API key. Both take `model` as the first argument. - -### `geminiText(model, config?)` / `createGeminiChat(model, apiKey, config?)` - -Creates a Gemini text/chat adapter. - -**Parameters:** - -- `model` - Gemini chat model id (e.g. `"gemini-3.1-pro-preview"`) -- `config?.baseURL` - Custom base URL (optional) - -### `geminiTextInteractions(model, config?)` / `createGeminiTextInteractions(model, apiKey, config?)` (experimental) - -Creates a Gemini Interactions API text adapter. Backs the stateful conversation pattern via `previous_interaction_id`. Exported from `@tanstack/ai-gemini/experimental`. +## API reference -**Parameters:** +Short factories use env key; `create*` takes explicit key. -- `model` - Gemini chat model id (e.g. `"gemini-3.5-flash"`) -- `config?.baseURL` - Custom base URL (optional) +| Factory | Purpose | +| --- | --- | +| `geminiText` / `createGeminiChat` | Chat | +| `geminiTextInteractions` / `createGeminiTextInteractions` | Interactions (experimental) | +| `geminiSummarize` / `createGeminiSummarize` | Summarize | +| `geminiImage` / `createGeminiImage` | Image | +| `geminiSpeech` / `createGeminiSpeech` | TTS (experimental) | +| `geminiAudio` / `createGeminiAudio` | Lyria music (experimental) | -### `geminiSummarize(model, config?)` / `createGeminiSummarize(model, apiKey, config?)` +## Provider tools -Creates a Gemini summarization adapter. - -### `geminiImage(model, config?)` / `createGeminiImage(model, apiKey, config?)` - -Creates a Gemini image adapter. Automatically routes to the correct API based on the model name — `gemini-*` models use `generateContent`, `imagen-*` models use `generateImages`. - -### `geminiSpeech(model, config?)` / `createGeminiSpeech(model, apiKey, config?)` - -Creates a Gemini text-to-speech adapter. _Experimental._ - -### `geminiAudio(model, config?)` / `createGeminiAudio(model, apiKey, config?)` - -Creates a Gemini Lyria music generation adapter. _Experimental._ - -## Next Steps - -- [Image Generation Guide](../media/image-generation) - Learn more about image generation -- [Media Generation Example](https://github.com/TanStack/ai/tree/main/examples/ts-react-media) - Full working example with Gemini and fal.ai -- [Getting Started](../getting-started/quick-start) - Learn the basics -- [Tools Guide](../tools/tools) - Learn about tools -- [Other Adapters](./openai) - Explore other providers - -## Provider Tools - -Google Gemini exposes several native tools beyond user-defined function calls. -Import them from `@tanstack/ai-gemini/tools` and pass them into -`chat({ tools: [...] })`. - -> For the full concept, a comparison matrix, and type-gating details, see -> [Provider Tools](../tools/provider-tools.md). +From `@tanstack/ai-gemini/tools`. Matrix: [Provider Tools](../tools/provider-tools.md). ### `codeExecutionTool` -Enables Gemini to execute Python code in a sandboxed environment and return -results inline. Takes no arguments — include it in the `tools` array to -activate code execution. - ```typescript import { chat } from "@tanstack/ai"; import { geminiText } from "@tanstack/ai-gemini"; @@ -613,13 +369,8 @@ const stream = chat({ }); ``` -**Supported models:** Gemini 1.5 Pro, Gemini 2.x, Gemini 2.5 and above. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). - ### `fileSearchTool` -Searches files that have been uploaded to the Gemini File API. Pass a -`FileSearch` config object with the corpus and file IDs to scope the search. - ```typescript import { chat } from "@tanstack/ai"; import { geminiText } from "@tanstack/ai-gemini"; @@ -636,14 +387,8 @@ const stream = chat({ }); ``` -**Supported models:** Gemini 2.x and above. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). - ### `googleSearchTool` -Enables Gemini to query Google Search and incorporate grounded search results -into its response. Pass an optional `GoogleSearch` config or call with no -arguments to use defaults. - ```typescript import { chat } from "@tanstack/ai"; import { geminiText } from "@tanstack/ai-gemini"; @@ -656,14 +401,8 @@ const stream = chat({ }); ``` -**Supported models:** Gemini 1.5 Pro, Gemini 2.x, Gemini 2.5. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). - ### `googleSearchRetrievalTool` -A retrieval-augmented variant of Google Search that returns ranked passages -from the web with configurable dynamic retrieval mode. Pass an optional -`GoogleSearchRetrieval` config. - ```typescript ignore import { chat } from "@tanstack/ai"; import { geminiText } from "@tanstack/ai-gemini"; @@ -680,14 +419,8 @@ const stream = chat({ }); ``` -**Supported models:** Gemini 1.5 Pro and above. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). - ### `googleMapsTool` -Connects Gemini to the Google Maps API for location-aware queries such as -directions, place search, and geocoding. Pass an optional `GoogleMaps` config -or call with no arguments. - ```typescript import { chat } from "@tanstack/ai"; import { geminiText } from "@tanstack/ai-gemini"; @@ -700,13 +433,8 @@ const stream = chat({ }); ``` -**Supported models:** Gemini 2.5 and above. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). - ### `urlContextTool` -Fetches and includes the content of URLs mentioned in the conversation so -Gemini can reason over live web pages. Takes no arguments. - ```typescript import { chat } from "@tanstack/ai"; import { geminiText } from "@tanstack/ai-gemini"; @@ -719,14 +447,8 @@ const stream = chat({ }); ``` -**Supported models:** Gemini 2.x and above. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). - ### `computerUseTool` -Allows Gemini to observe a virtual desktop via screenshots and interact with -it using predefined computer-use functions. Provide the `environment` and -optionally restrict callable functions via `excludedPredefinedFunctions`. - ```typescript ignore import { chat } from "@tanstack/ai"; import { geminiText } from "@tanstack/ai-gemini"; @@ -743,4 +465,10 @@ const stream = chat({ }); ``` -**Supported models:** Gemini 2.5 and above. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). +## Next steps + +- [Image Generation](../media/image-generation) +- [Media example](https://github.com/TanStack/ai/tree/main/examples/ts-react-media) +- [Getting Started](../getting-started/quick-start) +- [Tools](../tools/tools) +- [Other Adapters](./openai) diff --git a/docs/adapters/grok-build.md b/docs/adapters/grok-build.md index 5bfaef731..9d77cb8a3 100644 --- a/docs/adapters/grok-build.md +++ b/docs/adapters/grok-build.md @@ -2,7 +2,7 @@ title: Grok Build id: grok-build-adapter order: 15 -description: "Use xAI's Grok Build coding agent as a chat backend in TanStack AI — a sandbox harness that runs the grok CLI against a real workspace, with tool bridging via @tanstack/ai-grok-build." +description: "xAI Grok Build coding agent harness — sandbox-required, via @tanstack/ai-grok-build." keywords: - tanstack ai - grok @@ -14,40 +14,24 @@ keywords: - adapter --- -The Grok Build adapter runs xAI's **Grok Build** coding agent as a chat backend. -Unlike HTTP provider adapters, this is a **harness adapter**: Grok Build runs -its own agent loop and executes its own tools — shell commands, file edits, -search — by spawning the `grok` CLI **inside a sandbox**. Each `chat()` call runs -one full harness turn; the harness's tool activity streams back as -already-resolved tool-call events your UI can render. +If you need Grok Build as a coding agent → install, provide a sandbox, call `grokBuildText` with `withSandbox(...)`. -> **Requires a sandbox.** `grok-build` declares `requires: [SandboxCapability]`, -> so `chat()` errors at the call site unless you provide a sandbox with -> `withSandbox(...)` middleware. The sandbox — your laptop, a Docker container, -> or a cloud VM — is the filesystem and safety boundary the agent runs in. See -> the [Sandboxes overview](../sandbox/overview) for the full picture. +> **Requires a sandbox.** `chat()` errors without `withSandbox(...)`. Sandbox is the FS + safety boundary. See [Sandboxes](../sandbox/overview). -## Installation +## Install ```bash npm install @tanstack/ai-grok-build @tanstack/ai-sandbox ``` -You also need a sandbox provider (e.g. `@tanstack/ai-sandbox-docker`) and the -`grok` CLI available inside the sandbox image. +Also need a provider (e.g. `@tanstack/ai-sandbox-docker`) and `grok` CLI in the image. -## Authentication +## Auth -Grok Build resolves credentials the same way the `grok` CLI does: +1. `XAI_API_KEY` as workspace secret (headless/sandbox), or +2. grok.com browser login (local) -- the `XAI_API_KEY` environment variable (headless / sandbox — inject it as a - workspace secret), or -- an existing grok.com browser login on the machine (local dev). - -The two auth modes expose the model under slightly different ids; the adapter -maps the short alias for you (see [Models](#models)). - -## Basic Usage +## Do this ```ts import { chat } from '@tanstack/ai' @@ -82,47 +66,31 @@ const stream = chat({ ## Models -Grok Build accepts any xAI model id its backend supports; the known ids get -autocomplete (any string is allowed): - | Model id | Notes | | --- | --- | -| `grok-build` | The short alias. With a grok.com browser login the CLI lists it under this name. | -| `grok-build-0.1` | The fully-qualified id the CLI lists when authenticated with `XAI_API_KEY`. | -| `composer-2.5` | Also runnable through the Grok Build harness. | +| `grok-build` | Short alias (browser login); maps to `grok-build-0.1` | +| `grok-build-0.1` | Full id under `XAI_API_KEY` | +| `composer-2.5` | Also runnable | -Pass any of these to `grokBuildText(...)` — the adapter resolves `grok-build` to -the CLI's `grok-build-0.1` automatically, so the same code works under both auth -modes. +Any xAI model id works; known ids get autocomplete. ## Configuration -Adapter config (second argument to `grokBuildText`): - -| Option | Description | -| ---------------- | --------------------------------------------------------------------------- | -| `cwd` | Working directory inside the sandbox. Defaults to `/workspace`. | -| `grokExecutable` | Path/name of the `grok` executable inside the sandbox. Defaults to `grok`. | -| `env` | Extra environment variables for the `grok` process inside the sandbox. | -| `emitDiff` | Emit a `file.changed` CUSTOM event with the working-tree `git diff` after the run. Defaults to `true`. | -| `extraArgs` | Extra raw CLI flags appended verbatim (advanced). | - -Per-call overrides go through `modelOptions`: +| Option | Description | +| --- | --- | +| `cwd` | Working dir in sandbox (default `/workspace`) | +| `grokExecutable` | CLI path (default `grok`) | +| `env` | Extra env for `grok` | +| `emitDiff` | `file.changed` CUSTOM with git diff (default `true`) | +| `extraArgs` | Extra CLI flags | -| `modelOptions` | Description | -| --------------- | ------------------------------------------------------------ | -| `sessionId` | Resume an existing Grok Build session (see below). | -| `cwd` | Per-call override of the harness working directory. | -| `maxTurns` | Per-call cap on the number of harness turns. | +`modelOptions`: `sessionId`, `cwd`, `maxTurns`. -## Stateful Sessions +## Stateful sessions -Grok Build sessions are stateful — the harness keeps the working context (files -read, commands run, conclusions reached) between turns. The adapter surfaces the -session id of every fresh run as a custom stream event named -`grok-build.session-id`; thread it back via `modelOptions.sessionId` to resume. -When resuming, only the latest user message is sent — the harness already holds -the prior context. +1. Capture `grok-build.session-id` CUSTOM event. +2. Pass `modelOptions.sessionId`. +3. Send only the latest user message. ```ts import { chat, chatParamsFromRequest, toServerSentEventsResponse } from '@tanstack/ai' @@ -150,29 +118,14 @@ export async function POST(request: Request) { ## Tools -Two kinds of tools flow through this adapter: - -1. **Built-in harness tools** are executed by Grok Build itself (shell, file - edits, search) and stream back as tool-call events with results already - attached. Your code never executes them. -2. **Your TanStack tools** are bridged *into* the harness over an authenticated - MCP tool-proxy: define them with `toolDefinition().server()` and pass them to - `chat({ tools })`. Tool-call events come back under the names you registered. - Because the harness runs in a sandbox, see [Sandbox tools](../sandbox/tools) - for how the bridge reaches your host across providers (local/Docker vs cloud). - -**Client-side and approval-gated tools are not supported** — the harness runs -tools inside a live process and can't pause across an HTTP round-trip. A tool -without a server `execute()` (or marked `needsApproval`) fails fast; run those -with a regular provider adapter. - -## Limitations - -- **Requires a sandbox.** Always run it under `withSandbox(...)`; see the - [Sandboxes overview](../sandbox/overview). -- **Server-only (Node).** The harness spawns the `grok` CLI in a sandbox. -- **The harness owns the agent loop.** TanStack's agent-loop strategies and - per-iteration middleware don't apply inside a harness turn. -- **No sampling controls.** `temperature`-style options don't exist here. -- **Cold starts.** Each call runs a full harness turn; expect higher first-token - latency than HTTP adapters. +1. **Harness tools** — shell, edits, search (results attached). +2. **Your tools** — `toolDefinition().server()` + MCP bridge. See [Sandbox tools](../sandbox/tools). + +No client-side / `needsApproval` tools — fails fast. + +## Notes + +- Always `withSandbox(...)` +- Server-only (Node) +- Harness owns agent loop; no sampling controls +- Higher cold-start latency diff --git a/docs/adapters/grok.md b/docs/adapters/grok.md index 832ac9986..7ad562a1a 100644 --- a/docs/adapters/grok.md +++ b/docs/adapters/grok.md @@ -2,7 +2,7 @@ title: Grok (xAI) id: grok-adapter order: 5 -description: "Use xAI Grok models with TanStack AI — Grok 4.3, Grok Build 0.1, Grok Imagine image generation, and Grok Imagine video generation via @tanstack/ai-grok." +description: "xAI Grok text, image, video, TTS, STT, and realtime via @tanstack/ai-grok." keywords: - tanstack ai - grok @@ -15,15 +15,21 @@ keywords: - adapter --- -The Grok text and summarization adapters provide access to xAI's Responses API for `grok-4.3` and `grok-build-0.1`, plus Grok Imagine image generation and Grok Imagine video generation. +If you need xAI Grok → install, set `XAI_API_KEY`, call `grokText(model)`. -## Installation +Text uses xAI **Responses API**. Defaults: `store: false`, `include: ["reasoning.encrypted_content"]`. + +## Install ```bash npm install @tanstack/ai-grok ``` -## Basic Usage +```bash +XAI_API_KEY=xai-... +``` + +## Do this ```typescript import { chat } from "@tanstack/ai"; @@ -35,7 +41,7 @@ const stream = chat({ }); ``` -## Basic Usage - Custom API Key +### Explicit API key ```typescript import { chat } from "@tanstack/ai"; @@ -49,37 +55,7 @@ const stream = chat({ }); ``` -## Configuration - -```typescript -import { createGrokText, type GrokTextConfig } from "@tanstack/ai-grok"; - -const config: Omit = { - baseURL: "https://api.x.ai/v1", // Optional, this is the default -}; - -const adapter = createGrokText("grok-build-0.1", process.env.XAI_API_KEY!, config); -``` - -## Example: Chat Completion - -```typescript -import { chat, toServerSentEventsResponse } from "@tanstack/ai"; -import { grokText } from "@tanstack/ai-grok"; - -export async function POST(request: Request) { - const { messages } = await request.json(); - - const stream = chat({ - adapter: grokText("grok-build-0.1"), - messages, - }); - - return toServerSentEventsResponse(stream); -} -``` - -## Example: With Tools +### Server + tools ```typescript import { chat, toServerSentEventsResponse, toolDefinition } from "@tanstack/ai"; @@ -95,7 +71,6 @@ const getWeatherDef = toolDefinition({ }); const getWeather = getWeatherDef.server(async ({ location }) => { - // Fetch weather data return { temperature: 72, conditions: "sunny" }; }); @@ -112,9 +87,7 @@ export async function POST(request: Request) { } ``` -## Model Options - -Grok supports xAI Responses API options. Sampling parameters live here too — `temperature`, `top_p`, and `max_output_tokens` — rather than as root-level props on `chat()`: +## Model options ```typescript import { chat, toServerSentEventsResponse } from "@tanstack/ai"; @@ -139,17 +112,10 @@ export async function POST(request: Request) { } ``` -> If you previously passed `temperature` / `topP` / `maxTokens` at the root of `chat()`, see [Moving Sampling Options into modelOptions](../migration/sampling-options-to-model-options). +> Root-level sampling migration: [modelOptions](../migration/sampling-options-to-model-options). ## Summarization -Summarize long text content: - - - ```typescript ignore import { summarize } from "@tanstack/ai"; import { grokSummarize } from "@tanstack/ai-grok"; @@ -158,15 +124,13 @@ const result = await summarize({ adapter: grokSummarize("grok-4.3"), text: "Your long text to summarize...", maxLength: 100, - style: "concise", // "concise" | "bullet-points" | "paragraph" + style: "concise", }); console.log(result.summary); ``` -## Image Generation - -Generate images with Grok 2 Image: +## Image ```typescript import { generateImage } from "@tanstack/ai"; @@ -181,9 +145,7 @@ const result = await generateImage({ console.log(result.images); ``` -The grok-imagine models (`grok-imagine-image`, `grok-imagine-image-quality`) -are aspect-ratio sized — `size` takes an `aspectRatio_resolution` template -like `"16:9_2k"` (the `_2k` suffix is optional): +Imagine models use `aspectRatio_resolution` (`"16:9_2k"`, `_2k` optional): ```typescript import { generateImage } from "@tanstack/ai"; @@ -196,13 +158,9 @@ const result = await generateImage({ }); ``` -### Image Editing (image-to-image) +### Image editing -The grok-imagine models accept image prompt parts for image-conditioned -generation via xAI's `/v1/images/edits` endpoint — up to 3 source images, -addressed by xAI in the order they appear in the prompt. Per xAI's docs -there is no in-prompt referencing syntax; write the prompt naturally and -your text is sent verbatim: +Up to 3 source images (order = xAI order). URLs must be public; use `data` for private. `grok-2-image-1212` is text-only. ```typescript import { generateImage } from "@tanstack/ai"; @@ -227,21 +185,12 @@ const result = await generateImage({ }); ``` -URL sources are fetched by xAI's servers, so they must be publicly -reachable; use a `data` source for private images. `grok-2-image-1212` is -text-to-image only — image prompt parts are a compile-time type error and -throw at runtime. - -## Video Generation (Experimental) - -Generate short video clips (1–15 seconds, with audio) with the Grok Imagine video models via xAI's asynchronous jobs/polling API. - -Available models: +## Video (experimental) -- `grok-imagine-video` (v1.0) — text-to-video and image-to-video, $0.05 per second of video. -- `grok-imagine-video-1.5` — **image-to-video only**, $0.08 per second of video. A text-only prompt is rejected by the API; the adapter fails fast with a clear error telling you to add a starting-frame image or use `grok-imagine-video`. +1–15s with audio. Poll jobs. Full flow: [Video Generation](../media/video-generation). -Text-to-video with the base `grok-imagine-video` model: +- `grok-imagine-video` — T2V + I2V +- `grok-imagine-video-1.5` — **I2V only** (text-only prompt fails fast) ```typescript import { generateVideo, getVideoJobStatus } from "@tanstack/ai"; @@ -249,25 +198,23 @@ import { grokVideo } from "@tanstack/ai-grok"; const adapter = grokVideo("grok-imagine-video"); -// 1. Create the job const { jobId } = await generateVideo({ adapter, prompt: "A red panda balancing on a bamboo stalk in the rain", - size: "16:9_720p", // "aspectRatio" or "aspectRatio_resolution" - duration: 5, // integer seconds, 1–15 + size: "16:9_720p", + duration: 5, }); -// 2. Poll until complete, then read the video URL let status = await getVideoJobStatus({ adapter, jobId }); while (status.status !== "completed" && status.status !== "failed") { await new Promise((r) => setTimeout(r, 5000)); status = await getVideoJobStatus({ adapter, jobId }); } -console.log(status.url); // hosted .mp4 URL +console.log(status.url); ``` -For image-to-video (required for `grok-imagine-video-1.5`, optional for `grok-imagine-video`), include an `image` prompt part as the starting frame and describe the desired motion in the text part. URL sources are fetched by xAI's servers (so they must be publicly reachable); use a `data` source for a base64 starting frame: +I2V: ```typescript import { generateVideo } from "@tanstack/ai"; @@ -290,15 +237,9 @@ const { jobId } = await generateVideo({ }); ``` -Like the Grok Imagine image models, sizing is aspect-ratio based: the `size` option takes an `aspectRatio_resolution` template. Supported aspect ratios are `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `3:2`, and `2:3`; supported resolutions are `480p`, `720p`, and `1080p` (e.g. `"9:16_1080p"`). The resolution suffix is optional. - -When the job completes, the adapter reports usage on the result: `usage.unitsBilled` carries the billed seconds of video and `usage.cost` the exact cost in USD, both as returned by the xAI API. - -See [Video Generation](../media/video-generation) for the full jobs/polling flow, streaming mode, and the `useGenerateVideo` hook. - -## Text-to-Speech +Ratios: `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `3:2`, `2:3`. Res: `480p`, `720p`, `1080p`. Usage: `usage.unitsBilled` (seconds), `usage.cost` (USD). -Generate speech with Grok TTS: +## TTS / STT ```typescript import { generateSpeech } from "@tanstack/ai"; @@ -311,13 +252,9 @@ const result = await generateSpeech({ format: "mp3", }); -console.log(result.audio); // Base64-encoded audio +console.log(result.audio); ``` -## Transcription - -Transcribe audio with Grok STT: - ```typescript import { generateTranscription } from "@tanstack/ai"; import { grokTranscription } from "@tanstack/ai-grok"; @@ -331,94 +268,28 @@ const result = await generateTranscription({ console.log(result.text); ``` -## Realtime Voice - -Grok also exposes a Realtime voice adapter (`grokRealtime`) and a token issuer (`grokRealtimeToken`) for low-latency voice conversations. See [Realtime Voice Chat](../media/realtime-chat) for the end-to-end flow. - -## Environment Variables - -Set your API key in environment variables: - -```bash -XAI_API_KEY=xai-... -``` - -## Implementation Notes - -### Responses API - -The Grok text and summarize adapters use xAI's **Responses API** (`/v1/responses`). Requests default to `store: false` and include encrypted reasoning content with `include: ["reasoning.encrypted_content"]`; both can be overridden through `modelOptions`. - -The shared Responses implementation supports streaming text, reasoning events, structured output via `text.format`, and user-defined function tools. - -## API Reference - -### `grokText(model, config?)` - -Creates a Grok text adapter using environment variables. - -**Parameters:** - -- `model` - The model name (`'grok-4.3'` or `'grok-build-0.1'`) -- `config.baseURL?` - Custom base URL (optional) - -**Returns:** A Grok text adapter instance. - -### `createGrokText(model, apiKey, config?)` - -Creates a Grok text adapter with an explicit API key. - -**Parameters:** - -- `model` - The model name -- `apiKey` - Your xAI API key -- `config.baseURL?` - Custom base URL (optional) - -**Returns:** A Grok text adapter instance. - -### `grokSummarize(model, config?)` - -Creates a Grok summarization adapter using environment variables. - -**Returns:** A Grok summarize adapter instance. - -### `createGrokSummarize(model, apiKey, config?)` - -Creates a Grok summarization adapter with an explicit API key. - -**Returns:** A Grok summarize adapter instance. - -### `grokImage(model, config?)` / `createGrokImage(model, apiKey, config?)` - -Creates a Grok image generation adapter. - -### `grokVideo(model, config?)` / `createGrokVideo(model, apiKey, config?)` - -Creates a Grok video generation adapter (experimental) for the Grok Imagine video models (`'grok-imagine-video'`, `'grok-imagine-video-1.5'`). - -### `grokSpeech(model, config?)` / `createGrokSpeech(model, apiKey, config?)` - -Creates a Grok text-to-speech adapter. - -### `grokTranscription(model, config?)` / `createGrokTranscription(model, apiKey, config?)` - -Creates a Grok speech-to-text adapter. +## Realtime voice -### `grokRealtime(...)` / `grokRealtimeToken(...)` +`grokRealtime` / `grokRealtimeToken` — see [Realtime Voice Chat](../media/realtime-chat). -Realtime voice adapter and token issuer. See [Realtime Voice Chat](../media/realtime-chat) for usage. +## API reference -## Next Steps +| Factory | Purpose | +| --- | --- | +| `grokText` / `createGrokText` | Chat (`grok-4.3`, `grok-build-0.1`) | +| `grokSummarize` / `createGrokSummarize` | Summarize | +| `grokImage` / `createGrokImage` | Image | +| `grokVideo` / `createGrokVideo` | Video (experimental) | +| `grokSpeech` / `createGrokSpeech` | TTS | +| `grokTranscription` / `createGrokTranscription` | STT | +| `grokRealtime` / `grokRealtimeToken` | Realtime | -- [Getting Started](../getting-started/quick-start) - Learn the basics -- [Tools Guide](../tools/tools) - Learn about tools -- [Other Adapters](./openai) - Explore other providers +`config.baseURL?` optional (default `https://api.x.ai/v1`). -## Provider Tools +No provider-tool factories — use `toolDefinition()` ([tools](../tools/tools.md)). -Grok does not currently expose provider-specific tool factories. -Define your own tools with `toolDefinition()` from `@tanstack/ai`. +## Next steps -See [Tools](../tools/tools.md) for the general tool-definition flow, or -[Provider Tools](../tools/provider-tools.md) for other providers' -native-tool offerings. +- [Getting Started](../getting-started/quick-start) +- [Tools](../tools/tools) +- [Other Adapters](./openai) diff --git a/docs/adapters/groq.md b/docs/adapters/groq.md index 00729d06c..2f312266c 100644 --- a/docs/adapters/groq.md +++ b/docs/adapters/groq.md @@ -2,7 +2,7 @@ title: Groq id: groq-adapter order: 6 -description: "Use Groq's fast inference API with TanStack AI for low-latency LLM responses and Whisper transcription — Llama and other open-weight models via @tanstack/ai-groq." +description: "Groq fast chat inference and Whisper transcription via @tanstack/ai-groq." keywords: - tanstack ai - groq @@ -15,15 +15,19 @@ keywords: - transcription --- -The Groq adapter provides access to Groq's fast inference API, featuring the world's fastest LLM inference and Whisper-based audio transcription. +If you need Groq → install, set `GROQ_API_KEY`, call `groqText(model)`. -## Installation +## Install ```bash npm install @tanstack/ai-groq ``` -## Basic Usage +```bash +GROQ_API_KEY=gsk_... +``` + +## Do this ```typescript import { chat } from "@tanstack/ai"; @@ -35,15 +39,16 @@ const stream = chat({ }); ``` -## Basic Usage - Custom API Key +### Explicit API key ```typescript import { chat } from "@tanstack/ai"; import { createGroqText } from "@tanstack/ai-groq"; -const adapter = createGroqText("llama-3.3-70b-versatile", process.env.GROQ_API_KEY!, { - // ... your config options -}); +const adapter = createGroqText( + "llama-3.3-70b-versatile", + process.env.GROQ_API_KEY!, +); const stream = chat({ adapter, @@ -51,19 +56,7 @@ const stream = chat({ }); ``` -## Configuration - -```typescript -import { createGroqText, type GroqTextConfig } from "@tanstack/ai-groq"; - -const config: Omit = { - baseURL: "https://api.groq.com/openai/v1", // Optional, for custom endpoints -}; - -const adapter = createGroqText("llama-3.3-70b-versatile", process.env.GROQ_API_KEY!, config); -``` - -## Example: Chat Completion +### Server ```typescript import { chat, toServerSentEventsResponse } from "@tanstack/ai"; @@ -81,7 +74,7 @@ export async function POST(request: Request) { } ``` -## Example: With Tools +### With tools ```typescript import { chat, toolDefinition, type ModelMessage } from "@tanstack/ai"; @@ -97,11 +90,12 @@ const searchDatabaseDef = toolDefinition({ }); const searchDatabase = searchDatabaseDef.server(async ({ query }) => { - // Search database return { results: [] }; }); -const messages: Array = [{ role: "user", content: "Search for something" }]; +const messages: Array = [ + { role: "user", content: "Search for something" }, +]; const stream = chat({ adapter: groqText("llama-3.3-70b-versatile"), @@ -112,7 +106,7 @@ const stream = chat({ ## Transcription -Groq exposes Whisper-based speech-to-text via `groqTranscription()` and the `generateTranscription()` activity. The `audio` input accepts a `File`, `Blob`, `ArrayBuffer`, base64 string, data URL, or an `https://` URL (forwarded directly to Groq without re-uploading). +`audio`: `File`, `Blob`, `ArrayBuffer`, base64, data URL, or `https://` URL. Models: `whisper-large-v3-turbo`, `whisper-large-v3`. Formats: `json`, `text`, `verbose_json` (default). No `srt`/`vtt`. ```typescript import { generateTranscription } from "@tanstack/ai"; @@ -125,20 +119,16 @@ const result = await generateTranscription({ }); console.log(result.text); - -// verbose_json (the default) populates language, duration, and timestamped segments for (const segment of result.segments ?? []) { console.log(`[${segment.start}s → ${segment.end}s] ${segment.text}`); } ``` -Supported models: `whisper-large-v3-turbo`, `whisper-large-v3`. Supported `responseFormat` values: `json`, `text`, `verbose_json` (default). `srt` and `vtt` are not supported by Groq. - -See [Transcription](../media/transcription) for the full API. +See [Transcription](../media/transcription). -## Model Options +## Model options -Groq supports various provider-specific options. Sampling parameters live here too — `temperature`, `top_p`, and `max_completion_tokens` (Groq's token-limit key) — rather than as root-level props on `chat()`: +Token limit key: `max_completion_tokens`. ```typescript import { chat } from "@tanstack/ai"; @@ -155,11 +145,11 @@ const stream = chat({ }); ``` -> If you previously passed `temperature` / `topP` / `maxTokens` at the root of `chat()`, see [Moving Sampling Options into modelOptions](../migration/sampling-options-to-model-options). +> Root-level sampling migration: [modelOptions](../migration/sampling-options-to-model-options). ### Reasoning -Enable reasoning for models that support it (e.g., `openai/gpt-oss-120b`, `qwen/qwen3-32b`). This allows the model to show its reasoning process, which is streamed as `thinking` chunks: +e.g. `openai/gpt-oss-120b`, `qwen/qwen3-32b`: ```typescript ignore modelOptions: { @@ -167,89 +157,33 @@ modelOptions: { } ``` -## Supported Models - -Groq offers a diverse selection of models from multiple providers: - -### Meta Llama - -- `llama-3.3-70b-versatile` - Fast, capable model with 128K context -- `llama-3.1-8b-instant` - Fast, cost-effective model -- `meta-llama/llama-4-maverick-17b-128e-instruct` - Latest Llama 4 with vision support -- `meta-llama/llama-4-scout-17b-16e-instruct` - Efficient Llama 4 model - -### Security Models - -- `meta-llama/llama-guard-4-12b` - Content moderation -- `meta-llama/llama-prompt-guard-2-86m` - Prompt injection detection -- `meta-llama/llama-prompt-guard-2-22m` - Lightweight prompt guard - -### OpenAI GPT-OSS Models - -- `openai/gpt-oss-120b` - Large OSS model with reasoning support -- `openai/gpt-oss-20b` - Efficient OSS model -- `openai/gpt-oss-safeguard-20b` - Safety-tuned OSS model - -### Other Providers - -- `moonshotai/kimi-k2-instruct-0905` - Kimi K2 with 256K context -- `qwen/qwen3-32b` - Qwen 3 with reasoning support - -## Environment Variables - -Set your API key in environment variables: - -```bash -GROQ_API_KEY=gsk_... -``` - -## API Reference - -### `groqText(model, config?)` - -Creates a Groq chat adapter using environment variables. - -**Parameters:** - -- `model` - The model name (e.g., `llama-3.3-70b-versatile`) -- `config` (optional) - Optional configuration object. Supports the same options as `createGroqText` except `apiKey`, which is auto-detected from `GROQ_API_KEY` environment variable. Common options: - - `baseURL` - Custom base URL for API requests (optional) - -**Returns:** A Groq chat adapter instance. - -### `createGroqText(model, apiKey, config?)` - -Creates a Groq chat adapter with an explicit API key. - -**Parameters:** - -- `model` - The model name (e.g., `llama-3.3-70b-versatile`) -- `apiKey` - Your Groq API key -- `config` (optional) - Optional configuration object: - - `baseURL` - Custom base URL for API requests (optional) +## Models -**Returns:** A Groq chat adapter instance. +**Must-know:** -### `groqTranscription(model, config?)` / `createGroqTranscription(model, apiKey, config?)` +- `llama-3.3-70b-versatile` — 128K, capable +- `llama-3.1-8b-instant` — fast/cheap +- `meta-llama/llama-4-maverick-17b-128e-instruct` — vision +- `openai/gpt-oss-120b` — reasoning +- `qwen/qwen3-32b` — reasoning -Creates a Groq transcription (speech-to-text) adapter. The short form reads `GROQ_API_KEY` from the environment; the `create*` form takes an explicit API key. Supported models: `whisper-large-v3-turbo`, `whisper-large-v3`. +**Also:** Llama Guard / Prompt Guard, `openai/gpt-oss-20b`, `openai/gpt-oss-safeguard-20b`, `moonshotai/kimi-k2-instruct-0905`, Llama 4 Scout, etc. -## Limitations +## API reference -- **Text-to-Speech**: Groq does not currently expose a TTS adapter. Use OpenAI, Gemini, ElevenLabs, or fal for speech generation. -- **Image Generation**: Groq does not support image generation. Use OpenAI, Gemini, or fal for image generation. +| Factory | Purpose | +| --- | --- | +| `groqText` / `createGroqText` | Chat (`config.baseURL?`) | +| `groqTranscription` / `createGroqTranscription` | STT | -## Next Steps +## Notes -- [Getting Started](../getting-started/quick-start) - Learn the basics -- [Tools Guide](../tools/tools) - Learn about tools -- [Other Adapters](./openai) - Explore other providers +- No TTS / image generation — use OpenAI, Gemini, fal, or ElevenLabs -## Provider Tools +No provider-tool factories — use `toolDefinition()` ([tools](../tools/tools.md)). -Groq does not currently expose provider-specific tool factories. -Define your own tools with `toolDefinition()` from `@tanstack/ai`. +## Next steps -See [Tools](../tools/tools.md) for the general tool-definition flow, or -[Provider Tools](../tools/provider-tools.md) for other providers' -native-tool offerings. +- [Getting Started](../getting-started/quick-start) +- [Tools](../tools/tools) +- [Other Adapters](./openai) diff --git a/docs/adapters/mistral.md b/docs/adapters/mistral.md index a45474957..0ca5c1f2a 100644 --- a/docs/adapters/mistral.md +++ b/docs/adapters/mistral.md @@ -2,7 +2,7 @@ title: Mistral id: mistral-adapter order: 7 -description: "Use Mistral models with TanStack AI — Mistral Large, Mistral Medium, Pixtral vision models, Magistral reasoning models, and Codestral via @tanstack/ai-mistral." +description: "Mistral chat, vision (Pixtral), reasoning (Magistral), and Codestral via @tanstack/ai-mistral." keywords: - tanstack ai - mistral @@ -14,15 +14,21 @@ keywords: - llm --- -The Mistral adapter provides access to Mistral's chat models, including Mistral Large, the multimodal Pixtral family, the Magistral reasoning models, and the Codestral code-specialized model. +If you need Mistral → install, set `MISTRAL_API_KEY`, call `mistralText(model)`. -## Installation +## Install ```bash npm install @tanstack/ai-mistral ``` -## Basic Usage +```bash +MISTRAL_API_KEY=... +``` + +Key: [Mistral Console](https://console.mistral.ai/). + +## Do this ```typescript import { chat } from "@tanstack/ai"; @@ -34,7 +40,7 @@ const stream = chat({ }); ``` -## Basic Usage - Custom API Key +### Explicit API key ```typescript import { chat } from "@tanstack/ai"; @@ -51,7 +57,7 @@ const stream = chat({ }); ``` -## Configuration +### Config ```typescript import { @@ -60,7 +66,7 @@ import { } from "@tanstack/ai-mistral"; const config: Omit = { - serverURL: "https://api.mistral.ai", // Optional, this is the default + serverURL: "https://api.mistral.ai", defaultHeaders: { "X-Custom-Header": "value", }, @@ -73,28 +79,10 @@ const adapter = createMistralText( ); ``` -## Example: Chat Completion - -```typescript -import { chat, toServerSentEventsResponse } from "@tanstack/ai"; -import { mistralText } from "@tanstack/ai-mistral"; - -export async function POST(request: Request) { - const { messages } = await request.json(); - - const stream = chat({ - adapter: mistralText("mistral-large-latest"), - messages, - }); - - return toServerSentEventsResponse(stream); -} -``` - -## Example: With Tools +### Server + tools ```typescript -import { chat, toolDefinition } from "@tanstack/ai"; +import { chat, toServerSentEventsResponse, toolDefinition } from "@tanstack/ai"; import { mistralText } from "@tanstack/ai-mistral"; import { z } from "zod"; @@ -110,16 +98,22 @@ const getWeather = getWeatherDef.server(async ({ location }) => { return { temperature: 72, conditions: "sunny" }; }); -const stream = chat({ - adapter: mistralText("mistral-large-latest"), - messages: [{ role: "user", content: "What's the weather in Paris?" }], - tools: [getWeather], -}); +export async function POST(request: Request) { + const { messages } = await request.json(); + + const stream = chat({ + adapter: mistralText("mistral-large-latest"), + messages, + tools: [getWeather], + }); + + return toServerSentEventsResponse(stream); +} ``` -## Example: Multimodal (Vision) +## Vision -Use a vision-capable model — `pixtral-large-latest`, `pixtral-12b-2409`, `mistral-medium-latest`, or `mistral-small-latest` — to send images alongside text: +Models: `pixtral-large-latest`, `pixtral-12b-2409`, `mistral-medium-latest`, `mistral-small-latest`. ```typescript import { chat } from "@tanstack/ai"; @@ -145,45 +139,13 @@ const stream = chat({ }); ``` -For data-URL or base64 images, set `source.type` to `"data"` and provide `mimeType`: - -```typescript -import { chat } from "@tanstack/ai"; -import { mistralText } from "@tanstack/ai-mistral"; - -const base64String = "..."; // your base64-encoded image bytes - -const stream = chat({ - adapter: mistralText("pixtral-large-latest"), - messages: [ - { - role: "user", - content: [ - { type: "text", content: "What's in this image?" }, - { - type: "image", - source: { - type: "data", - mimeType: "image/png", - value: base64String, - }, - }, - ], - }, - ], -}); -``` - -See [Multimodal Content](../advanced/multimodal-content) for the full content-part shape. +Base64: `source.type: "data"` + `mimeType`. See [Multimodal Content](../advanced/multimodal-content). -## Example: Reasoning (Magistral) +## Reasoning (Magistral) -Magistral models (`magistral-medium-latest`, `magistral-small-latest`) stream their reasoning as separate events before the final answer. The adapter emits AG-UI `REASONING_*` chunks for the thinking content and `TEXT_MESSAGE_*` chunks for the answer: +`magistral-medium-latest`, `magistral-small-latest` — `REASONING_*` before `TEXT_MESSAGE_*`. Spec: [Thinking & Reasoning](../chat/thinking-content). ```typescript ignore -// ignore: narrowing the raw AG-UI stream by `chunk.type` relies on @ag-ui/core's -// discriminated-union `type` field, which kiira can't resolve in a source-only -// check. The runtime behaviour is exactly as shown. import { chat } from "@tanstack/ai"; import { mistralText } from "@tanstack/ai-mistral"; @@ -201,13 +163,7 @@ for await (const chunk of stream) { } ``` -Reasoning events are always closed before any text or tool output begins, so consumers see a complete `REASONING_START → REASONING_MESSAGE_START → REASONING_MESSAGE_CONTENT* → REASONING_MESSAGE_END → REASONING_END` sequence first. - -See [Thinking & Reasoning](../chat/thinking-content) for the cross-provider event spec. - -## Example: Structured Output - -Generate JSON that conforms to a Zod schema using Mistral's `json_schema` response format: +## Structured output ```typescript import { chat } from "@tanstack/ai"; @@ -228,14 +184,14 @@ const recipe = await chat({ outputSchema: recipeSchema, }); -console.log(recipe.name); // typed as z.infer +console.log(recipe.name); ``` -See [Structured Outputs](../chat/structured-outputs) for the full guide. +See [Structured Outputs](../chat/structured-outputs). -## Model Options +## Model options -Mistral exposes provider-specific options via `modelOptions`: +Snake_case Mistral names in `modelOptions`: ```typescript import { chat } from "@tanstack/ai"; @@ -259,94 +215,35 @@ const stream = chat({ }); ``` -> All sampling parameters — including `temperature`, `top_p`, and `max_tokens` — -> go inside `modelOptions` using Mistral's native (snake_case) names. - -## Environment Variables - -Set your API key in environment variables: - -```bash -MISTRAL_API_KEY=... -``` - -Get a key from the [Mistral Console](https://console.mistral.ai/). - -## Supported Models - -### Chat - -- `mistral-large-latest` — Flagship general-purpose model (128k context) -- `mistral-medium-latest` — Multimodal mid-tier model with vision -- `mistral-small-latest` — Fast, affordable multimodal model with vision -- `ministral-8b-latest` — 8B edge model -- `ministral-3b-latest` — 3B edge model -- `open-mistral-nemo` — Open 12B model - -### Code - -- `codestral-latest` — Code-specialized model (256k context) - -### Vision - -- `pixtral-large-latest` — Large vision model -- `pixtral-12b-2409` — 12B vision model - -### Reasoning - -Reasoning content is streamed as `REASONING_*` events before the final answer. - -- `magistral-medium-latest` — Mid-tier reasoning model -- `magistral-small-latest` — Small reasoning model - -See [Mistral's model comparison](https://docs.mistral.ai/getting-started/models/compare) for full details. - -## API Reference - -### `mistralText(model, config?)` - -Creates a Mistral text adapter using the `MISTRAL_API_KEY` environment variable. - -**Parameters:** - -- `model` — The model name (e.g., `'mistral-large-latest'`) -- `config.serverURL?` — Custom base URL (optional) -- `config.defaultHeaders?` — Headers to attach to every request (optional) - -**Returns:** A Mistral text adapter instance. - -### `createMistralText(model, apiKey, config?)` - -Creates a Mistral text adapter with an explicit API key. - -**Parameters:** +## Models -- `model` — The model name -- `apiKey` — Your Mistral API key -- `config.serverURL?` — Custom base URL (optional) -- `config.defaultHeaders?` — Headers to attach to every request (optional) +| Group | IDs | +| --- | --- | +| Chat | `mistral-large-latest`, `mistral-medium-latest`, `mistral-small-latest`, `ministral-8b-latest`, `ministral-3b-latest`, `open-mistral-nemo` | +| Code | `codestral-latest` | +| Vision | `pixtral-large-latest`, `pixtral-12b-2409` | +| Reasoning | `magistral-medium-latest`, `magistral-small-latest` | -**Returns:** A Mistral text adapter instance. +Compare: [Mistral docs](https://docs.mistral.ai/getting-started/models/compare). -## Limitations +## API reference -- **Embeddings**: Use the [Mistral SDK](https://github.com/mistralai/client-ts) directly for `mistral-embed`. -- **Image / Audio / Video Generation**: Mistral does not provide these endpoints. Use OpenAI, Gemini, or fal.ai. -- **Text-to-Speech / Transcription**: Not supported. Use OpenAI or ElevenLabs. +| Factory | Purpose | +| --- | --- | +| `mistralText(model, config?)` | Env key; `serverURL?`, `defaultHeaders?` | +| `createMistralText(model, apiKey, config?)` | Explicit key | -## Next Steps +## Notes -- [Getting Started](../getting-started/quick-start) — Learn the basics -- [Tools Guide](../tools/tools) — Define and call tools -- [Structured Outputs](../chat/structured-outputs) — Generate typed JSON -- [Multimodal Content](../advanced/multimodal-content) — Send images and other modalities -- [Other Adapters](./openai) — Explore other providers +- No embeddings adapter — use [Mistral SDK](https://github.com/mistralai/client-ts) for `mistral-embed` +- No image/audio/video generation, TTS, or STT -## Provider Tools +No provider-tool factories — use `toolDefinition()` ([tools](../tools/tools.md)). -Mistral does not currently expose provider-specific tool factories. -Define your own tools with `toolDefinition()` from `@tanstack/ai`. +## Next steps -See [Tools](../tools/tools.md) for the general tool-definition flow, or -[Provider Tools](../tools/provider-tools.md) for other providers' -native-tool offerings. +- [Getting Started](../getting-started/quick-start) +- [Tools](../tools/tools) +- [Structured Outputs](../chat/structured-outputs) +- [Multimodal Content](../advanced/multimodal-content) +- [Other Adapters](./openai) diff --git a/docs/adapters/ollama.md b/docs/adapters/ollama.md index cc821149b..1c998de55 100644 --- a/docs/adapters/ollama.md +++ b/docs/adapters/ollama.md @@ -2,7 +2,7 @@ title: Ollama id: ollama-adapter order: 4 -description: "Run local LLMs with Ollama in TanStack AI for private, no-cost AI on your own hardware via the @tanstack/ai-ollama adapter." +description: "Local Ollama models via @tanstack/ai-ollama — private, offline chat on your hardware." keywords: - tanstack ai - ollama @@ -14,15 +14,29 @@ keywords: - adapter --- -The Ollama adapter provides access to local models running via Ollama, allowing you to run AI models on your own infrastructure with full privacy and no API costs. +If you need local models → install Ollama, pull a model, call `ollamaText(model)`. -## Installation +## Setup Ollama + +1. Install Ollama (`brew install ollama`, [install.sh](https://ollama.com/install.sh), or [ollama.com](https://ollama.com)). +2. Pull: `ollama pull llama3` +3. Serve: `ollama serve` (default `http://localhost:11434`) + +List models: `ollama list`. + +## Install ```bash npm install @tanstack/ai-ollama ``` -## Basic Usage +Optional: + +```bash +OLLAMA_HOST=http://localhost:11434 +``` + +## Do this ```typescript import { chat } from "@tanstack/ai"; @@ -34,13 +48,14 @@ const stream = chat({ }); ``` -## Basic Usage - Custom Host +### Custom host ```typescript import { chat } from "@tanstack/ai"; import { createOllamaChat } from "@tanstack/ai-ollama"; const adapter = createOllamaChat("llama3", "http://your-server:11434"); +// or: createOllamaChat("llama3", { host: "...", headers: { Authorization: "Bearer ..." } }) const stream = chat({ adapter, @@ -48,59 +63,9 @@ const stream = chat({ }); ``` -## Configuration - -```typescript -import { createOllamaChat } from "@tanstack/ai-ollama"; - -// Custom host (URL string) -const adapter = createOllamaChat("llama3", "http://your-server:11434"); - -// Custom client config (e.g., custom headers, fetch) -const adapter2 = createOllamaChat("llama3", { - host: "http://your-server:11434", - headers: { Authorization: "Bearer ..." }, -}); -``` - -## Available Models - -To see available models on your Ollama instance: - -```bash -ollama list -``` - -### Popular Models +Network bind: `OLLAMA_HOST=0.0.0.0:11434 ollama serve`. -- `llama3` / `llama3.1` / `llama3.2` - Meta's Llama models -- `mistral` / `mistral:7b` - Mistral AI models -- `mixtral` - Mixtral MoE model -- `codellama` - Code-focused Llama -- `phi3` - Microsoft's Phi models -- `gemma` / `gemma2` - Google's Gemma models -- `qwen2` / `qwen2.5` - Alibaba's Qwen models -- `deepseek-coder` - DeepSeek coding model - -## Example: Chat Completion - -```typescript -import { chat, toServerSentEventsResponse } from "@tanstack/ai"; -import { ollamaText } from "@tanstack/ai-ollama"; - -export async function POST(request: Request) { - const { messages } = await request.json(); - - const stream = chat({ - adapter: ollamaText("llama3"), - messages, - }); - - return toServerSentEventsResponse(stream); -} -``` - -## Example: With Tools +### Server + tools ```typescript import { chat, toServerSentEventsResponse, toolDefinition } from "@tanstack/ai"; @@ -116,7 +81,6 @@ const getLocalDataDef = toolDefinition({ }); const getLocalData = getLocalDataDef.server(async ({ key }) => { - // Access local data return { data: "..." }; }); @@ -133,11 +97,11 @@ export async function POST(request: Request) { } ``` -**Note:** Tool support varies by model. Models like `llama3`, `mistral`, and `qwen2` generally have good tool calling support. +Tool support varies — `llama3`, `mistral`, `qwen2` generally work. -## Model Options +## Model options -Ollama supports various provider-specific options. Unlike the other providers, Ollama nests its sampling and runner parameters inside an `options` object **within** `modelOptions` — `temperature`, `top_p`, and `num_predict` (the token-limit key) all live under `modelOptions.options`: +Sampling lives under **`modelOptions.options`** (not root of `modelOptions`): ```typescript import { chat } from "@tanstack/ai"; @@ -151,59 +115,21 @@ const stream = chat({ temperature: 0.7, top_p: 0.9, top_k: 40, - num_predict: 1000, // Max tokens to generate + num_predict: 1000, repeat_penalty: 1.1, - num_ctx: 4096, // Context window size - num_gpu: -1, // GPU layers (-1 = auto) + num_ctx: 4096, + num_gpu: -1, }, }, }); ``` -> If you previously passed `temperature` / `topP` / `maxTokens` at the root of `chat()`, note that for Ollama they map to `modelOptions.options.temperature`, `modelOptions.options.top_p`, and `modelOptions.options.num_predict`. See [Moving Sampling Options into modelOptions](../migration/sampling-options-to-model-options). +> Migration map: `temperature` → `modelOptions.options.temperature`, token limit → `num_predict`. See [modelOptions](../migration/sampling-options-to-model-options). -### Advanced Options - -All sampling and runner parameters are nested under `modelOptions.options`: - -```typescript ignore -modelOptions: { - options: { - // Sampling - temperature: 0.7, - top_p: 0.9, - top_k: 40, - min_p: 0.05, - typical_p: 1.0, - - // Generation - num_predict: 1000, - repeat_penalty: 1.1, - repeat_last_n: 64, - penalize_newline: false, - - // Performance - num_ctx: 4096, - num_batch: 512, - num_gpu: -1, - num_thread: 0, // 0 = auto - - // Memory - use_mmap: true, - use_mlock: false, - - // Mirostat sampling - mirostat: 0, // 0 = disabled, 1 = Mirostat, 2 = Mirostat 2.0 - mirostat_tau: 5.0, - mirostat_eta: 0.1, - }, -} -``` +Also under `options`: `min_p`, `typical_p`, `repeat_last_n`, `penalize_newline`, `num_batch`, `num_thread`, `use_mmap`, `use_mlock`, `mirostat`, `mirostat_tau`, `mirostat_eta`. ## Summarization -Summarize long text content locally: - ```typescript ignore import { summarize } from "@tanstack/ai"; import { ollamaSummarize } from "@tanstack/ai-ollama"; @@ -212,101 +138,34 @@ const result = await summarize({ adapter: ollamaSummarize("llama3"), text: "Your long text to summarize...", maxLength: 100, - style: "concise", // "concise" | "bullet-points" | "paragraph" + style: "concise", }); console.log(result.summary); ``` -## Setting Up Ollama - -### 1. Install Ollama - -```bash -# macOS -brew install ollama - -# Linux -curl -fsSL https://ollama.com/install.sh | sh - -# Windows -# Download from https://ollama.com -``` - -### 2. Pull a Model - -```bash -ollama pull llama3 -``` - -### 3. Start Ollama Server - -```bash -ollama serve -``` - -The server runs on `http://localhost:11434` by default. - -## Running on a Remote Server - -```typescript -import { createOllamaChat } from "@tanstack/ai-ollama"; - -const adapter = createOllamaChat("llama3", "http://your-server:11434"); -``` - -To expose Ollama on a network interface: - -```bash -OLLAMA_HOST=0.0.0.0:11434 ollama serve -``` - -## Environment Variables - -Optionally set the host in environment variables: - -```bash -OLLAMA_HOST=http://localhost:11434 -``` - -## API Reference - -### `ollamaText(model)` - -Creates an Ollama text/chat adapter using `OLLAMA_HOST` from the environment (defaults to `http://localhost:11434`). - -**Parameters:** - -- `model` - Model name (e.g. `"llama3"`, `"mistral:7b"`) - -### `createOllamaChat(model, hostOrConfig?)` - -Creates an Ollama text/chat adapter with an explicit host or client config. - -**Parameters:** - -- `model` - Model name -- `hostOrConfig?` - Either an `OLLAMA_HOST`-style URL string, or an `OllamaClientConfig` object (e.g. `{ host, headers, fetch }`). +## Models -### `ollamaSummarize(model)` / `createOllamaSummarize(model, hostOrConfig?)` +Common: `llama3` / `llama3.1` / `llama3.2`, `mistral`, `mixtral`, `codellama`, `phi3`, `gemma` / `gemma2`, `qwen2` / `qwen2.5`, `deepseek-coder`. -Creates an Ollama summarization adapter — same signature shape as the chat adapter. +## API reference -## Benefits of Ollama +| Factory | Purpose | +| --- | --- | +| `ollamaText(model)` | Env host (default localhost) | +| `createOllamaChat(model, hostOrConfig?)` | URL string or `{ host, headers, fetch }` | +| `ollamaSummarize` / `createOllamaSummarize` | Same shape | -- ✅ **Privacy** - Data stays on your infrastructure -- ✅ **Cost** - No API costs after hardware -- ✅ **Customization** - Use any compatible model -- ✅ **Offline** - Works without internet -- ✅ **Speed** - No network latency for local deployment +OpenAI-compatible Ollama endpoint: prefer this adapter for native API, or [openai-compatible](./openai-compatible) for `/v1` surface. -## Limitations +## Notes -- **Image Generation**: Ollama does not support image generation. Use OpenAI or Gemini for image generation. -- **Performance**: Depends on your hardware (GPU recommended for larger models) +- Privacy / offline / no API cost after hardware +- No image generation +- Performance depends on GPU -## Next Steps +## Next steps -- [Getting Started](../getting-started/quick-start) - Learn the basics -- [Tools Guide](../tools/tools) - Learn about tools -- [Other Adapters](./openai) - Explore other providers +- [Getting Started](../getting-started/quick-start) +- [Tools](../tools/tools) +- [Other Adapters](./openai) diff --git a/docs/adapters/openai-compatible.md b/docs/adapters/openai-compatible.md index 91713aca3..71cd34c1a 100644 --- a/docs/adapters/openai-compatible.md +++ b/docs/adapters/openai-compatible.md @@ -1,7 +1,7 @@ --- title: OpenAI-Compatible Adapter id: openai-compatible-adapter -description: "Use any OpenAI-compatible provider (DeepSeek, Moonshot/Kimi, Together, Fireworks, Cerebras, Qwen, Perplexity, local servers, and more) in TanStack AI with one generic adapter." +description: "Any OpenAI Chat Completions provider (DeepSeek, Kimi, Together, local servers, LiteLLM) via one generic adapter." keywords: - tanstack ai - openai compatible @@ -19,28 +19,27 @@ keywords: - adapter --- -Many providers expose the OpenAI **Chat Completions** API (`/chat/completions`) — DeepSeek, Moonshot/Kimi, Together, Fireworks, Cerebras, Alibaba Qwen, Perplexity, NVIDIA NIM, and local servers like LM Studio, Ollama, and vLLM. Instead of a dedicated package per provider, TanStack AI ships one generic adapter: point it at any compatible `baseURL`, give it your models, and you get the same type-safe `chat()` experience as the first-class adapters. +If your provider speaks OpenAI `/chat/completions` but has no `@tanstack/ai-*` package → use `openaiCompatible`. Prefer dedicated adapters (OpenAI, Grok, Groq, OpenRouter) when they exist. -Use this when your provider speaks the OpenAI Chat Completions wire format but doesn't have its own `@tanstack/ai-*` package. If a dedicated adapter exists (OpenAI, Grok, Groq, OpenRouter), prefer it — those carry curated per-model metadata. +## Install -## Installation - -The adapter ships inside `@tanstack/ai-openai` under the `/compatible` subpath — no extra install: +Ships in `@tanstack/ai-openai` under `/compatible`: ```bash npm install @tanstack/ai-openai ``` -## Basic Usage +## Do this -Configure the provider once with `openaiCompatible({ baseURL, apiKey, models })`, then select a model per call. The returned model name is a type-safe union of the models you declared: +1. Configure once with `baseURL`, `apiKey`, `models`. +2. Select a model per call (type-safe union of declared models). ```typescript import { chat } from "@tanstack/ai"; import { openaiCompatible } from "@tanstack/ai-openai/compatible"; const deepseek = openaiCompatible({ - name: "deepseek", // optional label shown in devtools/errors (default: "openai-compatible") + name: "deepseek", baseURL: "https://api.deepseek.com/v1", apiKey: process.env.DEEPSEEK_API_KEY!, models: ["deepseek-chat", "deepseek-reasoner"], @@ -52,11 +51,7 @@ const stream = chat({ }); ``` -`deepseek("deepseek-reasoner")` is valid; `deepseek("gpt-4o")` is a type error — only declared models are accepted. - -## One-Shot Usage - -For a single model, skip the provider-factory and build the adapter inline with `openaiCompatibleText`: +### One-shot ```typescript import { chat } from "@tanstack/ai"; @@ -71,12 +66,10 @@ const stream = chat({ }); ``` -## Declaring Models +## Declaring models -The `models` array accepts two forms, which you can mix: - -- **A bare string** — gets optimistic defaults: `text` + `image` input, with `streaming`, `function_calling`, and `structured_outputs` support. Good for mainstream chat models. -- **A `createModel(name, capabilities)` definition** — declares precise per-model capabilities so the types match reality (e.g. a reasoning model with no image input). +- **String** → optimistic defaults: text+image input, streaming, function_calling, structured_outputs +- **`createModel(name, capabilities)`** → precise features (e.g. no image) ```typescript import { openaiCompatible } from "@tanstack/ai-openai/compatible"; @@ -86,20 +79,20 @@ const provider = openaiCompatible({ baseURL: "https://api.deepseek.com/v1", apiKey: process.env.DEEPSEEK_API_KEY!, models: [ - "deepseek-chat", // string → optimistic defaults + "deepseek-chat", createModel("deepseek-reasoner", { - input: ["text"], // text only + input: ["text"], features: ["reasoning", "structured_outputs"], }), ], }); ``` -> Capabilities are enforced at the type level. If a provider rejects a feature at runtime (e.g. tools on a model that doesn't support them), declare that model with `createModel` and omit the unsupported feature so the types stop you from calling it. +Omit unsupported features so types block bad calls. ## Configuration -`openaiCompatible` accepts every OpenAI SDK `ClientOptions` field besides `apiKey`/`baseURL` (which are required and promoted to the top level). The most useful are `defaultHeaders` and `defaultQuery`, for providers that need extra auth or routing parameters: +All OpenAI SDK `ClientOptions` except `apiKey`/`baseURL` (required at top level). Useful: `defaultHeaders`, `defaultQuery`. ```typescript import { openaiCompatible } from "@tanstack/ai-openai/compatible"; @@ -113,52 +106,36 @@ const provider = openaiCompatible({ }); ``` -## Chat Completions vs Responses +### Responses API -By default the adapter targets the **Chat Completions** API (`/chat/completions`) — the surface virtually every compatible provider implements. For the rare provider that also implements OpenAI's **Responses** API (e.g. Azure OpenAI), opt in with `api: "responses"`: +Default is Chat Completions. Opt in: ```typescript -import { openaiCompatible } from "@tanstack/ai-openai/compatible"; - -const provider = openaiCompatible({ - baseURL: "https://my-resource.openai.azure.com/openai/v1", - apiKey: process.env.AZURE_OPENAI_API_KEY!, - models: ["gpt-4o"], - api: "responses", // default is "chat-completions" -}); +api: "responses", // default: "chat-completions" ``` -## Supported Providers - -Any provider implementing the OpenAI Chat Completions API works. Common ones are below — **verify the `baseURL` and model ids against each provider's current docs**, since they change over time. Set the API key via the provider's own environment variable and pass it as `apiKey`. +## Providers (verify baseURL/models in their docs) | Provider | `baseURL` | Example model | | --- | --- | --- | -| DeepSeek | `https://api.deepseek.com/v1` | `deepseek-chat`, `deepseek-reasoner` | +| DeepSeek | `https://api.deepseek.com/v1` | `deepseek-chat` | | Moonshot / Kimi | `https://api.moonshot.ai/v1` | `kimi-k2-0711-preview` | -| Alibaba Qwen (DashScope, intl) | `https://dashscope-intl.aliyuncs.com/compatible-mode/v1` | `qwen-max`, `qwen-plus` | -| Alibaba Qwen (DashScope, China) | `https://dashscope.aliyuncs.com/compatible-mode/v1` | `qwen-max` | -| Together AI | `https://api.together.xyz/v1` | `meta-llama/Llama-3.3-70B-Instruct-Turbo` | -| Fireworks AI | `https://api.fireworks.ai/inference/v1` | `accounts/fireworks/models/llama-v3p3-70b-instruct` | +| Qwen (intl) | `https://dashscope-intl.aliyuncs.com/compatible-mode/v1` | `qwen-max` | +| Qwen (China) | `https://dashscope.aliyuncs.com/compatible-mode/v1` | `qwen-max` | +| Together | `https://api.together.xyz/v1` | `meta-llama/Llama-3.3-70B-Instruct-Turbo` | +| Fireworks | `https://api.fireworks.ai/inference/v1` | `accounts/fireworks/models/llama-v3p3-70b-instruct` | | Cerebras | `https://api.cerebras.ai/v1` | `llama-3.3-70b` | | DeepInfra | `https://api.deepinfra.com/v1/openai` | `meta-llama/Llama-3.3-70B-Instruct` | -| Perplexity | `https://api.perplexity.ai` | `sonar`, `sonar-pro` | -| Requesty | `https://router.requesty.ai/v1` | `openai/gpt-4o-mini` | -| Mistral | `https://api.mistral.ai/v1` | `mistral-large-latest` | -| Nebius | `https://api.studio.nebius.ai/v1` | `meta-llama/Llama-3.3-70B-Instruct` | -| Z.AI (GLM) | `https://api.z.ai/api/paas/v4` | `glm-4.6` | -| Baseten | `https://inference.baseten.co/v1` | model-dependent | -| Hugging Face (router) | `https://router.huggingface.co/v1` | `meta-llama/Llama-3.3-70B-Instruct` | +| Perplexity | `https://api.perplexity.ai` | `sonar` | | NVIDIA NIM | `https://integrate.api.nvidia.com/v1` | `meta/llama-3.3-70b-instruct` | -## Local & Self-Hosted Servers +Also: Requesty, Mistral, Nebius, Z.AI, Baseten, Hugging Face router, … -Point the adapter at any local OpenAI-compatible server. The API key is usually a placeholder: +## Local servers ```typescript import { openaiCompatible } from "@tanstack/ai-openai/compatible"; -// LM Studio const lmstudio = openaiCompatible({ name: "lmstudio", baseURL: "http://localhost:1234/v1", @@ -166,7 +143,6 @@ const lmstudio = openaiCompatible({ models: ["local-model"], }); -// vLLM const vllm = openaiCompatible({ name: "vllm", baseURL: "http://localhost:8000/v1", @@ -174,7 +150,6 @@ const vllm = openaiCompatible({ models: ["meta-llama/Llama-3.3-70B-Instruct"], }); -// Ollama's OpenAI-compatible endpoint const ollama = openaiCompatible({ name: "ollama", baseURL: "http://localhost:11434/v1", @@ -183,19 +158,17 @@ const ollama = openaiCompatible({ }); ``` -> Ollama also has a dedicated adapter, [`@tanstack/ai-ollama`](./ollama), which understands its native API. Use `openaiCompatible` only if you specifically want Ollama's OpenAI-compatible surface. +Ollama native API: [`@tanstack/ai-ollama`](./ollama). -## LiteLLM Proxy - -[LiteLLM](https://github.com/BerriAI/litellm) is a self-hosted gateway that exposes a single OpenAI Chat Completions endpoint in front of 100+ providers (OpenAI, Anthropic, Google, Azure, AWS Bedrock, Mistral, Groq, and more). Because the proxy speaks the OpenAI wire format, it needs no dedicated package — point `openaiCompatible` at your proxy's `baseURL` (default `http://localhost:4000/v1`) and route to a provider with LiteLLM's `provider/model` naming: +## LiteLLM ```typescript import { openaiCompatible } from "@tanstack/ai-openai/compatible"; const litellm = openaiCompatible({ name: "litellm", - baseURL: "http://localhost:4000/v1", // your LiteLLM proxy - apiKey: process.env.LITELLM_API_KEY!, // a virtual key issued by the proxy + baseURL: "http://localhost:4000/v1", + apiKey: process.env.LITELLM_API_KEY!, models: [ "anthropic/claude-sonnet-5", "openai/gpt-5.5", @@ -204,32 +177,26 @@ const litellm = openaiCompatible({ }); ``` -`litellm("anthropic/claude-sonnet-5")` selects the Anthropic route; `litellm("openai/gpt-5.5")` selects OpenAI — all through the one proxy. Declare only the model routes you configured on the proxy; for precise per-model capabilities (e.g. a reasoning route without image input), use `createModel` as shown under [Declaring Models](#declaring-models). - -> The proxy holds each upstream provider's real credentials; the `apiKey` here is the proxy's own virtual/master key, not the upstream provider's. +`apiKey` is the proxy virtual key, not upstream credentials. ## Azure OpenAI -Azure uses a resource-scoped URL and a separate API-version. Use the `/openai/v1` endpoint with `defaultQuery` for the version and `defaultHeaders` for the `api-key` header: - ```typescript import { openaiCompatible } from "@tanstack/ai-openai/compatible"; const azure = openaiCompatible({ name: "azure", baseURL: "https://YOUR_RESOURCE.openai.azure.com/openai/v1", - apiKey: process.env.AZURE_OPENAI_API_KEY!, // also sent as Bearer; Azure accepts the api-key header below - models: ["gpt-4o"], // your Azure deployment name + apiKey: process.env.AZURE_OPENAI_API_KEY!, + models: ["gpt-4o"], defaultQuery: { "api-version": "2026-01-01-preview" }, defaultHeaders: { "api-key": process.env.AZURE_OPENAI_API_KEY! }, }); ``` -> Confirm the current `api-version` and endpoint shape in Azure's documentation — Azure's API surface evolves independently of OpenAI's. - -## Example: With Tools +Confirm `api-version` in Azure docs. -Tools work exactly as they do with any other adapter, for models that support function calling: +## Tools ```typescript import { chat, toolDefinition } from "@tanstack/ai"; @@ -259,9 +226,9 @@ const stream = chat({ }); ``` -## Next Steps +## Next steps -- [OpenAI Adapter](./openai) - The first-class OpenAI adapter -- [OpenRouter Adapter](./openrouter) - Access 300+ models through one gateway -- [Tools Guide](../tools/tools) - Learn about tools -- [Extending Adapters](../advanced/extend-adapter) - Add custom models to any adapter +- [OpenAI Adapter](./openai) +- [OpenRouter Adapter](./openrouter) +- [Tools](../tools/tools) +- [Extending Adapters](../advanced/extend-adapter) diff --git a/docs/adapters/openai.md b/docs/adapters/openai.md index 776701950..931732783 100644 --- a/docs/adapters/openai.md +++ b/docs/adapters/openai.md @@ -2,7 +2,7 @@ title: OpenAI id: openai-adapter order: 1 -description: "Use OpenAI models with TanStack AI — GPT-4o, GPT-5, DALL-E image generation, TTS, and Whisper transcription via @tanstack/ai-openai." +description: "OpenAI GPT, images, TTS, Whisper, and provider tools via @tanstack/ai-openai." keywords: - tanstack ai - openai @@ -15,79 +15,40 @@ keywords: - chatgpt --- -The OpenAI adapter provides access to OpenAI's models, including GPT-4o, GPT-5, image generation (DALL-E), text-to-speech (TTS), and audio transcription (Whisper). +If you need OpenAI → install, set `OPENAI_API_KEY`, call `openaiText(model)`. -> Using a third-party provider that speaks the OpenAI API (DeepSeek, Moonshot/Kimi, Together, Fireworks, a local LM Studio/vLLM server, …)? See the [OpenAI-Compatible Adapter](./openai-compatible) for a generic `openaiCompatible({ baseURL, apiKey, models })` factory. +Third-party OpenAI-compatible APIs → [OpenAI-Compatible Adapter](./openai-compatible). -## Installation +## Install ```bash npm install @tanstack/ai-openai ``` -## Basic Usage - -```typescript -import { chat } from "@tanstack/ai"; -import { openaiText } from "@tanstack/ai-openai"; - -const stream = chat({ - adapter: openaiText("gpt-5.2"), - messages: [{ role: "user", content: "Hello!" }], -}); -``` - -## Chat Completions API - -`@tanstack/ai-openai` ships two text adapters that hit different OpenAI endpoints. `openaiText` (default) calls the Responses API (`/v1/responses`). `openaiChatCompletions` calls the older Chat Completions API (`/v1/chat/completions`). - -Pick whichever fits your wire format and feature needs: - -| | `openaiText` (Responses) | `openaiChatCompletions` (Chat Completions) | -|---|---|---| -| Endpoint | `/v1/responses` | `/v1/chat/completions` | -| Reasoning summaries | Yes — set `modelOptions.reasoning.summary: 'auto'` to surface reasoning text via `REASONING_*` events | No — reasoning tokens are still consumed but cannot be exposed | -| Wire-format compatibility | OpenAI-only | Matches the older de-facto industry shape (Grok, Groq, OpenRouter, many local model servers) | -| Structured output streaming | `text.format: { type: 'json_schema', strict: true }` + `stream: true` | `response_format: { type: 'json_schema', strict: true }` + `stream: true` | - -Use `openaiText` when you want reasoning-summary streaming or OpenAI-specific Responses features. Use `openaiChatCompletions` when you're migrating off a Chat-Completions-style provider, share request-building code with other Chat-Completions adapters in your stack, or want the more battle-tested wire format. - -```typescript -import { chat } from "@tanstack/ai"; -import { openaiChatCompletions } from "@tanstack/ai-openai"; - -const stream = chat({ - adapter: openaiChatCompletions("gpt-5.2"), - messages: [{ role: "user", content: "Hello!" }], -}); +```bash +OPENAI_API_KEY=sk-... ``` -With an explicit API key: +## Do this ```typescript import { chat } from "@tanstack/ai"; -import { createOpenaiChatCompletions } from "@tanstack/ai-openai"; - -const adapter = createOpenaiChatCompletions("gpt-5.2", process.env.OPENAI_API_KEY!, { - // organization, baseURL, headers — all optional -}); +import { openaiText } from "@tanstack/ai-openai"; const stream = chat({ - adapter, + adapter: openaiText("gpt-5.2"), messages: [{ role: "user", content: "Hello!" }], }); ``` -Both adapters work identically with [Structured Outputs](../structured-outputs/overview) — including `stream: true` — and accept the same `modelOptions` (temperature, top_p, max_tokens, stop, …). The reasoning section below applies to `openaiText`; `openaiChatCompletions` accepts `modelOptions.reasoning.effort` but cannot stream summary text. - -## Basic Usage - Custom API Key +### Explicit API key ```typescript import { chat } from "@tanstack/ai"; import { createOpenaiChat } from "@tanstack/ai-openai"; const adapter = createOpenaiChat("gpt-5.2", process.env.OPENAI_API_KEY!, { - // ... your config options + // organization, baseURL }); const stream = chat({ @@ -96,38 +57,7 @@ const stream = chat({ }); ``` -## Configuration - -```typescript -import { createOpenaiChat, type OpenAITextConfig } from "@tanstack/ai-openai"; - -const config: Omit = { - organization: "org-...", // Optional - baseURL: "https://api.openai.com/v1", // Optional, for custom endpoints -}; - -const adapter = createOpenaiChat("gpt-5.2", process.env.OPENAI_API_KEY!, config); -``` - -## Example: Chat Completion - -```typescript -import { chat, toServerSentEventsResponse } from "@tanstack/ai"; -import { openaiText } from "@tanstack/ai-openai"; - -export async function POST(request: Request) { - const { messages } = await request.json(); - - const stream = chat({ - adapter: openaiText("gpt-5.2"), - messages, - }); - - return toServerSentEventsResponse(stream); -} -``` - -## Example: With Tools +### Server + tools ```typescript import { chat, toServerSentEventsResponse, toolDefinition } from "@tanstack/ai"; @@ -143,7 +73,6 @@ const getWeatherDef = toolDefinition({ }); const getWeather = getWeatherDef.server(async ({ location }) => { - // Fetch weather data return { temperature: 72, conditions: "sunny" }; }); @@ -160,9 +89,44 @@ export async function POST(request: Request) { } ``` -## Model Options +## Chat Completions vs Responses + +| | `openaiText` (default) | `openaiChatCompletions` | +|---|---|---| +| Endpoint | `/v1/responses` | `/v1/chat/completions` | +| Reasoning summaries | Yes (`reasoning.summary: 'auto'`) | Effort only, no streamed summary | +| Wire format | OpenAI-only | Industry Chat Completions shape | +| Structured streaming | `text.format` + `stream: true` | `response_format` + `stream: true` | + +Use Responses for reasoning-summary streaming. Use Chat Completions when migrating or sharing code with other Completions adapters. + +```typescript +import { chat } from "@tanstack/ai"; +import { openaiChatCompletions } from "@tanstack/ai-openai"; + +const stream = chat({ + adapter: openaiChatCompletions("gpt-5.2"), + messages: [{ role: "user", content: "Hello!" }], +}); +``` + +```typescript +import { chat } from "@tanstack/ai"; +import { createOpenaiChatCompletions } from "@tanstack/ai-openai"; + +const adapter = createOpenaiChatCompletions("gpt-5.2", process.env.OPENAI_API_KEY!); + +const stream = chat({ + adapter, + messages: [{ role: "user", content: "Hello!" }], +}); +``` + +Both support [Structured Outputs](../structured-outputs/overview). + +## Model options -OpenAI supports various provider-specific options. Sampling parameters live here too — `temperature`, `top_p`, and `max_output_tokens` (the Responses API token-limit key) — rather than as root-level props on `chat()`: +Responses token limit: `max_output_tokens`. Completions: `max_tokens`. ```typescript import { chat } from "@tanstack/ai"; @@ -179,27 +143,23 @@ const stream = chat({ }); ``` -> The `openaiChatCompletions` adapter targets `/v1/chat/completions`, where the token-limit key is `max_tokens` (not `max_output_tokens`). If you previously passed `temperature` / `topP` / `maxTokens` at the root of `chat()`, see [Moving Sampling Options into modelOptions](../migration/sampling-options-to-model-options). +> Root-level sampling migration: [modelOptions](../migration/sampling-options-to-model-options). ### Reasoning -Enable reasoning for models that support it (e.g., GPT-5, O3). This allows the model to show its reasoning process, which is streamed as `thinking` chunks: - ```typescript ignore modelOptions: { reasoning: { effort: "medium", // "none" | "minimal" | "low" | "medium" | "high" - summary: "detailed", // "auto" | "detailed" (optional) + summary: "detailed", // "auto" | "detailed" }, } ``` -When reasoning is enabled, the model's reasoning process is streamed separately from the response text and appears as a collapsible thinking section in the UI. +Streams as thinking chunks. ## Summarization -Summarize long text content: - ```typescript import { summarize } from "@tanstack/ai"; import { openaiSummarize } from "@tanstack/ai-openai"; @@ -208,15 +168,13 @@ const result = await summarize({ adapter: openaiSummarize("gpt-5-mini"), text: "Your long text to summarize...", maxLength: 100, - style: "concise", // "concise" | "bullet-points" | "paragraph" + style: "concise", }); console.log(result.summary); ``` -## Image Generation - -Generate images: +## Image ```typescript import { generateImage } from "@tanstack/ai"; @@ -227,29 +185,17 @@ const result = await generateImage({ prompt: "A futuristic cityscape at sunset", numberOfImages: 1, size: "1024x1024", -}); - -console.log(result.images); -``` - -### Image Model Options - -```typescript -import { generateImage } from "@tanstack/ai"; -import { openaiImage } from "@tanstack/ai-openai"; - -const result = await generateImage({ - adapter: openaiImage("gpt-image-2"), - prompt: "...", modelOptions: { quality: "high", // "high" | "medium" | "low" | "auto" }, }); + +console.log(result.images); ``` -## Text-to-Speech +## Text-to-speech -Generate speech from text: +Voices: `alloy`, `echo`, `fable`, `onyx`, `nova`, `shimmer`, `ash`, `ballad`, `coral`, `sage`, `verse`. ```typescript import { generateSpeech } from "@tanstack/ai"; @@ -262,49 +208,13 @@ const result = await generateSpeech({ format: "mp3", }); -// result.audio contains base64-encoded audio -console.log(result.format); // "mp3" +console.log(result.format); ``` -### TTS Voices - -Available voices: `alloy`, `echo`, `fable`, `onyx`, `nova`, `shimmer`, `ash`, `ballad`, `coral`, `sage`, `verse` - -### TTS Model Options - -```typescript -import { generateSpeech } from "@tanstack/ai"; -import { openaiSpeech } from "@tanstack/ai-openai"; - -const result = await generateSpeech({ - adapter: openaiSpeech("tts-1-hd"), - text: "High quality speech", - modelOptions: { - instructions: "Speak slowly and clearly.", // voice instructions (not supported by tts-1/tts-1-hd) - }, -}); -``` +`instructions` in `modelOptions` not supported on `tts-1` / `tts-1-hd`. ## Transcription -Transcribe audio to text: - -```typescript -import { generateTranscription } from "@tanstack/ai"; -import { openaiTranscription } from "@tanstack/ai-openai"; -import { audioFile } from "./audio"; - -const result = await generateTranscription({ - adapter: openaiTranscription("whisper-1"), - audio: audioFile, // File object or base64 string - language: "en", -}); - -console.log(result.text); // Transcribed text -``` - -### Transcription Model Options - ```typescript import { generateTranscription } from "@tanstack/ai"; import { openaiTranscription } from "@tanstack/ai-openai"; @@ -313,6 +223,7 @@ import { audioFile } from "./audio"; const result = await generateTranscription({ adapter: openaiTranscription("whisper-1"), audio: audioFile, + language: "en", responseFormat: "verbose_json", prompt: "Technical terms: API, SDK", modelOptions: { @@ -321,13 +232,10 @@ const result = await generateTranscription({ }, }); -// Access the transcribed text console.log(result.text); ``` -### Speaker Diarization - -Use `gpt-4o-transcribe-diarize` for speaker-labeled transcripts: +### Speaker diarization ```typescript import { generateTranscription } from "@tanstack/ai"; @@ -351,87 +259,31 @@ for (const segment of result.segments ?? []) { } ``` -When no response format is specified, `gpt-4o-transcribe-diarize` requests default to `response_format: "diarized_json"` and `chunking_strategy: "auto"`; passing a top-level `responseFormat` of `"json"` or `"text"` opts out of speaker segments. `known_speaker_names` and `known_speaker_references` must be provided together (up to 4, matching lengths). OpenAI does not support `prompt`, `include`, or `timestamp_granularities` with diarized transcription. - -## Environment Variables - -Set your API key in environment variables: - -```bash -OPENAI_API_KEY=sk-... -``` - -## API Reference - -Every factory pair follows the same shape: the short factory (`openaiText`, `openaiImage`, …) reads `OPENAI_API_KEY` from the environment, while the `create*` variant takes an explicit API key. Both take `model` as the first argument. - -### `openaiText(model, config?)` - -Creates an OpenAI text adapter against the Responses API (`/v1/responses`) using `OPENAI_API_KEY` from the environment. - -**Parameters:** - -- `model` - OpenAI chat model id (e.g. `"gpt-5.2"`, `"gpt-4o-mini"`) -- `config?.organization` - Organization ID (optional) -- `config?.baseURL` - Custom base URL (optional) - -### `createOpenaiChat(model, apiKey, config?)` - -Creates an OpenAI text adapter (Responses API) with an explicit API key. - -### `openaiChatCompletions(model, config?)` +Default for diarize: `response_format: "diarized_json"` + `chunking_strategy: "auto"`. `known_speaker_names` + `known_speaker_references` together (≤4, matching lengths). No `prompt` / `include` / `timestamp_granularities` with diarized. -Creates an OpenAI text adapter that targets `/v1/chat/completions` instead of the Responses API. See [Chat Completions API](#chat-completions-api) for when to use this over `openaiText`. +## API reference -### `createOpenaiChatCompletions(model, apiKey, config?)` +Short factories use `OPENAI_API_KEY`; `create*` takes explicit key. -Creates an OpenAI chat-completions adapter with an explicit API key. +| Factory | Purpose | +| --- | --- | +| `openaiText` / `createOpenaiChat` | Responses API chat | +| `openaiChatCompletions` / `createOpenaiChatCompletions` | Chat Completions | +| `openaiSummarize` / `createOpenaiSummarize` | Summarize | +| `openaiImage` / `createOpenaiImage` | Image | +| `openaiSpeech` / `createOpenaiSpeech` | TTS | +| `openaiTranscription` / `createOpenaiTranscription` | STT | +| `openaiVideo` / `createOpenaiVideo` | Sora (experimental) | +| `openaiRealtime` / `openaiRealtimeToken` | [Realtime](../media/realtime-chat) | -### `openaiSummarize(model, config?)` / `createOpenaiSummarize(model, apiKey, config?)` +Config: `organization?`, `baseURL?`. -Creates an OpenAI summarization adapter. +## Provider tools -### `openaiImage(model, config?)` / `createOpenaiImage(model, apiKey, config?)` - -Creates an OpenAI image generation adapter (DALL-E, gpt-image). - -### `openaiSpeech(model, config?)` / `createOpenaiSpeech(model, apiKey, config?)` - -Creates an OpenAI text-to-speech adapter. - -### `openaiTranscription(model, config?)` / `createOpenaiTranscription(model, apiKey, config?)` - -Creates an OpenAI transcription adapter for Whisper, GPT-4o transcription, and GPT-4o diarized transcription models. - -### `openaiVideo(model, config?)` / `createOpenaiVideo(model, apiKey, config?)` - -Creates an OpenAI video generation adapter (Sora). _Experimental._ - -### `openaiRealtime(...)` / `openaiRealtimeToken(...)` - -Realtime voice adapters. See [Realtime Voice Chat](../media/realtime-chat) for usage. - -## Next Steps - -- [Getting Started](../getting-started/quick-start) - Learn the basics -- [Tools Guide](../tools/tools) - Learn about tools -- [Other Adapters](./anthropic) - Explore other providers - -## Provider Tools - -OpenAI exposes several native tools beyond user-defined function calls. -Import them from `@tanstack/ai-openai/tools` and pass them into -`chat({ tools: [...] })`. - -> For the full concept, a comparison matrix, and type-gating details, see -> [Provider Tools](../tools/provider-tools.md). +From `@tanstack/ai-openai/tools`. Matrix: [Provider Tools](../tools/provider-tools.md). ### `webSearchTool` -Enables the model to run a web search and return grounded results with -citations. Pass a `WebSearchToolConfig` object (typed from the OpenAI SDK) -to configure the tool. - ```typescript import { chat } from "@tanstack/ai"; import { openaiText } from "@tanstack/ai-openai"; @@ -444,14 +296,8 @@ const stream = chat({ }); ``` -**Supported models:** GPT-4o, GPT-5, and Responses API-capable models. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). - ### `webSearchPreviewTool` -The preview variant of web search with additional options for controlling -search context size and user location. Use this when you want fine-grained -control over the search context sent to the model. - ```typescript import { chat } from "@tanstack/ai"; import { openaiText } from "@tanstack/ai-openai"; @@ -469,14 +315,8 @@ const stream = chat({ }); ``` -**Supported models:** GPT-4o and above. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). - ### `fileSearchTool` -Searches OpenAI vector stores that you have pre-populated, letting the model -retrieve relevant document chunks. Provide the `vector_store_ids` to search -and optionally limit results with `max_num_results` (1–50). - ```typescript import { chat } from "@tanstack/ai"; import { openaiText } from "@tanstack/ai-openai"; @@ -495,13 +335,8 @@ const stream = chat({ }); ``` -**Supported models:** GPT-4o and above. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). - ### `imageGenerationTool` -Allows the model to generate images inline during a conversation using -DALL-E/GPT-Image. Pass quality, size, and style options via the config object. - ```typescript import { chat } from "@tanstack/ai"; import { openaiText } from "@tanstack/ai-openai"; @@ -519,14 +354,8 @@ const stream = chat({ }); ``` -**Supported models:** GPT-5 and GPT-Image-capable models. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). - ### `codeInterpreterTool` -Gives the model a sandboxed Python execution environment. The `container` -field configures the execution environment; pass the full -`CodeInterpreterToolConfig` object. - ```typescript import { chat } from "@tanstack/ai"; import { openaiText } from "@tanstack/ai-openai"; @@ -541,14 +370,8 @@ const stream = chat({ }); ``` -**Supported models:** GPT-4o and above. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). - ### `mcpTool` -Connects the model to a remote MCP (Model Context Protocol) server, exposing -all its capabilities as callable tools. Provide either `server_url` or -`connector_id` — not both. - ```typescript import { chat } from "@tanstack/ai"; import { openaiText } from "@tanstack/ai-openai"; @@ -566,14 +389,10 @@ const stream = chat({ }); ``` -**Supported models:** GPT-4o and above. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). +Provide `server_url` **or** `connector_id`, not both. ### `computerUseTool` -Lets the model observe a virtual desktop via screenshots and interact with -it using keyboard and mouse events. Provide the display dimensions and the -execution environment type. - ```typescript import { chat } from "@tanstack/ai"; import { openaiText } from "@tanstack/ai-openai"; @@ -593,17 +412,12 @@ const stream = chat({ }); ``` -**Supported models:** `computer-use-preview`. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). - -### `localShellTool` - -Provides the model with a local shell for executing system commands. Takes no -arguments — the tool is enabled simply by including it in the `tools` array. +### `localShellTool` / `shellTool` ```typescript import { chat } from "@tanstack/ai"; import { openaiText } from "@tanstack/ai-openai"; -import { localShellTool } from "@tanstack/ai-openai/tools"; +import { localShellTool, shellTool } from "@tanstack/ai-openai/tools"; const stream = chat({ adapter: openaiText("gpt-5.2"), @@ -612,33 +426,7 @@ const stream = chat({ }); ``` -**Supported models:** GPT-5.x and other agent-capable models. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). - -### `shellTool` - -A function-style shell tool that exposes shell execution as a structured -function call. Pass an `environment` object to attach container config and -hosted skills. - -```typescript -import { chat } from "@tanstack/ai"; -import { openaiText } from "@tanstack/ai-openai"; -import { shellTool } from "@tanstack/ai-openai/tools"; - -const stream = chat({ - adapter: openaiText("gpt-5.2"), - messages: [{ role: "user", content: "Count lines in all JS files" }], - tools: [shellTool()], -}); -``` - -**Supported models:** GPT-5.x and other agent-capable models. Responses API -only — Chat Completions does not support the shell tool. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). - -#### Attaching hosted skills - -Pass `environment.skills` to load provider-managed skill bundles into the -shell's container (Responses API only). +`shellTool` supports hosted skills via `environment.skills` (Responses only). See [Provider Skills](../tools/provider-skills.md). ```typescript import { chat, toServerSentEventsResponse } from "@tanstack/ai"; @@ -667,14 +455,8 @@ export async function POST(request: Request) { } ``` -For the full reference — skill shape, `version` string format, and the -Anthropic equivalent — see [Provider Skills](../tools/provider-skills.md). - ### `applyPatchTool` -Lets the model apply unified-diff patches to modify files directly. Takes no -arguments — include it in the `tools` array to enable patch application. - ```typescript import { chat } from "@tanstack/ai"; import { openaiText } from "@tanstack/ai-openai"; @@ -687,15 +469,8 @@ const stream = chat({ }); ``` -**Supported models:** GPT-5.x and other agent-capable models. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). - ### `customTool` -Defines a custom Responses API tool with an explicit name, description, and -format. Use this when none of the structured tool types fits your use case. -Unlike branded provider tools, `customTool` returns a plain `Tool` and is -accepted by any chat model. - ```typescript import { chat } from "@tanstack/ai"; import { openaiText } from "@tanstack/ai-openai"; @@ -714,4 +489,8 @@ const stream = chat({ }); ``` -**Supported models:** all Responses API models. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). +## Next steps + +- [Getting Started](../getting-started/quick-start) +- [Tools](../tools/tools) +- [Other Adapters](./anthropic) diff --git a/docs/adapters/opencode.md b/docs/adapters/opencode.md index 23ccbb453..5230b7f7e 100644 --- a/docs/adapters/opencode.md +++ b/docs/adapters/opencode.md @@ -2,7 +2,7 @@ title: OpenCode id: opencode-adapter order: 14 -description: "Use OpenCode as a chat backend in TanStack AI — agent harness with local tool execution, token-level streaming, stateful sessions, and tool bridging via @tanstack/ai-opencode." +description: "OpenCode harness adapter — local agent loop, token streaming, sessions via @tanstack/ai-opencode." keywords: - tanstack ai - opencode @@ -13,28 +13,23 @@ keywords: - adapter --- -The OpenCode adapter runs [OpenCode](https://opencode.ai) as a chat backend, driving it over its local HTTP server (`@opencode-ai/sdk`). Unlike HTTP provider adapters, this is a **harness adapter**: OpenCode runs its own agent loop and executes its own tools — shell commands, file reads and edits, search — locally on your server. Each `chat()` call runs one full harness turn; assistant text and reasoning stream as true token-level deltas, and the harness's tool activity streams back as already-resolved tool-call events your UI can render. +If you need OpenCode as a chat backend → **server-only**: install CLI + package, auth providers, call `opencodeText("provider/model", { directory, permissionMode })`. -> **Server-only.** The adapter spawns (or attaches to) an `opencode serve` process, so it only works in a Node.js server environment — never in the browser. Treat it like giving OpenCode a shell on the machine it runs on, and configure permissions accordingly. +> Spawns or attaches to `opencode serve`. Never in the browser. -## Installation +Demos: [`examples/sandbox-cloudflare`](https://github.com/TanStack/ai/tree/main/examples/sandbox-cloudflare), [`examples/sandbox-web`](https://github.com/TanStack/ai/tree/main/examples/sandbox-web). -```bash -npm install @tanstack/ai-opencode -``` - -The `opencode` CLI must be installed and its providers authenticated on the host: +## Install ```bash +npm install @tanstack/ai-opencode npm install -g opencode-ai opencode auth login ``` -A runnable demo lives at [`examples/sandbox-cloudflare`](https://github.com/TanStack/ai/tree/main/examples/sandbox-cloudflare) — pick Claude Code, Codex, or Grok Build in the UI, with session resume, the harness tool timeline, and tool bridging, wired into a TanStack Start app on Workers. For the same wiring on plain Node with durable, refresh-surviving runs (Claude Code on Docker), see [`examples/sandbox-web`](https://github.com/TanStack/ai/tree/main/examples/sandbox-web) — swapping in this adapter is a one-line change (`src/sandbox-agent.ts`). +## Do this -## Models - -OpenCode is provider-agnostic: it resolves any `provider/model` id its configured providers support. Address models as `provider/model` (the adapter splits on the first `/`): +Models: `provider/model` (split on first `/`). ```typescript import { chat } from "@tanstack/ai"; @@ -51,33 +46,32 @@ const stream = chat({ ## Configuration -| Option | Description | -| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `directory` | Working directory for the harness session. Defaults to `process.cwd()`. | -| `baseUrl` | Attach to an already-running `opencode serve` (e.g. `http://127.0.0.1:4096`) instead of spawning a new server per turn. | -| `hostname` | Hostname for the spawned server. Defaults to the SDK default (`127.0.0.1`). | -| `port` | Port for the spawned server. Defaults to the SDK default (`4096`). | -| `permissionMode` | `'default'` (bridged tools run, everything else that prompts is rejected), `'acceptEdits'` (also auto-approves file edits), or `'bypassPermissions'` (allow all). | -| `onPermissionRequest` | Custom permission handler; replaces the default policy entirely. | -| `config` | Extra OpenCode config merged with the adapter's MCP and permission config. | +| Option | Description | +| --- | --- | +| `directory` | Working dir (default `process.cwd()`) | +| `baseUrl` | Attach to existing `opencode serve` (e.g. `http://127.0.0.1:4096`) | +| `hostname` / `port` | Spawned server (defaults `127.0.0.1` / `4096`) | +| `permissionMode` | `'default'` \| `'acceptEdits'` \| `'bypassPermissions'` | +| `onPermissionRequest` | Custom permission handler | +| `config` | Extra OpenCode config (MCP, permissions) | -Per-call overrides — `sessionId`, `permissionMode`, `directory` — go through `modelOptions`. +`modelOptions`: `sessionId`, `permissionMode`, `directory`. ## Permissions -OpenCode asks for permission before mutating files or running commands. A headless server has no one to answer those prompts, so the adapter applies a policy automatically — it never hangs a turn: - -- **`'default'`** — bridged TanStack tools run; anything else that would prompt (edits, shell, web fetch) is rejected. -- **`'acceptEdits'`** — additionally auto-approves file-mutation requests (edit / write / patch). -- **`'bypassPermissions'`** — approves everything. Only use this against a sandbox or scratch directory. +Headless policy never hangs: -Provide `onPermissionRequest` to implement your own policy (e.g. allow-list specific commands). +- **`default`** — bridged tools run; edits/shell/fetch that would prompt → rejected +- **`acceptEdits`** — also auto-approves file mutations +- **`bypassPermissions`** — allow all (sandbox/scratch only) -## Stateful Sessions +## Stateful sessions -OpenCode sessions are stateful — the harness keeps the full working context (files read, commands run, conclusions reached) between turns. The adapter surfaces the session id of every fresh run as a custom stream event named `opencode.session-id`; thread it back via `modelOptions.sessionId` to resume. When resuming, only the latest user message is sent — the harness already holds the prior context. +1. Capture `opencode.session-id` CUSTOM event. +2. Pass `modelOptions.sessionId`. +3. Send only the latest user message. -Server endpoint: +**Server:** ```typescript import { @@ -90,7 +84,6 @@ import { opencodeText } from "@tanstack/ai-opencode"; export async function POST(request: Request) { const params = await chatParamsFromRequest(request); - // Extra fields the client puts in the connection `body` arrive here. const sessionId = typeof params.forwardedProps.sessionId === "string" ? params.forwardedProps.sessionId @@ -109,7 +102,7 @@ export async function POST(request: Request) { } ``` -Client (React) — capture the session id from the custom event and send it back on subsequent requests: +**Client:** ```typescript import { useState } from "react"; @@ -135,21 +128,15 @@ function CodingAssistant() { } }, }); - - // ... render messages; harness tool activity (bash, edit, read, ...) - // arrives as regular tool-call parts with results. } ``` -Sessions live on the server that ran them, so resuming only works against the same server instance (or a shared `baseUrl`). +Resume on same server (or shared `baseUrl`). ## Tools -Two kinds of tools flow through this adapter: - -1. **Built-in harness tools** are executed by OpenCode itself and stream back as tool-call events with results already attached: `bash`, `edit`, `write`, `read`, `grep`, and the agent's running todo plan (surfaced as an `opencode.todo` custom event). Your code never executes them. - -2. **Your TanStack tools** are bridged *into* the harness: the adapter starts a short-lived Streamable-HTTP MCP server on `127.0.0.1` for the duration of the turn and registers it with OpenCode. Define tools as usual with `toolDefinition().server()`; tool-call events come back under the names you registered (OpenCode prefixes MCP tools `tanstack_…` internally, which the adapter strips). +1. **Harness tools** — `bash`, `edit`, `write`, `read`, `grep`; todo as `opencode.todo` CUSTOM. +2. **Your tools** — Streamable-HTTP MCP on `127.0.0.1` for the turn (`tanstack_` prefix stripped). ```typescript import { z } from "zod"; @@ -171,16 +158,15 @@ const stream = chat({ }); ``` -**Client-side and approval-gated tools are not supported.** The harness executes tools inside a live process, which cannot pause across HTTP requests to wait for a browser round-trip or a human approval. Passing a tool without a server `execute()` implementation — or one marked `needsApproval` — fails fast with a descriptive error. Run those tools outside the harness with a regular provider adapter. +No client-side / `needsApproval` tools — fails fast. -## Structured Output +## Structured output -`structuredOutput()` is best-effort: OpenCode's prompt API has no native JSON-schema channel, so the schema is embedded as a prompt instruction in a fresh, one-shot session and the final text is parsed (markdown fences are stripped when present). It works for finalization after a chat, but a plain provider adapter (e.g. `@tanstack/ai-openai`) is the better choice when structured extraction is the primary job — it's faster, deterministic, and doesn't spawn a harness. +Best-effort: schema in prompt, parse final text. Prefer a plain provider for primary extraction. -## Limitations +## Notes -- **Server-only (Node).** The adapter spawns or attaches to an `opencode serve` process. -- **The harness owns the agent loop.** TanStack's agent-loop strategies and per-iteration middleware don't apply inside a harness turn. -- **No sampling controls.** `temperature`-style options don't exist here. -- **Sessions are server-local.** Resume requires hitting the same server instance (or a shared `baseUrl`). -- **Cold starts.** Spawning a server per turn adds first-token latency; point the adapter at a long-lived `baseUrl` to avoid it. +- Server-only (Node) +- Harness owns agent loop; no sampling controls +- Sessions server-local +- Reduce cold start with long-lived `baseUrl` diff --git a/docs/adapters/openrouter.md b/docs/adapters/openrouter.md index 8bc02dba9..5439231ab 100644 --- a/docs/adapters/openrouter.md +++ b/docs/adapters/openrouter.md @@ -1,7 +1,7 @@ --- title: OpenRouter Adapter id: openrouter-adapter -description: "Access 300+ LLMs from OpenAI, Anthropic, Google, Meta, Mistral, and more through a single API with OpenRouter in TanStack AI." +description: "300+ models through one OpenRouter key via @tanstack/ai-openrouter." keywords: - tanstack ai - openrouter @@ -12,27 +12,33 @@ keywords: - adapter --- -OpenRouter is TanStack AI's first official AI partner and the recommended starting point for most projects. It provides access to 300+ models from OpenAI, Anthropic, Google, Meta, Mistral, and many more — all through a single API key and unified interface. +If you need many providers via one key → install, set `OPENROUTER_API_KEY`, call `openRouterText("provider/model")`. -## Installation +Models: [openrouter.ai/models](https://openrouter.ai/models) (`openai/gpt-5.1`, `anthropic/claude-sonnet-4.5`, …). + +## Install ```bash npm install @tanstack/ai-openrouter ``` -## Basic Usage +```bash +OPENROUTER_API_KEY=sk-or-... +``` + +## Do this ```typescript import { chat } from "@tanstack/ai"; import { openRouterText } from "@tanstack/ai-openrouter"; - + const stream = chat({ adapter: openRouterText("openai/gpt-5"), - messages: [{ role: "user", content: "Hello!" }], + messages: [{ role: "user", content: "Hello!" }], }); ``` -## Configuration +### Explicit key / rankings headers ```typescript import { createOpenRouterText } from "@tanstack/ai-openrouter"; @@ -41,46 +47,14 @@ const adapter = createOpenRouterText( "openai/gpt-5", process.env.OPENROUTER_API_KEY!, { - serverURL: "https://openrouter.ai/api/v1", // Optional - httpReferer: "https://your-app.com", // Optional, for rankings - appTitle: "Your App Name", // Optional, for rankings + serverURL: "https://openrouter.ai/api/v1", + httpReferer: "https://your-app.com", + appTitle: "Your App Name", }, ); ``` -## Available Models - -OpenRouter provides access to 300+ models from various providers. Models use the format `provider/model-name`: - -```text -model: "openai/gpt-5.1" -model: "anthropic/claude-sonnet-4.5" -model: "google/gemini-3.1-pro-preview" -model: "meta-llama/llama-4-maverick" -model: "deepseek/deepseek-v3.2" -``` - -See the full list at [openrouter.ai/models](https://openrouter.ai/models). - -## Example: Chat Completion - -```typescript -import { chat, toServerSentEventsResponse } from "@tanstack/ai"; -import { openRouterText } from "@tanstack/ai-openrouter"; - -export async function POST(request: Request) { - const { messages } = await request.json(); - - const stream = chat({ - adapter: openRouterText("openai/gpt-5"), - messages, - }); - - return toServerSentEventsResponse(stream); -} -``` - -## Example: With Tools +### Server + tools ```typescript import { chat, toServerSentEventsResponse, toolDefinition } from "@tanstack/ai"; @@ -111,20 +85,8 @@ export async function POST(request: Request) { return toServerSentEventsResponse(stream); } ``` - - - -## Environment Variables - -Set your API key in environment variables: - -```bash -OPENROUTER_API_KEY=sk-or-... -``` -## Model Routing - -OpenRouter can automatically route requests to the best available provider: +## Model routing ```typescript import { chat, toServerSentEventsResponse } from "@tanstack/ai"; @@ -149,9 +111,9 @@ export async function POST(request: Request) { } ``` -## Model Options +## Model options -OpenRouter supports various provider-specific options. Sampling parameters live here too — `temperature`, `topP`, and `maxCompletionTokens` (OpenRouter's token-limit key for the chat adapter) — rather than as root-level props on `chat()`: +Token limit key: `maxCompletionTokens`. ```typescript import { chat, toServerSentEventsResponse } from "@tanstack/ai"; @@ -174,24 +136,16 @@ export async function POST(request: Request) { } ``` -> If you previously passed `temperature` / `topP` / `maxTokens` at the root of `chat()`, see [Moving Sampling Options into modelOptions](../migration/sampling-options-to-model-options). +> Root-level sampling migration: [modelOptions](../migration/sampling-options-to-model-options). ## Chat Completions vs Responses (beta) -OpenRouter exposes two OpenAI-compatible wire formats, and the adapter -package ships one of each: - -| Adapter | Endpoint | Status | When to use | -| -------------------------- | ------------------------- | -------- | ---------------------------------------------------------------------------- | -| `openRouterText` | `/v1/chat/completions` | Stable | Default for almost everything. Broadest model + tool support. | -| `openRouterResponsesText` | `/v1/responses` | Beta | OpenAI Responses-shaped request/response; richer multi-turn state on OpenAI-style models. | +| Adapter | Endpoint | Status | When | +| --- | --- | --- | --- | +| `openRouterText` | `/v1/chat/completions` | Stable | Default — broadest support | +| `openRouterResponsesText` | `/v1/responses` | Beta | Responses wire format | -Both adapters route to any underlying model OpenRouter supports -(`anthropic/...`, `google/...`, `meta-llama/...`, etc.) — the wire format -describes how your client talks to OpenRouter, not which provider answers. -`/v1/responses` is OpenAI's newer API surface; OpenRouter implements it so -clients that prefer that wire format can use it across the same 300+ -model catalogue. +Wire format is client↔OpenRouter, not which underlying model answers. ```typescript import { chat } from "@tanstack/ai"; @@ -203,24 +157,11 @@ const stream = chat({ }); ``` -Caveats while the Responses adapter is in beta: - -- Function tools are supported; OpenRouter's branded server-tools (web - search, file search, …) are not yet wired through this path — use - `openRouterText` if you need those. -- If in doubt, prefer `openRouterText`. The Chat Completions endpoint has - broader provider coverage and feature parity today. +**Beta caveats:** function tools OK; branded server-tools (web search, file search) not on Responses path yet — use `openRouterText`. -## Cost Tracking +## Cost tracking -OpenRouter reports the actual cost of each request inline on the streamed -response. When present, the adapter forwards it on the terminal `RUN_FINISHED` -event under `usage.cost`, with OpenRouter's per-request breakdown under -`usage.costDetails`. This is the cost OpenRouter itself reports for the -request — it is **not** computed locally from token counts, so it already -accounts for routing, fallback providers, BYOK, and cached-token pricing. See -OpenRouter's [Usage Accounting](https://openrouter.ai/docs/use-cases/usage-accounting) -docs for the meaning and units of these fields. +OpenRouter cost on `RUN_FINISHED` → `usage.cost` / `usage.costDetails` (provider-reported, not local token math). Also on middleware `onUsage` / `onFinish`. Absent when OpenRouter omits it. ```typescript ignore import { chat, type RunFinishedEvent, type StreamChunk } from "@tanstack/ai"; @@ -241,39 +182,16 @@ for await (const chunk of chat({ } ``` -The same `usage` (including `cost` / `costDetails`) is passed to middleware via -the `onUsage` and `onFinish` hooks. When OpenRouter does not report a cost, the -fields are simply absent and the stream completes normally. Both -`openRouterText` and `openRouterResponsesText` populate cost when OpenRouter -returns it. - -## Next Steps - -- [Getting Started](../getting-started/quick-start) - Learn the basics -- [Tools Guide](../tools/tools) - Learn about tools - -## Provider Tools +Docs: [Usage Accounting](https://openrouter.ai/docs/use-cases/usage-accounting). -> **Migrated from `createWebSearchTool`?** This factory was renamed to -> `webSearchTool` and moved to the `/tools` subpath in this release. -> See [Migration Guide §6](../migration/migration.md#6-provider-tools-moved-to-tools-subpath) -> for the exact before/after. +## Provider tools -OpenRouter's gateway exposes web search via a plugin that works across -any proxied chat model. Import it from `@tanstack/ai-openrouter/tools`. - -> For the full concept, a comparison matrix, and type-gating details, see -> [Provider Tools](../tools/provider-tools.md). +From `@tanstack/ai-openrouter/tools`. +`createWebSearchTool` → `webSearchTool` on `/tools` ([migration](../migration/migration.md#6-provider-tools-moved-to-tools-subpath)). +Matrix: [Provider Tools](../tools/provider-tools.md). ### `webSearchTool` -Adds web search capability to any OpenRouter-proxied chat model. The factory -accepts OpenRouter's `WebSearchConfig` directly — pick the `engine` -(`auto`, `native`, `exa`, `firecrawl`, or `parallel`), cap results with -`maxResults` / `maxTotalResults`, restrict which sites can appear in results -with `allowedDomains` / `excludedDomains`, and optionally pass -`searchContextSize` or `userLocation` for finer control. - ```typescript import { chat } from "@tanstack/ai"; import { openRouterText } from "@tanstack/ai-openrouter"; @@ -292,24 +210,10 @@ const stream = chat({ }); ``` -**Supported models:** all OpenRouter chat models. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). +Engines: `auto`, `native`, `exa`, `firecrawl`, `parallel`. ### `webFetchTool` -Lets any OpenRouter-proxied chat model fetch the full contents of a URL the -model chooses, instead of running a search. The factory accepts OpenRouter's -`WebFetchServerToolConfig` directly — pick the fetch `engine` (`auto` — the -default, `native`, `openrouter`, `exa`, or `firecrawl`), cap how much page -content the model receives with `maxContentTokens`, cap how many fetches the -model can make per request with `maxUses`, and restrict which URLs the model -can fetch with `allowedDomains` / `blockedDomains`. - -> The `native` engine routes to the underlying provider's own fetch (for -> example, Anthropic's `web_fetch` on Claude models). Native fetch -> capabilities vary, so `allowedDomains` and `blockedDomains` may be -> ignored. Use `openrouter`, `exa`, or `firecrawl` if you need consistent -> behaviour across models. - ```typescript import { chat } from "@tanstack/ai"; import { openRouterText } from "@tanstack/ai-openrouter"; @@ -330,5 +234,9 @@ const stream = chat({ }); ``` -**Supported models:** all OpenRouter chat models. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). +Engines: `auto`, `native`, `openrouter`, `exa`, `firecrawl`. `native` may ignore domain filters. + +## Next steps +- [Getting Started](../getting-started/quick-start) +- [Tools](../tools/tools) diff --git a/docs/advanced/built-in-middleware.md b/docs/advanced/built-in-middleware.md index 87cd59eec..6c6e2ac4b 100644 --- a/docs/advanced/built-in-middleware.md +++ b/docs/advanced/built-in-middleware.md @@ -2,7 +2,7 @@ title: Built-in Middleware id: built-in-middleware order: 2 -description: "Ready-made TanStack AI chat() middleware — toolCacheMiddleware for caching tool results, contentGuardMiddleware for redacting streamed text, and otelMiddleware for OpenTelemetry tracing." +description: "Drop-in chat() middleware — toolCacheMiddleware, contentGuardMiddleware, otelMiddleware." keywords: - tanstack ai - middleware @@ -13,21 +13,21 @@ keywords: - opentelemetry --- -TanStack AI ships ready-made middleware so you don't have to hand-roll the common cases. Each one is an ordinary [`ChatMiddleware`](./middleware) — drop it into the `middleware` array of any `chat()` call. This page documents every built-in. +If you need caching, redaction, or OTel → pass one of these into `chat({ middleware })`. Each is a normal [`ChatMiddleware`](./middleware). -| Middleware | Import | What it does | -|------------|--------|--------------| -| `toolCacheMiddleware` | `@tanstack/ai/middlewares` | Cache tool-call results by name + arguments | -| `contentGuardMiddleware` | `@tanstack/ai/middlewares` | Redact / transform / block streamed text content | -| `otelMiddleware` | `@tanstack/ai/middlewares/otel` | Emit OpenTelemetry spans + GenAI metrics | +| Middleware | Import | Role | +|------------|--------|------| +| `toolCacheMiddleware` | `@tanstack/ai/middlewares` | Cache tool results by name + args | +| `contentGuardMiddleware` | `@tanstack/ai/middlewares` | Redact / transform / block text chunks | +| `otelMiddleware` | `@tanstack/ai/middlewares/otel` | OpenTelemetry spans + GenAI metrics | -> `toolCacheMiddleware` and `contentGuardMiddleware` are exported from the main `@tanstack/ai/middlewares` barrel. `otelMiddleware` lives on its own subpath (`@tanstack/ai/middlewares/otel`) so that importing the barrel never eagerly pulls in `@opentelemetry/api` (an optional peer dependency). +`otelMiddleware` is on its own subpath so the barrel never pulls `@opentelemetry/api` (optional peer). -For app-owned policies (for example tool-call budgets), see the [tool-call budget recipe](../chat/agentic-cycle#tool-call-budgets-middleware-recipe) — those stay in your code, not in `@tanstack/ai/middlewares`. +App-owned policies (tool-call budgets, etc.) stay in your code — see the [tool-call budget recipe](../chat/agentic-cycle#tool-call-budgets-middleware-recipe). ## toolCacheMiddleware -Caches tool call results based on tool name and arguments. When a tool is called with the same name and arguments as a previous call, the cached result is returned immediately without re-executing the tool. +Skip re-running a tool when name + args match a prior successful call. ```typescript import { chat } from "@tanstack/ai"; @@ -41,32 +41,31 @@ const stream = chat({ tools: [weatherTool, stockTool], middleware: [ toolCacheMiddleware({ - ttl: 60_000, // Cache entries expire after 60 seconds - maxSize: 50, // Keep at most 50 entries (LRU eviction) - toolNames: ["getWeather"], // Only cache specific tools + ttl: 60_000, + maxSize: 50, + toolNames: ["getWeather"], }), ], }); ``` -**Options:** +### Options -| Option | Type | Default | Description | -|--------|------|---------|-------------| -| `maxSize` | `number` | `100` | Maximum cache entries. Oldest evicted first (LRU). Only applies to the default in-memory storage. | -| `ttl` | `number` | `Infinity` | Time-to-live in milliseconds. Expired entries are not served. | -| `toolNames` | `string[]` | All tools | Only cache these tools. Others pass through. | -| `keyFn` | `(toolName, args) => string` | `JSON.stringify([toolName, args])` | Custom cache key derivation. | -| `storage` | `ToolCacheStorage` | In-memory Map | Custom storage backend. When provided, `maxSize` is ignored — the storage manages its own capacity. | +| Option | Type | Default | Notes | +|--------|------|---------|-------| +| `maxSize` | `number` | `100` | LRU cap (default in-memory only) | +| `ttl` | `number` | `Infinity` | ms; expired entries not served | +| `toolNames` | `string[]` | all tools | Others pass through | +| `keyFn` | `(toolName, args) => string` | `JSON.stringify([toolName, args])` | Custom key | +| `storage` | `ToolCacheStorage` | in-memory Map | Custom backend; ignores `maxSize` | -**Behaviors:** +### Behavior -- Only successful tool calls are cached — errors are never stored -- Cache hits trigger `{ type: 'skip', result }` via `onBeforeToolCall` -- LRU eviction: when `maxSize` is reached, the oldest entry is removed (default storage only) -- Cache hits refresh the entry's LRU position (moved to most-recently-used) +1. Only successful calls are cached (errors never stored). +2. Hits return `{ type: 'skip', result }` via `onBeforeToolCall`. +3. LRU: at `maxSize`, oldest drops; hits move to most-recent. -**Custom key function** — useful when you want to ignore certain arguments: +### Custom key ```typescript import { toolCacheMiddleware } from "@tanstack/ai/middlewares"; @@ -77,8 +76,6 @@ function isRecord(value: unknown): value is Record { toolCacheMiddleware({ keyFn: (toolName, args) => { - // Ignore pagination, cache by query only. `args` is `unknown`, so - // narrow it with a type guard before destructuring. if (!isRecord(args)) return JSON.stringify([toolName, args]); const { page, ...rest } = args; return JSON.stringify([toolName, rest]); @@ -86,28 +83,24 @@ toolCacheMiddleware({ }); ``` -### Custom Storage +### Custom storage -By default the cache lives in-memory and is scoped to a single `toolCacheMiddleware()` instance. Pass a `storage` option to use an external backend like Redis, localStorage, or a database. This also enables **sharing a cache across multiple `chat()` calls**. - -The storage interface: +Default cache is per-instance in-memory. Pass `storage` for Redis, localStorage, DB, or a shared Map across `chat()` calls. ```typescript import { type ToolCacheEntry, type ToolCacheStorage } from "@tanstack/ai/middlewares"; -// Implement this interface (exported from `@tanstack/ai/middlewares`): +// ToolCacheEntry = { result: unknown; timestamp: number } interface MyStorage extends ToolCacheStorage { getItem: (key: string) => ToolCacheEntry | undefined | Promise; setItem: (key: string, value: ToolCacheEntry) => void | Promise; deleteItem: (key: string) => void | Promise; } - -// ToolCacheEntry is { result: unknown; timestamp: number } ``` -All methods may return a `Promise` for async backends. The middleware handles TTL checking — your storage just needs to store and retrieve entries. +Methods may be async. Middleware handles TTL; storage only stores/retrieves. -**Redis example:** +**Redis:** ```typescript import { chat } from "@tanstack/ai"; @@ -139,7 +132,7 @@ const stream = chat({ }); ``` -**Sharing a cache across requests:** +**Share across requests:** ```typescript import { chat, toServerSentEventsResponse } from "@tanstack/ai"; @@ -147,14 +140,16 @@ import { toolCacheMiddleware, type ToolCacheStorage } from "@tanstack/ai/middlew import { globalCache, app, adapter } from "./server"; import { weatherTool } from "./tools"; -// Create storage once, reuse across chat() calls const sharedStorage: ToolCacheStorage = { getItem: (key) => globalCache.get(key), - setItem: (key, value) => { globalCache.set(key, value); }, - deleteItem: (key) => { globalCache.delete(key); }, + setItem: (key, value) => { + globalCache.set(key, value); + }, + deleteItem: (key) => { + globalCache.delete(key); + }, }; -// Both requests share the same cache app.post("/api/chat", async (req: { body: { messages: unknown[] } }) => { const stream = chat({ adapter, @@ -168,7 +163,7 @@ app.post("/api/chat", async (req: { body: { messages: unknown[] } }) => { ## contentGuardMiddleware -Filters or transforms streamed text content as it flows through `onChunk`. Use it to redact sensitive data (SSNs, emails, API keys), enforce a profanity filter, or rewrite text on the fly. Rules are applied to `TEXT_MESSAGE_CONTENT` chunks; all other chunk types pass through untouched. +Filter or rewrite streamed text in `onChunk`. Targets `TEXT_MESSAGE_CONTENT` only; other chunk types pass through. ```typescript import { chat } from "@tanstack/ai"; @@ -181,9 +176,7 @@ const stream = chat({ middleware: [ contentGuardMiddleware({ rules: [ - // Regex + replacement { pattern: /\b\d{3}-\d{2}-\d{4}\b/g, replacement: "[SSN REDACTED]" }, - // Custom transform function { fn: (text) => text.replaceAll("badword", "****") }, ], strategy: "buffered", @@ -192,31 +185,32 @@ const stream = chat({ }); ``` -**Options:** +### Options -| Option | Type | Default | Description | -|--------|------|---------|-------------| -| `rules` | `ContentGuardRule[]` | — | **Required.** Applied in order; each rule receives the previous rule's output. A rule is either `{ pattern: RegExp; replacement: string }` or `{ fn: (text: string) => string }`. | -| `strategy` | `'delta' \| 'buffered'` | `'buffered'` | How content is matched. See below. | -| `bufferSize` | `number` | `50` | (Buffered only) Characters held back before emitting, so patterns spanning chunk boundaries still match. Set it ≥ the longest pattern you expect. Flushed at stream end. | -| `blockOnMatch` | `boolean` | `false` | When `true`, drop the entire chunk if any rule changes the content (instead of emitting the filtered version). | -| `onFiltered` | `(info: ContentFilteredInfo) => void` | — | Callback fired whenever a rule changes content. Receives `{ messageId, original, filtered, strategy }`. | +| Option | Type | Default | Notes | +|--------|------|---------|-------| +| `rules` | `ContentGuardRule[]` | — | **Required.** In order; each gets previous output. Rule = `{ pattern, replacement }` or `{ fn }` | +| `strategy` | `'delta' \| 'buffered'` | `'buffered'` | See below | +| `bufferSize` | `number` | `50` | Buffered only; look-behind so cross-chunk patterns match. ≥ longest pattern | +| `blockOnMatch` | `boolean` | `false` | Drop whole chunk if any rule changes content | +| `onFiltered` | `(info) => void` | — | Observability: `{ messageId, original, filtered, strategy }` | -**Matching strategies:** +### Strategies -- **`'buffered'` (default)** — Accumulates content and applies rules to the settled portion, holding back a `bufferSize` look-behind window so a pattern split across two chunks (`"...123-45"` then `"-6789..."`) is still caught. The buffer is flushed when the message or run ends. Use this for anything that can span deltas — which is most redaction. -- **`'delta'`** — Applies rules to each delta in isolation as it arrives. Fastest and lowest-latency, but a pattern split across a chunk boundary may slip through. Use only when your patterns are guaranteed to fit within a single delta. +| Strategy | Use when | +|----------|----------| +| `'buffered'` (default) | Patterns can span deltas (most redaction). Holds `bufferSize` chars; flushes at message/run end | +| `'delta'` | Patterns always fit one delta; lowest latency. Split patterns can slip through | -**Behaviors:** +### Behavior -- Only `TEXT_MESSAGE_CONTENT` chunks are inspected; every other chunk type passes through. -- A rule that doesn't change the text is a no-op — the chunk passes through unchanged. -- With `blockOnMatch: true`, a matched chunk is dropped entirely (returns `null` from `onChunk`) rather than emitting the redacted text. -- The `onFiltered` callback is for observability/audit — it fires with the before/after text but does not alter what is emitted. +1. Non-matching rules are no-ops. +2. `blockOnMatch: true` → return `null` from `onChunk` (drop), not redacted text. +3. `onFiltered` does not change what is emitted. ## otelMiddleware -Emits vendor-neutral OpenTelemetry traces and metrics for every `chat()` call — a root span per call, a child span per agent-loop iteration, and a grandchild span per tool execution, all tagged with [GenAI semantic-convention attributes](https://opentelemetry.io/docs/specs/semconv/gen-ai/). +Root span per call, child per agent-loop iteration, grandchild per tool — [GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). ```typescript import { chat } from "@tanstack/ai"; @@ -227,7 +221,7 @@ import { messages } from "./server"; const otel = otelMiddleware({ tracer: trace.getTracer("my-app"), - meter: metrics.getMeter("my-app"), // optional — enables GenAI histograms + meter: metrics.getMeter("my-app"), // optional GenAI histograms }); const result = await chat({ @@ -237,13 +231,13 @@ const result = await chat({ }); ``` -`otelMiddleware` has its own configuration surface (content capture, redaction, span-name formatting, attribute enrichment, lifecycle callbacks) and requires the optional `@opentelemetry/api` peer dependency. See the dedicated [OpenTelemetry](./otel) guide for full setup, the span/metric catalogue, and all options. +Full options, span/metric catalogue, privacy: [OpenTelemetry](./otel). -## Writing your own +## Write your own -These built-ins are just `ChatMiddleware` objects — nothing about them is privileged. To build your own, see the [Middleware](./middleware) guide for the full hook reference, the context object, and composition rules. +Built-ins are plain `ChatMiddleware`. Hook reference and composition: [Middleware](./middleware). -## Next Steps +## Related -- [Middleware](./middleware) — the full lifecycle and hook reference -- [OpenTelemetry](./otel) — `otelMiddleware` in depth \ No newline at end of file +- [Middleware](./middleware) — lifecycle + hooks +- [OpenTelemetry](./otel) — `otelMiddleware` in depth diff --git a/docs/advanced/debug-logging.md b/docs/advanced/debug-logging.md index 12867ecf9..fb5fcceb8 100644 --- a/docs/advanced/debug-logging.md +++ b/docs/advanced/debug-logging.md @@ -2,7 +2,7 @@ title: Debug Logging id: debug-logging order: 3 -description: "Turn on structured, category-toggleable debug logging to see every chunk, middleware transform, and tool call inside TanStack AI." +description: "Toggle structured debug logs for chunks, middleware, tools, and provider frames in TanStack AI." keywords: - tanstack ai - debug @@ -14,14 +14,10 @@ keywords: - middleware debugging --- -# Debug Logging - -You have a `chat()` that isn't behaving as expected — a missing chunk, a middleware that doesn't seem to fire, a tool call with wrong args. By the end of this guide, you'll have turned on debug logging and will see every chunk, middleware transform, and tool call flowing through your call. +If a `chat()` misbehaves → set `debug: true` (or a category map) and inspect the stream. ## Turn it on -Add `debug: true` to any activity call: - ```typescript import { chat } from "@tanstack/ai"; import { openaiText } from "@tanstack/ai-openai"; @@ -33,19 +29,18 @@ const stream = chat({ }); ``` -Every internal event now prints to the console with a `[tanstack-ai:]` prefix: +Console prefix: `[tanstack-ai:]` ``` [tanstack-ai:request] activity=chat provider=openai model=gpt-5.5 messages=1 tools=0 stream=true [tanstack-ai:agentLoop] run started [tanstack-ai:provider] provider=openai type=response.output_text.delta [tanstack-ai:output] type=TEXT_MESSAGE_CONTENT -... ``` -## Narrow what's printed +## Narrow categories -Pass a `DebugConfig` object instead of `true`. Every unspecified category defaults to `true`, so toggle by setting specific flags to `false`: +Omit a flag → it defaults to `true`. Turn off what you don't need: ```typescript import { chat } from "@tanstack/ai"; @@ -54,11 +49,11 @@ import { openaiText } from "@tanstack/ai-openai"; chat({ adapter: openaiText("gpt-5.5"), messages: [{ role: "user", content: "Hello" }], - debug: { middleware: false }, // everything except middleware + debug: { middleware: false }, }); ``` -If you want to see ONLY a specific set of categories, set the rest to `false` explicitly. Errors default to `true` — keep them on unless you really want total silence: +Only specific categories — set the rest to `false`. Keep `errors` on unless you want silence: ```typescript import { chat } from "@tanstack/ai"; @@ -74,15 +69,13 @@ chat({ tools: false, agentLoop: false, config: false, - errors: true, // keep errors on — they're cheap and important + errors: true, request: false, }, }); ``` -## Pipe into your own logger - -Pass a `Logger` implementation and all debug output flows through it instead of `console`: +## Pipe to your logger ```typescript import { chat, type Logger } from "@tanstack/ai"; @@ -93,29 +86,23 @@ import { messages } from "./server"; const pinoLogger = pino(); const logger: Logger = { debug: (msg, meta) => pinoLogger.debug(meta, msg), - info: (msg, meta) => pinoLogger.info(meta, msg), - warn: (msg, meta) => pinoLogger.warn(meta, msg), + info: (msg, meta) => pinoLogger.info(meta, msg), + warn: (msg, meta) => pinoLogger.warn(meta, msg), error: (msg, meta) => pinoLogger.error(meta, msg), }; chat({ adapter: openaiText("gpt-5.5"), messages, - debug: { logger }, // all categories on, piped to pino + debug: { logger }, }); ``` -The default logger is exported as `ConsoleLogger` if you want to wrap it: - -```typescript -import { ConsoleLogger } from "@tanstack/ai"; -``` - -### Your `Logger` is wrapped in a try/catch +Default export: `ConsoleLogger` from `@tanstack/ai`. -If your `Logger` implementation throws — a cyclic-meta `JSON.stringify`, a transport that rejects synchronously, a typo in a bound `this` — the exception is swallowed so it never masks the real error that triggered the log call (for example, a provider SDK failure inside the chat stream). You won't see the log line, but the pipeline error still surfaces through thrown exceptions and `RUN_ERROR` chunks. +### Logger failures are swallowed -If you need to know when your own logger is failing, guard inside your implementation: +Your `Logger` runs in try/catch so a broken logger never masks the real pipeline error. To notice logger failures, guard inside your implementation: ```typescript import { type Logger } from "@tanstack/ai"; @@ -127,45 +114,45 @@ const logger: Logger = { try { pinoLogger.debug(meta, msg); } catch (err) { - // surface to wherever you track infra errors process.stderr.write(`logger failed: ${String(err)}\n`); } }, - info: (msg, meta) => pinoLogger.info(meta, msg), - warn: (msg, meta) => pinoLogger.warn(meta, msg), + info: (msg, meta) => pinoLogger.info(meta, msg), + warn: (msg, meta) => pinoLogger.warn(meta, msg), error: (msg, meta) => pinoLogger.error(meta, msg), }; ``` -## Categories reference +## Categories -| Category | Logs | Applies to | -|----------|------|------------| -| `request` | Outgoing call to a provider (model, message count, tool count) | All activities | -| `provider` | Every raw chunk/frame received from a provider SDK | Streaming activities (`chat`, `realtime`, and streaming `generateAudio`/`generateSpeech`/`generateTranscription`) | -| `output` | Every chunk or result yielded to the caller | All activities | -| `middleware` | Inputs and outputs around every middleware hook | `chat()` only | -| `tools` | Before/after tool call execution | `chat()` only | -| `agentLoop` | Agent-loop iterations and phase transitions | `chat()` only | -| `config` | Config transforms returned by middleware `onConfig` hooks | `chat()` only | -| `errors` | Every caught error anywhere in the pipeline | All activities | +| Category | Logs | Scope | +|----------|------|-------| +| `request` | Outgoing call (model, message/tool counts) | All activities | +| `provider` | Raw SDK chunks/frames | Streaming (`chat`, `realtime`, streaming media) | +| `output` | Chunks/results to the caller | All activities | +| `middleware` | Hook in/out | `chat()` only | +| `tools` | Before/after tool execution | `chat()` only | +| `agentLoop` | Iterations and phase transitions | `chat()` only | +| `config` | `onConfig` transforms | `chat()` only | +| `errors` | Caught pipeline errors | All activities | -## Errors are always logged +## Errors without debug -Errors flow through the logger unconditionally — even when you omit `debug`: +Errors still log when you omit `debug`: ```typescript import { chat } from "@tanstack/ai"; import { adapter } from "./server"; -chat({ adapter, messages: [{ role: "user", content: "Hello" }] }); // still prints [tanstack-ai:errors] ... on failure +chat({ adapter, messages: [{ role: "user", content: "Hello" }] }); +// still prints [tanstack-ai:errors] on failure ``` -To fully silence (including errors), set `debug: false` or `debug: { errors: false }`. Errors also always reach the caller via thrown exceptions or `RUN_ERROR` stream chunks — the logger is additive, not the only surface. +Silence everything (including errors): `debug: false` or `debug: { errors: false }`. Errors also surface via thrown exceptions / `RUN_ERROR` chunks — the logger is additive. ## Non-chat activities -The same `debug` option works on every activity: +Same `debug` option: ```typescript import { @@ -174,7 +161,6 @@ import { generateSpeech, generateAudio, generateTranscription, - type Logger, } from "@tanstack/ai"; import { adapter } from "./server"; import { logger } from "./logger"; @@ -189,10 +175,8 @@ generateAudio({ adapter, prompt: "ambient piano", debug: true }); generateTranscription({ adapter, audio, debug: { provider: true } }); ``` -When streaming any of these (`generateAudio`, `generateSpeech`, `generateTranscription` with `stream: true`), the `provider` category emits the raw SDK chunks and `output` emits the AG-UI-shaped chunks yielded to the caller — useful when a media pipeline looks stuck or the bytes arriving don't match what you expected. - -The chat-only categories (`middleware`, `tools`, `agentLoop`, `config`) simply never fire for these activities because those concepts don't exist in their pipelines. +Streaming media: `provider` = raw SDK frames, `output` = AG-UI-shaped chunks. Chat-only categories never fire. ## Related -If you're building middleware and want to see chunks flow through it, `debug: { middleware: true }` is faster than writing a logging middleware. See [Middleware](./middleware) for writing your own middleware. +Middleware inspection: `debug: { middleware: true }` beats a hand-rolled logger. See [Middleware](./middleware). diff --git a/docs/advanced/extend-adapter.md b/docs/advanced/extend-adapter.md index bad7fc136..c6752c42a 100644 --- a/docs/advanced/extend-adapter.md +++ b/docs/advanced/extend-adapter.md @@ -2,7 +2,7 @@ title: Extend Adapter id: extend-adapter order: 8 -description: "Extend TanStack AI adapter factories with custom model IDs and fine-tuned models while keeping full type safety for input modalities and provider options." +description: "Add custom or fine-tuned model IDs to adapter factories with createModel + extendAdapter." keywords: - tanstack ai - extendAdapter @@ -13,85 +13,54 @@ keywords: - adapter factory --- -# Extending Adapters with Custom Models +If you need typed fine-tuned or proxy model IDs → `createModel` + `extendAdapter` on an existing factory. -The `extendAdapter` utility allows you to extend existing adapter factories (like `openaiText`, `anthropicText`) with custom model names while maintaining full type safety for input modalities and provider options. - -## Basic Usage +## Minimal path ```typescript -import { createModel, extendAdapter } from '@tanstack/ai' +import { chat, createModel, extendAdapter } from '@tanstack/ai' import { openaiText } from '@tanstack/ai-openai' -// Define your custom models using createModel helper -const myOpenaiModel = createModel('my-fine-tuned-gpt4',['text', 'image']); -const myOpenaiModelButCooler = createModel('my-fine-tuned-gpt5',['text', 'image']); - +const myOpenaiModel = createModel('my-fine-tuned-gpt4', ['text', 'image']) +const myOpenaiModelButCooler = createModel('my-fine-tuned-gpt5', ['text', 'image']) -// Create an extended adapter factory - simple API, no type parameters needed const myOpenai = extendAdapter(openaiText, [ myOpenaiModel, - myOpenaiModelButCooler + myOpenaiModelButCooler, ]) -// Use with original models - full type inference preserved +// Original models keep full inference const gpt5Adapter = myOpenai('gpt-5.5') -// Use with custom models - your custom types are applied +// Custom models use your modalities const customAdapter = myOpenai('my-fine-tuned-gpt4') -// Works seamlessly with chat() -import { chat } from '@tanstack/ai' - const stream = chat({ adapter: myOpenai('my-fine-tuned-gpt4'), - messages: [{ role: 'user', content: 'Hello!' }] + messages: [{ role: 'user', content: 'Hello!' }], }) ``` -## The `createModel` Helper +## createModel -The `createModel` function provides a clean way to define custom models with full type inference: +**Positional** — name + input modalities: ```typescript import { createModel } from '@tanstack/ai' -// Arguments define name and input modalities const model = createModel( - 'my-model', // model name (literal type inferred) - ['text', 'image'] // input modalities (tuple type inferred) + 'my-model', + ['text', 'image'], // 'text' | 'image' | 'audio' | 'video' | 'document' ) ``` - - -## Model Definition Structure - -A custom model definition (`ExtendedModelDef`) has the required properties `name`, `input`, and `modelOptions`, plus optional `features` and `tools`. The two `createModel` overloads let you fill these in two ways. - -### Defining Input Modalities - -The positional form takes a model name and an `input` array specifying which content types your model supports: - -```typescript -import { createModel } from '@tanstack/ai' - -const models = [ - createModel('text-only-model', ['text']), - createModel('multimodal-model', ['text', 'image', 'audio']), -] as const -``` - -Available modalities: `'text'`, `'image'`, `'audio'`, `'video'`, `'document'` - -### Capabilities-object form -To attach typed `modelOptions`, declared `features`, or provider `tools` to a custom model, use the second `createModel` overload, which takes a capabilities object as its second argument: +**Capabilities object** — modalities, features, tools, typed `modelOptions`: ```typescript import { createModel } from '@tanstack/ai' import type { OpenAITextProviderOptions } from '@tanstack/ai-openai' -// Type brand for provider options — the value is unused at runtime. +// Type brand only — value unused at runtime const modelOptions: OpenAITextProviderOptions = {} const reasoner = createModel('my-reasoner', { @@ -102,112 +71,84 @@ const reasoner = createModel('my-reasoner', { }) ``` -- `input` — supported input modalities (same as the positional form). -- `features` — declared feature flags (e.g. `'reasoning'`, `'structured_outputs'`). -- `tools` — declared provider tools (e.g. `'web_search'`). -- `modelOptions` — a type brand for the provider options accepted by this model; the value is unused at runtime, so declare an empty object typed as the provider options (e.g. `const modelOptions: OpenAITextProviderOptions = {}`). - -## Preserving Original Factory Behavior +`ExtendedModelDef` requires `name`, `input`, `modelOptions`; optional `features`, `tools`. -`extendAdapter` fully preserves the original factory's signature, including any configuration parameters: +## Factory config preserved ```typescript import { createModel, extendAdapter } from '@tanstack/ai' import { openaiText } from '@tanstack/ai-openai' -const customModels = [createModel('my-fine-tuned-gpt4', ['text', 'image'])] as const +const customModels = [ + createModel('my-fine-tuned-gpt4', ['text', 'image']), +] as const const myOpenai = extendAdapter(openaiText, customModels) -// Config parameter is preserved const adapter = myOpenai('my-fine-tuned-gpt4', { baseURL: 'https://my-proxy.com/v1', - timeout: 30000 + timeout: 30000, }) ``` -## Type Safety - -The extended adapter provides full type safety: +## Type safety ```typescript ignore import { extendAdapter, createModel } from '@tanstack/ai' import { openaiText } from '@tanstack/ai-openai' -const myOpenai = extendAdapter(openaiText, [createModel('custom-model', ['text'])]) - -// ✅ Original models work with their original types -const a1 = myOpenai('gpt-5.5') - -// ✅ Custom models work with your defined types -const a2 = myOpenai('custom-model') +const myOpenai = extendAdapter(openaiText, [ + createModel('custom-model', ['text']), +]) -// ❌ Type error: invalid model name -// Note: Type checking works when you assign the result to a variable -const invalid = myOpenai('nonexistent-model') // TypeScript error! +const a1 = myOpenai('gpt-5.5') // original +const a2 = myOpenai('custom-model') // custom +// const invalid = myOpenai('nonexistent-model') // TypeScript error ``` - -## Runtime Behavior +## Runtime -At runtime, `extendAdapter` simply passes through to the original factory: +No runtime validation of custom names — the original factory receives what you pass. Use for: -- No validation is performed on custom model names -- The original factory receives exactly what you pass -- This allows the original provider's API to handle the model name +1. Fine-tuned IDs the provider accepts but types don't list +2. Proxies with custom identifiers +3. Compile-time safety without runtime cost -This design is intentional - it allows you to: -- Use fine-tuned model names that the provider accepts but TypeScript doesn't know about -- Proxy requests to different backends that accept custom model identifiers -- Add type safety without runtime overhead - -## Example: OpenAI-Compatible Proxy - -A common use case is typing models for an OpenAI-compatible proxy: +## OpenAI-compatible proxy ```typescript import { extendAdapter, createModel } from '@tanstack/ai' import { openaiText } from '@tanstack/ai-openai' -// Models available through your proxy const proxyModels = [ - createModel( - 'llama-3.1-70b', - ['text'] - ), - createModel( - 'mixtral-8x7b', - ['text'] - ), + createModel('llama-3.1-70b', ['text']), + createModel('mixtral-8x7b', ['text']), ] as const const proxyAdapter = extendAdapter(openaiText, proxyModels) -// Use with your proxy's base URL const adapter = proxyAdapter('llama-3.1-70b', { - baseURL: 'https://my-llm-proxy.com/v1' + baseURL: 'https://my-llm-proxy.com/v1', }) ``` -## Example: Fine-tuned Models - -Adding type safety for your fine-tuned models: +## Fine-tuned Anthropic ```typescript import { chat, createModel, extendAdapter } from '@tanstack/ai' import { anthropicText } from '@tanstack/ai-anthropic' const fineTunedModels = [ - createModel( - 'ft:claude-3-opus:my-org:custom-task:abc123', - ['text', 'image'] - ), + createModel('ft:claude-3-opus:my-org:custom-task:abc123', [ + 'text', + 'image', + ]), ] as const const myAnthropic = extendAdapter(anthropicText, fineTunedModels) chat({ adapter: myAnthropic('ft:claude-3-opus:my-org:custom-task:abc123'), - messages: [{ role: 'user', content: 'Analyze this...' }] + messages: [{ role: 'user', content: 'Analyze this...' }], }) ``` diff --git a/docs/advanced/locks.md b/docs/advanced/locks.md index 0922c51a1..6f5d87f0a 100644 --- a/docs/advanced/locks.md +++ b/docs/advanced/locks.md @@ -2,7 +2,7 @@ title: Locks id: locks order: 3 -description: "Cross-instance mutual exclusion with LockStore and withLocks — coordination middleware for multi-worker critical sections (e.g. sandbox ensure)." +description: "Cross-instance mutual exclusion with LockStore and withLocks for multi-worker critical sections." keywords: - tanstack ai - locks @@ -16,33 +16,27 @@ keywords: - coordination --- -Locks answer a different question from persistence: +If multiple processes might enter the same critical section for one key → use locks. Locks are **not** persistence. | Concern | Question | Seam | | --- | --- | --- | | **State** | What is durable? | Stores + `withPersistence` | -| **Locks** | Who may run this critical section right now? | `LockStore` + `withLocks` | +| **Locks** | Who may run this critical section now? | `LockStore` + `withLocks` | -They live in **`@tanstack/ai`** as a middleware capability — not in -`@tanstack/ai-persistence`, and never as a key on `AIPersistence.stores`. +Lives in `@tanstack/ai` (middleware capability), not `@tanstack/ai-persistence`. ## When you need them -Use locks when **more than one process or isolate** might enter the same critical -section for the same key: +**Must use locks when** more than one process/isolate can hit the same key: -- **Sandbox resume-or-create** (`withSandbox` / `ensure`) — two concurrent runs - for the same thread must not both create a provider sandbox. See - [Sandbox Instance Durability](../sandbox/durability). -- **Your own middleware** — any multi-writer work you want to serialize across - workers (e.g. a custom “one active job per thread” gate). +1. Sandbox resume-or-create (`withSandbox` / `ensure`) — concurrent runs must not both create a sandbox. See [Sandbox Instance Durability](../sandbox/durability). +2. Custom multi-writer middleware (e.g. one active job per thread). -You do **not** need locks for: +**Skip locks for:** -- Single-process local dev (optional: `InMemoryLockStore` is fine). -- Ordinary chat state durability — that is stores, not mutexes. -- Automatically locking an entire `chat()` turn — `withLocks` only **provides** - the capability; consumers call `withLock` when they need exclusion. +- Single-process local dev (`InMemoryLockStore` is fine if you want one anyway) +- Chat state durability (use stores) +- Auto-locking a whole `chat()` turn — `withLocks` only **provides** the capability; consumers call `withLock` ## Wire it up @@ -58,18 +52,14 @@ chat({ adapter: grokBuildText('grok-build'), messages, middleware: [ - // Single process. Multi-instance: pass a distributed LockStore instead. - withLocks(new InMemoryLockStore()), - // later middleware can getLocks(ctx) / withSandbox will use the same token + withLocks(new InMemoryLockStore()), // multi-instance: pass a distributed LockStore ], }) ``` -Capability identity is by **object reference**. `withLocks` provides the shared -`LocksCapability` from core; any later middleware that reads that token (including -`@tanstack/ai-sandbox`) sees the same store. +Capability identity is by **object reference**. `withLocks` provides `LocksCapability`; later middleware (including `@tanstack/ai-sandbox`) read the same store. -Typical order when composing with sandbox: +With sandbox, provide locks first: ```ts import { withLocks, InMemoryLockStore } from '@tanstack/ai/locks' @@ -80,45 +70,36 @@ declare const sandbox: SandboxDefinition const middleware = [ withLocks(new InMemoryLockStore()), - withSandbox(sandbox), // after providers + withSandbox(sandbox), ] ``` -## The contract +## Contract ```ts import type { LockStore } from '@tanstack/ai/locks' declare const locks: LockStore -// Mutual exclusion for a key; lease-backed impls abort `signal` on loss. await locks.withLock('thread:abc', async (signal) => { - // critical section — pass `signal` to cancellable work when using leases + // critical section — pass signal to cancellable work under leases void signal }) ``` | Piece | Role | | --- | --- | -| `LockStore` | Interface: `withLock(key, fn)` | -| `withLocks(store)` | Chat middleware that provides `LocksCapability` | -| `InMemoryLockStore` | Process-local implementation (promise chain per key) | -| `getLocks` / `provideLocks` | Low-level capability accessors for custom middleware | +| `LockStore` | `withLock(key, fn)` | +| `withLocks(store)` | Middleware that provides `LocksCapability` | +| `InMemoryLockStore` | Process-local (promise chain per key) | +| `getLocks` / `provideLocks` | Capability accessors for custom middleware | -`InMemoryLockStore` is correct **within one process only**. It serializes -callers for the same key, does not poison the chain when a critical section -throws, and never aborts its signal (ownership cannot be lost in-process). +`InMemoryLockStore` only serializes **within one process**. Throws don't poison the chain; signal never aborts (ownership can't be lost in-process). -## Implement a lock - -Wrap your own mutual-exclusion primitive with `defineLock`. It types the object -against the contract inline (autocomplete, no `: LockStore` annotation). Acquire -the key, run `fn`, and release when `fn` settles (whether it resolves or throws): +## Implement a store ```ts import { defineLock } from '@tanstack/ai/locks' -// Your distributed primitive. `acquire` waits until the key is free and returns -// a `release` (plus, for leases, a `signal` that fires when ownership is lost). import { acquire } from './my-lock-backend' export const locks = defineLock({ @@ -133,29 +114,19 @@ export const locks = defineLock({ }) ``` -Wire it as middleware with `withLocks(locks)`. The requirements a production -store must meet are covered below. +Then: `withLocks(locks)`. ## Distributed locks and leases -Multi-instance deployments need a **distributed** implementation (Durable Object, -Redis, etc.). A good store: +Multi-instance needs a distributed backend (Durable Object, Redis, …). A production store must: -1. Serializes owners per `key`. -2. Uses **leases** (or equivalent) so a crashed owner cannot block forever. -3. Passes an `AbortSignal` into `fn`; when the lease is lost, **abort** so the - callback stops starting externally visible work and passes the signal to - cancellable dependencies. +1. Serialize owners per `key` +2. Use **leases** so a crashed owner can't block forever +3. Abort `signal` when the lease is lost so `fn` stops starting external work -Callbacks that ignore `signal` still type-check (`() => Promise` is -assignable), but lease-backed backends cannot protect you if the critical -section keeps mutating after abort. +Ignoring `signal` still type-checks; lease backends can't protect you if work continues after abort. -There is no shared lock conformance suite in the chat store testkit — write -targeted tests for concurrency, release-on-throw, and lease expiry for your -backend. The Cloudflare Durable Object recipe lives in the -`ai-persistence/build-cloudflare-adapter` agent skill (app-owned file, -not a shipped package). +No shared conformance suite — test concurrency, release-on-throw, and lease expiry yourself. Cloudflare Durable Object recipe: `ai-persistence/build-cloudflare-adapter` agent skill (app-owned, not a package). ## Consume in custom middleware @@ -169,20 +140,18 @@ const serializePerThread = defineChatMiddleware({ async onStart(ctx) { const locks = getLocks(ctx) await locks.withLock(`thread:${ctx.threadId}`, async (signal) => { - // critical section — honor `signal` under lease-backed locks void signal }) }, }) ``` -Or provide without `withLocks` by calling `provideLocks` in your own -`setup` hook if you already own a custom middleware. +Or call `provideLocks` in your own `setup` instead of `withLocks`. -## See also +## Related - [Middleware](./middleware) — capability bus and lifecycle -- [Sandboxes](../sandbox/overview) — sandbox middleware overview -- [Sandbox Instance Durability](../sandbox/durability) — primary product consumer (`withSandbox` / `ensure`) -- [Persistence Controls](../persistence/controls) — compose state stores from different systems +- [Sandboxes](../sandbox/overview) — sandbox middleware +- [Sandbox Instance Durability](../sandbox/durability) — `withSandbox` / `ensure` +- [Persistence Controls](../persistence/controls) — compose state stores - [Build Your Own Adapter](../persistence/build-your-own-adapter) — chat store contracts diff --git a/docs/advanced/middleware.md b/docs/advanced/middleware.md index 80fe5fed4..373308f4a 100644 --- a/docs/advanced/middleware.md +++ b/docs/advanced/middleware.md @@ -2,7 +2,7 @@ title: Middleware id: middleware order: 1 -description: "Hook into every stage of TanStack AI's chat() lifecycle with middleware — logging, analytics, stream transforms, tool interception, and side effects." +description: "Hook into chat() lifecycle stages — config, stream, tools, usage, and terminal outcomes." keywords: - tanstack ai - middleware @@ -14,19 +14,11 @@ keywords: - stream transform --- -Middleware lets you hook into every stage of the `chat()` lifecycle — from configuration to streaming, tool execution, usage tracking, and completion. You can observe, transform, or short-circuit behavior at each stage without modifying your adapter or tool implementations. +If you need to observe, transform, or short-circuit a `chat()` run → pass `middleware: ChatMiddleware[]`. -Common use cases include: +**Typical uses:** logging/usage, config transforms, stream redaction, tool interception, side effects. -- **Logging and observability** — track token usage, tool execution timing, errors -- **Configuration transforms** — inject system prompts, adjust temperature per iteration, filter tools -- **Stream processing** — redact sensitive content, transform chunks, drop unwanted events -- **Tool call interception** — validate arguments, cache results, abort on dangerous calls -- **Side effects** — send analytics, update databases, trigger notifications - -## Quick Start - -Pass an array of middleware to the `chat()` function: +## Quick start ```typescript import { chat, type ChatMiddleware } from "@tanstack/ai"; @@ -49,12 +41,9 @@ const stream = chat({ }); ``` -> **Just want to see chunks flowing through your middleware during development?** -> Use `debug: { middleware: true }` on your `chat()` call — no custom middleware required. See [Debug Logging](./debug-logging). - -## Lifecycle Overview +Dev-only chunk visibility without custom middleware: `debug: { middleware: true }` — [Debug Logging](./debug-logging). -Every `chat()` invocation follows a predictable lifecycle. Middleware hooks fire at specific phases: +## Lifecycle ```mermaid graph TD @@ -91,26 +80,24 @@ graph TD style P fill:#ffe1e1 ``` -### Phase Transitions +### Phases -The context's `phase` field tracks where you are in the lifecycle: - -| Phase | When | Hooks Called | -|-------|------|-------------| +| Phase | When | Hooks | +|-------|------|-------| | `init` | Once at startup | `onConfig` | -| `beforeModel` | Before each model call (per iteration) | `onConfig` | -| `modelStream` | While adapter streams chunks | `onChunk`, `onUsage` | -| `beforeTools` | Before tool execution | `onBeforeToolCall` | -| `afterTools` | After tool execution | `onAfterToolCall` | -| `structuredOutput` | During the final structured-output adapter call (when `outputSchema` is set **and** the adapter does not declare `supportsCombinedToolsAndSchema()`). Chunks from `adapter.structuredOutputStream` (or the synthesized non-streaming fallback) flow through `onChunk` with this phase, and `onUsage` fires for the final call's tokens. **Does not fire** for adapters that natively combine tools + schema in one streaming call (modern OpenAI Chat Completions, OpenAI Responses, Claude 4.5+, Gemini 3.x, Grok 4.x family — see issue #605); on that path middleware observes the run through `beforeModel` / `modelStream` as usual. | `onStructuredOutputConfig`, `onConfig`, `onChunk`, `onUsage` | +| `beforeModel` | Before each model call | `onConfig` | +| `modelStream` | Adapter streaming | `onChunk`, `onUsage` | +| `beforeTools` | Before tools | `onBeforeToolCall` | +| `afterTools` | After tools | `onAfterToolCall` | +| `structuredOutput` | Final structured-output call when `outputSchema` is set **and** adapter lacks `supportsCombinedToolsAndSchema()`. Does **not** fire for native-combined adapters (modern OpenAI Chat Completions/Responses, Claude 4.5+, Gemini 3.x, Grok 4.x — issue #605) | `onStructuredOutputConfig`, `onConfig`, `onChunk`, `onUsage` | -## Hooks Reference +With `outputSchema` on a single-iteration separate-finalization run, `onConfig` fires three times: `init` → `beforeModel` → `structuredOutput`. -### onConfig +## Hooks -Called once during `init` (startup) and once per iteration during `beforeModel` (before each model call). When `chat()` was invoked with `outputSchema`, `onConfig` additionally re-fires at the structured-output boundary with `ctx.phase === 'structuredOutput'`, receiving the post-`onStructuredOutputConfig` view of the config — so a single-iteration run with `outputSchema` fires `onConfig` three times (`init` + `beforeModel` + `structuredOutput`). Use it to transform the configuration that the model receives. +### onConfig -Return a **partial** config object with only the fields you want to change — they are shallow-merged with the current config automatically. No need to spread the existing config. +Transform config at `init`, each `beforeModel`, and (if applicable) `structuredOutput`. Return a **partial** object — shallow-merged. No need to spread the full config. ```typescript import { type ChatMiddleware } from "@tanstack/ai"; @@ -119,7 +106,6 @@ const dynamicTemperature: ChatMiddleware = { name: "dynamic-temperature", onConfig: (ctx, config) => { if (ctx.phase === "init") { - // Add a system prompt at startup — only systemPrompts is overwritten return { systemPrompts: [ ...config.systemPrompts, @@ -129,10 +115,6 @@ const dynamicTemperature: ChatMiddleware = { } if (ctx.phase === "beforeModel" && ctx.iteration > 0) { - // Increase temperature on retries. Sampling params live in the - // provider-native modelOptions object — `temperature` is universal, - // so it's the same key across providers. Spread the existing - // modelOptions so other model options stay unchanged. const current = typeof config.modelOptions?.temperature === "number" ? config.modelOptions.temperature @@ -148,9 +130,7 @@ const dynamicTemperature: ChatMiddleware = { }; ``` -> Sampling parameters (`temperature`, `top_p` / `topP`, the various `max*Tokens` keys) live inside `modelOptions` under each provider's native name — they are no longer root config fields. `temperature` happens to be spelled the same across every provider, so the example above is provider-agnostic; if you mutate a token limit instead, use the provider-native key (e.g. `max_output_tokens` for OpenAI, `num_predict` nested under `modelOptions.options` for Ollama). See [Moving Sampling Options into modelOptions](../migration/sampling-options-to-model-options). - -**Config fields you can transform:** +Sampling params live under `modelOptions` with provider-native keys — see [Moving Sampling Options into modelOptions](../migration/sampling-options-to-model-options). | Field | Type | Description | |-------|------|-------------| @@ -158,17 +138,13 @@ const dynamicTemperature: ChatMiddleware = { | `systemPrompts` | `string[]` | System prompts | | `tools` | `Tool[]` | Available tools | | `metadata` | `Record` | Request metadata | -| `modelOptions` | `Record` | Provider-native options — this is where sampling params (`temperature`, `top_p` / `topP`, the provider's `max*Tokens` key) now live, alongside every other model-specific knob. See [Moving Sampling Options into modelOptions](../migration/sampling-options-to-model-options). | +| `modelOptions` | `Record` | Provider-native options (incl. sampling) | -When multiple middleware define `onConfig`, the config is **piped** through them in order — each receives the merged config from the previous middleware. +Multiple `onConfig` hooks **pipe** in array order. ### onStructuredOutputConfig -Called once at the start of the final structured-output adapter call — only when `chat()` was invoked with `outputSchema` **and** the adapter takes the legacy finalization path (i.e. does not declare `supportsCombinedToolsAndSchema()`). Pipes through middleware in order, like `onConfig`, but with access to the **JSON Schema** being sent to the provider. Use this hook when you need to transform the schema (e.g., inject `$defs`, strip vendor-incompatible keywords) or apply structured-output-specific behavior (e.g., suppress system prompts on the final call). - -> Native-combined adapters (modern OpenAI, Claude 4.5+, Gemini 3.x, Grok 4.x — see issue #605) skip the separate finalization call and never invoke this hook. If you need to mutate the schema for a native-combined adapter, do it in `onConfig` (the schema is on `config.modelOptions` / the request — adapter-specific). - -Return a **partial** `StructuredOutputMiddlewareConfig` with only the fields you want to change — they are shallow-merged with the current config. Return `void` to pass through. +Fires once at structured-output finalization — only on the **legacy** path (no `supportsCombinedToolsAndSchema()`). Access/transform the JSON Schema. Native-combined adapters skip this hook; mutate schema in `onConfig` instead (adapter-specific location). ```typescript import { type ChatMiddleware } from "@tanstack/ai"; @@ -177,7 +153,6 @@ import { sharedDefs } from "./defs"; const injectDefs: ChatMiddleware = { name: "inject-defs", onStructuredOutputConfig: (_ctx, config) => { - // `config.outputSchema` is the JSON Schema being sent to the provider return { outputSchema: { ...config.outputSchema, @@ -188,26 +163,19 @@ const injectDefs: ChatMiddleware = { }; ``` -**Config fields you can transform:** - | Field | Type | Description | |-------|------|-------------| -| `messages` | `ModelMessage[]` | Conversation history sent to the final call | -| `systemPrompts` | `SystemPrompt[]` | System prompts on the final call | -| `metadata` | `Record` | Request metadata | -| `modelOptions` | `Record` | Provider-native options — this is where sampling params (`temperature`, `top_p` / `topP`, the provider's `max*Tokens` key) now live, alongside every other model-specific knob. See [Moving Sampling Options into modelOptions](../migration/sampling-options-to-model-options). | -| `outputSchema` | `JSONSchema` | JSON Schema being sent to the provider for structured output | - -**Ordering at the structured-output boundary:** +| `messages` | `ModelMessage[]` | Final-call history | +| `systemPrompts` | `SystemPrompt[]` | Final-call system prompts | +| `metadata` | `Record` | Metadata | +| `modelOptions` | `Record` | Provider options | +| `outputSchema` | `JSONSchema` | Schema sent to provider | -1. `onStructuredOutputConfig` fires first, piping through every middleware in array order. -2. `onConfig` then re-fires at the same boundary with `ctx.phase === 'structuredOutput'`, receiving the post-`onStructuredOutputConfig` view of the config (minus `outputSchema`). Use `onConfig` for general-purpose transforms that apply to every adapter call; use `onStructuredOutputConfig` when you need access to the schema. - -When multiple middleware define `onStructuredOutputConfig`, the config is **piped** through them in order — each receives the merged config from the previous middleware. +**Order at boundary:** (1) `onStructuredOutputConfig` pipes, (2) `onConfig` re-fires with `ctx.phase === 'structuredOutput'` (config minus `outputSchema`). ### onStart -Called once after the initial `onConfig` completes. Use it for setup tasks like initializing timers or logging. +Once after initial `onConfig`. Setup / logging. ```typescript import { type ChatMiddleware } from "@tanstack/ai"; @@ -222,7 +190,14 @@ const timer: ChatMiddleware = { ### onChunk -Called for every chunk streamed from the adapter. You can observe, transform, expand, or drop chunks. +Every streamed chunk. Return: + +| Return | Effect | +|--------|--------| +| `void` / `undefined` | Pass through | +| `StreamChunk` | Replace | +| `StreamChunk[]` | Expand | +| `null` | Drop (later middleware never see it) | ```typescript ignore import { type ChatMiddleware } from "@tanstack/ai"; @@ -231,51 +206,34 @@ const redactor: ChatMiddleware = { name: "redactor", onChunk: (ctx, chunk) => { if (chunk.type === "TEXT_MESSAGE_CONTENT") { - // Transform: redact sensitive content return { ...chunk, delta: chunk.delta.replace(/\b\d{3}-\d{2}-\d{4}\b/g, "[REDACTED]"), }; } - // Return void to pass through unchanged }, }; ``` -**Return values:** - -| Return | Effect | -|--------|--------| -| `void` / `undefined` | Chunk passes through unchanged | -| `StreamChunk` | Replaces the original chunk | -| `StreamChunk[]` | Expands into multiple chunks | -| `null` | Drops the chunk entirely | - -When multiple middleware define `onChunk`, chunks flow through them in order. If one middleware drops a chunk (returns `null`), subsequent middleware never see it. - -#### Chunk types you'll see +#### Chunk types -`onChunk` receives every [AG-UI event](https://docs.ag-ui.com/introduction) the run produces — not just text. Narrow on `chunk.type` (a discriminated union) before reading type-specific fields. The common ones: +[AG-UI events](https://docs.ag-ui.com/introduction). Narrow on `chunk.type`: | `chunk.type` | Meaning | Key fields | -|--------------|---------|-----------| -| `RUN_STARTED` / `RUN_FINISHED` / `RUN_ERROR` | Run lifecycle boundaries | `runId`, `finishReason`, `usage` (on finish), `message` (on error) | -| `TEXT_MESSAGE_START` / `TEXT_MESSAGE_CONTENT` / `TEXT_MESSAGE_END` | Assistant text streaming | `messageId`, `delta` (content) | -| `TOOL_CALL_START` / `TOOL_CALL_ARGS` / `TOOL_CALL_END` | Tool invocation streaming | `toolCallId`, `toolCallName`, `delta` (args), result on end | -| `STEP_STARTED` / `STEP_FINISHED` | Thinking / reasoning steps | `delta`, `signature` | -| `STATE_SNAPSHOT` / `STATE_DELTA` | Agent state sync | `snapshot`, `delta` | -| `CUSTOM` | Extensibility events (incl. structured-output — see below) | `name`, `value` | - -See the [AG-UI protocol docs](https://docs.ag-ui.com/introduction) for the full event catalogue and exact field shapes. +|--------------|---------|------------| +| `RUN_STARTED` / `RUN_FINISHED` / `RUN_ERROR` | Run lifecycle | `runId`, `finishReason`, `usage`, `message` | +| `TEXT_MESSAGE_START` / `CONTENT` / `END` | Assistant text | `messageId`, `delta` | +| `TOOL_CALL_START` / `ARGS` / `END` | Tool streaming | `toolCallId`, `toolCallName`, `delta` | +| `STEP_STARTED` / `STEP_FINISHED` | Reasoning steps | `delta`, `signature` | +| `STATE_SNAPSHOT` / `STATE_DELTA` | Agent state | `snapshot`, `delta` | +| `CUSTOM` | Extensibility (incl. structured-output) | `name`, `value` | -#### Transforming structured-output chunks +#### Structured-output chunks -There is **no separate `onStructuredOutputChunk` hook** — and you don't need one. When `chat()` is invoked with `outputSchema`, the structured-output chunks (the JSON `TEXT_MESSAGE_CONTENT` deltas, plus the `structured-output.start` / `structured-output.complete` CUSTOM events and any finalization `RUN_ERROR`) flow through the **same `onChunk` hook** as everything else. You transform, expand, or drop them exactly like any other chunk. +No separate chunk hook. With `outputSchema`, JSON deltas and `structured-output.start` / `structured-output.complete` CUSTOM events go through the same `onChunk`: -How you distinguish them depends on which finalization path the adapter takes: - -- **Separate-finalization adapters** (the legacy path — adapters that don't declare `supportsCombinedToolsAndSchema()`): `ctx.phase === 'structuredOutput'` during the finalization call. Discriminate on the phase. -- **Native-combined adapters** (modern OpenAI Chat Completions / Responses, Claude 4.5+, Gemini 3.x, Grok 4.x — see issue #605): the schema-constrained JSON is produced on the model's natural final turn, so **`ctx.phase` stays `'modelStream'`** — the `'structuredOutput'` phase never fires. Discriminate on the CUSTOM event name (`structured-output.start` / `structured-output.complete`) instead. +- **Separate finalization:** `ctx.phase === 'structuredOutput'` +- **Native-combined:** phase stays `'modelStream'` — key off CUSTOM event names ```typescript ignore import { type ChatMiddleware } from "@tanstack/ai"; @@ -283,10 +241,6 @@ import { type ChatMiddleware } from "@tanstack/ai"; const redactStructuredOutput: ChatMiddleware = { name: "redact-structured-output", onChunk: (ctx, chunk) => { - // Separate-finalization path: the JSON streams as TEXT_MESSAGE_CONTENT - // during the 'structuredOutput' phase. Transform the delta like any - // other text chunk — here, redact anything that looks like an SSN before - // it reaches the client. if ( ctx.phase === "structuredOutput" && chunk.type === "TEXT_MESSAGE_CONTENT" @@ -297,26 +251,16 @@ const redactStructuredOutput: ChatMiddleware = { }; } - // Both paths: the validated object arrives as a CUSTOM - // `structured-output.complete` event. On the native-combined path this is - // your only signal (ctx.phase never flips to 'structuredOutput'), so key - // off the event name, not the phase. `chunk.value` carries { object, raw }. if (chunk.type === "CUSTOM" && chunk.name === "structured-output.complete") { console.log("final structured output:", chunk.value); } - - // Return void to pass everything else through unchanged. }, }; ``` -> Why is there `onStructuredOutputConfig` but no `onStructuredOutputChunk`? Because the **config** shape genuinely differs at the structured-output boundary — it carries an `outputSchema` field that plain `ChatMiddlewareConfig` doesn't (see [onStructuredOutputConfig](#onstructuredoutputconfig)). **Chunks** are all just `StreamChunk` regardless of phase, so one `onChunk` plus `ctx.phase` (or the CUSTOM event name) covers every case — a parallel chunk hook would be redundant. - ### onShouldContinue -Called when the engine is deciding whether to start another agent-loop iteration (after a tool phase or between model turns). Combined with AND semantics across middleware **and** with `agentLoopStrategy` — any explicit `false` stops the loop. Return `true`, `void`, or `undefined` to allow continuation. - -Does **not** abort the run: the stream finishes normally with the current messages. Use `ctx.abort()` only for a hard abort. +Decides whether another agent-loop iteration starts. AND across middleware **and** `agentLoopStrategy` — any explicit `false` stops. Does **not** hard-abort (stream finishes normally). Use `ctx.abort()` for a hard stop. ```typescript import { type ChatMiddleware } from "@tanstack/ai"; @@ -324,17 +268,16 @@ import { type ChatMiddleware } from "@tanstack/ai"; const budget: ChatMiddleware = { name: "tool-budget", onShouldContinue: (_ctx, state) => { - // Stop further turns once 20 tool calls have been emitted if (state.toolCallCount >= 20) return false; }, }; ``` -For a full per-turn + cumulative tool budget recipe, see [Tool-call budgets](../chat/agentic-cycle#tool-call-budgets-middleware-recipe). +Full recipe: [Tool-call budgets](../chat/agentic-cycle#tool-call-budgets-middleware-recipe). ### onBeforeToolCall -Called before each tool executes. The first middleware that returns a non-void decision short-circuits — remaining middleware are skipped for that tool call. +First non-void decision wins; remaining middleware skip that call. ```typescript import { type ChatMiddleware } from "@tanstack/ai"; @@ -346,13 +289,15 @@ function isRecord(v: unknown): v is Record { const guard: ChatMiddleware = { name: "guard", onBeforeToolCall: (ctx, hookCtx) => { - // Block dangerous tools if (hookCtx.toolName === "deleteDatabase") { return { type: "abort", reason: "Dangerous operation blocked" }; } - // Validate and transform arguments - if (hookCtx.toolName === "search" && isRecord(hookCtx.args) && !hookCtx.args.limit) { + if ( + hookCtx.toolName === "search" && + isRecord(hookCtx.args) && + !hookCtx.args.limit + ) { return { type: "transformArgs", args: { ...hookCtx.args, limit: 10 }, @@ -362,28 +307,18 @@ const guard: ChatMiddleware = { }; ``` -**Decision types:** - | Decision | Effect | |----------|--------| -| `void` / `undefined` | Continue normally, next middleware can decide | -| `{ type: 'transformArgs', args }` | Replace tool arguments before execution | -| `{ type: 'skip', result }` | Skip execution entirely, use provided result | -| `{ type: 'abort', reason? }` | Abort the entire chat run | +| `void` / `undefined` | Continue; next middleware may decide | +| `{ type: 'transformArgs', args }` | Replace args | +| `{ type: 'skip', result }` | Skip execution; use result | +| `{ type: 'abort', reason? }` | Abort entire run | -The `hookCtx` provides: - -| Field | Type | Description | -|-------|------|-------------| -| `toolCall` | `ToolCall` | Raw tool call object | -| `tool` | `Tool \| undefined` | Resolved tool definition | -| `args` | `unknown` | Parsed arguments | -| `toolName` | `string` | Tool name | -| `toolCallId` | `string` | Tool call ID | +`hookCtx`: `toolCall`, `tool`, `args`, `toolName`, `toolCallId`. ### onAfterToolCall -Called after each tool execution (or skip). All middleware run — there is no short-circuiting. +Runs for every middleware (no short-circuit). ```typescript import { type ChatMiddleware } from "@tanstack/ai"; @@ -400,22 +335,11 @@ const toolLogger: ChatMiddleware = { }; ``` -The `info` object provides: - -| Field | Type | Description | -|-------|------|-------------| -| `toolCall` | `ToolCall` | Raw tool call object | -| `tool` | `Tool \| undefined` | Resolved tool definition | -| `toolName` | `string` | Tool name | -| `toolCallId` | `string` | Tool call ID | -| `ok` | `boolean` | Whether execution succeeded | -| `duration` | `number` | Execution time in milliseconds | -| `result` | `unknown` | Result (when `ok` is true) | -| `error` | `unknown` | Error (when `ok` is false) | +`info`: `toolCall`, `tool`, `toolName`, `toolCallId`, `ok`, `duration`, `result` / `error`. ### onUsage -Called once per model iteration when the `RUN_FINISHED` chunk includes usage data. Receives the usage object directly. +Once per model iteration when `RUN_FINISHED` includes usage. ```typescript import { type ChatMiddleware } from "@tanstack/ai"; @@ -423,41 +347,35 @@ import { type ChatMiddleware } from "@tanstack/ai"; const usageTracker: ChatMiddleware = { name: "usage-tracker", onUsage: (ctx, usage) => { - console.log( - `Iteration ${ctx.iteration}: ${usage.totalTokens} tokens` - ); + console.log(`Iteration ${ctx.iteration}: ${usage.totalTokens} tokens`); }, }; ``` -The `usage` object: +`usage`: `promptTokens`, `completionTokens`, `totalTokens`. -| Field | Type | Description | -|-------|------|-------------| -| `promptTokens` | `number` | Input tokens | -| `completionTokens` | `number` | Output tokens | -| `totalTokens` | `number` | Total tokens | - -### Terminal Hooks: onFinish, onAbort, onError - -Exactly **one** terminal hook fires per `chat()` invocation. They are mutually exclusive: - -| Hook | When it fires | -|------|--------------| -| `onFinish` | Run completed normally | -| `onAbort` | Run was aborted (via `ctx.abort()`, an external `AbortSignal`, or a `{ type: 'abort' }` decision from `onBeforeToolCall`) | -| `onError` | An unhandled error occurred | - -> **Structured-output lifecycle ordering:** When `chat()` is invoked with `outputSchema`, `onFinish` fires **after** the structured-output finalization call completes — not at the end of the agent loop. `onIteration` does **not** fire for the finalization step; it only fires for agent-loop iterations. -> -> **`onFinish` info fields and structured-output runs:** the `info` object reflects the **agent loop's** terminal state — finalization state is intentionally segregated to keep agent-loop semantics clean. -> -> - `info.content` — the agent loop's accumulated text. Finalization JSON deltas are **not** included here. The structured-output result is delivered via the `structured-output.complete` CUSTOM event, which middleware observes via `onChunk` (with `ctx.phase === 'structuredOutput'`). -> - `info.usage` — the agent loop's last `RUN_FINISHED.usage`. For a tools-less structured-output run (no agent-loop iteration produces `RUN_FINISHED`), this is `undefined`. To capture finalization tokens, use `onUsage` — that hook fires for **every** `RUN_FINISHED` carrying usage, including the finalization call. -> - `info.finishReason` — the agent loop's last `finishReason`. `null` when no agent-loop iteration produced `RUN_FINISHED` (e.g. a tools-less structured-output run). -> - `info.duration` — wall-clock duration of the entire `chat()` invocation, including finalization. -> -> To aggregate usage across the whole run, accumulate from `onUsage` callbacks rather than relying on `info.usage`. +### Terminal: onFinish / onAbort / onError + +Exactly **one** fires per run: + +| Hook | When | +|------|------| +| `onFinish` | Normal completion | +| `onAbort` | `ctx.abort()`, external `AbortSignal`, or `{ type: 'abort' }` from `onBeforeToolCall` | +| `onError` | Unhandled error | + +**Structured-output:** `onFinish` fires **after** finalization. `onIteration` does **not** fire for finalization. + +`FinishInfo` reflects the **agent loop**, not finalization: + +| Field | Notes | +|-------|-------| +| `content` | Agent-loop text only — not finalization JSON. Use `structured-output.complete` via `onChunk` | +| `usage` | Last agent-loop `RUN_FINISHED.usage`; may be `undefined`. Finalization tokens → `onUsage` | +| `finishReason` | Last agent-loop reason; `null` if no agent-loop `RUN_FINISHED` | +| `duration` | Wall-clock for entire `chat()`, including finalization | + +Aggregate full-run tokens from `onUsage`, not `info.usage` alone. ```typescript import { type ChatMiddleware } from "@tanstack/ai"; @@ -480,36 +398,25 @@ const terminal: ChatMiddleware = { }; ``` -The `info` object for `onFinish` (`FinishInfo`): +## Context object -| Field | Type | Description | -|-------|------|-------------| -| `finishReason` | `string \| null` | The agent loop's last `finishReason`. `null` when no agent-loop iteration produced `RUN_FINISHED` (e.g. a tools-less `chat({ outputSchema })` run). | -| `duration` | `number` | Total run duration in milliseconds, including any structured-output finalization. | -| `content` | `string` | The agent loop's accumulated text content. Does **not** include finalization JSON deltas — for that, observe the `structured-output.complete` CUSTOM event via `onChunk`. | -| `usage` | `{ promptTokens; completionTokens; totalTokens } \| undefined` | **Optional.** The agent loop's last `RUN_FINISHED.usage`. **Does not include finalization tokens** — use `onUsage` to observe those. Always guard with `if (info.usage)` or `info.usage?.`. | - -## Context Object - -Every hook receives a `ChatMiddlewareContext` as its first argument. It provides request-scoped information and control functions: +Every hook gets `ChatMiddlewareContext`: | Field | Type | Description | |-------|------|-------------| -| `requestId` | `string` | Unique ID for this chat request | -| `streamId` | `string` | Unique ID for this stream | -| `threadId` | `string` | AG-UI thread identifier. Resolves to caller-provided `threadId` (or legacy `conversationId`), or an auto-generated value if neither is supplied. Use this for event correlation. | -| `conversationId` | `string \| undefined` | **Deprecated** alias of `threadId`. Always equals `ctx.threadId`; retained so middleware written before the AG-UI rename keeps working. New middleware should read `ctx.threadId`. | -| `phase` | `ChatMiddlewarePhase` | Current lifecycle phase | -| `iteration` | `number` | Agent loop iteration (0-indexed) | -| `chunkIndex` | `number` | Running count of chunks yielded | -| `signal` | `AbortSignal \| undefined` | External abort signal | -| `abort(reason?)` | `function` | Abort the run from within middleware | -| `context` | `TContext` | User-provided runtime context value | -| `defer(promise)` | `function` | Register a non-blocking side-effect | - -## Typed Runtime Context - -`ChatMiddleware` accepts a context generic. This lets reusable middleware declared outside `chat()` access the same typed runtime context as your tools. +| `requestId` | `string` | Request ID | +| `streamId` | `string` | Stream ID | +| `threadId` | `string` | AG-UI thread (caller `threadId` / legacy `conversationId`, or generated) | +| `conversationId` | `string \| undefined` | **Deprecated** alias of `threadId` | +| `phase` | `ChatMiddlewarePhase` | Current phase | +| `iteration` | `number` | Agent loop (0-indexed) | +| `chunkIndex` | `number` | Chunks yielded | +| `signal` | `AbortSignal \| undefined` | External abort | +| `abort(reason?)` | `function` | Abort run | +| `context` | `TContext` | Runtime context | +| `defer(promise)` | `function` | Non-blocking side effect | + +### Typed runtime context ```typescript import { chat, type ChatMiddleware } from "@tanstack/ai"; @@ -530,7 +437,7 @@ export const auditMiddleware: ChatMiddleware = { ctx.context.audit.write({ userId: ctx.context.userId, requestId: ctx.requestId, - }) + }), ); }, }; @@ -546,13 +453,9 @@ chat({ }); ``` -When typed middleware or typed tools are present, `chat()` checks that the provided `context` matches the required shape. Existing middleware typed as plain `ChatMiddleware` still works; its `ctx.context` remains `unknown` and does not force a `context` option. - -Runtime context is process-local application state. It is separate from AG-UI `RunAgentInput.context`, which is protocol metadata parsed by `chatParamsFromRequest`. See [Runtime Context](./runtime-context) for server, client, and client-to-server handoff patterns. +Process-local only — not AG-UI protocol context. Patterns: [Runtime Context](./runtime-context). -### Aborting from Middleware - -Call `ctx.abort()` to gracefully stop the run. This triggers the `onAbort` terminal hook: +### Abort from middleware ```typescript import { type ChatMiddleware } from "@tanstack/ai"; @@ -567,9 +470,9 @@ const timeout: ChatMiddleware = { }; ``` -### Deferred Side Effects +### Deferred side effects -Use `ctx.defer()` to register promises that run after the terminal hook without blocking the stream: +`ctx.defer()` runs after the terminal hook without blocking the stream: ```typescript import { type ChatMiddleware } from "@tanstack/ai"; @@ -585,49 +488,31 @@ const analytics: ChatMiddleware = { duration: info.duration, tokens: info.usage?.totalTokens, }), - }) + }), ); }, }; ``` -## Composing Multiple Middleware - -Middleware execute in array order. The ordering matters for hooks that pipe or short-circuit: +## Composition -```typescript -import { chat, type ChatMiddleware } from "@tanstack/ai"; -import { openaiText } from "@tanstack/ai-openai"; -import { authMiddleware, loggingMiddleware, cachingMiddleware } from "./middleware"; +Execute in array order. -const stream = chat({ - adapter: openaiText("gpt-5.5"), - messages: [{ role: "user", content: "Hello" }], - middleware: [authMiddleware, loggingMiddleware, cachingMiddleware], -}); -``` - -### Composition Rules - -| Hook | Composition | Effect of Order | -|------|------------|----------------| -| `onConfig` | **Piped** — each receives previous output | Earlier middleware transforms first | -| `onStructuredOutputConfig` | **Piped** — each receives previous output | Earlier middleware transforms first | -| `onStart` | Sequential | All run in order | -| `onChunk` | **Piped** — chunks flow through each middleware | If first drops a chunk, later middleware never see it | -| `onBeforeToolCall` | **First-win** — first non-void decision wins | Earlier middleware has priority | -| `onShouldContinue` | **AND** — any explicit `false` stops the loop | Order only affects which middleware runs first when short-circuiting | -| `onAfterToolCall` | Sequential | All run in order | -| `onUsage` | Sequential | All run in order | -| `onFinish/onAbort/onError` | Sequential | All run in order | +| Hook | Composition | Order effect | +|------|-------------|--------------| +| `onConfig` | Piped | Earlier first | +| `onStructuredOutputConfig` | Piped | Earlier first | +| `onStart` | Sequential | All run | +| `onChunk` | Piped | Drop skips later | +| `onBeforeToolCall` | First-win | Earlier priority | +| `onShouldContinue` | AND | Any `false` stops | +| `onAfterToolCall` / `onUsage` / terminals | Sequential | All run | ## Capabilities -Middleware often need to **share state**. A provider middleware sets something up (a database handle, a per-request counter, a sandbox), and a consumer middleware reads it back later in the same run. Capabilities make that hand-off **type-safe and order-checked**: the consumer declares what it needs, the provider declares what it offers, and `chat()` refuses to run (at compile time _and_ at runtime) if a required capability was never provided. - -### Creating a capability +Share typed state across middleware in one run. Consumer declares needs; provider declares offers; `chat()` fails at compile time **and** runtime if a required capability is missing. -A capability is created with `createCapability()('name')` — a **curried** call: +### Create a capability ```typescript import { createCapability } from "@tanstack/ai"; @@ -636,48 +521,34 @@ const counterCapability = createCapability<{ value: number }>()("counter"); const [getCounter, provideCounter] = counterCapability; ``` -The currying is deliberate: you supply the **value type** explicitly (`<{ value: number }>`) while the **name literal** is inferred from the argument (`"counter"`). A single `createCapability('name')` call can't do both — supplying `T` explicitly stops TypeScript inferring the name, collapsing it to `string` and defeating the compile-time coverage check that keys on the literal name. - -The returned `counterCapability` is a hybrid value: - -- It **destructures to `[get, provide]`** — the two accessors you use inside hooks. -- It **is itself the identity** you list in `requires` / `provides`. There is no separate token to import. - -The accessors: +Curried: value type explicit, name inferred as literal. The handle **is** the identity for `requires` / `provides` and destructures to `[get, provide]`. | Accessor | Behavior | |----------|----------| -| `getCounter(ctx)` | Returns the value. **Throws** if the capability was never provided. | -| `getCounter(ctx, { optional: true })` | Returns `TValue \| undefined` — no throw when absent. | -| `provideCounter(ctx, value)` | Sets the value for this run. Call it from `setup`. | - -Equivalently, the context exposes `ctx.get(capability)`, `ctx.getOptional(capability)`, and `ctx.provide(capability, value)` — pass the capability handle directly. These are typed by the handle you pass (`ctx.get(counterCapability)` returns the value type), so `getCounter(ctx)` and `ctx.get(counterCapability)` are interchangeable — use whichever reads better in your hook. - -> **Capability names must be unique across your app.** The compile-time coverage check keys on the name literal (runtime keys on the handle reference), so two capabilities sharing a name will conflate in the type-level check. +| `getCounter(ctx)` | Value; **throws** if missing | +| `getCounter(ctx, { optional: true })` | `TValue \| undefined` | +| `provideCounter(ctx, value)` | Set for this run (call from `setup`) | -### The `setup` hook +Also: `ctx.get` / `ctx.getOptional` / `ctx.provide` with the handle. -Provisioning happens in a dedicated `setup(ctx)` hook. It **runs first** — before any `onConfig` (init), across all middleware in array order — so that by the time the rest of the lifecycle begins, every capability is in place. `setup` receives the stable `ChatMiddlewareContext` (not the mutable config), and may be async. +Capability names must be unique app-wide (type check keys on the name literal). -### `requires` / `provides` / `optionalRequires` +### setup / requires / provides -Three array fields on a middleware declare its capability contract. Each is a `ReadonlyArray` — you list the capability handles themselves: +`setup(ctx)` runs **first** (before any `onConfig` init), in array order. May be async. | Field | Meaning | |-------|---------| -| `provides` | Capabilities this middleware sets up. Each one **must** be `provide`d inside `setup`, or `chat()` throws after the setup phase. | -| `requires` | Capabilities this middleware reads. `chat()` validates (compile time + runtime) that some earlier middleware provides each one. | -| `optionalRequires` | Capabilities used **if present** but not required. Non-gating — never causes a validation error. Read with `getX(ctx, { optional: true })`. | +| `provides` | Must `provide` each in `setup` or `chat()` throws | +| `requires` | Must be provided earlier (compile + runtime) | +| `optionalRequires` | Non-gating; use optional get | ### Array example -Author middleware with `defineChatMiddleware` — it sharpens the `requires` / `provides` tuple types so the coverage check and builder can read them precisely. Here a **provider** sets up a counter in `setup`, and a **consumer** reads it in a hook: - ```typescript import { chat, createCapability, - createChatMiddleware, defineChatMiddleware, } from "@tanstack/ai"; import { openaiText } from "@tanstack/ai-openai"; @@ -685,7 +556,6 @@ import { openaiText } from "@tanstack/ai-openai"; const counterCapability = createCapability<{ value: number }>()("counter"); const [getCounter, provideCounter] = counterCapability; -// Provider: declares `provides` and provisions the value in `setup`. const withCounter = defineChatMiddleware({ name: "with-counter", provides: [counterCapability], @@ -694,7 +564,6 @@ const withCounter = defineChatMiddleware({ }, }); -// Consumer: declares `requires` and reads the value via `get` in a hook. const countsChunks = defineChatMiddleware({ name: "counts-chunks", requires: [counterCapability], @@ -709,16 +578,13 @@ const countsChunks = defineChatMiddleware({ const stream = chat({ adapter: openaiText("gpt-5.5"), messages: [{ role: "user", content: "Hello" }], - // Provider must come before the consumer. middleware: [withCounter, countsChunks], }); ``` -If you drop `withCounter` from the array, `chat()` reports a compile-time error at the `middleware` option naming the missing `"counter"` capability — and throws at runtime before the adapter is ever called. - -### Builder example +### Builder -`createChatMiddleware()` builds the array through chained `.use()` calls and enforces **provider-before-consumer ordering at compile time**: each `.use()` requires that the middleware's `requires` are already covered by capabilities provided by earlier `.use()` calls. +`createChatMiddleware()` enforces provider-before-consumer at each `.use()`: ```typescript import { chat, createChatMiddleware } from "@tanstack/ai"; @@ -726,8 +592,8 @@ import { openaiText } from "@tanstack/ai-openai"; import { withCounter, countsChunks } from "./counter-middleware"; const middleware = createChatMiddleware() - .use(withCounter) // provides "counter" - .use(countsChunks) // requires "counter" — OK, already provided above + .use(withCounter) + .use(countsChunks) .build(); const stream = chat({ @@ -737,35 +603,26 @@ const stream = chat({ }); ``` -Swap the two `.use()` calls (`.use(countsChunks).use(withCounter)`) and the builder rejects it at the `.use(countsChunks)` line — the consumer is ordered before its provider, so `"counter"` isn't in the provided set yet. +### Validation -### Validation guarantees +1. Compile-time coverage on array + builder +2. Runtime throw before adapter if required capability missing +3. Post-`setup` throw if `provides` never called `provide` +4. Duplicate provide → last-wins + dev warning -The capability system fails loudly and early: +## Built-in middleware -- **Compile-time coverage.** A required capability that nothing provides surfaces as a type error at the `middleware` option. This is enforced two ways: an **array coverage check** on `middleware: [...]`, and the order-aware **`createChatMiddleware()` builder** (which additionally enforces ordering). -- **Runtime coverage.** Even if types are bypassed, `chat()` validates coverage and **throws before the adapter runs** if a required capability is missing. -- **Post-`setup` assertion.** If a middleware declares a capability in `provides` but never calls its `provide` accessor during `setup`, `chat()` throws after the setup phase — you can't silently forget to provision. -- **Duplicate provide → last-wins + warning.** If two middleware provide the same capability, the last write wins and a development warning is emitted. -- **Unique names.** Capability `name`s must be unique across your app; the compile-time coverage check keys on the name literal (runtime keys on the handle reference). +| Middleware | Import | Role | +|------------|--------|------| +| `toolCacheMiddleware` | `@tanstack/ai/middlewares` | Cache tool results | +| `contentGuardMiddleware` | `@tanstack/ai/middlewares` | Redact/transform/block text | +| `otelMiddleware` | `@tanstack/ai/middlewares/otel` | OTel spans + metrics | -## Built-in Middleware - -TanStack AI ships ready-made middleware for common cases — caching tool results, redacting streamed text, and OpenTelemetry tracing: - -| Middleware | Import | What it does | -|------------|--------|--------------| -| `toolCacheMiddleware` | `@tanstack/ai/middlewares` | Cache tool-call results by name + arguments | -| `contentGuardMiddleware` | `@tanstack/ai/middlewares` | Redact / transform / block streamed text content | -| `otelMiddleware` | `@tanstack/ai/middlewares/otel` | Emit OpenTelemetry spans + GenAI metrics | - -See [Built-in Middleware](./built-in-middleware) for full options and examples for each. The recipes below show how to build your own. +Details: [Built-in Middleware](./built-in-middleware). ## Recipes -### Rate Limiting - -Limit the number of tool calls per request: +### Rate limit tool calls ```typescript import { type ChatMiddleware } from "@tanstack/ai"; @@ -774,7 +631,7 @@ function rateLimitMiddleware(maxCalls: number): ChatMiddleware { let toolCallCount = 0; return { name: "rate-limit", - onBeforeToolCall: (ctx, hookCtx) => { + onBeforeToolCall: () => { toolCallCount++; if (toolCallCount > maxCalls) { return { @@ -787,9 +644,7 @@ function rateLimitMiddleware(maxCalls: number): ChatMiddleware { } ``` -### Audit Trail - -Log every action for compliance: +### Audit trail ```typescript import { type ChatMiddleware } from "@tanstack/ai"; @@ -803,7 +658,7 @@ const auditTrail: ChatMiddleware = { requestId: ctx.requestId, event: "chat_started", timestamp: Date.now(), - }) + }), ); }, onAfterToolCall: (ctx, info) => { @@ -815,7 +670,7 @@ const auditTrail: ChatMiddleware = { success: info.ok, duration: info.duration, timestamp: Date.now(), - }) + }), ); }, onFinish: (ctx, info) => { @@ -826,15 +681,13 @@ const auditTrail: ChatMiddleware = { duration: info.duration, tokens: info.usage?.totalTokens, timestamp: Date.now(), - }) + }), ); }, }; ``` -### Per-Iteration Tool Swapping - -Expose different tools at different stages of the agent loop: +### Per-iteration tools ```typescript import { type ChatMiddleware } from "@tanstack/ai"; @@ -843,21 +696,16 @@ const toolSwapper: ChatMiddleware = { name: "tool-swapper", onConfig: (ctx, config) => { if (ctx.phase !== "beforeModel") return; - if (ctx.iteration === 0) { - // First iteration: only allow search return { tools: config.tools.filter((t) => t.name === "search"), }; } - // Later iterations: allow all tools }, }; ``` -### Content Filtering - -Drop or transform chunks before they reach the consumer: +### Drop filtered content ```typescript ignore import { type ChatMiddleware } from "@tanstack/ai"; @@ -868,7 +716,6 @@ const contentFilter: ChatMiddleware = { onChunk: (ctx, chunk) => { if (chunk.type === "TEXT_MESSAGE_CONTENT") { if (containsProfanity(chunk.delta)) { - // Drop the chunk entirely return null; } } @@ -876,7 +723,7 @@ const contentFilter: ChatMiddleware = { }; ``` -### Error Recovery with Retry Logging +### Error alerting ```typescript import { type ChatMiddleware } from "@tanstack/ai"; @@ -890,15 +737,13 @@ const errorRecovery: ChatMiddleware = { level: "error", message: `Chat ${ctx.requestId} failed after ${info.duration}ms`, error: String(info.error), - }) + }), ); }, }; ``` -## TypeScript Types - -The core middleware types are exported from `@tanstack/ai`: +## Types ```typescript import type { @@ -919,7 +764,7 @@ import type { } from "@tanstack/ai"; ``` -The option/type surfaces for the [built-in middleware](./built-in-middleware) are exported from the `@tanstack/ai/middlewares` subpath (not the main barrel): +Built-in option types from `@tanstack/ai/middlewares` (not main barrel): ```typescript import type { @@ -932,9 +777,10 @@ import type { } from "@tanstack/ai/middlewares"; ``` -## Next Steps +## Related -- [Built-in Middleware](./built-in-middleware) — `toolCacheMiddleware`, `contentGuardMiddleware`, `otelMiddleware` -- [OpenTelemetry](./otel) — emit traces and metrics via `otelMiddleware`- [Tools](../tools/tools) — Learn about the isomorphic tool system -- [Agentic Cycle](../chat/agentic-cycle) — Understand the multi-step agent loop -- [Streaming](../chat/streaming) — How streaming works in TanStack AI +- [Built-in Middleware](./built-in-middleware) +- [OpenTelemetry](./otel) +- [Tools](../tools/tools) +- [Agentic Cycle](../chat/agentic-cycle) +- [Streaming](../chat/streaming) diff --git a/docs/advanced/multimodal-content.md b/docs/advanced/multimodal-content.md index 0bea1a795..9c0019338 100644 --- a/docs/advanced/multimodal-content.md +++ b/docs/advanced/multimodal-content.md @@ -2,7 +2,7 @@ title: Multimodal Content id: multimodal-content order: 4 -description: "Send images, audio, video, and documents alongside text in TanStack AI messages with typed ContentPart primitives for multimodal models." +description: "Send images, audio, video, and documents with typed ContentPart in TanStack AI messages." keywords: - tanstack ai - multimodal @@ -15,56 +15,45 @@ keywords: - ImagePart --- -TanStack AI supports multimodal content in messages, allowing you to send images, audio, video, and documents alongside text to AI models that support these modalities. +If you need to send non-text media → set `content` to a `ContentPart[]` (string content still works). -When sending messages to AI models, you can include different types of content: -- **Text** - Plain text messages -- **Images** - JPEG, PNG, GIF, WebP images -- **Audio** - Audio files (model-dependent support) -- **Video** - Video files (model-dependent support) -- **Documents** - PDFs and other document types +Modalities: text, image, audio, video, document — model-dependent. -## Content Parts - -Multimodal messages use the `ContentPart` type to represent different content types: +## Content parts ```typescript -import type { ContentPart, ImagePart, TextPart } from '@tanstack/ai' +import type { ImagePart, TextPart } from '@tanstack/ai' -// Text content const textPart: TextPart = { type: 'text', - content: 'What do you see in this image?' + content: 'What do you see in this image?', } -// Image from base64 data (mimeType is required for data sources) +// Data source: mimeType required const imagePart: ImagePart = { type: 'image', source: { type: 'data', value: 'base64EncodedImageData...', - mimeType: 'image/jpeg' // Required for data sources + mimeType: 'image/jpeg', }, metadata: { - // Provider-specific metadata - detail: 'high' // OpenAI detail level - } + detail: 'high', // OpenAI detail level + }, } -// Image from URL (mimeType is optional for URL sources) +// URL source: mimeType optional const imageUrlPart: ImagePart = { type: 'image', source: { type: 'url', value: 'https://example.com/image.jpg', - mimeType: 'image/jpeg' // Optional hint for URL sources - } + mimeType: 'image/jpeg', + }, } ``` -## Using Multimodal Content in Messages - -Messages can have `content` as either a string or an array of `ContentPart`: +## In chat() ```typescript import { chat } from '@tanstack/ai' @@ -81,48 +70,43 @@ const response = await chat({ type: 'image', source: { type: 'url', - value: 'https://example.com/photo.jpg' - } - } - ] - } - ] + value: 'https://example.com/photo.jpg', + }, + }, + ], + }, + ], }) ``` -## Provider Support +## Provider notes ### OpenAI -OpenAI supports images and audio in their vision and audio models: - ```typescript import { openaiText } from '@tanstack/ai-openai' import { imageBase64 } from './data' const adapter = openaiText('gpt-5.5') -// Image with detail level metadata const message = { - role: 'user' , + role: 'user', content: [ - { type: 'text' , content: 'Describe this image' }, + { type: 'text', content: 'Describe this image' }, { - type: 'image' , - source: { type: 'data' , value: imageBase64, mimeType: 'image/jpeg' }, - metadata: { detail: 'high' } // 'auto' | 'low' | 'high' - } - ] + type: 'image', + source: { type: 'data', value: imageBase64, mimeType: 'image/jpeg' }, + metadata: { detail: 'high' }, // 'auto' | 'low' | 'high' + }, + ], } ``` -**Supported modalities by model:** -- `gpt-5.2`, `gpt-5-mini`: text, image -- `gpt-4o-audio`: text, audio +Examples: `gpt-5.2` / `gpt-5-mini` → text, image; `gpt-4o-audio` → text, audio. ### Anthropic -Anthropic's Claude models support images and PDF documents: +Images + PDF documents on Claude models (e.g. `claude-sonnet-4-6`). Authoritative list: `supports.input` in `@tanstack/ai-anthropic` `model-meta.ts`. ```typescript import { anthropicText } from '@tanstack/ai-anthropic' @@ -130,178 +114,124 @@ import { imageBase64, pdfBase64 } from './data' const adapter = anthropicText('claude-sonnet-4-6') -// Image with mimeType in source const imageMessage = { - role: 'user' , + role: 'user', content: [ - { type: 'text' , content: 'What do you see?' }, + { type: 'text', content: 'What do you see?' }, { - type: 'image' , - source: { type: 'data' , value: imageBase64, mimeType: 'image/jpeg' } - } - ] + type: 'image', + source: { type: 'data', value: imageBase64, mimeType: 'image/jpeg' }, + }, + ], } -// PDF document const docMessage = { role: 'user', content: [ { type: 'text', content: 'Summarize this document' }, { type: 'document', - source: { type: 'data', value: pdfBase64, mimeType: 'application/pdf' } - } - ] + source: { + type: 'data', + value: pdfBase64, + mimeType: 'application/pdf', + }, + }, + ], } ``` -**Supported modalities:** -- All Claude models (e.g. `claude-haiku-4-5`, `claude-sonnet-5`, `claude-opus-4-8`, `claude-fable-5`): text, image, and document (PDF) - -Check each model's `supports.input` in `@tanstack/ai-anthropic`'s `model-meta.ts` for the authoritative per-model list. - ### Gemini -Google's Gemini models support a wide range of modalities: - ```typescript import { geminiText } from '@tanstack/ai-gemini' import { imageBase64 } from './data' const adapter = geminiText('gemini-3-flash-preview') -// Image with mimeType in source const message = { role: 'user', content: [ { type: 'text', content: 'Analyze this image' }, { type: 'image', - source: { type: 'data', value: imageBase64, mimeType: 'image/png' } - } - ] + source: { type: 'data', value: imageBase64, mimeType: 'image/png' }, + }, + ], } ``` -**Supported modalities:** -- `gemini-2.5-flash`: text, image, audio, video +Example: `gemini-2.5-flash` → text, image, audio, video. ### Ollama -Ollama supports images in compatible models: +Host from `OLLAMA_HOST` (default `http://localhost:11434`). Multimodal support is model-specific. ```typescript import { ollamaText } from '@tanstack/ai-ollama' import { imageBase64 } from './data' -// `ollamaText(model)` takes a model name. The host is read from the -// `OLLAMA_HOST` environment variable (defaults to http://localhost:11434). const adapter = ollamaText('llama3.2-vision') -// Image as base64 const message = { role: 'user', content: [ { type: 'text', content: 'What is in this image?' }, { type: 'image', - source: { type: 'data', value: imageBase64, mimeType: 'image/jpeg' } - } - ] + source: { type: 'data', value: imageBase64, mimeType: 'image/jpeg' }, + }, + ], } ``` -**Note:** Ollama support varies by model. Check the specific model documentation for multimodal capabilities. - -## Source Types - -Content can be provided as either inline data or a URL: +## Source types -### Data (Base64) - -Use `type: 'data'` for inline base64-encoded content. **The `mimeType` field is required** to ensure providers receive proper content type information: +| Source | `mimeType` | When | +|--------|------------|------| +| `type: 'data'` | **Required** | Inline base64 | +| `type: 'url'` | Optional | Hosted content; not all providers/modalities support URLs | ```typescript -const imagePart = { +const imageData = { type: 'image', source: { type: 'data', - value: 'iVBORw0KGgoAAAANSUhEUgAAAAUA...', // Base64 string - mimeType: 'image/png' // Required for data sources - } -} - -const audioPart = { - type: 'audio', - source: { - type: 'data', - value: 'base64AudioData...', - mimeType: 'audio/mp3' // Required for data sources - } + value: 'iVBORw0KGgoAAAANSUhEUgAAAAUA...', + mimeType: 'image/png', + }, } -``` -### URL - -Use `type: 'url'` for content hosted at a URL. The `mimeType` field is **optional** as providers can often infer it from the URL or response headers: - -```typescript -const imagePart = { - type: 'image' , +const imageUrl = { + type: 'image', source: { - type: 'url' , + type: 'url', value: 'https://example.com/image.jpg', - mimeType: 'image/jpeg' // Optional hint - } -} -``` - -**Note:** Not all providers support URL-based content for all modalities. Check provider documentation for specifics. - -## Backward Compatibility - -String content continues to work as before: - -```typescript -// This still works -const message = { - role: 'user', - content: 'Hello, world!' -} - -// And this works for multimodal -const multimodalMessage = { - role: 'user', - content: [ - { type: 'text', content: 'Hello, world!' }, - { type: 'image', source: { type: 'url', value: '...' } } - ] + mimeType: 'image/jpeg', + }, } ``` -## Type Safety - -The multimodal types are fully typed. Provider-specific metadata types are available: +## Types ```typescript -import type { +import type { ContentPart, ImagePart, DocumentPart, AudioPart, VideoPart, - TextPart + TextPart, } from '@tanstack/ai' -// Provider-specific metadata types import type { OpenAIImageMetadata } from '@tanstack/ai-openai' import type { AnthropicImageMetadata } from '@tanstack/ai-anthropic' import type { GeminiImageMetadata } from '@tanstack/ai-gemini' ``` -### Validating Dynamic Messages +## Validate dynamic messages -When receiving messages from external sources (like `request.json()`), the data is typed as `any`. TanStack AI does not ship a runtime message validator — define a schema with your preferred Standard-Schema library (Zod, Valibot, ArkType, …) and parse the body before handing it to `chat()`. +No built-in runtime message validator. Parse with a Standard Schema library before `chat()`: ```typescript ignore import { chat } from '@tanstack/ai' @@ -312,21 +242,22 @@ const ContentPartSchema = z.discriminatedUnion('type', [ z.object({ type: z.literal('text'), content: z.string() }), z.object({ type: z.literal('image'), - source: z.object({ type: z.enum(['url', 'data']), value: z.string() }), + source: z.object({ + type: z.enum(['url', 'data']), + value: z.string(), + }), }), ]) const MessageSchema = z.object({ - // `ModelMessage.role` is 'user' | 'assistant' | 'tool' — there is no - // 'system' role. System instructions are passed separately via the - // `systemPrompts` option on `chat()`, not as messages. + // roles: 'user' | 'assistant' | 'tool' — no 'system' role. + // System instructions → systemPrompts on chat(). role: z.enum(['user', 'assistant', 'tool']), content: z.union([z.string(), z.array(ContentPartSchema)]), }) const BodySchema = z.object({ messages: z.array(MessageSchema) }) -// In an API route handler const { messages } = BodySchema.parse(await request.json()) const stream = chat({ @@ -335,25 +266,21 @@ const stream = chat({ }) ``` -The TypeScript types on `chat()` still constrain anything you append at the call site to the modalities supported by the selected model. +`chat()` still type-checks call-site content against the selected model's modalities. -## Best Practices +## Must / optional practices -1. **Use appropriate source type**: Use `data` for small content or when you need to include content inline. Use `url` for large files or when the content is already hosted. +**Must:** -2. **Include metadata**: Provide relevant metadata (like `mimeType` or `detail`) to help the model process the content correctly. +1. Confirm the model supports the modality +2. Require `mimeType` on data sources +3. Handle unsupported-modality errors -3. **Check model support**: Not all models support all modalities. Verify the model you're using supports the content types you want to send. +**Optional:** URL sources for large/hosted files; provider `metadata` (e.g. OpenAI `detail`). -4. **Handle errors gracefully**: When a model doesn't support a particular modality, it may throw an error. Handle these cases in your application. +## Client: ChatClient / useChat -## Client-Side Multimodal Messages - -When using the `ChatClient` from `@tanstack/ai-client`, you can send multimodal messages directly from your UI using the `sendMessage` method. - -### Basic Usage - -The `sendMessage` method accepts either a simple string or a `MultimodalContent` object: +`sendMessage` accepts a string or multimodal payload: ```typescript import { ChatClient, fetchServerSentEvents } from '@tanstack/ai-client' @@ -362,62 +289,45 @@ const client = new ChatClient({ connection: fetchServerSentEvents('/api/chat'), }) -// Simple text message await client.sendMessage('Hello!') -// Multimodal message with image await client.sendMessage({ content: [ { type: 'text', content: 'What is in this image?' }, { type: 'image', - source: { type: 'url', value: 'https://example.com/photo.jpg' } - } - ] + source: { type: 'url', value: 'https://example.com/photo.jpg' }, + }, + ], }) ``` -### Custom Message ID - -You can provide a custom ID for the message: +Custom id: ```typescript -import { ChatClient, fetchServerSentEvents } from '@tanstack/ai-client' - -const client = new ChatClient({ - connection: fetchServerSentEvents('/api/chat'), -}) - await client.sendMessage({ content: 'Hello!', - id: 'custom-message-id-123' + id: 'custom-message-id-123', }) ``` -### Per-Message Forwarded Props - -The second parameter allows you to pass additional `forwardedProps` for that specific request. These are shallow-merged with the client's base `forwardedProps` configuration, with per-message values taking priority: +Per-message `forwardedProps` (second arg) shallow-merge over client base; per-message wins. Legacy `body` constructor option is deprecated — use `forwardedProps`. ```typescript import { ChatClient, fetchServerSentEvents } from '@tanstack/ai-client' const client = new ChatClient({ connection: fetchServerSentEvents('/api/chat'), - forwardedProps: { model: 'gpt-5' }, // Base forwarded props + forwardedProps: { model: 'gpt-5' }, }) -// Override model for this specific message await client.sendMessage('Analyze this complex problem', { model: 'gpt-5', temperature: 0.2, }) ``` -> **Note:** The legacy `body` constructor option is still supported but deprecated. New code should use `forwardedProps`. Both populate the same wire field. - -### React Example - -Here's how to use multimodal messages in a React component: +### React: image URL ```tsx import { useChat } from '@tanstack/ai-react' @@ -426,19 +336,18 @@ import { useState } from 'react' function ChatWithImages() { const [imageUrl, setImageUrl] = useState('') - const { sendMessage, messages } = useChat({ + const { sendMessage } = useChat({ connection: fetchServerSentEvents('/api/chat'), }) const handleSendWithImage = () => { - if (imageUrl) { - sendMessage({ - content: [ - { type: 'text', content: 'What do you see in this image?' }, - { type: 'image', source: { type: 'url', value: imageUrl } } - ] - }) - } + if (!imageUrl) return + sendMessage({ + content: [ + { type: 'text', content: 'What do you see in this image?' }, + { type: 'image', source: { type: 'url', value: imageUrl } }, + ], + }) } return ( @@ -455,9 +364,7 @@ function ChatWithImages() { } ``` -### File Upload Example - -Here's how to handle file uploads and send them as multimodal content: +### React: file upload (base64) ```tsx import { useChat } from '@tanstack/ai-react' @@ -469,18 +376,25 @@ function ChatWithFileUpload() { }) const handleFileUpload = async (file: File) => { - // Convert file to base64 - const base64 = await new Promise((resolve) => { + const base64 = await new Promise((resolve, reject) => { const reader = new FileReader() reader.onload = () => { - const result = reader.result as string - // Remove data URL prefix (e.g., "data:image/png;base64,") - resolve(result.split(',')[1]!) + const result = reader.result + if (typeof result !== 'string') { + reject(new Error('Expected data URL string')) + return + } + const payload = result.split(',')[1] + if (payload === undefined) { + reject(new Error('Invalid data URL')) + return + } + resolve(payload) } + reader.onerror = () => reject(reader.error ?? new Error('read failed')) reader.readAsDataURL(file) }) - // Determine content type based on file type const type = file.type.startsWith('image/') ? 'image' : file.type.startsWith('audio/') @@ -494,9 +408,9 @@ function ChatWithFileUpload() { { type: 'text', content: `Please analyze this ${type}` }, { type, - source: { type: 'data', value: base64, mimeType: file.type } - } - ] + source: { type: 'data', value: base64, mimeType: file.type }, + }, + ], }) } @@ -506,10 +420,9 @@ function ChatWithFileUpload() { accept="image/*,audio/*,video/*,.pdf" onChange={(e) => { const file = e.target.files?.[0] - if (file) handleFileUpload(file) + if (file) void handleFileUpload(file) }} /> ) } ``` - diff --git a/docs/advanced/otel.md b/docs/advanced/otel.md index 3bdc1513b..c2ff883fb 100644 --- a/docs/advanced/otel.md +++ b/docs/advanced/otel.md @@ -2,7 +2,7 @@ title: OpenTelemetry id: otel order: 3 -description: "Emit vendor-neutral OpenTelemetry traces and metrics from every TanStack AI chat() call, following the OTel GenAI semantic conventions." +description: "Emit OpenTelemetry traces and GenAI metrics from chat() and media activities via otelMiddleware." keywords: - tanstack ai - opentelemetry @@ -14,17 +14,15 @@ keywords: - semantic conventions --- -The `otelMiddleware` factory wires TanStack AI into your existing OpenTelemetry setup. Every `chat()` call produces a root span, one child span per agent-loop iteration, and one grandchild span per tool call — all with [GenAI semantic-convention attributes](https://opentelemetry.io/docs/specs/semconv/gen-ai/). It also records GenAI token and duration histograms when a `Meter` is provided. +If you need OTel traces/metrics on `chat()` (and media activities) → install `@opentelemetry/api`, create a tracer/meter, pass `otelMiddleware`. -## Setup - -Install `@opentelemetry/api` — it's an optional peer dependency of `@tanstack/ai`: +Spans follow [GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). Import from `@tanstack/ai/middlewares/otel` so the main package never requires OTel. -```bash -pnpm add @opentelemetry/api -``` +## Setup -Wire up your OTel SDK however you already do (e.g. `@opentelemetry/sdk-node`). Then pass a `Tracer` (and optionally a `Meter`) into the middleware. The OTel middleware lives on its own subpath — importing it never affects users who don't need OTel: +1. `pnpm add @opentelemetry/api` +2. Wire your OTel SDK (e.g. `@opentelemetry/sdk-node`) +3. Pass tracer (+ optional meter) into middleware ```ts import { chat } from '@tanstack/ai' @@ -57,57 +55,41 @@ chat gpt-5.5 (root, kind: INTERNAL) └── chat gpt-5.5 #1 (iteration, kind: CLIENT) ``` -Iteration spans are numbered (`#0`, `#1`, ...) so distinct iterations of the same chat are easy to pick apart in trace viewers. +Iterations are numbered (`#0`, `#1`, …). -### Attribute reference +### Attributes (selected) | Level | Attribute | Value | | --- | --- | --- | -| root / iteration | `gen_ai.system` | `openai`, `anthropic`, ... | +| root / iteration | `gen_ai.system` | `openai`, `anthropic`, … | | iteration | `gen_ai.operation.name` | `chat` | | root / iteration | `gen_ai.request.model` | requested model | | iteration | `gen_ai.response.model` | actual model | -| iteration | `gen_ai.request.temperature` | from config | -| iteration | `gen_ai.request.top_p` | from config | -| iteration | `gen_ai.request.max_tokens` | from config | -| iteration | `gen_ai.usage.input_tokens` | per iteration | -| iteration | `gen_ai.usage.output_tokens` | per iteration | -| root / iteration | `gen_ai.usage.total_tokens` | provider-reported total | -| root / iteration | `gen_ai.usage.cost` | provider-reported cost, when available | -| root / iteration | `gen_ai.usage.cache_read.input_tokens` | cached prompt tokens, when reported | -| root / iteration | `gen_ai.usage.cache_creation.input_tokens` | cache-write prompt tokens, when reported | -| root / iteration | `gen_ai.usage.reasoning.output_tokens` | reasoning/thinking tokens, when reported | -| root / iteration | `tanstack.ai.usage.duration_seconds` | duration-based billing (e.g. transcription), when reported | -| root / iteration | `tanstack.ai.usage.upstream_cost` | gateway upstream cost (e.g. OpenRouter), when reported | -| root / iteration | `tanstack.ai.usage.upstream_input_cost` | upstream input cost split, when reported | -| root / iteration | `tanstack.ai.usage.upstream_output_cost` | upstream output cost split, when reported | -| iteration | `gen_ai.response.finish_reasons` | `[stop]`, `[tool_calls]`, ... | -| root | `gen_ai.usage.input_tokens` | rolled up | -| root | `gen_ai.usage.output_tokens` | rolled up | -| root | `tanstack.ai.iterations` | iteration count | -| tool | `gen_ai.tool.name` | tool name | -| tool | `gen_ai.tool.call.id` | tool call id | -| tool | `gen_ai.tool.type` | `function` | +| iteration | `gen_ai.request.temperature` / `top_p` / `max_tokens` | from config | +| iteration | `gen_ai.usage.input_tokens` / `output_tokens` | per iteration | +| root / iteration | `gen_ai.usage.total_tokens` / `cost` | when reported | +| root / iteration | `gen_ai.usage.cache_read.input_tokens` / `cache_creation.input_tokens` | when reported | +| root / iteration | `gen_ai.usage.reasoning.output_tokens` | when reported | +| root / iteration | `tanstack.ai.usage.duration_seconds` / `upstream_cost` / splits | when reported | +| iteration | `gen_ai.response.finish_reasons` | e.g. `[stop]`, `[tool_calls]` | +| root | `gen_ai.usage.*` rolled up; `tanstack.ai.iterations` | totals | +| tool | `gen_ai.tool.name` / `call.id` / `type` | tool identity | | tool | `tanstack.ai.tool.outcome` | `success` / `error` | -Usage attributes beyond input/output tokens are emitted only when the provider reports them, so spans stay clean otherwise. Cache and reasoning breakdowns use the official GenAI semconv names; `gen_ai.usage.cost` and `gen_ai.usage.total_tokens` are de-facto extensions consumed directly by backends like PostHog — without them, backends re-derive cost from their own price tables and lose cache discounts and gateway markup. Fields with no established convention (duration-based billing, the upstream cost split) are TanStack-namespaced. +Extra usage fields emit only when the provider reports them. Cache/reasoning use GenAI names; cost/total-tokens are de-facto extensions used by backends like PostHog. TanStack-namespaced fields cover duration billing and upstream cost splits. ### Metrics -Two GenAI-standard histograms: - -- `gen_ai.client.operation.duration` (seconds) — recorded **once per `chat()` call**, covering all agent-loop iterations and tool execution. On error or abort the record carries an `error.type` attribute (the thrown error's `name`, or `"cancelled"` for aborts). -- `gen_ai.client.token.usage` (tokens) — recorded **once per iteration** (two records: input and output), tagged with `gen_ai.token.type`. - -Both `gen_ai.response.id` and `gen_ai.response.model` are deliberately excluded from metric attributes to keep cardinality low (per-request custom-model names and request IDs would blow up the series set). +Requires a `Meter`: -## Privacy: capturing prompts and completions +- `gen_ai.client.operation.duration` (seconds) — once per `chat()` call (all iterations + tools). Errors/aborts set `error.type` (error `name`, or `"cancelled"`). +- `gen_ai.client.token.usage` — once per iteration, twice (input + output) with `gen_ai.token.type`. -By default, only metadata lands on spans. To record prompt and completion content, set `captureContent: true`. Content is captured as OTel span events following the GenAI convention: +`gen_ai.response.id` / `gen_ai.response.model` are **not** on metrics (cardinality). -- `gen_ai.user.message`, `gen_ai.system.message`, `gen_ai.assistant.message`, `gen_ai.tool.message`, `gen_ai.choice` +## Privacy: prompts and completions -Pass a `redact` function to strip PII before anything is recorded: +Default: metadata only. To record content: ```ts import { otelMiddleware } from '@tanstack/ai/middlewares/otel' @@ -122,21 +104,19 @@ otelMiddleware({ }) ``` -If `redact` throws, the middleware writes the literal sentinel `"[redaction_failed]"` into the span event and logs a warning — it never falls back to the raw content. This is the load-bearing invariant for users who ship traces to third-party backends: a broken redactor should shut off capture, not leak prompts. +Events: `gen_ai.user.message`, `gen_ai.system.message`, `gen_ai.assistant.message`, `gen_ai.tool.message`, `gen_ai.choice`. -Accumulated assistant text (the `gen_ai.choice` event) is capped at `maxContentLength` characters (default `100 000`); longer completions are truncated with a trailing `"…"` marker. +If `redact` throws → span event gets `"[redaction_failed]"` + warning log — **never** raw content. -Multimodal content (images, audio, video, documents) is represented as placeholder strings (`[image]`, `[audio]`, ...) to preserve message order without dumping binary data onto spans. Use `onSpanEnd` if you need richer multimodal capture. +Assistant text (`gen_ai.choice`) capped at `maxContentLength` (default 100_000); longer → truncate with `"…"`. Multimodal → placeholders (`[image]`, …). Richer capture: `onSpanEnd`. -Prompt/system/user message events fire from `onConfig` at the start of every iteration, which means the full conversation history (as the adapter will re-send it) is re-emitted on each iteration span. This mirrors what the provider actually sees on the wire. +Prompt events re-fire from `onConfig` each iteration (full history the adapter re-sends). ## Extension points -All four extensions are optional. Each wraps user code in try/catch — a thrown callback becomes a log line, never a broken chat. +All optional; throws become log lines, never break chat. -### `spanNameFormatter(info)` - -Override default span names. `info.kind` is `'chat' | 'iteration' | 'tool'`. +### spanNameFormatter ```ts import { otelMiddleware } from '@tanstack/ai/middlewares/otel' @@ -151,9 +131,9 @@ otelMiddleware({ }) ``` -### `attributeEnricher(info)` +`info.kind`: `'chat' | 'iteration' | 'tool'` (media: `'generation'`). -Add custom attributes to every span. Fires once per span. +### attributeEnricher ```ts import { otelMiddleware } from '@tanstack/ai/middlewares/otel' @@ -170,13 +150,11 @@ otelMiddleware({ }) ``` -### `onBeforeSpanStart(info, options)` - -Mutate `SpanOptions` immediately before `tracer.startSpan(...)`. Useful for adding links, custom start times, or extra default attributes. +### onBeforeSpanStart / onSpanEnd -### `onSpanEnd(info, span)` +`onBeforeSpanStart(info, options)` — mutate `SpanOptions` before `startSpan`. -Fires just before every `span.end()`. Common uses: record custom events, emit per-tool metrics via your own `Meter`. +`onSpanEnd(info, span)` — just before `span.end()`: ```ts import { otelMiddleware } from '@tanstack/ai/middlewares/otel' @@ -184,22 +162,21 @@ import { trace, metrics } from '@opentelemetry/api' const tracer = trace.getTracer('my-app') const meter = metrics.getMeter('my-app') - const toolDuration = meter.createHistogram('tool.duration') + otelMiddleware({ tracer, onSpanEnd: (info, span) => { if (info.kind === 'tool') { - // span is still recording; read timestamps from your own store if needed toolDuration.record(1, { 'tool.name': info.toolName }) } }, }) ``` -## Beyond chat: media activities +## Media activities -`otelMiddleware` is not chat-only. The media activities — `generateImage`, `generateVideo`, `generateAudio`, `generateSpeech`, and `generateTranscription` — accept the **same** `otelMiddleware` value on their `middleware` option. Each is a single request → response (or submit → poll for video), so the middleware emits one span per call instead of the chat span tree: +Same `otelMiddleware` instance works on `generateImage`, `generateVideo`, `generateAudio`, `generateSpeech`, `generateTranscription`, `summarize` — shared hooks use activity-agnostic `GenerationMiddlewareContext`. One span per call (not a chat tree): ```ts import { generateImage } from '@tanstack/ai' @@ -219,10 +196,6 @@ const result = await generateImage({ }) ``` -The same `otel` value can be passed to `chat()` and to any media activity — its shared lifecycle hooks (`onStart` / `onUsage` / `onFinish` / `onAbort` / `onError`) are authored against the activity-agnostic `GenerationMiddlewareContext`, so the one instance works everywhere. - -Each media call produces one `CLIENT` span tagged with the activity's `gen_ai.operation.name`: - | Activity | `gen_ai.operation.name` | | --- | --- | | `generateImage` | `image_generation` | @@ -232,11 +205,11 @@ Each media call produces one `CLIENT` span tagged with the activity's `gen_ai.op | `generateTranscription` | `transcription` | | `summarize` | `summarize` | -The span carries `gen_ai.system` and `gen_ai.request.model` at start and, on finish, the same `gen_ai.usage.*` / `tanstack.ai.usage.*` attributes documented above — including `tanstack.ai.usage.units_billed` for unit-billed media. When a `Meter` is supplied it records the `gen_ai.client.operation.duration` histogram, tagged per activity. For streaming video the span covers the full create → poll → complete lifecycle. Non-streaming video is two calls, so the submit itself emits no span — the run opens once the provider accepts the job, and the `getVideoJobStatus()` poll that observes a terminal state ends it. If a streaming video consumer abandons the stream before completion, the span is ended via `onAbort` (status `ERROR`, `tanstack.ai.completion.reason = cancelled`) rather than leaked. +Same usage attributes + `tanstack.ai.usage.units_billed` when applicable. Meter records operation duration per activity. Streaming video spans create→poll→complete; non-streaming video opens when the job is accepted and ends on terminal poll. Abandoned streams → `onAbort` (`cancelled`), not a leak. -`otelMiddleware` applies the same `spanNameFormatter`, `attributeEnricher`, `onBeforeSpanStart`, and `onSpanEnd` extension points to media spans — the span info is discriminated by `kind`, where media spans report `kind: 'generation'`. For a custom backend, implement the base `GenerationMiddleware` contract directly; its hooks (`onStart` / `onUsage` / `onFinish` / `onAbort` / `onError`) receive the `GenerationMiddlewareContext` and fire for every activity, chat included. The `GenerationMiddleware` types are exported from the package root, while the `otelMiddleware` value lives on the `@tanstack/ai/middlewares/otel` subpath so importing `@tanstack/ai` never requires the optional `@opentelemetry/api` peer. +Custom backends: implement `GenerationMiddleware` (exported from package root). `otelMiddleware` stays on the otel subpath. ## Related -- [Middleware](./middleware) — the lifecycle this middleware hooks into -- [Debug Logging](./debug-logging) — quick console-output diagnostics, complementary to OTel \ No newline at end of file +- [Middleware](./middleware) — lifecycle hooks +- [Debug Logging](./debug-logging) — console diagnostics diff --git a/docs/advanced/per-model-type-safety.md b/docs/advanced/per-model-type-safety.md index 6e9d92c41..43777cce1 100644 --- a/docs/advanced/per-model-type-safety.md +++ b/docs/advanced/per-model-type-safety.md @@ -2,7 +2,7 @@ title: Per-Model Type Safety id: per-model-type-safety order: 5 -description: "TanStack AI narrows modelOptions and content types to the specific model you select, enforcing capabilities at compile time." +description: "modelOptions and content types narrow to the selected model at compile time." keywords: - tanstack ai - type safety @@ -13,32 +13,27 @@ keywords: - compile-time --- -The AI SDK provides **model-specific type safety** for `modelOptions`. Each model's capabilities determine which model options are allowed, and TypeScript will enforce this at compile time. +If you pick a model → only that model's options and modalities type-check. Prefer first-class `chat({ outputSchema })` for structured output; use raw provider `text` only when you need provider-specific control. -> **Tip:** For structured outputs, most users should prefer the first-class `chat({ outputSchema })` option over the raw provider `text` option shown below — it works across providers and validates the result for you. The raw `text` option is for when you need provider-specific control. +## How it works -## How It Works +1. Adapter factory captures the model literal: `openaiText(model)`. +2. `modelOptions` resolves via `ResolveProviderOptions` — only options that model supports. +3. Excess properties (e.g. `text` on a model without structured output) fail at compile time. -Each adapter factory captures the model literal as a type parameter — `openaiText(model)` — so the adapter carries the exact model you selected at the type level. +Same mechanism as [Typed Pre-Configured Options](./typed-options) and [Extend Adapter](./extend-adapter). -The `modelOptions` you pass are then resolved against a per-model map (`ResolveProviderOptions`). Each model's entry declares only the options that model actually supports. A model without a structured-output capability simply has no `text` property in its resolved options type, so TypeScript's excess-property checking rejects `text` for that model — at compile time, with zero runtime cost. - -This is the same mechanism described in [Typed Pre-Configured Options](./typed-options) (which captures these resolved options in a reusable object) and [Extend Adapter](./extend-adapter) (which lets you attach the same typed `modelOptions` to custom models). - -## Usage Examples - -### ✅ Correct Usage +## Valid ```typescript import { chat } from "@tanstack/ai"; import { openaiText } from "@tanstack/ai-openai"; -// ✅ gpt-5 supports structured outputs - `text` is allowed +// gpt-5 supports structured outputs — `text` allowed const validCall = chat({ adapter: openaiText("gpt-5"), messages: [], modelOptions: { - // OK - text is included for gpt-5 text: { format: { type: "json_schema", @@ -52,31 +47,24 @@ const validCall = chat({ }); ``` -### ❌ Incorrect Usage +## Invalid ```typescript ignore import { chat } from "@tanstack/ai"; import { openaiText } from "@tanstack/ai-openai"; -// ❌ gpt-4-turbo does NOT support structured outputs - `text` is rejected +// gpt-4-turbo does not support structured outputs const invalidCall = chat({ adapter: openaiText("gpt-4-turbo"), messages: [], modelOptions: { - text: {}, // ❌ TypeScript error: 'text' does not exist in type + text: {}, // TS2353: 'text' does not exist }, }); ``` -TypeScript will produce: - -``` -error TS2353: Object literal may only specify known properties, and 'text' does not exist in type ...'. -``` - -## Benefits +## What you get -- **Compile-time safety**: Catch incorrect model options before deployment -- **Better IDE experience**: Autocomplete shows only valid options for each model -- **Self-documenting**: Model capabilities are explicit in the type system -- **Zero runtime overhead**: All type checking happens at compile time +1. Compile-time rejection of invalid options +2. Autocomplete limited to the selected model +3. Zero runtime overhead diff --git a/docs/advanced/runtime-adapter-switching.md b/docs/advanced/runtime-adapter-switching.md index e5c6189c9..76c9f839d 100644 --- a/docs/advanced/runtime-adapter-switching.md +++ b/docs/advanced/runtime-adapter-switching.md @@ -2,7 +2,7 @@ title: Runtime Adapter Switching id: runtime-adapter-switching order: 6 -description: "Let users switch between LLM providers at runtime in TanStack AI while keeping full TypeScript type safety for each adapter's model options." +description: "Switch LLM providers at runtime with a factory map while keeping per-adapter model types." keywords: - tanstack ai - runtime switching @@ -12,13 +12,9 @@ keywords: - dynamic adapter --- -# Runtime Adapter Switching with Type Safety +If users pick a provider at runtime → map provider keys to adapter factories that already bake in the model. -Learn how to build interfaces where users can switch between LLM providers at runtime while maintaining full TypeScript type safety. - -## The Simple Approach - -With TanStack AI, the model is passed directly to the adapter factory function. This gives you full type safety and autocomplete at the point of definition: +## Pattern ```typescript import { chat, toServerSentEventsResponse } from '@tanstack/ai' @@ -27,14 +23,12 @@ import { openaiText } from '@tanstack/ai-openai' type Provider = 'openai' | 'anthropic' -// Define adapters with their models - autocomplete works here! const adapters = { - anthropic: () => anthropicText('claude-sonnet-4-6'), // ✅ Autocomplete! - openai: () => openaiText('gpt-5.5'), // ✅ Autocomplete! + anthropic: () => anthropicText('claude-sonnet-4-6'), + openai: () => openaiText('gpt-5.5'), } async function handleRequest(request: Request) { - // In your request handler: const body = await request.json() const provider: Provider = body.forwardedProps?.provider || 'openai' @@ -42,37 +36,31 @@ async function handleRequest(request: Request) { adapter: adapters[provider](), messages: body.messages, }) + + return toServerSentEventsResponse(stream) } ``` -## Why This Works - -Each adapter factory function accepts a model name as its first argument and returns a fully typed adapter: +Model is the first factory arg; `chat()` uses `adapter.model`. Autocomplete and invalid-name errors happen at the factory call site. ```typescript import { openaiText, OpenAITextAdapter } from '@tanstack/ai-openai' -// These are equivalent: +// Equivalent: const adapter1 = openaiText('gpt-5.5') -const adapter2 = new OpenAITextAdapter({ apiKey: process.env.OPENAI_API_KEY! }, 'gpt-5.5') +const adapter2 = new OpenAITextAdapter( + { apiKey: process.env.OPENAI_API_KEY! }, + 'gpt-5.5', +) -// The model is stored on the adapter console.log(adapter1.model) // 'gpt-5.5' ``` -When you pass an adapter to `chat()`, it uses the model from `adapter.model`. This means: - -- **Full autocomplete** - When typing the model name, TypeScript knows valid options -- **Type validation** - Invalid model names cause compile errors -- **Clean code** - No separate `model` parameter needed - -## Full Example - -Here's a complete example showing a multi-provider chat API: +## Full multi-provider route ```typescript ignore import { createFileRoute } from '@tanstack/react-router' -import { chat, maxIterations, toServerSentEventsResponse } from '@tanstack/ai' +import { chat, toServerSentEventsResponse } from '@tanstack/ai' import { openaiText } from '@tanstack/ai-openai' import { anthropicText } from '@tanstack/ai-anthropic' import { geminiText } from '@tanstack/ai-gemini' @@ -80,7 +68,6 @@ import { ollamaText } from '@tanstack/ai-ollama' type Provider = 'openai' | 'anthropic' | 'gemini' | 'ollama' -// Define adapters with their models const adapters = { anthropic: () => anthropicText('claude-sonnet-4-6'), gemini: () => geminiText('gemini-3-flash-preview'), @@ -94,9 +81,7 @@ export const Route = createFileRoute('/api/chat')({ POST: async ({ request }) => { const abortController = new AbortController() const body = await request.json() - // `forwardedProps` is the AG-UI field set by `useChat({ forwardedProps })`. - // The legacy `body.data.provider` access still works (mirrored on the - // wire for backward compatibility) but `forwardedProps` is preferred. + // Prefer forwardedProps (AG-UI). Legacy body.data.provider still mirrored. const provider: Provider = body.forwardedProps?.provider || 'openai' const stream = chat({ @@ -114,9 +99,7 @@ export const Route = createFileRoute('/api/chat')({ }) ``` -## Using with Image Adapters - -The same pattern works for image generation: +## Image adapters ```typescript import { generateImage } from '@tanstack/ai' @@ -125,7 +108,7 @@ import { geminiImage } from '@tanstack/ai-gemini' type ImageProvider = 'openai' | 'gemini' -const imageAdapters: Record ReturnType> = { +const imageAdapters = { openai: () => openaiImage('gpt-image-2'), gemini: () => geminiImage('gemini-3.1-flash-image-preview'), } @@ -144,9 +127,7 @@ export async function POST(request: Request) { } ``` -## Using with Summarize Adapters - -And for summarization: +## Summarize adapters ```typescript import { summarize } from '@tanstack/ai' @@ -155,7 +136,7 @@ import { anthropicSummarize } from '@tanstack/ai-anthropic' type SummarizeProvider = 'openai' | 'anthropic' -const summarizeAdapters: Record ReturnType> = { +const summarizeAdapters = { openai: () => openaiSummarize('gpt-5.4-mini'), anthropic: () => anthropicSummarize('claude-sonnet-4-6'), } @@ -176,36 +157,11 @@ export async function POST(request: Request) { } ``` -## Migration from Switch Statements - -If you have existing code using switch statements, here's how to migrate: - -### Before - -```typescript ignore -let adapter -let model - -switch (provider) { - case 'anthropic': - adapter = anthropicText() - model = 'claude-sonnet-4-6' - break - case 'openai': - default: - adapter = openaiText() - model = 'gpt-5.5' - break -} +## Migrate from switch + casts -const stream = chat({ - adapter: adapter as any, - model: model as any, - messages, -}) -``` +**Before** (avoid): switch sets adapter/model separately, often with type casts. -### After +**After:** ```typescript import { chat, toServerSentEventsResponse } from '@tanstack/ai' @@ -232,8 +188,6 @@ export async function POST(request: Request) { } ``` -The key changes: - -1. Replace the switch statement with an object of factory functions -2. Each factory function creates an adapter with the model included -3. No more `as any` casts - full type safety! +1. Replace switch with factory object +2. Include the model in each factory +3. Drop type casts diff --git a/docs/advanced/runtime-context.md b/docs/advanced/runtime-context.md index cb9df6b84..6a3e8b325 100644 --- a/docs/advanced/runtime-context.md +++ b/docs/advanced/runtime-context.md @@ -2,7 +2,7 @@ title: Runtime Context id: runtime-context order: 2 -description: "Pass typed runtime dependencies to TanStack AI tools and middleware without serializing them to the model or AG-UI protocol context." +description: "Pass typed runtime deps to tools and middleware without sending them to the model or AG-UI context." keywords: - tanstack ai - runtime context @@ -12,24 +12,25 @@ keywords: - ag-ui context --- -Runtime context is application state you pass to tool implementations and middleware. Use it for request-scoped or client-local dependencies such as authenticated users, database clients, tenancy, feature flags, audit loggers, or browser services. +If tools/middleware need request-local deps (user, db, toast) → pass `context` on `chat()` / `useChat`. It is **not** prompt context and **not** AG-UI `RunAgentInput.context`. Never sent to the model automatically. -Runtime context is not prompt context and is not the AG-UI `RunAgentInput.context` field. It is never sent to the model automatically. +## Type safety -## How Type Safety Works +Consumers declare needs; call site must satisfy the merge: -Runtime context is checked from the point of view of the code that consumes it. Tools and middleware declare the context shape they need, and `chat()`, `ChatClient`, and framework hooks check that the `context` value you pass satisfies those requirements. +1. `toolDefinition(...).server(...)` / `.client(...)` +2. `ChatMiddleware` +3. `chat()` / hooks check `context` against every typed consumer -The source of truth is: - -- `toolDefinition(...).server(...)` for server tools. -- `toolDefinition(...).client(...)` for client tools. -- `ChatMiddleware` for middleware. - -This means the context value is the implementation detail you provide at runtime, while tools and middleware are the contract. TanStack AI infers the required context from every typed tool and middleware in the call, merges those requirements, and checks your `context` option against the result. +Untyped consumers get `unknown` and do not force `context`. Optional: declare `TContext | undefined` so `context` may be omitted. ```typescript -import { chat, toServerSentEventsResponse, toolDefinition, type ChatMiddleware } from "@tanstack/ai"; +import { + chat, + toServerSentEventsResponse, + toolDefinition, + type ChatMiddleware, +} from "@tanstack/ai"; import { openaiText } from "@tanstack/ai-openai"; type UserContext = { @@ -70,13 +71,7 @@ export async function POST(request: Request) { } ``` -In this example, the tool requires `UserContext` and the middleware requires `TenantContext`, so the `context` value must satisfy both. If you remove `tenantId`, TypeScript reports an error because `tenantMiddleware` declared that it needs it. - -This is intentional. The `context` object alone should not decide what tools and middleware are allowed to read. The consumers define their requirements, and the call site proves that it supplied them. Untyped tools and middleware still work; they receive `unknown` context and do not force a `context` option. - -This inference also works when reusable tools or middleware are declared outside the `chat()` call and passed in as arrays. A consumer can opt into optional runtime context by declaring `TContext | undefined`; then the `context` option can be omitted when all typed consumers accept `undefined`. If a context value is provided, it still has to satisfy every typed consumer. - -The same rule applies on the client: +Client tools force client `context` the same way: ```typescript import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; @@ -106,11 +101,7 @@ useChat({ }); ``` -Because the client tool declares `ClientRuntimeContext & { mode: "debug" }`, `useChat()` requires a `context` value with both `currentTabId` and the literal `mode: "debug"`. - -## Server Runtime Context - -Define the context type once, use it in server tools and middleware, then pass the matching `context` value to `chat()`. +## Server runtime context ```typescript import { @@ -128,7 +119,10 @@ type AppContext = { tenantId: string; db: { notes: { - findMany(args: { userId: string; tenantId: string }): Promise>; + findMany(args: { + userId: string; + tenantId: string; + }): Promise>; }; }; }; @@ -176,11 +170,9 @@ export async function POST(request: Request) { } ``` -When any tool or middleware in a `chat()` call declares a concrete context type, TypeScript checks the `context` value against that type. Existing untyped tools and middleware continue to work; their `ctx.context` type remains `unknown`. - -## Client Runtime Context +## Client runtime context -Client runtime context is local to `ChatClient` and framework hooks. It is passed to client tool implementations and is not serialized to the server. +Local to `ChatClient` / hooks — not serialized to the server. ```typescript import { createChatClientOptions } from "@tanstack/ai-client"; @@ -212,11 +204,11 @@ const chatOptions = createChatClientOptions({ const chat = useChat(chatOptions); ``` -Use client context for local dependencies only. Do not put values there expecting the server to receive them. +## Client → server handoff -## Client-to-Server Handoff +Serializable data → `forwardedProps`. Validate on the server; map into server `context` yourself. -To send serializable client data to the server, use `forwardedProps`, validate it in your route, and explicitly map it into the server runtime context. +**Client:** ```typescript import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; @@ -235,7 +227,6 @@ const notifyUser = toolDefinition({ return { ok: true }; }); -// Client useChat({ connection: fetchServerSentEvents("/api/chat"), tools: [notifyUser], @@ -249,8 +240,9 @@ useChat({ }); ``` +**Server:** + ```typescript -// Server import { chat, chatParamsFromRequest, @@ -288,16 +280,18 @@ export async function POST(request: Request) { } ``` -Treat `forwardedProps` as client-controlled input. Validate and allowlist every field before using it to build server runtime context. - -## AG-UI Context +Treat `forwardedProps` as client-controlled: validate and allowlist before use. -AG-UI also defines `RunAgentInput.context`, usually as protocol-level context entries for interoperable agents. TanStack AI surfaces that field through `chatParamsFromRequest`, but it is separate from `chat({ context })`. +## AG-UI context -TanStack AI does not automatically copy AG-UI `params.aguiContext` into runtime context. If you want to use AG-UI context values, validate and map them yourself. `params.context` is a deprecated alias of `params.aguiContext` kept for backward compatibility. +`RunAgentInput.context` is protocol metadata via `chatParamsFromRequest` (`params.aguiContext`; `params.context` is a deprecated alias). TanStack AI does **not** auto-copy it into runtime `context` — map it yourself: ```typescript -import { chat, chatParamsFromRequest, toServerSentEventsResponse } from "@tanstack/ai"; +import { + chat, + chatParamsFromRequest, + toServerSentEventsResponse, +} from "@tanstack/ai"; import { openaiText } from "@tanstack/ai-openai"; import { buildRuntimeContextFrom } from "./context"; import { serverTools } from "./tools"; diff --git a/docs/advanced/tree-shaking.md b/docs/advanced/tree-shaking.md index 950380d71..6cc9c2bb9 100644 --- a/docs/advanced/tree-shaking.md +++ b/docs/advanced/tree-shaking.md @@ -2,7 +2,7 @@ title: Tree-Shaking id: tree-shaking order: 7 -description: "TanStack AI's tree-shakeable architecture — import only the activities and adapters you use for minimal bundle size across chat, image, and speech." +description: "Import only the activities and adapters you use for minimal TanStack AI bundle size." keywords: - tanstack ai - tree-shaking @@ -12,41 +12,22 @@ keywords: - tree-shakeable --- -# Tree-Shaking & Bundle Optimization +If you care about bundle size → import named activity functions and per-activity adapters only. Never `import *`. -TanStack AI is designed from the ground up for maximum tree-shakeability. The entire system—from activity functions to adapters—uses a functional, modular architecture that ensures you only bundle the code you actually use. - -## Design Philosophy - -Instead of a monolithic API that includes everything, TanStack AI provides: - -- **Individual activity functions** - Import only the activities you need (`chat`, `summarize`, etc.) -- **Individual adapter functions** - Import only the adapters you need (`openaiText`, `openaiSummarize`, etc.) -- **Functional API design** - Pure functions that can be easily eliminated by bundlers -- **Separate modules** - Each activity and adapter lives in its own module - -This design means that if you only use `chat` with OpenAI, you won't bundle code for summarization, image generation, or other providers. - -## Activity Functions - -Each AI activity is exported as a separate function from `@tanstack/ai`: +## Import what you use ```ts -// Import only the activities you need -import { chat } from '@tanstack/ai' // Chat/text generation -import { summarize } from '@tanstack/ai' // Summarization -import { generateImage } from '@tanstack/ai' // Image generation -import { generateSpeech } from '@tanstack/ai' // Text-to-speech -import { generateTranscription } from '@tanstack/ai' // Audio transcription -import { generateVideo } from '@tanstack/ai' // Video generation +import { chat } from '@tanstack/ai' +import { summarize } from '@tanstack/ai' +import { generateImage } from '@tanstack/ai' +import { generateSpeech } from '@tanstack/ai' +import { generateTranscription } from '@tanstack/ai' +import { generateVideo } from '@tanstack/ai' ``` -### Example: Chat Only - -If you only need chat functionality: +Chat-only app: ```ts -// Only chat code is bundled import { chat } from '@tanstack/ai' import { openaiText } from '@tanstack/ai-openai' @@ -56,100 +37,52 @@ const stream = chat({ }) ``` -Your bundle will **not** include: -- Summarization logic -- Image generation logic -- Other activity implementations - -## Adapter Functions +Bundles `chat` + `openaiText` — not summarize, image, or other providers. -Each provider package exports individual adapter functions for each activity type: +## Adapter imports by provider -### OpenAI +**OpenAI** ```ts import { - openaiText, // Chat/text generation - openaiSummarize, // Summarization - openaiImage, // Image generation - openaiSpeech, // Text-to-speech - openaiTranscription, // Audio transcription - openaiVideo, // Video generation + openaiText, + openaiSummarize, + openaiImage, + openaiSpeech, + openaiTranscription, + openaiVideo, } from '@tanstack/ai-openai' ``` -### Anthropic +**Anthropic** ```ts -import { - anthropicText, // Chat/text generation - anthropicSummarize, // Summarization -} from '@tanstack/ai-anthropic' +import { anthropicText, anthropicSummarize } from '@tanstack/ai-anthropic' ``` -### Gemini +**Gemini** ```ts import { - geminiText, // Chat/text generation - geminiSummarize, // Summarization - geminiImage, // Image generation - geminiSpeech, // Text-to-speech (experimental) + geminiText, + geminiSummarize, + geminiImage, + geminiSpeech, // experimental } from '@tanstack/ai-gemini' ``` -### Ollama +**Ollama** ```ts -import { - ollamaText, // Chat/text generation - ollamaSummarize, // Summarization -} from '@tanstack/ai-ollama' -``` - -## Complete Example - -Here's how the tree-shakeable design works in practice: - -```ts -// Only import what you need -import { chat } from '@tanstack/ai' -import { openaiText } from '@tanstack/ai-openai' - -// Chat generation - returns AsyncIterable -const chatResult = chat({ - adapter: openaiText('gpt-5.5'), - messages: [{ role: 'user', content: 'Hello!' }], -}) - -for await (const chunk of chatResult) { - console.log(chunk) -} +import { ollamaText, ollamaSummarize } from '@tanstack/ai-ollama' ``` -**What gets bundled:** -- ✅ `chat` function and its dependencies -- ✅ `openaiText` adapter and its dependencies -- ✅ Chat-specific streaming and tool handling logic - -**What doesn't get bundled:** -- ❌ `summarize` function -- ❌ `generateImage` function -- ❌ Other adapter implementations (Anthropic, Gemini, etc.) -- ❌ Other activity implementations - -## Using Multiple Activities - -If you need multiple activities, import only what you use: +## Multiple activities ```ts import { chat, summarize } from '@tanstack/ai' -import { - openaiText, - openaiSummarize -} from '@tanstack/ai-openai' +import { openaiText, openaiSummarize } from '@tanstack/ai-openai' -// Each activity is independent const chatResult = chat({ adapter: openaiText('gpt-5.5'), messages: [{ role: 'user', content: 'Hello!' }], @@ -161,140 +94,72 @@ const summarizeResult = await summarize({ }) ``` -Each activity is in its own module, so bundlers can eliminate unused ones. +## Typed options helpers -## Type Safety - -The tree-shakeable design doesn't sacrifice type safety. Each adapter provides full type safety for its supported models: - -```ts ignore -import { openaiText, type OpenAIChatModel } from '@tanstack/ai-openai' - -const adapter = openaiText('gpt-5.5') - -// TypeScript knows the exact models supported -const model: OpenAIChatModel = 'gpt-5.5' // ✓ Valid -const model2: OpenAIChatModel = 'invalid' // ✗ Type error -``` - -## Create Options Functions - -The `create___Options` functions are also tree-shakeable: +Also tree-shakeable: ```ts -import { - createChatOptions, - createImageOptions -} from '@tanstack/ai' +import { createChatOptions } from '@tanstack/ai' import { openaiText } from '@tanstack/ai-openai' -// Only import what you need const chatOptions = createChatOptions({ adapter: openaiText('gpt-5.5'), }) ``` -## Bundle Size Benefits - -The functional, modular design provides significant bundle size benefits: - -### Importing Everything (Less Efficient) - -```ts -// ❌ Importing more than needed -import * as ai from '@tanstack/ai' -import * as openai from '@tanstack/ai-openai' - -// This bundles all exports from both packages -``` - -### Importing Only What You Need (Recommended) +## Do / don't ```ts -// ✅ Only what you use gets bundled +// Do import { chat } from '@tanstack/ai' import { openaiText } from '@tanstack/ai-openai' -// You only get: -// - Chat activity implementation -// - OpenAI text adapter -// - Chat-specific dependencies +// Don't +import * as ai from '@tanstack/ai' +import * as openai from '@tanstack/ai-openai' ``` -### Real-World Impact - -For a typical chat application, importing a single activity and one adapter pulls in substantially less code than bundling every activity and every provider adapter. Because each activity and adapter lives in its own side-effect-free module, your bundler drops everything you don't reference — so the more providers and activities the library supports, the larger the difference between a focused import and a namespace import. +**Must:** -## How It Works +1. Named imports only +2. Specific adapters (`openaiText`, not a monolith) +3. Keep provider SDKs server-side; mobile chat screens import `useChat` + connection adapters only — [Quick Start: React Native](../getting-started/quick-start-react-native) -The tree-shakeability is achieved through: +**Optional:** -1. **ES Module exports** - Each function is a named export, not a default export -2. **Separate modules** - Each activity and adapter lives in its own file -3. **No side effects** - Functions are pure and don't have module-level side effects -4. **Functional composition** - Functions compose together, allowing dead code elimination -5. **Type-only imports** - Type imports are stripped at build time +- Split activities by route +- Dynamic import for code-split routes -Modern bundlers (Vite, Webpack, Rollup, esbuild) can easily eliminate unused code because: +## Adapter kinds -- Functions are statically analyzable -- No dynamic imports of unused code -- No module-level side effects -- Clear dependency graphs - -## Best Practices - -1. **Import only what you need** - Don't import entire namespaces -2. **Use specific adapter functions** - Import `openaiText` not `openai` -3. **Separate activities by route** - Different API routes can use different activities -4. **Lazy load when possible** - Use dynamic imports for code-split routes -5. **Keep mobile chat bundles client-only** - React Native and Expo chat screens - should import `useChat` and chat connection adapters, not provider SDKs, - server response helpers, React DOM UI, devtools UI, or other framework - packages. See [Quick Start: React Native](../getting-started/quick-start-react-native) - for the server-only provider boundary and mobile transport setup. +| Interface | Method | `kind` example | +|-----------|--------|----------------| +| `ChatAdapter` | `chatStream()` | `'text'` | +| `SummarizeAdapter` | `summarize()` | `'summarize'` | +| `ImageAdapter` | `generateImage()` | — | +| `TTSAdapter` | `generateSpeech()` | — | +| `TranscriptionAdapter` | `generateTranscription()` | — | +| `VideoAdapter` | `generateVideo()` | — | ```ts -// ✅ Good - Only imports chat -import { chat } from '@tanstack/ai' -import { openaiText } from '@tanstack/ai-openai' +import { openaiText, openaiSummarize } from '@tanstack/ai-openai' -// ❌ Bad - Imports everything -import * as ai from '@tanstack/ai' -import * as openai from '@tanstack/ai-openai' +console.log(openaiText('gpt-5.5').kind) // 'text' +console.log(openaiSummarize('gpt-5.4-mini').kind) // 'summarize' ``` -## Adapter Types +## Why bundlers drop unused code -Each adapter type implements a specific interface: +1. Named ES module exports +2. Separate modules per activity/adapter +3. No module-level side effects +4. Type-only imports stripped at build -- `ChatAdapter` - Provides `chatStream()` method for streaming chat responses -- `SummarizeAdapter` - Provides `summarize()` method for text summarization -- `ImageAdapter` - Provides `generateImage()` method for image generation -- `TTSAdapter` - Provides `generateSpeech()` method for text-to-speech -- `TranscriptionAdapter` - Provides `generateTranscription()` method for audio transcription -- `VideoAdapter` - Provides `generateVideo()` method for video generation +Type safety is unchanged — each adapter still narrows models: -All adapters have a `kind` property that indicates their type: - -```ts -import { openaiText, openaiSummarize } from '@tanstack/ai-openai' - -const chatAdapter = openaiText('gpt-5.5') -console.log(chatAdapter.kind) // 'text' +```ts ignore +import { openaiText, type OpenAIChatModel } from '@tanstack/ai-openai' -const summarizeAdapter = openaiSummarize('gpt-5.4-mini') -console.log(summarizeAdapter.kind) // 'summarize' +const model: OpenAIChatModel = 'gpt-5.5' // ok +// const model2: OpenAIChatModel = 'invalid' // error ``` - -## Summary - -TanStack AI's tree-shakeable design means: - -- ✅ **Smaller bundles** - Only include code you actually use -- ✅ **Faster load times** - Less JavaScript to download and parse -- ✅ **Better performance** - Less code means faster execution -- ✅ **Type safety** - Full TypeScript support without runtime overhead -- ✅ **Flexibility** - Mix and match activities and adapters as needed - -The functional, modular architecture ensures that modern bundlers can eliminate unused code effectively, resulting in optimal bundle sizes for your application. diff --git a/docs/advanced/typed-options.md b/docs/advanced/typed-options.md index cafa8a904..9ec201006 100644 --- a/docs/advanced/typed-options.md +++ b/docs/advanced/typed-options.md @@ -2,7 +2,7 @@ title: Typed Pre-Configured Options id: typed-options order: 11 -description: "Define typed, reusable option objects for chat, summarize, image, video, audio, speech, and transcription with createChatOptions and friends — share configuration across routes without losing per-model type safety." +description: "Reuse typed activity options with createChatOptions and siblings without losing per-model inference." keywords: - tanstack ai - createChatOptions @@ -16,11 +16,11 @@ keywords: - shared configuration --- -You have a `chat()` (or `generateImage()`, `generateSpeech()`, …) configuration you want to reuse — across multiple routes, between a server function and its caller, or simply factored out of a handler for clarity. By the end of this guide, you'll have a single typed options object that infers the adapter's model, modalities, and provider options, and that you can spread into any call site without losing type safety. +If you reuse the same activity config across routes or layers → wrap it with `createXxxOptions`, then spread at the call site. -## The pattern +## Pattern -Every activity in `@tanstack/ai` ships a paired `createXxxOptions` helper that takes the exact same options object as the activity itself and returns it unchanged — at runtime it's the identity function. The point is **type inference**: the returned object carries the adapter's full type, so when you spread it into the activity, TypeScript still narrows `modelOptions`, content modalities, and `outputSchema` to the adapter you chose. +Helpers are the identity function at runtime. Point is **type inference**: returned object keeps adapter/model generics so `modelOptions`, modalities, and schemas stay narrowed. ```typescript import { chat, createChatOptions } from '@tanstack/ai' @@ -28,50 +28,42 @@ import { openaiText } from '@tanstack/ai-openai' const chatOptions = createChatOptions({ adapter: openaiText('gpt-5.5'), - // modelOptions, systemPrompts, tools — all type-checked against the - // adapter+model pair above. Sampling params (temperature, top_p, - // max_output_tokens, …) live inside modelOptions, under each provider's - // native key. modelOptions: { temperature: 0.3, reasoning: { effort: 'medium' }, }, }) -// Later, anywhere in your codebase: -const stream = chat({ ...chatOptions, messages: [{ role: 'user', content: 'Hello' }] }) +const stream = chat({ + ...chatOptions, + messages: [{ role: 'user', content: 'Hello' }], +}) ``` -Without the helper you'd have to either inline the configuration at every call site, or hand-write the full chat options type with its adapter/model generics resolved manually — `createChatOptions` does that for you. - -## When to reach for it +Skip the helper if you only call the activity once inline. -- **Sharing a configuration across multiple routes** — define once, spread into each handler. -- **Passing options through a layer** (a server function, a wrapper, a test fixture) without erasing the adapter's model-specific types. -- **Branching on a runtime value while keeping types intact** — build different options objects and choose between them, instead of weaving conditionals into a single `chat({...})` call. -- **Co-locating tools, system prompts, and middleware** with the adapter they target. +## When to use -If you only call an activity once at one site, you don't need this helper. Inline the options. +1. Share config across routes +2. Pass options through a layer without erasing adapter types +3. Branch on runtime value with separate options objects +4. Co-locate tools, system prompts, middleware with the adapter -## Available helpers +## Helpers -Each helper mirrors the activity it pairs with. Same options, same return type. - -| Helper | Activity | Adapter | +| Helper | Activity | Adapter example | |---|---|---| -| `createChatOptions` | `chat()` | text adapter (e.g. `openaiText`, `anthropicText`) | -| `createSummarizeOptions` | `summarize()` | summarize adapter (e.g. `openaiSummarize`) | -| `createImageOptions` | `generateImage()` | image adapter (e.g. `openaiImage`, `falImage`) | -| `createAudioOptions` | `generateAudio()` | audio adapter (e.g. `falAudio`, `geminiAudio`) | -| `createVideoOptions` | `generateVideo()` / `getVideoJobStatus()` | video adapter (e.g. `falVideo`, `openaiVideo`) | -| `createSpeechOptions` | `generateSpeech()` | speech adapter (e.g. `openaiSpeech`, `elevenlabsSpeech`) | -| `createTranscriptionOptions` | `generateTranscription()` | transcription adapter (e.g. `openaiTranscription`, `falTranscription`) | - -All helpers are exported from `@tanstack/ai`. +| `createChatOptions` | `chat()` | `openaiText`, `anthropicText` | +| `createSummarizeOptions` | `summarize()` | `openaiSummarize` | +| `createImageOptions` | `generateImage()` | `openaiImage`, `falImage` | +| `createAudioOptions` | `generateAudio()` | `falAudio`, `geminiAudio` | +| `createVideoOptions` | `generateVideo()` / `getVideoJobStatus()` | `falVideo`, `openaiVideo` | +| `createSpeechOptions` | `generateSpeech()` | `openaiSpeech`, `elevenlabsSpeech` | +| `createTranscriptionOptions` | `generateTranscription()` | `openaiTranscription`, `falTranscription` | -## Example: shared chat configuration across routes +All from `@tanstack/ai`. -Suppose you have several routes that all hit the same model with the same provider options and tool set. Factor the configuration out once: +## Shared chat config across routes ```typescript // lib/ai/chat-options.ts @@ -113,7 +105,7 @@ export async function POST(request: Request) { ``` ```typescript ignore -// routes/api/support/draft-reply.ts — same adapter+tools, different schema +// routes/api/support/draft-reply.ts import { chat } from '@tanstack/ai' import { supportChatOptions } from '@/lib/ai/chat-options' import { z } from 'zod' @@ -130,9 +122,9 @@ export async function POST(request: Request) { } ``` -Both routes share the adapter, system prompt, tools, and reasoning settings; each adds what it needs. Override or omit any field at the call site — the spread wins on the right. +Right-hand spread overrides shared fields. -## Example: typed pre-configured image generation +## Image options ```typescript import { createImageOptions, generateImage } from '@tanstack/ai' @@ -148,16 +140,16 @@ const heroImageOptions = createImageOptions({ const result = await generateImage(heroImageOptions) ``` -The same pattern works for `createVideoOptions`, `createSpeechOptions`, `createTranscriptionOptions`, `createAudioOptions`, and `createSummarizeOptions` — the adapter is captured in the typed options object and every downstream call is narrowed to it. +Same pattern for video, speech, transcription, audio, summarize. -## What the helper does NOT do +## What helpers do not do -- **No runtime behavior.** `createChatOptions(opts)` is `opts`. There is no validation, freezing, cloning, or memoization. If you mutate the returned object after creation, the next call sees the mutation. Treat the result as immutable by convention. -- **No partial typing.** The helper expects the full options shape it'll be spread into. If you need to build options up incrementally, type the intermediate state yourself (a `Partial<>` of the full chat options shape) and only call the helper at the boundary where the shape is complete. -- **No request execution.** The helper does not call the model. Only the activity function (`chat`, `generateImage`, …) makes the request. +1. **No runtime behavior** — no validate/freeze/clone. Treat as immutable by convention. +2. **No partial typing** — pass a complete options shape; build partials yourself until the boundary. +3. **No request** — only the activity function (`chat`, `generateImage`, …) calls the model. ## Related -- [Per-Model Type Safety](./per-model-type-safety) — how the adapter+model pair drives `modelOptions` inference. -- [Tree-Shaking](./tree-shaking) — why each adapter is exported separately, and how the typed-options pattern keeps your bundle small. -- [Extend Adapter](./extend-adapter) — when you need to add custom models to an adapter without losing the same typed-options ergonomics. +- [Per-Model Type Safety](./per-model-type-safety) — adapter+model drives `modelOptions` +- [Tree-Shaking](./tree-shaking) — separate adapter imports +- [Extend Adapter](./extend-adapter) — custom models with the same options ergonomics diff --git a/docs/api/ai-angular.md b/docs/api/ai-angular.md index 2ae10c658..eb1285c95 100644 --- a/docs/api/ai-angular.md +++ b/docs/api/ai-angular.md @@ -2,7 +2,7 @@ title: "@tanstack/ai-angular" id: ai-angular order: 6 -description: "API reference for @tanstack/ai-angular — Angular signal-based injectables including injectChat for streaming chat with full type safety." +description: "Angular injectChat, generation injectables, reactive options, structured output." keywords: - tanstack ai - "@tanstack/ai-angular" @@ -13,19 +13,15 @@ keywords: - api reference --- -Angular signal-based bindings for TanStack AI, providing convenient Angular bindings for the headless client. - -> **Injection context requirement:** Every `inject*` function in this package calls Angular's `inject()` internally. They **must** be called within an Angular injection context — a component or directive class field initializer, the constructor, or inside `runInInjectionContext`. Calling them outside an injection context will throw a runtime error. - -## Installation +If you need streaming chat in Angular → call `injectChat` **inside an injection context**. ```bash npm install @tanstack/ai-angular ``` -## `injectChat(options?)` +> Every `inject*` function calls Angular `inject()`. Valid sites: class field initializer, constructor, or `runInInjectionContext`. Outside → runtime error. -Main injectable for managing chat state in Angular with full type safety. +## `injectChat(options?)` ```typescript import { Component } from "@angular/core"; @@ -37,38 +33,28 @@ import { injectChat, fetchServerSentEvents } from "@tanstack/ai-angular"; template: `...`, }) export class ChatComponent { - // injectChat is called in a field initializer — valid injection context. chat = injectChat({ connection: fetchServerSentEvents("/api/chat"), }); } ``` +Read signals by calling them: `chat.messages()`, `chat.isLoading()`. Cleanup via `DestroyRef.onDestroy`. Client tools auto-run (no `onToolCall`). + ### Options -Extends `ChatClientOptions` from `@tanstack/ai-client` (minus internal state callbacks): - -- `connection` - Connection adapter (required, or use `fetcher`) -- `fetcher?` - Direct async function for one-shot generation (alternative to `connection`) -- `tools?` - Array of client tool implementations (with `.client()` method) -- `initialMessages?` - Initial messages array -- `id?` - Unique identifier for this chat instance -- `threadId?` - Thread ID for AG-UI run correlation. Persists across sends; auto-generated if omitted -- `forwardedProps?` - Arbitrary client-controlled JSON forwarded to the server in the AG-UI `RunAgentInput.forwardedProps` field. Reactive — accepts a plain value, an Angular `Signal`, or a zero-arg getter; changes sync automatically via `effect` -- `body?` - **Deprecated.** Use `forwardedProps` instead. Still works for backward compatibility; values are merged into `forwardedProps` on the wire. Reactive (same forms as `forwardedProps`) -- `context?` - Typed client-local runtime context passed to client tool implementations. Reactive (same forms). This value is not serialized to the server -- `live?` - Enable live subscription mode (auto-subscribes/unsubscribes). Reactive (same forms) -- `outputSchema?` - Standard-schema-compatible schema (Zod, Valibot, ArkType, or JSON Schema). When provided, adds typed `partial` and `final` signals to the return value -- `persistence?` - Persistence configuration -- `devtools?` - Display options for TanStack AI Devtools -- `onResponse?` - Callback when response is received -- `onChunk?` - Callback when stream chunk is received -- `onFinish?` - Callback when response finishes -- `onError?` - Callback when error occurs -- `onCustomEvent?` - Callback for custom stream events -- `streamProcessor?` - Stream processing configuration - -**Reactive options** (`body`, `forwardedProps`, `context`, `live`) accept a `ReactiveOption`, which is one of: +Extends `ChatClientOptions` (minus internal state callbacks): + +- `connection` — required (or use `fetcher` for one-shot) +- `tools?` — `.client()` implementations +- `initialMessages?` / `id?` / `threadId?` — seed + AG-UI thread +- `forwardedProps?` — reactive (`T | Signal | () => T`) +- `context?` / `live?` — reactive; `context` is client-local (not serialized) + +Also: `outputSchema?` (adds typed `partial` / `final`), `persistence?`, `devtools?`, `onResponse?`, `onChunk?`, `onFinish?`, `onError?`, `onCustomEvent?`, `streamProcessor?`. +`body?` is **deprecated** (still reactive + merged into `forwardedProps`). + +### Reactive options ```typescript import type { Signal } from "@angular/core"; @@ -76,9 +62,7 @@ import type { Signal } from "@angular/core"; type ReactiveOption = T | Signal | (() => T); ``` -A plain value becomes a constant; a `Signal` is read directly; a zero-arg getter is wrapped in `computed` so any signals read inside it are tracked. - -**Note:** Client tools are automatically executed — no `onToolCall` callback needed! +Plain value = constant. `Signal` read directly. Zero-arg getter → wrapped in `computed`. ### Returns @@ -118,17 +102,15 @@ interface InjectChatResult { isSubscribed: Signal; connectionStatus: Signal; sessionGenerating: Signal; - // Only present when outputSchema is supplied: + // Only when outputSchema is set: partial: Signal>>; final: Signal | null>; } ``` -**Note:** All reactive state (`messages`, `isLoading`, `error`, `status`, `isSubscribed`, `connectionStatus`, `sessionGenerating`) is exposed as read-only Angular `Signal`s. Read them by calling them as functions (e.g., `chat.messages()`, `chat.isLoading()`). Cleanup is automatic via `DestroyRef.onDestroy`. - -## Connection Adapters +--- -Re-exported from `@tanstack/ai-client` for convenience: +## Connection adapters ```typescript import { @@ -142,7 +124,9 @@ import { } from "@tanstack/ai-angular"; ``` -## Example: Basic Chat +--- + +## Basic chat ```typescript import { Component } from "@angular/core"; @@ -187,7 +171,7 @@ export class ChatComponent { } ``` -## Example: Tool Approval +## Tool approval ```typescript import { Component } from "@angular/core"; @@ -227,16 +211,12 @@ export class ApprovalChatComponent { } ``` -## Example: Client Tools with Type Safety +## Client tools (typed) ```typescript import { Component } from "@angular/core"; import { CommonModule } from "@angular/common"; import { injectChat, fetchServerSentEvents } from "@tanstack/ai-angular"; -import { - createChatClientOptions, - type InferChatMessages, -} from "@tanstack/ai-client"; import { toolDefinition } from "@tanstack/ai"; import { z } from "zod"; @@ -267,9 +247,7 @@ const saveToStorageDef = toolDefinition({ `, }) export class TypedChatComponent { - // Create client implementations private updateUI = updateUIDef.client((input) => { - // input is fully typed! return { success: true }; }); @@ -278,17 +256,16 @@ export class TypedChatComponent { return { saved: true }; }); - // Create typed tools array (no 'as const' needed!) private tools = [this.updateUI, this.saveToStorage]; chat = injectChat({ connection: fetchServerSentEvents("/api/chat"), - tools: this.tools, // Automatic execution, full type safety + tools: this.tools, }); } ``` -## Example: Reactive Options with Signals +## Reactive options ```typescript import { Component, signal } from "@angular/core"; @@ -307,7 +284,6 @@ import { injectChat, fetchServerSentEvents } from "@tanstack/ai-angular"; export class ReactiveChatComponent { language = signal("en"); - // forwardedProps is reactive — the signal is read on every request chat = injectChat({ connection: fetchServerSentEvents("/api/chat"), forwardedProps: () => ({ language: this.language() }), @@ -319,7 +295,7 @@ export class ReactiveChatComponent { } ``` -## Example: Structured Output +## Structured output ```typescript import { Component } from "@angular/core"; @@ -357,13 +333,13 @@ export class RecipeChatComponent { } ``` -## Generation Injectables +--- -Angular injectables for one-shot generation tasks (images, audio, speech, transcription, summarization, video). All share the same pattern: provide a `connection` or `fetcher`, call `generate()`, and read reactive signals. +## Generation injectables -### `injectGeneration(options)` +Provide `connection` or `fetcher`, call `generate()`, read signals. Cleanup via `DestroyRef.onDestroy`. -Base injectable for custom generation types. All specialized injectables below are built on this. +### `injectGeneration(options)` ```typescript import { Component } from "@angular/core"; @@ -375,18 +351,24 @@ export class CustomGenerationComponent { gen = injectGeneration({ connection: fetchServerSentEvents("/api/generate/custom"), }); - - // Call gen.generate(input), read gen.result(), gen.isLoading(), etc. } ``` **Options:** `connection?`, `fetcher?`, `id?`, `body?` (reactive), `devtools?`, `onResult?`, `onError?`, `onProgress?`, `onChunk?` -**Returns:** `generate`, `result`, `isLoading`, `error`, `status`, `stop`, `reset`, `runId`. All reactive state is a read-only `Signal`. +**Returns:** `generate`, `result`, `isLoading`, `error`, `status`, `stop`, `reset`, `runId` (signals). + +### Specialized -### `injectGenerateImage(options)` +| Injectable | Input | Notes | +| --- | --- | --- | +| `injectGenerateImage` | `ImageGenerateInput` | `ImageGenerationResult` | +| `injectGenerateAudio` | `AudioGenerateInput` | `AudioGenerationResult` | +| `injectGenerateSpeech` | `SpeechGenerateInput` | `TTSResult` | +| `injectTranscription` | `TranscriptionGenerateInput` | `TranscriptionResult` | +| `injectSummarize` / `injectGenerateVideo` | summarize / video | video adds `jobId`, `videoStatus` | -Image generation injectable. `generate()` accepts `ImageGenerateInput`, result is `ImageGenerationResult`. +### Image example ```typescript import { Component } from "@angular/core"; @@ -412,9 +394,7 @@ export class ImageComponent { } ``` -### `injectGenerateAudio(options)` - -Audio generation injectable (music, sound effects). `generate()` accepts `AudioGenerateInput`, result is `AudioGenerationResult`. +### Audio example ```typescript import { Component } from "@angular/core"; @@ -440,21 +420,7 @@ export class AudioComponent { } ``` -### `injectGenerateSpeech(options)` - -Text-to-speech injectable. `generate()` accepts `SpeechGenerateInput`, result is `TTSResult`. - -### `injectTranscription(options)` - -Audio transcription injectable. `generate()` accepts `TranscriptionGenerateInput`, result is `TranscriptionResult`. - -### `injectSummarize(options)` - -Text summarization injectable. `generate()` accepts `SummarizeGenerateInput`, result is `SummarizationResult`. - -### `injectGenerateVideo(options)` - -Video generation injectable with job polling. Returns additional `jobId` and `videoStatus` signals. Accepts extra `onJobCreated?` and `onStatusUpdate?` callbacks. +### Video example ```typescript import { Component } from "@angular/core"; @@ -484,15 +450,11 @@ export class VideoComponent { } ``` -**Additional returns (video only):** -- `jobId: Signal` — The polling job ID, once the server creates it -- `videoStatus: Signal` — Real-time status updates from the polling loop +Video-only returns: `jobId: Signal`, `videoStatus: Signal`. -All generation injectables automatically clean up via `DestroyRef.onDestroy`. - -## Injection Context +--- -Angular's DI system requires that `inject()` is called during component construction. Every `inject*` function in this package calls `inject()` internally. Valid call sites: +## Injection context ```typescript import { inject, runInInjectionContext, Injector } from "@angular/core"; @@ -513,7 +475,7 @@ export class MyComponentAlt { } } -// Inside runInInjectionContext +// runInInjectionContext const chat = runInInjectionContext(injector, () => injectChat({ connection: fetchServerSentEvents("/api/chat") }), ); @@ -521,8 +483,6 @@ const chat = runInInjectionContext(injector, () => ## `createChatClientOptions(options)` -Helper to create typed chat options (re-exported from `@tanstack/ai-client`). - ```typescript import { createChatClientOptions, @@ -531,7 +491,6 @@ import { import { fetchServerSentEvents } from "@tanstack/ai-angular"; import { tool1, tool2 } from "./tools"; -// Create typed tools array (no 'as const' needed!) const tools = [tool1, tool2]; const chatOptions = createChatClientOptions({ @@ -544,36 +503,12 @@ type Messages = InferChatMessages; ## Types -Re-exported from `@tanstack/ai-angular` (sourced from `@tanstack/ai-client`): - -- `UIMessage` - Message type with tool type parameter -- `InjectChatOptions` - Chat injectable options -- `InjectChatResult` - Chat injectable return type -- `ReactiveOption` - Union of `T | Signal | (() => T)` for reactive option fields -- `DeepPartial` - Recursive partial; used to type the in-flight `partial` value -- `ChatRequestBody` - Request body type -- `MultimodalContent` - Multimodal content type for `sendMessage` -- `ConnectionAdapter` - Connection adapter interface -- `InferChatMessages` - Extract message type from options -- `GenerationClientState` - Generation lifecycle state -- `ImageGenerateInput` - Image generation input type -- `AudioGenerateInput` - Audio generation input type -- `SpeechGenerateInput` - Speech generation input type -- `TranscriptionGenerateInput` - Transcription input type -- `SummarizeGenerateInput` - Summarization input type -- `VideoGenerateInput` - Video generation input type -- `VideoGenerateResult` - Video generation result type -- `VideoStatusInfo` - Video job status info - -Tool authoring types — import directly from `@tanstack/ai` (not re-exported by `@tanstack/ai-angular`): - -- `toolDefinition()` - Create isomorphic tool definition -- `ToolDefinitionInstance` - Tool definition type -- `ClientTool` - Client tool type -- `ServerTool` - Server tool type +From `@tanstack/ai-angular` / client: `UIMessage`, `InjectChatOptions`, `InjectChatResult`, `ReactiveOption`, `DeepPartial`, `ChatRequestBody`, `MultimodalContent`, `ConnectionAdapter`, `InferChatMessages`, generation types (`GenerationClientState`, `ImageGenerateInput`, `AudioGenerateInput`, `SpeechGenerateInput`, `TranscriptionGenerateInput`, `SummarizeGenerateInput`, `VideoGenerateInput`, `VideoGenerateResult`, `VideoStatusInfo`). + +Tool authoring — import from `@tanstack/ai` (not re-exported here): `toolDefinition()`, `ToolDefinitionInstance`, `ClientTool`, `ServerTool`. ## Next Steps -- [Getting Started](../getting-started/quick-start) - Learn the basics -- [Tools Guide](../tools/tools) - Learn about the isomorphic tool system -- [Client Tools](../tools/client-tools) - Learn about client-side tools +- [Getting Started](../getting-started/quick-start) +- [Tools Guide](../tools/tools) +- [Client Tools](../tools/client-tools) diff --git a/docs/api/ai-client.md b/docs/api/ai-client.md index fda57f7e7..db849265f 100644 --- a/docs/api/ai-client.md +++ b/docs/api/ai-client.md @@ -2,7 +2,7 @@ title: "@tanstack/ai-client" slug: /api/ai-client order: 2 -description: "API reference for @tanstack/ai-client — the framework-agnostic headless client for managing chat state and streaming transports." +description: "Headless ChatClient, connection adapters, and typed helpers for any framework." keywords: - tanstack ai - "@tanstack/ai-client" @@ -13,17 +13,20 @@ keywords: - api reference --- -Framework-agnostic headless client for managing chat state and streaming. - -## Installation +If you need chat state without a framework, or to build a custom UI → use `ChatClient`. ```bash npm install @tanstack/ai-client ``` -## `ChatClient` +Framework hooks (`@tanstack/ai-react`, `-vue`, `-solid`, `-svelte`, `-preact`, `-angular`) wrap this client and call `attach()` / `detach()` for you. + +## Quick start -The main client class for managing chat state. +1. Create a client with a connection adapter. +2. Call `attach()` when the view appears. +3. Call `sendMessage()` / `stop()` / etc. +4. Call `detach()` when the view goes away (not `stop()` — that ends the run). ```typescript import { @@ -42,174 +45,84 @@ const client = new ChatClient({ }, }); -// A new client is IDLE. Attach it when your view appears, detach when it goes. client.attach(); ``` -### Lifecycle: `attach()` and `detach()` - -One page can hold many chats. A browser allows only about six connections to one -origin, and a chat that is tailing a run holds one for as long as that run lasts. If -every chat held a connection, a handful of open views would use every slot and every -other request would queue behind them, including the request that loads your messages. +### Why attach/detach? -So the connection follows the view. A new client holds none, `attach()` starts it, and -`detach()` stops it. +Browsers limit ~6 connections per origin. A tailing chat holds one slot for the whole run. Attach only while a view is watching; detach drops the socket but keeps transcript + resume pointer so re-attach can rejoin. -If you use a framework package (`@tanstack/ai-react`, `-vue`, `-solid`, `-svelte`, -`-preact`, `-angular`), the hook already does this: it attaches when its view mounts -and detaches when it unmounts. Call these yourself only when you use `ChatClient` -directly. - -```typescript -import { ChatClient, fetchServerSentEvents } from "@tanstack/ai-client"; +| Method | Effect | +| --- | --- | +| `attach()` | Start tailing; rejoin in-progress run; load thread if persisted. Idempotent. | +| `detach()` | Drop connection; keep messages, run id, resume pointer. Not `stop()` or `dispose()`. | +| `stop()` | End the current generation. | -const client = new ChatClient({ - connection: fetchServerSentEvents("/api/chat"), - threadId: "thread-1", - persistence: true, -}); +No persistence → `attach()` issues no request. -client.attach(); // start: rejoin a run in progress, and load the thread -client.detach(); // stop: drop the connection, keep messages and the run pointer -``` +**Migration:** older versions tailed in the constructor. If you use `ChatClient` directly, add `attach()` / `detach()` at view mount/unmount. Framework hooks need no change. -What each one guarantees: +--- -- `attach()` is safe to call more than once. Attaching an attached client does nothing. -- `detach()` keeps the transcript, the resume pointer and the run id. The run keeps - going on the server while nobody watches, so re-attaching repaints at once and picks - it back up from the durable log. -- `detach()` is neither `stop()` (which ends the run) nor `dispose()` (which ends the - client). It says only that no view is watching right now. -- A chat with no persistence has no resume pointer and no stored thread, so `attach()` - issues no request at all. +## Constructor options -#### Migrating from constructor tailing +### Required / core -Earlier versions started tailing inside the constructor. If you build a `ChatClient` -yourself, add `client.attach()` where your view appears and `client.detach()` where it -goes away. Users of the framework hooks need no change. +- `connection` — streaming adapter +- `tools?` — `.client()` tools (auto-executed on match) +- `initialMessages?` / `id?` / `threadId?` — seed + AG-UI thread (auto thread id if omitted) +- `forwardedProps?` — client JSON → server `RunAgentInput.forwardedProps` +- `context?` — client-local typed context for client tools (not sent to server) -### Constructor Options +### Callbacks & processing -- `connection` - Connection adapter for streaming -- `initialMessages?` - Initial messages array -- `id?` - Unique identifier for this chat instance -- `threadId?` - Thread ID for AG-UI run correlation. Persists across sends; auto-generated if omitted -- `forwardedProps?` - Arbitrary client-controlled JSON forwarded to the server in the AG-UI `RunAgentInput.forwardedProps` field -- `body?` - **Deprecated.** Use `forwardedProps` instead. Still works — values are merged into `forwardedProps` on the wire and mirrored under the legacy `data` field for backward compatibility -- `context?` - Typed client-local runtime context passed to client tool implementations. This value is not serialized to the server -- `tools?` - Registered `.client()` tool implementations. The client automatically executes matching tools when the model calls them -- `onResponse?` - Callback when response is received -- `onChunk?` - Callback when stream chunk is received -- `onFinish?` - Callback when response finishes -- `onError?` - Callback when error occurs -- `onMessagesChange?` - Callback when messages change -- `onLoadingChange?` - Callback when loading state changes -- `onErrorChange?` - Callback when error state changes -- `streamProcessor?` - Stream processing configuration +- `onResponse?` / `onChunk?` / `onFinish?` / `onError?` +- `onMessagesChange?` / `onLoadingChange?` / `onErrorChange?` +- `streamProcessor?` — chunk strategy config +- `body?` — **Deprecated.** Prefer `forwardedProps` (still merged on the wire + legacy `data` mirror) -### Methods +--- -#### `sendMessage(content: string)` +## Methods -Sends a user message and gets a response. +### `sendMessage(content)` ```typescript -import { client } from "./client"; - await client.sendMessage("Hello!"); ``` -#### `append(message: ModelMessage | UIMessage)` - -Appends a message to the conversation. +### `append(message)` ```typescript -import { client } from "./client"; - -await client.append({ - role: "user", - content: "Additional context", -}); +await client.append({ role: "user", content: "Additional context" }); ``` -#### `reload()` - -Reloads the last assistant message. +### `reload()` / `stop()` / `clear()` ```typescript -import { client } from "./client"; - await client.reload(); -``` - -#### `attach()` - -Start tailing. Rejoins a run that is still in progress and, in server-authoritative -mode, loads the stored thread. Idempotent. See -[Lifecycle](#lifecycle-attach-and-detach). - -#### `detach()` - -Stop tailing and drop the connection. Keeps messages, the run pointer and the run -id, so a later `attach()` continues where it left off. See -[Lifecycle](#lifecycle-attach-and-detach). - -#### `stop()` - -Stops the current response generation. - -```typescript -import { client } from "./client"; - client.stop(); -``` - -#### `clear()` - -Clears all messages. - -```typescript -import { client } from "./client"; - client.clear(); ``` -#### `setMessagesManually(messages: UIMessage[])` - -Manually sets the messages array. +### `setMessagesManually(messages)` ```typescript -import { client } from "./client"; import type { UIMessage } from "@tanstack/ai-client"; const newMessages: UIMessage[] = []; client.setMessagesManually([...newMessages]); ``` -#### `addToolResult(result)` - -Adds the result of a client-side tool execution. +### Tool result / approval ```typescript -import { client } from "./client"; - await client.addToolResult({ toolCallId: "call_123", tool: "toolName", output: { result: "..." }, state: "output-available", }); -``` - -#### `addToolApprovalResponse(response)` - -Responds to a tool approval request. - -```typescript -import { client } from "./client"; await client.addToolApprovalResponse({ id: "approval_123", @@ -219,128 +132,83 @@ await client.addToolApprovalResponse({ ### Properties -- `messages: UIMessage[]` - Current messages -- `isLoading: boolean` - Whether a response is being generated -- `error: Error | undefined` - Current error, if any - -## Connection Adapters +- `messages` — current `UIMessage[]` +- `isLoading` — generation in flight +- `error` — current error, if any -For a complete transport walkthrough, see -[Connection Adapters](../chat/connection-adapters). For React Native and Expo, -see [Quick Start: React Native](../getting-started/quick-start-react-native). +--- -### `fetchServerSentEvents(url, options?)` +## Connection adapters -Creates an SSE connection adapter. +Full guide: [Connection Adapters](../chat/connection-adapters). React Native: [Quick Start: React Native](../getting-started/quick-start-react-native). -```typescript -import { fetchServerSentEvents } from "@tanstack/ai-client"; +### Pick an adapter -const adapter = fetchServerSentEvents("/api/chat", { - headers: { - Authorization: "Bearer token", - }, -}); -``` +| Need | Adapter | Pair with (server) | +| --- | --- | --- | +| Browser SSE | `fetchServerSentEvents` | `toServerSentEventsResponse()` | +| Browser NDJSON | `fetchHttpStream` | `toHttpResponse()` | +| React Native / Expo (default) | `xhrHttpStream` | `toHttpResponse()` | +| RN SSE | `xhrServerSentEvents` | `toServerSentEventsResponse()` | +| Custom | `stream(connectFn)` | your protocol | -### `fetchHttpStream(url, options?)` +`fetchHttpStream` needs streaming `fetch` + `getReader()` + `TextDecoder`. Missing support → `UnsupportedResponseStreamError` — use XHR adapters on RN/Expo. -Creates a newline-delimited JSON HTTP stream connection adapter. Pair it with -`toHttpResponse()` on the server. +### Examples ```typescript -import { fetchHttpStream } from "@tanstack/ai-client"; - -const adapter = fetchHttpStream("/api/chat"); -``` - -`fetchHttpStream()` requires a runtime with streaming `fetch`, -`Response.body.getReader()`, and `TextDecoder`. If the runtime cannot expose an -incremental response body, it throws `UnsupportedResponseStreamError`; use the -XHR adapters in React Native or Expo. - -### `xhrHttpStream(url, options?)` +import { + fetchServerSentEvents, + fetchHttpStream, + xhrHttpStream, + xhrServerSentEvents, + stream, +} from "@tanstack/ai-client"; -Creates an `XMLHttpRequest`-backed newline-delimited JSON stream adapter. This -is the recommended default for React Native and Expo chat screens. Pair it with -`toHttpResponse()` on the server. +const sse = fetchServerSentEvents("/api/chat", { + headers: { Authorization: "Bearer token" }, +}); -```typescript -import { xhrHttpStream } from "@tanstack/ai-client"; +const http = fetchHttpStream("/api/chat"); -const adapter = xhrHttpStream("http://192.168.1.10:8787/chat/http", { +const xhrHttp = xhrHttpStream("http://192.168.1.10:8787/chat/http", { headers: { Authorization: "Bearer token" }, withCredentials: true, }); -``` -### `xhrServerSentEvents(url, options?)` +const xhrSse = xhrServerSentEvents("http://192.168.1.10:8787/chat/sse"); -Creates an `XMLHttpRequest`-backed SSE adapter for runtimes where XHR progress -events are more reliable than streaming `fetch`. Pair it with -`toServerSentEventsResponse()` on the server. - -```typescript -import { xhrServerSentEvents } from "@tanstack/ai-client"; - -const adapter = xhrServerSentEvents("http://192.168.1.10:8787/chat/sse"); +const custom = stream(async (messages, data, signal) => { + // `data` is merged forwardedProps + const response = await fetch("/api/chat", { + method: "POST", + body: JSON.stringify({ messages, forwardedProps: data }), + signal, + }); + return processStream(response); +}); ``` ### Adapter options -Fetch adapters accept: - -- `headers?: Record | Headers` -- `credentials?: RequestCredentials` -- `signal?: AbortSignal` -- `body?: Record` -- `fetchClient?: typeof globalThis.fetch` +**Fetch:** `headers?`, `credentials?`, `signal?`, `body?`, `fetchClient?` -XHR adapters accept: +**XHR:** `headers?`, `withCredentials?`, `signal?`, `body?`, `xhrFactory?` -- `headers?: Record | Headers` -- `withCredentials?: boolean` -- `signal?: AbortSignal` -- `body?: Record` -- `xhrFactory?: () => XMLHttpRequest` - -`body` is merged into the AG-UI `forwardedProps` payload. Values from -`forwardedProps` on the client and per-message `sendMessage(..., data)` calls -override static adapter `body` values. +`body` merges into AG-UI `forwardedProps`. Client `forwardedProps` and per-message `sendMessage(..., data)` override static adapter `body`. ### Stream errors -- `UnsupportedResponseStreamError` - thrown by fetch-based adapters when - `Response.body`, `Response.body.getReader()`, or `TextDecoder` is missing. -- `StreamTruncatedError` - thrown when an SSE or NDJSON stream ends with - unterminated trailing data, usually because the server, proxy, or network cut - the connection mid-line. - -### `stream(connectFn)` +- `UnsupportedResponseStreamError` — no streaming body/reader/decoder +- `StreamTruncatedError` — stream ended mid-line -Creates a custom connection adapter. - -```typescript ignore -import { stream } from "@tanstack/ai-client"; - -const adapter = stream(async (messages, data, signal) => { - // `data` here carries the merged forwardedProps. The fetch-based - // adapters serialize it as the AG-UI `RunAgentInput.forwardedProps` - // field on the wire (with a backward-compat `data` mirror). - const response = await fetch("/api/chat", { - method: "POST", - body: JSON.stringify({ messages, forwardedProps: data }), - signal, - }); - return processStream(response); -}); -``` +--- -## Helper Functions +## Helpers ### `clientTools(...tools)` -**Optional.** A plain array — `tools: [tool1, tool2]` — already narrows tool names, inputs and outputs without any wrapper or `as const`. `clientTools()` is an identity helper that performs the same capture explicitly; reach for it only when you want to build a shared, reusable tools tuple outside the hook/options call. +Optional. A plain array already narrows types. Use this for an explicit reusable tools tuple. ```typescript import { @@ -361,40 +229,21 @@ const myTool1 = toolDefinition({ outputSchema: z.object({ result: z.string() }), }); -const myTool2 = toolDefinition({ - name: "myTool2", - description: "Second tool", - inputSchema: z.object({ query: z.string() }), - outputSchema: z.object({ result: z.string() }), -}); - -// Create client implementations const tool1Client = myTool1.client((input) => { - // Implementation return { result: input.query }; }); -const tool2Client = myTool2.client((input) => { - // Implementation - return { result: input.query }; -}); +const tools = clientTools(tool1Client); -// The explicit-capture form (equivalent to `[tool1Client, tool2Client]`). -const tools = clientTools(tool1Client, tool2Client); - -// Now when you use these tools in chat options: const chatOptions = createChatClientOptions({ connection: fetchServerSentEvents("/api/chat"), - tools, // Fully typed with literal tool names + tools, }); -// In your component: messages.forEach((message) => { message.parts.forEach((part) => { if (part.type === "tool-call" && part.name === "myTool1") { - // ✅ TypeScript knows part.name is literally "myTool1" - // ✅ part.input is typed from myTool1's input schema - // ✅ part.output is typed from myTool1's output schema + // part.input / part.output typed from schemas } }); }); @@ -402,7 +251,7 @@ messages.forEach((message) => { ### `createChatClientOptions(options)` -Helper function to create typed chat client options with proper type inference. +Preserve tool + context types for `InferChatMessages`. ```typescript import { @@ -410,32 +259,10 @@ import { fetchServerSentEvents, type InferChatMessages, } from "@tanstack/ai-client"; -import { tool1, tool2 } from "./tools"; - -const tools = [tool1, tool2]; - -const chatOptions = createChatClientOptions({ - connection: fetchServerSentEvents("/api/chat"), - tools, -}); - -// Use InferChatMessages to extract message types -type ChatMessages = InferChatMessages; -``` - -`createChatClientOptions` also preserves typed client runtime context: - -```typescript -import { - createChatClientOptions, - fetchServerSentEvents, -} from "@tanstack/ai-client"; import { toolDefinition } from "@tanstack/ai"; import { z } from "zod"; -type ClientContext = { - activeProjectId: string; -}; +type ClientContext = { activeProjectId: string }; const projectTool = toolDefinition({ name: "projectAction", @@ -452,17 +279,19 @@ const tool = projectTool.client((input, ctx: { context: ClientCon const chatOptions = createChatClientOptions({ connection: fetchServerSentEvents("/api/chat"), tools: [tool], - context: { - activeProjectId: "project_123", - }, + context: { activeProjectId: "project_123" }, }); + +type ChatMessages = InferChatMessages; ``` -Client runtime context is local to the client instance. Use `forwardedProps` for explicit client-to-server handoff of serializable values, then validate and map those values into server `chat({ context })`. +Client `context` stays local. For server values, send `forwardedProps`, then map into server `chat({ context })`. + +--- ## Types -### `UIMessage` +### `UIMessage` / parts ```typescript ignore interface UIMessage { @@ -471,52 +300,50 @@ interface UIMessage { parts: MessagePart[]; createdAt?: Date; } -``` - -### `MessagePart` -```typescript ignore type MessagePart = TextPart | ThinkingPart | ToolCallPart | ToolResultPart; ``` -### `TextPart` - ```typescript interface TextPart { type: "text"; content: string; } -``` - -### `ThinkingPart` -```typescript interface ThinkingPart { type: "thinking"; content: string; } ``` -Thinking parts represent the model's internal reasoning process. They are typically displayed in a collapsible format and automatically collapse when the response text appears. Thinking parts are UI-only and are not sent back to the model in subsequent requests. - -**Note:** Thinking parts are only available when using models that support reasoning/thinking (e.g., Anthropic Claude with thinking enabled, OpenAI GPT-5 with reasoning enabled). +Thinking is UI-only (not resent to the model). Only models with reasoning/thinking support emit it. -### `ToolCallPart` +### `ToolCallPart` / states ```typescript ignore interface ToolCallPart { type: "tool-call"; id: string; name: string; - arguments: string; // JSON string (may be incomplete during streaming) - input?: any; // Parsed tool input (typed from tool's inputSchema) + arguments: string; // may be incomplete while streaming + input?: any; // typed from inputSchema state: ToolCallState; - approval?: ApprovalRequest; // only on tools declared `needsApproval: true` - output?: any; // Tool execution output (typed from tool's outputSchema) + approval?: ApprovalRequest; // only if needsApproval: true + output?: any; // typed from outputSchema } + +type ToolCallState = + | "awaiting-input" + | "input-streaming" + | "input-complete" + | "approval-requested" + | "approval-responded" + | "complete"; + +type ToolResultState = "streaming" | "complete" | "error"; ``` -When you pass a typed `tools` array (a plain array works — `clientTools()` is optional), the `input` and `output` fields are automatically typed based on your tool's Zod schemas, and `name` becomes a discriminated union enabling type narrowing. The `approval` field is present **only** on parts for tools declared with `needsApproval: true` — narrow by `part.name` (or guard with `'approval' in part`) before accessing it. +With a typed `tools` array, narrow on `part.name` for `input` / `output` / `approval`. ### `ToolResultPart` @@ -530,30 +357,9 @@ interface ToolResultPart { } ``` -### `ToolCallState` - -```typescript ignore -type ToolCallState = - | "awaiting-input" - | "input-streaming" - | "input-complete" - | "approval-requested" - | "approval-responded" - | "complete"; -``` - -### `ToolResultState` - -```typescript ignore -type ToolResultState = - | "streaming" - | "complete" - | "error"; -``` - -## Stream Processing +--- -Configure stream processing with chunk strategies: +## Stream processing ```typescript import { @@ -565,13 +371,13 @@ import { const client = new ChatClient({ connection: fetchServerSentEvents("/api/chat"), streamProcessor: { - chunkStrategy: new ImmediateStrategy(), // Emit every chunk + chunkStrategy: new ImmediateStrategy(), }, }); ``` ## Next Steps -- [Getting Started](../getting-started/quick-start) - Learn the basics -- [Connection Adapters](../chat/connection-adapters) - Learn about adapters -- [@tanstack/ai-react API](./ai-react) - React hooks wrapper +- [Getting Started](../getting-started/quick-start) +- [Connection Adapters](../chat/connection-adapters) +- [@tanstack/ai-react API](./ai-react) diff --git a/docs/api/ai-preact.md b/docs/api/ai-preact.md index 8876a6e62..23e1678b6 100644 --- a/docs/api/ai-preact.md +++ b/docs/api/ai-preact.md @@ -2,7 +2,7 @@ title: "@tanstack/ai-preact" slug: /api/ai-preact order: 5 -description: "API reference for @tanstack/ai-preact — Preact hooks including useChat for streaming chat with full type safety in Preact apps." +description: "Preact useChat hook, connection adapters, and typed client tools." keywords: - tanstack ai - "@tanstack/ai-preact" @@ -12,9 +12,7 @@ keywords: - api reference --- -Preact hooks for TanStack AI, providing convenient Preact bindings for the headless client. - -## Installation +If you need streaming chat in Preact → `useChat` + a connection adapter. ```bash npm install @tanstack/ai-preact @@ -22,13 +20,15 @@ npm install @tanstack/ai-preact ## `useChat(options?)` -Main hook for managing chat state in Preact with full type safety. +1. Wire a connection. +2. Pass `.client()` tools if needed (auto-executed). +3. Render `messages` / call `sendMessage`. ```tsx import { useChat, fetchServerSentEvents } from "@tanstack/ai-preact"; -import { - createChatClientOptions, - type InferChatMessages +import { + createChatClientOptions, + type InferChatMessages, } from "@tanstack/ai-client"; import { toolDefinition } from "@tanstack/ai"; import { z } from "zod"; @@ -42,13 +42,12 @@ const updateUIDef = toolDefinition({ function ChatComponent() { const [, setNotification] = useState(null); - // Create client tool implementations + const updateUI = updateUIDef.client((input) => { setNotification(input.message); return { success: true }; }); - // Create typed tools array (no 'as const' needed!) const tools = [updateUI]; const chatOptions = createChatClientOptions({ @@ -56,7 +55,6 @@ function ChatComponent() { tools, }); - // Fully typed messages! type ChatMessages = InferChatMessages; const { messages, sendMessage, isLoading, error, addToolApprovalResponse } = @@ -70,21 +68,14 @@ function ChatComponent() { Extends `ChatClientOptions` from `@tanstack/ai-client`: -- `connection` - Connection adapter (required) -- `tools?` - Array of client tool implementations (with `.client()` method) -- `initialMessages?` - Initial messages array -- `id?` - Unique identifier for this chat instance -- `threadId?` - Thread ID for AG-UI run correlation. Persists across sends; auto-generated if omitted -- `forwardedProps?` - Arbitrary client-controlled JSON forwarded to the server in the AG-UI `RunAgentInput.forwardedProps` field (e.g., `{ provider: 'openai', model: 'gpt-4o' }`) -- `body?` - **Deprecated.** Use `forwardedProps` instead. Still works for backward compatibility; values are merged into `forwardedProps` on the wire -- `context?` - Typed client-local runtime context passed to client tool implementations. This value is not serialized to the server -- `onResponse?` - Callback when response is received -- `onChunk?` - Callback when stream chunk is received -- `onFinish?` - Callback when response finishes -- `onError?` - Callback when error occurs -- `streamProcessor?` - Stream processing configuration - -**Note:** Client tools are now automatically executed - no `onToolCall` callback needed! +- `connection` — required adapter +- `tools?` — `.client()` implementations (auto-run; no `onToolCall`) +- `initialMessages?` / `id?` / `threadId?` — seed + AG-UI thread +- `forwardedProps?` — client JSON → server +- `context?` — client-local tool context (not serialized) + +Also: `onResponse?`, `onChunk?`, `onFinish?`, `onError?`, `streamProcessor?`. +`body?` is **deprecated** — use `forwardedProps`. ### Returns @@ -116,9 +107,9 @@ interface UseChatReturn { } ``` -## Connection Adapters +--- -Re-exported from `@tanstack/ai-client` for convenience: +## Connection adapters ```typescript import { @@ -129,7 +120,9 @@ import { } from "@tanstack/ai-preact"; ``` -## Example: Basic Chat +--- + +## Basic chat ```tsx import { useState } from "preact/hooks"; @@ -187,7 +180,7 @@ export function Chat() { } ``` -## Example: Tool Approval +## Tool approval ```tsx import { useChat, fetchServerSentEvents } from "@tanstack/ai-preact"; @@ -240,14 +233,10 @@ export function ChatWithApproval() { } ``` -## Example: Client Tools with Type Safety +## Client tools (typed) ```tsx import { useChat, fetchServerSentEvents } from "@tanstack/ai-preact"; -import { - createChatClientOptions, - type InferChatMessages -} from "@tanstack/ai-client"; import { toolDefinition } from "@tanstack/ai"; import { z } from "zod"; import { useState } from "preact/hooks"; @@ -265,11 +254,12 @@ const saveToStorageDef = toolDefinition({ }); export function ChatWithClientTools() { - const [notification, setNotification] = useState<{ message: string; type: string } | null>(null); + const [notification, setNotification] = useState<{ + message: string; + type: string; + } | null>(null); - // Create client implementations const updateUI = updateUIDef.client((input) => { - // ✅ input is fully typed! setNotification({ message: input.message, type: input.type }); return { success: true }; }); @@ -279,12 +269,11 @@ export function ChatWithClientTools() { return { saved: true }; }); - // Create typed tools array (no 'as const' needed!) const tools = [updateUI, saveToStorage]; const { messages, sendMessage } = useChat({ connection: fetchServerSentEvents("/api/chat"), - tools, // ✅ Automatic execution, full type safety + tools, }); return ( @@ -292,7 +281,6 @@ export function ChatWithClientTools() { {messages.map((message) => message.parts.map((part) => { if (part.type === "tool-call" && part.name === "updateUI") { - // ✅ part.input and part.output are fully typed! return
Tool executed: {part.name}
; } return null; @@ -305,17 +293,14 @@ export function ChatWithClientTools() { ## `createChatClientOptions(options)` -Helper to create typed chat options (re-exported from `@tanstack/ai-client`). - ```typescript -import { - createChatClientOptions, - type InferChatMessages +import { + createChatClientOptions, + type InferChatMessages, } from "@tanstack/ai-client"; import { fetchServerSentEvents } from "@tanstack/ai-preact"; import { tool1, tool2 } from "./tools"; -// Create typed tools array (no 'as const' needed!) const tools = [tool1, tool2]; const chatOptions = createChatClientOptions({ @@ -328,27 +313,12 @@ type Messages = InferChatMessages; ## Types -Re-exported from `@tanstack/ai-client`: - -- `UIMessage` - Message type with tool type parameter -- `MessagePart` - Message part with tool type parameter -- `TextPart` - Text content part -- `ThinkingPart` - Thinking content part -- `ToolCallPart` - Tool call part (discriminated union) -- `ToolResultPart` - Tool result part -- `ChatClientOptions` - Chat client options with typed client runtime context -- `ConnectionAdapter` - Connection adapter interface -- `InferChatMessages` - Extract message type from options - -Re-exported from `@tanstack/ai`: +From `@tanstack/ai-client`: `UIMessage`, `MessagePart`, `TextPart`, `ThinkingPart`, `ToolCallPart`, `ToolResultPart`, `ChatClientOptions`, `ConnectionAdapter`, `InferChatMessages`. -- `toolDefinition()` - Create isomorphic tool definition -- `ToolDefinitionInstance` - Tool definition type -- `ClientTool` - Client tool type -- `ServerTool` - Server tool type +From `@tanstack/ai`: `toolDefinition()`, `ToolDefinitionInstance`, `ClientTool`, `ServerTool`. ## Next Steps -- [Getting Started](../getting-started/quick-start) - Learn the basics -- [Tools Guide](../tools/tools) - Learn about the isomorphic tool system -- [Client Tools](../tools/client-tools) - Learn about client-side tools +- [Getting Started](../getting-started/quick-start) +- [Tools Guide](../tools/tools) +- [Client Tools](../tools/client-tools) diff --git a/docs/api/ai-react.md b/docs/api/ai-react.md index 01eb1587b..c9b34e188 100644 --- a/docs/api/ai-react.md +++ b/docs/api/ai-react.md @@ -2,7 +2,7 @@ title: "@tanstack/ai-react" slug: /api/ai-react order: 3 -description: "API reference for @tanstack/ai-react — React hooks including useChat for streaming chat with full type safety in React apps." +description: "React useChat hook, connection adapters, and typed client tools." keywords: - tanstack ai - "@tanstack/ai-react" @@ -12,29 +12,25 @@ keywords: - api reference --- -React hooks for TanStack AI, providing convenient React bindings for the headless client. -For React Native, the documented support surface is narrow: `useChat` with chat -connection adapters. React DOM-specific UI packages and TanStack AI devtools UI -are not part of the React Native support surface. - -For a complete native journey, see -[Quick Start: React Native](../getting-started/quick-start-react-native). - -## Installation +If you need streaming chat in React → `useChat` + a connection adapter. ```bash npm install @tanstack/ai-react ``` +**React Native:** supported surface is `useChat` + connection adapters only (no DOM/devtools UI packages). See [Quick Start: React Native](../getting-started/quick-start-react-native). + ## `useChat(options?)` -Main hook for managing chat state in React with full type safety. +1. Wire a connection. +2. Pass `.client()` tools if needed (auto-executed). +3. Render `messages` / call `sendMessage`. ```tsx import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; -import { - createChatClientOptions, - type InferChatMessages +import { + createChatClientOptions, + type InferChatMessages, } from "@tanstack/ai-client"; import { toolDefinition } from "@tanstack/ai"; import { z } from "zod"; @@ -43,22 +39,18 @@ import { useState } from "react"; const updateUIDef = toolDefinition({ name: "updateUI", description: "Update the UI with a notification", - inputSchema: z.object({ - message: z.string(), - }), + inputSchema: z.object({ message: z.string() }), outputSchema: z.object({ success: z.boolean() }), }); function ChatComponent() { const [notification, setNotification] = useState(null); - // Create client tool implementations const updateUI = updateUIDef.client((input) => { setNotification(input.message); return { success: true }; }); - // Create typed tools array (no 'as const' needed!) const tools = [updateUI]; const chatOptions = createChatClientOptions({ @@ -66,7 +58,6 @@ function ChatComponent() { tools, }); - // Fully typed messages! type ChatMessages = InferChatMessages; const { messages, sendMessage, isLoading, error, addToolApprovalResponse } = @@ -80,21 +71,14 @@ function ChatComponent() { Extends `ChatClientOptions` from `@tanstack/ai-client`: -- `connection` - Connection adapter (required) -- `tools?` - Array of client tool implementations (with `.client()` method) -- `initialMessages?` - Initial messages array -- `id?` - Unique identifier for this chat instance -- `threadId?` - Thread ID for AG-UI run correlation. Persists across sends; auto-generated if omitted -- `forwardedProps?` - Arbitrary client-controlled JSON forwarded to the server in the AG-UI `RunAgentInput.forwardedProps` field (e.g., `{ provider: 'openai', model: 'gpt-4o' }`) -- `body?` - **Deprecated.** Use `forwardedProps` instead. Still works for backward compatibility; values are merged into `forwardedProps` on the wire -- `context?` - Typed client-local runtime context passed to client tool implementations. This value is not serialized to the server -- `onResponse?` - Callback when response is received -- `onChunk?` - Callback when stream chunk is received -- `onFinish?` - Callback when response finishes -- `onError?` - Callback when error occurs -- `streamProcessor?` - Stream processing configuration - -**Note:** Client tools are now automatically executed - no `onToolCall` callback needed! +- `connection` — required adapter +- `tools?` — `.client()` implementations (auto-run; no `onToolCall`) +- `initialMessages?` / `id?` / `threadId?` — seed + AG-UI thread +- `forwardedProps?` — client JSON → server (e.g. `{ provider: 'openai', model: 'gpt-4o' }`) +- `context?` — client-local tool context (not serialized) + +Also: `onResponse?`, `onChunk?`, `onFinish?`, `onError?`, `streamProcessor?`. +`body?` is **deprecated** — use `forwardedProps` (still merged for compat). ### Returns @@ -126,9 +110,11 @@ interface UseChatReturn { } ``` -## Connection Adapters +--- + +## Connection adapters -Re-exported from `@tanstack/ai-client` for convenience: +Re-exported from `@tanstack/ai-client`: ```typescript import { @@ -143,22 +129,24 @@ import { } from "@tanstack/ai-react"; ``` -For React Native or Expo chat screens, use an absolute server URL and prefer -`xhrHttpStream()` with a server route that returns `toHttpResponse()`. Use -`xhrServerSentEvents()` with `toServerSentEventsResponse()` when you want SSE. -Use `fetchHttpStream()` only when the runtime supports streaming `fetch`, -`Response.body.getReader()`, and `TextDecoder`; otherwise it throws -`UnsupportedResponseStreamError`. +| Runtime | Prefer | Server | +| --- | --- | --- | +| Browser SSE | `fetchServerSentEvents` | `toServerSentEventsResponse()` | +| Browser NDJSON | `fetchHttpStream` | `toHttpResponse()` | +| React Native / Expo | `xhrHttpStream` (absolute URL) | `toHttpResponse()` | +| RN SSE | `xhrServerSentEvents` | `toServerSentEventsResponse()` | + +`fetchHttpStream` needs streaming `fetch` + reader + `TextDecoder`; otherwise `UnsupportedResponseStreamError`. -XHR adapter options include `headers`, `withCredentials`, `signal`, `body`, and -`xhrFactory`. Fetch adapter options include `headers`, `credentials`, `signal`, -`body`, and `fetchClient`. Both option objects may be provided directly or as a -function that resolves per request. +Options may be a static object or a per-request function. +Fetch: `headers`, `credentials`, `signal`, `body`, `fetchClient`. +XHR: `headers`, `withCredentials`, `signal`, `body`, `xhrFactory`. -For error narrowing, import `UnsupportedResponseStreamError` and -`StreamTruncatedError` from `@tanstack/ai-client`. +Narrow stream errors from `@tanstack/ai-client`: `UnsupportedResponseStreamError`, `StreamTruncatedError`. -## Example: Basic Chat +--- + +## Basic chat ```tsx import { useState } from "react"; @@ -216,7 +204,7 @@ export function Chat() { } ``` -## Example: Tool Approval +## Tool approval ```tsx import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; @@ -269,14 +257,10 @@ export function ChatWithApproval() { } ``` -## Example: Client Tools with Type Safety +## Client tools (typed) ```tsx import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; -import { - createChatClientOptions, - type InferChatMessages -} from "@tanstack/ai-client"; import { toolDefinition } from "@tanstack/ai"; import { z } from "zod"; import { useState } from "react"; @@ -302,11 +286,12 @@ const saveToStorageDef = toolDefinition({ }); export function ChatWithClientTools() { - const [notification, setNotification] = useState<{ message: string; type: string } | null>(null); + const [notification, setNotification] = useState<{ + message: string; + type: string; + } | null>(null); - // Create client implementations const updateUI = updateUIDef.client((input) => { - // ✅ input is fully typed! setNotification({ message: input.message, type: input.type }); return { success: true }; }); @@ -316,12 +301,11 @@ export function ChatWithClientTools() { return { saved: true }; }); - // Create typed tools array (no 'as const' needed!) const tools = [updateUI, saveToStorage]; const { messages, sendMessage } = useChat({ connection: fetchServerSentEvents("/api/chat"), - tools, // ✅ Automatic execution, full type safety + tools, }); return ( @@ -329,7 +313,6 @@ export function ChatWithClientTools() { {messages.map((message) => message.parts.map((part) => { if (part.type === "tool-call" && part.name === "updateUI") { - // ✅ part.input and part.output are fully typed! return
Tool executed: {part.name}
; } return null; @@ -342,17 +325,16 @@ export function ChatWithClientTools() { ## `createChatClientOptions(options)` -Helper to create typed chat options (re-exported from `@tanstack/ai-client`). +Re-export for typed options + `InferChatMessages`. ```typescript -import { - createChatClientOptions, +import { + createChatClientOptions, fetchServerSentEvents, - type InferChatMessages + type InferChatMessages, } from "@tanstack/ai-client"; import { tool1, tool2 } from "./tools"; -// Create typed tools array (no 'as const' needed!) const tools = [tool1, tool2]; const chatOptions = createChatClientOptions({ @@ -365,27 +347,12 @@ type Messages = InferChatMessages; ## Types -Re-exported from `@tanstack/ai-client`: - -- `UIMessage` - Message type with tool type parameter -- `MessagePart` - Message part with tool type parameter -- `TextPart` - Text content part -- `ThinkingPart` - Thinking content part -- `ToolCallPart` - Tool call part (discriminated union) -- `ToolResultPart` - Tool result part -- `ChatClientOptions` - Chat client options with typed client runtime context -- `ConnectionAdapter` - Connection adapter interface -- `InferChatMessages` - Extract message type from options - -Re-exported from `@tanstack/ai`: +From `@tanstack/ai-client`: `UIMessage`, `MessagePart`, `TextPart`, `ThinkingPart`, `ToolCallPart`, `ToolResultPart`, `ChatClientOptions`, `ConnectionAdapter`, `InferChatMessages`. -- `toolDefinition()` - Create isomorphic tool definition -- `ToolDefinitionInstance` - Tool definition type -- `ClientTool` - Client tool type -- `ServerTool` - Server tool type +From `@tanstack/ai`: `toolDefinition()`, `ToolDefinitionInstance`, `ClientTool`, `ServerTool`. ## Next Steps -- [Getting Started](../getting-started/quick-start) - Learn the basics -- [Tools Guide](../tools/tools) - Learn about the isomorphic tool system -- [Client Tools](../tools/client-tools) - Learn about client-side tools +- [Getting Started](../getting-started/quick-start) +- [Tools Guide](../tools/tools) +- [Client Tools](../tools/client-tools) diff --git a/docs/api/ai-solid.md b/docs/api/ai-solid.md index 46de176a9..ccf75fb63 100644 --- a/docs/api/ai-solid.md +++ b/docs/api/ai-solid.md @@ -2,7 +2,7 @@ title: "@tanstack/ai-solid" slug: /api/ai-solid order: 4 -description: "API reference for @tanstack/ai-solid — SolidJS primitives including useChat for streaming chat with full type safety." +description: "SolidJS useChat primitive, connection adapters, and typed client tools." keywords: - tanstack ai - "@tanstack/ai-solid" @@ -13,9 +13,7 @@ keywords: - api reference --- -SolidJS primitives for TanStack AI, providing convenient SolidJS bindings for the headless client. - -## Installation +If you need streaming chat in SolidJS → `useChat` + a connection adapter. ```bash npm install @tanstack/ai-solid @@ -23,13 +21,15 @@ npm install @tanstack/ai-solid ## `useChat(options?)` -Main primitive for managing chat state in SolidJS with full type safety. +1. Wire a connection. +2. Pass `.client()` tools if needed (auto-executed). +3. Read accessors with `()` — `messages()`, `isLoading()`, `error()`. ```tsx import { useChat, fetchServerSentEvents } from "@tanstack/ai-solid"; -import { - createChatClientOptions, - type InferChatMessages +import { + createChatClientOptions, + type InferChatMessages, } from "@tanstack/ai-client"; import { toolDefinition } from "@tanstack/ai"; import { z } from "zod"; @@ -43,13 +43,12 @@ const updateUIDef = toolDefinition({ function ChatComponent() { const [, setNotification] = createSignal(null); - // Create client tool implementations + const updateUI = updateUIDef.client((input) => { setNotification(input.message); return { success: true }; }); - // Create typed tools array (no 'as const' needed!) const tools = [updateUI]; const chatOptions = createChatClientOptions({ @@ -57,7 +56,6 @@ function ChatComponent() { tools, }); - // Fully typed messages! type ChatMessages = InferChatMessages; const { messages, sendMessage, isLoading, error, addToolApprovalResponse } = @@ -71,21 +69,14 @@ function ChatComponent() { Extends `ChatClientOptions` from `@tanstack/ai-client`: -- `connection` - Connection adapter (required) -- `tools?` - Array of client tool implementations (with `.client()` method) -- `initialMessages?` - Initial messages array -- `id?` - Unique identifier for this chat instance -- `threadId?` - Thread ID for AG-UI run correlation. Persists across sends; auto-generated if omitted -- `forwardedProps?` - Arbitrary client-controlled JSON forwarded to the server in the AG-UI `RunAgentInput.forwardedProps` field (e.g., `{ provider: 'openai', model: 'gpt-4o' }`) -- `body?` - **Deprecated.** Use `forwardedProps` instead. Still works for backward compatibility; values are merged into `forwardedProps` on the wire -- `context?` - Typed client-local runtime context passed to client tool implementations. This value is not serialized to the server -- `onResponse?` - Callback when response is received -- `onChunk?` - Callback when stream chunk is received -- `onFinish?` - Callback when response finishes -- `onError?` - Callback when error occurs -- `streamProcessor?` - Stream processing configuration - -**Note:** Client tools are now automatically executed - no `onToolCall` callback needed! +- `connection` — required adapter +- `tools?` — `.client()` implementations (auto-run; no `onToolCall`) +- `initialMessages?` / `id?` / `threadId?` — seed + AG-UI thread +- `forwardedProps?` — client JSON → server +- `context?` — client-local tool context (not serialized) + +Also: `onResponse?`, `onChunk?`, `onFinish?`, `onError?`, `streamProcessor?`. +`body?` is **deprecated** — use `forwardedProps`. ### Returns @@ -118,11 +109,9 @@ interface UseChatReturn { } ``` -**Note:** Unlike React, `messages`, `isLoading`, and `error` are SolidJS `Accessor` functions, so you need to call them to get their values (e.g., `messages()` instead of just `messages`). - -## Connection Adapters +--- -Re-exported from `@tanstack/ai-client` for convenience: +## Connection adapters ```typescript import { @@ -133,7 +122,9 @@ import { } from "@tanstack/ai-solid"; ``` -## Example: Basic Chat +--- + +## Basic chat ```tsx import { createSignal, For } from "solid-js"; @@ -195,10 +186,10 @@ export function Chat() { } ``` -## Example: Tool Approval +## Tool approval ```tsx -import { For, Show } from "solid-js"; +import { For } from "solid-js"; import { useChat, fetchServerSentEvents } from "@tanstack/ai-solid"; export function ChatWithApproval() { @@ -253,14 +244,10 @@ export function ChatWithApproval() { } ``` -## Example: Client Tools with Type Safety +## Client tools (typed) ```tsx import { useChat, fetchServerSentEvents } from "@tanstack/ai-solid"; -import { - createChatClientOptions, - type InferChatMessages -} from "@tanstack/ai-client"; import { toolDefinition } from "@tanstack/ai"; import { z } from "zod"; import { createSignal, For } from "solid-js"; @@ -278,11 +265,12 @@ const saveToStorageDef = toolDefinition({ }); export function ChatWithClientTools() { - const [notification, setNotification] = createSignal<{ message: string; type: string } | null>(null); + const [notification, setNotification] = createSignal<{ + message: string; + type: string; + } | null>(null); - // Create client implementations const updateUI = updateUIDef.client((input) => { - // ✅ input is fully typed! setNotification({ message: input.message, type: input.type }); return { success: true }; }); @@ -292,12 +280,11 @@ export function ChatWithClientTools() { return { saved: true }; }); - // Create typed tools array (no 'as const' needed!) const tools = [updateUI, saveToStorage]; const { messages, sendMessage } = useChat({ connection: fetchServerSentEvents("/api/chat"), - tools, // ✅ Automatic execution, full type safety + tools, }); return ( @@ -307,7 +294,6 @@ export function ChatWithClientTools() { {(part) => { if (part.type === "tool-call" && part.name === "updateUI") { - // ✅ part.input and part.output are fully typed! return
Tool executed: {part.name}
; } return null; @@ -322,17 +308,14 @@ export function ChatWithClientTools() { ## `createChatClientOptions(options)` -Helper to create typed chat options (re-exported from `@tanstack/ai-client`). - ```typescript -import { - createChatClientOptions, - type InferChatMessages +import { + createChatClientOptions, + type InferChatMessages, } from "@tanstack/ai-client"; import { fetchServerSentEvents } from "@tanstack/ai-solid"; import { tool1, tool2 } from "./tools"; -// Create typed tools array (no 'as const' needed!) const tools = [tool1, tool2]; const chatOptions = createChatClientOptions({ @@ -345,28 +328,12 @@ type Messages = InferChatMessages; ## Types -Re-exported from `@tanstack/ai-client`: - -- `UIMessage` - Message type with tool type parameter -- `MessagePart` - Message part with tool type parameter -- `TextPart` - Text content part -- `ThinkingPart` - Thinking content part -- `ToolCallPart` - Tool call part (discriminated union) -- `ToolResultPart` - Tool result part -- `ChatClientOptions` - Chat client options with typed client runtime context -- `ConnectionAdapter` - Connection adapter interface -- `InferChatMessages` - Extract message type from options -- `ChatRequestBody` - Request body type - -Re-exported from `@tanstack/ai`: +From `@tanstack/ai-client`: `UIMessage`, `MessagePart`, `TextPart`, `ThinkingPart`, `ToolCallPart`, `ToolResultPart`, `ChatClientOptions`, `ConnectionAdapter`, `InferChatMessages`, `ChatRequestBody`. -- `toolDefinition()` - Create isomorphic tool definition -- `ToolDefinitionInstance` - Tool definition type -- `ClientTool` - Client tool type -- `ServerTool` - Server tool type +From `@tanstack/ai`: `toolDefinition()`, `ToolDefinitionInstance`, `ClientTool`, `ServerTool`. ## Next Steps -- [Getting Started](../getting-started/quick-start) - Learn the basics -- [Tools Guide](../tools/tools) - Learn about the isomorphic tool system -- [Client Tools](../tools/client-tools) - Learn about client-side tools +- [Getting Started](../getting-started/quick-start) +- [Tools Guide](../tools/tools) +- [Client Tools](../tools/client-tools) diff --git a/docs/api/ai-svelte.md b/docs/api/ai-svelte.md index 179221594..c9ef9a763 100644 --- a/docs/api/ai-svelte.md +++ b/docs/api/ai-svelte.md @@ -2,7 +2,7 @@ title: "@tanstack/ai-svelte" id: ai-svelte order: 6 -description: "API reference for @tanstack/ai-svelte — Svelte 5 reactive factory functions for streaming chat built on runes." +description: "Svelte 5 createChat factory, generation helpers, and typed client tools." keywords: - tanstack ai - "@tanstack/ai-svelte" @@ -13,17 +13,23 @@ keywords: - api reference --- -Svelte 5 bindings for TanStack AI, providing reactive factory functions for the headless client using Svelte runes. - -## Installation +If you need streaming chat in Svelte 5 → `createChat` (factory, not a hook). ```bash npm install @tanstack/ai-svelte ``` -## `createChat(options)` +## Svelte-specific rules -Factory function for managing chat state in Svelte 5 with full type safety. +1. Call `createChat` in ` @@ -278,13 +266,13 @@ import { ``` -## Generation Functions +--- -Factory functions for one-shot generation tasks (images, speech, transcription, summarization, video). All share the same pattern: provide a `connection` or `fetcher`, call `generate()`, and read reactive state. +## Generation functions -### `createGeneration(options)` +Provide `connection` or `fetcher`, call `generate()`, read reactive getters. **No auto-cleanup** — call `.stop()` when done. -Base factory for custom generation types. All specialized functions below are built on this. +### `createGeneration(options)` ```typescript import { createGeneration, fetchServerSentEvents } from "@tanstack/ai-svelte"; @@ -299,34 +287,22 @@ const gen = createGeneration({ **Options:** `connection?`, `fetcher?`, `id?`, `body?`, `onResult?`, `onError?`, `onProgress?`, `onChunk?` -**Returns:** `generate`, `result`, `isLoading`, `error`, `status`, `stop`, `reset`, `runId`, `updateBody` -- all state properties are reactive getters. - -### `createGenerateImage(options)` - -Image generation factory. `generate()` accepts `ImageGenerateInput`, result is `ImageGenerationResult`. - -### `createGenerateSpeech(options)` - -Text-to-speech factory. `generate()` accepts `SpeechGenerateInput`, result is `TTSResult`. +**Returns:** `generate`, `result`, `isLoading`, `error`, `status`, `stop`, `reset`, `runId`, `updateBody`. -### `createTranscription(options)` +### Specialized -Audio transcription factory. `generate()` accepts `TranscriptionGenerateInput`, result is `TranscriptionResult`. +| Factory | Input | Result notes | +| --- | --- | --- | +| `createGenerateImage` | `ImageGenerateInput` | `ImageGenerationResult` | +| `createGenerateSpeech` | `SpeechGenerateInput` | `TTSResult` | +| `createTranscription` | `TranscriptionGenerateInput` | `TranscriptionResult` | +| `createSummarize` | `SummarizeGenerateInput` | `SummarizationResult` | +| `createGenerateVideo` | video input | + `jobId`, `videoStatus`; `onJobCreated?`, `onStatusUpdate?` | -### `createSummarize(options)` - -Text summarization factory. `generate()` accepts `SummarizeGenerateInput`, result is `SummarizationResult`. - -### `createGenerateVideo(options)` - -Video generation factory with job polling. Returns additional `jobId` and `videoStatus` reactive getters. Accepts extra `onJobCreated?` and `onStatusUpdate?` callbacks. - -No generation function includes automatic cleanup. Call `.stop()` manually when done. +--- ## `createChatClientOptions(options)` -Helper to create typed chat options (re-exported from `@tanstack/ai-client`). - ```typescript import { createChatClientOptions, @@ -335,7 +311,6 @@ import { } from "@tanstack/ai-client"; import { tool1, tool2 } from "./tools"; -// Create typed tools array (no 'as const' needed!) const tools = [tool1, tool2]; const chatOptions = createChatClientOptions({ @@ -348,36 +323,12 @@ type Messages = InferChatMessages; ## Types -Re-exported from `@tanstack/ai-client`: - -- `UIMessage` - Message type with tool type parameter -- `MessagePart` - Message part with tool type parameter -- `TextPart` - Text content part -- `ThinkingPart` - Thinking content part -- `ToolCallPart` - Tool call part (discriminated union) -- `ToolResultPart` - Tool result part -- `ChatClientOptions` - Chat client options with typed client runtime context -- `ConnectionAdapter` - Connection adapter interface -- `InferChatMessages` - Extract message type from options -- `ChatRequestBody` - Request body type -- `GenerationClientState` - Generation lifecycle state -- `ImageGenerateInput` - Image generation input type -- `SpeechGenerateInput` - Speech generation input type -- `TranscriptionGenerateInput` - Transcription input type -- `SummarizeGenerateInput` - Summarization input type -- `VideoGenerateInput` - Video generation input type -- `VideoGenerateResult` - Video generation result type -- `VideoStatusInfo` - Video job status info - -Re-exported from `@tanstack/ai`: - -- `toolDefinition()` - Create isomorphic tool definition -- `ToolDefinitionInstance` - Tool definition type -- `ClientTool` - Client tool type -- `ServerTool` - Server tool type +From `@tanstack/ai-client`: `UIMessage`, `MessagePart`, `TextPart`, `ThinkingPart`, `ToolCallPart`, `ToolResultPart`, `ChatClientOptions`, `ConnectionAdapter`, `InferChatMessages`, `ChatRequestBody`, generation types (`GenerationClientState`, `ImageGenerateInput`, `SpeechGenerateInput`, `TranscriptionGenerateInput`, `SummarizeGenerateInput`, `VideoGenerateInput`, `VideoGenerateResult`, `VideoStatusInfo`). + +From `@tanstack/ai`: `toolDefinition()`, `ToolDefinitionInstance`, `ClientTool`, `ServerTool`. ## Next Steps -- [Getting Started](../getting-started/quick-start) - Learn the basics -- [Tools Guide](../tools/tools) - Learn about the isomorphic tool system -- [Client Tools](../tools/client-tools) - Learn about client-side tools +- [Getting Started](../getting-started/quick-start) +- [Tools Guide](../tools/tools) +- [Client Tools](../tools/client-tools) diff --git a/docs/api/ai-vue.md b/docs/api/ai-vue.md index 7358000cb..b524d5ffb 100644 --- a/docs/api/ai-vue.md +++ b/docs/api/ai-vue.md @@ -2,7 +2,7 @@ title: "@tanstack/ai-vue" id: ai-vue order: 5 -description: "API reference for @tanstack/ai-vue — Vue 3 composables including useChat for streaming chat with full type safety." +description: "Vue 3 useChat composable, generation helpers, and typed client tools." keywords: - tanstack ai - "@tanstack/ai-vue" @@ -13,9 +13,7 @@ keywords: - api reference --- -Vue composables for TanStack AI, providing convenient Vue 3 bindings for the headless client. - -## Installation +If you need streaming chat in Vue 3 → `useChat` + a connection adapter. ```bash npm install @tanstack/ai-vue @@ -23,7 +21,9 @@ npm install @tanstack/ai-vue ## `useChat(options?)` -Main composable for managing chat state in Vue with full type safety. +1. Call inside ` @@ -296,13 +281,15 @@ const { messages, sendMessage } = useChat({ ``` -## Generation Composables +--- + +## Generation composables -Vue composables for one-shot generation tasks (images, speech, transcription, summarization, video). All share the same pattern: provide a `connection` or `fetcher`, call `generate()`, and read reactive state. +One-shot tasks: provide `connection` or `fetcher`, call `generate()`, read reactive state. Cleanup via `onScopeDispose`. ### `useGeneration(options)` -Base composable for custom generation types. All specialized composables below are built on this. +Base composable; specialized ones below wrap it. ```typescript import { useGeneration } from "@tanstack/ai-vue"; @@ -316,34 +303,22 @@ const { generate, result, isLoading, error, status, stop, reset } = **Options:** `connection?`, `fetcher?`, `id?`, `body?`, `onResult?`, `onError?`, `onProgress?`, `onChunk?` -**Returns:** `generate`, `result`, `isLoading`, `error`, `status`, `stop`, `reset`, `runId` -- all reactive state is `DeepReadonly>`. - -### `useGenerateImage(options)` - -Image generation composable. `generate()` accepts `ImageGenerateInput`, result is `ImageGenerationResult`. - -### `useGenerateSpeech(options)` - -Text-to-speech composable. `generate()` accepts `SpeechGenerateInput`, result is `TTSResult`. - -### `useTranscription(options)` +**Returns:** `generate`, `result`, `isLoading`, `error`, `status`, `stop`, `reset`, `runId` (state is `DeepReadonly>`). -Audio transcription composable. `generate()` accepts `TranscriptionGenerateInput`, result is `TranscriptionResult`. +### Specialized -### `useSummarize(options)` +| Composable | Input | Result notes | +| --- | --- | --- | +| `useGenerateImage` | `ImageGenerateInput` | `ImageGenerationResult` | +| `useGenerateSpeech` | `SpeechGenerateInput` | `TTSResult` | +| `useTranscription` | `TranscriptionGenerateInput` | `TranscriptionResult` | +| `useSummarize` | `SummarizeGenerateInput` | `SummarizationResult` | +| `useGenerateVideo` | video input | + `jobId`, `videoStatus`; `onJobCreated?`, `onStatusUpdate?` | -Text summarization composable. `generate()` accepts `SummarizeGenerateInput`, result is `SummarizationResult`. - -### `useGenerateVideo(options)` - -Video generation composable with job polling. Returns additional `jobId` and `videoStatus` refs. Accepts extra `onJobCreated?` and `onStatusUpdate?` callbacks. - -All generation composables automatically clean up via `onScopeDispose`. +--- ## `createChatClientOptions(options)` -Helper to create typed chat options (re-exported from `@tanstack/ai-client`). - ```typescript import { createChatClientOptions, @@ -352,7 +327,6 @@ import { import { fetchServerSentEvents } from "@tanstack/ai-vue"; import { tool1, tool2 } from "./tools"; -// Create typed tools array (no 'as const' needed!) const tools = [tool1, tool2]; const chatOptions = createChatClientOptions({ @@ -365,36 +339,12 @@ type Messages = InferChatMessages; ## Types -Re-exported from `@tanstack/ai-client`: - -- `UIMessage` - Message type with tool type parameter -- `MessagePart` - Message part with tool type parameter -- `TextPart` - Text content part -- `ThinkingPart` - Thinking content part -- `ToolCallPart` - Tool call part (discriminated union) -- `ToolResultPart` - Tool result part -- `ChatClientOptions` - Chat client options with typed client runtime context -- `ConnectionAdapter` - Connection adapter interface -- `InferChatMessages` - Extract message type from options -- `ChatRequestBody` - Request body type -- `GenerationClientState` - Generation lifecycle state -- `ImageGenerateInput` - Image generation input type -- `SpeechGenerateInput` - Speech generation input type -- `TranscriptionGenerateInput` - Transcription input type -- `SummarizeGenerateInput` - Summarization input type -- `VideoGenerateInput` - Video generation input type -- `VideoGenerateResult` - Video generation result type -- `VideoStatusInfo` - Video job status info - -Re-exported from `@tanstack/ai`: - -- `toolDefinition()` - Create isomorphic tool definition -- `ToolDefinitionInstance` - Tool definition type -- `ClientTool` - Client tool type -- `ServerTool` - Server tool type +From `@tanstack/ai-client`: `UIMessage`, `MessagePart`, `TextPart`, `ThinkingPart`, `ToolCallPart`, `ToolResultPart`, `ChatClientOptions`, `ConnectionAdapter`, `InferChatMessages`, `ChatRequestBody`, generation input/result types (`ImageGenerateInput`, `SpeechGenerateInput`, `TranscriptionGenerateInput`, `SummarizeGenerateInput`, `VideoGenerateInput`, `VideoGenerateResult`, `VideoStatusInfo`, `GenerationClientState`). + +From `@tanstack/ai`: `toolDefinition()`, `ToolDefinitionInstance`, `ClientTool`, `ServerTool`. ## Next Steps -- [Getting Started](../getting-started/quick-start) - Learn the basics -- [Tools Guide](../tools/tools) - Learn about the isomorphic tool system -- [Client Tools](../tools/client-tools) - Learn about client-side tools +- [Getting Started](../getting-started/quick-start) +- [Tools Guide](../tools/tools) +- [Client Tools](../tools/client-tools) diff --git a/docs/api/ai.md b/docs/api/ai.md index c70aeb861..2c768b512 100644 --- a/docs/api/ai.md +++ b/docs/api/ai.md @@ -2,7 +2,7 @@ title: "@tanstack/ai" id: tanstack-ai-api order: 1 -description: "API reference for @tanstack/ai — the core TanStack AI library providing chat(), generateImage(), toolDefinition(), and streaming utilities." +description: "Core API: chat(), toolDefinition(), summarize(), SSE helpers, agent strategies." keywords: - tanstack ai - "@tanstack/ai" @@ -13,9 +13,7 @@ keywords: - core library --- -The core AI library for TanStack AI. - -## Installation +If you need server-side chat, tools, or SSE responses → install and call the APIs below. ```bash npm install @tanstack/ai @@ -23,7 +21,7 @@ npm install @tanstack/ai ## `chat(options)` -Creates a streaming chat response. +Stream (or await) a chat completion. ```typescript import { chat, maxIterations } from "@tanstack/ai"; @@ -39,56 +37,32 @@ const stream = chat({ }); ``` -### Parameters - -- `adapter` - An AI adapter instance with model (e.g., `openaiText('gpt-5.2')`, `anthropicText('claude-sonnet-4-5')`) -- `messages` - Array of chat messages. Accepts mixed `UIMessage | ModelMessage` arrays — internal conversion handles AG-UI fan-out dedup, drops `reasoning`/`activity`, and collapses `developer` → `system` -- `tools?` - Array of tools for function calling -- `context?` - Typed runtime context passed to server tools and middleware. If a tool or middleware declares a concrete context type, `chat()` requires a compatible value here -- `systemPrompts?` - System prompts to prepend to messages -- `agentLoopStrategy?` - Strategy for agent loops (default: `maxIterations(5)`). Strategies receive `{ iterationCount, finishReason, messages, toolCallCount, lastTurnToolCallCount }` and run between model turns. Iterations are model turns, not tool calls — for tool-call budgets use middleware (`onBeforeToolCall` + `onShouldContinue`); see [Tool-call budgets](../chat/agentic-cycle#tool-call-budgets-middleware-recipe). -- `middleware?` - Array of chat middleware. Use `onShouldContinue` / `onBeforeToolCall` for app-owned tool budgets. -- `abortController?` - AbortController for cancellation -- `modelOptions?` - Provider-native model options. This is where sampling parameters live — `temperature`, `top_p`/`topP`, and the provider's token-limit key (`max_output_tokens`, `max_tokens`, `maxOutputTokens`, …) — under each provider's canonical name, rather than as generic root-level props. See [Moving Sampling Options into modelOptions](../migration/sampling-options-to-model-options). (Renamed from `providerOptions`.) -- `threadId?` - AG-UI thread identifier propagated into `RUN_STARTED` events for run correlation -- `runId?` - AG-UI run identifier (auto-generated if omitted) -- `parentRunId?` - AG-UI parent run identifier for nested runs - -### Returns +### Required -An async iterable of `StreamChunk`. +- `adapter` — adapter + model (e.g. `openaiText('gpt-5.2')`, `anthropicText('claude-sonnet-4-5')`) +- `messages` — `UIMessage | ModelMessage[]` (mixed OK; converts AG-UI fan-out, drops `reasoning`/`activity`, maps `developer` → `system`) -## `summarize(options)` - -Creates a text summarization. +### Common options -```typescript -import { summarize } from "@tanstack/ai"; -import { openaiSummarize } from "@tanstack/ai-openai"; - -const result = await summarize({ - adapter: openaiSummarize("gpt-5.2"), - text: "Long text to summarize...", - maxLength: 100, - style: "concise", -}); -``` +- `tools?` — server/client tools for function calling +- `context?` — typed runtime context for tools/middleware (required if those declare a concrete type) +- `systemPrompts?` — prepended system prompts +- `agentLoopStrategy?` — model-turn limit (default `maxIterations(5)`). Turns ≠ tool calls; use middleware for tool budgets ([recipe](../chat/agentic-cycle#tool-call-budgets-middleware-recipe)) +- `middleware?` — chat middleware (`onShouldContinue`, `onBeforeToolCall`, …) -### Parameters +### More options -- `adapter` - An AI adapter instance with model -- `text` - Text to summarize -- `maxLength?` - Maximum length of summary -- `style?` - Summary style ("concise" | "detailed") -- `modelOptions?` - Model-specific options +- `modelOptions?` — sampling & provider limits (`temperature`, `top_p`/`topP`, `max_tokens`, …) under provider names. See [modelOptions migration](../migration/sampling-options-to-model-options) +- `abortController?` — cancel the run +- `threadId?` / `runId?` / `parentRunId?` — AG-UI run correlation (`runId` auto-generated if omitted) -### Returns +**Returns:** async iterable of `StreamChunk`. With `stream: false`, a one-shot result; with `outputSchema`, a parsed structured result. -A `SummarizationResult` with the summary text. +--- ## `toolDefinition(config)` -Creates an isomorphic tool definition that can be instantiated for server or client execution. +Define once; attach `.server()` or `.client()` implementations. ```typescript import { chat, toolDefinition } from "@tanstack/ai"; @@ -98,35 +72,19 @@ import { z } from "zod"; const myToolDef = toolDefinition({ name: "my_tool", description: "Tool description", - inputSchema: z.object({ - param: z.string(), - }), - outputSchema: z.object({ - result: z.string(), - }), - needsApproval: false, // Optional + inputSchema: z.object({ param: z.string() }), + outputSchema: z.object({ result: z.string() }), + needsApproval: false, }); -// Or create client implementation -const myClientTool = myToolDef.client(async ({ param }) => { - // Client-side implementation +const myServerTool = myToolDef.server(async ({ param }) => { return { result: "..." }; }); -// Use directly in chat() (server-side, no execute) -chat({ - adapter: openaiText("gpt-5.2"), - tools: [myToolDef], - messages: [{ role: "user", content: "..." }], -}); - -// Or create server implementation -const myServerTool = myToolDef.server(async ({ param }) => { - // Server-side implementation +const myClientTool = myToolDef.client(async ({ param }) => { return { result: "..." }; }); -// Use directly in chat() (server-side, no execute) chat({ adapter: openaiText("gpt-5.2"), tools: [myServerTool], @@ -134,7 +92,7 @@ chat({ }); ``` -Tools can declare typed runtime context for request-scoped dependencies: +### Typed context for server tools ```typescript import { chat, toolDefinition, toServerSentEventsResponse } from "@tanstack/ai"; @@ -165,22 +123,44 @@ export async function POST(request: Request) { } ``` -### Parameters +### Config + +- `name` — unique tool name +- `description` — for the model +- `inputSchema` — Zod (or schema) for inputs +- `outputSchema?` — Zod for outputs +- `needsApproval?` / `metadata?` — approval gate and extra metadata + +**Returns:** `ToolDefinition` with `.server()` / `.client()`. + +--- + +## `summarize(options)` + +```typescript +import { summarize } from "@tanstack/ai"; +import { openaiSummarize } from "@tanstack/ai-openai"; + +const result = await summarize({ + adapter: openaiSummarize("gpt-5.2"), + text: "Long text to summarize...", + maxLength: 100, + style: "concise", +}); +``` -- `name` - Tool name (must be unique) -- `description` - Tool description for the model -- `inputSchema` - Zod schema for input validation -- `outputSchema?` - Zod schema for output validation -- `needsApproval?` - Whether tool requires user approval -- `metadata?` - Additional metadata +- `adapter` — summarize adapter + model +- `text` — source text +- `maxLength?` / `style?` — `"concise" | "detailed"` +- `modelOptions?` — model-specific options -### Returns +**Returns:** `SummarizationResult`. -A `ToolDefinition` object with `.server()` and `.client()` methods for creating concrete implementations. +--- -## `toServerSentEventsStream(stream, abortController?)` +## SSE / HTTP helpers -Converts a stream to a ReadableStream in Server-Sent Events format. +### `toServerSentEventsStream(stream, abortController?)` ```typescript import { chat, toServerSentEventsStream } from "@tanstack/ai"; @@ -193,21 +173,9 @@ const stream = chat({ const readableStream = toServerSentEventsStream(stream); ``` -### Parameters - -- `stream` - Async iterable of `StreamChunk` -- `abortController?` - Optional AbortController to abort when stream is cancelled - -### Returns - -A `ReadableStream` in Server-Sent Events format. Each chunk is: -- Prefixed with `"data: "` -- Followed by `"\n\n"` -- Stream ends with `"data: [DONE]\n\n"` - -## `toServerSentEventsResponse(stream, init?)` +SSE format: `data: …\n\n`, ends with `data: [DONE]\n\n`. -Converts a stream to an HTTP Response with proper SSE headers. +### `toServerSentEventsResponse(stream, init?)` ```typescript import { chat, toServerSentEventsResponse } from "@tanstack/ai"; @@ -222,18 +190,15 @@ async function POST() { } ``` -### Parameters +Sets SSE headers (`Content-Type: text/event-stream`, etc.). `init?` may include `abortController`. -- `stream` - Async iterable of `StreamChunk` -- `init?` - Optional ResponseInit options (including `abortController`) - -### Returns +--- -A `Response` object suitable for HTTP endpoints with SSE headers (`Content-Type: text/event-stream`, `Cache-Control: no-cache`, `Connection: keep-alive`). +## Request parsing -## `chatParamsFromRequest(req)` +### `chatParamsFromRequest(req)` -Reads an HTTP `Request`, parses its JSON body, and validates it against AG-UI `RunAgentInputSchema`. Returns parsed chat parameters ready to spread into `chat()`. On a malformed body, **throws a 400 `Response`** that frameworks like TanStack Start, SolidStart, Remix, and React Router 7 return to the client automatically. +Parse + validate AG-UI `RunAgentInput` from a `Request`. Malformed body → throws a **400 `Response`** (auto-handled by TanStack Start, SolidStart, Remix, React Router 7). ```typescript import { chat, chatParamsFromRequest, toServerSentEventsResponse } from "@tanstack/ai"; @@ -251,23 +216,16 @@ export async function POST(req: Request) { } ``` -### Parameters - -- `req` - An incoming `Request` whose JSON body conforms to AG-UI `RunAgentInput` - -### Returns - -A promise resolving to `{ messages, threadId, runId, parentRunId?, tools, forwardedProps, state, aguiContext, context }`. +**Returns:** `{ messages, threadId, runId, parentRunId?, tools, forwardedProps, state, aguiContext, context }`. -The returned `aguiContext` is the AG-UI protocol `RunAgentInput.context` field. It is not the same as TanStack AI runtime `chat({ context })`; validate and map it explicitly if you want those values available to tools or middleware. +- Prefer `aguiContext` (AG-UI protocol context). Map it yourself into `chat({ context })` if tools need it. +- `context` is a deprecated alias of `aguiContext`. -The returned `context` field is a deprecated alias of `aguiContext` kept for backward compatibility. Prefer `aguiContext` in new code. +> **Next.js / SvelteKit / Hono / Node:** no auto-throw handling. Use try/catch or `chatParamsFromRequestBody(await req.json())`. -> **Framework note.** Next.js Route Handlers, SvelteKit, Hono, and raw Node do not auto-handle thrown `Response` objects. In those, wrap with try/catch or use `chatParamsFromRequestBody(await req.json())` directly. +### `chatParamsFromRequestBody(body)` -## `chatParamsFromRequestBody(body)` - -Lower-level variant of `chatParamsFromRequest` that validates an already-parsed body. Rejects with an `AGUIError` on malformed input. Use this when you need explicit error handling control. +Same validation on an already-parsed body. Rejects with `AGUIError`. ```typescript import { chatParamsFromRequestBody } from "@tanstack/ai"; @@ -285,9 +243,9 @@ async function handler(req: Request): Promise { } ``` -## `mergeAgentTools(serverTools, clientTools)` +### `mergeAgentTools(serverTools, clientTools)` -Merges a server-side tool registry with the AG-UI client-declared tools received in the request payload. Server tools win on name collision; client-only tools become no-execute stubs that the runtime dispatches via `ClientToolRequest` events. +Server tools win on name collision. Client-only tools become no-execute stubs (`ClientToolRequest` events). ```typescript import { chat, chatParamsFromRequest, mergeAgentTools } from "@tanstack/ai"; @@ -304,18 +262,11 @@ async function handler(req: Request) { } ``` -### Parameters - -- `serverTools` - The server's `toolDefinition().server(...)` registry, keyed by tool name -- `clientTools` - The `tools` array from `chatParamsFromRequest`'s return value - -### Returns - -A merged tool record suitable for `chat({ tools })`. +--- ## `maxIterations(count)` -Creates an agent loop strategy that limits **model turns** (iterations), not tool calls. One turn can still emit many parallel tool calls — use middleware for tool-call budgets ([recipe](../chat/agentic-cycle#tool-call-budgets-middleware-recipe)). +Limit **model turns**, not tool calls. One turn can still fire many parallel tools — use middleware for tool budgets ([recipe](../chat/agentic-cycle#tool-call-budgets-middleware-recipe)). ```typescript import { chat, maxIterations } from "@tanstack/ai"; @@ -328,15 +279,9 @@ const stream = chat({ }); ``` -### Parameters - -- `count` - Maximum number of model turns - -### Returns - -An `AgentLoopStrategy` function. +--- -## Types +## Types (brief) ### `ModelMessage` @@ -358,27 +303,11 @@ type StreamChunk = | ToolResultStreamChunk | DoneStreamChunk | ErrorStreamChunk; - -interface ThinkingStreamChunk { - type: "thinking"; - id: string; - model: string; - timestamp: number; - delta?: string; // Incremental thinking token - content: string; // Accumulated thinking content -} ``` -Stream chunks represent different types of data in the stream: +Chunk kinds: content, thinking (reasoning models), tool call, tool result, done, error. -- **Content chunks** - Text content being generated -- **Thinking chunks** - Model's reasoning process (when supported by the model) -- **Tool call chunks** - When the model calls a tool -- **Tool result chunks** - Results from tool execution -- **Done chunks** - Stream completion -- **Error chunks** - Stream errors - -### `Tool` +### `Tool` / `ToolExecutionContext` ```typescript import type { SchemaInput, ToolExecutionContext } from "@tanstack/ai"; @@ -398,75 +327,15 @@ interface Tool { } ``` -### `ToolExecutionContext` - -```typescript ignore -type ToolExecutionContext = { - toolCallId?: string; - emitCustomEvent: (eventName: string, value: Record) => void; -} & (unknown extends TContext ? { context?: TContext } : { context: TContext }); -``` - -`context` is the runtime value from `chat({ context })` for server tools, or from `ChatClient` / framework hook options for client tools. It is required when a tool declares a concrete `TContext` and optional for untyped tools where the context type is `unknown`. +`context` comes from `chat({ context })` (server) or client options (client). Required when `TContext` is concrete. -### `ChatMiddleware` +### `ChatMiddleware` -```typescript -import type { - StreamChunk, - ChatMiddlewarePhase, - ToolCallHookContext, - BeforeToolCallDecision, - AfterToolCallInfo, - FinishInfo, - AbortInfo, - ErrorInfo, -} from "@tanstack/ai"; - -interface ChatMiddlewareContext { - requestId: string; - streamId: string; - threadId: string; - phase: ChatMiddlewarePhase; - iteration: number; - context: TContext; - abort(reason?: string): void; - defer(promise: Promise): void; -} - -interface ChatMiddleware { - name?: string; - onStart?: (ctx: ChatMiddlewareContext) => void | Promise; - onChunk?: ( - ctx: ChatMiddlewareContext, - chunk: StreamChunk - ) => void | StreamChunk | StreamChunk[] | null | Promise; - onBeforeToolCall?: ( - ctx: ChatMiddlewareContext, - hookCtx: ToolCallHookContext - ) => BeforeToolCallDecision | Promise; - onAfterToolCall?: ( - ctx: ChatMiddlewareContext, - info: AfterToolCallInfo - ) => void | Promise; - onFinish?: ( - ctx: ChatMiddlewareContext, - info: FinishInfo - ) => void | Promise; - onAbort?: ( - ctx: ChatMiddlewareContext, - info: AbortInfo - ) => void | Promise; - onError?: ( - ctx: ChatMiddlewareContext, - info: ErrorInfo - ) => void | Promise; -} -``` +Hooks: `onStart`, `onChunk`, `onBeforeToolCall`, `onAfterToolCall`, `onFinish`, `onAbort`, `onError`. See [Runtime Context](../advanced/runtime-context). -See [Runtime Context](../advanced/runtime-context) for the recommended context patterns. +--- -## Usage Examples +## Patterns ```typescript import { chat, summarize, generateImage, toolDefinition } from "@tanstack/ai"; @@ -477,47 +346,42 @@ import { } from "@tanstack/ai-openai"; import { z } from "zod"; -// --- Streaming chat -const stream = chat({ - adapter: openaiText("gpt-5.2"), - messages: [{ role: "user", content: "Hello!" }], -}); - -// --- Structured response with tools const weatherTool = toolDefinition({ name: "getWeather", description: "Get the current weather for a city", - inputSchema: z.object({ - city: z.string(), - }), + inputSchema: z.object({ city: z.string() }), }).server(async ({ city }) => { - // Implementation that fetches weather info return JSON.stringify({ temperature: 72, condition: "Sunny" }); }); async function examples() { - // --- One-shot chat response (stream: false) + // One-shot (no stream) const response = await chat({ adapter: openaiText("gpt-5.2"), messages: [{ role: "user", content: "What's the capital of France?" }], - stream: false, // Returns a Promise instead of AsyncIterable + stream: false, }); - // --- Structured response with outputSchema + // Structured output const parsed = await chat({ adapter: openaiText("gpt-5.2"), - messages: [{ role: "user", content: "Summarize this text in JSON with keys 'summary' and 'keywords': ... " }], + messages: [ + { + role: "user", + content: + "Summarize this text in JSON with keys 'summary' and 'keywords': ... ", + }, + ], outputSchema: z.object({ summary: z.string(), keywords: z.array(z.string()), }), }); - const toolResult = await chat({ + // Tools + structured output + await chat({ adapter: openaiText("gpt-5.2"), - messages: [ - { role: "user", content: "What's the weather in Paris?" } - ], + messages: [{ role: "user", content: "What's the weather in Paris?" }], tools: [weatherTool], outputSchema: z.object({ answer: z.string(), @@ -528,15 +392,13 @@ async function examples() { }), }); - // --- Summarization - const summary = await summarize({ + await summarize({ adapter: openaiSummarize("gpt-5.2"), text: "Long text to summarize...", maxLength: 100, }); - // --- Image generation - const image = await generateImage({ + await generateImage({ adapter: openaiImage("dall-e-3"), prompt: "A futuristic city skyline at sunset", numberOfImages: 1, @@ -547,6 +409,6 @@ async function examples() { ## Next Steps -- [Getting Started](../getting-started/quick-start) - Learn the basics -- [Tools Guide](../tools/tools) - Learn about tools -- [Adapters](../adapters/openai) - Explore adapter options +- [Getting Started](../getting-started/quick-start) +- [Tools Guide](../tools/tools) +- [Adapters](../adapters/openai) diff --git a/docs/architecture/approval-flow-processing.md b/docs/architecture/approval-flow-processing.md index 44f74ccfb..ee2061679 100644 --- a/docs/architecture/approval-flow-processing.md +++ b/docs/architecture/approval-flow-processing.md @@ -1,7 +1,7 @@ --- title: Approval Flow Processing Architecture id: approval-flow-processing -description: "Internal architecture of TanStack AI tool approvals, interrupts, persistence, and typed client resume handling." +description: "Internal interrupt → validate → resume pipeline for tool approvals (ephemeral by default)." keywords: - tanstack ai - approval flow @@ -12,8 +12,7 @@ keywords: # Approval Flow Processing Architecture -Tool approval is an interrupt-and-resume protocol. A run that needs user input -ends with one canonical event: +If a tool needs user input → the run ends with one interrupt terminal. Public guide: [Interrupts](../interrupts/overview). Deprecated readers: [Migrate to AG-UI interrupts](../interrupts/migration). ```ts const interruptTerminal = { @@ -40,57 +39,32 @@ const interruptTerminal = { } ``` -The canonical event stream is the only native approval event stream. It works -ephemerally without persistence. When server state persistence is configured, -it stores the complete descriptor/binding batch before the terminal is exposed; -SSE delivery durability separately assigns opaque resume offsets to delivered -events. Native paths do not emit `approval-requested` or -`tool-input-available` custom events. +Canonical event stream only. Works ephemerally without persistence. With server state persistence, the descriptor/binding batch is stored before the terminal is exposed. Native paths do **not** emit `approval-requested` or `tool-input-available` custom events. -See [Interrupts](../interrupts/overview) for the public server/client guide and -[Migrate to AG-UI interrupts](../interrupts/migration) for deprecated readers. - -## Responsibilities +## Layer responsibilities | Layer | Responsibility | | --- | --- | -| Tool definition | Declares `needsApproval: true` for a sensitive operation. | -| Chat engine | Stops before tool execution and emits the interrupt outcome. | -| Chat client | Binds descriptors to typed methods, stages drafts, and submits one exact resume batch. | -| Application UI | Explains the operation and uses `resolveInterrupt`, `cancel`, or root batch controls. | -| Delivery adapter | Optionally replays SSE events by opaque adapter-owned offsets. | - -## Descriptor to continuation pipeline - -The invariant is **descriptor → validate all → continuation → history**: - -1. The engine builds public descriptors and bindings. Output includes - `MESSAGES_SNAPSHOT`, optional `STATE_SNAPSHOT`, and the interrupt - `RUN_FINISHED` terminal. -2. The client binds only descriptors whose reason, tool identity, call ID, - schema hashes, interrupted run, and generation match its tool registry. - Anything untrusted degrades to `generic` rather than gaining a typed tool - resolver. -3. Item methods validate and stage local drafts. The submit boundary contains - every pending interrupt ID exactly once. -4. The client submits a fresh run with the full current message history, the - interrupted `parentRunId`, and the complete resume batch. -5. The server validates **all** payloads, edited inputs, outputs, hashes, and - correlation before executing anything, reconstructing the expected batch - from the client-provided history and its current tool definitions. -6. Resumed tool calls emit results only; they do not replay synthetic tool-call - start/argument events. Successful history belongs to the continuation run. - -Because the batch is rebuilt from client-provided history, ephemeral mode does -not provide replay, exactly-once, restart, or cross-instance guarantees; the -message history is validated but remains client-provided input. - -## Server setup - -Define the tool normally. The following route is the **ephemeral** flow (no -persistence middleware): `chat` + `chatParamsFromRequest` resume the interrupt -batch from client message history and tool definitions. Durable recovery is a -separate optional layer and is not required for tool approvals. +| Tool definition | `needsApproval: true` | +| Chat engine | Stop before execute; emit interrupt outcome | +| Chat client | Bind descriptors, stage drafts, submit one resume batch | +| Application UI | `resolveInterrupt` / `cancel` / root batch controls | +| Delivery adapter | Optional SSE replay by opaque offsets | + +## Pipeline: descriptor → validate all → continuation → history + +1. Engine builds descriptors/bindings → `MESSAGES_SNAPSHOT`, optional `STATE_SNAPSHOT`, interrupt `RUN_FINISHED` +2. Client binds only descriptors that match reason, tool identity, call ID, schema hashes, interrupted run, generation — untrusted → `generic` +3. Item methods validate and stage drafts; submit includes every pending interrupt ID exactly once +4. Client starts a fresh run with full message history, interrupted `parentRunId`, full resume batch +5. Server validates **all** payloads, edits, hashes, correlation; rebuilds expected batch from client history + current tool defs +6. Resumed tools emit results only (no synthetic tool-call start/args replay) + +Ephemeral mode: no replay / exactly-once / restart / cross-instance guarantees — history is client-provided and validated. + +## Server setup (ephemeral) + +No storage required. Continuation rebuilds the paused call from browser-sent history. ```ts // tools.ts @@ -139,32 +113,20 @@ export async function POST(request: Request) { } ``` -No server storage is required to emit or resolve interrupts. The route above is -the complete flow: the browser sends the full message history back on the -continuation request, and the engine rebuilds the paused call from it. - ## Client state machine -A single approval follows this sequence: - -1. The model emits a tool call. -2. The client tool-call part reaches `approval-requested`. -3. The run ends with `RUN_FINISHED.outcome.type === 'interrupt'`. -4. `useChat` exposes a bound item in `interrupts`. -5. The UI calls `resolveInterrupt(...)` or `cancel()`; a singleton - submits immediately, while a multi-item batch waits for every valid draft. -6. The next request carries a fresh `runId`, the interrupted `parentRunId`, and - the exact AG-UI `resume` array. -7. The server validates the full set before the engine continues the tool call. +1. Model emits tool call +2. Tool-call part → `approval-requested` +3. Run ends: `RUN_FINISHED.outcome.type === 'interrupt'` +4. `useChat` exposes bound item in `interrupts` +5. UI: `resolveInterrupt(...)` or `cancel()` — singleton submits immediately; multi-item batch waits for every valid draft +6. Next request: fresh `runId`, interrupted `parentRunId`, exact AG-UI `resume` array +7. Server validates full set, then continues -Normal input is rejected at step 4. This prevents a second branch from being -created while the existing run still waits for a decision. +Normal input is rejected at step 4 while a decision is pending. ## React approval UI -Use the bound values returned by `useChat`. Rendering `interrupts` keeps IDs, -tool types, drafts, and errors connected to the hook that owns the run. - ```tsx group=approval-ui import type { ItemInterruptError } from '@tanstack/ai' import { fetchServerSentEvents, useChat } from '@tanstack/ai-react' @@ -201,7 +163,7 @@ export function ApprovalQueue() { } ``` -For a batch, stage every resolution in one synchronous root callback: +Batch — stage every resolution in one synchronous root callback: ```tsx group=approval-ui function ResolveAll({ approved }: { approved: boolean }) { @@ -233,11 +195,11 @@ function ResolveAll({ approved }: { approved: boolean }) { } ``` -## State durability versus delivery durability +## State durability vs delivery durability + +| Kind | What it does | +| --- | --- | +| State (ephemeral default) | Rebuild paused call from message history on continue | +| Delivery | Replay live byte stream after drop — configure on `toServerSentEventsResponse` (opaque SSE id per chunk; not NDJSON) | -Interrupts run ephemerally: the paused call is rebuilt from the message history -the browser replays on the continuation request. That is separate from -*delivery* durability, which makes the live byte stream replayable after a -dropped connection. Delivery durability is configured on -`toServerSentEventsResponse` and assigns one opaque SSE id per chunk (it is not -available for NDJSON). See [Resumable Streams](../resumable-streams/overview). +See [Resumable Streams](../resumable-streams/overview). diff --git a/docs/chat/agentic-cycle.md b/docs/chat/agentic-cycle.md index 2b3ddd07d..07273a482 100644 --- a/docs/chat/agentic-cycle.md +++ b/docs/chat/agentic-cycle.md @@ -2,7 +2,7 @@ title: Agentic Cycle id: agentic-cycle order: 1 -description: "The agentic cycle in TanStack AI — how the LLM loops through tool calls, results, and reasoning until it produces a final answer." +description: "How chat() loops tool calls → results → reasoning until a final answer — and how to bound the loop." keywords: - tanstack ai - agentic cycle @@ -12,9 +12,9 @@ keywords: - ai agents --- -The agentic cycle is the pattern where the LLM repeatedly calls tools, receives results, and continues reasoning until it can provide a final answer. This enables complex multi-step operations. +If the model needs tools → it calls them, gets results, and continues until it can answer. That loop is the agentic cycle. -> **Tip:** Code Mode can reduce agent loop iterations by letting the LLM write a program that calls multiple tools in a single execution. See [Code Mode](../code-mode/code-mode). +> **Tip:** Need fewer iterations for multi-tool work? Use [Code Mode](../code-mode/code-mode) so the model writes one program that calls many tools. ```mermaid graph TD @@ -41,7 +41,7 @@ graph TD style L fill:#fff4e1 ``` -### Detailed Agentic Flow +## Multi-step flow ```mermaid sequenceDiagram @@ -76,31 +76,21 @@ sequenceDiagram Client->>User: Display answer ``` -### Multi-Step Example +**Example:** "Find flights to Paris under $500 and book the cheapest" -Here's a real-world example of the agentic cycle: +1. Call `searchFlights({destination: "Paris", maxPrice: 500})` → two flights +2. Call `bookFlight({flightId: "F1"})` — may need [Tool Approval](../tools/tool-approval) +3. Final text with booking ID -**User**: "Find me flights to Paris under $500 and book the cheapest one" +The loop continues while finish reason is `tool_calls` (pending tools) **and** the agent loop strategy allows another iteration. It stops on a normal `stop` finish reason. -**Cycle 1**: LLM calls `searchFlights({destination: "Paris", maxPrice: 500})` -- Tool returns: `[{id: "F1", price: 450}, {id: "F2", price: 480}]` - -**Cycle 2**: LLM analyzes results and calls `bookFlight({flightId: "F1"})` -- Tool requires approval (sensitive operation) — see [Tool Approval](../tools/tool-approval) -- User approves -- Tool returns: `{bookingId: "B123", confirmed: true}` - -**Cycle 3**: LLM generates final response -- "I found 2 flights under $500. I've booked the cheapest one (Flight F1) for $450. Your booking ID is B123." - -### Code Example: Agentic Weather Assistant +## Wire tools into chat ```typescript import { chat, toolDefinition, toServerSentEventsResponse } from "@tanstack/ai"; import { openaiText } from "@tanstack/ai-openai"; import { z } from "zod"; -// Tool definitions const getWeatherDef = toolDefinition({ name: "get_weather", description: "Get current weather for a city", @@ -118,21 +108,18 @@ const getClothingAdviceDef = toolDefinition({ }), }); -// Server implementations const getWeather = getWeatherDef.server(async ({ city }) => { const response = await fetch(`https://api.weather.com/v1/${city}`); return await response.json(); }); const getClothingAdvice = getClothingAdviceDef.server(async ({ temperature, conditions }) => { - // Business logic for clothing recommendations if (temperature < 50) { return { recommendation: "Wear a warm jacket" }; } return { recommendation: "Light clothing is fine" }; }); -// Server route export async function POST(request: Request) { const { messages } = await request.json(); @@ -146,25 +133,25 @@ export async function POST(request: Request) { } ``` -**User**: "What should I wear in San Francisco today?" - -**Agentic Cycle**: -1. LLM calls `get_weather({city: "San Francisco"})` → Returns `{temp: 62, conditions: "cloudy"}` -2. LLM calls `get_clothing_advice({temperature: 62, conditions: "cloudy"})` → Returns `{recommendation: "Light jacket recommended"}` -3. LLM generates: "The weather in San Francisco is 62°F and cloudy. I recommend wearing a light jacket." +**User:** "What should I wear in San Francisco today?" -The loop continues only while the model's finish reason is `tool_calls` (with pending tool calls) **and** the agent loop strategy permits another iteration; it ends as soon as the model returns a normal `stop` finish reason. +1. `get_weather({city: "San Francisco"})` → `{temp: 62, conditions: "cloudy"}` +2. `get_clothing_advice({temperature: 62, conditions: "cloudy"})` → recommendation +3. Model streams the final answer -### Controlling the loop +## Bound the loop -By default the loop is bounded by `maxIterations(5)` — after five **model turns** it stops even if the model would keep calling tools. Override this with the `agentLoopStrategy` option. +Default: `maxIterations(5)` — stops after five **model turns**, even if the model would keep calling tools. -Other built-in strategies: +**Built-in strategies:** -- **`untilFinishReason([...])`** — continue until the model returns one of the given finish reasons (e.g. `untilFinishReason(["stop", "length"])`). -- **`combineStrategies([...])`** — combine multiple strategies with AND logic; the loop continues only while every strategy agrees. +| Strategy | Use when | +| --- | --- | +| `maxIterations(n)` | Cap model turns (default 5) | +| `untilFinishReason([...])` | Stop on specific finish reasons | +| `combineStrategies([...])` | AND multiple strategies | -A strategy is just a function that receives `{ iterationCount, finishReason, messages, toolCallCount, lastTurnToolCallCount }` and returns `true` to allow another iteration or `false` to stop, so you can also write your own: +A strategy receives `{ iterationCount, finishReason, messages, toolCallCount, lastTurnToolCallCount }` and returns `true` to continue or `false` to stop: ```typescript import { chat, combineStrategies, maxIterations, toServerSentEventsResponse } from "@tanstack/ai"; @@ -187,14 +174,14 @@ export async function POST(request: Request) { } ``` -### Tool-call budgets (middleware recipe) +## Cap tool calls (middleware) -> **Iterations ≠ tool calls.** One model turn can emit many parallel tool calls. `maxIterations` only bounds **model turns**. Strategies run *between* turns, so without a per-turn cap a single runaway turn can still fan out unbounded. +> **Iterations ≠ tool calls.** One model turn can emit many parallel tool calls. `maxIterations` only bounds **model turns**. -There is no built-in `maxToolCalls` strategy. Cap tools with middleware: +No built-in `maxToolCalls`. Cap with middleware: -- **`onBeforeToolCall`** — skip excess calls inside one turn (`maxPerTurn`) -- **`onShouldContinue`** — stop further turns once cumulative **emitted** tools hit a budget (`max`); skipped calls still count toward `toolCallCount` +1. **`onBeforeToolCall`** — skip excess calls inside one turn (`maxPerTurn`) +2. **`onShouldContinue`** — stop further turns once cumulative **emitted** tools hit `max` (skipped calls still count toward `toolCallCount`) ```typescript import { @@ -252,8 +239,8 @@ export async function POST(request: Request) { agentLoopStrategy: maxIterations(20), // model turns middleware: [ toolCallBudget({ - maxPerTurn: 10, // cap parallel fan-out inside one turn - max: 20, // stop further turns once cumulative emitted tools hit 20 + maxPerTurn: 10, + max: 20, }), ], }); @@ -261,4 +248,4 @@ export async function POST(request: Request) { } ``` -Place this **before** `toolCacheMiddleware` so over-budget skips win over cache hits. See [`onShouldContinue`](../advanced/middleware#onshouldcontinue) for the hook contract. +Place this **before** `toolCacheMiddleware` so over-budget skips win over cache hits. See [`onShouldContinue`](../advanced/middleware#onshouldcontinue). diff --git a/docs/chat/connection-adapters.md b/docs/chat/connection-adapters.md index 2c3009ec7..d12449daa 100644 --- a/docs/chat/connection-adapters.md +++ b/docs/chat/connection-adapters.md @@ -2,7 +2,7 @@ title: Connection Adapters id: connection-adapters order: 3 -description: "Connection adapters bridge your client and server in TanStack AI — SSE, HTTP streaming, server functions, RPC, and persistent transports like WebSockets via subscribe/send." +description: "Pick and wire how StreamChunks reach ChatClient — SSE, HTTP stream, server functions, RPC, WebSockets." keywords: - tanstack ai - connection adapters @@ -18,29 +18,27 @@ keywords: - subscribe send --- -A **connection adapter** is the piece that decides _how_ chunks get from your server to the `ChatClient` (and through it, to your framework's `useChat`). Everything else in TanStack AI — chunk processing, message reassembly, tool calls, UI updates — is transport-agnostic. The adapter is the only thing that touches the network. +If you need chunks from server → `ChatClient` / `useChat` → pick a connection adapter. Everything else (reassembly, tools, UI) is transport-agnostic. -This page covers every supported transport, when to pick which, and how to build a custom one. - -## Pick a Transport +## Pick a transport | You have… | Use | | --- | --- | -| A normal HTTP server and want the default | [`fetchServerSentEvents`](#server-sent-events-sse) | -| An environment that blocks SSE (some edge runtimes, strict proxies) | [`fetchHttpStream`](#http-streaming-ndjson) | -| React Native or Expo | [`xhrHttpStream`](#react-native-and-expo) by default, [`xhrServerSentEvents`](#react-native-and-expo) for SSE, or [`fetchHttpStream`](#http-streaming-ndjson) only when streaming `fetch` is available | -| Code that **synchronously** returns an `AsyncIterable` (in-process `chat()`, an RSC stream, tests) | [`stream`](#server-functions-and-direct-async-iterables) | -| An **async** call — a TanStack Start server function or any `Promise`-returning function — resolving to a `Response` or an `AsyncIterable` | [`fetcher`](#server-functions-via-fetcher) | -| An RPC framework like Cap'n Web, gRPC-Web, or tRPC | [`rpcStream`](#rpc-streams) | -| A single long-lived WebSocket (or BroadcastChannel, postMessage, shared worker) serving many runs | [Custom `subscribe` / `send` adapter](#persistent-transports-websockets-and-friends) | -| Standard SSE but with custom fetch wrapping (auth refresh, retries) | [`fetchServerSentEvents` with `fetchClient`](#custom-fetch-client) | -| Something else entirely (HTTP/3, Server-Sent Events over a different protocol, etc.) | [Custom `connect` adapter](#custom-request-scoped-adapters) | - -All adapters produce the same `StreamChunk` events ([AG-UI Protocol](../migration/ag-ui-compliance)) — the choice is purely about transport. +| Normal HTTP + default | [`fetchServerSentEvents`](#server-sent-events-sse) | +| SSE blocked (edge/proxy) | [`fetchHttpStream`](#http-streaming-ndjson) | +| React Native / Expo | [`xhrHttpStream`](#react-native-and-expo) (default); `xhrServerSentEvents` for SSE; `fetchHttpStream` only if streaming `fetch` works | +| Sync `AsyncIterable` (in-process `chat()`, RSC, tests) | [`stream`](#server-functions-and-direct-async-iterables) | +| Async fn → `Response` or iterable (TanStack Start server fn) | [`fetcher`](#server-functions-via-fetcher) | +| Cap'n Web / gRPC-Web / tRPC | [`rpcStream`](#rpc-streams) | +| Long-lived WebSocket / BroadcastChannel / shared worker | [Custom `subscribe` / `send`](#persistent-transports-websockets-and-friends) | +| SSE + custom fetch (auth refresh, retries) | [`fetchServerSentEvents` + `fetchClient`](#custom-fetch-client) | +| Other request-scoped transport | [Custom `connect`](#custom-request-scoped-adapters) | + +All adapters yield the same `StreamChunk` events ([AG-UI Protocol](../migration/ag-ui-compliance)). ## Server-Sent Events (SSE) -The default. SSE is well-supported across browsers, transparent through most proxies, and easy to debug. Pair it with `toServerSentEventsResponse()` on the server. +Default. Pair with `toServerSentEventsResponse()` on the server. ```typescript import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; @@ -50,7 +48,7 @@ const { messages, sendMessage } = useChat({ }); ``` -**Dynamic URL and headers.** Pass functions when the value depends on per-request state (current user, fresh token): +**Dynamic URL / headers** — pass functions for per-request values: ```typescript import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; @@ -66,7 +64,7 @@ const { messages } = useChat({ }); ``` -**Static body.** Anything in `options.body` is merged into the AG-UI `forwardedProps` payload sent to your server. Per-message data passed to `sendMessage` wins over this: +**Static body** — merged into AG-UI `forwardedProps`. Per-message `sendMessage` data wins: ```typescript import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; @@ -78,24 +76,15 @@ const { messages } = useChat({ }); ``` -> **Tip:** `body` and `forwardedProps` populate the same wire field. Use `body` for static defaults, the `forwardedProps` constructor option (or per-`sendMessage` `data`) for dynamic values. Runtime values always win. +> **Tip:** `body` and `forwardedProps` share the same wire field. Use `body` for static defaults; runtime values always win. ### Resumable SSE -`fetchServerSentEvents` watches SSE `id:` values. If a connection drops after -receiving an id, it reconnects with `Last-Event-ID` and de-duplicates the -replayed prefix. `joinRun(runId)` performs a read-only GET with `offset=-1` and -the run id, replaying an in-flight or finished run from the start. +`fetchServerSentEvents` tracks SSE `id:` values. On drop after an id, it reconnects with `Last-Event-ID` and de-duplicates replay. `joinRun(runId)` does a read-only GET with `offset=-1` and the run id. -The ids only appear when the server passes a durability adapter to -`toServerSentEventsResponse`. They are opaque tokens owned by that adapter; the -chat client does not create, parse, or persist them. Without ids, behavior is -identical to a plain single fetch. See -[Resumable Streams](../resumable-streams/overview). +Ids appear only when the server passes a durability adapter to `toServerSentEventsResponse`. Without ids → plain single fetch. See [Resumable Streams](../resumable-streams/overview). -Your route needs a `GET` handler alongside `POST` for `joinRun` (second tab or -reload) to work. `POST` handles fresh runs and auto-reconnects (it re-sends the -same body with `Last-Event-ID`); `GET` replays a known run from the start: +**Must:** add a `GET` handler for `joinRun` (second tab / reload). `POST` = fresh runs + auto-reconnect; `GET` = replay only: ```typescript import { @@ -115,26 +104,19 @@ export async function POST(request: Request) { }); } -// joinRun hits GET ?offset=-1&runId=... (replay only, no messages sent). +// joinRun hits GET ?offset=-1&runId=... (replay only) export async function GET(request: Request) { return resumeServerSentEventsResponse({ adapter: memoryStream(request) }); } ``` -The `GET` handler calls no provider: on a replay the durability adapter's -`resumeFrom()` is non-null (from `?offset`), so the log is replayed instead. -`resumeServerSentEventsResponse` returns a 400 when the request has no resume -offset. Use `resumeHttpResponse` for the NDJSON adapters. +`GET` does not call a provider — durability `resumeFrom()` replays the log. No resume offset → 400. Use `resumeHttpResponse` for NDJSON adapters. -`fetchHttpStream` and `xhrHttpStream` resume the same way over NDJSON, where the -offset rides in an `{ id, chunk }` envelope (see below) instead of an SSE `id:` -line. Enable it by passing a durability adapter to `toHttpResponse`. -`xhrServerSentEvents` resumes over SSE exactly like `fetchServerSentEvents` -(paired with `toServerSentEventsResponse` and its `id:` lines). +`fetchHttpStream` / `xhrHttpStream` resume the same way over NDJSON (`{ id, chunk }` envelopes via `toHttpResponse`). `xhrServerSentEvents` resumes over SSE like `fetchServerSentEvents`. ## HTTP Streaming (NDJSON) -For environments that don't speak SSE — some edge runtimes, certain mobile WebViews, or anywhere a proxy strips `text/event-stream` — use raw newline-delimited JSON. The wire format is one JSON `StreamChunk` per line: +Use when SSE is stripped. Wire format: one JSON `StreamChunk` per line. ```typescript import { useChat, fetchHttpStream } from "@tanstack/ai-react"; @@ -144,17 +126,15 @@ const { messages } = useChat({ }); ``` -Server-side, write each chunk as `JSON.stringify(chunk) + "\n"` to the response body (or use `toHttpResponse(stream)`). Options (`url`, `headers`, `body`, `fetchClient`, dynamic functions) match `fetchServerSentEvents` exactly. +Server: `toHttpResponse(stream)` (or `JSON.stringify(chunk) + "\n"`). Options match `fetchServerSentEvents` (`url`, `headers`, `body`, `fetchClient`). -`fetchHttpStream` is also resumable: pass a durability adapter to `toHttpResponse` and each line becomes an `{ id, chunk }` envelope. A dropped connection reconnects with `Last-Event-ID`, de-duplicates the replayed prefix, and `joinRun(runId)` attaches to an existing run. Same guarantees as [Resumable SSE](#resumable-sse), over NDJSON. +Pass a durability adapter to `toHttpResponse` for resumability — same guarantees as [Resumable SSE](#resumable-sse) over NDJSON. ## React Native and Expo -You have a native app that needs to call your own backend rather than a -same-origin browser route. Use `useChat` from `@tanstack/ai-react` with an -explicit chat transport and an absolute URL. By the end of this section, the -client adapter and server response helper will be paired correctly for React -Native or Expo. +1. Point at an absolute backend URL your runtime can reach +2. Prefer `xhrHttpStream` + `toHttpResponse` +3. Keep provider SDKs off the mobile bundle ```typescript const baseUrl = @@ -164,15 +144,13 @@ const httpUrl = `${baseUrl}/chat/http` const sseUrl = `${baseUrl}/chat/sse` ``` -Use the URL your runtime can reach: - -- **iOS simulator**: often `localhost` or `127.0.0.1`. -- **Android emulator**: commonly `10.0.2.2` to reach the host machine. -- **A physical device**: a LAN or tunneled URL. +| Runtime | Host URL tip | +| --- | --- | +| iOS simulator | `localhost` / `127.0.0.1` | +| Android emulator | `10.0.2.2` for host machine | +| Physical device | LAN or tunnel | -Prefer `xhrHttpStream()` for Expo and React Native. It pairs with -`toHttpResponse()` and reads newline-delimited JSON through incremental XHR -progress events: +**Preferred — XHR + NDJSON:** ```typescript import { useChat, xhrHttpStream } from "@tanstack/ai-react"; @@ -185,12 +163,7 @@ const chat = useChat({ }); ``` -Mobile connections drop often, so this is where resumability pays off most. -Both XHR adapters reconnect and `joinRun` when the server adds a durability -adapter. See [Resumable Streams](../resumable-streams/overview). - -Use `xhrServerSentEvents()` when your server returns `text/event-stream` via -`toServerSentEventsResponse()`: +**SSE via XHR** (server uses `toServerSentEventsResponse`): ```typescript import { useChat, xhrServerSentEvents } from "@tanstack/ai-react"; @@ -203,62 +176,35 @@ const chat = useChat({ }); ``` -Only use `fetchHttpStream()` if your exact React Native runtime exposes -streaming `fetch` responses, `Response.body.getReader()`, and `TextDecoder`. -The server still returns newline-delimited JSON with `toHttpResponse()`: - -```typescript -import { useChat, fetchHttpStream } from "@tanstack/ai-react"; - -const baseUrl = process.env.EXPO_PUBLIC_TANSTACK_AI_BASE_URL ?? 'http://127.0.0.1:8787'; -const httpUrl = `${baseUrl}/chat/http`; - -const chat = useChat({ - connection: fetchHttpStream(httpUrl), -}); -``` - -If one of those fetch-streaming APIs is missing, `fetchHttpStream()` throws -`UnsupportedResponseStreamError`. A polyfill that buffers the response does not -make fetch streaming compatible; the adapter needs incremental bytes. Switch to -`xhrHttpStream()` or `xhrServerSentEvents()` instead. +**`fetchHttpStream` only if** the RN runtime has streaming `fetch`, `Response.body.getReader()`, and `TextDecoder`. Missing any → `UnsupportedResponseStreamError`. Buffering polyfills do not count — switch to XHR adapters. -Keep provider SDKs and server helpers on your backend. The React Native bundle -should import hooks and connection adapters, not OpenAI/Anthropic/Gemini SDKs, -React DOM UI, devtools UI, or other framework packages. For a complete mobile -walkthrough, see [Quick Start: React Native](../getting-started/quick-start-react-native). +Full walkthrough: [Quick Start: React Native](../getting-started/quick-start-react-native). Resumability: [Resumable Streams](../resumable-streams/overview). -## Server Functions and Direct Async Iterables +## Server functions and direct async iterables -When your client can call into your server without going over HTTP — RSC streams, in-process tests, a direct in-process `chat()` call — skip the transport entirely. `stream()` takes a factory that returns an `AsyncIterable` **synchronously** and wires it straight into the client. (A [TanStack Start](https://tanstack.com/start) server function returns a `Promise`, so it needs [`fetcher`](#server-functions-via-fetcher), not `stream()` — see the next section.) +If the factory returns `AsyncIterable` **synchronously** → `stream()`: ```typescript import { useChat, stream } from "@tanstack/ai-react"; import { chatServerFn } from "./server/chat.server"; -// `chatServerFn` is an in-process server-side function that synchronously -// returns an AsyncIterable — e.g. the result of -// `chat({ adapter, model, messages })` on the server. +// chatServerFn must return AsyncIterable synchronously const { messages } = useChat({ connection: stream((messages, data) => chatServerFn({ messages, ...data })), }); ``` -The factory receives the conversation messages plus any per-request `data` you passed to `sendMessage`. Return any async iterable that yields `StreamChunk` objects — a generator, the output of `chat()` on the server, a transformed stream, anything. - -> **Tip:** `stream()` is **request-scoped**. The factory is invoked once per `sendMessage`, the iterable runs to completion, and the connection closes. If you need a single long-lived channel that multiplexes many sends — for example a WebSocket — use [`subscribe` / `send`](#persistent-transports-websockets-and-friends) instead. +> **Tip:** TanStack Start server functions return a `Promise` → use [`fetcher`](#server-functions-via-fetcher), not `stream()`. -`stream()` also takes an optional second argument of persistence handlers, spread onto the adapter, so server-driven persistence (`persistence: true`) works without an HTTP endpoint. Each is typically a one-line call into your server: +`stream()` is **request-scoped** (one factory call per `sendMessage`). For a long-lived multiplexed channel → [`subscribe` / `send`](#persistent-transports-websockets-and-friends). -- `hydrate`: restores a chat thread. -- `hydrateGeneration`: restores a generation's last run. -- `joinRun`: replays a run still in flight. +Optional second arg: persistence handlers (`hydrate`, `hydrateGeneration`, `joinRun`). See [Generation Persistence](../persistence/generation-persistence#server-functions--direct). -See [Generation Persistence](../persistence/generation-persistence#server-functions--direct) for the full server-function wiring. +## Server functions via `fetcher` -## Server Functions via `fetcher` +For **async** handlers (TanStack Start server fn → `Promise`). Provide `fetcher` **or** `connection`, not both. -When you call into your server with an **async** function — the universal case for a [TanStack Start](https://tanstack.com/start) server function, which always returns a `Promise` — use the top-level `fetcher` option instead of a connection adapter. `fetcher` is a sibling of `connection` (provide exactly one), and it accepts a plain async function. It mirrors the `fetcher` option on the [generation hooks](../media/generation-hooks). The most common shape is a handler that ends with `toServerSentEventsResponse(...)` and resolves to a `Response`: +**Server:** ```typescript ignore // server/chat.server.ts @@ -276,6 +222,8 @@ export const chatFn = createServerFn({ method: "POST" }) ); ``` +**Client:** + ```typescript import { useChat } from "@tanstack/ai-react"; import { chatFn } from "./server/chat.server"; @@ -285,26 +233,23 @@ const { messages, sendMessage } = useChat({ }); ``` -The fetcher receives `{ messages, data, threadId, runId }` plus an `AbortSignal` (triggered by `stop()` or when a send is superseded). Return either: - -- a `Response`: the chat client parses its SSE body for you. -- an `AsyncIterable`: yielded directly. This covers a server function that returns the stream itself rather than wrapping it in a `Response`. +Fetcher input: `{ messages, data, threadId, runId }` + `AbortSignal`. Return: -Sync and `Promise`-wrapped returns are both accepted. +1. `Response` — client parses SSE body +2. `AsyncIterable` — yielded directly -> **Tip:** The generation hooks (`useGenerateImage` and siblings) take the same server-function shape a step further: alongside their `fetcher` they accept `hydrateGeneration` and `joinRun` options, so `persistence: true` hydrates and rejoins through server functions with no HTTP route at all. See [Generation Persistence — Server functions / direct](../persistence/generation-persistence#server-functions--direct). +Sync and `Promise`-wrapped returns both work. -> **Tip:** The choice between `fetcher` and [`stream()`](#server-functions-and-direct-async-iterables) is about **async vs sync**, not `Response`-vs-iterable — both can yield an `AsyncIterable`. `stream()`'s factory must return that iterable **synchronously**, so a server-function call (which returns a `Promise`) won't typecheck there — that's the gap `fetcher` fills ([issue #509](https://github.com/TanStack/ai/issues/509)). Use `stream()` when you can hand back an async iterable synchronously (in-process `chat()`, an RPC client, tests); use `fetcher` for anything you have to `await`. Both normalize to the same request-scoped adapter, so `stop()`/abort, error handling, and tool calls behave identically. +> **`fetcher` vs `stream()`:** about **async vs sync**, not Response-vs-iterable. `stream()` factory must return the iterable synchronously; server-fn `Promise`s need `fetcher` ([issue #509](https://github.com/TanStack/ai/issues/509)). -## RPC Streams +## RPC streams -`rpcStream()` is identical in behavior to `stream()` but reads better at call sites that hand off to an RPC client. Use it when integrating with Cap'n Web, gRPC-Web, tRPC subscriptions, or any RPC framework that already returns an async iterable: +Same behavior as `stream()`, clearer at RPC call sites: ```typescript import { useChat, rpcStream } from "@tanstack/ai-react"; import { api } from "./rpc-client"; -// `api.chat.stream` is your RPC method; it must return an AsyncIterable. const { messages } = useChat({ connection: rpcStream((messages, data) => api.chat.stream({ messages, ...data }), @@ -312,27 +257,16 @@ const { messages } = useChat({ }); ``` -Like `stream()`, `rpcStream()` takes an optional second argument of persistence handlers (`{ hydrate, hydrateGeneration, joinRun }`) so server-driven persistence works over RPC — each handler is usually a one-line RPC call. - -## Persistent Transports (WebSockets and Friends) - -A persistent transport — WebSocket, BroadcastChannel, postMessage between iframes, a shared worker — is fundamentally different from request/response. You open the channel **once**, then send and receive over it for the lifetime of the client. `stream()`/`connect()` can't model this cleanly because they assume one async iterable per request. - -For these cases, implement the `SubscribeConnectionAdapter` interface directly. The shape (full definition in [The Adapter Interface](#the-adapter-interface)): +Optional persistence handlers: `{ hydrate, hydrateGeneration, joinRun }`. -```typescript -import type { SubscribeConnectionAdapter } from "@tanstack/ai-react"; - -// subscribe(abortSignal?): AsyncIterable — long-lived -// send(messages, data?, abortSignal?, runContext?): Promise — one per user message -``` +## Persistent transports (WebSockets) -- `subscribe()` is called **once** by the `ChatClient` and returns a long-lived async iterable of every chunk the channel produces. -- `send()` is called **once per user message** to push a request frame onto the channel. It returns when the frame has been written — chunks arrive separately through `subscribe()`. +Open once, send/receive for the client lifetime. Implement `SubscribeConnectionAdapter`: -The runtime correlates them: chunks emitted on the subscription queue between `send()` and the next terminal event (`RUN_FINISHED` / `RUN_ERROR`) are attributed to that run. +- `subscribe()` — once; long-lived async iterable of all chunks +- `send()` — once per user message; returns when the frame is written -### WebSocket example +Chunks between `send()` and next terminal (`RUN_FINISHED` / `RUN_ERROR`) belong to that run. ```typescript import { useChat, type SubscribeConnectionAdapter } from "@tanstack/ai-react"; @@ -369,16 +303,11 @@ function websocketConnection(url: string): SubscribeConnectionAdapter { return { async *subscribe(abortSignal) { - // Register the abort listener once (not per-iteration) so it can't - // accumulate on a long-lived socket. const onAbort = () => deliver(null); abortSignal?.addEventListener("abort", onAbort, { once: true }); try { while (!abortSignal?.aborted) { - // Drain buffered chunks BEFORE honoring `closed`: a burst of messages - // followed by a close event (common within one macrotask) must still - // deliver the queued chunks (including a trailing RUN_FINISHED), - // otherwise the client would hang waiting for a terminal it dropped. + // Drain queue before honoring close so trailing RUN_FINISHED is not dropped const buffered = queue.shift(); if (buffered !== undefined) { yield buffered; @@ -415,21 +344,19 @@ const { messages } = useChat({ }); ``` -> **Tip:** Your server is responsible for emitting `RUN_FINISHED` (or `RUN_ERROR`) at the end of each run. Without it, the client will not know the assistant turn has ended and will wait indefinitely. See [Streaming](./streaming) for the full event lifecycle. - -### When to choose persistent over request-scoped +> **Must:** server emits `RUN_FINISHED` (or `RUN_ERROR`) per run or the client waits forever. See [Streaming](./streaming). -Pick `subscribe` / `send` when **any** of these are true: +**Choose persistent when any of:** -- A single connection multiplexes many runs (chat thread keeps the socket open across messages). -- The server pushes chunks outside of a request (presence updates, server-initiated tool calls, broadcast notifications). -- You want to share one connection across multiple tabs (BroadcastChannel) or workers. +1. One connection multiplexes many runs +2. Server pushes outside request/response +3. Shared connection across tabs/workers -Otherwise, prefer `fetchServerSentEvents` or `stream()` — they're simpler and require no connection lifecycle management. +Otherwise prefer `fetchServerSentEvents` or `stream()`. -## Custom Fetch Client +## Custom fetch client -If you're keeping SSE or HTTP streaming but need to wrap `fetch` — for auth refresh, retries, logging, or routing through an edge proxy — pass a `fetchClient`: +Wrap `fetch` for auth refresh, retries, logging: ```typescript import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; @@ -451,11 +378,11 @@ const { messages } = useChat({ }); ``` -The `fetchClient` must satisfy the standard `fetch` signature. `fetchHttpStream` accepts the same option. +Same option on `fetchHttpStream`. -## Custom Request-Scoped Adapters +## Custom request-scoped adapters -When none of the built-ins fit but the transport is still request-scoped (one request per user message), implement `ConnectConnectionAdapter` directly. This is the lowest-level escape hatch short of going persistent: +Implement `ConnectConnectionAdapter` when built-ins do not fit but transport is still one-request-per-message: ```typescript import { useChat, type ConnectConnectionAdapter } from "@tanstack/ai-react"; @@ -478,8 +405,6 @@ const myAdapter: ConnectConnectionAdapter = { if (!response.ok) throw new Error(`HTTP ${response.status}`); if (!response.body) throw new Error("Response has no body"); - // Example: newline-delimited JSON. Replace this loop with whatever - // framing your wire format uses, yielding one `StreamChunk` per event. const reader = response.body.getReader(); const decoder = new TextDecoder(); let buffer = ""; @@ -502,16 +427,11 @@ const myAdapter: ConnectConnectionAdapter = { const { messages } = useChat({ connection: myAdapter }); ``` -`runContext` carries `threadId`, `runId`, `clientTools`, and `forwardedProps`. Include them in your request payload so the server can build an AG-UI-compliant response. +Include `runContext` (`threadId`, `runId`, `clientTools`, `forwardedProps`) in the payload. Runtime synthesizes `RUN_FINISHED` if missing, or `RUN_ERROR` on throw. -The runtime covers the terminal event either way: +## Adapter interface -- Your `connect` stream completes without emitting `RUN_FINISHED`: one is synthesized for you. -- Your `connect` stream throws: a `RUN_ERROR` is synthesized. - -## The Adapter Interface - -A `ConnectionAdapter` is a union — provide **either** `connect`, **or** both `subscribe` and `send`. Never both modes. +Provide **either** `connect` **or** both `subscribe` + `send` — never both modes: ```typescript import type { UIMessage } from "@tanstack/ai-client"; @@ -549,14 +469,11 @@ export type ConnectionAdapter = | SubscribeConnectionAdapter; ``` -Internally, `ChatClient` normalizes both shapes to a single `subscribe`/`send` pair via `normalizeConnectionAdapter()`: - -- Provide `connect` and it gets wrapped in an async queue. -- Provide `subscribe` + `send` natively and they are used as-is. +`ChatClient` normalizes both via `normalizeConnectionAdapter()`. -## Authentication +## Auth -Static headers go in `options.headers`: +Static headers: ```typescript import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; @@ -569,7 +486,7 @@ const { messages } = useChat({ }); ``` -For tokens that change per request (refresh tokens, short-lived JWTs), pass a function — it's called on every send, so the header always reflects the latest token: +Per-request tokens — pass a function (called every send): ```typescript import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; @@ -582,24 +499,24 @@ const { messages } = useChat({ }); ``` -Cookies are sent automatically when `credentials` is `"same-origin"` (default) or `"include"`. +Cookies: automatic when `credentials` is `"same-origin"` (default) or `"include"`. ## Cancellation -Every adapter — built-in or custom — receives an `AbortSignal`. Built-ins propagate it to `fetch`; custom adapters must honor it themselves. `useChat`'s `stop()` aborts the current run by triggering the signal: +Every adapter gets an `AbortSignal`. Built-ins pass it to `fetch`. `stop()` aborts the current run: ```typescript import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; const { stop } = useChat({ connection: fetchServerSentEvents("/api/chat") }); -stop(); // aborts the active stream +stop(); ``` -For `SubscribeConnectionAdapter`, the signal in `subscribe()` ends the entire subscription (component unmount); the signal in `send()` ends just the in-flight send. +`SubscribeConnectionAdapter`: `subscribe` signal ends the subscription (unmount); `send` signal ends that send only. -## Error Handling +## Errors -Adapters should throw on transport errors (HTTP non-2xx, parse failures, dropped sockets). The `ChatClient` catches the throw, emits a `RUN_ERROR` chunk if none has been emitted yet, and surfaces it via `onError` / the `error` state: +Throw on transport failures. Client emits `RUN_ERROR` and surfaces via `onError` / `error`: ```typescript import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; @@ -610,19 +527,19 @@ const { error } = useChat({ }); ``` -Don't swallow `AbortError` — let it propagate so the client knows the abort succeeded. +Do not swallow `AbortError`. -## Best Practices +## Must-do -- **Default to SSE.** It's the most compatible and the easiest to debug. Switch only when something blocks it. -- **Use `stream()` when you can.** If you control both sides and don't need HTTP semantics, server functions are faster to wire up than building a custom adapter. -- **Reach for `subscribe`/`send` only when you need persistence.** WebSockets are powerful but require you to handle reconnection, run correlation, and lifecycle yourself. -- **Always honor `abortSignal`.** It's how the client cleans up on unmount and on `stop()`. -- **Emit `RUN_FINISHED` from the server.** Without it, the client never knows the turn ended. +1. Default to SSE unless something blocks it +2. Prefer `stream()` / `fetcher` when you control both sides +3. Use `subscribe`/`send` only when you need a persistent channel +4. Honor `abortSignal` +5. Emit `RUN_FINISHED` from the server -## Next Steps +## Next -- [Streaming](./streaming) — the full event lifecycle and `StreamChunk` types -- [AG-UI Client Compliance](../migration/ag-ui-compliance) — the wire protocol your server emits -- [Cloudflare Adapter](../community-adapters/cloudflare) — example of a custom `fetchClient` in production -- [API Reference: `@tanstack/ai-client`](../api/ai-client) — full type signatures +- [Streaming](./streaming) +- [AG-UI Client Compliance](../migration/ag-ui-compliance) +- [Cloudflare Adapter](../community-adapters/cloudflare) +- [API Reference: `@tanstack/ai-client`](../api/ai-client) diff --git a/docs/chat/streaming.md b/docs/chat/streaming.md index e04a75999..972577b20 100644 --- a/docs/chat/streaming.md +++ b/docs/chat/streaming.md @@ -2,7 +2,7 @@ title: Streaming id: streaming-responses order: 2 -description: "Stream AI responses in real time with TanStack AI — async iterable chunks, chunk strategies, and partial JSON for responsive chat UIs." +description: "Stream chat chunks in real time — server/client wiring, AG-UI events, cancel, and message queueing." keywords: - tanstack ai - streaming @@ -13,30 +13,9 @@ keywords: - partial json --- -TanStack AI supports streaming responses for real-time chat experiences. Streaming allows you to display responses as they're generated, rather than waiting for the complete response. +If you need live token-by-token UI → stream `chat()` and wire it with a connection adapter. -## How Streaming Works - -When you use `chat()`, it returns an async iterable stream of chunks: - -```typescript -import { chat } from "@tanstack/ai"; -import { openaiText } from "@tanstack/ai-openai"; - -const stream = chat({ - adapter: openaiText("gpt-5.5"), - messages: [{ role: "user", content: "Hello!" }], -}); - -// Stream contains chunks as they arrive -for await (const chunk of stream) { - console.log(chunk); // Process each chunk -} -``` - -## Server-Side Streaming - -Convert the stream to an HTTP response using `toServerSentEventsResponse`: +## 1. Stream on the server ```typescript import { chat, toServerSentEventsResponse } from "@tanstack/ai"; @@ -50,14 +29,29 @@ export async function POST(request: Request) { messages, }); - // Convert to HTTP response with proper headers return toServerSentEventsResponse(stream); } ``` -## Client-Side Streaming +Iterate chunks yourself when needed: + +```typescript +import { chat } from "@tanstack/ai"; +import { openaiText } from "@tanstack/ai-openai"; -The `useChat` hook automatically handles streaming: +const stream = chat({ + adapter: openaiText("gpt-5.5"), + messages: [{ role: "user", content: "Hello!" }], +}); + +for await (const chunk of stream) { + console.log(chunk); +} +``` + +## 2. Consume on the client + +`useChat` updates `messages` as chunks arrive: ```typescript import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; @@ -65,43 +59,30 @@ import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; const { messages, sendMessage, isLoading } = useChat({ connection: fetchServerSentEvents("/api/chat"), }); - -// Messages update in real-time as chunks arrive -messages.forEach((message) => { - // Message content updates incrementally -}); ``` -## Stream Events (AG-UI Protocol) - -TanStack AI implements the [AG-UI Protocol](https://docs.ag-ui.com/introduction) for streaming. Stream events contain different types of data: +## Stream events (AG-UI) -### AG-UI Events +TanStack AI uses the [AG-UI Protocol](https://docs.ag-ui.com/introduction). -- **RUN_STARTED** - Emitted when a run begins -- **TEXT_MESSAGE_START/CONTENT/END** - Text content streaming lifecycle -- **TOOL_CALL_START/ARGS/END** - Tool invocation lifecycle -- **STEP_STARTED/STEP_FINISHED** - Thinking/reasoning steps -- **CUSTOM** - Namespaced extension events (sandbox file changes, Code Mode progress, structured-output completion, and your own `emitCustomEvent` calls) — see the [Custom Events Reference](../protocol/custom-events) for the full typed taxonomy and how to narrow `chunk.value` with a plain `if` -- **RUN_FINISHED** - Run completion with finish reason and usage -- **RUN_ERROR** - Error occurred during the run +| Event | When | +| --- | --- | +| `RUN_STARTED` | Run begins | +| `TEXT_MESSAGE_START/CONTENT/END` | Text streaming | +| `TOOL_CALL_START/ARGS/END` | Tool invocation | +| `STEP_STARTED/STEP_FINISHED` | Thinking/reasoning (legacy) | +| `CUSTOM` | Extensions — [Custom Events](../protocol/custom-events) | +| `RUN_FINISHED` | Success + usage | +| `RUN_ERROR` | Failure | -> **Tip:** Some models expose their internal reasoning as thinking content that streams before the response. See [Thinking & Reasoning](./thinking-content). +Thinking streams before text. See [Thinking & Reasoning](./thinking-content). -### Threads and runs +### Threads vs runs -Two ids frame every stream, and they come from the AG-UI protocol itself — not -from any storage layer: +- **`threadId`** — conversation identity across reloads/devices +- **`runId`** — one execution (`RUN_STARTED` → `RUN_FINISHED` / `RUN_ERROR`) -- A **thread** (`threadId`) is the conversation: the stable identity across - every exchange, reload, and device. -- A **run** (`runId`) is one execution inside it: everything between one - `RUN_STARTED` and its `RUN_FINISHED` (or `RUN_ERROR`). Every start mints a - fresh run id, so a thread accumulates many runs over its life. - -A run is not limited to a single model response. Tool calls and their -follow-up responses stream inside the same run — the whole -[agentic cycle](./agentic-cycle), however many loops it takes, is one run: +A run includes the full [agentic cycle](./agentic-cycle) (tool calls + follow-ups), not just one model response. ```mermaid flowchart LR @@ -123,16 +104,11 @@ flowchart LR end ``` -Because run ids are ephemeral, anything long-lived anchors on the thread: -[resumable streams](../resumable-streams/overview) log delivery per `runId`, -while [server persistence](../persistence/chat-persistence#threads-runs-and-turns) -stores the transcript per `threadId`. The media generation hooks take a -`threadId` too, where it names a slot rather than a conversation. See -[Id map](../persistence/id-map). +Resumable streams log per `runId`; [server persistence](../persistence/chat-persistence#threads-runs-and-turns) stores per `threadId`. See [Id map](../persistence/id-map). -### Type-Safe Tool Call Events +### Type-safe tool call events -When you pass typed tools (defined with `toolDefinition()` and Zod schemas) to `chat()`, the stream chunks automatically carry type information for tool call events. Prefer the AG-UI field `toolCallName` (or the deprecated `toolName` alias) — both narrow to the union of your tool name literals. The `input` field on `TOOL_CALL_END` is typed as the union of your tool input schemas (typically set on the adapter-emitted END once arguments are complete): +Pass tools from `toolDefinition()` + Zod → `toolCallName` / `input` narrow on the stream: ```typescript import { chat, toolDefinition } from "@tanstack/ai"; @@ -159,19 +135,15 @@ const stream = chat({ }); for await (const chunk of stream) { - // `'type' in chunk` is required for control-flow narrowing across the - // StreamChunk union (AG-UI event types from `@ag-ui/core` use Zod - // passthrough, which otherwise hides the discriminant from property access). + // `'type' in chunk` required for control-flow narrowing on StreamChunk if ("type" in chunk && chunk.type === "TOOL_CALL_END") { - chunk.toolCallName; // ✅ typed as "get_weather" (not string) - chunk.input; // ✅ typed as { location: string; unit?: "celsius" | "fahrenheit" } | undefined + chunk.toolCallName; // "get_weather" + chunk.input; // { location: string; unit?: ... } | undefined } } ``` -Without typed tools, names default to `string` and `input`/`output` default to `unknown` — the same behavior as before. The type narrowing is automatic when you use `toolDefinition()` with Zod schemas. - -When multiple tools are provided, tool call events form a **discriminated union** — checking `toolCallName` (or `toolName`) narrows `input` / `output` to that specific tool's type: +With multiple tools, check `toolCallName` to narrow `input` / `output` per tool: ```typescript import { chat, toolDefinition } from "@tanstack/ai"; @@ -206,22 +178,20 @@ const stream = chat({ for await (const chunk of stream) { if ("type" in chunk && chunk.type === "TOOL_CALL_END") { if (chunk.toolCallName === "get_weather") { - // ✅ input is narrowed to { location: string; unit?: "celsius" | "fahrenheit" } console.log(`Weather in ${chunk.input?.location}`); } if (chunk.toolCallName === "search") { - // ✅ input is narrowed to { query: string } console.log(`Searched for: ${chunk.input?.query}`); } } } ``` -> **Tip:** The typed stream type is exported as `TypedStreamChunk`. The default (no type args) matches `ChatStream`: standard chunks plus the known framework `CUSTOM` event union. Free-form `emitCustomEvent` names still flow at runtime; cast to `StreamChunk` if you need to read them. +Without typed tools, names are `string` and `input`/`output` are `unknown`. Typed stream type: `TypedStreamChunk`. -### Thinking Chunks +### Thinking parts -Adapters emit reasoning as both the canonical `REASONING_MESSAGE_*` events and the older `STEP_STARTED` / `STEP_FINISHED` events. Rather than parsing those raw events yourself, read the reconciled `ThinkingPart` from `message.parts` — the stream processor merges both event families into a single part for you: +Adapters emit `REASONING_MESSAGE_*` and legacy `STEP_*` events. Read the reconciled `ThinkingPart` from `message.parts` — do not hand-parse raw events: ```typescript import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; @@ -233,19 +203,21 @@ const { messages } = useChat({ for (const message of messages) { for (const part of message.parts) { if (part.type === "thinking") { - console.log("Thinking:", part.content); // Accumulated thinking content + console.log("Thinking:", part.content); } } } ``` -Thinking content is automatically converted to `ThinkingPart` in `UIMessage` objects. It is UI-only and excluded from messages sent back to the model. See [Thinking & Reasoning](./thinking-content) for the full rendering pattern. - -## Connection Adapters +Thinking is UI-only — never sent back to the model. See [Thinking & Reasoning](./thinking-content). -TanStack AI provides connection adapters for different streaming protocols: +## Connection adapters -### Server-Sent Events (SSE) +| Transport | Import | +| --- | --- | +| SSE (default) | `fetchServerSentEvents("/api/chat")` | +| NDJSON HTTP | `fetchHttpStream("/api/chat")` | +| Async / server fn | `fetcher` option (sibling of `connection`) | ```typescript import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; @@ -255,8 +227,6 @@ const { messages } = useChat({ }); ``` -### HTTP Stream - ```typescript import { useChat, fetchHttpStream } from "@tanstack/ai-react"; @@ -265,9 +235,7 @@ const { messages } = useChat({ }); ``` -### Custom Stream - -For a fully custom request, use the `fetcher` transport. The fetcher receives the request input plus an `AbortSignal`, and returns a `Response` (whose SSE body the client parses) or an `AsyncIterable`. It may return that value synchronously, as a `Promise`, or as an `async function*`: +`fetcher` for a custom request (returns `Response` or `AsyncIterable`): ```typescript import { useChat } from "@tanstack/ai-react"; @@ -282,11 +250,9 @@ const { messages } = useChat({ }); ``` -> **Note:** The lower-level `stream()` connection adapter takes a factory that must return an `AsyncIterable` **synchronously** (e.g. a generator) — it does not accept an `async (...) => {...}` function that returns a `Promise`. Prefer the `fetcher` transport above unless you specifically need the connection adapter. - -## Monitoring Stream Progress +> **Note:** `stream()` needs a factory that returns `AsyncIterable` **synchronously**. Prefer `fetcher` for anything you `await`. Full matrix: [Connection Adapters](./connection-adapters). -You can monitor stream progress with callbacks: +## Monitor progress ```typescript import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; @@ -302,9 +268,7 @@ const { messages } = useChat({ }); ``` -## Cancelling Streams - -Cancel ongoing streams: +## Cancel ```typescript import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; @@ -313,13 +277,10 @@ const { stop } = useChat({ connection: fetchServerSentEvents("/api/chat"), }); -// Cancel the current stream -stop(); +stop(); // aborts active stream (expected AbortError) ``` -Calling `stop()` aborts the underlying fetch; the resulting `AbortError` is expected and normal. This differs from a connection being cut mid-line: a truncated stream throws a `StreamTruncatedError` and moves the client into its `error` state. See [Connection Adapters](./connection-adapters) for the underlying behavior. - -On the server, pass an `AbortController` to `toServerSentEventsResponse(stream, { abortController })` so the chat run is cancelled when the client disconnects: +Truncated mid-line streams throw `StreamTruncatedError` instead. On the server, pass `abortController` so the run cancels on client disconnect: ```typescript import { chat, toServerSentEventsResponse } from "@tanstack/ai"; @@ -334,13 +295,9 @@ export async function POST(request: Request) { } ``` -## Queueing Messages - -By default, calling `sendMessage` while a stream is already in flight **queues** the message instead of dropping it. It sends automatically once the current run settles **successfully**. Configure this with the `queue` option, which accepts any of three forms: +## Queue messages while streaming -- a `QueueConfig` object -- a plain shorthand string -- a strategy function +Default: `sendMessage` while busy **queues** (does not drop). Sends after a **successful** settle. ```tsx group=queueing-messages import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; @@ -351,23 +308,28 @@ const { messages, queue, sendMessage, cancelQueued, isLoading } = useChat({ }); ``` -- **`whenBusy`** — what happens to a send that arrives while the client is busy (streaming, claiming a send, or draining the queue): - - `"queue"` (default) — hold the message; it sends once the run settles **successfully**. Clear the composer once the message appears in `queue` or `messages`. - - `"drop"` — ignore the send (promise still resolves; does not throw). The message never appears in `queue` or `messages` — keep the composer text and show feedback if you want the user to retry. - - `"interrupt"` — abort the current stream and send the new message immediately. Unlike `stop()`, this does **not** clear already-queued messages — they still drain after the interrupting send succeeds. -- **`drain`** — how queued items leave the queue: `"fifo"` (default) sends them one at a time in order; `"batch"` merges everything currently queued into a single send once the run settles successfully (string contents joined with `\n`, multimodal content concatenated in order; when sending via `ChatClient` with per-message `body`, the last item's `body` wins — framework hooks do not forward per-send `body`). -- **`maxSize`** — caps how many messages can be queued (`0` means never queue). -- **`onOverflow`** — `"reject"` (default) silently ignores a send once `maxSize` is reached (does not throw); `"drop-oldest"` evicts the oldest queued item to make room. +**Must configure if you care:** -You can also pass a plain `WhenBusy` string (e.g. `queue: "interrupt"`) as shorthand for `{ whenBusy: "interrupt" }`, or a `QueueStrategy` function for per-send action control. Strategy form always drains FIFO (no `batch`); actions are `'queue' | 'drop' | 'interrupt'` (no concurrent streams). Per-call `whenBusy` overrides both config and strategy. +| Option | Values | Default | +| --- | --- | --- | +| `whenBusy` | `"queue"` \| `"drop"` \| `"interrupt"` | `"queue"` | +| `drain` | `"fifo"` \| `"batch"` | `"fifo"` | +| `maxSize` | number (`0` = never queue) | — | +| `onOverflow` | `"reject"` \| `"drop-oldest"` | `"reject"` | -### When the queue drains vs flushes +- **`interrupt`** — abort current stream, send immediately; does **not** clear the queue +- Shorthand: `queue: "interrupt"` → `{ whenBusy: "interrupt" }` +- Per-send override: `sendMessage("…", { whenBusy: "interrupt" })` -- **Drain (auto-send)** — only after a **successful** stream settle (including after tool continuations finish). -- **Flush (discard without sending)** — on **error/abort of the active generation** (user `stop()`, real stream errors), `clear()`, `unsubscribe()`, and `reload()`. Interrupt aborts the old run without flushing; remaining items drain after a **successful** interrupting turn. -- **`interrupt` does not flush** — existing queued items remain and drain after the interrupting turn succeeds. +**Drain vs flush:** -`useChat` exposes the pending queue as `queue` so you can render it distinctly from `messages`, along with `cancelQueued(id)` to cancel an item before it sends: +| Action | When | +| --- | --- | +| Drain (auto-send) | After successful stream settle | +| Flush (discard) | Error/abort of active generation, `clear()`, `unsubscribe()`, `reload()` | +| Interrupt | Keeps queue; drains after successful interrupting turn | + +Render pending items from `queue`: ```tsx group=queueing-messages function PendingQueue() { @@ -384,24 +346,16 @@ function PendingQueue() { } ``` -Override the configured policy for a single send with the second argument to `sendMessage`: - -```tsx group=queueing-messages -sendMessage("Never mind, do this instead", { whenBusy: "interrupt" }); -``` - -> **Note:** This is a default-behavior change — messages sent while streaming used to be silently dropped. They are now queued unless you opt into `queue: "drop"` (or `{ whenBusy: "drop" }`) to restore the old behavior, or `queue: "interrupt"`. +> **Note:** Busy sends used to drop silently. Opt into `queue: "drop"` to restore that. -## Best Practices +## Must-do checklist -1. **Handle loading states** - Use `isLoading` to show loading indicators -2. **Handle errors** - Check `error` state for stream failures -3. **Cancel on unmount** - Clean up streams when components unmount -4. **Optimize rendering** - Batch updates if needed for performance -5. **Show progress** - Display partial content as it streams -6. **Render queued messages distinctly** - Use `queue` to show pending sends separately from `messages` +1. Use `isLoading` / `error` for UI feedback +2. Call `stop()` (or honor abort) on unmount +3. Render partial content as it streams +4. Render `queue` separately from `messages` when queueing is on -## Next Steps +## Next -- [Connection Adapters](./connection-adapters) - Learn about different connection types -- [API Reference](../api/ai) - Explore streaming APIs +- [Connection Adapters](./connection-adapters) +- [API Reference](../api/ai) diff --git a/docs/chat/structured-outputs.md b/docs/chat/structured-outputs.md index 425dc56ab..74f4f1eb4 100644 --- a/docs/chat/structured-outputs.md +++ b/docs/chat/structured-outputs.md @@ -2,7 +2,7 @@ title: Structured Outputs (Moved) id: structured-outputs order: 4 -description: "Structured outputs documentation has moved to its own top-level Structured Outputs section, with separate guides for one-shot extraction, streaming UIs, multi-turn chat, and combining with tools." +description: "Redirect — structured outputs live under /structured-outputs/* by journey (one-shot, streaming, multi-turn, tools)." keywords: - tanstack ai - structured outputs @@ -10,12 +10,12 @@ keywords: - redirect --- -The structured-outputs guide has moved to its own top-level section, split by what you're building. Pick the journey that fits: +This page moved. Pick a journey: -- **[Overview](../structured-outputs/overview)** — what structured output is, schema library options, provider support, and "which page do I read?" -- **[One-Shot Extraction](../structured-outputs/one-shot)** — single prompt in, single typed object out. Use this when you don't need streaming or chat history. -- **[Streaming UIs](../structured-outputs/streaming)** — `useChat({ outputSchema })` with `partial` and `final` populating a UI field by field. -- **[Multi-Turn Chat](../structured-outputs/multi-turn)** — every assistant turn carries its own typed `StructuredOutputPart`, history stays renderable, and `messages[i].parts.find(p => p.type === "structured-output").data` is typed by your schema. -- **[With Tools](../structured-outputs/with-tools)** — combining `outputSchema` with the agent loop, including pause/resume for server-tool approvals and client-tool invocations. +1. **[Overview](../structured-outputs/overview)** — what it is, schemas, provider support +2. **[One-Shot Extraction](../structured-outputs/one-shot)** — single prompt → typed object +3. **[Streaming UIs](../structured-outputs/streaming)** — `useChat({ outputSchema })` field-by-field +4. **[Multi-Turn Chat](../structured-outputs/multi-turn)** — per-turn `StructuredOutputPart` +5. **[With Tools](../structured-outputs/with-tools)** — `outputSchema` + agent loop / approvals -> **Note:** This URL is kept for backward compatibility. New content lives under `/structured-outputs/*` — update existing bookmarks when you can. +> Kept for backward compatibility. Prefer `/structured-outputs/*`. diff --git a/docs/chat/thinking-content.md b/docs/chat/thinking-content.md index fdaa6864f..0fdde85ff 100644 --- a/docs/chat/thinking-content.md +++ b/docs/chat/thinking-content.md @@ -2,7 +2,7 @@ title: Thinking & Reasoning id: thinking-content order: 5 -description: "Render reasoning tokens from thinking models (Claude extended thinking, OpenAI o-series) as streamed ThinkingPart in TanStack AI chat UIs." +description: "Stream and render ThinkingPart from Claude extended thinking and OpenAI reasoning models." keywords: - tanstack ai - thinking @@ -14,13 +14,11 @@ keywords: - ThinkingPart --- -Some models expose their internal reasoning as "thinking" content -- Claude with extended thinking, OpenAI o-series models with reasoning, and others. TanStack AI captures this as `ThinkingPart` in messages, streamed to your UI in real-time alongside text and tool calls. +If the model exposes reasoning tokens → read `ThinkingPart` on `message.parts`. It is **UI-only** (never sent back to the model). -Thinking content is **UI-only**. It is never sent back to the model in subsequent requests. +## Shape -## How It Works - -When a model emits reasoning tokens, the adapter emits AG-UI events for them. Adapters emit `REASONING_MESSAGE_*` events (the preferred, canonical form) **and** the older `STEP_STARTED` / `STEP_FINISHED` events. The stream processor reconciles both into a single `ThinkingPart` on the assistant's `UIMessage`, deduplicating overlapping content. You should rely on the `ThinkingPart` in `message.parts` rather than hand-parsing the raw events: +Adapters emit `REASONING_MESSAGE_*` (canonical) and legacy `STEP_*` events. The stream processor merges both into one `ThinkingPart` — use that, not raw events: ```typescript interface ThinkingPart { @@ -31,15 +29,11 @@ interface ThinkingPart { } ``` -The `ThinkingPart` appears in `UIMessage.parts` alongside `TextPart` and `ToolCallPart` entries. As reasoning tokens arrive, its `content` accumulates token by token. - -## Enabling Thinking +## Enable thinking -How you enable thinking depends on the provider. +### Anthropic (extended thinking) -### Anthropic (Extended Thinking) - -Pass the `thinking` option in `modelOptions` with `type: "enabled"` and a `budget_tokens` (minimum 1024). Keep `budget_tokens` below `modelOptions.max_tokens` so there is room for the visible response in addition to the thinking budget: +`budget_tokens` ≥ 1024 and **below** `max_tokens`: ```typescript import { chat, toServerSentEventsResponse } from "@tanstack/ai"; @@ -52,7 +46,6 @@ export async function POST(request: Request) { messages, modelOptions: { max_tokens: 32000, - // budget_tokens must be at least 1024 and below max_tokens thinking: { type: "enabled", budget_tokens: 10000 }, }, }); @@ -60,9 +53,9 @@ export async function POST(request: Request) { } ``` -### OpenAI (Reasoning Models) +### OpenAI (reasoning models) -OpenAI o-series models (o1, o3, o3-mini, o3-pro) perform reasoning automatically. You can control the depth with the `reasoning` option: +o-series reasons automatically. Control depth with `reasoning`: ```typescript import { chat, toServerSentEventsResponse } from "@tanstack/ai"; @@ -84,9 +77,9 @@ export async function POST(request: Request) { } ``` -When `reasoning.summary` is set, the adapter streams reasoning summary text as thinking content. Without it, reasoning tokens are still used internally but may not be surfaced depending on the model. +With `reasoning.summary` set, summary text streams as thinking. Without it, reasoning may stay internal. -GPT-5 and later models also support reasoning. Their `reasoning.effort` accepts `"none" | "minimal" | "low" | "medium" | "high"`, and reasoning activates on any non-`none` value: +GPT-5+ also supports reasoning — any non-`none` effort activates it: ```typescript import { chat, toServerSentEventsResponse } from "@tanstack/ai"; @@ -105,9 +98,9 @@ export async function POST(request: Request) { } ``` -## Rendering in React +## Render in React -Thinking parts appear in `message.parts` just like text and tool calls. A common pattern is to render them in a collapsible element so they don't dominate the UI: +Collapse thinking so it does not dominate the UI: ```tsx import type { UIMessage } from "@tanstack/ai-react"; @@ -134,24 +127,22 @@ function MessageContent({ message }: { message: UIMessage }) { } ``` -The [Quick Start](../getting-started/quick-start) guide shows a simpler inline pattern where thinking is rendered as italic text above the response. - -## Streaming Behavior +Inline italic pattern: [Quick Start](../getting-started/quick-start). -Thinking content streams **before** the final text response. As reasoning tokens arrive, `ThinkingPart.content` accumulates token by token, the same way `TextPart.content` does for the response text. +## Streaming order -The typical streaming order is: +Thinking accumulates **before** visible text: -1. The reasoning block begins (`REASONING_MESSAGE_START`, plus a legacy `STEP_STARTED`) -2. Reasoning tokens stream in (`REASONING_MESSAGE_CONTENT`, plus legacy `STEP_FINISHED` events), accumulating into `ThinkingPart.content` -3. `TEXT_MESSAGE_START` -- the model begins its visible response -4. `TEXT_MESSAGE_CONTENT` (repeated) -- the response text streams in +1. Reasoning block starts (`REASONING_MESSAGE_START` + legacy `STEP_STARTED`) +2. Tokens stream (`REASONING_MESSAGE_CONTENT` + legacy `STEP_FINISHED`) → `ThinkingPart.content` +3. `TEXT_MESSAGE_START` — visible response begins +4. `TEXT_MESSAGE_CONTENT` — text streams -Adapters emit both the canonical `REASONING_MESSAGE_*` events and the older `STEP_*` events; the stream processor reconciles them into one `ThinkingPart` so you never have to hand-parse the raw events. If you use `useChat` from `@tanstack/ai-react` (or the Solid/Vue/Svelte equivalents), your `messages` array updates automatically with both thinking and text parts as they arrive. +`useChat` (and Solid/Vue/Svelte hooks) update `messages` automatically. -## Next Steps +## Next -- [Streaming](./streaming) -- Connection adapters and stream events -- [Agentic Cycle](./agentic-cycle) -- How thinking interacts with tool-calling loops -- [Anthropic Adapter](../adapters/anthropic) -- Full Anthropic provider options -- [OpenAI Adapter](../adapters/openai) -- Full OpenAI provider options +- [Streaming](./streaming) +- [Agentic Cycle](./agentic-cycle) +- [Anthropic Adapter](../adapters/anthropic) +- [OpenAI Adapter](../adapters/openai) diff --git a/docs/code-mode/client-integration.md b/docs/code-mode/client-integration.md index 6e6928c45..856a5434d 100644 --- a/docs/code-mode/client-integration.md +++ b/docs/code-mode/client-integration.md @@ -2,7 +2,7 @@ title: Showing Code Mode in the UI id: code-mode-client-integration order: 2 -description: "Stream Code Mode execution events to your React app — console output, external calls, and results as they happen, via onCustomEvent." +description: "Stream Code Mode events to React — console, external calls, results via onCustomEvent." keywords: - tanstack ai - code mode @@ -13,27 +13,23 @@ keywords: - execution progress --- -You have [Code Mode](./code-mode) working on your server — the LLM writes and executes TypeScript, and you get results back. But your users see nothing while the sandbox runs. By the end of this guide, your React app will show real-time execution progress: console output, external function calls, and final results as they stream in. +# Showing Code Mode in the UI -## How events reach the client +If Code Mode works on the server but the UI is silent → wire `onCustomEvent` and render an execution panel. -When code runs inside the sandbox, Code Mode emits **custom events** through the AG-UI streaming protocol. These events travel alongside normal chat chunks (text, tool calls) and arrive in your client via the `onCustomEvent` callback. +## Events -The events emitted during each `execute_typescript` call: +Events ride the AG-UI stream with chat chunks. Each includes `toolCallId` for the matching `execute_typescript` call. | Event | When | Key fields | |-------|------|------------| -| `code_mode:execution_started` | Sandbox begins executing | `timestamp`, `codeLength` | -| `code_mode:console` | Each `console.log/error/warn/info` | `level`, `message`, `timestamp` | -| `code_mode:external_call` | Before an `external_*` function runs | `function`, `args`, `timestamp` | -| `code_mode:external_result` | After a successful `external_*` call | `function`, `result`, `duration` | -| `code_mode:external_error` | When an `external_*` call fails | `function`, `error`, `duration` | +| `code_mode:execution_started` | Sandbox start | `timestamp`, `codeLength` | +| `code_mode:console` | console.* | `level`, `message`, `timestamp` | +| `code_mode:external_call` | Before `external_*` | `function`, `args`, `timestamp` | +| `code_mode:external_result` | After success | `function`, `result`, `duration` | +| `code_mode:external_error` | After failure | `function`, `error`, `duration` | -Every event includes a `toolCallId` that ties it to the specific `execute_typescript` tool call, so you can render events alongside the right message. - -## Listening to events with useChat - -Pass an `onCustomEvent` callback to `useChat`. The callback receives the event type, payload, and a context object with the `toolCallId`: +## 1. Listen with `useChat` ```tsx group=code-mode-client import { useCallback, useRef, useState } from "react"; @@ -83,15 +79,11 @@ export function CodeModeChat() { onCustomEvent: handleCustomEvent, }); - // Render messages with events — see next section + // pass messages + toolCallEvents to MessageList } ``` -Events are keyed by `toolCallId` so each `execute_typescript` call gets its own event timeline. - -## Rendering execution progress - -When rendering messages, check for `execute_typescript` tool calls and display their events: +## 2. Render `execute_typescript` parts ```tsx group=code-mode-client function MessageList({ @@ -138,9 +130,7 @@ function MessageList({ } ``` -## Building an execution panel - -Here's a complete `CodeExecutionPanel` component that shows the generated code, live event stream, and final result: +## 3. Execution panel ```tsx group=code-mode-client function CodeExecutionPanel({ @@ -151,12 +141,16 @@ function CodeExecutionPanel({ }: { code?: string; events: Array; - result?: { success: boolean; result?: unknown; logs?: string[]; error?: { message: string } }; + result?: { + success: boolean; + result?: unknown; + logs?: string[]; + error?: { message: string }; + }; isRunning: boolean; }) { return (
- {/* Generated code */} {code && (
@@ -168,7 +162,6 @@ function CodeExecutionPanel({
)} - {/* Live event stream */} {events.length > 0 && (
@@ -185,7 +178,6 @@ function CodeExecutionPanel({
)} - {/* Final result */} {result && (
- → {String(data.function)}( - {JSON.stringify(data.args)}) + → {String(data.function)}({JSON.stringify(data.args)})
); @@ -270,17 +261,14 @@ function EventLine({ event }: { event: VMEvent }) { } ``` -This gives you: -- A collapsible code block showing the TypeScript the model wrote -- A live event log showing console output, external function calls with arguments, results with durations, and errors -- A status-colored result panel with logs and the return value +Panel shows: generated code · live console/calls · final result/logs/error. -## Adapting for other frameworks +## Other frameworks -The `onCustomEvent` callback is available through `ChatClient` from `@tanstack/ai-client`, which all framework integrations use under the hood. In Solid, Vue, or Svelte, pass `onCustomEvent` in the same way you pass it to `useChat` in React — the callback signature is identical: +Same callback signature via `ChatClient` / framework wrappers: ```typescript ignore (eventType: string, data: unknown, context: { toolCallId?: string }) => void ``` -See [Code Mode](./code-mode) for setting up the server side, and [Code Mode with Skills](./code-mode-with-skills) for adding persistent skill libraries. +Server setup: [Code Mode](./code-mode). Skills events: [Code Mode with Skills](./code-mode-with-skills). diff --git a/docs/code-mode/code-mode-isolates.md b/docs/code-mode/code-mode-isolates.md index c7dcaec4e..72e956b6b 100644 --- a/docs/code-mode/code-mode-isolates.md +++ b/docs/code-mode/code-mode-isolates.md @@ -2,7 +2,7 @@ title: Code Mode Isolate Drivers id: code-mode-isolates order: 4 -description: "Compare Code Mode sandbox drivers — Node isolated-vm, QuickJS WASM, and Cloudflare Workers — and choose the right runtime for your deployment." +description: "Pick a Code Mode sandbox: Node isolated-vm, QuickJS WASM, or Cloudflare Workers." keywords: - tanstack ai - code mode @@ -14,108 +14,76 @@ keywords: - secure execution --- -Isolate drivers provide the secure sandbox runtimes that [Code Mode](./code-mode.md) uses to execute generated TypeScript. All drivers implement the same `IsolateDriver` interface, so you can swap them without changing any other code. +# Code Mode isolate drivers -## Choosing a Driver +If you need a sandbox for [Code Mode](./code-mode.md) → pick a driver. All implement `IsolateDriver` (swappable). + +## Choose | | Node (`isolated-vm`) | QuickJS (WASM) | Cloudflare Workers | |---|---|---|---| -| **Best for** | Server-side Node.js apps | Browsers, edge, portability | Edge deployments on Cloudflare | -| **Performance** | Fast (V8 JIT) | Slower (interpreted) | Fast (V8 on Cloudflare edge) | -| **Native deps** | Yes (C++ addon) | None | None | -| **Browser support** | No | Yes | N/A | -| **Memory limit** | Configurable | Configurable | N/A | -| **Stack size limit** | N/A | Configurable | N/A | -| **Setup** | `pnpm add` | `pnpm add` | Deploy a Worker first | - ---- - -## Node.js Driver (`@tanstack/ai-isolate-node`) - -Uses V8 isolates via the [`isolated-vm`](https://github.com/laverdet/isolated-vm) native addon. This is the fastest option for server-side Node.js applications because generated code runs in the same V8 engine as the host, under JIT compilation, with no serialization overhead beyond tool call boundaries. +| **Best for** | Node servers | Browser / portable edge | Cloudflare edge | +| **Speed** | Fast (V8 JIT) | Slower (interpreted) | Fast (edge V8) | +| **Native deps** | Yes (C++) | None | None | +| **Browser** | No | Yes | N/A | +| **Setup** | `pnpm add` | `pnpm add` | Deploy Worker first | -### Installation +## Node (`@tanstack/ai-isolate-node`) ```bash pnpm add @tanstack/ai-isolate-node ``` -`isolated-vm` is a native C++ addon and must be compiled for your platform. It requires Node.js 18 or later. - -### Usage +Requires Node 18+ (native addon). ```typescript import { createNodeIsolateDriver } from '@tanstack/ai-isolate-node' const driver = createNodeIsolateDriver({ - memoryLimit: 128, // MB - timeout: 30_000, // ms + memoryLimit: 128, // MB + timeout: 30_000, // ms }) ``` -### Options - | Option | Type | Default | Description | |--------|------|---------|-------------| -| `memoryLimit` | `number` | `128` | Maximum heap size for the V8 isolate, in megabytes. Execution is terminated if this limit is exceeded. | -| `timeout` | `number` | `30000` | Maximum wall-clock time per execution, in milliseconds. | +| `memoryLimit` | `number` | `128` | Heap MB | +| `timeout` | `number` | `30000` | Wall-clock ms | -### How it works +Each `execute_typescript` gets a fresh V8 isolate. Tools bridge as async refs (`external_*`). Console captured. Isolate destroyed after the call. -Each `execute_typescript` call creates a fresh V8 isolate. Your tools are bridged into the isolate as async reference functions — when generated code calls `external_myTool(...)`, the call crosses the isolate boundary back into the host Node.js process, executes your tool implementation, and returns the result. Console output (`log`, `error`, `warn`, `info`) is captured and returned in the execution result. The isolate is destroyed after each call. - ---- - -## QuickJS Driver (`@tanstack/ai-isolate-quickjs`) - -Uses [QuickJS](https://bellard.org/quickjs/) compiled to WebAssembly via Emscripten. Because the sandbox is a WASM module, it has no native dependencies and runs anywhere JavaScript runs: Node.js, browsers, Deno, Bun, and Cloudflare Workers (without deploying a separate Worker). - -### Installation +## QuickJS (`@tanstack/ai-isolate-quickjs`) ```bash pnpm add @tanstack/ai-isolate-quickjs ``` -### Usage - ```typescript import { createQuickJSIsolateDriver } from '@tanstack/ai-isolate-quickjs' const driver = createQuickJSIsolateDriver({ - memoryLimit: 128, // MB - timeout: 30_000, // ms - maxStackSize: 524288, // bytes (512 KiB) + memoryLimit: 128, + timeout: 30_000, + maxStackSize: 524288, // 512 KiB }) ``` -### Options - | Option | Type | Default | Description | |--------|------|---------|-------------| -| `memoryLimit` | `number` | `128` | Maximum heap memory for the QuickJS VM, in megabytes. | -| `timeout` | `number` | `30000` | Maximum wall-clock time per execution, in milliseconds. | -| `maxStackSize` | `number` | `524288` | Maximum call stack size in bytes (default: 512 KiB). Increase for deeply recursive code; decrease to catch runaway recursion sooner. | +| `memoryLimit` | `number` | `128` | Heap MB | +| `timeout` | `number` | `30000` | Wall-clock ms | +| `maxStackSize` | `number` | `524288` | Stack bytes | -### How it works +Asyncified WASM; executions serialized through a global queue. Fatals dispose the VM and return a structured error. Compute-heavy scripts slower than Node; tool-wait heavy scripts are fine. -QuickJS WASM uses an asyncified execution model — the WASM module can pause while awaiting host async functions (your tools). Executions are serialized through a global queue to prevent concurrent WASM calls, which the asyncify model does not support. Fatal errors (memory exhaustion, stack overflow) are detected, the VM is disposed, and a structured error is returned. Console output is captured and returned with the result. +## Cloudflare (`@tanstack/ai-isolate-cloudflare`) -> **Performance note:** QuickJS interprets JavaScript rather than JIT-compiling it, so compute-heavy scripts run slower than with the Node driver. For typical LLM-generated scripts that are mostly waiting on `external_*` tool calls, this difference is not significant. - ---- - -## Cloudflare Workers Driver (`@tanstack/ai-isolate-cloudflare`) - -Runs generated code inside a [Cloudflare Worker](https://workers.cloudflare.com/) at the edge. Your application server sends code and tool schemas to the Worker via HTTP; the Worker executes the code and calls back when it needs a tool result. This keeps your tool implementations on your server while sandboxed execution happens on Cloudflare's global network. - -### Installation +Tools stay on your server; code runs on the edge Worker. ```bash pnpm add @tanstack/ai-isolate-cloudflare ``` -### Usage - ```typescript import { createCloudflareIsolateDriver } from '@tanstack/ai-isolate-cloudflare' @@ -127,18 +95,14 @@ const driver = createCloudflareIsolateDriver({ }) ``` -### Options - | Option | Type | Default | Description | |--------|------|---------|-------------| -| `workerUrl` | `string` | — | **Required.** Full URL of the deployed Cloudflare Worker. | -| `authorization` | `string` | — | Optional value sent as the `Authorization` header on every request. Use this to prevent unauthorized access to your Worker. | -| `timeout` | `number` | `30000` | Maximum wall-clock time for the entire execution (including all tool round-trips), in milliseconds. | -| `maxToolRounds` | `number` | `10` | Maximum number of tool-call/result cycles per execution. Prevents infinite loops when generated code calls tools in a loop. | - -### Deploying the Worker +| `workerUrl` | `string` | required | Worker URL | +| `authorization` | `string` | — | `Authorization` header | +| `timeout` | `number` | `30000` | Full run incl. tool RTs | +| `maxToolRounds` | `number` | `10` | Max tool cycles | -The package exports a ready-made Worker handler at `@tanstack/ai-isolate-cloudflare/worker`. Create a `wrangler.toml` and a worker entry file: +### Deploy Worker ```toml # wrangler.toml @@ -155,36 +119,13 @@ bindings = [{ name = "eval", type = "eval" }] export { default } from '@tanstack/ai-isolate-cloudflare/worker' ``` -Deploy: - ```bash wrangler deploy ``` -### How it works - -The driver implements a request/response loop for tool execution: - -``` -Driver (your server) Worker (Cloudflare edge) -───────────────────── ───────────────────────── -Send: code + tool schemas ──────▶ Execute code - ◀────── Return: needs tool X with args Y -Execute tool X locally -Send: tool result ──────▶ Resume execution - ◀────── Return: final result / needs tool Z -...repeat until done... -``` - -Each round-trip adds network latency, so the `maxToolRounds` limit both prevents runaway scripts and caps the maximum number of cross-continent hops. Console output from all rounds is aggregated and returned in the final result. - -> **Security:** The Worker requires `UNSAFE_EVAL` (local dev) or the `eval` unsafe binding (production) to execute arbitrary code. Restrict access using the `authorization` option or Cloudflare Access policies. - ---- - -## The `IsolateDriver` Interface +Loop: send code + schemas → Worker needs tool → host runs tool → send result → repeat. Logs aggregated. Secure with `authorization` or Cloudflare Access. Needs `UNSAFE_EVAL` / `eval` binding. -All three drivers satisfy this interface, exported from `@tanstack/ai-code-mode`: +## `IsolateDriver` interface ```typescript import type { ToolBinding, NormalizedError } from "@tanstack/ai-code-mode"; @@ -212,10 +153,8 @@ interface ExecutionResult { } ``` -You can implement this interface to build a custom driver — for example, a Docker-based sandbox or a Deno subprocess. +Implement for custom sandboxes (Docker, Deno, etc.). -## Next Steps +## Next -- [Code Mode](./code-mode) — Core setup, API reference, and getting started guide -- [Showing Code Mode in the UI](./client-integration) — Display execution progress in your React app -- [Code Mode with Skills](./code-mode-with-skills) — Add persistent, reusable skill libraries +- [Code Mode](./code-mode) · [Client UI](./client-integration) · [Skills](./code-mode-with-skills) diff --git a/docs/code-mode/code-mode-with-skills.md b/docs/code-mode/code-mode-with-skills.md index f5dfea65a..a58a9a981 100644 --- a/docs/code-mode/code-mode-with-skills.md +++ b/docs/code-mode/code-mode-with-skills.md @@ -2,7 +2,7 @@ title: Code Mode with Skills id: code-mode-with-skills order: 3 -description: "Teach Code Mode to save and reuse working code as named skills backed by persistent storage — faster follow-up requests and composable agent memory." +description: "Save and reuse TypeScript snippets as skills — storage, selection, register_skill, trust." keywords: - tanstack ai - code mode @@ -14,66 +14,38 @@ keywords: - skill storage --- -Skills extend [Code Mode](./code-mode.md) with a persistent library of reusable TypeScript snippets. When the LLM writes a useful piece of code — say, a function that fetches and ranks NPM packages — it can save that code as a _skill_. On future requests, relevant skills are loaded from storage and made available as first-class tools the LLM can call without re-writing the logic. +# Code Mode with Skills -> **Different from agent-authoring skills.** The skills on this page are _runtime_ snippets the chat LLM saves and reuses. If you're looking to teach your coding assistant (Claude Code, Cursor, etc.) how TanStack AI itself works, see [Agent Skills (TanStack Intent)](../getting-started/agent-skills). +If Code Mode works and you want reusable snippets → skills: the model saves working code and reloads it on later requests. -## Overview +> Runtime skills (this page) ≠ [Agent Skills / TanStack Intent](../getting-started/agent-skills) for coding assistants. -The skills system has two integration paths: +## Paths -| Approach | Entry point | Skill selection | Best for | -|----------|-------------|----------------|----------| -| **High-level** | `codeModeWithSkills()` | Automatic (LLM-based) | New projects, turnkey setup | -| **Manual** | Individual functions (`skillsToTools`, `createSkillManagementTools`, etc.) | You decide which skills to load | Full control, existing setups | +| Approach | Entry | Selection | When | +|----------|-------|-----------|------| +| High-level | `codeModeWithSkills()` | Auto (cheap LLM) | New projects | +| Manual | `skillsToTools`, etc. | You load skills | Full control | -Both paths share the same storage, trust, and execution primitives — they differ only in how skills are selected and assembled. +## Request flow (high-level) -## How It Works +1. Load skill index (metadata only) +2. Select relevant skills (cheap model) +3. Build registry: `execute_typescript` + management tools + selected skill tools +4. System prompt = Code Mode stubs + skill docs +5. Main `chat()` with strong model -A request with skills enabled goes through these stages: +**Two LLM calls:** selection (metadata, last 5 messages) + main chat. Empty storage/messages short-circuits selection. -``` -┌─────────────────────────────────────────────────────┐ -│ 1. Load skill index (metadata only, no code) │ -├─────────────────────────────────────────────────────┤ -│ 2. Select relevant skills (LLM call — fast model) │ -├─────────────────────────────────────────────────────┤ -│ 3. Build tool registry │ -│ ├── execute_typescript (Code Mode sandbox) │ -│ ├── search_skills / get_skill / register_skill │ -│ └── skill tools (one per selected skill) │ -├─────────────────────────────────────────────────────┤ -│ 4. Generate system prompt │ -│ ├── Code Mode type stubs │ -│ └── Skill library documentation │ -├─────────────────────────────────────────────────────┤ -│ 5. Main chat() call (strong model) │ -│ ├── Can call skill tools directly │ -│ ├── Can write code via execute_typescript │ -│ └── Can register new skills for future use │ -└─────────────────────────────────────────────────────┘ -``` - -### LLM calls - -There are **two** LLM interactions per request when using the high-level API: - -1. **Skill selection** (`selectRelevantSkills`) — A single chat call using the adapter you provide. It sends the last 5 conversation messages plus a catalog of skill names/descriptions, and asks the model to return a JSON array of relevant skill names. This should be a cheap/fast model (e.g., `gpt-4o-mini`, `claude-haiku-4-5`). - -2. **Main chat** — The primary `chat()` call with your full model. This is where the LLM reasons, calls tools, writes code, and registers skills. +## High-level: `codeModeWithSkills()` -The selection call is lightweight — it only sees skill metadata (names, descriptions, usage hints), not full code. If there are no skills in storage or no messages, it short-circuits and skips the LLM call entirely. - -## High-Level API: `codeModeWithSkills()` - -### Installation +### 1. Install ```bash pnpm add @tanstack/ai-code-mode-skills ``` -### Usage +### 2. Wire ```typescript import { chat, maxIterations, toServerSentEventsStream } from '@tanstack/ai' @@ -87,23 +59,24 @@ const messages = [{ role: 'user' as const, content: 'Hello' }] const storage = createFileSkillStorage({ directory: './.skills' }) const driver = createNodeIsolateDriver() -const { toolsRegistry, systemPrompt, selectedSkills } = await codeModeWithSkills({ - config: { - driver, - tools: [myTool1, myTool2], - timeout: 60_000, - memoryLimit: 128, - }, - adapter: openaiText('gpt-5-mini'), // cheap model for skill selection - skills: { - storage, - maxSkillsInContext: 5, - }, - messages, // current conversation -}) +const { toolsRegistry, systemPrompt, selectedSkills } = + await codeModeWithSkills({ + config: { + driver, + tools: [myTool1, myTool2], + timeout: 60_000, + memoryLimit: 128, + }, + adapter: openaiText('gpt-5-mini'), // selection + skills: { + storage, + maxSkillsInContext: 5, + }, + messages, + }) const stream = chat({ - adapter: openaiText('gpt-5.5'), // strong model for reasoning + adapter: openaiText('gpt-5.5'), // main tools: toolsRegistry.getTools(), messages, systemPrompts: ['You are a helpful assistant.', systemPrompt], @@ -111,27 +84,17 @@ const stream = chat({ }) ``` -`codeModeWithSkills` returns: - | Property | Type | Description | |----------|------|-------------| -| `toolsRegistry` | `ToolRegistry` | Mutable registry containing all tools. Pass to `chat()` via `tools: toolsRegistry.getTools()`. | -| `systemPrompt` | `string` | Combined Code Mode + skill library documentation. | -| `selectedSkills` | `Array` | Skills the selection model chose for this conversation. | - -### What goes into the registry - -The registry is populated with: +| `toolsRegistry` | `ToolRegistry` | `tools: toolsRegistry.getTools()` | +| `systemPrompt` | `string` | Code Mode + skill docs | +| `selectedSkills` | `Array` | Chosen for this turn | -- **`execute_typescript`** — The Code Mode sandbox tool. Inside the sandbox, skills are also available as `skill_*` functions (loaded dynamically at execution time). -- **`search_skills`** — Search the skill library by query. Returns matching skill metadata. -- **`get_skill`** — Retrieve full details (including code) for a specific skill. -- **`register_skill`** — Save working code as a new skill. Newly registered skills are immediately added to the registry as callable tools. -- **One tool per selected skill** — Each selected skill becomes a direct tool (prefixed with `[SKILL]` in its description) that the LLM can call without going through `execute_typescript`. +**Registry contents:** `execute_typescript` · `search_skills` · `get_skill` · `register_skill` · one tool per selected skill (`[SKILL]` prefix). Skills also bind as `skill_*` inside the sandbox. ## Manual API -If you want full control — for example, loading all skills instead of using LLM-based selection — use the lower-level functions directly. This is the approach used in the `ts-code-mode-web` example. +Skip selection LLM; load what you want (as in `ts-code-mode-web`): ```typescript import { chat, maxIterations } from '@tanstack/ai' @@ -155,44 +118,39 @@ const storage = createFileSkillStorage({ }) const driver = createNodeIsolateDriver() -// 1. Create Code Mode tool + prompt -const { tool: codeModeTool, systemPrompt: codeModePrompt } = - createCodeMode({ - driver, - tools: [myTool1, myTool2], - timeout: 60_000, - memoryLimit: 128, - }) +const { tool: codeModeTool, systemPrompt: codeModePrompt } = createCodeMode({ + driver, + tools: [myTool1, myTool2], + timeout: 60_000, + memoryLimit: 128, +}) -// 2. Load all skills and convert to tools const allSkills = await storage.loadAll() const skillIndex = await storage.loadIndex() -const skillTools = allSkills.length > 0 - ? skillsToTools({ - skills: allSkills, - driver, - tools: [myTool1, myTool2], - storage, - timeout: 60_000, - memoryLimit: 128, - }) - : [] +const skillTools = + allSkills.length > 0 + ? skillsToTools({ + skills: allSkills, + driver, + tools: [myTool1, myTool2], + storage, + timeout: 60_000, + memoryLimit: 128, + }) + : [] -// 3. Create management tools const managementTools = createSkillManagementTools({ storage, trustStrategy, }) -// 4. Generate skill library prompt const skillsPrompt = createSkillsSystemPrompt({ selectedSkills: allSkills, totalSkillCount: skillIndex.length, skillsAsTools: true, }) -// 5. Assemble and call chat() const stream = chat({ adapter: openaiText('gpt-5.5'), tools: [codeModeTool, ...managementTools, ...skillTools], @@ -202,44 +160,23 @@ const stream = chat({ }) ``` -This approach skips the selection LLM call entirely — you load whichever skills you want and pass them in directly. - -## Skill Storage - -Skills are persisted through the `SkillStorage` interface. Two implementations are provided: +## Storage -### File storage (production) - -`createFileSkillStorage` is Node-only — it imports `node:fs` / `node:path` — so -it lives behind the `/storage` subpath rather than the package root. This keeps -the root export safe to bundle for Cloudflare Workers and browser builds; only -reach for the subpath in a Node runtime. +### File (Node only — `/storage` subpath) ```typescript import { createFileSkillStorage } from '@tanstack/ai-code-mode-skills/storage' import { createDefaultTrustStrategy } from '@tanstack/ai-code-mode-skills' -const trustStrategy = createDefaultTrustStrategy() const storage = createFileSkillStorage({ directory: './.skills', - trustStrategy, // optional, defaults to createDefaultTrustStrategy() + trustStrategy: createDefaultTrustStrategy(), }) ``` -Creates a directory structure: - -``` -.skills/ - _index.json # Lightweight catalog for fast loading - fetch_github_stats/ - meta.json # Description, schemas, hints, stats - code.ts # TypeScript source - compare_npm_packages/ - meta.json - code.ts -``` +Layout: `_index.json` + per-skill `meta.json` / `code.ts`. -### Memory storage (testing & edge runtimes) +### Memory (tests / edge / Workers) ```typescript import { createMemorySkillStorage } from '@tanstack/ai-code-mode-skills' @@ -247,27 +184,25 @@ import { createMemorySkillStorage } from '@tanstack/ai-code-mode-skills' const storage = createMemorySkillStorage() ``` -Keeps everything in memory — no `node:fs` dependency, so it is re-exported from -the package root and is safe to use in Workers and browsers. Useful for tests, -demos, and edge deployments. (It is also available from the `/storage` subpath.) - -### Storage interface - -Both implementations satisfy this interface: +### Interface | Method | Description | |--------|-------------| -| `loadIndex()` | Load lightweight metadata for all skills (no code) | -| `loadAll()` | Load all skills with full details including code | -| `get(name)` | Get a single skill by name | -| `save(skill)` | Create or update a skill | -| `delete(name)` | Remove a skill | -| `search(query, options?)` | Search skills by text query | -| `updateStats(name, success)` | Record an execution result for trust tracking | +| `loadIndex()` / `loadAll()` / `get(name)` | Read | +| `save(skill)` / `delete(name)` | Write | +| `search(query, options?)` | Text search | +| `updateStats(name, success)` | Trust stats | -## Trust Strategies +## Trust strategies -Skills start untrusted and earn trust through successful executions. The trust level is metadata only — it does not currently gate execution. Four built-in strategies are available: +Metadata only — does not gate execution. + +| Strategy | Initial | Provisional | Trusted | +|----------|---------|-------------|---------| +| Default | `untrusted` | 10+ runs, ≥90% | 100+ runs, ≥95% | +| Relaxed | `untrusted` | 3+ / ≥80% | 10+ / ≥90% | +| Always trusted | `trusted` | — | — | +| Custom | configurable | configurable | configurable | ```typescript group=code-mode-with-skills import { @@ -276,16 +211,7 @@ import { createRelaxedTrustStrategy, createCustomTrustStrategy, } from '@tanstack/ai-code-mode-skills' -``` - -| Strategy | Initial level | Provisional | Trusted | -|----------|--------------|-------------|---------| -| **Default** | `untrusted` | 10+ runs, ≥90% success | 100+ runs, ≥95% success | -| **Relaxed** | `untrusted` | 3+ runs, ≥80% success | 10+ runs, ≥90% success | -| **Always trusted** | `trusted` | — | — | -| **Custom** | Configurable | Configurable | Configurable | -```typescript group=code-mode-with-skills const strategy = createCustomTrustStrategy({ initialLevel: 'untrusted', provisionalThreshold: { executions: 5, successRate: 0.85 }, @@ -293,76 +219,37 @@ const strategy = createCustomTrustStrategy({ }) ``` -## Skill Lifecycle - -### Registration - -When the LLM produces useful code via `execute_typescript`, the system prompt instructs it to call `register_skill` with: - -- `name` — snake_case identifier (becomes the tool name) -- `description` — what the skill does -- `code` — TypeScript source that receives an `input` variable -- `inputSchema` / `outputSchema` — JSON Schema strings -- `usageHints` — when to use this skill -- `dependsOn` — other skills this one calls - -The skill is saved to storage and (if a `ToolRegistry` was provided) immediately added as a callable tool in the current session. - -### Execution - -When a skill tool is called, the system: - -1. Wraps the skill code with `const input = ;` -2. Strips TypeScript syntax to plain JavaScript -3. Creates a fresh sandbox context with `external_*` bindings -4. Executes the code and returns the result -5. Updates execution stats (success/failure count) asynchronously - -### Selection (high-level API only) - -On each new request, `selectRelevantSkills`: - -1. Takes the last 5 conversation messages as context -2. Builds a catalog from the skill index (name + description + first usage hint) -3. Asks the adapter to return a JSON array of relevant skill names (max `maxSkillsInContext`) -4. Loads full skill data for the selected names - -If parsing fails or the model returns invalid JSON, it falls back to an empty selection — the request proceeds without pre-loaded skills, but the LLM can still search and use skills via the management tools. +## Lifecycle -## Skills as Tools vs. Sandbox Bindings +**Register** via `register_skill`: `name`, `description`, `code`, schemas, `usageHints`, `dependsOn`. High-level registry adds the tool immediately. -The `skillsAsTools` option (default: `true`) controls how skills are exposed: +**Execute:** wrap input → strip TS → sandbox with `external_*` → return → async stats. -| Mode | How the LLM calls a skill | Pros | Cons | -|------|--------------------------|------|------| -| **As tools** (`true`) | Direct tool call: `skill_name({ ... })` | Simpler for the LLM, shows in tool-call UI, proper input validation | One tool per skill in the tool list | -| **As bindings** (`false`) | Inside `execute_typescript`: `await skill_fetch_data({ ... })` | Skills composable in code, fewer top-level tools | LLM must write code to use them | +**Select (high-level):** last 5 messages + catalog → JSON names (max `maxSkillsInContext`) → load full skills. Parse failure → empty selection; management tools still work. -When `skillsAsTools` is enabled, the system prompt documents each skill with its schema, usage hints, and example calls. When disabled, skills appear as typed `skill_*` functions in the sandbox type stubs. +## Skills as tools vs bindings -## Custom Events +| Mode | Call style | Tradeoff | +|------|------------|----------| +| `skillsAsTools: true` (default) | Direct tool call | Simpler UI; more tools | +| `false` | Inside `execute_typescript` as `skill_*` | Composable in code; fewer top-level tools | -Skill execution emits events through the TanStack AI event system: +## Events -| Event | When | Payload | -|-------|------|---------| -| `code_mode:skill_call` | Skill tool invoked | `{ skill, input, timestamp }` | -| `code_mode:skill_result` | Skill completed successfully | `{ skill, result, duration, timestamp }` | -| `code_mode:skill_error` | Skill execution failed | `{ skill, error, duration, timestamp }` | -| `skill:registered` | New skill saved via `register_skill` | `{ id, name, description, timestamp }` | +| Event | When | +|-------|------| +| `code_mode:skill_call` / `:skill_result` / `:skill_error` | Skill tool lifecycle | +| `skill:registered` | New skill saved | -To render these events in your React app alongside Code Mode execution events, see [Showing Code Mode in the UI](./client-integration). +Render with Code Mode events: [Client integration](./client-integration). ## Tips -- **Use a cheap model for selection.** The selection call only needs to match skill names to conversation context — `gpt-4o-mini` or `claude-haiku-4-5` work well. -- **Start without skills.** Get Code Mode working first, then add `@tanstack/ai-code-mode-skills` once you have tools that produce reusable patterns. -- **Monitor the skill count.** As the library grows, consider increasing `maxSkillsInContext` or switching to the manual API where you control which skills load. -- **Newly registered skills are available on the next message,** not in the current turn's tool list (unless using `ToolRegistry` with the high-level API, which adds them immediately). -- **Skills can call other skills.** Inside the sandbox, both `external_*` and `skill_*` functions are available. Set `dependsOn` when registering to document these relationships. +1. Cheap model for selection (`gpt-4o-mini`, `claude-haiku-4-5`). +2. Get Code Mode working before adding skills. +3. Watch library size — raise `maxSkillsInContext` or use manual API. +4. Skills can call `external_*` and `skill_*`; set `dependsOn` when registering. -## Next Steps +## Next -- [Code Mode](./code-mode) — Core Code Mode setup and API reference -- [Showing Code Mode in the UI](./client-integration) — Display execution progress in your React app -- [Isolate Drivers](./code-mode-isolates) — Compare sandbox runtimes +- [Code Mode](./code-mode) · [Client UI](./client-integration) · [Isolates](./code-mode-isolates) diff --git a/docs/code-mode/code-mode.md b/docs/code-mode/code-mode.md index 04f4ff5f7..636dbfca9 100644 --- a/docs/code-mode/code-mode.md +++ b/docs/code-mode/code-mode.md @@ -2,7 +2,7 @@ title: Code Mode id: code-mode order: 1 -description: "Let LLMs write and execute TypeScript programs that orchestrate tools in a secure sandbox with TanStack AI Code Mode — fewer loops, richer logic." +description: "LLM writes TypeScript that orchestrates tools in a sandbox — fewer agent loops, typed stubs." keywords: - tanstack ai - code mode @@ -13,55 +13,35 @@ keywords: - ai agents --- -Code Mode lets an LLM write and execute TypeScript programs inside a secure sandbox. Instead of making one tool call at a time, the model writes a short script that orchestrates multiple tools with loops, conditionals, `Promise.all`, and data transformations — then returns a single result. +# Code Mode -You already have a chat app that uses [tools](../tools/tools). By the end of this guide, you'll have Code Mode set up so the LLM can compose those tools in TypeScript and execute them in a single sandbox call. +If tools already work in chat → let the model write one TypeScript program that orchestrates them in a sandbox instead of multi-step tool loops. -## Why Code Mode? +## Why -### Reduced context window usage +1. **Fewer tokens** — one `execute_typescript` call vs many tool round-trips +2. **Logic up front** — filter, `Promise.all`, branch inside the sandbox +3. **Typed stubs** — tools become typed `external_*` functions in the prompt +4. **Sandbox** — V8 / QuickJS / Cloudflare Worker; timeouts + memory limits -In a traditional agentic loop, every tool call adds a round-trip of messages: the model's tool-call request, the tool result, then the model's next reasoning step. A task that touches five tools can easily consume thousands of tokens in back-and-forth. +## Setup -With Code Mode the model emits one `execute_typescript` call containing a complete program. The five tool invocations happen inside the sandbox, and only the final result comes back — one request, one response. - -### The LLM decides how to interpret tool output - -When tools are called individually, the model must decide what to do with each result in a new turn. With Code Mode, the model writes the logic up front: filter, aggregate, compare, branch. It can `Promise.all` ten API calls, pick the best result, and return a summary — all in a single execution. - -### Type-safe tool execution - -Tools you pass to Code Mode are converted to typed function stubs that appear in the system prompt. The model sees exact input/output types, so it generates correct calls without guessing parameter names or shapes. TypeScript annotations in the generated code are stripped automatically before execution. - -### Secure sandboxing - -Generated code runs in an isolated environment (V8 isolate, QuickJS WASM, or Cloudflare Worker) with no access to the host file system, network, or process. The sandbox has configurable timeouts and memory limits. - -## Getting Started - -### 1. Install packages +### 1. Install ```bash pnpm add @tanstack/ai @tanstack/ai-code-mode zod ``` -Pick an isolate driver: +Pick a driver: ```bash -# Node.js — fastest, uses V8 isolates (requires native compilation) -pnpm add @tanstack/ai-isolate-node - -# QuickJS WASM — no native deps, works in browsers and edge runtimes -pnpm add @tanstack/ai-isolate-quickjs - -# Cloudflare Workers — run on the edge -pnpm add @tanstack/ai-isolate-cloudflare +pnpm add @tanstack/ai-isolate-node # Node V8 (fastest) +pnpm add @tanstack/ai-isolate-quickjs # WASM, portable +pnpm add @tanstack/ai-isolate-cloudflare # Cloudflare edge ``` ### 2. Define tools -Define your tools with `toolDefinition()` and provide a server-side implementation with `.server()`. These become the `external_*` functions available inside the sandbox. - ```typescript group=code-mode import { toolDefinition } from "@tanstack/ai"; import { z } from "zod"; @@ -80,7 +60,7 @@ const fetchWeather = toolDefinition({ }); ``` -### 3. Create the Code Mode tool and system prompt +### 3. Create Code Mode ```typescript group=code-mode import { createCodeMode } from "@tanstack/ai-code-mode"; @@ -115,7 +95,7 @@ const result = await chat({ }); ``` -The model will generate something like: +Model may emit something like: ```typescript ignore const cities = ["Tokyo", "Paris", "New York City"]; @@ -137,114 +117,85 @@ return { }; ``` -All three API calls happen in parallel inside the sandbox. The model receives one structured result instead of three separate tool-call round-trips. - -## API Reference +## API ### `createCodeMode(config)` -Creates both the `execute_typescript` tool and its matching system prompt from a single config object. This is the recommended entry point. - ```typescript ignore const { tool, systemPrompt } = createCodeMode({ - driver, // IsolateDriver — required - tools, // Array — required, at least one - timeout, // number — execution timeout in ms (default: 30000) - memoryLimit, // number — memory limit in MB (default: 128, Node + QuickJS drivers) - getSkillBindings, // () => Promise> — optional dynamic bindings + driver, // IsolateDriver — required + tools, // ServerTool[] — required, need .server() + timeout, // ms (default 30000) + memoryLimit, // MB (default 128; Node + QuickJS) + getSkillBindings, // optional dynamic bindings }); ``` -**Config properties:** - | Property | Type | Description | |----------|------|-------------| -| `driver` | `IsolateDriver` | The sandbox runtime to execute code in | -| `tools` | `Array` | Tools exposed as `external_*` functions. Must have `.server()` implementations | -| `timeout` | `number` | Execution timeout in milliseconds (default: 30000) | -| `memoryLimit` | `number` | Memory limit in MB (default: 128). Supported by Node and QuickJS drivers | -| `getSkillBindings` | `() => Promise>` | Optional function returning additional bindings at execution time | +| `driver` | `IsolateDriver` | Sandbox runtime | +| `tools` | `Array` | Become `external_*` | +| `timeout` | `number` | Execution timeout ms | +| `memoryLimit` | `number` | Heap MB | +| `getSkillBindings` | `() => Promise>` | Extra bindings | -The tool returns a `CodeModeToolResult`: +Result: ```typescript interface CodeModeToolResult { success: boolean; - result?: unknown; // Return value from the executed code - logs?: Array; // Captured console output - error?: { - message: string; - name?: string; - line?: number; - }; + result?: unknown; + logs?: Array; + error?: { message: string; name?: string; line?: number }; } ``` -### `createCodeModeTool(config)` / `createCodeModeSystemPrompt(config)` - -Lower-level functions if you need only the tool or only the prompt. `createCodeMode` calls both internally. +### Split helpers ```typescript -import { createCodeModeTool, createCodeModeSystemPrompt } from "@tanstack/ai-code-mode"; +import { + createCodeModeTool, + createCodeModeSystemPrompt, +} from "@tanstack/ai-code-mode"; import { config } from "./config"; const tool = createCodeModeTool(config); const prompt = createCodeModeSystemPrompt(config); ``` -### `IsolateDriver` - -The interface that sandbox runtimes implement. You do not implement this yourself — pick one of the provided drivers: - -```typescript -import type { IsolateConfig, IsolateContext } from "@tanstack/ai-code-mode"; - -interface IsolateDriver { - createContext(config: IsolateConfig): Promise; -} -``` - -**Available drivers:** +### Drivers -| Package | Factory function | Environment | -|---------|-----------------|-------------| +| Package | Factory | Environment | +|---------|---------|-------------| | `@tanstack/ai-isolate-node` | `createNodeIsolateDriver()` | Node.js | -| `@tanstack/ai-isolate-quickjs` | `createQuickJSIsolateDriver()` | Node.js, browser, edge | +| `@tanstack/ai-isolate-quickjs` | `createQuickJSIsolateDriver()` | Node, browser, edge | | `@tanstack/ai-isolate-cloudflare` | `createCloudflareIsolateDriver()` | Cloudflare Workers | -For full configuration options for each driver, see [Isolate Drivers](./code-mode-isolates.md). - -### Advanced - -These utilities are used internally and are exported for custom pipelines: - -- **`stripTypeScript(code)`** — Strips TypeScript syntax using sucrase (edge-safe, no native binary), converting to plain JavaScript. -- **`toolsToBindings(tools, prefix?)`** — Converts TanStack AI tools into `Record` for sandbox injection. -- **`generateTypeStubs(bindings, options?)`** — Generates TypeScript type declarations from tool bindings for system prompts. +Full options: [Isolate Drivers](./code-mode-isolates.md). -## Choosing a Driver +**Rule of thumb:** Node for servers · QuickJS for portable/edge without Workers deploy · Cloudflare when you already run Workers. -For a full comparison of drivers with all configuration options, see [Isolate Drivers](./code-mode-isolates.md). +### Internals (exported) -In brief: use the **Node driver** for server-side Node.js (fastest, V8 JIT), **QuickJS** for browsers or portable edge deployments (no native deps), and the **Cloudflare driver** when you deploy to Cloudflare Workers. +- `stripTypeScript(code)` — sucrase strip to JS +- `toolsToBindings(tools, prefix?)` — tool → bindings +- `generateTypeStubs(bindings, options?)` — prompt type stubs -## Custom Events - -Code Mode emits custom events during execution that you can observe through the TanStack AI event system. These are useful for building UIs that show execution progress, debugging, or logging. +## Custom events | Event | When | Payload | |-------|------|---------| -| `code_mode:execution_started` | Code execution begins | `{ timestamp, codeLength }` | -| `code_mode:console` | Each `console.log/error/warn/info` call | `{ level, message, timestamp }` | -| `code_mode:external_call` | Before an `external_*` function runs | `{ function, args, timestamp }` | -| `code_mode:external_result` | After a successful `external_*` call | `{ function, result, duration }` | -| `code_mode:external_error` | When an `external_*` call fails | `{ function, error, duration }` | +| `code_mode:execution_started` | Start | `timestamp`, `codeLength` | +| `code_mode:console` | log/error/warn/info | `level`, `message`, `timestamp` | +| `code_mode:external_call` | Before tool | `function`, `args`, `timestamp` | +| `code_mode:external_result` | After tool | `function`, `result`, `duration` | +| `code_mode:external_error` | Tool fail | `function`, `error`, `duration` | -To display these events in your React app, see [Showing Code Mode in the UI](./client-integration). +UI: [Showing Code Mode in the UI](./client-integration). -## Model Compatibility +## Model compatibility -Code Mode asks the model to write valid TypeScript that calls your tools through the sandbox bridge. Not every model handles this equally — many small or older models mishandle the `external_*` calling conventions even when the system prompt is explicit. We track a single multi-step benchmark (joining three tables, filtering customers who bought from every product category, aggregating spend per category) against a gold reference. The full harness lives at `packages/ai-code-mode/models-eval/`. +Single multi-step benchmark (join/filter/aggregate). Harness: `packages/ai-code-mode/models-eval/`. | Rank | Model | Stars | Acc | Comp | TS | CME | Latency | Tokens | |------|-------|:-----:|:---:|:----:|:--:|:---:|--------:|-------:| @@ -256,40 +207,32 @@ Code Mode asks the model to write valid TypeScript that calls your tools through | 6 | `openai:gpt-4o-mini` | ★★☆ | 10 | 8 | 8 | 10 | 19.2s | 8.7k | | 7 | `ollama:gemma4:31b` | ★★☆ | 10 | 8 | 4 | 5 | 264.2s | 6.4k | -**Columns** - -- **Stars** — overall weighted rating (1-3) combining accuracy, comprehensiveness, code quality, code-mode efficiency, speed, token efficiency, and stability. -- **Acc / Comp / TS / CME** — Anthropic-judged subscores out of 10: accuracy vs gold, comprehensiveness, TypeScript quality, code-mode efficiency (fewer wasted attempts is better). -- **Latency** — wall-clock time for the full agentic loop. -- **Tokens** — total prompt + completion tokens. Grok's adapter does not report usage. +- **Stars** — weighted overall (1–3) +- **Acc / Comp / TS / CME** — accuracy, comprehensiveness, TS quality, code-mode efficiency ( /10) +- **Latency / Tokens** — full loop wall time and usage (Grok adapter omits usage) **Takeaways** -- **Strongest cloud picks:** Grok 4.1 Fast, Claude Haiku 4.5, and Gemini 2.5 Flash all finish under 10s and handle the multi-step task cleanly. Claude Haiku 4.5 has the highest comprehensiveness score (10/10). -- **Strongest local pick:** `ollama:gpt-oss:20b` is the best local performer at 45s with zero compilation failures. `ollama:nemotron-cascade-2` is a close second. -- **Avoid:** the smaller `gemma4` (9.6 GB) and the other local models commented out at the top of `eval-config.ts` (`granite4:3b`, `ministral-3`, `mistral:7b`, `qwen3:8b`, etc.) — they either ignore the `external_queryTable` shape, hallucinate results, or refuse to invoke `execute_typescript`. -- **Caveat:** this is a single‑prompt benchmark. Local model results can vary noticeably between runs; use these as a rough capability filter rather than a definitive ranking. - -Reproduce locally: +1. Cloud under 10s: Grok 4.1 Fast, Claude Haiku 4.5, Gemini 2.5 Flash +2. Best local: `ollama:gpt-oss:20b` (~45s) +3. Avoid tiny locals that ignore `external_*` / refuse `execute_typescript` +4. Single-prompt bench — use as a filter, not a ranking ```bash cd packages/ai-code-mode/models-eval pnpm install -pnpm eval # full suite (needs cloud API keys + Anthropic for judging) -pnpm eval -- --ollama-only # local models only -pnpm eval -- --no-judge # skip Anthropic-based judging +pnpm eval +pnpm eval -- --ollama-only +pnpm eval -- --no-judge ``` ## Tips -- **Start simple.** Give the model 2-3 tools and a clear task. Code Mode works best when the model has a focused set of capabilities. -- **Prefer `Promise.all` tasks.** Code Mode shines when the model can parallelize work that would otherwise be sequential tool calls. -- **Use `console.log` for debugging.** Logs are captured and returned in the result, making it easy to see what happened inside the sandbox. -- **Keep tools focused.** Each tool should do one thing well. The model will compose them in code. -- **Check the system prompt.** Call `createCodeModeSystemPrompt(config)` and inspect the output to see exactly what the model will see, including generated type stubs. +1. Start with 2–3 focused tools. +2. Prefer tasks that benefit from `Promise.all`. +3. Use `console.log` — logs return in the result. +4. Inspect `createCodeModeSystemPrompt(config)` to see model-facing stubs. -## Next Steps +## Next -- [Showing Code Mode in the UI](./client-integration) — Display execution progress in your React app -- [Code Mode with Skills](./code-mode-with-skills) — Add persistent, reusable skill libraries -- [Isolate Drivers](./code-mode-isolates) — Compare Node, QuickJS, and Cloudflare sandbox runtimes +- [Client UI](./client-integration) · [Skills](./code-mode-with-skills) · [Isolates](./code-mode-isolates) diff --git a/docs/code-mode/lazy-tools.md b/docs/code-mode/lazy-tools.md index 52957b88d..0784dd129 100644 --- a/docs/code-mode/lazy-tools.md +++ b/docs/code-mode/lazy-tools.md @@ -2,7 +2,7 @@ title: Lazy Tools id: lazy-tools order: 5 -description: "Keep large tool catalogs out of the Code Mode system prompt with lazy tools — the model fetches TypeScript signatures on demand via a discover_tools call." +description: "Mark tools lazy: true so signatures load on demand via discover_tools — smaller Code Mode prompts." keywords: - tanstack ai - code mode @@ -13,19 +13,16 @@ keywords: - tool catalog --- -Large tool catalogs bloat the `execute_typescript` system prompt. Every tool you pass to `createCodeMode` becomes a full TypeScript type stub in that prompt — and at 50+ tools, those stubs can push the effective prompt into the tens of thousands of tokens before the model has even seen your user message. +# Lazy Tools -Lazy tools fix this with **progressive disclosure**: mark rarely-used tools `lazy: true` and they are withheld from the initial system prompt. The model sees only their names in a short "Discoverable APIs" catalog. When it needs one, it calls the `discover_tools` sibling tool to fetch the TypeScript signature on demand, then uses it inside `execute_typescript`. All sandbox bindings are always injected — lazy only defers _documentation_, not callability. +If a large tool catalog bloats the `execute_typescript` prompt → mark rare tools `lazy: true`. Bindings stay injected; only **docs** are deferred until `discover_tools`. -## Marking a Tool Lazy - -Add `lazy: true` to the `toolDefinition` config for any tool you want to defer: +## 1. Mark tools lazy ```typescript group=lazy-tools import { toolDefinition } from "@tanstack/ai"; import { z } from "zod"; -// Always eager — documented upfront const fetchWeather = toolDefinition({ name: "fetchWeather", description: "Get current weather for a city", @@ -36,7 +33,6 @@ const fetchWeather = toolDefinition({ return res.json(); }); -// Lazy — kept out of the system prompt until discovered const fetchArchive = toolDefinition({ name: "fetchArchive", description: "Retrieve historical weather archive data for a date range", @@ -45,21 +41,23 @@ const fetchArchive = toolDefinition({ from: z.string(), to: z.string(), }), - outputSchema: z.array(z.object({ date: z.string(), temperature: z.number() })), + outputSchema: z.array( + z.object({ date: z.string(), temperature: z.number() }), + ), lazy: true, }).server(async ({ location, from, to }) => { const res = await fetch( - `https://api.weather.example/v1/archive?city=${location}&from=${from}&to=${to}` + `https://api.weather.example/v1/archive?city=${location}&from=${from}&to=${to}`, ); return res.json(); }); ``` -Eager tools continue to receive full type stubs in the system prompt. Lazy tools appear only by name. +Eager tools get full type stubs. Lazy tools appear by name only until discovered. -## Server Setup +## 2. Server setup -Pass both eager and lazy tools to `createCodeMode`. When at least one tool is lazy, `createCodeMode` also returns a `discover_tools` sibling tool — include it in the `tools` array you pass to `chat()`: +Any lazy tool makes `createCodeMode` return `discover_tools` — spread `tools` into `chat()`: ```typescript group=lazy-tools // server/route.ts @@ -70,11 +68,10 @@ import { openaiText } from "@tanstack/ai-openai"; const { tools, systemPrompt } = createCodeMode({ driver: createNodeIsolateDriver(), - tools: [fetchWeather, fetchArchive], // fetchArchive is lazy + tools: [fetchWeather, fetchArchive], }); -// tools is [execute_typescript, discover_tools] -// — discover_tools is included automatically because fetchArchive is lazy +// tools = [execute_typescript, discover_tools] export async function POST(req: Request) { const { messages } = await req.json(); @@ -91,30 +88,24 @@ export async function POST(req: Request) { } ``` -`createCodeMode` returns `{ tool, discoveryTool, tools, systemPrompt }`: - | Field | Type | Description | |-------|------|-------------| -| `tool` | `ServerTool` | The `execute_typescript` tool (backward compatible) | -| `discoveryTool` | `ServerTool \| null` | The `discover_tools` tool, or `null` when there are no lazy tools | -| `tools` | `Array` | `[tool]` or `[tool, discoveryTool]` — spread into `chat({ tools })` | -| `systemPrompt` | `string` | The matching system prompt | - -If no tools are lazy, `discoveryTool` is `null` and `tools` contains only `execute_typescript`. +| `tool` | `ServerTool` | `execute_typescript` | +| `discoveryTool` | `ServerTool \| null` | `discover_tools` if any lazy | +| `tools` | `Array` | `[tool]` or `[tool, discoveryTool]` | +| `systemPrompt` | `string` | Matching prompt | -## The `discover_tools` Flow +## `discover_tools` flow -When the model encounters a task that requires a lazy tool, it: +1. Model calls `discover_tools` with bare name (no `external_` prefix) +2. Gets TypeScript stub + description +3. Writes `execute_typescript` using `external_fetchArchive(...)` -1. Calls `discover_tools` with the tool name (bare name, no `external_` prefix). -2. Receives the TypeScript type stub and description for that tool. -3. Writes `execute_typescript` code using the now-documented `external_fetchArchive(...)` call. +Discovery is documentation only — bindings always work. -The bindings are always injected into the sandbox — discovering a tool only retrieves documentation, it does not enable the binding. The model could call `external_fetchArchive` without discovering it first, but it would be writing blind without the type signature. +## Catalog detail -## Tuning the Discoverable APIs Catalog - -By default, lazy tools appear in the system prompt as bare names with no description: +Default catalog is bare names: ```text ### Discoverable APIs @@ -124,7 +115,7 @@ By default, lazy tools appear in the system prompt as bare names with no descrip - external_exportData ``` -If you want the model to have a hint about what each tool does before deciding whether to discover it, use `lazyToolsConfig.includeDescription`: +Add hints with `lazyToolsConfig.includeDescription`: ```typescript import { createCodeMode } from "@tanstack/ai-code-mode"; @@ -145,34 +136,23 @@ const { tools, systemPrompt } = createCodeMode({ }); ``` -With `'first-sentence'` the catalog becomes: - -```text -### Discoverable APIs - -- external_fetchArchive — Retrieve historical weather archive data for a date range. -- external_runReport — Generate a summary report for a given time period. -- external_exportData — Export query results to CSV or JSON format. -``` - | Value | Effect | |-------|--------| -| `'none'` (default) | Bare names only — smallest possible prompt addition | -| `'first-sentence'` | Name plus the first sentence of the tool's description | -| `'full'` | Name plus the complete description | +| `'none'` (default) | Names only | +| `'first-sentence'` | Name + first sentence of description | +| `'full'` | Name + full description | -The full type stub and input/output schema are always returned on discovery — `includeDescription` only affects the pre-discovery catalog. +Full stubs still return on discovery. -## Lazy Tools with Plain `chat()` +## Lazy tools with plain `chat()` -The same `lazyToolsConfig` option works for lazy tools used directly with `chat()`, outside of Code Mode. Tools marked `lazy: true` are withheld from the `__lazy__tool__discovery__` catalog description until the model calls for them. Pass `lazyToolsConfig` directly to `chat()`: +Same option outside Code Mode: ```typescript import { chat, maxIterations, toServerSentEventsStream } from "@tanstack/ai"; import { openaiText } from "@tanstack/ai-openai"; import { fetchWeather, fetchArchive, runReport } from "./tools"; -// Non-code-mode: lazy tools in a regular chat agent export async function POST(req: Request) { const { messages } = await req.json(); @@ -190,17 +170,13 @@ export async function POST(req: Request) { } ``` -The `includeDescription` behavior is identical — `'none'` lists bare tool names, `'first-sentence'` appends the first sentence, `'full'` appends the complete description. - ## Tips -- **Start with `'none'`.** The bare-names catalog is enough for models that reason well about tool names. Add `'first-sentence'` only if the model frequently discovers irrelevant tools. -- **Lazy tools are always callable.** Their `external_*` bindings are injected into the sandbox regardless of whether the model has called `discover_tools`. Discovery only reveals documentation. -- **Use `discoveryTool` for observability.** You can inspect `discoveryTool.name` (`"discover_tools"`) to confirm the tool is wired up, or log its calls for analytics. -- **Partition by frequency, not capability.** Mark tools lazy when they are rarely needed for a typical request. Core tools that most requests use should stay eager. +1. Start with `'none'`; add `'first-sentence'` if the model discovers the wrong tools. +2. Lazy tools remain callable without discovery. +3. Log `discoveryTool` for observability. +4. Lazy by **frequency**, not capability — keep core tools eager. -## Next Steps +## Next -- [Code Mode](./code-mode) — Core Code Mode setup and API reference -- [Code Mode with Skills](./code-mode-with-skills) — Persistent reusable skill libraries -- [Isolate Drivers](./code-mode-isolates) — Compare Node, QuickJS, and Cloudflare sandbox runtimes +- [Code Mode](./code-mode) · [Skills](./code-mode-with-skills) · [Isolates](./code-mode-isolates) diff --git a/docs/community-adapters/cencori.md b/docs/community-adapters/cencori.md index ef4bda30c..8ff5dbd09 100644 --- a/docs/community-adapters/cencori.md +++ b/docs/community-adapters/cencori.md @@ -2,7 +2,7 @@ title: Cencori id: cencori-adapter order: 3 -description: "Access 14+ AI providers (OpenAI, Anthropic, Google, xAI, and more) through Cencori's unified interface with built-in security, observability, and cost tracking in TanStack AI." +description: "Multi-provider chat via Cencori (security, observability, cost tracking) for TanStack AI." keywords: - tanstack ai - cencori @@ -13,15 +13,23 @@ keywords: - community adapter --- -The Cencori adapter provides access to 14+ AI providers (OpenAI, Anthropic, Google, xAI, and more) through a unified interface with built-in security, observability, and cost tracking. +# Cencori -## Installation +If you want one key across many providers with Cencori security/observability → install the adapter and pass a model id. + +## Install ```bash npm install @cencori/ai-sdk ``` -## Basic Usage +```bash +CENCORI_API_KEY=csk_your_api_key_here +``` + +API key: [cencori.com](https://cencori.com). + +## Usage ```typescript ignore // ignore: @cencori/ai-sdk/tanstack is a subpath export; kiira's paths["*"] wildcard maps it @@ -42,7 +50,7 @@ for await (const chunk of chat({ } ``` -## Configuration +## Config ```typescript ignore // ignore: @cencori/ai-sdk/tanstack subpath not resolvable via kiira's paths["*"] wildcard. @@ -50,35 +58,13 @@ import { createCencori } from "@cencori/ai-sdk/tanstack"; const myCencori = createCencori({ apiKey: process.env.CENCORI_API_KEY!, - baseUrl: "https://cencori.com", // Optional + baseUrl: "https://cencori.com", // optional }); const adapter = myCencori("o1"); ``` -## Streaming - -```typescript ignore -// ignore: @cencori/ai-sdk/tanstack subpath not resolvable via kiira's paths["*"] wildcard. -import { chat } from "@tanstack/ai"; -import { cencori } from "@cencori/ai-sdk/tanstack"; - -const adapter = cencori("claude-3-5-sonnet"); - -for await (const chunk of chat({ - adapter, - messages: [{ role: "user", content: "Tell me a story" }], -})) { - if (chunk.type === "TEXT_MESSAGE_CONTENT") { - process.stdout.write(chunk.delta); - } else if (chunk.type === "RUN_FINISHED") { - console.log("\nDone"); - } -} -``` - - -## Tool Calling +## Tools ```typescript ignore // ignore: @cencori/ai-sdk/tanstack subpath not resolvable via kiira's paths["*"] wildcard. @@ -86,121 +72,55 @@ import { chat, toolDefinition } from "@tanstack/ai"; import { cencori } from "@cencori/ai-sdk/tanstack"; import { z } from "zod"; -const adapter = cencori("o1"); - -const getWeatherDef = toolDefinition({ +const getWeather = toolDefinition({ name: "getWeather", description: "Get weather for a location", inputSchema: z.object({ location: z.string() }), -}); - -const getWeather = getWeatherDef.server(async ({ location }) => { - // Look up the weather for `location` +}).server(async ({ location }) => { return { temperature: 72, conditions: "Sunny" }; }); for await (const chunk of chat({ - adapter, + adapter: cencori("o1"), messages: [{ role: "user", content: "What's the weather in NYC?" }], tools: [getWeather], })) { if (chunk.type === "TOOL_CALL_START") { console.log("Tool call:", chunk.toolCallName); - } else if (chunk.type === "TOOL_CALL_END") { - console.log("Tool call finished:", chunk.toolCallId); } } ``` +## Models -## Multi-Provider Support - -Switch between providers with a single parameter: +Switch provider by model id: ```typescript ignore // ignore: @cencori/ai-sdk/tanstack subpath not resolvable via kiira's paths["*"] wildcard. import { cencori } from "@cencori/ai-sdk/tanstack"; -// OpenAI-compatible -const openaiCompat = cencori("o1"); - -// Anthropic -const anthropic = cencori("claude-3-5-sonnet"); - -// Google -const google = cencori("gemini-2.5-flash"); - -// xAI -const grok = cencori("grok-3"); - -// DeepSeek -const deepseek = cencori("deepseek-v3.2"); +cencori("o1"); +cencori("claude-3-5-sonnet"); +cencori("gemini-2.5-flash"); +cencori("grok-3"); +cencori("deepseek-v3.2"); ``` -All responses use the same unified format regardless of provider. - -## Supported Models - -| Provider | Models | +| Provider | Models (verify against Cencori catalogue) | |----------|--------| | OpenAI | `gpt-5`, `gpt-4o`, `gpt-4o-mini`, `o3`, `o1` | | Anthropic | `claude-opus-4`, `claude-sonnet-4`, `claude-3-5-sonnet` | | Google | `gemini-3-pro`, `gemini-2.5-flash`, `gemini-2.0-flash` | -| xAI | `grok-4`, `grok-3` | -| Mistral | `mistral-large`, `codestral`, `devstral` | -| DeepSeek | `deepseek-v3.2`, `deepseek-reasoner` | -| + More | Groq, Cohere, Perplexity, Together, Qwen, OpenRouter | - -> **Note:** Cencori is an external package and its catalogue changes over time. Verify the model ids above against [Cencori's current catalogue](https://cencori.com/docs) before relying on them. - -## Environment Variables - -```bash -CENCORI_API_KEY=csk_your_api_key_here -``` - -## Getting an API Key - -1. Go to [Cencori](https://cencori.com) -2. Create an account and generate an API key -3. Add it to your environment variables - -## Why Cencori? - -- **🔒 Security** — PII filtering, jailbreak detection, content moderation -- **📊 Observability** — Request logs, latency metrics, cost tracking -- **💰 Cost Control** — Budgets, alerts, per-route analytics -- **🔌 Multi-Provider** — One API key for 14+ AI providers -- **🛠️ Tool Calling** — Full support for function calling across providers -- **🔄 Failover** — Automatic retry and fallback to alternative providers - -## API Reference - -### `cencori(model)` - -Creates a Cencori adapter using environment variables. - -**Parameters:** - -- `model` - Model name (e.g., `"gpt-4o"`, `"claude-3-5-sonnet"`, `"gemini-2.5-flash"`) - -**Returns:** A Cencori TanStack AI adapter instance. - -### `createCencori(config)` - -Creates a custom Cencori adapter factory. +| xAI / DeepSeek / others | `grok-4`, `deepseek-v3.2`, plus Groq, Cohere, Perplexity, Together, Qwen, OpenRouter | -**Parameters:** +Confirm ids at [Cencori docs](https://cencori.com/docs). -- `config.apiKey` - Your Cencori API key -- `config.baseUrl?` - Custom base URL (optional) +## API -**Returns:** A function that creates adapter instances for specific models. +- `cencori(model)` — env-based adapter +- `createCencori({ apiKey, baseUrl? })` — factory returning model → adapter -## Next Steps +## Links -- [Cencori Dashboard](https://cencori.com) — View analytics, logs, and costs -- [Documentation](https://cencori.com/docs) — Complete API reference -- [GitHub Repository](https://github.com/cencori/cencori) — SDK source code -- [Streaming Guide](../chat/streaming) — Learn about streaming responses -- [Tools Guide](../tools/tools) — Learn about tool calling +- [Dashboard](https://cencori.com) · [Docs](https://cencori.com/docs) · [GitHub](https://github.com/cencori/cencori) +- TanStack: [Streaming](../chat/streaming) · [Tools](../tools/tools) diff --git a/docs/community-adapters/cloudflare.md b/docs/community-adapters/cloudflare.md index fcb4c0aed..bf5a8cf32 100644 --- a/docs/community-adapters/cloudflare.md +++ b/docs/community-adapters/cloudflare.md @@ -2,7 +2,7 @@ title: Cloudflare id: cloudflare-adapter order: 3 -description: "Use Cloudflare Workers AI and AI Gateway with TanStack AI for edge inference, caching, rate limiting, and unified billing across providers." +description: "Workers AI and AI Gateway with TanStack AI — edge chat, image, STT, TTS, summarize, caching." keywords: - tanstack ai - cloudflare @@ -14,50 +14,25 @@ keywords: - community adapter --- -The Cloudflare adapter provides access to [Cloudflare Workers AI](https://developers.cloudflare.com/workers-ai/) models and [AI Gateway](https://developers.cloudflare.com/ai-gateway/) for routing requests to OpenAI, Anthropic, Gemini, Grok, and OpenRouter with caching, rate limiting, and unified billing. +# Cloudflare -## Installation +If you run on Workers AI or need AI Gateway routing → use `@cloudflare/tanstack-ai`. + +## Install ```bash npm install @cloudflare/tanstack-ai @tanstack/ai ``` -For AI Gateway with third-party providers, install the provider SDKs you need: +Gateway + third-party providers (as needed): ```bash -npm install @tanstack/ai-openai # For OpenAI via Gateway -npm install @tanstack/ai-anthropic # For Anthropic via Gateway -npm install @tanstack/ai-gemini # For Gemini via Gateway -npm install @tanstack/ai-grok # For Grok via Gateway -npm install @tanstack/ai-openrouter # For OpenRouter via Gateway -``` - -## Basic Usage - -```typescript -import { chat, toHttpResponse } from "@tanstack/ai"; -import { createWorkersAiChat } from "@cloudflare/tanstack-ai"; -import { env } from "./env"; - -const adapter = createWorkersAiChat( - "@cf/meta/llama-4-scout-17b-16e-instruct", - { binding: env.AI }, -); - -const response = chat({ - adapter, - stream: true, - messages: [{ role: "user", content: "Hello!" }], -}); - -const httpResponse = toHttpResponse(response); +npm install @tanstack/ai-openai @tanstack/ai-anthropic @tanstack/ai-gemini @tanstack/ai-grok @tanstack/ai-openrouter ``` -## Workers AI +## Workers AI chat -The simplest way to use AI in a Cloudflare Worker. No API keys needed when using the `env.AI` binding. - -### Chat +No API key with `env.AI` binding: ```typescript import { chat, toHttpResponse } from "@tanstack/ai"; @@ -78,110 +53,69 @@ const response = chat({ const httpResponse = toHttpResponse(response); ``` -### Chat with REST Credentials - -If you're not running inside a Worker, use account ID and API key instead: +Outside a Worker — REST credentials: ```typescript import { createWorkersAiChat } from "@cloudflare/tanstack-ai"; const adapter = createWorkersAiChat( "@cf/meta/llama-4-scout-17b-16e-instruct", - { - accountId: "your-account-id", - apiKey: "your-api-key", - }, + { accountId: "your-account-id", apiKey: "your-api-key" }, ); ``` -### Image Generation +## Other Workers AI capabilities ```typescript +// Image import { generateImage } from "@tanstack/ai"; import { createWorkersAiImage } from "@cloudflare/tanstack-ai"; -import { env } from "./env"; - -const adapter = createWorkersAiImage( - "@cf/stabilityai/stable-diffusion-xl-base-1.0", - { binding: env.AI }, -); const result = await generateImage({ - adapter, + adapter: createWorkersAiImage( + "@cf/stabilityai/stable-diffusion-xl-base-1.0", + { binding: env.AI }, + ), prompt: "a cat in space", }); -console.log(result.images[0]?.b64Json); -``` - -### Transcription (Speech-to-Text) - -Supports Whisper and Deepgram models: - -```typescript +// Transcription (Whisper / Deepgram) import { generateTranscription } from "@tanstack/ai"; import { createWorkersAiTranscription } from "@cloudflare/tanstack-ai"; -import { env, audioArrayBuffer } from "./env"; - -const adapter = createWorkersAiTranscription( - "@cf/openai/whisper-large-v3-turbo", - { binding: env.AI }, -); -const result = await generateTranscription({ - adapter, +const transcript = await generateTranscription({ + adapter: createWorkersAiTranscription( + "@cf/openai/whisper-large-v3-turbo", + { binding: env.AI }, + ), audio: audioArrayBuffer, }); -console.log(result.text); -console.log(result.segments); -``` - -Supported transcription models: `@cf/openai/whisper`, `@cf/openai/whisper-tiny-en`, `@cf/openai/whisper-large-v3-turbo`, `@cf/deepgram/nova-3` - -### Text-to-Speech - -```typescript +// TTS import { generateSpeech } from "@tanstack/ai"; import { createWorkersAiTts } from "@cloudflare/tanstack-ai"; -import { env } from "./env"; - -const adapter = createWorkersAiTts("@cf/deepgram/aura-2-en", { - binding: env.AI, -}); -const result = await generateSpeech({ - adapter, +const speech = await generateSpeech({ + adapter: createWorkersAiTts("@cf/deepgram/aura-2-en", { binding: env.AI }), text: "Hello world", }); -console.log(result.audio); -``` - -### Summarization - -```typescript +// Summarize import { summarize, type AnySummarizeAdapter } from "@tanstack/ai"; import { createWorkersAiSummarize } from "@cloudflare/tanstack-ai"; -import { env } from "./env"; - -const adapter: AnySummarizeAdapter = createWorkersAiSummarize("@cf/facebook/bart-large-cnn", { - binding: env.AI, -}); - -const result = await summarize({ - adapter, - text: "Long article here...", -}); -console.log(result.summary); +const adapter: AnySummarizeAdapter = createWorkersAiSummarize( + "@cf/facebook/bart-large-cnn", + { binding: env.AI }, +); +const summary = await summarize({ adapter, text: "Long article here..." }); ``` -## AI Gateway +Transcription models: `@cf/openai/whisper`, `@cf/openai/whisper-tiny-en`, `@cf/openai/whisper-large-v3-turbo`, `@cf/deepgram/nova-3`. -Route AI requests through Cloudflare's AI Gateway for caching, rate limiting, and unified billing. Supports both Workers AI and third-party providers. +## AI Gateway -### Workers AI through Gateway +Workers AI through gateway: ```typescript import { createWorkersAiChat } from "@cloudflare/tanstack-ai"; @@ -196,15 +130,12 @@ const adapter = createWorkersAiChat( ); ``` -### Third-Party Providers through Gateway - -Use the binding approach (recommended for Cloudflare Workers): +Third-party via gateway binding: ```typescript import { createOpenAiChat, createAnthropicChat, - createGeminiChat, createGrokChat, createOpenRouterChat, } from "@cloudflare/tanstack-ai"; @@ -213,21 +144,18 @@ import { env } from "./env"; const openai = createOpenAiChat("gpt-4o", { binding: env.AI.gateway("my-gateway-id"), }); - const anthropic = createAnthropicChat("claude-sonnet-4-5", { binding: env.AI.gateway("my-gateway-id"), }); - const grok = createGrokChat("grok-4.3", { binding: env.AI.gateway("my-gateway-id"), }); - const openrouter = createOpenRouterChat("openai/gpt-4o", { binding: env.AI.gateway("my-gateway-id"), }); ``` -Or use credentials for non-Worker environments: +REST gateway (non-Worker): ```typescript import { createOpenAiChat } from "@cloudflare/tanstack-ai"; @@ -240,80 +168,43 @@ const adapter = createOpenAiChat("gpt-4o", { }); ``` -### Cache Options - -Both binding and credentials modes support cache configuration: - -```typescript -import { createOpenAiChat } from "@cloudflare/tanstack-ai"; -import { env } from "./env"; - -const adapter = createOpenAiChat("gpt-4o", { - binding: env.AI.gateway("my-gateway-id"), - skipCache: false, - cacheTtl: 3600, - customCacheKey: "my-key", - metadata: { user: "test" }, -}); -``` - -## Configuration Modes - -Workers AI supports four configuration modes: +Cache options: `skipCache`, `cacheTtl`, `customCacheKey`, `metadata`. -| Mode | Config | Description | -|------|--------|-------------| -| Plain binding | `{ binding: env.AI }` | Direct access, no gateway | -| Plain REST | `{ accountId, apiKey }` | REST API, no gateway | -| Gateway binding | `{ binding: env.AI.gateway(id) }` | Through AI Gateway via binding | -| Gateway REST | `{ accountId, gatewayId, ... }` | Through AI Gateway via REST | +## Config modes -Third-party providers (OpenAI, Anthropic, Gemini, Grok, OpenRouter) only support the gateway modes. +| Mode | Config | +|------|--------| +| Plain binding | `{ binding: env.AI }` | +| Plain REST | `{ accountId, apiKey }` | +| Gateway binding | `{ binding: env.AI.gateway(id) }` | +| Gateway REST | `{ accountId, gatewayId, … }` | -## Supported Capabilities +Third-party providers support gateway modes only. -| Provider | Chat | Summarize | Image Gen | Transcription | TTS | -|----------|------|-----------|-----------|---------------|-----| -| **Workers AI** | ✅ | ✅ | ✅ | ✅ | ✅ | -| **OpenAI** | ✅ | ✅ | ✅ | ✅ | ✅ | -| **Anthropic** | ✅ | ✅ | ❌ | ❌ | ❌ | -| **Gemini** | ✅ | ✅ | ✅ | ❌ | ✅ | -| **Grok** | ✅ | ✅ | ✅ | ❌ | ❌ | -| **OpenRouter** | ✅ | ✅ | ✅ | ❌ | ❌ | +## Capabilities -## Environment Variables +| Provider | Chat | Summarize | Image | STT | TTS | +|----------|------|-----------|-------|-----|-----| +| Workers AI | ✅ | ✅ | ✅ | ✅ | ✅ | +| OpenAI | ✅ | ✅ | ✅ | ✅ | ✅ | +| Anthropic | ✅ | ✅ | ❌ | ❌ | ❌ | +| Gemini | ✅ | ✅ | ✅ | ❌ | ✅ | +| Grok / OpenRouter | ✅ | ✅ | ✅ | ❌ | ❌ | -For the REST credential path (outside of Cloudflare Workers): +REST env (outside Workers): ```bash CLOUDFLARE_ACCOUNT_ID=your-account-id CLOUDFLARE_API_KEY=your-api-key ``` -When using the `env.AI` binding inside a Worker, no environment variables are needed. - -## API Reference - -### Workers AI - -- `createWorkersAiChat(model, config)` — Chat and structured output -- `createWorkersAiImage(model, config)` — Image generation -- `createWorkersAiTranscription(model, config)` — Speech-to-text (Whisper, Deepgram) -- `createWorkersAiTts(model, config)` — Text-to-speech (Deepgram Aura) -- `createWorkersAiSummarize(model, config)` — Summarization (BART-large-CNN) +## API surface -### Gateway Providers +**Workers AI:** `createWorkersAiChat` · `createWorkersAiImage` · `createWorkersAiTranscription` · `createWorkersAiTts` · `createWorkersAiSummarize` -- `createOpenAiChat(model, config)` / `createOpenAiSummarize` / `createOpenAiImage` / `createOpenAiTranscription` / `createOpenAiTts` -- `createAnthropicChat(model, config)` / `createAnthropicSummarize` -- `createGeminiChat(model, config)` / `createGeminiSummarize` / `createGeminiImage` / `createGeminiTts` -- `createGrokChat(model, config)` / `createGrokSummarize` / `createGrokImage` -- `createOpenRouterChat(model, config)` / `createOpenRouterSummarize` / `createOpenRouterImage` +**Gateway:** `createOpenAi*` / `createAnthropic*` / `createGemini*` / `createGrok*` / `createOpenRouter*` (Chat, Summarize, Image, Transcription, Tts as supported) -## Next Steps +## Links -- [Cloudflare Workers AI](https://developers.cloudflare.com/workers-ai/) — Workers AI documentation -- [Cloudflare AI Gateway](https://developers.cloudflare.com/ai-gateway/) — AI Gateway documentation -- [GitHub Repository](https://github.com/cloudflare/ai) — Source code and issues -- [Streaming Guide](../chat/streaming) — Learn about streaming responses -- [Tools Guide](../tools/tools) — Learn about tool calling +- [Workers AI](https://developers.cloudflare.com/workers-ai/) · [AI Gateway](https://developers.cloudflare.com/ai-gateway/) · [GitHub](https://github.com/cloudflare/ai) +- [Streaming](../chat/streaming) · [Tools](../tools/tools) diff --git a/docs/community-adapters/decart.md b/docs/community-adapters/decart.md index d18765294..9ec4529ab 100644 --- a/docs/community-adapters/decart.md +++ b/docs/community-adapters/decart.md @@ -2,7 +2,7 @@ title: Decart id: decart-adapter order: 2 -description: "Generate images and videos with Decart's AI models in TanStack AI via the Decart community adapter." +description: "Generate images and videos with Decart models in TanStack AI." keywords: - tanstack ai - decart @@ -11,55 +11,23 @@ keywords: - community adapter --- -The Decart adapter provides access to Decart's image and video generation models. +# Decart -## Installation +If you need Decart image/video generation → install the adapter and call `generateImage` / `generateVideo`. + +## Install ```bash npm install @decartai/tanstack-ai-adapter ``` -## Basic Usage - -```typescript -import { generateImage } from "@tanstack/ai"; -import { decartImage } from "@decartai/tanstack-ai-adapter"; - -const result = await generateImage({ - adapter: decartImage("lucy-pro-t2i"), - prompt: "A serene mountain landscape at sunset", -}); -``` - -## Basic Usage - Custom API Key - -```typescript -import { generateImage } from "@tanstack/ai"; -import { createDecartImage } from "@decartai/tanstack-ai-adapter"; - -const adapter = createDecartImage("lucy-pro-t2i", process.env.DECART_API_KEY!); - -const result = await generateImage({ - adapter, - prompt: "A serene mountain landscape at sunset", -}); -``` - -## Configuration - -```typescript -import { createDecartImage, type DecartImageConfig } from "@decartai/tanstack-ai-adapter"; - -const config: Omit = { - baseUrl: "https://api.decart.ai", // Optional, for custom endpoints -}; - -const adapter = createDecartImage("lucy-pro-t2i", process.env.DECART_API_KEY!, config); +```bash +DECART_API_KEY=your-api-key-here ``` -## Image Generation +Key: [platform.decart.ai](https://platform.decart.ai). -Generate images with `lucy-pro-t2i`: +## Image ```typescript import { generateImage } from "@tanstack/ai"; @@ -67,73 +35,40 @@ import { decartImage } from "@decartai/tanstack-ai-adapter"; const result = await generateImage({ adapter: decartImage("lucy-pro-t2i"), - prompt: "A futuristic cityscape at night", -}); - -console.log(result.images[0]?.b64Json); -``` - -### Image Model Options - -```typescript -import { generateImage } from "@tanstack/ai"; -import { decartImage } from "@decartai/tanstack-ai-adapter"; - -const result = await generateImage({ - adapter: decartImage("lucy-pro-t2i"), - prompt: "A portrait of a robot artist", + prompt: "A serene mountain landscape at sunset", modelOptions: { resolution: "720p", orientation: "portrait", seed: 42, }, }); -``` - -| Option | Type | Default | Description | -|--------|------|---------|-------------| -| `resolution` | `"720p"` | `"720p"` | Output resolution | -| `orientation` | `"portrait" \| "landscape"` | `"landscape"` | Image orientation | -| `seed` | `number` | - | Seed for reproducible generation | - -## Video Generation - -Video generation uses an async job/polling architecture. - -### Creating a Video Job - -```typescript -import { generateVideo } from "@tanstack/ai"; -import { decartVideo } from "@decartai/tanstack-ai-adapter"; - -const { jobId } = await generateVideo({ - adapter: decartVideo("lucy-pro-t2v"), - prompt: "A cat playing with a ball of yarn", -}); -console.log("Job started:", jobId); +console.log(result.images[0]?.b64Json); ``` -### Polling for Status +Explicit key: ```typescript -import { getVideoJobStatus } from "@tanstack/ai"; -import { decartVideo } from "@decartai/tanstack-ai-adapter"; +import { generateImage } from "@tanstack/ai"; +import { createDecartImage } from "@decartai/tanstack-ai-adapter"; -const jobId = "example-job-id"; -const status = await getVideoJobStatus({ - adapter: decartVideo("lucy-pro-t2v"), - jobId, +const adapter = createDecartImage("lucy-pro-t2i", process.env.DECART_API_KEY!, { + baseUrl: "https://api.decart.ai", // optional }); -console.log("Status:", status.status); // "pending" | "processing" | "completed" | "failed" - -if (status.status === "completed" && status.url) { - console.log("Video URL:", status.url); -} +const result = await generateImage({ + adapter, + prompt: "A futuristic cityscape at night", +}); ``` -### Complete Example with Polling +| Option | Type | Default | +|--------|------|---------| +| `resolution` | `"720p"` | `"720p"` | +| `orientation` | `"portrait" \| "landscape"` | `"landscape"` | +| `seed` | `number` | — | + +## Video (job + poll) ```typescript import { generateVideo, getVideoJobStatus } from "@tanstack/ai"; @@ -141,122 +76,40 @@ import { decartVideo } from "@decartai/tanstack-ai-adapter"; async function createVideo(prompt: string) { const adapter = decartVideo("lucy-pro-t2v"); - - // Create the job - const { jobId } = await generateVideo({ adapter, prompt }); - console.log("Job created:", jobId); - - // Poll for completion - let status = "pending"; - while (status !== "completed" && status !== "failed") { + const { jobId } = await generateVideo({ + adapter, + prompt, + modelOptions: { + resolution: "720p", + orientation: "landscape", + seed: 42, + }, + }); + + for (;;) { await new Promise((resolve) => setTimeout(resolve, 5000)); - const result = await getVideoJobStatus({ adapter, jobId }); - status = result.status; - console.log(`Status: ${status}`); - - if (result.status === "failed") { - throw new Error("Video generation failed"); - } - - if (result.status === "completed" && result.url) { - return result.url; - } + if (result.status === "failed") throw new Error("Video generation failed"); + if (result.status === "completed" && result.url) return result.url; } } const videoUrl = await createVideo("A drone shot over a tropical beach"); ``` -### Video Model Options - -```typescript -import { generateVideo } from "@tanstack/ai"; -import { decartVideo } from "@decartai/tanstack-ai-adapter"; - -const { jobId } = await generateVideo({ - adapter: decartVideo("lucy-pro-t2v"), - prompt: "A timelapse of a blooming flower", - modelOptions: { - resolution: "720p", - orientation: "landscape", - seed: 42, - }, -}); -``` - -| Option | Type | Default | Description | -|--------|------|---------|-------------| -| `resolution` | `"720p" \| "480p"` | `"720p"` | Output resolution | -| `orientation` | `"portrait" \| "landscape"` | `"landscape"` | Video orientation | -| `seed` | `number` | - | Seed for reproducible generation | - -## Environment Variables - -Set your API key in environment variables: - -```bash -DECART_API_KEY=your-api-key-here -``` - -## Getting an API Key - -1. Go to [Decart Platform](https://platform.decart.ai) -2. Create an account and generate an API key -3. Add it to your environment variables - -## API Reference - -### `decartImage(model, config?)` - -Creates a Decart image adapter using environment variables. - -**Parameters:** - -- `model` - Model name (`"lucy-pro-t2i"`) -- `config.baseUrl?` - Custom base URL (optional) - -**Returns:** A Decart image adapter instance. - -### `createDecartImage(model, apiKey, config?)` - -Creates a Decart image adapter with an explicit API key. - -**Parameters:** - -- `model` - Model name (`"lucy-pro-t2i"`) -- `apiKey` - Your Decart API key -- `config.baseUrl?` - Custom base URL (optional) - -**Returns:** A Decart image adapter instance. - -### `decartVideo(model, config?)` - -Creates a Decart video adapter using environment variables. - -**Parameters:** - -- `model` - Model name (`"lucy-pro-t2v"`) -- `config.baseUrl?` - Custom base URL (optional) - -**Returns:** A Decart video adapter instance. - -### `createDecartVideo(model, apiKey, config?)` - -Creates a Decart video adapter with an explicit API key. - -**Parameters:** +| Option | Type | Default | +|--------|------|---------| +| `resolution` | `"720p" \| "480p"` | `"720p"` | +| `orientation` | `"portrait" \| "landscape"` | `"landscape"` | +| `seed` | `number` | — | -- `model` - Model name (`"lucy-pro-t2v"`) -- `apiKey` - Your Decart API key -- `config.baseUrl?` - Custom base URL (optional) +## API -**Returns:** A Decart video adapter instance. +- `decartImage(model, config?)` / `createDecartImage(model, apiKey, config?)` — model `"lucy-pro-t2i"` +- `decartVideo(model, config?)` / `createDecartVideo(model, apiKey, config?)` — model `"lucy-pro-t2v"` +- `config.baseUrl?` optional -## Next Steps +## Links -- [Decart Platform](https://platform.decart.ai) - Visit Decart's platform to generate API keys -- [API Documentation](https://docs.platform.decart.ai) - View complete API reference -- [GitHub Repository](https://github.com/decartai/tanstack-ai) - Explore the adapter source code -- [Image Generation Guide](../media/image-generation) - Learn about image generation -- [Video Generation Guide](../media/video-generation) - Learn about video generation +- [Platform](https://platform.decart.ai) · [API docs](https://docs.platform.decart.ai) · [GitHub](https://github.com/decartai/tanstack-ai) +- [Image generation](../media/image-generation) · [Video generation](../media/video-generation) diff --git a/docs/community-adapters/guide.md b/docs/community-adapters/guide.md index 7690cb09a..7966d5367 100644 --- a/docs/community-adapters/guide.md +++ b/docs/community-adapters/guide.md @@ -2,7 +2,7 @@ title: "Community Adapters Guide" slug: /community-adapters/guide order: 1 -description: "Build and publish a community adapter for TanStack AI — package conventions, implementing the adapter interface, and publishing to npm." +description: "Build and publish a community adapter — package layout, model metadata, capability typing, npm + docs PR." keywords: - tanstack ai - community adapters @@ -15,82 +15,28 @@ keywords: # Community Adapters Guide -This guide explains how to create and contribute community adapters for the TanStack AI ecosystem. +If you need a provider TanStack AI does not ship → implement an activity adapter, publish to npm, PR the docs list. -Community adapters extend TanStack AI by integrating external services, APIs, or custom model logic. They are authored and maintained by the community and can be reused across projects. +Community adapters are community-maintained (not core team). -## What is a Community Adapter? +## Steps -A community adapter is a reusable module that connects TanStack AI to an external provider or system. +1. **Study existing adapters** — [packages/](https://github.com/tanstack/ai/tree/main/packages); full reference: [OpenAI adapter](https://github.com/tanstack/ai/tree/main/packages/ai-openai). +2. **Define model metadata** — name/id, input/output modalities, features (stream, tools, structured output), pricing if known, provider limits. Example: [model-meta.ts](https://github.com/TanStack/ai/blob/main/packages/ai-openai/src/model-meta.ts). +3. **Export capability arrays** — only models that fully support each activity: -Common use cases include: -- Integrating third-party AI model providers -- Implementing custom inference or routing logic -- Exposing provider-specific tools or capabilities -- Connecting to non-LLM AI services (e.g. images, embeddings, video) - -Community adapters are **not maintained by the core TanStack AI team**, and can be reused across different projects. - -## Creating a Community Adapter - -Follow the steps below to build a well-structured, type-safe adapter. - -### 1. Set up your project - -Start by reviewing the [existing internal adapter implementations in the TanStack AI GitHub repository](https://github.com/tanstack/ai/tree/main/packages). These define the expected structure, conventions, and integration patterns. - -For a complete, detailed reference, use the [OpenAI adapter](https://github.com/tanstack/ai/tree/main/packages/ai-openai), which is the most fully featured implementation. - -### 2. Define model metadata - -Model metadata describes each model’s capabilities and constraints and is used by TanStack AI for compatibility checks and feature selection. - -Your metadata should define, at a minimum: - -- Model name and identifier -- Supported input and output modalities -- Supported features (e.g. streaming, tools, structured output) -- Pricing or cost information (if available) -- Any provider-specific notes or limitations - -Refer to the [OpenAI adapter’s model metadata](https://github.com/TanStack/ai/blob/main/packages/ai-openai/src/model-meta.ts) for a concrete example. - -### 3. Define model capability arrays - -After defining metadata, group models by supported functionality using exported arrays. These arrays allow TanStack AI to automatically select compatible models for a given task. - -Example: ```typescript ignore export const OPENAI_CHAT_MODELS = [ - // Frontier models GPT5_2.name, GPT5_2_PRO.name, - GPT5_2_CHAT.name, - GPT5_1.name, - GPT5_1_CODEX.name, - GPT5.name, - GPT5_MINI.name, - GPT5_NANO.name, - GPT5_PRO.name, - GPT5_CODEX.name, - // ...other models -] as const -export const OPENAI_IMAGE_MODELS = [ - GPT_IMAGE_1.name, - GPT_IMAGE_1_MINI.name, - DALL_E_3.name, - DALL_E_2.name, + // ... ] as const - +export const OPENAI_IMAGE_MODELS = [GPT_IMAGE_1.name, DALL_E_3.name] as const export const OPENAI_VIDEO_MODELS = [SORA2.name, SORA2_PRO.name] as const ``` -Each array should only include models that fully support the associated functionality. - -### 4. Define model provider options -Each model exposes a different set of configurable options. These options must be typed per model name so that users only see valid configuration options. +4. **Type options per model** — map model name → option fragments: -Example: ```typescript ignore export type OpenAIChatModelProviderOptionsByName = { [GPT5_2.name]: OpenAIBaseOptions & @@ -99,119 +45,57 @@ export type OpenAIChatModelProviderOptionsByName = { OpenAIToolsOptions & OpenAIStreamingOptions & OpenAIMetadataOptions - [GPT5_2_CHAT.name]: OpenAIBaseOptions & - OpenAIReasoningOptions & - OpenAIStructuredOutputOptions & - OpenAIToolsOptions & - OpenAIStreamingOptions & - OpenAIMetadataOptions - // ... repeat for each model + // repeat per model } - ``` -This ensures strict type safety and feature correctness at compile time. - -### 5. Define supported input modalities -Models typically support different input modalities (e.g. text, images, audio). These must be defined per model to prevent invalid usage. +5. **Type input modalities per model**: -Example: ```typescript ignore export type OpenAIModelInputModalitiesByName = { [GPT5_2.name]: typeof GPT5_2.supports.input [GPT5_2_PRO.name]: typeof GPT5_2_PRO.supports.input - [GPT5_2_CHAT.name]: typeof GPT5_2_CHAT.supports.input - // ... repeat for each model + // ... } ``` -### 6. Define model option fragments +## Option fragments -Model options should be composed from reusable fragments rather than duplicated per model. +Compose reusable pieces (base + feature) rather than duplicating per model. Example: [text-provider-options.ts](https://github.com/TanStack/ai/blob/main/packages/ai-openai/src/text/text-provider-options.ts). -A common pattern is: -- Base options shared by all models -- Feature fragments that are stitched together per model - -Example (based on [OpenAI models](https://github.com/TanStack/ai/blob/main/packages/ai-openai/src/text/text-provider-options.ts)): ```typescript export interface OpenAIBaseOptions { - // base options that every chat model supports + // shared by all chat models } -// Feature fragments that can be stitched per-model - -/** - * Reasoning options for models - */ export interface OpenAIReasoningOptions { - //... -} - -/** - * Structured output options for models. - */ -export interface OpenAIStructuredOutputOptions { - //... + // ... } -``` - -Models can then opt into only the features they support: - -```typescript ignore -export type OpenAIChatModelProviderOptionsByName = { - [GPT5_2.name]: OpenAIBaseOptions & - OpenAIReasoningOptions & - OpenAIStructuredOutputOptions & - OpenAIToolsOptions & - OpenAIStreamingOptions & - OpenAIMetadataOptions +export interface OpenAIStructuredOutputOptions { + // ... } ``` -There is no single correct composition; this structure should reflect the capabilities of the provider you are integrating. - -### 7. Implement adapter logic - -Finally, implement the adapter’s runtime logic. - -This includes: -- Sending requests to the external service -- Handling streaming and non-streaming responses -- Mapping provider responses to TanStack AI types -- Enforcing model-specific options and constraints - -Adapters are implemented per capability, so only implement what your provider supports: - -- Text adapter -- Chat adapter -- Image adapter -- Embeddings adapter -- Video adapter - -Refer to the [OpenAI adapter](https://github.com/TanStack/ai/blob/main/packages/ai-openai/src/adapters/text.ts) for a complete, end-to-end implementation example. - -### 8. Publish and submit a PR - -Once your adapter is complete: -1. Publish it as an npm package -2. Open a PR to the [TanStack AI repository](https://github.com/TanStack/ai/pulls) -3. Add your adapter to the [Community Adapters list in the documentation](https://github.com/TanStack/ai/tree/main/docs/community-adapters) +## Runtime logic -### 9. Sync documentation configuration +Implement only capabilities your provider supports: -After adding your adapter, run the `pnpm run sync-docs-config` in the root of the TanStack AI monorepo. This ensures your adapter appears correctly in the documentation navigation. Open a PR with the generated changes. +- Text / chat +- Image / embeddings / video (as applicable) -### 10. Maintain your adapter +Handle request mapping, streaming vs non-streaming, response → TanStack types, model constraints. Reference: [text adapter](https://github.com/TanStack/ai/blob/main/packages/ai-openai/src/adapters/text.ts). -As a community adapter author, you are responsible for ongoing maintenance. +## Publish and list -This includes: +1. Publish the npm package. +2. Open a PR to [TanStack AI](https://github.com/TanStack/ai/pulls). +3. Add the adapter under [docs/community-adapters](https://github.com/TanStack/ai/tree/main/docs/community-adapters). +4. Run `pnpm run sync-docs-config` at monorepo root; PR the nav changes. -- Tracking upstream provider API changes -- Keeping compatibility with TanStack AI releases -- Addressing issues and feedback from users -- Updating documentation when features change +## Maintain -If you add new features or breaking changes, open a follow-up PR to keep the docs in sync. \ No newline at end of file +- Track provider API changes +- Stay compatible with TanStack AI releases +- Fix user issues +- Update docs on feature or breaking changes diff --git a/docs/community-adapters/mynth.md b/docs/community-adapters/mynth.md index 5686bc87c..fad3a06a6 100644 --- a/docs/community-adapters/mynth.md +++ b/docs/community-adapters/mynth.md @@ -1,7 +1,7 @@ --- title: Mynth id: mynth-adapter -description: "Generate images with Mynth models like Flux, Recraft, Gemini, Qwen, Seedream, Wan, and Grok Imagine in TanStack AI via the Mynth community adapter." +description: "Image generation with Mynth (Flux, Recraft, Gemini, Qwen, Seedream, Wan, Grok Imagine) in TanStack AI." keywords: - tanstack ai - mynth @@ -15,42 +15,22 @@ keywords: # Mynth -The Mynth adapter gives you access to Mynth image generation models through TanStack AI. It is a community adapter for `generateImage()` with typed model IDs, normalized image results, image-to-image support, and Mynth-specific request options through `modelOptions`. +If you need Mynth image models via `generateImage()` → install `@mynthio/tanstack-ai-adapter` (image-only; `@tanstack/ai` ≥ 0.34). Public beta — model list may change. -Mynth is image-only in this package. Reach for it when you want TanStack AI's image generation workflow with Mynth models such as Flux, Recraft, Gemini, Qwen, Seedream, Wan, and Grok Imagine. - -Quick note: Mynth is in public beta, so the model lineup and a few request options are still settling. The adapter tracks the Mynth SDK closely, and we welcome feedback on the API and integration experience. - -## Installation +## Install ```sh -# bun -bun add @mynthio/tanstack-ai-adapter @tanstack/ai - -# pnpm pnpm add @mynthio/tanstack-ai-adapter @tanstack/ai - -# npm -npm install @mynthio/tanstack-ai-adapter @tanstack/ai +# or: bun add … / npm install … ``` -The adapter targets `@tanstack/ai` 0.34 and newer. - -## Authentication - -Set your Mynth API key in the environment: - ```sh MYNTH_API_KEY=mak_... ``` -Keep `MYNTH_API_KEY` on the server only. Never expose it in browser code or public client environment variables, or it may end up in a client bundle. - -You can also pass `apiKey` directly in the adapter config. `baseUrl` is optional and useful for proxies, tests, or custom deployments. +Server-only. Key: [Mynth dashboard](https://mynth.io/dashboard/keys). Optional `apiKey` / `baseUrl` on adapter config. -If you need a key, create one in the [Mynth API keys dashboard](https://mynth.io/dashboard/keys). - -## Quick Start +## Quick start ```ts import { generateImage } from "@tanstack/ai"; @@ -63,16 +43,10 @@ const result = await generateImage({ size: "square", }); -console.log(result.id); -console.log(result.model); console.log(result.images[0]?.url); ``` -TanStack AI adapters are model-bound, so you choose the Mynth model when you create the adapter. - -## Reusable Provider - -Use `createMynthImage()` when you want to share config across multiple adapters: +## Reusable provider ```ts import { generateImage } from "@tanstack/ai"; @@ -87,25 +61,13 @@ const result = await generateImage({ adapter: mynth("google/gemini-3.1-flash-image"), prompt: "A playful paper-cut illustration of a city park in spring", }); - -console.log(result.images[0]?.url); ``` -You can still override shared config per adapter: - -```ts -import { createMynthImage } from "@mynthio/tanstack-ai-adapter"; - -const mynth = createMynthImage(); - -const adapter = mynth("auto", { - baseUrl: "https://proxy.example.com", -}); -``` +Per-adapter override: `mynth("auto", { baseUrl: "https://proxy.example.com" })`. -## Model Options +## Model options -Use TanStack's top-level fields for common options such as `prompt`, `numberOfImages`, and shorthand `size`. Use `modelOptions` for Mynth-specific options: +Top-level: `prompt`, `numberOfImages`, shorthand `size`. Mynth-specific via `modelOptions`: ```ts import { generateImage } from "@tanstack/ai"; @@ -124,30 +86,21 @@ const result = await generateImage({ aspectRatio: "4:5", scale: "4k", }, - output: { - format: "png", - quality: 90, - }, + output: { format: "png", quality: 90 }, rating: true, - metadata: { - requestId: "req_123", - }, + metadata: { requestId: "req_123" }, }, }); ``` -Notes: - -- `modelOptions.negativePrompt` maps to Mynth's `negative_prompt`, and `modelOptions.magicPrompt` maps to `magic_prompt` -- `modelOptions.rating` configures content rating on the result. The older `contentRating` name still works as an alias -- `modelOptions.promptStructured` is still supported for compatibility and expands into `prompt`, `negative_prompt`, and `magic_prompt`. When set, its `positive` overrides the plain `prompt` -- `modelOptions.size` overrides the top-level `size`. Use it when you need structured Mynth size objects, including aspect ratios and an optional `scale: "4k"` -- Top-level `size` is for shorthand values such as `"auto"` and preset strings like `"square"` or `"landscape"` -- `modelOptions.destination` delivers the generation to a configured Mynth destination, overriding any adapter-level or env default +- `negativePrompt` / `magicPrompt` → Mynth `negative_prompt` / `magic_prompt` +- `modelOptions.size` overrides top-level `size` (structured ratios / `scale: "4k"`) +- `promptStructured` still expands to prompt fields; `positive` overrides plain `prompt` +- `destination` delivers to a configured Mynth destination -## Image Inputs (image-to-image) +## Image-to-image -Models that accept image inputs work with TanStack AI's content-part prompts, so you can mix instruction text with reference images for image-to-image, reference-guided, edit, and try-on flows. The adapter maps the image parts onto Mynth's `inputs`: +Models in `MYNTH_IMAGE_INPUT_MODELS` accept content-part prompts: ```ts import { generateImage } from "@tanstack/ai"; @@ -165,63 +118,20 @@ const result = await generateImage({ }); ``` -A few things worth knowing: +URL or data sources supported. Finer intents: `modelOptions.inputs` with explicit `as`. Prompt parts + `modelOptions.inputs` combine (prompt first). Image parts on text-only models are a compile-time error. -- Only models in `MYNTH_IMAGE_INPUT_MODELS` accept image parts. Passing image parts to a text-only model is a compile-time error -- Both URL sources (`{ type: "url", value }`) and inline data sources (`{ type: "data", value, mimeType }`, encoded as a data URI) are supported -- A part's `metadata.role` maps to Mynth's input intent. TanStack's `"character"` maps directly, and the other generic roles fall back to Mynth's automatic detection -- For Mynth's finer-grained intents (`person`, `garment`, `pose`, `style`, `background`, `product`, `object`), pass `modelOptions.inputs` with an explicit `as` -- Image parts from the prompt and entries in `modelOptions.inputs` are combined, with prompt parts first - -## Available Models - -The adapter exports a runtime list and a type union for supported image models: +## Models list + live catalog ```ts import { MYNTH_IMAGE_MODELS, type MynthImageModel } from "@mynthio/tanstack-ai-adapter"; const defaultModel: MynthImageModel = "auto"; - -for (const model of MYNTH_IMAGE_MODELS) { - console.log(model); -} -``` - -This is handy for model selectors, validation, and keeping client and server code in sync. There is a matching `MYNTH_IMAGE_INPUT_MODELS` list (and `MynthImageInputModel` type) for the subset that accepts image inputs. - -Mynth supports model IDs across multiple providers, including `auto`, Flux, Recraft, Gemini, Qwen, Seedream, Imagine, Wan, Grok Imagine, and try-on models. The exported list is a fixed snapshot for type safety. For the live catalog with pricing, use the models endpoint below. - -### Models endpoint - -Mynth exposes a public catalog at `https://api.mynth.io/models`. It does not require an API key, and it carries pricing today with room for more metadata over time. This is the source of truth if you want to render a picker with live pricing rather than the static exported list. - -```ts -const response = await fetch("https://api.mynth.io/models"); -const { data } = await response.json(); - -for (const model of data) { - console.log(model.id, model.displayName, model.pricing); -} +for (const model of MYNTH_IMAGE_MODELS) console.log(model); ``` -Each entry looks roughly like this: +Live pricing: `GET https://api.mynth.io/models` (no key) or SDK `new Mynth().models.list()`. -```jsonc -{ - "id": "black-forest-labs/flux.2-dev", - "displayName": "FLUX.2 Dev", - "pricing": { - "perImage": { "base": "0.01", "4k": "0.04" }, - "perInput": "0.002", - }, -} -``` - -If you already use the Mynth SDK, the same data is available through `new Mynth().models.list()`. - -## Streaming Example - -This adapter also works with TanStack AI's streaming image workflow: +## Streaming endpoint ```ts import { generateImage, toServerSentEventsResponse } from "@tanstack/ai"; @@ -229,75 +139,33 @@ import { mynthImage } from "@mynthio/tanstack-ai-adapter"; export async function POST(request: Request) { const { prompt, model } = await request.json(); - const stream = generateImage({ adapter: mynthImage(model ?? "auto"), prompt, numberOfImages: 1, stream: true, }); - return toServerSentEventsResponse(stream); } ``` -For a full example using `useGenerateImage()`, see the [TanStack Start + Mynth adapter demo](https://github.com/mynthio/oss/tree/main/examples/tanstack-start-ai-mynth-adapter). - -## Supported Capabilities - -- Image generation with `generateImage()` -- Image-to-image with content-part prompts on input-capable models -- Streaming image generation with `stream: true` -- Typed model IDs through `MYNTH_IMAGE_MODELS` and `MynthImageModel` -- Mynth-specific request options through `modelOptions` - -The adapter returns TanStack AI's normalized image result shape: - -- `id`: the Mynth task id -- `model`: the resolved model returned by Mynth, or the requested model as a fallback -- `images`: only successful images are included -- `images[*].revisedPrompt`: included when Mynth enhances the prompt - -## API Reference - -### `mynthImage(model, config?)` - -Creates a Mynth image adapter directly. - -- `model`: a `MynthImageModel` -- `config.apiKey?`: optional override for `MYNTH_API_KEY` -- `config.baseUrl?`: optional base URL override -- `config.destination?`: optional default destination for generated images - -Returns a `MynthImageAdapter` for use with `generateImage()`. - -### `createMynthImage(config?)` - -Creates a reusable provider factory that returns model-bound adapters. - -### `MYNTH_IMAGE_MODELS` - -Readonly array of supported Mynth image model IDs. - -### `MynthImageModel` - -Type union of supported Mynth image model IDs. - -### `MYNTH_IMAGE_INPUT_MODELS` +Demo: [TanStack Start + Mynth](https://github.com/mynthio/oss/tree/main/examples/tanstack-start-ai-mynth-adapter). -Readonly array of model IDs that accept image inputs (image-to-image, try-on). +## Result shape -### `MynthImageInputModel` +- `id` — Mynth task id +- `model` — resolved model (or requested fallback) +- `images` — successful images only; `revisedPrompt` when enhanced -Type union of the model IDs that accept image inputs. +## API -## Limitations +- `mynthImage(model, config?)` — `apiKey?`, `baseUrl?`, `destination?` +- `createMynthImage(config?)` — factory for model-bound adapters +- `MYNTH_IMAGE_MODELS` / `MynthImageModel` +- `MYNTH_IMAGE_INPUT_MODELS` / `MynthImageInputModel` -- This package only provides an image adapter for `generateImage()` -- It does not provide chat or text-generation adapters +Image-only package — no chat/text adapter. -## Next Steps +## Links -- [Mynth SDK README](https://github.com/mynthio/oss/tree/main/packages/sdk) -- [TanStack Start + Mynth adapter demo](https://github.com/mynthio/oss/tree/main/examples/tanstack-start-ai-mynth-adapter) -- [Mynth](https://mynth.io) +- [Mynth SDK](https://github.com/mynthio/oss/tree/main/packages/sdk) · [mynth.io](https://mynth.io) diff --git a/docs/community-adapters/soniox.md b/docs/community-adapters/soniox.md index 5305dea5f..cc88c18a1 100644 --- a/docs/community-adapters/soniox.md +++ b/docs/community-adapters/soniox.md @@ -2,7 +2,7 @@ title: Soniox id: soniox-adapter order: 3 -description: "Transcribe audio with Soniox speech-to-text models in TanStack AI via the Soniox community adapter." +description: "Transcribe audio with Soniox STT models in TanStack AI." keywords: - tanstack ai - soniox @@ -12,19 +12,19 @@ keywords: - community adapter --- -The Soniox adapter provides access to Soniox transcription models. +# Soniox -## Installation +If you need Soniox speech-to-text → install the adapter and call `generateTranscription`. + +## Install ```bash npm install @soniox/tanstack-ai-adapter ``` -## Authentication - -Set `SONIOX_API_KEY` in your environment or pass `apiKey` when creating the adapter. Get your API key from the [Soniox Console](https://console.soniox.com). +Set `SONIOX_API_KEY` or pass `apiKey`. Key: [Soniox Console](https://console.soniox.com). -## Basic Usage +## Basic usage ```typescript import { generateTranscription } from "@tanstack/ai"; @@ -44,24 +44,22 @@ console.log(result.text); console.log(result.segments); ``` -## Basic Usage - Custom API Key +Explicit key: ```typescript import { generateTranscription } from "@tanstack/ai"; import { createSonioxTranscription } from "@soniox/tanstack-ai-adapter"; import { audioFile } from "./audio"; -const adapter = createSonioxTranscription("stt-async-v3", process.env.SONIOX_API_KEY!); +const adapter = createSonioxTranscription( + "stt-async-v3", + process.env.SONIOX_API_KEY!, +); -const result = await generateTranscription({ - adapter, - audio: audioFile, -}); +const result = await generateTranscription({ adapter, audio: audioFile }); ``` -## Adapter Configuration - -Use `createSonioxTranscription` to customize the adapter instance: +## Adapter config ```typescript import { createSonioxTranscription } from "@soniox/tanstack-ai-adapter"; @@ -73,19 +71,11 @@ const adapter = createSonioxTranscription("stt-async-v3", process.env.SONIOX_API }); ``` -Options: - -- `apiKey` - Override `SONIOX_API_KEY` (required when using `createSonioxTranscription`) -- `baseUrl` - Custom API base URL. Default is `https://api.soniox.com` -- `headers` - Additional request headers -- `timeout` - Transcription timeout in milliseconds (default: 180000) -- `pollingIntervalMs` - Transcription polling interval in milliseconds (default: 1000) - -See the [Soniox regional endpoints](https://soniox.com/docs/stt/data-residency#regional-endpoints) if you need data residency. +- `apiKey` — required with `createSonioxTranscription` +- `baseUrl` — default `https://api.soniox.com` ([regional endpoints](https://soniox.com/docs/stt/data-residency#regional-endpoints)) +- `headers` · `timeout` (default 180000) · `pollingIntervalMs` (default 1000) -## Transcription Options - -Per-request options are passed via `modelOptions`: +## modelOptions ```typescript import { generateTranscription } from "@tanstack/ai"; @@ -99,61 +89,20 @@ const result = await generateTranscription({ languageHints: ["en", "es"], enableLanguageIdentification: true, enableSpeakerDiarization: true, - context: { - terms: ["Soniox", "TanStack"], - }, + context: { terms: ["Soniox", "TanStack"] }, }, }); ``` -Available options: - -- `languageHints` - Array of ISO language codes to bias recognition -- `languageHintsStrict` - When true, rely more heavily on language hints (not supported by all models) -- `enableLanguageIdentification` - Automatically detect spoken language -- `enableSpeakerDiarization` - Identify and separate different speakers -- `context` - Additional context to improve accuracy -- `clientReferenceId` - Optional client-defined reference ID -- `webhookUrl` - Webhook URL for completion notifications -- `webhookAuthHeaderName` - Webhook authentication header name -- `webhookAuthHeaderValue` - Webhook authentication header value -- `translation` - Translation configuration - -Check the [Soniox API reference](https://soniox.com/docs/stt/api-reference/transcriptions/create_transcription) for more details. - -## Language Hints - -Soniox automatically detects and transcribes speech in [60+ languages](https://soniox.com/docs/stt/concepts/supported-languages). When you know which languages are likely to appear in your audio, provide `languageHints` to improve accuracy by biasing recognition toward those languages. +- `languageHints` / `languageHintsStrict` — bias ([60+ languages](https://soniox.com/docs/stt/concepts/supported-languages)); TanStack `language` merges into hints +- `enableLanguageIdentification` · `enableSpeakerDiarization` +- `context` · `clientReferenceId` · webhook fields · `translation` -Language hints do not restrict recognition. If you pass the TanStack `language` option, this adapter merges it into `languageHints`. - -```typescript -import { generateTranscription } from "@tanstack/ai"; -import { sonioxTranscription } from "@soniox/tanstack-ai-adapter"; -import { audio } from "./audio"; - -const result = await generateTranscription({ - adapter: sonioxTranscription("stt-async-v3"), - audio, - modelOptions: { - languageHints: ["en", "es"], - }, -}); -``` - -For more details, see the [Soniox language hints documentation](https://soniox.com/docs/stt/concepts/language-hints). +API: [create transcription](https://soniox.com/docs/stt/api-reference/transcriptions/create_transcription). ## Context -Provide custom context to improve transcription and translation accuracy. Context helps the model understand your domain, recognize important terms, and apply custom vocabulary. - -The `context` object supports four optional sections: - ```typescript -import { generateTranscription } from "@tanstack/ai"; -import { sonioxTranscription } from "@soniox/tanstack-ai-adapter"; -import { audio } from "./audio"; - const result = await generateTranscription({ adapter: sonioxTranscription("stt-async-v3"), audio, @@ -162,96 +111,34 @@ const result = await generateTranscription({ general: [ { key: "domain", value: "Healthcare" }, { key: "topic", value: "Diabetes management consultation" }, - { key: "doctor", value: "Dr. Martha Smith" }, ], text: "The patient has a history of...", - terms: ["Celebrex", "Zyrtec", "Xanax"], - translationTerms: [ - { source: "Mr. Smith", target: "Sr. Smith" }, - { source: "MRI", target: "RM" }, - ], + terms: ["Celebrex", "Zyrtec"], + translationTerms: [{ source: "MRI", target: "RM" }], }, }, }); ``` -For more details, see the [Soniox context documentation](https://soniox.com/docs/stt/concepts/context). +Details: [Soniox context](https://soniox.com/docs/stt/concepts/context). ## Translation -Configure translation for your transcriptions: - -```typescript -import { generateTranscription } from "@tanstack/ai"; -import { sonioxTranscription } from "@soniox/tanstack-ai-adapter"; -import { audio } from "./audio"; - -const result = await generateTranscription({ - adapter: sonioxTranscription("stt-async-v3"), - audio, - modelOptions: { - translation: { - type: "one_way", - targetLanguage: "es", - }, - }, -}); -``` - -For two-way translation: - ```typescript -import { generateTranscription } from "@tanstack/ai"; -import { sonioxTranscription } from "@soniox/tanstack-ai-adapter"; -import { audio } from "./audio"; - -const result = await generateTranscription({ - adapter: sonioxTranscription("stt-async-v3"), - audio, - modelOptions: { - translation: { - type: "two_way", - languageA: "en", - languageB: "es", - }, - }, -}); -``` - -When using translation, the API returns both transcription tokens and translation tokens. The `segments` array always includes only transcription tokens. To access translation tokens, use `providerMetadata` and filter by `translation_status === "translation"`. - -### Accessing Raw Tokens - -When using translation or working with multilingual audio, you may need access to raw tokens with per-token language information and translation status. The adapter attaches a non-standard `providerMetadata` field at runtime: - -```typescript ignore -// ignore: providerMetadata is a non-standard runtime extension not on TranscriptionResult; -// accessing it requires a cast that cannot be expressed without `as`. -const result = await generateTranscription({ - adapter: sonioxTranscription("stt-async-v3"), - audio, - modelOptions: { - translation: { type: "one_way", targetLanguage: "es" }, - }, -}); - -const rawTokens = (result as any).providerMetadata?.soniox?.tokens; +// one-way +modelOptions: { + translation: { type: "one_way", targetLanguage: "es" }, +} -if (rawTokens) { - rawTokens.forEach((token) => { - // token.text - token text - // token.start_ms - start time in milliseconds - // token.end_ms - end time in milliseconds - // token.language - detected language for this token - // token.translation_status - translation status (if translation enabled) - // token.speaker - speaker identifier - // token.confidence - confidence score - }); +// two-way +modelOptions: { + translation: { type: "two_way", languageA: "en", languageB: "es" }, } ``` -## Next Steps +`segments` are transcription tokens only. Translation tokens live on runtime `providerMetadata` (non-standard field — see package docs / filter `translation_status === "translation"`). + +## Links -- [Soniox Console](https://console.soniox.com) - Manage API keys and projects -- [Soniox API docs](https://soniox.com/docs) - Complete Soniox API reference -- [Transcription Guide](../media/transcription) - Learn about transcription in TanStack AI +- [Console](https://console.soniox.com) · [Soniox docs](https://soniox.com/docs) +- [Transcription Guide](../media/transcription) diff --git a/docs/comparison/vercel-ai-sdk.md b/docs/comparison/vercel-ai-sdk.md index 0375e2d0c..ef7223028 100644 --- a/docs/comparison/vercel-ai-sdk.md +++ b/docs/comparison/vercel-ai-sdk.md @@ -2,7 +2,7 @@ title: TanStack AI vs Vercel AI SDK id: vercel-ai-sdk order: 1 -description: "How TanStack AI compares to the Vercel AI SDK — feature matrix, philosophy, type safety, tool calling, streaming, and framework support." +description: "Feature matrix and philosophy: TanStack AI vs Vercel AI SDK — types, tools, streaming, frameworks." keywords: - tanstack ai - vercel ai sdk @@ -14,60 +14,50 @@ keywords: - llm --- -Both TanStack AI and Vercel AI SDK are open-source TypeScript toolkits for building AI-powered applications. They share common ground - streaming chat, tool calling, multi-provider support, and deploy-anywhere flexibility - but they approach the problem from fundamentally different directions. +If you need to pick an SDK → scan the matrix, then jump to “When to choose” at the bottom. Trying TanStack → [Quick Start](../getting-started/quick-start). -TanStack AI treats AI as a **library composition problem**. Every piece - adapters, tools, agent loops, transport, UI - is a composable building block. You import what you need, compose it how you want, and ship it wherever you want. No platform layer, no gateway abstraction, no implicit associations. +Both are open-source TypeScript toolkits (streaming chat, tools, multi-provider). **TanStack AI** is library composition (import adapters/tools/transport/UI; no platform layer). **Vercel AI SDK** is a broader full-stack surface with optional platform/gateway integration. -Vercel AI SDK treats AI as a **full-stack platform problem**. It provides a broad surface area of primitives with optional platform integration for gateway routing, observability, and deployment optimization. +Versions: TanStack AI as of this writing; Vercel AI SDK `ai@6.x` (v6 Dec 2025; v7 pre-release at writing). -This article compares the two SDKs from TanStack AI's perspective, with honest acknowledgment of where each excels. - -## Feature Comparison - -Versions referenced below: TanStack AI as of this writing; Vercel AI SDK `ai@6.x` (v6.0.0 shipped December 2025; v7 is in pre-release at the time of writing). +## Feature matrix | Feature | TanStack AI | Vercel AI SDK | |---------|------------|---------------| | License | MIT | Apache 2.0 | | Hosting | Works anywhere | Works anywhere | -| Providers | 9 official + community; OpenRouter routes to 100s of models and the `openaiCompatible` adapter connects to any OpenAI-compatible endpoint | ~38 first-party provider packages (plus community providers); 100+ models via AI Gateway | -| Framework Hooks | React, Solid, Svelte, Vue, Preact (+ React Native) | React, Vue, Svelte, Angular (Solid is community-maintained) | +| Providers | 9 official + community; OpenRouter → 100s of models; `openaiCompatible` for any OpenAI-compatible endpoint | ~38 first-party packages (+ community); 100+ models via AI Gateway | +| Framework Hooks | React, Solid, Svelte, Vue, Preact (+ React Native) | React, Vue, Svelte, Angular (Solid community-maintained) | | Generation UI Hooks | One hook per activity: chat, structured output, image, audio, speech, transcription, summarize, video, realtime | `useChat`, `useCompletion`, `useObject` | -| Wire Protocol | Native AG-UI events end to end | Proprietary UI Message Stream; AG-UI via external translation layer | -| Streaming | Built-in with configurable chunk strategies | Built-in with progressive delivery | -| Tool Calling | Isomorphic `.server()` / `.client()` system | `tool()` objects; client execution via `onToolCall` | -| Agent Loop Control | Composable strategy functions `(state) => boolean` | `stopWhen` conditions + `Agent` (`ToolLoopAgent`) class | -| Tool Approval | Per-tool `needsApproval` with batched approval flow | Per-tool `needsApproval` (human-in-the-loop) | -| Type Safety | Per-model type narrowing | Per-provider types | -| Tree-Shaking | Separate adapter per activity (text, image, speech, etc.) | Monolithic provider packages | -| Lazy Tool Discovery | Built-in - works across every provider | Anthropic-only tool search with `deferLoading` (provider-hosted) | -| Connection Adapters | SSE, HTTP stream, XHR (SSE/stream), RPC, direct async iterables, `fetcher`, custom | SSE-based data stream protocol (`ChatTransport`) | -| Middleware | App-level lifecycle hooks (config, iterations, chunks, tool calls, usage, errors) | Model-level wrapping via `wrapLanguageModel()` | -| Extend Adapter | Add custom/fine-tuned models with per-model type narrowing | `customProvider()` / `createProviderRegistry()` (string model ids) | -| Structured Outputs | Typed `StructuredOutputPart`, streamed alongside tools and preserved per turn in message history | `generateObject()` / `streamObject()` / `Output` API (per-call; no structured-output message part) | -| Image Generation | Stable API with per-model type safety (OpenAI, Gemini, Grok, OpenRouter, fal.ai) | `generateImage()` (stable) | -| Video Generation | Stable API with async job lifecycle (OpenAI, fal.ai) | `experimental_generateVideo()` | -| Text-to-Speech | Stable API, 6 output formats, speed control (OpenAI, Gemini, Grok, ElevenLabs, fal.ai) | `generateSpeech()` (experimental) | -| Transcription | Stable API with word timestamps and diarization (OpenAI, Grok, ElevenLabs, fal.ai) | `transcribe()` (experimental) | -| Audio / Music Generation | `generateAudio()` for music & sound effects (Gemini, ElevenLabs, fal.ai) | - | -| Summarization | Dedicated `summarize()` with streaming and style options | - | -| Code Execution | Node.js, Cloudflare Workers, QuickJS sandboxes you run yourself | Provider-hosted code-execution tools (Anthropic, xAI, OpenAI) | -| Code Mode Skills | LLM-writable persistent skill library | - | -| Coding Agent Sandboxes | First-party Grok Build, Claude Code, Codex, OpenCode harnesses **+ any ACP agent** via `acpCompatible`; runs on local-process, Docker, Daytona, Vercel, Sprites, or Cloudflare | `HarnessAgent` (experimental) — Claude Code, Codex, Pi, OpenCode, Deep Agents; centered on Vercel Sandbox | -| Realtime Voice | OpenAI, Grok, and ElevenLabs with VAD modes and tool support | - | -| DevTools | Isomorphic in-app panel via TanStack DevTools (all frameworks, media previews) | `devToolsMiddleware` + local inspector (server-side, dev-only) | -| Debug Logging | One flag, per-category toggles, pluggable logger | Warning logs + experimental telemetry hooks | -| MCP Client | Standalone host-side client (`@tanstack/ai-mcp`) + provider-routed `mcpTool()` | Built-in (`@ai-sdk/mcp`, stable) | -| MCP Apps (Interactive Widgets) | `ui://` widgets via `@mcp-ui/client`; React + Preact + framework-agnostic bridge; multi-server routing, pluggable session store, link-scheme hardening | `experimental_MCPAppRenderer` (React only); model-vs-app tool split; iframe sandbox + allowlist | -| Platform Association | None - pure library | Optional Vercel integration | - -## Where TanStack AI Excels - -### Per-Model Type Safety - -When you select a provider and model, TypeScript narrows the exact options, capabilities, and input modalities available for that specific model - not a union of everything the provider supports. - -Each provider adapter contains a comprehensive `model-meta.ts` that maps every model to its capabilities: supported input modalities, context windows, and provider-specific options. When you write `openaiText('gpt-5.5')`, the type system knows exactly what that model can do. +| Wire Protocol | Native AG-UI events end to end | Proprietary UI Message Stream; AG-UI via external translation | +| Streaming | Built-in, configurable chunk strategies | Built-in progressive delivery | +| Tool Calling | Isomorphic `.server()` / `.client()` | `tool()`; client via `onToolCall` | +| Agent Loop Control | `(state) => boolean` strategies + `combineStrategies` | `stopWhen` + `Agent` (`ToolLoopAgent`) | +| Tool Approval | Per-tool `needsApproval`, batched | Per-tool `needsApproval` | +| Type Safety | Per-model narrowing | Per-provider types | +| Tree-Shaking | Separate adapter per activity | Monolithic provider packages | +| Lazy Tool Discovery | Built-in, every provider | Anthropic-only tool search + `deferLoading` | +| Connection Adapters | SSE, HTTP stream, XHR, RPC, async iterables, `fetcher`, custom | SSE data stream (`ChatTransport`) | +| Middleware | App-level lifecycle hooks | Model wrap via `wrapLanguageModel()` | +| Extend Adapter | Custom models + per-model type narrowing | `customProvider()` / registry (string ids) | +| Structured Outputs | Typed `StructuredOutputPart` in message history | `generateObject` / `streamObject` / `Output` (per-call) | +| Image / Video / TTS / Transcription | Stable APIs; video async job lifecycle; multi-provider | `generateImage` stable; video/speech/transcribe experimental or batch-only | +| Audio / Music | `generateAudio()` | — | +| Summarization | Dedicated `summarize()` | — (prompt `generateText`) | +| Code Execution | Node / Cloudflare / QuickJS isolates you run | Provider-hosted code tools | +| Code Mode Skills | LLM-writable skill library | — | +| Coding Agent Sandboxes | Grok Build, Claude Code, Codex, OpenCode + any ACP via `acpCompatible`; local/Docker/Daytona/Vercel/Sprites/CF | `HarnessAgent` (experimental); centered on Vercel Sandbox | +| Realtime Voice | OpenAI, Grok, ElevenLabs | — | +| DevTools | In-app panel, all frameworks | Server-side inspector + middleware | +| MCP Client | `@tanstack/ai-mcp` + provider-routed `mcpTool()` | `@ai-sdk/mcp` | +| MCP Apps | React + Preact + agnostic bridge; multi-server | React experimental renderer | +| Platform Association | None | Optional Vercel integration | + +## Where TanStack AI excels + +### Per-model type safety + +Selecting a model narrows options, modalities, and capabilities for **that** model (from adapter `model-meta.ts`), not a provider union. ```ts import { chat } from '@tanstack/ai' @@ -86,11 +76,11 @@ const stream = chat({ }) ``` -If you pass an image content part to a text-only model, TypeScript catches it at compile time. +Image part on a text-only model → compile error. -### Tree-Shakeable Adapters +### Tree-shakeable adapters -Every AI activity - chat, summarization, image generation, speech, transcription, video - is a separate import. Every provider exposes separate adapter functions per activity. If your app only uses chat, image generation code never enters your bundle. +Each activity is a separate import. Chat-only apps do not bundle image/speech code. ```ts ignore // Only chat code is bundled - nothing else @@ -102,11 +92,9 @@ import { chat, generateImage } from '@tanstack/ai' import { openaiText, openaiImage } from '@tanstack/ai-openai' ``` -This is architectural, not incidental. Each adapter implements a specific interface (`TextAdapter`, `ImageAdapter`, `TTSAdapter`, etc.) and lives in its own module. Modern bundlers eliminate everything you don't import. - -### Isomorphic Tools +### Isomorphic tools -`toolDefinition()` creates a shared contract - name, description, input schema, output schema - that can be implemented for different runtimes. `.server()` adds a server-side implementation with access to databases and APIs. `.client()` adds a client-side implementation that runs in the browser. +One contract; `.server()` / `.client()` implementations. Shared Zod validation; compile-time `ServerTool` vs `ClientTool`. ```ts import { toolDefinition } from '@tanstack/ai' @@ -143,13 +131,11 @@ const addToCartClient = addToCartDef.client(async ({ itemId, quantity }) => { }) ``` -The same schema validates inputs and outputs on both sides. The type system tracks whether a tool is a `ServerTool` or `ClientTool` at compile time. +Vercel: `tool()` + browser `onToolCall` / `addToolOutput` when there is no `execute`. No single definition that yields both `.server()` and `.client()`. -Vercel AI SDK defines tools with a `tool()` helper and does support client-side execution - a tool with no `execute` function is handled in the browser via the UI hook's `onToolCall` callback, with the result returned through `addToolOutput` (renamed from `addToolResult` in v6). What it doesn't have is a single shared contract that produces separate `.server()` and `.client()` implementations: server and client tool code are declared independently rather than derived from one definition. +### Agent loop strategies -### Composable Agent Loop Strategies - -TanStack AI provides agent loop control as composable pure functions. Each strategy is `(state) => boolean` - return `true` to continue, `false` to stop. +Continue/stop as `(state) => boolean`; AND via `combineStrategies`. ```ts import { chat, maxIterations, untilFinishReason, combineStrategies } from '@tanstack/ai' @@ -169,7 +155,7 @@ const stream = chat({ }) ``` -`combineStrategies` composes them with AND logic - all strategies must agree to continue. You can add custom strategies alongside built-in ones: +Custom predicates inline: ```ts import { maxIterations, untilFinishReason, combineStrategies } from '@tanstack/ai' @@ -183,13 +169,11 @@ combineStrategies([ ]) ``` -Vercel AI SDK (v5+) controls agent loops via `stopWhen`, which accepts composable stopping conditions like `stepCountIs(n)` and `hasToolCall(name)` (the default is `stepCountIs(20)`), and v6 adds a dedicated `Agent` abstraction (the `ToolLoopAgent` class) that bundles model, tools, instructions, and loop settings into a reusable object. The remaining difference is in the composition model: TanStack AI's strategies are arbitrary `(state) => boolean` predicates you write inline and combine with `combineStrategies`, so a stopping condition can encode any business logic (token budgets, cost ceilings, custom state checks) without waiting for a built-in condition to exist. Vercel's `stopWhen` also accepts custom functions, so the gap here is smaller than it once was. - -### Lazy Tool Discovery +Vercel: `stopWhen` (`stepCountIs`, `hasToolCall`, custom fns) + v6 `ToolLoopAgent`. Custom stop fns exist on both sides; TanStack treats arbitrary predicates as first-class. -When your application has dozens of tools, sending all their schemas to the LLM on every request wastes tokens. TanStack AI solves this with lazy tool discovery. +### Lazy tool discovery -Mark tools as `lazy: true` and they won't be sent to the LLM initially. Instead, a synthetic discovery tool is injected that lets the LLM request tool schemas on demand: +`lazy: true` tools stay off the initial prompt; a synthetic discovery tool loads schemas on demand (all providers). ```ts import { toolDefinition } from '@tanstack/ai' @@ -204,24 +188,16 @@ const searchProducts = toolDefinition({ }) ``` -The LLM sees a lightweight `__lazy__tool__discovery__` tool listing available tool names. When it needs one, it calls the discovery tool to get the full schema, then uses the real tool. For applications with large tool inventories, this significantly reduces per-request token costs. +Vercel v6: Anthropic tool search + `deferLoading` only (provider-hosted). -Vercel AI SDK 6 added a provider-specific analogue for Anthropic: the tool search provider tool (`toolSearchBm25` / `toolSearchRegex`) with per-tool `deferLoading`, where deferred tools are excluded from the initial prompt and discovered on demand. It's Anthropic-only and runs provider-side; TanStack AI's lazy discovery works across every provider and runs in your own agent loop. +### MCP -### Model Context Protocol (MCP) +**Two paths (mixable in one `chat()`)** -TanStack AI connects to MCP servers two ways, and you can mix them in a single `chat()` run: +1. Host-side: `@tanstack/ai-mcp` — `createMCPClient` / `createMCPClients`, HTTP/SSE/stdio, OAuth +2. Provider-routed: `mcpTool()` — provider connects (OpenAI Responses, Anthropic) -- **Host-side client** (`@tanstack/ai-mcp`) - your server connects directly to any MCP server. `createMCPClient` (single server) and `createMCPClients` (multi-server pool) discover and execute tools, read resources, and fetch prompts over Streamable HTTP, SSE, or stdio transports, with OAuth 2.1 (`authProvider`) and static-token auth. -- **Provider-routed** (`mcpTool()`) - the *provider* connects to the MCP server on your behalf (OpenAI Responses API, Anthropic), so no MCP traffic flows through your server at all. - -The host-side client goes beyond basic discovery: - -- **Managed lifecycle** - hand clients to `chat()` via the `mcp` option and it discovers tools and closes connections when the run ends - no `try/finally` per route. -- **Multi-server pools** - `createMCPClients` connects to many servers in parallel, auto-prefixing each server's tools to prevent name collisions. -- **Three modes of type safety** - untyped auto-discovery, `toolDefinition()`-typed allowlists with Zod validation, or fully generated per-server types via the `tanstack-ai-mcp` CLI. -- **Lazy discovery** - `tools({ lazy: true })` defers sending tool schemas to the LLM, plugging into TanStack AI's lazy tool discovery to cut token usage on tool-heavy servers. -- **Resources & prompts** - inject MCP resources and prompts into a run with `mcpResourceToContentPart` and `mcpPromptToMessages`. +**Host-side extras:** managed lifecycle via `mcp` on `chat()`, multi-server pools + prefixes, typed allowlists/codegen CLI, lazy discovery, resources/prompts helpers. ```ts import { chat } from '@tanstack/ai' @@ -242,15 +218,15 @@ const stream = chat({ }) ``` -Vercel AI SDK's `@ai-sdk/mcp` (`createMCPClient`) is a stable host-side client with HTTP/SSE transports, OAuth, resource reading, and prompt templates. TanStack AI's `@tanstack/ai-mcp` matches that surface and adds generated end-to-end types, multi-server pools, lazy discovery, a managed `chat()` lifecycle, and the provider-routed `mcpTool()` alternative. +Vercel `@ai-sdk/mcp` covers host-side HTTP/SSE, OAuth, resources, prompts. TanStack adds pools, codegen types, lazy discovery, managed lifecycle, and `mcpTool()`. -### MCP Apps (Interactive Widgets) +### MCP Apps (widgets) -Both SDKs implement [MCP Apps](https://modelcontextprotocol.io) — the ratified MCP extension (standardized 2026-01-26) where a server returns a `ui://` resource so a tool result renders as an interactive widget in a sandboxed iframe instead of raw JSON. Both keep the widget HTML out of model input and render it in a sandboxed iframe with a tool allowlist and safe link handling. TanStack AI's implementation is more built out along three axes: +Both implement [MCP Apps](https://modelcontextprotocol.io) (`ui://` sandboxed widgets). TanStack differences: -- **More than one framework.** Widgets render via `@tanstack/ai-react/mcp-apps` and `@tanstack/ai-preact/mcp-apps`, and the bridge that routes widget actions (`createMcpAppBridge`) lives in the framework-agnostic `@tanstack/ai-client`, so a new framework only needs a thin renderer. Vercel's `experimental_MCPAppRenderer` and its bridge live in `@ai-sdk/react` — React only. -- **Multi-server routing.** Each `UIResourcePart` carries a `serverId` (the pool prefix from `createMCPClients`), and interactive calls route back to the exact server that produced the widget — automatically when you run a multi-server pool. The call handler also enforces an unconditional same-server exposure check (`toolName` must be a tool that server actually exposes) with an optional `allowTool` restriction AND-ed on top. -- **Session persistence & serverless-safety.** The call handler reconnects per call from a transport descriptor (stateless, serverless-safe by default), and stateful transports opt into a pluggable `McpSessionStore` — an `inMemoryMcpSessionStore` ships, and SQL/KV backends drop in behind the same interface. +1. React + Preact renderers; bridge in framework-agnostic `@tanstack/ai-client` +2. Multi-server routing via `serverId` + same-server tool checks +3. Stateless reconnect by default; pluggable `McpSessionStore` for stateful transports ```tsx import { useChat, useMcpAppBridge } from '@tanstack/ai-react' @@ -289,25 +265,13 @@ export function Chat() { } ``` -Vercel AI SDK 7 covers the core flow and adds one thing TanStack AI doesn't: `splitMCPAppTools`, which separates *model-visible* tools from *app-only* tools the widget can call but the model never sees. Both implementations are new — Vercel marks its renderer `experimental_`, and TanStack AI's writeback of widget tool-calls into chat history is still out of scope. See the [MCP Apps guide](../mcp/apps) for the full API. - -### Headless Client Architecture - -`ChatClient` is a framework-agnostic class that manages the entire chat lifecycle - streaming, message state, tool execution, approval flows, and connection management. Every framework integration wraps this single client: +Vercel: `experimental_MCPAppRenderer` (React only) + `splitMCPAppTools` (model-visible vs app-only). Full API: [MCP Apps guide](../mcp/apps). -- `@tanstack/ai-react` - `useChat` hook wraps `ChatClient` -- `@tanstack/ai-solid` - `useChat` hook wraps `ChatClient` -- `@tanstack/ai-vue` - `useChat` composable wraps `ChatClient` -- `@tanstack/ai-svelte` - `createChat` wraps `ChatClient` (Svelte 5 runes) -- `@tanstack/ai-preact` - `useChat` hook wraps `ChatClient` +### Headless client -No framework-specific logic in the core. If a new framework emerges, it only needs a thin reactive wrapper. +`ChatClient` owns streaming, messages, tools, approvals, connections. Framework packages are thin wrappers: React/Solid/Vue/Preact `useChat`, Svelte `createChat`. Persistence adapter + typed runtime `context` supported. -`ChatClient` also accepts a persistence adapter (`ChatClientPersistence`) for saving and restoring conversations client-side, and a typed runtime `context` that flows through to tools and middleware. - -### Connection Adapters - -TanStack AI ships six built-in connection adapters plus a custom adapter interface: +### Connection adapters ```ts import { @@ -340,11 +304,9 @@ rpcStream((messages, data) => api.streamResponse(messages, data)) // Or implement your own ConnectionAdapter ``` -Each adapter accepts static or dynamic (function-based) URLs and options. There's also a lighter-weight `fetcher` option on `ChatClient` / `useChat` for wiring a server function directly without a full adapter. Swap transport without changing application code. Vercel AI SDK centers on its SSE-based data stream protocol and a `ChatTransport` interface for extensibility, but doesn't ship the same breadth of built-in adapters - notably the XHR variants for React Native. - -### Extend Adapter +Static or dynamic URLs/options. Lighter `fetcher` option on `ChatClient` / `useChat` for server functions without a full adapter. Vercel centers on SSE data stream + `ChatTransport`; fewer built-ins (no XHR RN pair). -When you use fine-tuned models, OpenAI-compatible proxies, or custom model endpoints, `extendAdapter()` lets you add them to any provider adapter with full type safety: +### Extend adapter ```ts import { extendAdapter, createModel } from '@tanstack/ai' @@ -361,11 +323,11 @@ const myOpenai = extendAdapter(openaiText, customModels) const adapter = myOpenai('my-fine-tuned-gpt4') ``` -Your custom models appear in autocomplete alongside official ones. Vercel AI SDK covers the registration half of this with the now-stable `customProvider()` (custom and aliased model ids, settings overrides) and `createProviderRegistry()`; the difference is type-safety depth - registry model ids are plain strings, while `extendAdapter()` gives custom models the same literal-type narrowing and per-model option gating as official ones. +Vercel: `customProvider()` / `createProviderRegistry()` with string model ids — registration without the same literal-type narrowing. ### Middleware -TanStack AI's middleware system hooks into every stage of the `chat()` lifecycle: configuration, streaming, tool execution, usage tracking, and completion. Each middleware is a plain object with named hooks that fire at specific phases. +App-level hooks across config, chunks, tools, usage, finish/abort/error: ```ts import { chat, EventType, type ChatMiddleware, type StreamChunk } from '@tanstack/ai' @@ -406,53 +368,43 @@ const stream = chat({ }) ``` -The available hooks cover the full lifecycle: - | Hook | Purpose | |------|---------| -| `onConfig` | Transform messages, tools, temperature, system prompts per iteration | -| `onStructuredOutputConfig` | Transform the structured-output schema/config before the call | -| `onStart` | Setup tasks (timers, logging) | -| `onIteration` | Observe each agent-loop iteration | -| `onChunk` | Transform, expand, or drop individual stream chunks | -| `onBeforeToolCall` | Intercept tool calls: transform args, skip execution, or abort the run | -| `onAfterToolCall` | Observe tool results, timing, and errors | -| `onToolPhaseComplete` | Observe the full batch of tool results for an iteration (e.g. aggregate approval state) | -| `onUsage` | Track token usage per iteration | -| `onFinish` / `onAbort` / `onError` | Terminal hooks (exactly one fires per run) | +| `onConfig` | Transform messages, tools, temperature, system per iteration | +| `onStructuredOutputConfig` | Transform structured-output schema/config | +| `onStart` / `onIteration` | Setup; observe agent-loop iterations | +| `onChunk` | Transform/expand/drop stream chunks | +| `onBeforeToolCall` / `onAfterToolCall` / `onToolPhaseComplete` | Intercept tools; observe batches | +| `onUsage` / `onFinish` / `onAbort` / `onError` | Usage + terminals | -Middleware compose naturally. `onConfig` pipes through each middleware in order. `onChunk` pipes chunks through each middleware (if one drops a chunk, later middleware never see it). `onBeforeToolCall` uses first-win semantics: the first middleware that returns a decision short-circuits the rest. +Compose: `onConfig`/`onChunk` pipe; `onBeforeToolCall` first-win short-circuit. -TanStack AI ships several built-in middleware. `toolCacheMiddleware` and `contentGuardMiddleware` come from the `@tanstack/ai/middlewares` subpath, and `otelMiddleware` from `@tanstack/ai/middlewares/otel` (kept on its own subpath so `@opentelemetry/api` stays an optional peer). `toolCacheMiddleware` caches tool results by name and arguments with configurable TTL, LRU eviction, and pluggable storage backends (Redis, localStorage, etc.). +Built-ins: `toolCacheMiddleware`, `contentGuardMiddleware` from `@tanstack/ai/middlewares`; `otelMiddleware` from `@tanstack/ai/middlewares/otel`. ```ts import { toolCacheMiddleware, contentGuardMiddleware } from '@tanstack/ai/middlewares' import { otelMiddleware } from '@tanstack/ai/middlewares/otel' ``` -Vercel AI SDK takes a different approach: `wrapLanguageModel()` wraps a model instance with middleware that can intercept and transform calls (and v6 adds `wrapEmbeddingModel()`). It ships several built-in middleware (`extractReasoningMiddleware`, `simulateStreamingMiddleware`, `defaultSettingsMiddleware`, and the new `devToolsMiddleware`), but these all operate at the model level rather than the application level. v6 also exposes per-call options that cover slices of this surface: `experimental_transform` for stream transforms, `experimental_onToolCallStart` / `experimental_onToolCallFinish` callbacks, `prepareStep` for per-step config changes, and `experimental_repairToolCall`. What it doesn't have is a unified middleware system at the application level - named, reusable middleware objects whose hooks span the whole lifecycle, compose in order, and can short-circuit tool calls with first-win semantics. - -### No Platform Association +Vercel: model-level `wrapLanguageModel()` (+ `wrapEmbeddingModel` in v6), built-in middleware, and per-call experimental callbacks — not a single app-lifecycle middleware object model. -TanStack AI is a pure library. There's no optional platform layer, no gateway abstraction, no hosting-specific features, and no deployment-specific optimizations. Your AI code carries no implicit association with any deployment platform. +### No platform association -This isn't just philosophical - it means no accidental dependencies on platform-specific features, no gateway abstractions that subtly encourage vendor adoption, and no marketing surface embedded in your technical stack. +Pure library: no gateway, hosting-specific features, or deployment coupling. -### Code Execution Sandboxes +### Code execution sandboxes -TanStack AI provides three isolate drivers for safe code execution in AI workflows: +| Package | Runtime | +| --- | --- | +| `@tanstack/ai-isolate-node` | Node via `isolated-vm` | +| `@tanstack/ai-isolate-cloudflare` | Cloudflare Workers | +| `@tanstack/ai-isolate-quickjs` | QuickJS | -- **`@tanstack/ai-isolate-node`** - Node.js sandbox via `isolated-vm` -- **`@tanstack/ai-isolate-cloudflare`** - Cloudflare Workers sandbox -- **`@tanstack/ai-isolate-quickjs`** - QuickJS lightweight sandbox +Same `IsolateDriver` interface. Powers code mode + `@tanstack/ai-code-mode-skills` (LLM-writable persistent skills). Vercel has no first-party isolate drivers (provider-hosted tools only). -All three implement the same `IsolateDriver` interface, so you can swap execution environments without changing application code. This powers TanStack AI's code mode - where the LLM writes and executes code as part of the agent loop. A companion `@tanstack/ai-code-mode-skills` package lets you give code mode a persistent, reusable library of runtime skills. Skills are LLM-writable: the model can save working TypeScript snippets, list and reuse them across sessions, with trust strategies controlling what gets promoted to a first-class tool. The closest AI SDK analogues - Anthropic's provider-hosted code execution and developer-uploaded skills, or pre-authored file skills loaded into a sandbox - are provider-specific and static; none give the model a persistent, provider-agnostic skill library it builds itself. +### Coding agent sandboxes -Vercel AI SDK does not provide built-in code execution sandboxes (though some providers expose their own server-side code execution as provider-executed tools). - -### Coding Agent Sandboxes - -Separately from the JS isolates above, TanStack AI can put a full **coding-agent CLI** — Claude Code, Codex, Grok Build, OpenCode, or any ACP-compliant agent — inside an isolated sandbox with a real filesystem, shell, and a cloned repo, and stream its work back through `chat()` like any other run. A sandboxed run composes three swappable pieces: a **provider** (where it runs), a **workspace** (what the agent sees), and a **harness adapter** (which agent runs). The sandbox is a `chat()` middleware, so the agent's edits and commands arrive as the same AG-UI stream every `useChat` UI already renders. +Run Claude Code / Codex / Grok Build / OpenCode / any ACP agent in a sandbox; stream via `chat()` middleware. ```ts import { chat } from '@tanstack/ai' @@ -478,20 +430,11 @@ const stream = chat({ }) ``` -Two axes are open where the AI SDK's is narrower: - -- **Any agent, not a fixed list.** Grok Build, Claude Code, Codex, and OpenCode ship as first-party harness packages, and `acpCompatible` (from `@tanstack/ai-acp`) turns *any* [Agent Client Protocol](https://agentclientprotocol.com) agent — `pi`, `gemini --acp`, and [dozens of others](https://agentclientprotocol.com/get-started/agents) — into a harness by describing how to launch it. Adding an agent doesn't require a dedicated adapter to exist. -- **Any sandbox, not one cloud.** The same run executes on `localProcessSandbox` (host dev loop), `dockerSandbox` (real container isolation), Daytona, Vercel Sandbox, Sprites, or Cloudflare — swap the provider without touching the harness or workspace. Providers declare their `capabilities()` (`fs`, `exec`, `ports`, `snapshots`, `fork`, `durableFilesystem`, …) so code degrades gracefully across them. - -Vercel AI SDK 7 added a `HarnessAgent` API for the same idea — running a coding-agent harness in a sandbox and returning AI SDK-compatible `generate()` / `stream()` results. It's marked experimental, ships harnesses for Claude Code, Codex, Pi, OpenCode, and Deep Agents, and the documented path runs them in Vercel Sandbox. There's no generic ACP-compatible escape hatch (each supported harness is its own dedicated package), and sandbox support centers on Vercel's own microVM rather than a provider-swappable contract. +**Axes:** any ACP agent via `acpCompatible`; providers (local, Docker, Daytona, Vercel, Sprites, Cloudflare) with `capabilities()`. Vercel `HarnessAgent` (experimental) is narrower (fixed harness list; Vercel Sandbox–centered). -### Media Generation +### Media generation -TanStack AI provides stable, dedicated APIs for every media generation activity - image, video, speech, transcription, and summarization. Each is a separate, tree-shakeable function with its own adapter per provider. - -Vercel AI SDK has added several of these capabilities. As of v6, `generateImage()` is stable; video generation is still experimental (`experimental_generateVideo()`); and `generateSpeech()` / `transcribe()` are exported without the `experimental_` prefix but are still documented as experimental features. TanStack AI's media APIs are stable across the board and go further in several areas: - -**Image generation** - `generateImage()` with per-model type safety. TypeScript knows that `gpt-image-2` and `dall-e-3` expose different size constraints. Five providers ship adapters: OpenAI (GPT Image, DALL-E), Gemini (Imagen), Grok, OpenRouter, and fal.ai (600+ community models including Flux, SDXL, and more). +Stable, tree-shakeable APIs: image, video (async job + poll), speech (6 formats), transcription, `generateAudio()`, `summarize()`, realtime voice. Vercel: image stable; video/speech/transcribe experimental or no realtime/summarize/music equivalents. ```ts import { generateImage } from '@tanstack/ai' @@ -505,8 +448,6 @@ const result = await generateImage({ }) ``` -**Video generation** - `generateVideo()` handles the full async job lifecycle automatically. Video generation APIs are inherently asynchronous - you submit a job, poll for status, and eventually get a result. TanStack AI manages this entire lifecycle with configurable polling intervals and timeouts, streaming status updates back to the client. - ```ts import { generateVideo } from '@tanstack/ai' import { openaiVideo } from '@tanstack/ai-openai' @@ -525,10 +466,6 @@ for await (const chunk of stream) { } ``` -Vercel AI SDK's `experimental_generateVideo()` returns the video directly without exposing the job lifecycle or streaming status updates. - -**Text-to-speech** - `generateSpeech()` supports 6 audio output formats (mp3, opus, aac, flac, wav, pcm), speed control (0.25x to 4x), and five providers: OpenAI (11 voices), Gemini (30+ voices with language hints), Grok, ElevenLabs, and fal.ai. - ```ts import { generateSpeech } from '@tanstack/ai' import { openaiSpeech } from '@tanstack/ai-openai' @@ -542,8 +479,6 @@ const result = await generateSpeech({ }) ``` -**Transcription** - `generateTranscription()` supports common output formats (json, text, srt, verbose_json, vtt), word-level timestamps with confidence scores, and four providers (OpenAI, Grok, ElevenLabs, fal.ai), with speaker diarization via OpenAI's `gpt-4o-transcribe-diarize` model. - ```ts import { generateTranscription } from '@tanstack/ai' import { openaiTranscription } from '@tanstack/ai-openai' @@ -558,57 +493,43 @@ const result = await generateTranscription({ // result.words → [{ word: 'Hello', start: 0.0, end: 0.42 }, ...] ``` -**Audio & music generation** - `generateAudio()` generates music and sound effects across Gemini (Lyria), ElevenLabs (music + sound effects), and fal.ai. Vercel AI SDK has no equivalent. - -**Summarization** - `summarize()` is a dedicated activity with style control (`bullet-points`, `paragraph`, `concise`), focus topics, and streaming support. Vercel AI SDK has no equivalent - summarization requires calling `generateText()` with a prompt. - -**Realtime voice** - `realtimeToken()` enables bidirectional audio streaming with Voice Activity Detection modes (server, semantic, manual), tool calling during voice sessions, and simultaneous audio + text output. Three providers ship realtime adapters: OpenAI (Realtime API), Grok, and ElevenLabs. Vercel AI SDK has no realtime/bidirectional voice primitive - its audio support is batch-only (`generateSpeech` and `transcribe`). - -All media activities follow the same adapter pattern as chat - tree-shakeable imports, per-model type safety, and streaming support. If your app only uses chat, none of this media code enters your bundle. +### Native AG-UI -### Native AG-UI Protocol +Wire format is AG-UI (`@ag-ui/core`) end to end. Vercel uses proprietary UI Message Stream; AG-UI needs `@ag-ui/vercel-ai-sdk`. -The events TanStack AI streams between server and client are [AG-UI](https://docs.ag-ui.com/) events (`RUN_STARTED`, `TEXT_MESSAGE_*`, `TOOL_CALL_*`, `RUN_FINISHED`), imported directly from `@ag-ui/core` - not a bespoke format with an AG-UI export bolted on. Anything that speaks AG-UI can sit on either side of the wire: AG-UI-compliant agent frameworks behind a TanStack AI frontend, or a TanStack AI client in front of an agent server written in another language entirely. +### Hooks for every activity -Vercel AI SDK streams its own proprietary UI Message Stream protocol. AG-UI interop requires an external translation layer (`@ag-ui/vercel-ai-sdk`, built and maintained by the AG-UI project), and native support remains an open feature request on the AI SDK repo. +`useGeneration`, `useGenerateImage`, `useGenerateAudio`, `useGenerateSpeech`, `useTranscription`, `useSummarize`, `useGenerateVideo`, `useRealtimeChat` — same connection/devtools story as `useChat` across frameworks. Vercel UI: `useChat` / `useCompletion` / `useObject`; media is server-side only. -### Hooks for Every Activity +### Multi-turn structured output -Chat isn't the only activity with a hook. Every activity ships one - `useGeneration` (streaming structured output), `useGenerateImage`, `useGenerateAudio`, `useGenerateSpeech`, `useTranscription`, `useSummarize`, `useGenerateVideo`, and `useRealtimeChat` - with the same connection-adapter wiring and devtools integration as `useChat`, across React, Solid, Vue, Svelte, and Preact. +`outputSchema` on `useChat` → typed `StructuredOutputPart` per turn (partial → final) in history. Vercel structured APIs are per-call results without a structured message part in the union. -Vercel AI SDK's UI layer has three hooks: `useChat`, `useCompletion`, and `useObject`. Its media functions (`generateImage()`, `experimental_generateVideo()`, speech, transcription) are server-side only - surfacing them in a UI means hand-rolling your own routes and client state. +### Debug logging -### Multi-Turn Structured Output +`debug: true` with per-category toggles + pluggable logger. Vercel: warnings + experimental telemetry / dev inspector. -Structured output in TanStack AI is part of the conversation, not a separate call. Pass `outputSchema` to `useChat` and every assistant turn carries its own typed `StructuredOutputPart` - streamed as a `partial`, validated as a `final`, preserved in message history, with the schema generic threading all the way down to `messages[i].parts[j].data`. +### Community adapters -Vercel AI SDK's structured output (`generateObject` / `streamObject` / `Output`) is per-call: the typed object lives on the call result, the message-part union has no structured-output type, and combining `useChat` with typed structured output means manually parsing model text into custom data parts. +Open adapter spec; community adapters (e.g. Decart, Cencori, Cloudflare, Soniox, Mynth). Guide: [build your own](../community-adapters/guide). -### Debug Logging +## Where Vercel AI SDK excels -Set `debug: true` on any activity and the pipeline prints itself: raw provider chunks, post-middleware output, middleware hook inputs and outputs, tool execution, agent-loop iterations, config transforms, and request metadata - each category individually toggleable, with a pluggable `logger` for structured output. Vercel AI SDK's built-in logging covers provider warnings; richer observability goes through experimental telemetry hooks or the dev-only DevTools recorder rather than a debug log you can flip on anywhere. +**Provider packages.** ~38 first-party typed packages. TanStack reaches large model counts via OpenRouter + `openaiCompatible`, not the same package count. -### Community Adapter Ecosystem +**Angular.** Official Vercel Angular integration; TanStack has React/Solid/Svelte/Vue/Preact (no Angular). Solid flips the other way: TanStack official; AI SDK Solid is community/older major. -TanStack AI publishes an open adapter specification. The community has already built adapters for Decart, Cencori, Cloudflare, Soniox, and Mynth - with a [guide for building your own](../community-adapters/guide). The adapter interface is simple enough that adding a new provider is a focused, self-contained task. +**Agent class.** v6 `ToolLoopAgent` packages model/tools/instructions/loop. TanStack composes per call. -## Where Vercel AI SDK Excels +**AI Gateway.** Centralized failover/caching/keys on Vercel. TanStack recommends OpenRouter for multi-model routing without a platform layer. -**Provider breadth.** Vercel AI SDK ships ~38 first-party, individually typed provider packages, plus a large community list. If you want a specific provider as a dedicated, maintained package without writing an adapter, their coverage is broader today. Raw model *count* is not the differentiator, though - TanStack AI's OpenRouter adapter reaches OpenRouter's full catalog (several hundred models), and the `openaiCompatible` adapter connects to any OpenAI-compatible endpoint. +**RSC.** `@ai-sdk/rsc` (experimental; Vercel recommends AI SDK UI for production). -**Angular support.** Vercel AI SDK has an official Angular integration. TanStack AI supports React, Solid, Svelte, Vue, and Preact, but not Angular. (Solid now cuts the other way: AI SDK's Solid package is community-maintained and pinned to an older SDK major, while TanStack AI ships an official, current Solid integration.) +## Side-by-side snippets -**Agent abstraction.** Vercel AI SDK v6 ships a dedicated `Agent` abstraction (the `ToolLoopAgent` class) that packages a model, tools, instructions, and loop settings into a reusable object with `.generate()` and `.stream()` methods, plus `InferAgentUIMessage` for end-to-end type safety. TanStack AI composes these pieces per call rather than offering a single agent class. +### Tool definition -**AI Gateway.** Vercel's optional AI Gateway adds centralized provider management - failover routing, caching, and a single key across providers - integrated with the Vercel platform (and used by default when no provider is configured). TanStack AI ships no gateway of its own; for the same centralized routing across a large model catalog, it recommends its first-class OpenRouter adapter, with no platform association attached. - -**React Server Components.** Vercel AI SDK has an RSC integration via `@ai-sdk/rsc` (`AIState`, `StreamableValue`, `streamUI`). Note that Vercel documents this as experimental and recommends AI SDK UI for production - so it's an option for Next.js RSC apps rather than the primary path. - -## Side-by-Side: Key Differences - -### Tool Definition - -**TanStack AI** - Isomorphic definitions with separate runtime implementations: +**TanStack** ```ts import { toolDefinition } from '@tanstack/ai' @@ -635,7 +556,7 @@ const getWeatherClient = getWeather.client(async ({ city }) => { }) ``` -**Vercel AI SDK** - Tool objects via the `tool()` helper: +**Vercel** ```ts import { generateText, tool } from 'ai' @@ -659,11 +580,9 @@ const result = await generateText({ }) ``` -The TanStack approach separates the tool contract from its implementation, making tools reusable across server and client contexts. - -### Agent Loop Control +### Agent loop -**TanStack AI** - Composable strategies: +**TanStack** ```ts import { chat, combineStrategies, maxIterations, untilFinishReason } from '@tanstack/ai' @@ -685,7 +604,7 @@ const stream = chat({ }) ``` -**Vercel AI SDK** - `stopWhen` conditions (v5+): +**Vercel** ```ts import { generateText, stepCountIs } from 'ai' @@ -700,52 +619,37 @@ const result = await generateText({ }) ``` -Both let you compose multiple stopping conditions - `stopWhen` accepts an array of conditions including custom functions, and v6 adds a reusable `Agent` class. The remaining nuance is ergonomic: TanStack AI's strategies are plain `(state) => boolean` predicates combined with `combineStrategies`, so token budgets and custom business logic are first-class without reaching for a built-in condition. - -### Tree-Shaking - -**TanStack AI** - Separate adapters per activity: +### Tree-shaking ```ts -// Only bundles chat + OpenAI text adapter +// TanStack — only chat + OpenAI text import { chat } from '@tanstack/ai' import { openaiText } from '@tanstack/ai-openai' ``` -**Vercel AI SDK** - Single provider import: - ```ts -// Provider package includes all model types +// Vercel — provider package includes model surface import { openai } from '@ai-sdk/openai' ``` -In TanStack AI, each activity (chat, image, speech, video, transcription, summarization) is a separate adapter function. You import `openaiText` for chat and `openaiImage` for image generation - they're independent modules. Vercel AI SDK's provider packages are more monolithic. - -## When to Choose TanStack AI +## When to choose TanStack AI -- **Bundle size matters** - Tree-shakeable adapters per activity mean smaller bundles -- **AG-UI native** - The wire protocol is AG-UI end to end; interoperate with the agent-UI ecosystem and non-TypeScript agent servers without a translation layer -- **Solid, Preact, or React Native** - One headless core covers React, Solid, Vue, Svelte, Preact, and React Native (via XHR adapters), all officially maintained -- **Hooks beyond chat** - `useGeneration`, `useGenerateImage`, `useSummarize`, and the rest of the generation hook family across every supported framework -- **Isomorphic tools** - Define a tool once and derive `.server()` / `.client()` implementations from one contract -- **App-level middleware** - Lifecycle hooks for chunks, tool calls, usage, and errors - not just model wrapping -- **Realtime voice** - Bidirectional audio across OpenAI, Grok, and ElevenLabs -- **No vendor association** - Pure library with no platform layer -- **Per-model type safety** - TypeScript narrows options per model, not per provider -- **Code execution** - Built-in sandboxed execution environments -- **Coding agent sandboxes** - Run Claude Code, Codex, Grok Build, OpenCode, or any ACP agent in a swappable sandbox (local, Docker, Daytona, Vercel, Sprites, Cloudflare), streamed through `chat()` -- **Flexible transport** - SSE, HTTP streams, XHR, RPC, direct iterables, or custom adapters -- **MCP, two ways** - A standalone host-side client (`@tanstack/ai-mcp`) with pools, codegen, and managed `chat()` lifecycle, plus a provider-routed `mcpTool()` +- Bundle size / tree-shakeable activity adapters +- Native AG-UI wire protocol +- Solid, Preact, or React Native (XHR adapters) with official support +- Hooks beyond chat (image, summarize, video, realtime, …) +- Isomorphic tools + app-level middleware +- Realtime voice, code/coding-agent sandboxes, flexible transport, dual MCP paths -## When to Choose Vercel AI SDK +## When to choose Vercel AI SDK -- **Need a first-party package for a specific provider** - ~38 dedicated, individually typed provider packages today (TanStack reaches comparable model breadth via OpenRouter + `openaiCompatible`) -- **Angular support** - Official Angular integration -- **Agent abstraction** - A reusable `Agent` (`ToolLoopAgent`) class with end-to-end UI message types -- **Vercel platform** - AI Gateway, observability, and deployment optimization -- **React Server Components** - RSC primitives via `@ai-sdk/rsc` (experimental; AI SDK UI is the recommended production path) +- First-party package for a specific niche provider (~38 packages) +- Official Angular +- Reusable `ToolLoopAgent` + UI message types +- Vercel AI Gateway / platform observability +- RSC primitives via `@ai-sdk/rsc` (experimental) -## Getting Started +## Start with TanStack ```bash npm install @tanstack/ai @tanstack/ai-openai @@ -753,4 +657,5 @@ npm install @tanstack/ai @tanstack/ai-openai pnpm add @tanstack/ai @tanstack/ai-openai ``` -See the [Quick Start Guide](../getting-started/quick-start) to build your first chat application, or explore the [full documentation](../getting-started/overview). +- [Quick Start](../getting-started/quick-start) +- [Overview](../getting-started/overview) diff --git a/docs/config.json b/docs/config.json index 67c19a8db..9dc815a40 100644 --- a/docs/config.json +++ b/docs/config.json @@ -13,49 +13,55 @@ "label": "Overview", "to": "getting-started/overview", "addedAt": "2026-04-15", - "updatedAt": "2026-07-31" + "updatedAt": "2026-08-06" }, { "label": "Quick Start: React", "to": "getting-started/quick-start", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "Quick Start: React Native", "to": "getting-started/quick-start-react-native", - "addedAt": "2026-05-28" + "addedAt": "2026-05-28", + "updatedAt": "2026-08-06" }, { "label": "Devtools", "to": "getting-started/devtools", "addedAt": "2026-04-15", - "updatedAt": "2026-07-22" + "updatedAt": "2026-08-06" }, { "label": "Quick Start: Vue", "to": "getting-started/quick-start-vue", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "Quick Start: Svelte", "to": "getting-started/quick-start-svelte", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "Quick Start: Angular", "to": "getting-started/quick-start-angular", - "addedAt": "2026-06-15" + "addedAt": "2026-06-15", + "updatedAt": "2026-08-06" }, { "label": "Quick Start: Server Only", "to": "getting-started/quick-start-server", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "Agent Skills (TanStack Intent)", "to": "getting-started/agent-skills", "addedAt": "2026-04-17", - "updatedAt": "2026-07-26" + "updatedAt": "2026-08-06" } ] }, @@ -66,7 +72,7 @@ "label": "TanStack AI vs Vercel AI SDK", "to": "comparison/vercel-ai-sdk", "addedAt": "2026-04-15", - "updatedAt": "2026-07-01" + "updatedAt": "2026-08-06" } ] }, @@ -77,47 +83,49 @@ "label": "Tools", "to": "tools/tools", "addedAt": "2026-04-15", - "updatedAt": "2026-07-08" + "updatedAt": "2026-08-06" }, { "label": "Provider Tools", "to": "tools/provider-tools", "addedAt": "2026-04-21", - "updatedAt": "2026-07-02" + "updatedAt": "2026-08-06" }, { "label": "Provider Skills", "to": "tools/provider-skills", - "addedAt": "2026-06-04" + "addedAt": "2026-06-04", + "updatedAt": "2026-08-06" }, { "label": "Tool Architecture", "to": "tools/tool-architecture", "addedAt": "2026-04-15", - "updatedAt": "2026-07-08" + "updatedAt": "2026-08-06" }, { "label": "Server Tools", "to": "tools/server-tools", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "Client Tools", "to": "tools/client-tools", "addedAt": "2026-04-15", - "updatedAt": "2026-07-21" + "updatedAt": "2026-08-06" }, { "label": "Tool Approval Flow", "to": "tools/tool-approval", "addedAt": "2026-04-15", - "updatedAt": "2026-07-24" + "updatedAt": "2026-08-06" }, { "label": "Lazy Tool Discovery", "to": "tools/lazy-tool-discovery", "addedAt": "2026-04-15", - "updatedAt": "2026-06-08" + "updatedAt": "2026-08-06" } ] }, @@ -127,28 +135,32 @@ { "label": "MCP Server Tools", "to": "tools/mcp", - "addedAt": "2026-06-05" + "addedAt": "2026-06-05", + "updatedAt": "2026-08-06" }, { "label": "Managed MCP with chat()", "to": "tools/mcp-managed", - "addedAt": "2026-06-05" + "addedAt": "2026-06-05", + "updatedAt": "2026-08-06" }, { "label": "Manual MCP", "to": "tools/mcp-manual", - "addedAt": "2026-06-05" + "addedAt": "2026-06-05", + "updatedAt": "2026-08-06" }, { "label": "MCP Type Generation", "to": "tools/mcp-codegen", - "addedAt": "2026-06-05" + "addedAt": "2026-06-05", + "updatedAt": "2026-08-06" }, { "label": "MCP Apps", "to": "mcp/apps", "addedAt": "2026-06-24", - "updatedAt": "2026-07-03" + "updatedAt": "2026-08-06" } ] }, @@ -159,24 +171,25 @@ "label": "Agentic Cycle", "to": "chat/agentic-cycle", "addedAt": "2026-04-15", - "updatedAt": "2026-07-21" + "updatedAt": "2026-08-06" }, { "label": "Streaming", "to": "chat/streaming", "addedAt": "2026-04-15", - "updatedAt": "2026-07-30" + "updatedAt": "2026-08-06" }, { "label": "Connection Adapters", "to": "chat/connection-adapters", "addedAt": "2026-04-15", - "updatedAt": "2026-07-30" + "updatedAt": "2026-08-06" }, { "label": "Thinking & Reasoning", "to": "chat/thinking-content", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" } ] }, @@ -187,27 +200,32 @@ { "label": "Overview", "to": "interrupts/overview", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Tool Approval", "to": "interrupts/tool-approval", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Multiple Interrupts", "to": "interrupts/multiple", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Generic Interrupts", "to": "interrupts/generic", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Migration", "to": "interrupts/migration", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" } ] }, @@ -218,17 +236,20 @@ { "label": "Overview", "to": "resumable-streams/overview", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Advanced", "to": "resumable-streams/advanced", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Custom Durability Adapter", "to": "resumable-streams/custom-adapter", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" } ] }, @@ -239,37 +260,44 @@ { "label": "Overview", "to": "persistence/overview", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Chat Persistence", "to": "persistence/chat-persistence", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Client Persistence", "to": "persistence/client-persistence", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Generation Persistence", "to": "persistence/generation-persistence", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Controls", "to": "persistence/controls", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Build Your Own Adapter", "to": "persistence/build-your-own-adapter", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Migrations", "to": "persistence/migrations", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" } ] }, @@ -282,37 +310,44 @@ { "label": "Id Map", "to": "persistence/id-map", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Keep Generated Files", "to": "persistence/keep-generated-files", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Build a Chat Adapter", "to": "persistence/build-your-own-chat-adapter", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Build a Generation Adapter", "to": "persistence/build-your-own-generation-adapter", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Build a Sandbox Adapter", "to": "persistence/build-a-sandbox-adapter", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Store Reference", "to": "persistence/store-reference", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "How Persistence Works", "to": "persistence/internals", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" } ] }, @@ -322,7 +357,8 @@ { "label": "Custom Events Reference", "to": "protocol/custom-events", - "addedAt": "2026-07-03" + "addedAt": "2026-07-03", + "updatedAt": "2026-08-06" } ] }, @@ -334,28 +370,31 @@ "label": "Overview", "to": "structured-outputs/overview", "addedAt": "2026-05-19", - "updatedAt": "2026-06-10" + "updatedAt": "2026-08-06" }, { "label": "One-Shot Extraction", "to": "structured-outputs/one-shot", - "addedAt": "2026-05-19" + "addedAt": "2026-05-19", + "updatedAt": "2026-08-06" }, { "label": "Streaming UIs", "to": "structured-outputs/streaming", - "addedAt": "2026-05-19" + "addedAt": "2026-05-19", + "updatedAt": "2026-08-06" }, { "label": "Multi-Turn Chat", "to": "structured-outputs/multi-turn", - "addedAt": "2026-05-19" + "addedAt": "2026-05-19", + "updatedAt": "2026-08-06" }, { "label": "With Tools", "to": "structured-outputs/with-tools", "addedAt": "2026-05-19", - "updatedAt": "2026-07-08" + "updatedAt": "2026-08-06" } ] }, @@ -365,28 +404,32 @@ { "label": "Code Mode", "to": "code-mode/code-mode", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "Showing Code Mode in the UI", "to": "code-mode/client-integration", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "Code Mode with Skills", "to": "code-mode/code-mode-with-skills", "addedAt": "2026-04-15", - "updatedAt": "2026-06-10" + "updatedAt": "2026-08-06" }, { "label": "Code Mode Isolate Drivers", "to": "code-mode/code-mode-isolates", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "Lazy Tools", "to": "code-mode/lazy-tools", - "addedAt": "2026-06-08" + "addedAt": "2026-06-08", + "updatedAt": "2026-08-06" } ] }, @@ -397,55 +440,55 @@ "label": "Generations", "to": "media/generations", "addedAt": "2026-04-15", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" }, { "label": "Realtime Voice Chat", "to": "media/realtime-chat", "addedAt": "2026-04-15", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" }, { "label": "Text-to-Speech", "to": "media/text-to-speech", "addedAt": "2026-04-15", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" }, { "label": "Transcription", "to": "media/transcription", "addedAt": "2026-04-15", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" }, { "label": "Audio Recording", "to": "media/audio-recording", "addedAt": "2026-06-23", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" }, { "label": "Audio Generation", "to": "media/audio-generation", "addedAt": "2026-04-23", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" }, { "label": "Image Generation", "to": "media/image-generation", "addedAt": "2026-04-15", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" }, { "label": "Video Generation", "to": "media/video-generation", "addedAt": "2026-04-15", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" }, { "label": "Generation Hooks", "to": "media/generation-hooks", "addedAt": "2026-04-15", - "updatedAt": "2026-07-30" + "updatedAt": "2026-08-06" } ] }, @@ -456,24 +499,25 @@ "label": "Middleware", "to": "advanced/middleware", "addedAt": "2026-04-15", - "updatedAt": "2026-07-21" + "updatedAt": "2026-08-06" }, { "label": "Built-in Middleware", "to": "advanced/built-in-middleware", "addedAt": "2026-06-03", - "updatedAt": "2026-07-21" + "updatedAt": "2026-08-06" }, { "label": "Locks", "to": "advanced/locks", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "OpenTelemetry", "to": "advanced/otel", "addedAt": "2026-05-08", - "updatedAt": "2026-07-31" + "updatedAt": "2026-08-06" } ] }, @@ -485,65 +529,67 @@ "label": "Overview", "to": "sandbox/overview", "addedAt": "2026-06-16", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" }, { "label": "Quick Start", "to": "sandbox/quick-start", "addedAt": "2026-06-29", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" }, { "label": "Providers", "to": "sandbox/providers", "addedAt": "2026-06-29", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" }, { "label": "Harnesses", "to": "sandbox/harnesses", "addedAt": "2026-06-30", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" }, { "label": "Workspace", "to": "sandbox/workspace", "addedAt": "2026-06-29", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" }, { "label": "Tools", "to": "sandbox/tools", "addedAt": "2026-06-29", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" }, { "label": "Policy", "to": "sandbox/policy", "addedAt": "2026-06-29", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" }, { "label": "Lifecycle & Snapshots", "to": "sandbox/lifecycle", "addedAt": "2026-06-29", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" }, { "label": "Instance Durability", "to": "sandbox/durability", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Durable Runs", "to": "sandbox/durable-runs", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Events", "to": "sandbox/events", "addedAt": "2026-06-29", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" } ] }, @@ -556,35 +602,38 @@ { "label": "The Run Journal", "to": "sandbox/journal", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Takeover & Detached Runs", "to": "sandbox/takeover", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Reaping & Retention", "to": "sandbox/reaping", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Provisioning", "to": "sandbox/provisioning", "addedAt": "2026-06-29", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" }, { "label": "Observability", "to": "sandbox/observability", "addedAt": "2026-06-29", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" }, { "label": "Cloudflare (Edge)", "to": "sandbox/cloudflare", "addedAt": "2026-06-29", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" } ] }, @@ -595,27 +644,32 @@ { "label": "Overview", "to": "memory/overview", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Quickstart", "to": "memory/quickstart", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Adapters", "to": "memory/adapters", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Custom Adapter", "to": "memory/custom-adapter", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" }, { "label": "Operating", "to": "memory/operating", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" } ] }, @@ -626,48 +680,55 @@ "label": "Runtime Context", "to": "advanced/runtime-context", "addedAt": "2026-06-01", - "updatedAt": "2026-07-08" + "updatedAt": "2026-08-06" }, { "label": "Debug Logging", "to": "advanced/debug-logging", - "addedAt": "2026-04-22" + "addedAt": "2026-04-22", + "updatedAt": "2026-08-06" }, { "label": "Multimodal Content", "to": "advanced/multimodal-content", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "Per-Model Type Safety", "to": "advanced/per-model-type-safety", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "Runtime Adapter Switching", "to": "advanced/runtime-adapter-switching", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "Tree-Shaking", "to": "advanced/tree-shaking", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "Extend Adapter", "to": "advanced/extend-adapter", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "Typed Pre-Configured Options", "to": "advanced/typed-options", - "addedAt": "2026-05-25" + "addedAt": "2026-05-25", + "updatedAt": "2026-08-06" }, { "label": "Approval Flow Processing", "to": "architecture/approval-flow-processing", "addedAt": "2026-04-15", - "updatedAt": "2026-07-21" + "updatedAt": "2026-08-06" } ] }, @@ -677,24 +738,26 @@ { "label": "Migration Guide", "to": "migration/migration", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "From Vercel AI SDK", "to": "migration/migration-from-vercel-ai", "addedAt": "2026-04-20", - "updatedAt": "2026-07-08" + "updatedAt": "2026-08-06" }, { "label": "AG-UI Client Compliance", "to": "migration/ag-ui-compliance", "addedAt": "2026-05-16", - "updatedAt": "2026-07-31" + "updatedAt": "2026-08-06" }, { "label": "Sampling → modelOptions", "to": "migration/sampling-options-to-model-options", - "addedAt": "2026-06-03" + "addedAt": "2026-06-03", + "updatedAt": "2026-08-06" } ] }, @@ -705,49 +768,49 @@ "label": "@tanstack/ai", "to": "api/ai", "addedAt": "2026-04-15", - "updatedAt": "2026-07-21" + "updatedAt": "2026-08-06" }, { "label": "@tanstack/ai-client", "to": "api/ai-client", "addedAt": "2026-04-15", - "updatedAt": "2026-08-04" + "updatedAt": "2026-08-06" }, { "label": "@tanstack/ai-react", "to": "api/ai-react", "addedAt": "2026-04-15", - "updatedAt": "2026-07-08" + "updatedAt": "2026-08-06" }, { "label": "@tanstack/ai-solid", "to": "api/ai-solid", "addedAt": "2026-04-15", - "updatedAt": "2026-07-08" + "updatedAt": "2026-08-06" }, { "label": "@tanstack/ai-preact", "to": "api/ai-preact", "addedAt": "2026-04-15", - "updatedAt": "2026-07-08" + "updatedAt": "2026-08-06" }, { "label": "@tanstack/ai-vue", "to": "api/ai-vue", "addedAt": "2026-04-15", - "updatedAt": "2026-07-30" + "updatedAt": "2026-08-06" }, { "label": "@tanstack/ai-svelte", "to": "api/ai-svelte", "addedAt": "2026-04-15", - "updatedAt": "2026-07-30" + "updatedAt": "2026-08-06" }, { "label": "@tanstack/ai-angular", "to": "api/ai-angular", "addedAt": "2026-06-15", - "updatedAt": "2026-07-30" + "updatedAt": "2026-08-06" } ] }, @@ -758,99 +821,108 @@ "label": "OpenAI", "to": "adapters/openai", "addedAt": "2026-04-15", - "updatedAt": "2026-07-03" + "updatedAt": "2026-08-06" }, { "label": "Anthropic", "to": "adapters/anthropic", "addedAt": "2026-04-15", - "updatedAt": "2026-07-04" + "updatedAt": "2026-08-06" }, { "label": "Google Gemini", "to": "adapters/gemini", "addedAt": "2026-04-15", - "updatedAt": "2026-07-01" + "updatedAt": "2026-08-06" }, { "label": "Ollama", "to": "adapters/ollama", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "Grok (xAI)", "to": "adapters/grok", "addedAt": "2026-04-15", - "updatedAt": "2026-06-24" + "updatedAt": "2026-08-06" }, { "label": "Groq", "to": "adapters/groq", "addedAt": "2026-04-15", - "updatedAt": "2026-07-01" + "updatedAt": "2026-08-06" }, { "label": "Mistral", - "to": "adapters/mistral" + "to": "adapters/mistral", + "updatedAt": "2026-08-06" }, { "label": "ElevenLabs", "to": "adapters/elevenlabs", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "fal.ai", "to": "adapters/fal", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "OpenRouter Adapter", "to": "adapters/openrouter", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "OpenAI-Compatible", "to": "adapters/openai-compatible", "addedAt": "2026-06-01", - "updatedAt": "2026-07-20" + "updatedAt": "2026-08-06" }, { "label": "Claude Code", "to": "adapters/claude-code", "addedAt": "2026-06-12", - "updatedAt": "2026-06-30" + "updatedAt": "2026-08-06" }, { "label": "Codex", "to": "adapters/codex", "addedAt": "2026-06-12", - "updatedAt": "2026-06-30" + "updatedAt": "2026-08-06" }, { "label": "OpenCode", "to": "adapters/opencode", "addedAt": "2026-06-12", - "updatedAt": "2026-06-30" + "updatedAt": "2026-08-06" }, { "label": "Grok Build", "to": "adapters/grok-build", - "addedAt": "2026-06-29" + "addedAt": "2026-06-29", + "updatedAt": "2026-08-06" }, { "label": "ACP-Compatible", "to": "adapters/acp-compatible", - "addedAt": "2026-06-30" + "addedAt": "2026-06-30", + "updatedAt": "2026-08-06" }, { "label": "Amazon Bedrock", "to": "adapters/bedrock", - "addedAt": "2026-06-25" + "addedAt": "2026-06-25", + "updatedAt": "2026-08-06" }, { "label": "BytePlus", "to": "adapters/byteplus", - "addedAt": "2026-08-04" + "addedAt": "2026-08-04", + "updatedAt": "2026-08-06" } ] }, @@ -861,32 +933,38 @@ { "label": "Community Adapters Guide", "to": "community-adapters/guide", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "Decart", "to": "community-adapters/decart", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "Cencori", "to": "community-adapters/cencori", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "Cloudflare", "to": "community-adapters/cloudflare", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "Soniox", "to": "community-adapters/soniox", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" }, { "label": "Mynth", "to": "community-adapters/mynth", - "addedAt": "2026-04-15" + "addedAt": "2026-04-15", + "updatedAt": "2026-08-06" } ] }, diff --git a/docs/getting-started/agent-skills.md b/docs/getting-started/agent-skills.md index da8c55eeb..41008eb88 100644 --- a/docs/getting-started/agent-skills.md +++ b/docs/getting-started/agent-skills.md @@ -2,7 +2,7 @@ title: Agent Skills (TanStack Intent) id: agent-skills order: 6 -description: "Use TanStack Intent to wire TanStack AI's bundled Agent Skills into Claude Code, Cursor, GitHub Copilot, and other AI coding assistants." +description: "Wire TanStack AI Agent Skills into Claude Code, Cursor, Copilot, and other coding agents via Intent." keywords: - tanstack ai - tanstack intent @@ -14,60 +14,45 @@ keywords: - SKILL.md - AGENTS.md --- -> **Looking for runtime skills inside Code Mode?** Those are a different feature — see [Code Mode with Skills](../code-mode/code-mode-with-skills). This page is about _agent-authoring_ skills: markdown files that teach your coding assistant how TanStack AI works. -## Step 1: Install TanStack AI -If you haven't already, install `@tanstack/ai` plus any adapter packages you need. See the [Quick Start](./quick-start) for a full walkthrough. +If you need your coding agent to use TanStack AI correctly → install packages, run `npx @tanstack/intent@latest install`, confirm the `intent-skills` block. + +Runtime Code Mode skills are different → [Code Mode with Skills](../code-mode/code-mode-with-skills). + +## 1. Install TanStack AI ```bash pnpm add @tanstack/ai ``` -## Step 2: Run `intent install` +Full walkthrough: [Quick Start](./quick-start). + +## 2. Install Intent mappings -From the root of your project, run: +From the project root: ```bash npx @tanstack/intent@latest install ``` +## What you installed -## What are Agent Skills? - -Agent Skills are markdown documents (`SKILL.md`) that ship inside npm packages and tell AI coding agents how to use a library correctly — which functions to use, which patterns to avoid, and when to reach for which module. The format is an open standard supported by Claude Code, Cursor, GitHub Copilot, Codex, and others. - -TanStack AI publishes skills inside its packages so the guidance travels with `npm update` instead of being pinned in a model's training data or copy-pasted into `CLAUDE.md` manually. - -## Skills Shipped by TanStack AI - -| Package | Skill | What it teaches | -|---------|-------|-----------------| -| `@tanstack/ai` | `ai-core` | Chat experience, browser persistence on `useChat`, tool calling, adapters, middleware, locks, structured outputs, media generation, AG-UI protocol, custom backends | -| `@tanstack/ai-persistence` | `ai-persistence` | Server chat state (`withPersistence`), the store contracts, and per-stack recipes that write a `chat-persistence.ts` into your app against your existing Drizzle, Prisma, or Cloudflare D1 setup | -| `@tanstack/ai-memory` | `tanstack-ai-memory` | `memoryMiddleware`, the recall/save adapter contract, and the in-memory / Redis / Hindsight / Mem0 / Honcho adapters | -| `@tanstack/ai-mcp` | `ai-mcp` | Connecting to MCP servers, running their tools inside `chat()`, resources, prompts, and the type-generating CLI | -| `@tanstack/ai-sandbox` | `ai-sandbox` | Running harness adapters inside isolated sandboxes with `defineSandbox` / `withSandbox` | -| `@tanstack/ai-code-mode` | `ai-code-mode` | Setting up Code Mode with a sandbox driver and registering server tools | - -Skills route to each other: `ai-core` points at the companion packages' -skills, and `ai-persistence` is an entry point that routes to its own -sub-skills (`server`, `stores`, and the -`build-{drizzle,prisma,cloudflare,custom}-adapter` recipes) under -`skills/ai-persistence/`, same nesting style as `ai-core`. Multi-instance locks -ship with the code they teach, so `ai-core/locks` lives in `@tanstack/ai` -alongside `withLocks`. - -Each skill ships with the code it teaches. Browser persistence lives in the -framework packages, so `ai-core/client-persistence` is in `@tanstack/ai` rather -than in `@tanstack/ai-persistence` — an app that persists only in the browser -never installs the server package. +Agent Skills are `SKILL.md` files in npm packages that tell coding agents which APIs and patterns to use. Supported by Claude Code, Cursor, Copilot, Codex, and others. They update with `npm update` instead of stale training data or hand-pasted docs. -Each skill lives under `node_modules//skills//SKILL.md` once the package is installed. +| Package | Skill | Teaches | +|---------|-------|---------| +| `@tanstack/ai` | `ai-core` | Chat, tools, adapters, middleware, locks, structured output, media, AG-UI | +| `@tanstack/ai-persistence` | `ai-persistence` | Server chat state, store contracts, Drizzle/Prisma/D1 recipes | +| `@tanstack/ai-memory` | `tanstack-ai-memory` | `memoryMiddleware`, adapters (in-memory, Redis, …) | +| `@tanstack/ai-mcp` | `ai-mcp` | MCP servers, tools in `chat()`, CLI types | +| `@tanstack/ai-sandbox` | `ai-sandbox` | `defineSandbox` / `withSandbox` | +| `@tanstack/ai-code-mode` | `ai-code-mode` | Code Mode + sandbox driver + server tools | +Skills cross-link. Each skill ships with the package that owns the code (e.g. browser persistence under `ai-core`, not `@tanstack/ai-persistence`). Files live at `node_modules//skills//SKILL.md`. -## Step 3: Review the Generated Mappings +## 3. Review mappings -The install command appends (or creates) an `intent-skills` block that looks like this: +Install appends (or creates) an `intent-skills` block: ```yaml @@ -82,37 +67,37 @@ skills: ``` -Check that the `task:` descriptions match areas you actually work in. Tighten or reword them if needed — they're how your agent decides when to pull the skill into context. +Tighten `task:` lines so they match work you actually do — agents use them to decide when to load a skill. -## Step 4: Confirm It's Wired Up +## 4. Smoke-test the agent -Open a fresh session in your coding agent and ask it to build something with TanStack AI — for example: _"Add a streaming chat endpoint using `@tanstack/ai` and the OpenAI adapter."_ +New session; ask something like: _Add a streaming chat endpoint with `@tanstack/ai` and the OpenAI adapter._ -You should see: +Expect: -- The agent uses `chat()`, not `streamText()`. -- The adapter is imported as `openaiText()` from `@tanstack/ai-openai`, not `createOpenAI()`. -- The response is wrapped with `toServerSentEventsResponse()` instead of manual SSE wiring. -- Middleware is used for lifecycle events (no `onFinish` callback on `chat()`). +- `chat()`, not `streamText()` +- `openaiText()` from `@tanstack/ai-openai`, not `createOpenAI()` +- `toServerSentEventsResponse()` for SSE +- Middleware for lifecycle events (no `onFinish` on `chat()`) -If the agent still falls back to other-SDK patterns, re-open its config file and confirm the `intent-skills` block is present and the `task:` descriptions clearly cover the area you're asking about. +Still wrong patterns → re-check the config file for `intent-skills` and clearer `task:` coverage. -## Keeping Skills Current +## Keep skills current -Skills are versioned with the package. When you bump `@tanstack/ai`, the `SKILL.md` files under `node_modules` update with it — no CLI re-run needed. Re-run `npx @tanstack/intent@latest install` only when you _add_ a new intent-enabled package (for example, adding `@tanstack/ai-code-mode` later) or want to refresh the task mappings. +`SKILL.md` updates when you bump the package — no re-run needed. Re-run Intent install only when you add a new intent-enabled package or want fresh task mappings. -## Using Skills Without the CLI +## Without the CLI -If you'd rather wire skills in yourself, you can reference them directly from `node_modules` in any agent config file. The minimum your agent needs is a pointer to the file: +Point the agent at the file directly: ```markdown When working on TanStack AI code, read and follow: node_modules/@tanstack/ai/skills/ai-core/SKILL.md ``` -The CLI is recommended because it discovers packages automatically and stays consistent with the agent-skills standard, but the underlying file paths are stable. +CLI is preferred for discovery and standard layout; paths above stay stable. -## Learn More +## Related -- [TanStack Intent documentation](https://tanstack.com/intent/latest/docs/overview) — the CLI's full reference, including `scaffold`, `validate`, and CI setup for library maintainers. -- [Agent Skills registry](https://tanstack.com/intent/registry) — browse other intent-enabled packages. +- [TanStack Intent docs](https://tanstack.com/intent/latest/docs/overview) — `scaffold`, `validate`, CI for library authors +- [Agent Skills registry](https://tanstack.com/intent/registry) diff --git a/docs/getting-started/devtools.md b/docs/getting-started/devtools.md index fcae3b717..c38884dac 100644 --- a/docs/getting-started/devtools.md +++ b/docs/getting-started/devtools.md @@ -2,7 +2,7 @@ title: Devtools id: devtools order: 3 -description: "Inspect and debug TanStack AI apps with the TanStack Devtools panel — live chat messages, tool call inputs and outputs, state, and errors." +description: "Inspect TanStack AI hooks, runs, tools, memory, and stream events in the Devtools panel." keywords: - tanstack ai - devtools @@ -13,27 +13,69 @@ keywords: - observability --- -TanStack Devtools is a unified devtools panel for inspecting and debugging TanStack libraries, including TanStack AI. It provides real-time insights into AI interactions, tool calls, and state changes, making it easier to develop and troubleshoot AI-powered applications. +If you need to debug chat/tools → install the AI Devtools plugin, mount `TanStackDevtools`, set `connectToServerBus: true`. -## Features -- **Hook dashboard** - Discover every active TanStack AI hook on the page, including chat, structured output, image, video, audio, speech, transcription, and summarize hooks. -- **Run timeline** - Inspect user turns, linked runs, stream events, client snapshots, and server-only events by `threadId` and `runId`. -- **Real-time Monitoring** - View live chat messages, tool invocations, and AI responses. -- **Tool Call Inspection** - Inspect input and output of tool calls. -- **Tool Fixture Replay** - Build tool payloads from a tool's standard-schema input, append the result into chat messages, and save fixtures in localStorage for repeated UI iteration. -- **State Visualization** - Visualize chat state and message history. -- **Memory Inspector** - For chats wired with `memoryMiddleware`, see what memory recalled and injected each turn plus the current stored records and facts. -- **Error Tracking** - Monitor errors and exceptions in AI interactions. +## What you get -## Hook Dashboard +**Do now** -The AI devtools panel listens for active TanStack AI clients and shows them in the left sidebar. Hooks register when they are created, emit a snapshot immediately, and respond again whenever the devtools panel opens or requests state. This keeps hooks discoverable even when the panel is opened after the app has already rendered. +- Hook dashboard — every active TanStack AI hook (chat, image, video, speech, …) +- Run timeline — turns, stream events, client/server snapshots by `threadId` / `runId` +- Tool call I/O inspection +- Error tracking on AI interactions -Each hook entry includes its type, lifecycle, message count, run count, and the latest linked `threadId`. Selecting a hook opens the full timeline for that hook. Chat hooks keep the current turn-based view: a user message wraps every run and event that happened while answering that turn. The details view also includes lightweight client/server state snapshots between runs so you can see exactly what changed. +**Later / when relevant** -### Naming Hooks +- Tool fixture replay (schema form → append tool result; fixtures in localStorage) +- Memory tab when using [`memoryMiddleware`](../memory/overview.md) -When a page has more than one AI hook, pass `devtools.name` to give each hook a user-facing label in the dashboard. The configured name is display-only; hook type, framework, thread id, and run correlation still come from the TanStack AI client. +Hooks register on create, emit a snapshot, and answer when the panel opens — discoverable even if you open Devtools after render. Multiple hooks → set `devtools.name` for labels. + +## 1. Install + +React: + +```bash +npm install -D @tanstack/react-ai-devtools @tanstack/react-devtools +``` + +Solid: + +```bash +npm install -D @tanstack/solid-ai-devtools @tanstack/solid-devtools +``` + +Preact: + +```bash +npm install -D @tanstack/preact-ai-devtools @tanstack/preact-devtools +``` + +## 2. Mount the panel + +```tsx +import { TanStackDevtools } from '@tanstack/react-devtools' +import { aiDevtoolsPlugin } from '@tanstack/react-ai-devtools' + +const App = () => { + return ( + <> + + + ) +} +``` + +## 3. Name hooks (multi-hook pages) ```tsx import { fetchServerSentEvents, useChat } from '@tanstack/ai-react' @@ -51,7 +93,7 @@ export function SupportChat() { } ``` -The same display option works for specialized generation hooks: +Same option on generation hooks: ```tsx import { fetchServerSentEvents, useGenerateImage } from '@tanstack/ai-react' @@ -69,73 +111,26 @@ export function ImageStudio() { } ``` -## Tool Fixtures +## Tool fixtures -When a `useChat` hook receives tools, the devtools panel lists those tools and their schemas. For standard-schema-compatible inputs, the panel renders a small form from the input schema so you can create a tool call payload without hand-writing JSON. +When `useChat` has tools, the panel lists them and builds a form from standard-schema inputs. Applying a fixture appends tool call + result into real messages; saved fixtures live under the AI Devtools localStorage namespace. -Applying a tool fixture appends the tool call and result into the real chat messages for that hook. Saved fixtures are stored in browser localStorage under the AI devtools namespace so they are available the next time you open the panel. +## Memory inspector -## Memory Inspector +With [`memoryMiddleware`](../memory/overview.md), the **Memory** tab shows: -When a chat is wired with [`memoryMiddleware`](../memory/overview.md), the hook's **Memory** tab shows what the server-side memory backend did for that conversation, grouped by scope (session): +- **Operations** — per-turn recall query, fragment count, chars injected, tools exposed, duration +- **Stored records & facts** — when the adapter implements `inspect` / `listFacts` (`inMemory()`, `redis()` do) -- **Operations timeline** - Each turn's recall: the query, how many fragments came back, how many characters were injected into the system prompt, whether memory-provided tools were exposed, and the recall duration. -- **Stored records & facts** - The current contents of the memory store for the scope, when the adapter implements the optional `inspect`/`listFacts` methods (the built-in `inMemory()` and `redis()` adapters do). Adapters without introspection still show the operations timeline. +State rides the chat stream (`CUSTOM` event). Snapshot is as of turn start; a turn’s writes show on the next turn. Opening the panel mid-conversation replays the latest memory state. -Because memory runs on the server, its state is transported to the panel over the chat stream (a `CUSTOM` event the client re-emits) rather than a separate channel — the same way generation results reach the panel. The snapshot reflects memory as of the start of each turn, so a turn's own writes appear in the next turn's snapshot. Opening the panel after a turn replays the latest memory state, so the tab is populated even when you open devtools mid-conversation. - -## Event Sources - -Client-visible state is emitted by the headless client. Server-only details, such as middleware and provider stream events that never exist on the client, are emitted from the server counterpart. Events include a source descriptor and stable envelope id so the panel can link related events and avoid displaying duplicates. - -## Installation -To use TanStack Devtools with TanStack AI, install the `@tanstack/react-ai-devtools` package: - -```bash -npm install -D @tanstack/react-ai-devtools @tanstack/react-devtools -``` - -Or the `@tanstack/solid-ai-devtools` package for SolidJS: -```bash -npm install -D @tanstack/solid-ai-devtools @tanstack/solid-devtools -``` - -Or the `@tanstack/preact-ai-devtools` package for Preact: -```bash -npm install -D @tanstack/preact-ai-devtools @tanstack/preact-devtools -``` - -## Usage - -Import and include the TanStackDevtools component in your application: - -```tsx -import { TanStackDevtools } from '@tanstack/react-devtools' -import { aiDevtoolsPlugin } from '@tanstack/react-ai-devtools' - -const App = () => { - return ( - <> - - - ) -} -``` +## Event sources -## Using with Next.js (or without a Vite plugin) +Client state comes from the headless client. Server-only middleware/provider events come from the server side. Events carry a source descriptor and stable envelope id so the panel can link and dedupe. -`connectToServerBus: true` relies on a WebSocket/SSE server on port 4206 that is normally started by `@tanstack/devtools-vite`. If you're using Next.js (or any non-Vite bundler), you need to start `ServerEventBus` manually at server boot. +## Next.js (no Vite plugin) -In Next.js, do this in `instrumentation.ts`: +`connectToServerBus: true` needs the event bus on port 4206 (normally from `@tanstack/devtools-vite`). On Next.js, start `ServerEventBus` in `instrumentation.ts`: ```ts ignore export async function register() { @@ -152,4 +147,4 @@ export async function register() { } ``` -This sets globalThis.__TANSTACK_EVENT_TARGET__ so the server-side devtoolsMiddleware (which runs automatically inside every chat() call) can emit tool call events to the bus, which then forwards them to the devtools panel. +Sets `globalThis.__TANSTACK_EVENT_TARGET__` so server-side `devtoolsMiddleware` (auto inside `chat()`) can emit tool events to the panel. diff --git a/docs/getting-started/overview.md b/docs/getting-started/overview.md index 50ab9c21b..20e6e33fc 100644 --- a/docs/getting-started/overview.md +++ b/docs/getting-started/overview.md @@ -2,7 +2,7 @@ title: Overview id: overview order: 1 -description: "TanStack AI is a type-safe, provider-agnostic TypeScript SDK for building streaming chat, tool calling, and AI features that work across any framework." +description: "Type-safe, provider-agnostic TypeScript SDK for streaming chat, tools, and AI features." keywords: - tanstack ai - ai sdk @@ -14,29 +14,25 @@ keywords: - llm sdk --- -TanStack AI is a lightweight, type-safe SDK for building production-ready AI experiences. Its framework-agnostic core provides type-safe tool/function calling, streaming responses, and first-class React and Solid integrations, with adapters for multiple LLM providers — enabling predictable, composable, and testable AI features across any stack. +If you need a working chat UI → [Quick Start](./quick-start). Server only → [Server Quick Start](./quick-start-server). -## Key Features +TanStack AI is a type-safe TypeScript SDK for streaming chat, tool calling, and multi-provider AI apps. Core is framework-agnostic; React, Solid, Vue, Svelte, and Preact ship first-class clients. -- ✅ **Type-Safe** - Full TypeScript support with Zod schema inference -- ✅ **Streaming** - Built-in streaming support for real-time responses -- ✅ **Isomorphic Tools** - Define once with `toolDefinition()`, implement with `.server()` or `.client()` -- ✅ **Framework Agnostic** - Core library works anywhere -- ✅ **Multiple Providers** - OpenRouter, OpenAI, Anthropic, Gemini, Ollama, and more -- ✅ **Approval Flow** - Built-in support for tool approval workflows -- ✅ **Automatic Execution** - Both server and client tools execute automatically +## When to use it -## Framework Agnostic +**Must-have capabilities** -The framework-agnostic core of TanStack AI provides the building blocks for creating AI experiences in any environment, including: +- Type-safe tools and model options (Zod + per-model narrowing) +- Streaming chat with automatic tool execution +- Isomorphic tools: define once, `.server()` / `.client()` +- Adapters for OpenRouter, OpenAI, Anthropic, Gemini, Ollama, and more +- Approval flows for human-in-the-loop tools -- **Next.js** - API routes and App Router -- **TanStack Start** - React Start or Solid Start (recommended!) -- **React Native / Expo** - Native chat screens with `useChat`, absolute server URLs, and XHR streaming transports -- **Express** - Node.js server -- **React Router v7** - Loaders and actions +**Runs in** -TanStack AI lets you define a tool once and provide environment-specific implementations. Using `toolDefinition()` to declare the tool's input/output types and the server behavior with `.server()` (or a client implementation with `.client()`). These isomorphic tools can be invoked from the AI runtime regardless of framework. +Next.js, TanStack Start, Express, React Router v7, React Native / Expo (absolute URL + XHR stream). + +## Define a tool once ```typescript import { chat, toolDefinition } from '@tanstack/ai' @@ -44,7 +40,6 @@ import { openaiText } from '@tanstack/ai-openai' import { z } from 'zod' import { db } from './db' -// Define a tool const getProductsDef = toolDefinition({ name: 'getProducts', description: 'Search for products by query', @@ -52,12 +47,10 @@ const getProductsDef = toolDefinition({ outputSchema: z.array(z.object({ id: z.string(), name: z.string() })), }) -// Create server implementation const getProducts = getProductsDef.server(async ({ query }) => { return await db.products.search(query) }) -// Use in AI chat chat({ adapter: openaiText('gpt-5.2'), messages: [{ role: 'user', content: 'Find products' }], @@ -65,60 +58,34 @@ chat({ }) ``` -## Core Packages - -The TanStack AI ecosystem consists of several packages: - -### `@tanstack/ai` -The core AI library that provides: -- AI adapter interface for connecting to LLM providers -- Chat completion and streaming -- Isomorphic tool/function calling system -- Agent loop strategies -- Type-safe tool definitions with `toolDefinition()` -- Type-safe Model Options based on adapter & model selection -- Type-safe content modalities (text, image, audio, video, document) based on model capabilities - -### `@tanstack/ai-client` -A framework-agnostic headless client for managing chat state: -- Message management with full type safety -- Streaming support -- Connection adapters (SSE, HTTP stream, custom) -- Automatic tool execution (server and client) -- Tool approval flow handling - -### `@tanstack/ai-react` -React hooks for TanStack AI: -- `useChat` hook for chat interfaces -- Automatic state management -- Tool approval flow support -- Type-safe message handling with `InferChatMessages` - -### `@tanstack/ai-solid` -Solid hooks for TanStack AI: -- `useChat` hook for chat interfaces -- Automatic state management -- Tool approval flow support -- Type-safe message handling with `InferChatMessages` +## Packages + +| Package | Use for | +| --- | --- | +| `@tanstack/ai` | `chat()`, adapters, tools, agent loops, typed modalities | +| `@tanstack/ai-client` | Headless chat state, SSE/HTTP/custom connections | +| `@tanstack/ai-react` | React `useChat` | +| `@tanstack/ai-solid` | Solid `useChat` | + +Framework packages wrap the same headless client (messages, streaming, auto tool execution, approvals). ## Adapters -With the help of adapters, TanStack AI can connect to various LLM providers. Available adapters include: - -- **@tanstack/ai-openrouter** - OpenRouter (300+ models via a single API key — recommended) -- **@tanstack/ai-openai** - OpenAI (GPT series) -- **@tanstack/ai-anthropic** - Anthropic (Claude) -- **@tanstack/ai-gemini** - Google Gemini -- **@tanstack/ai-ollama** - Ollama (local models) -- **@tanstack/ai-groq** - Groq -- **@tanstack/ai-grok** - xAI Grok -- **@tanstack/ai-bedrock** - Amazon Bedrock (Claude, Nova, Llama, and more via AWS) -- **@tanstack/ai-byteplus** - BytePlus (Seed chat, Seedance video, Seedream image, Seed Speech) -- **@tanstack/ai-fal** - fal (image & video generation) - -## Next Steps - -- [Quick Start Guide](./quick-start) - Get up and running in minutes -- [Quick Start: React Native](./quick-start-react-native) - Add mobile chat with Expo and a server-owned provider boundary -- [Tools Guide](../tools/tools) - Learn about the isomorphic tool system -- [API Reference](../api/ai) - Explore the full API +**Do now (common)** + +- `@tanstack/ai-openrouter` — 300+ models, one API key (recommended to start) +- `@tanstack/ai-openai` — GPT series +- `@tanstack/ai-anthropic` — Claude +- `@tanstack/ai-gemini` — Google Gemini +- `@tanstack/ai-ollama` — local models + +**Also available** + +`@tanstack/ai-groq`, `@tanstack/ai-grok`, `@tanstack/ai-bedrock`, `@tanstack/ai-byteplus`, `@tanstack/ai-fal` + +## Next + +- [Quick Start: React](./quick-start) +- [Quick Start: React Native](./quick-start-react-native) +- [Tools](../tools/tools) +- [API Reference](../api/ai) diff --git a/docs/getting-started/quick-start-angular.md b/docs/getting-started/quick-start-angular.md index 46b8752b4..6899ec26d 100644 --- a/docs/getting-started/quick-start-angular.md +++ b/docs/getting-started/quick-start-angular.md @@ -2,7 +2,7 @@ title: "Quick Start: Angular" id: quick-start-angular order: 4 -description: "Build a streaming TanStack AI chat component in an Angular app using the injectChat function and the OpenAI adapter." +description: "Streaming chat in Angular with injectChat signals and an OpenAI backend." keywords: - tanstack ai - angular @@ -13,11 +13,11 @@ keywords: - signals --- -You have an Angular app and want to add AI chat. By the end of this guide, you'll have a streaming chat component powered by TanStack AI and OpenAI. +If you need Angular chat → install packages, stream from a backend, call `injectChat` in an injection context. -> **Tip:** If you'd prefer not to sign up with individual AI providers, [OpenRouter](../adapters/openrouter) gives you access to 300+ models with a single API key and is the easiest way to get started. +Prefer one key for many models → [OpenRouter](../adapters/openrouter). -## Installation +## 1. Install ```bash npm install @tanstack/ai @tanstack/ai-angular @tanstack/ai-openai @@ -27,9 +27,9 @@ pnpm add @tanstack/ai @tanstack/ai-angular @tanstack/ai-openai yarn add @tanstack/ai @tanstack/ai-angular @tanstack/ai-openai ``` -## Server Setup +## 2. Server -Angular apps typically use a separate backend. Here's an Express server that streams chat responses: +Express example (Fastify, Hono, Nitro work if they return TanStack AI SSE): ```typescript ignore import express from 'express' @@ -82,11 +82,9 @@ app.post('/api/chat', async (req, res) => { app.listen(3000, () => console.log('Server running on port 3000')) ``` -> **Tip:** Any backend that returns the TanStack AI SSE format works — you can use Fastify, Hono, Nitro, or any other Node.js framework. +## 3. Chat component -## Client Setup - -Create a standalone `ChatComponent` using the `injectChat` function: +Call `injectChat` in a field initializer (or constructor) — not in `ngOnInit`. ```typescript group=quick-start-angular import { Component, signal } from '@angular/core' @@ -148,9 +146,9 @@ export class ChatComponent { } ``` -## Environment Variables +## 4. API keys -Create a `.env` file (or `.env.local` depending on your setup) with your API key: +`.env` or `.env.local` — server only: ```bash # OpenRouter (recommended — access 300+ models with one key) @@ -160,11 +158,9 @@ OPENROUTER_API_KEY=sk-or-... OPENAI_API_KEY=your-openai-api-key ``` -Your server reads this key at runtime. Never expose it to the browser. - -## Angular-Specific Notes +## Angular notes (when you hit them) -**State is exposed as Angular `Signal`s.** The `injectChat` function returns state wrapped in read-only `Signal`s. Read them by calling them as functions: +**State is read-only signals** — call them as functions: ```typescript ignore // In component class @@ -182,7 +178,7 @@ const count = this.chat.messages().length {{ chat.messages().length }} messages ``` -**`injectChat` must be called in an injection context.** Angular's dependency injection requires that `inject()` is called during component construction. The recommended approach is a field initializer (shown above). You can also call it in the constructor or inside `runInInjectionContext`: +**Injection context only** ```typescript import { injectChat } from '@tanstack/ai-angular' @@ -202,23 +198,10 @@ export class MyComponentB { } ``` -Calling `injectChat` outside an injection context — for example, in a lifecycle hook like `ngOnInit` — will throw a runtime error. - -**Automatic cleanup.** The function subscribes to `DestroyRef` internally, so in-flight requests are stopped when the component is destroyed. No manual cleanup needed. - -**Same API shape as React and Vue.** If you're coming from `@tanstack/ai-react` or `@tanstack/ai-vue`, `injectChat` returns the same properties (`messages`, `sendMessage`, `isLoading`, `error`, `status`, `stop`, `reload`, `clear`). The only difference is that each property is an Angular `Signal` rather than a React state value or a Vue `ShallowRef`. - -## That's It! - -You now have a working Angular chat application. The `injectChat` function handles: - -- Message state management -- Streaming responses -- Loading states -- Error handling +Destroy stops in-flight requests (`DestroyRef`). Same API shape as React/Vue (`messages`, `sendMessage`, `isLoading`, `error`, `status`, `stop`, `reload`, `clear`) as signals. -## Next Steps +## Next -- Learn about [Tools](../tools/tools) to add function calling -- Check out the [Adapters](../adapters/openai) to connect to different providers -- See the [React Quick Start](./quick-start) if you're comparing frameworks +- [Tools](../tools/tools) +- [Adapters](../adapters/openai) +- [React Quick Start](./quick-start) diff --git a/docs/getting-started/quick-start-react-native.md b/docs/getting-started/quick-start-react-native.md index a34f7952b..90b1a17c9 100644 --- a/docs/getting-started/quick-start-react-native.md +++ b/docs/getting-started/quick-start-react-native.md @@ -2,7 +2,7 @@ title: "Quick Start: React Native" id: quick-start-react-native order: 3 -description: "Build a React Native or Expo chat screen with TanStack AI's useChat hook, a server-only OpenAI backend, and mobile-compatible streaming transports." +description: "Expo/React Native chat with useChat, a server-only OpenAI route, and XHR streaming." keywords: - tanstack ai - react native @@ -14,39 +14,27 @@ keywords: - openai --- -You have a React Native or Expo app and you want to add streaming AI chat -without putting provider SDKs or API keys in the native bundle. By the end of -this guide, your app will call a server-owned Hono route with `useChat` from -`@tanstack/ai-react`, stream responses over a mobile-compatible transport, and -keep `OPENAI_API_KEY` / `OPENAI_MODEL` on the server. +If you need mobile chat → keep the provider on a server, use an absolute backend URL, start with `xhrHttpStream()`. -> **Coming from the web quick start?** The hook is the same, but the URL and -> transport are different. React Native needs an absolute backend URL, not -> `/api/chat`, and most Expo runtimes should start with `xhrHttpStream()`. +Web quick start differs: no relative `/api/chat`; prefer XHR over streaming `fetch` on Expo. -## 1. Install packages +## 1. Install -If you are starting from scratch, create an Expo app first: +New Expo app (optional): ```bash npx create-expo-app@latest my-ai-chat ``` -Install TanStack AI, the React hook package, the OpenAI adapter for your -server, and Hono for the example backend: - ```bash pnpm add @tanstack/ai @tanstack/ai-react @tanstack/ai-openai hono @hono/node-server zod ``` -If your Expo app lives in a workspace, run the command from the app package or -use your workspace filter. +In a monorepo, run this in the app package (or use a workspace filter). -## 2. Keep OpenAI on the server +## 2. Server: OpenAI stays off the device -Create a Hono route that owns the model, API key, and response format. The -native app sends chat messages to this route; it never imports -`@tanstack/ai-openai` and never receives `OPENAI_API_KEY`. +Native app never imports `@tanstack/ai-openai` or sees `OPENAI_API_KEY`. ```ts // server.ts @@ -100,15 +88,14 @@ serve({ }) ``` -Set server-only environment variables where the Hono process runs: +Server env (where Hono runs): ```env OPENAI_API_KEY=sk-... OPENAI_MODEL=gpt-5.2 ``` -Run the Hono server before starting the native app. For a TypeScript-only -example, install `tsx` and add a script: +Start the server: ```bash pnpm add -D tsx @@ -116,36 +103,28 @@ pnpm pkg set scripts.dev:server="tsx server.ts" pnpm dev:server ``` -> **Route pairing matters:** `xhrHttpStream()` and `fetchHttpStream()` expect -> the newline-delimited JSON response from `toHttpResponse()`. -> `xhrServerSentEvents()` expects the `text/event-stream` response from -> `toServerSentEventsResponse()`. +**Route pairing** -## 3. Configure a native-reachable URL +| Client | Server | +| --- | --- | +| `xhrHttpStream()` / `fetchHttpStream()` | `toHttpResponse()` (NDJSON) | +| `xhrServerSentEvents()` | `toServerSentEventsResponse()` | -React Native is not served from your backend origin, so `/api/chat` cannot work -as a default. Expose the backend URL to Expo with a public variable: +## 3. Absolute URL for the device ```env EXPO_PUBLIC_TANSTACK_AI_BASE_URL=http://192.168.1.10:8787 ``` -Use the address your device can reach: - -- iOS simulator: `http://127.0.0.1:8787` often works. -- Android emulator: use `http://10.0.2.2:8787`. -- Physical device: use your computer's LAN IP, for example - `http://192.168.1.10:8787`, or a tunneled HTTPS URL. - -Only `EXPO_PUBLIC_*` values are bundled into the app. Keep provider keys as -plain server variables such as `OPENAI_API_KEY`. +| Runtime | Base URL | +| --- | --- | +| iOS simulator | `http://127.0.0.1:8787` | +| Android emulator | `http://10.0.2.2:8787` | +| Physical device | LAN IP or HTTPS tunnel | -## 4. Use `useChat` in your native screen +Only `EXPO_PUBLIC_*` is bundled. Keep `OPENAI_API_KEY` server-only. -Start with `xhrHttpStream()` for Expo and React Native. It reads the same -newline-delimited JSON produced by `toHttpResponse()` and relies on XHR progress -events, which are usually more reliable on phone runtimes than streaming -`fetch`. +## 4. Chat screen with `xhrHttpStream` ```tsx // ChatScreen.tsx @@ -199,107 +178,40 @@ export function ChatScreen() { } ``` -You now have a native chat screen that calls your server endpoint, streams -assistant text, and keeps provider credentials outside the app. +## 5. Pick a transport -## 5. Choose a transport deliberately - -Use the transport that matches your server route and runtime: - -| Native runtime | Client adapter | Server response | +| Native runtime | Client | Server | | --- | --- | --- | -| Most Expo / React Native apps | `xhrHttpStream(url)` with `/chat/http` | `toHttpResponse(stream)` | -| SSE-compatible native runtime or proxy path | `xhrServerSentEvents(url)` with `/chat/sse` | `toServerSentEventsResponse(stream)` | -| Runtime with streaming `fetch` support | `fetchHttpStream(url)` with `/chat/http` | `toHttpResponse(stream)` | - -Only use `fetchHttpStream()` when your exact runtime supports all of: +| Most Expo / RN | `xhrHttpStream(url)` → `/chat/http` | `toHttpResponse` | +| SSE-capable path | `xhrServerSentEvents(url)` → `/chat/sse` | `toServerSentEventsResponse` | +| Streaming `fetch` works | `fetchHttpStream(url)` → `/chat/http` | `toHttpResponse` | -- `Response.body` -- `Response.body.getReader()` -- `TextDecoder` +Use `fetchHttpStream()` only if the runtime has `Response.body`, `getReader()`, and `TextDecoder`. Missing any → `UnsupportedResponseStreamError`. Buffered polyfills are not enough. -If any of those are missing, the adapter throws -`UnsupportedResponseStreamError`. A polyfilled `fetch` that buffers the whole -response is not enough; TanStack AI needs incremental response bytes to update -the chat while the model is streaming. +More options (headers, credentials, dynamic URLs): [Connection Adapters](../chat/connection-adapters). -For deeper adapter options such as headers, credentials, `withCredentials`, and -dynamic URLs, see [Connection Adapters](../chat/connection-adapters). +## 6. Try the Expo example -## 6. Try the Expo recipe example - -If you are evaluating React Native support, use the included Expo app. It runs a -local Hono/OpenAI server, shows a transport selector, and streams structured -recipe cards so you can verify native chat and structured output behavior -together. - -Create `examples/ts-react-native-chat/.env`: +`examples/ts-react-native-chat/.env`: ```env OPENAI_API_KEY=sk-... OPENAI_MODEL=gpt-5.2 ``` -Run the example: - ```bash pnpm --filter ts-react-native-chat dev ``` -The command starts: - -- Hono on `0.0.0.0:8787` -- Expo/Metro in LAN mode -- `EXPO_PUBLIC_TANSTACK_AI_BASE_URL=http://:8787` when a LAN address is detected - -Scan the Expo Go QR code from a phone on the same Wi-Fi network. In the app, -use the Testing mode panel to switch between Fetch HTTP, XHR HTTP, and XHR SSE. -The main recipe card streams structured fields such as title, ingredients, -steps, tips, warnings, and revision across follow-up prompts. - -For example-specific commands and network overrides, see -`examples/ts-react-native-chat/README.md`. +Starts Hono on `0.0.0.0:8787`, Expo in LAN mode, and sets `EXPO_PUBLIC_TANSTACK_AI_BASE_URL` when a LAN IP is found. Use the Testing panel to switch Fetch HTTP / XHR HTTP / XHR SSE. Details: `examples/ts-react-native-chat/README.md`. ## Troubleshooting -### `http://localhost:8081` shows JSON - -That is normal. Port `8081` is Metro's manifest and bundle server, not a web UI. -Launch the app from Expo Go, an Android emulator, or an iOS simulator instead. - -### A physical device cannot reach the backend - -Open `http://:8787/health` from the phone browser. If it does not return -`{"ok":true}`, confirm the phone and computer are on the same Wi-Fi network, -client isolation is disabled, and your firewall allows Node.js on the Hono port -and Metro port `8081`. - -### Android emulator cannot reach `127.0.0.1` - -Use `http://10.0.2.2:8787` for -`EXPO_PUBLIC_TANSTACK_AI_BASE_URL`. Android emulators map `10.0.2.2` to the -host machine. - -### Expo prints Android SDK or `adb` warnings - -This is an Android tooling issue, not a TanStack AI transport issue. Confirm -Android Studio installed the SDK, an emulator exists in Device Manager, and -`adb` is on `PATH`. On Windows, check -`%LOCALAPPDATA%\Android\Sdk\platform-tools\adb.exe`. - -### The phone logs `UnsupportedResponseStreamError` - -Your runtime does not expose streaming `fetch`, `Response.body.getReader()`, or -`TextDecoder`. Switch from `fetchHttpStream()` to `xhrHttpStream()` or -`xhrServerSentEvents()`. Do not rely on fetch polyfills unless they provide a -real incremental readable stream. - -### XHR reports a server error - -Check the Hono server terminal first. Common causes are missing -`OPENAI_API_KEY`, an unsupported `OPENAI_MODEL`, or pointing -`xhrServerSentEvents()` at `/chat/http` instead of `/chat/sse` (or the reverse). - -You now have the full React Native path: a server-owned provider boundary, a -native-reachable URL, a mobile-compatible transport, and an Expo example that -proves the setup on a real device. +| Symptom | Fix | +| --- | --- | +| `http://localhost:8081` is JSON | Metro only — open Expo Go / emulator / simulator | +| Device can't reach backend | Open `http://:8787/health` → `{"ok":true}`; same Wi‑Fi, no client isolation, firewall allows 8787 + 8081 | +| Android emulator + `127.0.0.1` | Use `http://10.0.2.2:8787` | +| Android SDK / `adb` warnings | Android Studio SDK + Device Manager; `adb` on `PATH` | +| `UnsupportedResponseStreamError` | Switch to `xhrHttpStream()` or `xhrServerSentEvents()` | +| XHR server error | Check Hono logs: key, model, HTTP vs SSE route mismatch | diff --git a/docs/getting-started/quick-start-server.md b/docs/getting-started/quick-start-server.md index 65291a95c..be24f5961 100644 --- a/docs/getting-started/quick-start-server.md +++ b/docs/getting-started/quick-start-server.md @@ -2,7 +2,7 @@ title: "Quick Start: Server Only" id: quick-start-server order: 5 -description: "Add a streaming AI chat endpoint to a Node.js backend with TanStack AI — no UI framework required." +description: "Streaming AI chat endpoints in Node.js — no UI framework." keywords: - tanstack ai - node.js @@ -14,11 +14,11 @@ keywords: - sse --- -You have a Node.js backend and want to add AI capabilities. By the end of this guide, you'll have a working chat endpoint powered by TanStack AI and OpenAI -- no UI framework required. +If you need AI on a Node backend only → install core + adapter, call `chat()`, return SSE or NDJSON. -> **Tip:** If you'd prefer not to sign up with individual AI providers, [OpenRouter](../adapters/openrouter) gives you access to 300+ models with a single API key and is the easiest way to get started. +Prefer one key for many models → [OpenRouter](../adapters/openrouter). -## Installation +## 1. Install ```bash npm install @tanstack/ai @tanstack/ai-openai @@ -28,9 +28,7 @@ pnpm add @tanstack/ai @tanstack/ai-openai yarn add @tanstack/ai @tanstack/ai-openai ``` -## Basic Chat - -The simplest way to get a response -- call `chat()` and collect the text: +## 2. One-shot text ```typescript import { chat, streamToText } from '@tanstack/ai' @@ -45,11 +43,11 @@ const text = await streamToText(stream) console.log(text) ``` -`chat()` returns an `AsyncIterable`. `streamToText` consumes it and returns the accumulated text content. +`chat()` → `AsyncIterable`. `streamToText` accumulates text. -## HTTP Endpoint +## 3. Streaming HTTP (SSE) -Here's an Express server that exposes a streaming chat endpoint using Server-Sent Events: +Compatible with `@tanstack/ai-react` / `ai-vue` / `ai-svelte` later. Fastify/Hono also work if they return TanStack AI SSE. ```typescript ignore import express from 'express' @@ -89,13 +87,9 @@ app.post('/api/chat', async (req, res) => { app.listen(3000, () => console.log('Server running on port 3000')) ``` -> **Tip:** Any backend that returns the TanStack AI SSE format works -- you can use Fastify, Hono, or any other Node.js framework. - -This endpoint is compatible with TanStack AI's client-side `useChat` hooks (`@tanstack/ai-react`, `@tanstack/ai-vue`, `@tanstack/ai-svelte`), so you can pair it with any frontend later. +## 4. Tools (optional) -## With Tools - -Define a server tool with `toolDefinition` and pass it to `chat()`. The agent loop automatically calls your tool and feeds the result back to the model: +Agent loop calls the tool and continues in one `chat()`: ```typescript import { chat, toolDefinition, streamToText } from '@tanstack/ai' @@ -121,13 +115,9 @@ const text = await streamToText(stream) console.log(text) ``` -The model decides when to call `getWeather`, receives the result, and incorporates it into its response -- all within a single `chat()` call. - -## Alternative Response Formats +## Other response shapes -TanStack AI ships several ways to return a stream over HTTP: - -**`toHttpResponse()`** returns a `Response` using newline-delimited JSON instead of SSE. Pair it with `fetchHttpStream` on the client: +**NDJSON** — pair with `fetchHttpStream` on the client: ```typescript import { chat, toHttpResponse } from '@tanstack/ai' @@ -143,7 +133,7 @@ export async function POST(request: Request) { } ``` -**Raw stream consumption** -- iterate the `AsyncIterable` directly with `for await`: +**Raw chunks** ```typescript import { stream } from './stream' @@ -155,11 +145,7 @@ for await (const chunk of stream) { } ``` -This gives you full control over every chunk type (text deltas, tool calls, run lifecycle events, etc.). - -## Environment Variables - -Create a `.env` file with your API key: +## API keys ```bash # OpenRouter (recommended — access 300+ models with one key) @@ -169,11 +155,11 @@ OPENROUTER_API_KEY=sk-or-... OPENAI_API_KEY=your-openai-api-key ``` -The adapter reads `OPENAI_API_KEY` at runtime. Never expose it to the browser. +Adapter reads the key at runtime. Never expose to the browser. -## Next Steps +## Next -- Learn about [Tools](../tools/tools) to add function calling and agent loops -- Explore [StreamProcessor](../reference/classes/StreamProcessor) for fine-grained stream control -- Check out the [Adapters](../adapters/openai) to connect to different providers -- See the [React Quick Start](./quick-start) to add a frontend +- [Tools](../tools/tools) +- [StreamProcessor](../reference/classes/StreamProcessor) +- [Adapters](../adapters/openai) +- [React Quick Start](./quick-start) diff --git a/docs/getting-started/quick-start-svelte.md b/docs/getting-started/quick-start-svelte.md index f74a9867b..6ba02a520 100644 --- a/docs/getting-started/quick-start-svelte.md +++ b/docs/getting-started/quick-start-svelte.md @@ -2,7 +2,7 @@ title: "Quick Start: Svelte" id: quick-start-svelte order: 4 -description: "Add a streaming TanStack AI chat component to a SvelteKit app using Svelte 5 runes and the OpenAI adapter." +description: "Streaming chat in SvelteKit with createChat, Svelte 5 runes, and OpenAI." keywords: - tanstack ai - svelte @@ -14,11 +14,11 @@ keywords: - runes --- -You have a SvelteKit app and want to add AI chat. By the end of this guide, you'll have a streaming chat component powered by TanStack AI and OpenAI. +If you need SvelteKit chat → install packages, add `/api/chat`, wire `createChat` on the page. -> **Tip:** If you'd prefer not to sign up with individual AI providers, [OpenRouter](../adapters/openrouter) gives you access to 300+ models with a single API key and is the easiest way to get started. +Prefer one key for many models → [OpenRouter](../adapters/openrouter). -## Installation +## 1. Install ```bash npm install @tanstack/ai @tanstack/ai-svelte @tanstack/ai-openai @@ -28,9 +28,7 @@ pnpm add @tanstack/ai @tanstack/ai-svelte @tanstack/ai-openai yarn add @tanstack/ai @tanstack/ai-svelte @tanstack/ai-openai ``` -## Server Setup - -Create a SvelteKit API route that streams chat responses: +## 2. Server route ```typescript ignore // src/routes/api/chat/+server.ts @@ -68,11 +66,11 @@ export const POST: RequestHandler = async ({ request }) => { } ``` -> **Tip:** `toServerSentEventsResponse` returns a standard `Response`, so it works with any server that speaks the Web Response API -- SvelteKit, Hono, Cloudflare Workers, etc. +`toServerSentEventsResponse` returns a standard `Response` (SvelteKit, Hono, Workers, etc.). -## Client Setup +## 3. Page component -Create a Svelte 5 component using `createChat`: +Use `createChat` (not `useChat`): ```svelte @@ -112,9 +110,9 @@ function handleSubmit() {
``` -## Environment Variables +`chat.messages` / `chat.isLoading` are reactive getters (no `.value`). Same surface as React/Vue: `messages`, `sendMessage`, `isLoading`, `error`, `status`, `stop`, `reload`, `clear`. -Create a `.env` file with your API key: +## 4. API keys ```bash # OpenRouter (recommended -- access 300+ models with one key) @@ -124,15 +122,11 @@ OPENROUTER_API_KEY=sk-or-... OPENAI_API_KEY=your-openai-api-key ``` -Your SvelteKit server reads this key at runtime. Never expose it to the browser. - -## Svelte-Specific Notes - -**`createChat`, not `useChat`.** The Svelte integration uses `createChat` instead of `useChat` to follow Svelte's naming conventions. The returned object has the same properties as the React and Vue versions (`messages`, `sendMessage`, `isLoading`, `error`, `status`, `stop`, `reload`, `clear`). +Server-only. Never expose to the browser. -**Svelte 5 runes.** The examples above use Svelte 5 runes (`$state`). The `createChat` return object uses reactive getters internally, so `chat.messages` and `chat.isLoading` are reactive without any extra wrappers -- no `.value` like Vue, no signals to unwrap. +## Cleanup on unmount -**No automatic cleanup.** Unlike the React and Vue integrations, `createChat` does not register automatic cleanup. If your component can unmount while a response is streaming, call `chat.stop()` in an `onDestroy` callback: +`createChat` does **not** auto-stop streams. If the page can unmount mid-stream: ```svelte ``` -## That's It! - -You now have a working SvelteKit chat application. The `createChat` function handles: - -- Message state management -- Streaming responses -- Loading states -- Error handling - -## Next Steps +## Next -- Learn about [Tools](../tools/tools) to add function calling -- Check out the [Adapters](../adapters/openai) to connect to different providers -- See the [React Quick Start](./quick-start) if you're comparing frameworks +- [Tools](../tools/tools) +- [Adapters](../adapters/openai) +- [React Quick Start](./quick-start) diff --git a/docs/getting-started/quick-start-vue.md b/docs/getting-started/quick-start-vue.md index 14afbb5ab..59acd1490 100644 --- a/docs/getting-started/quick-start-vue.md +++ b/docs/getting-started/quick-start-vue.md @@ -2,7 +2,7 @@ title: "Quick Start: Vue" id: quick-start-vue order: 3 -description: "Build a streaming TanStack AI chat component in a Vue 3 app using the useChat composable and the OpenAI adapter." +description: "Streaming chat in Vue 3 with useChat and an OpenAI (or OpenRouter) backend." keywords: - tanstack ai - vue @@ -14,11 +14,11 @@ keywords: - composable --- -You have a Vue 3 app and want to add AI chat. By the end of this guide, you'll have a streaming chat component powered by TanStack AI and OpenAI. +If you need Vue chat → install packages, stream from a backend, wire `useChat` in a component. -> **Tip:** If you'd prefer not to sign up with individual AI providers, [OpenRouter](../adapters/openrouter) gives you access to 300+ models with a single API key and is the easiest way to get started. +Prefer one key for many models → [OpenRouter](../adapters/openrouter). -## Installation +## 1. Install ```bash npm install @tanstack/ai @tanstack/ai-vue @tanstack/ai-openai @@ -28,9 +28,9 @@ pnpm add @tanstack/ai @tanstack/ai-vue @tanstack/ai-openai yarn add @tanstack/ai @tanstack/ai-vue @tanstack/ai-openai ``` -## Server Setup +## 2. Server -Vue apps typically use a separate backend. Here's an Express server that streams chat responses: +Express example (Fastify, Hono, Nitro work if they return TanStack AI SSE): ```typescript ignore import express from 'express' @@ -83,11 +83,7 @@ app.post('/api/chat', async (req, res) => { app.listen(3000, () => console.log('Server running on port 3000')) ``` -> **Tip:** Any backend that returns the TanStack AI SSE format works -- you can use Fastify, Hono, Nitro, or any other Node.js framework. - -## Client Setup - -Create a `Chat.vue` component using the `useChat` composable: +## 3. `Chat.vue` ```vue