feat(ai): native Files API support across providers (upload adapters + file content source) - #915
Open
tombeckenham wants to merge 5 commits into
Conversation
Contributor
📝 WalkthroughWalkthroughChangesNative Files API support now spans shared file contracts, upload and lifecycle activities, OpenAI/Anthropic/Gemini/fal adapters, provider-specific content mapping, fail-closed capability checks, documentation, examples, and end-to-end validation. Native Files API
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Application
participant FilesAdapter
participant ProviderFilesAPI
participant ChatActivity
participant ProviderAdapter
Application->>FilesAdapter: uploadFile(input)
FilesAdapter->>ProviderFilesAPI: upload normalized file
ProviderFilesAPI-->>FilesAdapter: provider metadata
FilesAdapter-->>Application: FileHandle
Application->>ChatActivity: send fileSourceFromHandle(handle)
ChatActivity->>ProviderAdapter: validate and convert file source
ProviderAdapter-->>ChatActivity: provider-specific request content
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
🚀 Changeset Version Preview21 package(s) bumped directly, 25 bumped as dependents. 🟥 Major bumps
🟨 Minor bumps
🟩 Patch bumps
|
|
View your CI Pipeline Execution ↗ for commit aaeb090
☁️ Nx Cloud last updated this comment at |
|
View your CI Pipeline Execution ↗ for commit 83e6d90
☁️ Nx Cloud last updated this comment at |
@tanstack/ai
@tanstack/ai-acp
@tanstack/ai-angular
@tanstack/ai-anthropic
@tanstack/ai-bedrock
@tanstack/ai-byteplus
@tanstack/ai-claude-code
@tanstack/ai-client
@tanstack/ai-code-mode
@tanstack/ai-code-mode-skills
@tanstack/ai-codex
@tanstack/ai-devtools-core
@tanstack/ai-durable-stream
@tanstack/ai-elevenlabs
@tanstack/ai-event-client
@tanstack/ai-fal
@tanstack/ai-gemini
@tanstack/ai-grok
@tanstack/ai-grok-build
@tanstack/ai-groq
@tanstack/ai-isolate-cloudflare
@tanstack/ai-isolate-node
@tanstack/ai-isolate-quickjs
@tanstack/ai-mcp
@tanstack/ai-memory
@tanstack/ai-mistral
@tanstack/ai-ollama
@tanstack/ai-openai
@tanstack/ai-opencode
@tanstack/ai-openrouter
@tanstack/ai-persistence
@tanstack/ai-preact
@tanstack/ai-react
@tanstack/ai-react-ui
@tanstack/ai-sandbox
@tanstack/ai-sandbox-cloudflare
@tanstack/ai-sandbox-daytona
@tanstack/ai-sandbox-docker
@tanstack/ai-sandbox-local-process
@tanstack/ai-sandbox-sprites
@tanstack/ai-sandbox-vercel
@tanstack/ai-solid
@tanstack/ai-solid-ui
@tanstack/ai-svelte
@tanstack/ai-utils
@tanstack/ai-vue
@tanstack/ai-vue-ui
@tanstack/openai-base
@tanstack/preact-ai-devtools
@tanstack/react-ai-devtools
@tanstack/solid-ai-devtools
commit: |
tombeckenham
force-pushed
the
909-featai-native-files-api-support-across-providers-upload-adapters-+-file-content-source
branch
from
July 10, 2026 11:10
d440425 to
160e1d0
Compare
…+ `file` content source)
Add first-class support for provider Files / storage APIs so callers can
upload media once and reference it by a provider-issued handle instead of
re-sending base64 or a public URL each request.
- New tree-shakeable `files` adapter kind: openaiFiles(), anthropicFiles(),
geminiFiles(), falFiles() — each with upload(), plus get()/delete() where
the provider has a lifecycle API (fal is upload-only). Driven by the new
uploadFile()/getFile()/deleteFile() activity functions.
- New `{ type: 'file' }` arm on ContentPartSource. Adapters map it to the
provider's native reference: OpenAI (Responses) input_image/input_file
file_id, Anthropic file_id source (sends the files-api-2025-04-14 beta),
Gemini fileData.fileUri, fal storage URL. fileSourceFromHandle() builds
the source from an uploaded FileHandle.
- Runtime provider routing: a handle only routes to its issuing provider;
cross-provider handles and endpoints that require raw bytes (image edits,
Veo, Chat Completions images, Bedrock, Mistral, Grok, OpenRouter, Ollama)
throw a clear error instead of silently mis-mapping.
Closes #909
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… + Nitro note Upload reference images (Gemini) and image-to-video start frames (fal) once via the Files API (`geminiFiles()` / `falFiles()`) and reference them by handle, instead of re-sending the base64 payload inline on every generation request. Bump the example's `nitro` to `latest` (3.0.260610-beta) — older Nitro rejected `@google/genai`'s resumable upload (explicit `Content-Length` on a Blob body) with "invalid content-length header", surfaced as "fetch failed". Document that runtime requirement in the Files API guide. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…thread provider-literal handle types Post-review completion pass for the Files API PR (rebased onto main): - ai-byteplus (landed on main after this branch): file-source guards in text/image/video adapters + tests — a handle would previously have been sent to Ark as a URL. - Gemini Interactions video path: add the missing assertOwnFileSource guard (the sibling text-interactions adapter already had it) + tests. - openai-base Responses: new `supportsFileIdInput` gate (true only for OpenAI itself) so Grok/Bedrock/compatible subclasses reject file sources instead of inheriting the file_id mapping; chat-completions error copy no longer points non-OpenAI providers at openaiText. Tests for both. - Provider-literal handle types: FileHandle<TProvider>/FilesAdapter<TName> thread each adapter's name literal through uploadFile; getFile/deleteFile accept the handle itself, making cross-provider lifecycle calls compile errors. fileSourceFromHandle + FileHandle now also exported from the browser-safe @tanstack/ai/client entry. - Docs: fix broken openaiText import in multimodal-content.md, document the uploadFile/getFile/deleteFile dispatchers, correct the fal get/delete and explicit-key claims; kiira green. Skills: adapter-configuration §7 + chat-experience/media-generation notes. - Tests: gemini files-source mapping/rejection, OpenAI Responses document- arm behavior, toFileHandle normalizers (openai/gemini/fal), handle-object lifecycle dispatch, and an aimock e2e (file-source-wire) covering the round-trip + cross-provider rejection end-to-end. - Polish: unsupportedFileSourceError detail no longer contradicted by the generic tail, ollama uses this.name, fal expiresIn !== undefined, comment rot fixes; changeset updated (adds ai-byteplus patch).
tombeckenham
force-pushed
the
909-featai-native-files-api-support-across-providers-upload-adapters-+-file-content-source
branch
from
August 7, 2026 06:41
160e1d0 to
7f0f9d5
Compare
Contributor
Author
|
Rebased onto main (was 47 behind / conflicting) and completed a post-review pass. Summary of what changed beyond the rebase: Sweep completion
Types
Docs / skills
Tests
Quality gates: |
…y preflight
Redesigns the `{ type: 'file' }` content source before it ships:
- The source now carries a per-provider reference record —
`{ type: 'file', reference: { openai: 'file-…', gemini: 'https://…' } }` —
instead of a single { value, provider } pair. Each adapter reads only its
own entry (`fileReferenceFor`), and a lookup miss throws naming the
providers that are present. `fileSourceFromHandle(...handles)` merges
handles from several providers into one source that routes to any of
them (upload once per provider, replay the same conversation anywhere).
- New fail-closed preflight: adapters that can consume file references
declare `supportsFileSources`; chat()/generateImage()/generateVideo()
reject file sources for every other adapter before a request is built.
Adapters written before this feature existed can no longer silently
mis-map a reference onto their URL/data branch — the failure class the
per-adapter sweep was policing by convention is now structural.
(openai-base's supportsFileIdInput gate is folded into the same flag.)
- Removing `value` from the file arm also makes fall-through code a
compile error — caught two latent OpenRouter paths that would have sent
a reference as a URL, now restructured with narrowed sources.
- Docs, agent skills, changeset, and all files-source tests updated to the
record shape; new preflight unit tests; e2e spec asserts the record
round-trip and lookup-miss rejection end-to-end.
tombeckenham
marked this pull request as ready for review
August 7, 2026 08:07
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 10
🧹 Nitpick comments (6)
packages/ai-byteplus/tests/files-source.test.ts (1)
1-5: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winPlace these unit tests beside the adapters they cover.
packages/ai-byteplus/tests/files-source.test.ts#L1-L5: move this test beside the BytePlus text adapter underpackages/ai-byteplus/src/adapters/.packages/ai-grok/tests/files-source.test.ts#L1-L5: move this test beside the Grok text adapter underpackages/ai-grok/src/adapters/.As per coding guidelines, “Test files should be placed alongside source code as *.test.ts files using Vitest.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/ai-byteplus/tests/files-source.test.ts` around lines 1 - 5, Move the BytePlus test from packages/ai-byteplus/tests/files-source.test.ts to packages/ai-byteplus/src/adapters/files-source.test.ts, keeping its contents unchanged. Also move the Grok test from packages/ai-grok/tests/files-source.test.ts to packages/ai-grok/src/adapters/files-source.test.ts so each test sits beside the adapter it covers.Source: Coding guidelines
packages/ai-anthropic/tests/files-source.test.ts (1)
1-4: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winPlace the new tests beside their source modules.
Both tests use the same unsupported package-level test-directory layout.
packages/ai-anthropic/tests/files-source.test.ts#L1-L4: move the test besidepackages/ai-anthropic/src/adapters/text.tsas a*.test.tsfile.packages/ai-gemini/tests/files-adapter.test.ts#L1-L2: move the test besidepackages/ai-gemini/src/adapters/files.tsas a*.test.tsfile.As per coding guidelines: Test files should be placed alongside source code as
*.test.tsfiles.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/ai-anthropic/tests/files-source.test.ts` around lines 1 - 4, Move the test from packages/ai-anthropic/tests/files-source.test.ts beside packages/ai-anthropic/src/adapters/text.ts, preserving its *.test.ts name and contents. Also move packages/ai-gemini/tests/files-adapter.test.ts beside packages/ai-gemini/src/adapters/files.ts as a *.test.ts file; no test logic changes are required.Source: Coding guidelines
packages/ai-openai/tests/files-source.test.ts (1)
1-4: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winPlace this test alongside the source.
Move this test beside the exercised adapter, such as
packages/ai-openai/src/adapters/text.files-source.test.ts. The test already uses Vitest.As per coding guidelines, “Test files should be placed alongside source code as *.test.ts files using Vitest with happy-dom for DOM testing.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/ai-openai/tests/files-source.test.ts` around lines 1 - 4, Move the test from the package-level test location to sit beside the exercised OpenAITextAdapter source, using the adjacent *.files-source.test.ts naming pattern under the adapters directory. Preserve its existing Vitest implementation and assertions.Source: Coding guidelines
packages/ai-gemini/tests/files-source.test.ts (1)
1-4: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueMove this test next to its source module.
Place this new
*.test.tsfile beside the tested adapter underpackages/ai-gemini/src/adapters/. Keep Vitest.As per coding guidelines, “Test files should be placed alongside source code as *.test.ts files using Vitest with happy-dom for DOM testing.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/ai-gemini/tests/files-source.test.ts` around lines 1 - 4, Move the test file beside the tested GeminiTextAdapter source under src/adapters, preserving its *.test.ts name and Vitest-based implementation.Source: Coding guidelines
packages/ai/tests/files-source.test.ts (1)
1-220: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚖️ Poor tradeoffMove new unit tests next to their source modules.
packages/ai/tests/files-source.test.ts#L1-L220: colocate the tests with the file-source utilities.packages/ai-fal/tests/content-source-to-fal-url.test.ts#L1-L35: colocate the tests withsrc/image/image-inputs.ts.packages/ai-fal/tests/files-adapter.test.ts#L1-L69: colocate the tests withsrc/adapters/files.ts.packages/ai-openai/tests/files-adapter.test.ts#L1-L75: colocate the tests withsrc/adapters/files.ts.As per coding guidelines, “Test files should be placed alongside source code as *.test.ts files using Vitest with happy-dom for DOM testing.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/ai/tests/files-source.test.ts` around lines 1 - 220, Move the tests alongside their source modules: relocate packages/ai/tests/files-source.test.ts (lines 1-220) next to the file-source utilities, packages/ai-fal/tests/content-source-to-fal-url.test.ts (lines 1-35) next to src/image/image-inputs.ts, packages/ai-fal/tests/files-adapter.test.ts (lines 1-69) next to src/adapters/files.ts, and packages/ai-openai/tests/files-adapter.test.ts (lines 1-75) next to src/adapters/files.ts. Preserve the existing Vitest tests and update imports as needed after relocation; each site requires a direct move.Source: Coding guidelines
packages/ai-openai/src/index.ts (1)
81-87: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd the
/adapterssubpath for OpenAI adapter imports.
packages/ai-openaihas nosrc/adapters/index.tsor./adaptersexport, so users cannot import the required tree-shakeableopenaiFilesfactory from@tanstack/ai-openai/adapters. Add the subpath barrel and export entry.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/ai-openai/src/index.ts` around lines 81 - 87, Add an adapters barrel at src/adapters/index.ts that re-exports the OpenAI files adapter symbols, and add the corresponding ./adapters package export entry so `@tanstack/ai-openai/adapters` resolves to that barrel while preserving the existing root exports.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/ts-react-chat/src/routes/index.tsx`:
- Around line 337-339: Update hasRenderablePart to count an image as renderable
only when its source contains a value, matching the guard in the image rendering
branch. Preserve visibility for other supported part types and keep the existing
<img> rendering behavior unchanged.
In `@examples/ts-react-media/src/lib/server-functions.ts`:
- Around line 115-139: The uploadInlineImageInputs flow in
examples/ts-react-media/src/lib/server-functions.ts must cache uploaded handles
per inline image, reuse fileSourceFromHandle(handle) on later requests, and
remove temporary uploads when a request fails. Update
examples/ts-react-media/src/components/ImageGenerator.tsx:185-186 to preserve
and pass the reusable handles, and update docs/advanced/files-api.md:92-110 to
document the reuse and failure-cleanup behavior.
In `@packages/ai-anthropic/src/adapters/files.ts`:
- Around line 71-76: Ensure the explicit apiKey argument remains authoritative
in every provider factory by spreading config before it: update
createAnthropicFiles in packages/ai-anthropic/src/adapters/files.ts (lines
71-76), the corresponding factory in packages/ai-gemini/src/adapters/files.ts
(lines 69-74), and the corresponding factory in
packages/ai-openai/src/adapters/files.ts (lines 72-77) to construct adapters
with config first and apiKey last.
In `@packages/ai-openai/tests/files-source.test.ts`:
- Around line 148-178: Update the foreign-provider file-handle test around chat
and the create mock to assert that the OpenAI client is never called. Keep the
existing RUN_ERROR and OpenAI-message assertions, and add a call-count
expectation on the mocked create function after consuming the stream.
In `@packages/ai/src/activities/chat/index.ts`:
- Around line 1339-1345: Validate file-source support in
runStructuredFinalization after structured-output middleware configuration and
immediately before each structured-output provider call, including
structuredOutputStream and structuredOutput. Reuse
assertMessagesFileSourceSupport(this.adapter, this.messages) so schema-only
requests fail closed for adapters without file-source support.
In `@packages/ai/src/activities/generateImage/index.ts`:
- Around line 285-288: Move the existing try block in the generation flow to
begin before assertPromptFileSourceSupport(adapter, rest.prompt), while keeping
the preflight validation inside that try. Ensure failures after
runGenerationStart are routed through the existing runGenerationError and
logger.errors handling.
In `@packages/ai/src/types.ts`:
- Around line 279-286: Update the documentation for ContentPartFileSource in the
multimodal content source specification to describe reference as a
provider-keyed record, rather than a singular issuing provider. Keep the
descriptions for data and url sources unchanged.
In `@packages/ai/src/utilities/tool-result.ts`:
- Line 14: The file-source validation must reject array references and only
accept a non-empty provider-to-reference record. Update the relevant type
guard/schema in tool-result handling to require !Array.isArray(reference) and
validate record values with Zod rather than Object.values() on untrusted input;
revise the nearby docstring to describe a non-empty reference record instead of
a string value.
In `@testing/e2e/src/routes/api.file-source-wire.ts`:
- Around line 25-29: Validate the file-source wire HTTP contract with Zod: in
testing/e2e/src/routes/api.file-source-wire.ts lines 25-29, parse provider,
handleProvider, and testId before adapter creation and return HTTP 400 for
invalid input; in testing/e2e/tests/file-source-wire.spec.ts lines 36, 54, 67,
82, and 93, replace response type assertions with the appropriate Zod result
schema parsing for each OpenAI, Anthropic, and Gemini success or rejection
response.
- Around line 71-76: Update the RUN_ERROR handling in the route’s
chunk-processing logic to assign runError for every chunk with type RUN_ERROR,
regardless of whether its message contains “file”; preserve the existing message
value for the failure response.
---
Nitpick comments:
In `@packages/ai-anthropic/tests/files-source.test.ts`:
- Around line 1-4: Move the test from
packages/ai-anthropic/tests/files-source.test.ts beside
packages/ai-anthropic/src/adapters/text.ts, preserving its *.test.ts name and
contents. Also move packages/ai-gemini/tests/files-adapter.test.ts beside
packages/ai-gemini/src/adapters/files.ts as a *.test.ts file; no test logic
changes are required.
In `@packages/ai-byteplus/tests/files-source.test.ts`:
- Around line 1-5: Move the BytePlus test from
packages/ai-byteplus/tests/files-source.test.ts to
packages/ai-byteplus/src/adapters/files-source.test.ts, keeping its contents
unchanged. Also move the Grok test from
packages/ai-grok/tests/files-source.test.ts to
packages/ai-grok/src/adapters/files-source.test.ts so each test sits beside the
adapter it covers.
In `@packages/ai-gemini/tests/files-source.test.ts`:
- Around line 1-4: Move the test file beside the tested GeminiTextAdapter source
under src/adapters, preserving its *.test.ts name and Vitest-based
implementation.
In `@packages/ai-openai/src/index.ts`:
- Around line 81-87: Add an adapters barrel at src/adapters/index.ts that
re-exports the OpenAI files adapter symbols, and add the corresponding
./adapters package export entry so `@tanstack/ai-openai/adapters` resolves to that
barrel while preserving the existing root exports.
In `@packages/ai-openai/tests/files-source.test.ts`:
- Around line 1-4: Move the test from the package-level test location to sit
beside the exercised OpenAITextAdapter source, using the adjacent
*.files-source.test.ts naming pattern under the adapters directory. Preserve its
existing Vitest implementation and assertions.
In `@packages/ai/tests/files-source.test.ts`:
- Around line 1-220: Move the tests alongside their source modules: relocate
packages/ai/tests/files-source.test.ts (lines 1-220) next to the file-source
utilities, packages/ai-fal/tests/content-source-to-fal-url.test.ts (lines 1-35)
next to src/image/image-inputs.ts, packages/ai-fal/tests/files-adapter.test.ts
(lines 1-69) next to src/adapters/files.ts, and
packages/ai-openai/tests/files-adapter.test.ts (lines 1-75) next to
src/adapters/files.ts. Preserve the existing Vitest tests and update imports as
needed after relocation; each site requires a direct move.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 3eca19b3-6c88-4f64-97c0-9999d5ce0532
📒 Files selected for processing (71)
.changeset/native-files-api-support.mddocs/advanced/files-api.mddocs/advanced/multimodal-content.mddocs/config.jsonexamples/ts-react-chat/src/routes/index.tsxexamples/ts-react-media/src/components/ImageGenerator.tsxexamples/ts-react-media/src/lib/server-functions.tspackages/ai-anthropic/src/adapters/files.tspackages/ai-anthropic/src/adapters/text.tspackages/ai-anthropic/src/index.tspackages/ai-anthropic/src/text/text-provider-options.tspackages/ai-anthropic/tests/files-source.test.tspackages/ai-bedrock/src/converse/message-converter.tspackages/ai-byteplus/src/adapters/image.tspackages/ai-byteplus/src/adapters/text.tspackages/ai-byteplus/src/adapters/video.tspackages/ai-byteplus/tests/files-source.test.tspackages/ai-event-client/src/index.tspackages/ai-fal/src/adapters/files.tspackages/ai-fal/src/adapters/image.tspackages/ai-fal/src/adapters/video.tspackages/ai-fal/src/image/image-inputs.tspackages/ai-fal/src/index.tspackages/ai-fal/tests/content-source-to-fal-url.test.tspackages/ai-fal/tests/files-adapter.test.tspackages/ai-gemini/src/adapters/files.tspackages/ai-gemini/src/adapters/image.tspackages/ai-gemini/src/adapters/text.tspackages/ai-gemini/src/adapters/video.tspackages/ai-gemini/src/experimental/text-interactions/adapter.tspackages/ai-gemini/src/index.tspackages/ai-gemini/tests/files-adapter.test.tspackages/ai-gemini/tests/files-source.test.tspackages/ai-grok/src/adapters/image.tspackages/ai-grok/src/adapters/video.tspackages/ai-grok/tests/files-source.test.tspackages/ai-mistral/src/adapters/text.tspackages/ai-ollama/src/adapters/text.tspackages/ai-openai/src/adapters/files.tspackages/ai-openai/src/adapters/text.tspackages/ai-openai/src/image/image-input-to-file.tspackages/ai-openai/src/index.tspackages/ai-openai/tests/files-adapter.test.tspackages/ai-openai/tests/files-source.test.tspackages/ai-openrouter/src/adapters/image.tspackages/ai-openrouter/src/adapters/responses-text.tspackages/ai-openrouter/src/adapters/text.tspackages/ai/skills/ai-core/adapter-configuration/SKILL.mdpackages/ai/skills/ai-core/chat-experience/SKILL.mdpackages/ai/skills/ai-core/media-generation/SKILL.mdpackages/ai/src/activities/chat/adapter.tspackages/ai/src/activities/chat/index.tspackages/ai/src/activities/files/adapter.tspackages/ai/src/activities/files/index.tspackages/ai/src/activities/generateImage/adapter.tspackages/ai/src/activities/generateImage/index.tspackages/ai/src/activities/generateVideo/adapter.tspackages/ai/src/activities/generateVideo/index.tspackages/ai/src/activities/index.tspackages/ai/src/client.tspackages/ai/src/index.tspackages/ai/src/types.tspackages/ai/src/utilities/content-source.tspackages/ai/src/utilities/tool-result.tspackages/ai/tests/files-source.test.tspackages/ai/tests/media-prompt.test.tspackages/openai-base/src/adapters/chat-completions-text.tspackages/openai-base/src/adapters/responses-text.tstesting/e2e/src/routeTree.gen.tstesting/e2e/src/routes/api.file-source-wire.tstesting/e2e/tests/file-source-wire.spec.ts
Comment on lines
+337
to
+339
| // Render image parts (file references have no local bytes | ||
| // or URL to render, so only url/data sources get an <img>) | ||
| if (part.type === 'image' && 'value' in part.source) { |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate the file and inspect the relevant sections without running repo code.
file="examples/ts-react-chat/src/routes/index.tsx"
if [ ! -f "$file" ]; then
echo "missing $file"
fd -a 'index\.tsx$' .
exit 1
fi
echo "== file size =="
wc -l "$file"
echo "== outline around render helpers =="
ast-grep outline "$file" --match 'hasRenderablePart' --view expanded || true
echo "== relevant lines 300-360 =="
sed -n '300,360p' "$file" | nl -ba -v300
echo "== occurrences of hasRenderablePart and image/file rendering =="
rg -n "hasRenderablePart|part.type === 'image'|'value' in part.source|type === 'file'|RenderablePart" "$file"Repository: TanStack/ai
Length of output: 377
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="examples/ts-react-chat/src/routes/index.tsx"
echo "== TypeScript declarations/types around MessagePart =="
rg -n -C 4 "type .*Part|interface .*Part|MessagePart|FileReference|ImagePart" "$file"
echo "== standalone semantic probe from source text =="
python3 - <<'PY'
from pathlib import Path
p = Path("examples/ts-react-chat/src/routes/index.tsx")
text = p.read_text()
checks = {
"hasRenderablePart_exists": "function hasRenderablePart" in text or "const hasRenderablePart" in text,
"image_unconditional_in_hasRenderablePart": "if (part.type === 'image') return true" in text,
"image_conditional_with_value": "if (part.type === 'image' && 'value' in part.source) return true" in text,
"image_render_guards_value": "if (part.type === 'image' && 'value' in part.source)" in text,
}
for k, v in checks.items():
print(f"{k}={v}")
PYRepository: TanStack/ai
Length of output: 918
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="examples/ts-react-chat/src/routes/index.tsx"
echo "== relevant source slices =="
sed -n '260,375p' "$file"
echo
sed -n '375,500p' "$file"
echo "== type declarations around ContentPart / file/image =="
sed -n '20,35p' "$file"
rg -n "ContentPart|messageParts|Image.*Source|File.*Source|source" "$file"Repository: TanStack/ai
Length of output: 10571
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate messageParts/visible messages helpers =="
rg -n "messageParts|visibleMessages|hasRenderablePart|ContentPart" examples/ts-react-chat/src/routes/index.tsx
echo "== source slices around helpers/usages =="
sed -n '1,120p' examples/ts-react-chat/src/routes/index.tsx
echo
sed -n '540,590p' examples/ts-react-chat/src/routes/index.tsxRepository: TanStack/ai
Length of output: 5310
Align message visibility with the image rendering guard.
hasRenderablePart treats every image part as visible, but the image render branch only produces an <img> when part.source has a value. A message containing only an image part without a value returns the surrounding container with no content. Require a value in hasRenderablePart, or render a placeholder for missing image sources.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/ts-react-chat/src/routes/index.tsx` around lines 337 - 339, Update
hasRenderablePart to count an image as renderable only when its source contains
a value, matching the guard in the image rendering branch. Preserve visibility
for other supported part types and keep the existing <img> rendering behavior
unchanged.
Comment on lines
+115
to
+139
| /** | ||
| * Upload each inline (base64 `data`) image input to the provider's Files API and | ||
| * swap in a `{ type: 'file' }` handle. A reference image / start frame is then | ||
| * uploaded once via the tree-shakeable files adapter (`geminiFiles()` / | ||
| * `falFiles()`) instead of being re-sent inline as base64 on the generation | ||
| * request — the memory-safe path for large inputs. URL and already-uploaded | ||
| * sources pass through untouched. | ||
| */ | ||
| async function uploadInlineImageInputs( | ||
| prompt: string | Array<TextPart | ImagePart<MediaInputMetadata>>, | ||
| files: FilesAdapter, | ||
| ): Promise<string | Array<TextPart | ImagePart<MediaInputMetadata>>> { | ||
| if (typeof prompt === 'string') return prompt | ||
| return Promise.all( | ||
| prompt.map(async (part) => { | ||
| if (part.type !== 'image' || part.source.type !== 'data') return part | ||
| const handle = await uploadFile({ | ||
| adapter: files, | ||
| input: { data: part.source.value, mimeType: part.source.mimeType }, | ||
| }) | ||
| return { ...part, source: fileSourceFromHandle(handle) } | ||
| }), | ||
| ) | ||
| } | ||
|
|
Contributor
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== repo files matching relevant names =="
git ls-files | rg '(^|/)server-functions\.ts$|(^|/)ImageGenerator\.tsx$|(^|/)files-api\.md$' || true
echo
echo "== server-functions outline/lines =="
if [ -f examples/ts-react-media/src/lib/server-functions.ts ]; then
wc -l examples/ts-react-media/src/lib/server-functions.ts
sed -n '1,220p' examples/ts-react-media/src/lib/server-functions.ts | nl -ba
fi
echo
echo "== ImageGenerator relevant lines =="
if [ -f examples/ts-react-media/src/components/ImageGenerator.tsx ]; then
wc -l examples/ts-react-media/src/components/ImageGenerator.tsx
sed -n '1,280p' examples/ts-react-media/src/components/ImageGenerator.tsx | nl -ba
fi
echo
echo "== files-api doc relevant lines =="
if [ -f docs/advanced/files-api.md ]; then
wc -l docs/advanced/files-api.md
sed -n '1,150p' docs/advanced/files-api.md | nl -ba
fi
echo
echo "== uploadFile and fileSourceFromHandle references =="
rg -n "uploadFile|fileSourceFromHandle|geminiFiles\(|falFiles\(|anthropicFiles\(|uploadInlineImageInputs|uploaded_files|files" examples/ts-react-media docs -S || trueRepository: TanStack/ai
Length of output: 502
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== lock/package refs for ai sdk versions =="
for f in package.json examples/ts-react-media/package.json pnpm-lock.yaml package-lock.json yarn.lock bun.lockb; do
if [ -f "$f" ]; then
echo "--- $f ---"
if [ "${f##*.}" = "json" ]; then
sed -n '1,220p' "$f"
else
rg -n "ai/|`@ai-sdk/`|versions:|typescript" "$f" | sed -n '1,220p'
fi
fi
done
echo
echo "== TypeScript dependency declarations =="
rg -n '"`@ai-sdk/`|packageManager|typescript|vite' package.json examples/ts-react-media/package.json 2>/dev/null || trueRepository: TanStack/ai
Length of output: 26695
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== server-functions relevant sections =="
sed -n '90,170p' examples/ts-react-media/src/lib/server-functions.ts
echo "----"
sed -n '1,140p' examples/ts-react-media/src/lib/server-functions.ts
echo
echo "== ImageGenerator relevant sections =="
sed -n '150,230p' examples/ts-react-media/src/components/ImageGenerator.tsx
echo "----"
sed -n '1,120p' examples/ts-react-media/src/components/ImageGenerator.tsx
echo
echo "== files-api doc relevant sections =="
sed -n '70,130p' docs/advanced/files-api.md
echo "----"
sed -n '1,60p' docs/advanced/files-api.md
echo
echo "== handle/upload related source references with context =="
rg -n "uploadInlineImageInputs|uploadFile|fileSourceFromHandle|stored.*file|persist|upload.*once|uploaded.*files|uploaded_files|downloadFile|deleteFile|deleteFiles|FilesAdapter|FileHandle|uploaded" \
examples/ts-react-media/src examples/ts-react-media -g '!**/node_modules/**' -S || trueRepository: TanStack/ai
Length of output: 26084
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files-api.md lines 85-115 =="
sed -n '85,115p' docs/advanced/files-api.md
echo
echo "== server-functions generate image/video reference handling around uploadInlineImageInputs =="
sed -n '210,260p' examples/ts-react-media/src/lib/server-functions.ts
echo "----"
sed -n '390,455p' examples/ts-react-media/src/lib/server-functions.ts
echo
echo "== media helpers around toImagePart =="
fd -a 'media' examples/ts-react-media/src/lib | sed 's#^\./##'
rg -n "readMediaFile|toImagePart|AttachedMedia|dataUrl" examples/ts-react-media/src/lib -SRepository: TanStack/ai
Length of output: 6103
Reuse uploaded handles instead of uploading inline data for every request.
uploadInlineImageInputs() uploads a new provider file on each request and then discards the handle. Store the handle, send fileSourceFromHandle(handle) for later requests, and clean up temporary uploads if a request fails.
📍 Affects 3 files
examples/ts-react-media/src/lib/server-functions.ts#L115-L139(this comment)examples/ts-react-media/src/components/ImageGenerator.tsx#L185-L186docs/advanced/files-api.md#L92-L110
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/ts-react-media/src/lib/server-functions.ts` around lines 115 - 139,
The uploadInlineImageInputs flow in
examples/ts-react-media/src/lib/server-functions.ts must cache uploaded handles
per inline image, reuse fileSourceFromHandle(handle) on later requests, and
remove temporary uploads when a request fails. Update
examples/ts-react-media/src/components/ImageGenerator.tsx:185-186 to preserve
and pass the reusable handles, and update docs/advanced/files-api.md:92-110 to
document the reuse and failure-cleanup behavior.
Comment on lines
+71
to
+76
| export function createAnthropicFiles( | ||
| apiKey: string, | ||
| config?: Omit<AnthropicFilesConfig, 'apiKey'>, | ||
| ): AnthropicFilesAdapter { | ||
| return new AnthropicFilesAdapter({ apiKey, ...config }) | ||
| } |
Contributor
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Keep explicit API keys authoritative in every provider factory.
Each factory spreads config after its explicit apiKey argument. A config.apiKey value can redirect uploads to another account.
packages/ai-anthropic/src/adapters/files.ts#L71-L76: change{ apiKey, ...config }to{ ...config, apiKey }.packages/ai-gemini/src/adapters/files.ts#L69-L74: change{ apiKey, ...config }to{ ...config, apiKey }.packages/ai-openai/src/adapters/files.ts#L72-L77: change{ apiKey, ...config }to{ ...config, apiKey }.
📍 Affects 3 files
packages/ai-anthropic/src/adapters/files.ts#L71-L76(this comment)packages/ai-gemini/src/adapters/files.ts#L69-L74packages/ai-openai/src/adapters/files.ts#L72-L77
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/ai-anthropic/src/adapters/files.ts` around lines 71 - 76, Ensure the
explicit apiKey argument remains authoritative in every provider factory by
spreading config before it: update createAnthropicFiles in
packages/ai-anthropic/src/adapters/files.ts (lines 71-76), the corresponding
factory in packages/ai-gemini/src/adapters/files.ts (lines 69-74), and the
corresponding factory in packages/ai-openai/src/adapters/files.ts (lines 72-77)
to construct adapters with config first and apiKey last.
Comment on lines
+148
to
+178
| it('errors when a foreign provider file handle reaches the openai adapter', async () => { | ||
| const create = vi.fn().mockResolvedValueOnce(mockResponsesStream()) | ||
| const adapter = withMockClient(create) | ||
|
|
||
| const chunks: Array<StreamChunk> = [] | ||
| for await (const chunk of chat({ | ||
| adapter, | ||
| messages: [ | ||
| { | ||
| role: 'user', | ||
| content: [ | ||
| { | ||
| type: 'image', | ||
| source: { | ||
| type: 'file', | ||
| reference: { gemini: 'files/gemini-xyz' }, | ||
| }, | ||
| }, | ||
| ], | ||
| }, | ||
| ], | ||
| })) { | ||
| chunks.push(chunk) | ||
| } | ||
|
|
||
| const runError = chunks.find((c) => c.type === 'RUN_ERROR') | ||
| expect(runError).toBeDefined() | ||
| if (runError?.type === 'RUN_ERROR') { | ||
| expect(runError.message).toMatch(/openai/) | ||
| } | ||
| }) |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Assert that the client is not called.
This test verifies the error only. It does not verify the fail-closed requirement that the OpenAI client receives no request for a foreign handle.
Proposed fix
if (runError?.type === 'RUN_ERROR') {
expect(runError.message).toMatch(/openai/)
}
+ expect(create).not.toHaveBeenCalled()
})📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| it('errors when a foreign provider file handle reaches the openai adapter', async () => { | |
| const create = vi.fn().mockResolvedValueOnce(mockResponsesStream()) | |
| const adapter = withMockClient(create) | |
| const chunks: Array<StreamChunk> = [] | |
| for await (const chunk of chat({ | |
| adapter, | |
| messages: [ | |
| { | |
| role: 'user', | |
| content: [ | |
| { | |
| type: 'image', | |
| source: { | |
| type: 'file', | |
| reference: { gemini: 'files/gemini-xyz' }, | |
| }, | |
| }, | |
| ], | |
| }, | |
| ], | |
| })) { | |
| chunks.push(chunk) | |
| } | |
| const runError = chunks.find((c) => c.type === 'RUN_ERROR') | |
| expect(runError).toBeDefined() | |
| if (runError?.type === 'RUN_ERROR') { | |
| expect(runError.message).toMatch(/openai/) | |
| } | |
| }) | |
| it('errors when a foreign provider file handle reaches the openai adapter', async () => { | |
| const create = vi.fn().mockResolvedValueOnce(mockResponsesStream()) | |
| const adapter = withMockClient(create) | |
| const chunks: Array<StreamChunk> = [] | |
| for await (const chunk of chat({ | |
| adapter, | |
| messages: [ | |
| { | |
| role: 'user', | |
| content: [ | |
| { | |
| type: 'image', | |
| source: { | |
| type: 'file', | |
| reference: { gemini: 'files/gemini-xyz' }, | |
| }, | |
| }, | |
| ], | |
| }, | |
| ], | |
| })) { | |
| chunks.push(chunk) | |
| } | |
| const runError = chunks.find((c) => c.type === 'RUN_ERROR') | |
| expect(runError).toBeDefined() | |
| if (runError?.type === 'RUN_ERROR') { | |
| expect(runError.message).toMatch(/openai/) | |
| } | |
| expect(create).not.toHaveBeenCalled() | |
| }) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/ai-openai/tests/files-source.test.ts` around lines 148 - 178, Update
the foreign-provider file-handle test around chat and the create mock to assert
that the OpenAI client is never called. Keep the existing RUN_ERROR and
OpenAI-message assertions, and add a call-count expectation on the mocked create
function after consuming the stream.
Comment on lines
+1339
to
+1345
| // Fail closed on `{ type: 'file' }` sources for adapters that haven't | ||
| // declared support — an adapter written before the file arm existed would | ||
| // otherwise fall through to its URL/data branch and silently mis-map the | ||
| // reference. Checked per model call so tool results added mid-loop are | ||
| // covered too. | ||
| assertMessagesFileSourceSupport(this.adapter, this.messages) | ||
|
|
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Validate file sources before structured-output calls.
Line 1344 validates only chatStream calls. A schema-only request can skip this path and call structuredOutputStream or structuredOutput from runStructuredFinalization with an unsupported file source.
Run assertMessagesFileSourceSupport(this.adapter, this.messages) after structured-output middleware configuration and before the provider call.
Proposed fix
this.applyMiddlewareConfig(postOnConfig)
+ assertMessagesFileSourceSupport(this.adapter, this.messages)
// Build the StructuredOutputOptions the adapter expects.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| // Fail closed on `{ type: 'file' }` sources for adapters that haven't | |
| // declared support — an adapter written before the file arm existed would | |
| // otherwise fall through to its URL/data branch and silently mis-map the | |
| // reference. Checked per model call so tool results added mid-loop are | |
| // covered too. | |
| assertMessagesFileSourceSupport(this.adapter, this.messages) | |
| this.applyMiddlewareConfig(postOnConfig) | |
| assertMessagesFileSourceSupport(this.adapter, this.messages) | |
| // Build the StructuredOutputOptions the adapter expects. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/ai/src/activities/chat/index.ts` around lines 1339 - 1345, Validate
file-source support in runStructuredFinalization after structured-output
middleware configuration and immediately before each structured-output provider
call, including structuredOutputStream and structuredOutput. Reuse
assertMessagesFileSourceSupport(this.adapter, this.messages) so schema-only
requests fail closed for adapters without file-source support.
Comment on lines
+285
to
+288
| // Fail closed on `{ type: 'file' }` sources for adapters that haven't | ||
| // declared support (see assertPromptFileSourceSupport). | ||
| assertPromptFileSourceSupport(adapter, rest.prompt) | ||
|
|
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Route preflight failures through the generation error lifecycle.
Line 287 throws after runGenerationStart but before the try block at Line 318. Unsupported file sources therefore bypass runGenerationError and logger.errors.
Start the existing try block before this preflight, so every started generation receives a matching error callback.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/ai/src/activities/generateImage/index.ts` around lines 285 - 288,
Move the existing try block in the generation flow to begin before
assertPromptFileSourceSupport(adapter, rest.prompt), while keeping the preflight
validation inside that try. Ensure failures after runGenerationStart are routed
through the existing runGenerationError and logger.errors handling.
Comment on lines
279
to
286
| /** | ||
| * Source specification for multimodal content. | ||
| * Discriminated union supporting both inline data (base64) and URL-based content. | ||
| * Discriminated union supporting inline data (base64), URL-based content, and | ||
| * provider-issued file handles. | ||
| * - For 'data' sources: mimeType is required | ||
| * - For 'url' sources: mimeType is optional | ||
| * - For 'file' sources: a provider-issued handle plus its issuing `provider` | ||
| */ |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Correct the file source description.
Line 285 describes a singular issuing provider, but ContentPartFileSource contains a provider-keyed reference record. This can lead callers to construct an invalid source shape. Describe the source as a provider-keyed reference record instead.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/ai/src/types.ts` around lines 279 - 286, Update the documentation
for ContentPartFileSource in the multimodal content source specification to
describe reference as a provider-keyed record, rather than a singular issuing
provider. Keep the descriptions for data and url sources unchanged.
| * Structural check for a single `ContentPart`. A text part must carry a string | ||
| * `content`; every other modality must carry a `source` with `type` of | ||
| * `'url' | 'data'` and a string `value`. | ||
| * `'url' | 'data' | 'file'` and a string `value`. |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the declared Zod dependency and existing validation conventions.
fd -a '^package\.json$' . -x rg -n '"zod"' {}
rg -n -C 3 'z\.record|safeParse|isContentPart' packages/ai/src packages/ai/testsRepository: TanStack/ai
Length of output: 992
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== tool-result.ts outline =="
ast-grep outline packages/ai/src/utilities/tool-result.ts --view expanded || true
echo "== tool-result.ts relevant lines =="
cat -n packages/ai/src/utilities/tool-result.ts
echo "== related file validation references =="
rg -n -C 3 'isContentPart|file.*reference|FileSource|source' packages/ai/src packages/ai/tests || true
echo "== zod declarations =="
for f in package.json packages/ai/package.json; do
if [ -f "$f" ]; then
echo "--- $f ---"
sed -n '1,90p' "$f" | rg -n -C 2 '"zod"|dependencies|devDependencies'
fi
done
echo "== deterministic object check behavior =="
node - <<'JS'
for (const reference of [['file-abc'], ['file-abc', 'file-def'], [''], [], { provider: 'file-abc' }]) {
console.log(JSON.stringify(reference), 'object=', typeof reference === 'object', 'array=', Array.isArray(reference));
}
JSRepository: TanStack/ai
Length of output: 50368
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== package manager and zod availability =="
node - <<'JS'
const { existsSync } = require('node:fs')
for (const p of ['node_modules/zod/package.json', 'packages/ai/node_modules/zod/package.json', 'packages/ai-event-client/node_modules/zod/package.json']) {
console.log(p, existsSync(p))
}
if (existsSync('node_modules/zod/package.json')) {
const pkg = require('node:fs').readFileSync('node_modules/zod/package.json', 'utf8')
console.log(JSON.parse(pkg).version)
}
JS
echo "== fileSource helpers outline =="
cd packages/ai/src && ast-grep outline utilities/file-source.ts --view expanded || true
cd - >/dev/null
echo "== file-source.ts relevant lines =="
cat -n packages/ai/src/utilities/file-source.ts
echo "== focused file-source tests around contract mentions =="
sed -n '1,160p' packages/ai/tests/files-source.test.tsRepository: TanStack/ai
Length of output: 564
Reject array references for file sources.
typeof reference === 'object' accepts arrays such as reference: ['file-abc'], but file sources require a provider-to-reference record. Add !Array.isArray(reference) and validate the record values with Zod instead of using Object.values() over an untrusted array value. Update the docstring to describe a non-empty reference record, not a string value.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/ai/src/utilities/tool-result.ts` at line 14, The file-source
validation must reject array references and only accept a non-empty
provider-to-reference record. Update the relevant type guard/schema in
tool-result handling to require !Array.isArray(reference) and validate record
values with Zod rather than Object.values() on untrusted input; revise the
nearby docstring to describe a non-empty reference record instead of a string
value.
Source: Coding guidelines
Comment on lines
+25
to
+29
| const provider = (url.searchParams.get('provider') ?? | ||
| 'openai') as Provider | ||
| const handleProvider = | ||
| url.searchParams.get('handleProvider') ?? provider | ||
| const testId = url.searchParams.get('testId') ?? undefined |
Contributor
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Validate the file-source wire HTTP contract with Zod.
provider is currently an unchecked cast, so an invalid query value can reach createTextAdapter and fail outside the route error handler. Parse provider, handleProvider, and testId with Zod and return a 400 response for invalid values. Parse each E2E response with a Zod result schema instead of type assertions.
testing/e2e/src/routes/api.file-source-wire.ts#L25-L29: validate query parameters before adapter creation.testing/e2e/tests/file-source-wire.spec.ts#L36-L36: validate the OpenAI response.testing/e2e/tests/file-source-wire.spec.ts#L54-L54: validate the OpenAI rejection response.testing/e2e/tests/file-source-wire.spec.ts#L67-L67: validate the Anthropic response.testing/e2e/tests/file-source-wire.spec.ts#L82-L82: validate the Gemini response.testing/e2e/tests/file-source-wire.spec.ts#L93-L93: validate the Gemini rejection response.
As per coding guidelines, “Use Zod for schema validation.”
📍 Affects 2 files
testing/e2e/src/routes/api.file-source-wire.ts#L25-L29(this comment)testing/e2e/tests/file-source-wire.spec.ts#L36-L36testing/e2e/tests/file-source-wire.spec.ts#L54-L54testing/e2e/tests/file-source-wire.spec.ts#L67-L67testing/e2e/tests/file-source-wire.spec.ts#L82-L82testing/e2e/tests/file-source-wire.spec.ts#L93-L93
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@testing/e2e/src/routes/api.file-source-wire.ts` around lines 25 - 29,
Validate the file-source wire HTTP contract with Zod: in
testing/e2e/src/routes/api.file-source-wire.ts lines 25-29, parse provider,
handleProvider, and testId before adapter creation and return HTTP 400 for
invalid input; in testing/e2e/tests/file-source-wire.spec.ts lines 36, 54, 67,
82, and 93, replace response type assertions with the appropriate Zod result
schema parsing for each OpenAI, Anthropic, and Gemini success or rejection
response.
Source: Coding guidelines
Comment on lines
+71
to
+76
| if ( | ||
| chunk.type === 'RUN_ERROR' && | ||
| /file/.test(chunk.message ?? '') | ||
| ) { | ||
| runError = chunk.message | ||
| } |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Treat every RUN_ERROR as a route failure.
If a provider request fails with an error message that does not contain file, this route returns { ok: true }. Set runError for every RUN_ERROR chunk.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@testing/e2e/src/routes/api.file-source-wire.ts` around lines 71 - 76, Update
the RUN_ERROR handling in the route’s chunk-processing logic to assign runError
for every chunk with type RUN_ERROR, regardless of whether its message contains
“file”; preserve the existing message value for the failure response.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #909.
Adds first-class support for provider Files / storage APIs so callers can upload media once and reference it by a provider-issued handle instead of re-sending base64 (or relying on the provider to re-fetch a public URL) each request — lower latency/bandwidth, no re-buffering of base64 on memory-constrained runtimes (the #907 OOM class), and access to provider-side file lifecycle.
What's included
1. Tree-shakeable
filesadapter kind (packages/ai/src/activities/files/)openaiFiles(),anthropicFiles(),geminiFiles(),falFiles()— each withupload(), plusget()/delete()where the provider has a lifecycle API. fal storage is upload-only (getFile/deleteFilethrow a clear error for it).uploadFile()/getFile()/deleteFile()activity dispatchers. Handles are provider-literal typed (FileHandle<'openai'>), and the lifecycle calls accept the handle itself — a cross-provider lifecycle call is a compile error.2. New
{ type: 'file' }arm onContentPartSource— a per-provider reference record{ type: 'file', reference: { openai: 'file-…', gemini: 'https://…' } }. Each adapter reads only its own entry and maps it to its native wire field: OpenAI Responsesinput_image/input_filefile_id, Anthropicfile_idsource (auto-sends thefiles-api-2025-04-14beta), GeminifileData.fileUri, fal storage URL.fileSourceFromHandle(...handles)builds the source and merges handles from several providers — upload the same bytes to each, and one source routes the same conversation to any of them. A supporting adapter with no entry for its provider throws a lookup error naming the providers that are present.fileSourceFromHandle+FileHandleare also exported from the browser-safe@tanstack/ai/cliententry.3. Fail-closed capability preflight (chosen over per-adapter convention — see below)
supportsFileSources;chat()/generateImage()/generateVideo()reject file sources for every other adapter (Bedrock, Mistral, Grok, Groq, OpenRouter, Ollama, BytePlus, and any future adapter that doesn't opt in) before a request is built.input_reference, Gemini Veo, Chat Completions images — throw endpoint-specific errors.4. Docs, skills, and a worked example
docs/advanced/files-api.mdguide (incl. the multi-provider merge pattern) + a "File Handle" section in the multimodal-content page; agent skills updated (adapter-configuration§7, chat-experience, media-generation).examples/ts-react-mediauploads reference images (Gemini) and image-to-video start frames (fal) once viauploadFile()and references them by handle.Notes / decisions
{ value, provider }pair. A scalar handle makes cross-provider use an error to police at every mapping site; a record makes routing self-describing (each adapter looks up its own key), composes across providers (merge + replay), and turns a wrong-provider send into an inherent lookup miss.source.typediscrimination site is fragile by construction — an adapter added later (BytePlus proved this mid-review) silently falls through to its URL/data branch. ThesupportsFileSourcesgate runs at the activity layer (same place modality is validated), so unsupported adapters fail closed by default; the in-adapter guards remain as backstops. Removingvaluefrom the file arm also makes any leftover fall-through a compile error (it caught two latent OpenRouter paths).client.beta.messages.createalready, so consumingfile_idsources meant switching that adapter's content block types to the Beta param variants and auto-adding the Files beta when a file source is present.@google/genai's resumable upload sets an explicitContent-Lengthon a Blob-body request, which older Nitro runtimes reject ("invalid content-length header" / "fetch failed"). Verified fixed onnitro@3.0.260610-beta(the version the example pins);docs/advanced/files-api.mddocuments the requirement.supports.filescapability typing.Testing
toFileHandlenormalizers (seconds→ms expiry, ISO parse, missing-name throw), preflight rejection for Grok/BytePlus.testing/e2e/tests/file-source-wire.spec.ts): round-trip per issuer against aimock + cross-provider lookup-miss rejection end-to-end.test:pr(sherif, knip, docs, kiira, oxlint, lib, types, build across all packages, examples, and testing/) and the full e2e suite (447 passed). The only failing tasks on this machine (ai-grok-builddurability-warning tests under nx,ai-sandbox-local-processreaper conformance) fail identically on cleanmain— pre-existing, unrelated.🤖 Generated with Claude Code
Summary by CodeRabbit