fix(ai-gemini): align createGeminiImage URL handling with chat — stop fetching arbitrary HTTPS URLs

[tombeckenham]

Reported by

Thanks to @L4Ph for surfacing this.

Are there plans to support the Files API in functions like createGeminiImage()?

I've observed that running image generation on Cloudflare Workers consistently triggers fetch operations using base64 data, which leads to Out-of-Memory (OOM) errors.

Summary

createGeminiImage() (Gemini-native image models using generateContent) fetches arbitrary https:// URL image inputs locally and re-encodes them as base64 inlineData. This differs from createGeminiChat(), which passes URL sources through as fileData.fileUri without fetching.

On memory-constrained runtimes (e.g. Cloudflare Workers), the fetch + double-buffering path can cause OOM.

Current behavior

In packages/ai-gemini/src/adapters/image.ts, imagePartToGeminiPart:

Input	Behavior
source.type: 'data'	inlineData — no fetch
gs://... or https://generativelanguage.googleapis.com/...	fileData.fileUri passthrough — no fetch
Any other https://... URL	fetch() → blob → arrayBuffer → base64 inlineData

The same conservative pattern exists in the Veo video adapter (imagePartToVeoImage in packages/ai-gemini/src/adapters/video.ts).

Expected / consistent behavior

The Gemini chat adapter already passes all URL sources through:

return {
  fileData: {
    fileUri: part.source.value,
    mimeType: part.source.mimeType ?? defaultMimeType,
  },
}

(packages/ai-gemini/src/adapters/text.ts)

Google’s docs also describe passing public HTTPS URLs and letting Gemini fetch server-side (External HTTP / Signed URLs). PR #624 introduced the image-gen multimodal prompt path with an explicit triage of fileData vs inlineData vs fetch+inline — likely modeled on legacy generateContent cookbook examples that fetch URLs client-side, rather than matching chat.

Impact

• Workers / edge: OOM when image-conditioned generation uses type: 'url' with public HTTPS URLs
• Inconsistency: Same ImagePart shape behaves differently in chat vs generateImage
• Workaround today: Pre-upload via Files API and pass the generativelanguage.googleapis.com/... URI, or use type: 'data' (still holds base64 in memory)

Proposed fix

1. Align imagePartToGeminiPart with the text adapter: pass arbitrary public HTTPS URLs as fileData.fileUri (no local fetch), keeping inlineData for type: 'data'.
2. Consider the same change for imagePartToVeoImage where applicable.
3. Update tests in packages/ai-gemini/tests/image-adapter.test.ts (the test "fetches arbitrary URL sources and inlines them as base64" would flip to passthrough).
4. Optional follow-up: automatic Files API upload helper for large inputs (separate from URL passthrough).

Files

• packages/ai-gemini/src/adapters/image.ts — imagePartToGeminiPart
• packages/ai-gemini/src/adapters/text.ts — reference implementation
• packages/ai-gemini/src/adapters/video.ts — same pattern for Veo
• packages/ai-gemini/tests/image-adapter.test.ts

Related

• Introduced in feat: multimodal prompt for generateImage/generateVideo (image-to-image, image-to-video) #624 (multimodal prompt for generateImage / generateVideo)
• Gap analysis notes Files API as a broader Gemini gap (.agent/gap-analysis/)