Skip to content

fix: ensure vision-capable models for image attachments #388

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ngoiyaeric merged 2 commits into from
Dec 30, 2025
Merged

fix: ensure vision-capable models for image attachments #388

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f89ae98
fix: ensure vision-capable models for image attachments
ngoiyaeric Dec 30, 2025
8ec42ae
Merge branch 'main' into fix/image-attachment-tokens
ngoiyaeric Dec 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions app/actions.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -250,8 +250,9 @@ async function submit(formData?: FormData, skip?: boolean) {
}

const hasImage = messageParts.some(part => part.type === 'image')
const content = hasImage
? (messageParts as any)
// Properly type the content based on whether it contains images
const content: CoreMessage['content'] = hasImage
? messageParts as CoreMessage['content']
: messageParts.map(part => part.text).join('\n')
Comment on lines 252 to 256
Copy link
Copy Markdown

@charliecreates charliecreates Bot Dec 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

content is typed as CoreMessage['content'], but the code only checks hasImage to decide whether to send an array vs. a joined string. This can silently produce an invalid payload when non-image non-text parts exist (e.g., future file, audio, tool parts) or if a text part is missing .text. In that case, the non-image branch will do part.text and potentially drop/flatten data.

Given the stated goal (multimodal correctness), it’s safer to build content by validating/normalizing parts and only joining when you’re sure all parts are text.

Suggestion

Tighten the branching to ensure the string path only runs when every part is text, otherwise keep the array form:

const isAllText = messageParts.every(p => p.type === 'text')
const content: CoreMessage['content'] = isAllText
  ? messageParts.map(p => p.text).join('\n')
  : (messageParts as CoreMessage['content'])

This avoids lossy conversion if additional part types are introduced later. Reply with "@CharlieHelps yes please" if you'd like me to add a commit with this change.


const type = skip
Expand All @@ -278,7 +279,7 @@ async function submit(formData?: FormData, skip?: boolean) {
messages.push({
role: 'user',
content
})
} as CoreMessage)
}

const userId = 'anonymous'
Expand Down
8 changes: 7 additions & 1 deletion lib/agents/researcher.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,8 +96,14 @@ export async function researcher(
? dynamicSystemPrompt
: getDefaultSystemPrompt(currentDate)

// Check if any message contains an image
const hasImage = messages.some(message =>
Array.isArray(message.content) &&
message.content.some(part => part.type === 'image')
)

const result = await nonexperimental_streamText({
model: getModel() as LanguageModel,
model: getModel(hasImage) as LanguageModel,
maxTokens: 4096,
system: systemPromptToUse,
messages,
Expand Down
8 changes: 7 additions & 1 deletion lib/agents/resolution-search.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,15 @@ Analyze the user's prompt and the image to provide a holistic understanding of t

const filteredMessages = messages.filter(msg => msg.role !== 'system');

// Check if any message contains an image (resolution search is specifically for image analysis)
const hasImage = messages.some(message =>
Array.isArray(message.content) &&
message.content.some(part => part.type === 'image')
)
Comment on lines +42 to +46
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Dec 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Consider extracting duplicate image detection logic.

This exact image detection pattern appears in both lib/agents/researcher.tsx (lines 99-103) and here. Consider extracting it to a shared utility function to maintain consistency and reduce duplication.

🔎 Example helper function

Add to lib/utils/index.ts:

+export function hasImageContent(messages: CoreMessage[]): boolean {
+  return messages.some(message => 
+    Array.isArray(message.content) && 
+    message.content.some(part => part.type === 'image')
+  )
+}

Then use it in both files:

-  const hasImage = messages.some(message => 
-    Array.isArray(message.content) && 
-    message.content.some(part => part.type === 'image')
-  )
+  const hasImage = hasImageContent(messages)

Committable suggestion skipped: line range outside the PR's diff.

🤖 Prompt for AI Agents 
In lib/agents/resolution-search.tsx around lines 42 to 46 the image-detection
logic (checking message.content is an array and any part.type === 'image') is
duplicated elsewhere (lib/agents/researcher.tsx lines 99-103); extract this into
a shared utility (e.g., export function hasImage(messages: Message[]): boolean)
placed in lib/utils/index.ts, implement the same array-and-part-type checks once
in that function, export it, then replace the inline detection in both files
with an import of the new hasImage utility and call it (adjust imports/typing
accordingly).


// Use generateObject to get the full object at once.
const { object } = await generateObject({
model: getModel(),
model: getModel(hasImage),
Comment on lines +42 to +50
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Dec 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Consider validating that images are actually present.

Since resolutionSearch is specifically designed for satellite image analysis (per the system prompt), consider adding a validation check to ensure images are present before proceeding. This would provide clearer error messages if called incorrectly.

🔎 Suggested validation 
   const hasImage = messages.some(message => 
     Array.isArray(message.content) && 
     message.content.some(part => part.type === 'image')
   )
+
+  if (!hasImage) {
+    throw new Error('resolutionSearch requires at least one image in the messages')
+  }

   const { object } = await generateObject({
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// Check if any message contains an image (resolution search is specifically for image analysis)
const hasImage = messages.some(message =>
Array.isArray(message.content) &&
message.content.some(part => part.type === 'image')
)
// Use generateObject to get the full object at once.
const { object } = await generateObject({
model: getModel(),
model: getModel(hasImage),
// Check if any message contains an image (resolution search is specifically for image analysis)
const hasImage = messages.some(message =>
Array.isArray(message.content) &&
message.content.some(part => part.type === 'image')
)
if (!hasImage) {
throw new Error('resolutionSearch requires at least one image in the messages')
}
// Use generateObject to get the full object at once.
const { object } = await generateObject({
model: getModel(hasImage),
🤖 Prompt for AI Agents 
In lib/agents/resolution-search.tsx around lines 42 to 50, the code only detects
image parts but does not halt when none exist; add an explicit validation after
computing hasImage that checks if hasImage is false and then return or throw a
clear, contextual error (e.g., "resolutionSearch requires at least one image in
messages") so the function short-circuits before calling
generateObject/getModel; ensure the validation uses the same
Array.isArray(message.content) && message.content.some(...) check and that the
error path provides a helpful message for callers and logs where appropriate.

Comment on lines 40 to +50
Copy link
Copy Markdown

@charliecreates charliecreates Bot Dec 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

filteredMessages is computed, but hasImage is checked against messages (unfiltered). If a system message ever contains image parts (or the caller passes different arrays), you could incorrectly select a vision model when it’s not needed, or vice versa.

For correctness and consistency, compute hasImage from the actual messages array you pass to the model (filteredMessages).

Suggestion

Base the check on filteredMessages:

const hasImage = filteredMessages.some(message =>
  Array.isArray(message.content) &&
  message.content.some(part => part.type === 'image')
)

Reply with "@CharlieHelps yes please" if you'd like me to add a commit with this fix.

system: systemPrompt,
messages: filteredMessages,
schema: resolutionSearchSchema,
Expand Down
9 changes: 5 additions & 4 deletions lib/utils/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,16 @@ export function generateUUID(): string {
return uuidv4();
}

export function getModel() {
export function getModel(requireVision: boolean = false) {
const xaiApiKey = process.env.XAI_API_KEY
const gemini3ProApiKey = process.env.GEMINI_3_PRO_API_KEY
const awsAccessKeyId = process.env.AWS_ACCESS_KEY_ID
const awsSecretAccessKey = process.env.AWS_SECRET_ACCESS_KEY
const awsRegion = process.env.AWS_REGION
const bedrockModelId = ''
const bedrockModelId = process.env.BEDROCK_MODEL_ID || 'anthropic.claude-3-5-sonnet-20241022-v2:0'

if (xaiApiKey) {
// If vision is required, skip models that don't support it
if (!requireVision && xaiApiKey) {
Comment on lines +27 to +28
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Dec 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Good vision-gating logic.

Correctly skips xAI when vision is required, as xAI models don't support multimodal inputs.

Optional: Consider clarifying the comment 
-  // If vision is required, skip models that don't support it
+  // If vision is required, skip xAI (no vision support) and try vision-capable providers
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// If vision is required, skip models that don't support it
if (!requireVision && xaiApiKey) {
// If vision is required, skip xAI (no vision support) and try vision-capable providers
if (!requireVision && xaiApiKey) {
🤖 Prompt for AI Agents 
In lib/utils/index.ts around lines 27 to 28, the existing comment "If vision is
required, skip models that don't support it" is misleading given the
conditional; update the comment to accurately describe the logic (e.g., explain
that when vision is not required and an xAI API key is present, xAI-only models
are considered, otherwise xAI is skipped) so future readers understand the
condition and intent.

const xai = createXai({
apiKey: xaiApiKey,
baseURL: 'https://api.x.ai/v1',
Expand Down Expand Up @@ -67,7 +68,7 @@ export function getModel() {
return model
}

// Default fallback (OpenAI)
// Default fallback (OpenAI gpt-4o supports vision)
const openai = createOpenAI({
apiKey: process.env.OPENAI_API_KEY,
})
Expand Down