Skip to content

Implement intent-driven object detection and iterative zoom #596

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ngoiyaeric wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Implement intent-driven object detection and iterative zoom #596

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
58df5f7
feat: implement intent-driven object detection and iterative zoom
google-labs-jules[bot] May 7, 2026
48ff9d2
feat: fix camera state types and populate map bounds
google-labs-jules[bot] May 7, 2026
b806fa4
feat: implement intent-driven object detection and iterative zoom
google-labs-jules[bot] May 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 29 additions & 11 deletions app/actions.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,14 @@ async function submit(formData?: FormData, skip?: boolean) {
const timezone = (formData?.get('timezone') as string) || 'UTC';
const lat = formData?.get('latitude') ? parseFloat(formData.get('latitude') as string) : undefined;
const lng = formData?.get('longitude') ? parseFloat(formData.get('longitude') as string) : undefined;
const location = (lat !== undefined && lng !== undefined) ? { lat, lng } : undefined;
const boundsString = formData?.get('bounds') as string;
let bounds = undefined;
try {
bounds = boundsString ? JSON.parse(boundsString) : undefined;
} catch (e) {
console.error('Failed to parse bounds:', e);
}
const location = (lat !== undefined && lng !== undefined) ? { lat, lng, bounds } : undefined;

if (!file) {
throw new Error('No file provided for resolution search.');
Expand All @@ -81,7 +88,7 @@ async function submit(formData?: FormData, skip?: boolean) {
message.type !== 'resolution_search_result'
);

const userInput = 'Analyze this map view.';
const userInput = (formData?.get('input') as string) || 'Analyze this map view.';
const content: CoreMessage['content'] = [
{ type: 'text', text: userInput },
{ type: 'image', image: dataUrl, mimeType: file.type }
Expand All @@ -96,32 +103,43 @@ async function submit(formData?: FormData, skip?: boolean) {
});
messages.push({ role: 'user', content });

const summaryStream = createStreamableValue<string>('Analyzing map view...');
const summaryStream = createStreamableValue<string>(`Analyzing: ${userInput}`);
const groupeId = nanoid();

async function processResolutionSearch() {
try {
const streamResult = await resolutionSearch(messages, timezone, drawnFeatures, location);

uiStream.append(
<GeoJsonLayer
id={groupeId}
data={{ type: 'FeatureCollection', features: [] } as FeatureCollection}
/>
);

let fullSummary = '';
for await (const partialObject of streamResult.partialObjectStream) {
if (partialObject.summary) {
fullSummary = partialObject.summary;
summaryStream.update(fullSummary);
if (fullSummary.includes('ZOOM PASSContext') || fullSummary.includes('zoomed-in CROP')) {
summaryStream.update(`${fullSummary}\n\n[Zoom Pass in progress...]`);
}
Comment on lines +125 to +127
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot May 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

'ZOOM PASSContext' is missing a space — this branch can never be entered.

The string 'ZOOM PASSContext' has no space between PASS and Context. The system prompt uses **ZOOM PASS Context:** (with a space), but even so, the AI's summary output is unlikely to reproduce that system-prompt phrase verbatim. The [Zoom Pass in progress...] indicator therefore never appears in the streamed summary.

🐛 Proposed fix 
-            if (fullSummary.includes('ZOOM PASSContext') || fullSummary.includes('zoomed-in CROP')) {
+            if (fullSummary.includes('ZOOM PASS') || fullSummary.includes('zoomed-in crop')) {
               summaryStream.update(`${fullSummary}\n\n[Zoom Pass in progress...]`);
             }

A more reliable alternative is to track zoom-pass state in a flag set when extractRegion is called by the agent, rather than checking for keywords in the AI-generated text.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if (fullSummary.includes('ZOOM PASSContext') || fullSummary.includes('zoomed-in CROP')) {
summaryStream.update(`${fullSummary}\n\n[Zoom Pass in progress...]`);
}
if (fullSummary.includes('ZOOM PASS') || fullSummary.includes('zoomed-in crop')) {
summaryStream.update(`${fullSummary}\n\n[Zoom Pass in progress...]`);
}
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/actions.tsx` around lines 125 - 127, The branch checking fullSummary for
the misspelled literal 'ZOOM PASSContext' will never match (missing space) and
relying on AI-generated text is brittle; instead add and use an explicit
zoom-pass boolean flag (e.g., isZoomPass or zoomInProgress) that is set when
extractRegion is invoked by the agent and cleared when the zoom pass completes,
then change the code path that currently inspects fullSummary (the block calling
summaryStream.update and referencing fullSummary) to consult that flag and call
summaryStream.update(`${fullSummary}\n\n[Zoom Pass in progress...]`) only when
the flag is true.

}
if (partialObject.geoJson) {
uiStream.update(
<GeoJsonLayer
id={groupeId}
data={partialObject.geoJson as FeatureCollection}
/>
);
Comment on lines +129 to +135
Copy link
Copy Markdown
Contributor

@qodo-code-review qodo-code-review Bot May 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

2. Geojson update blanks response 🐞 Bug ≡ Correctness

During streaming, processResolutionSearch() calls uiStream.update(<GeoJsonLayer .../>), which
replaces the already-rendered response section with a component that returns null, making the chat
response disappear.
Agent Prompt 
### Issue description
`uiStream.update(<GeoJsonLayer .../>)` replaces the previously rendered `<Section>` (carousel + BotMessage) with a component that renders `null`, so users lose the visible response while streaming.

### Issue Context
`GeoJsonLayer` is meant to be a side-effect-only component (map overlay), not the message body.

### Fix Focus Areas
- app/actions.tsx[109-137]
- app/actions.tsx[223-233]

### Suggested fix
- Render `GeoJsonLayer` *inside* the same `<Section>` and call `uiStream.update()` with the entire `<Section>` whenever geoJson changes, e.g. a helper `renderResolutionUI({geoJson})`.
- Alternatively, keep a single UI tree and drive GeoJSON updates via a `StreamableValue<FeatureCollection>` prop (if you have a pattern for that), rather than replacing the entire stream with `GeoJsonLayer`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

}
}

const analysisResult = await streamResult.object;
summaryStream.done(analysisResult.summary || 'Analysis complete.');

if (analysisResult.geoJson) {
uiStream.append(
<GeoJsonLayer
id={groupeId}
data={analysisResult.geoJson as FeatureCollection}
/>
);
}


messages.push({ role: 'assistant', content: analysisResult.summary || 'Analysis complete.' });

Expand Down
1 change: 1 addition & 0 deletions bun.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions components/chat.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ export function Chat({ id }: ChatProps) {
if (isMobile) {
return (
<MapDataProvider> {/* Add Provider */}
<HeaderSearchButton />
<HeaderSearchButton input={input} setInput={setInput} />
<div className="mobile-layout-container">
<div className="mobile-map-section">
{activeView ? <SettingsView /> : isUsageOpen ? <UsageView /> : <MapProvider />}
Expand Down Expand Up @@ -184,7 +184,7 @@ export function Chat({ id }: ChatProps) {
// Desktop layout
return (
<MapDataProvider> {/* Add Provider */}
<HeaderSearchButton />
<HeaderSearchButton input={input} setInput={setInput} />
<div className="flex justify-start items-start">
{/* This is the new div for scrolling */}
<div className="w-1/2 flex flex-col space-y-3 md:space-y-4 px-8 sm:px-12 pt-16 md:pt-20 pb-4 h-[calc(100vh-0.5in)] overflow-y-auto">
Expand Down
4 changes: 4 additions & 0 deletions components/followup-panel.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@ export function FollowupPanel() {

// Include drawn features in the form data
formData.append('drawnFeatures', JSON.stringify(mapData.drawnFeatures || []))
const bounds = mapData.cameraState?.bounds;
if (bounds) {
formData.append('bounds', JSON.stringify(bounds))
}
Comment thread
qodo-code-review[bot] marked this conversation as resolved.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

const responseMessage = await submit(formData)
setMessages(currentMessages => [
Expand Down
13 changes: 11 additions & 2 deletions components/header-search-button.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ interface HeaderActions {
submit: (formData: FormData) => Promise<any>;
}

export function HeaderSearchButton() {
export function HeaderSearchButton({ input, setInput }: { input?: string, setInput?: (value: string) => void }) {
const { map } = useMap()
const { mapProvider } = useSettingsStore()
const { mapData } = useMapData()
Expand Down Expand Up @@ -72,7 +72,7 @@ export function HeaderSearchButton() {
...currentMessages,
{
id: nanoid(),
component: <UserMessage content={[{ type: 'text', text: 'Analyze this map view.' }]} />
component: <UserMessage content={input || 'Analyze this map view.'} />
}
])

Expand Down Expand Up @@ -146,6 +146,7 @@ export function HeaderSearchButton() {
// Keep 'file' for backward compatibility if needed, or just use the first available
formData.append('file', (mapboxBlob || googleBlob)!, 'map_capture.png')

if (input) formData.append('input', input)
formData.append('action', 'resolution_search')
formData.append('timezone', mapData.currentTimezone || 'UTC')
formData.append('drawnFeatures', JSON.stringify(mapData.drawnFeatures || []))
Expand All @@ -154,10 +155,18 @@ export function HeaderSearchButton() {
if (center) {
formData.append('latitude', center.lat.toString())
formData.append('longitude', center.lng.toString())
const bounds = mapProvider === 'mapbox' && map ? map.getBounds() : null;
if (bounds) {
formData.append('bounds', JSON.stringify({
sw: { lat: bounds.getSouthWest().lat, lng: bounds.getSouthWest().lng },
ne: { lat: bounds.getNorthEast().lat, lng: bounds.getNorthEast().lng }
}))
}
}
Comment on lines +158 to 165
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot May 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Bounds are not captured for the Google Maps provider.

For mapProvider === 'google', no bounds field is appended to formData. As a result, location.bounds is always undefined in resolution-search.tsx for Google Maps users. The zoom-pass coordinate remapping (lines 271–286 of resolution-search.tsx) requires location.bounds to map detected features back to lat/lng; without it, coordinates fall back to raw normalized (0–1) image-space values, which are not valid GeoJSON coordinates.

🛠️ Suggested fix — capture bounds for Google Maps provider 
       const center = mapProvider === 'mapbox' && map ? map.getCenter() : mapData.cameraState?.center;
       if (center) {
         formData.append('latitude', center.lat.toString())
         formData.append('longitude', center.lng.toString())
         const bounds = mapProvider === 'mapbox' && map ? map.getBounds() : null;
         if (bounds) {
           formData.append('bounds', JSON.stringify({
             sw: { lat: bounds.getSouthWest().lat, lng: bounds.getSouthWest().lng },
             ne: { lat: bounds.getNorthEast().lat, lng: bounds.getNorthEast().lng }
           }))
+        } else if (mapData.cameraState?.bounds) {
+          formData.append('bounds', JSON.stringify(mapData.cameraState.bounds))
         }
       }
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/header-search-button.tsx` around lines 158 - 165, The code only
captures bounds when mapProvider === 'mapbox'; update the branch that builds
formData so it also captures bounds for mapProvider === 'google' by calling
map.getBounds() (or equivalent) and appending the same JSON structure via
formData.append('bounds', JSON.stringify({...})). Ensure you use the Google Maps
LatLngBounds methods (getSouthWest/getNorthEast or toJSON) to extract lat/lng
into the sw/ne shape, and keep the existing variable names (mapProvider, map,
bounds, formData.append) so resolution-search.tsx receives location.bounds for
the zoom-pass coordinate remapping.


const responseMessage = await actions.submit(formData)
setMessages((currentMessages: any[]) => [...currentMessages, responseMessage as any])
if (setInput) setInput('')
} catch (error) {
console.error('Failed to perform resolution search:', error)
toast.error('An error occurred while analyzing the map.')
Expand Down
1 change: 1 addition & 0 deletions components/map/map-data-context.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ export interface CameraState {
range?: number;
tilt?: number;
heading?: number;
bounds?: { sw: { lat: number; lng: number }; ne: { lat: number; lng: number } };
}

export interface MapData {
Expand Down
8 changes: 7 additions & 1 deletion components/map/mapbox-map.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -343,6 +343,8 @@ export const Mapbox: React.FC<{ position?: { latitude: number; longitude: number
currentMapCenterRef.current = { center: [center.lng, center.lat], zoom, pitch };

const timezone = tzlookup(center.lat, center.lng);
const bounds = map.current.getBounds();
if (!bounds) return;

setMapData(prevData => ({
...prevData,
Expand All @@ -351,7 +353,11 @@ export const Mapbox: React.FC<{ position?: { latitude: number; longitude: number
center: { lat: center.lat, lng: center.lng },
zoom,
pitch,
bearing
bearing,
bounds: {
sw: { lat: bounds.getSouthWest().lat, lng: bounds.getSouthWest().lng },
ne: { lat: bounds.getNorthEast().lat, lng: bounds.getNorthEast().lng }
}
}
}));
}
Expand Down
19 changes: 19 additions & 0 deletions fix_types.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
import sys

with open('lib/agents/resolution-search.tsx', 'r') as f:
content = f.read()

old_vars = """ let finalSummary = '';
let finalCogInfo = undefined;
let finalNewsContext = undefined;
let finalExtractedCoordinates = undefined;"""

new_vars = """ let finalSummary = '';
let finalCogInfo: z.infer<typeof resolutionSearchSchema>['cogInfo'] = undefined;
let finalNewsContext: z.infer<typeof resolutionSearchSchema>['newsContext'] = undefined;
let finalExtractedCoordinates: z.infer<typeof resolutionSearchSchema>['extractedCoordinates'] = undefined;"""

content = content.replace(old_vars, new_vars)

with open('lib/agents/resolution-search.tsx', 'w') as f:
f.write(content)
Loading