th-3be564: multimodal Big Smooth chat — attach + see images - #169
Merged
Conversation
Web: paperclip/paste/drag-drop image+PDF attachments, composer thumbnails,
inline <img> render in the user bubble. Sends {content, attachments:[{name,
mime,data(base64)}]} — optional field, text-only sends unchanged.
Daemon: chat routes accept attachments, content-address bytes under
~/.smooth/attachments/, build a multimodal turn via Message::user_with_images.
Image turns auto-route to gemini-2.5-flash-lite (cheap vision, native PDF);
text-only keeps the default model; explicit body.model still wins. Documents
are marker-only (Phase 4 does real ingest).
Engine: pins smooth-operator-core rev b43c04fe (multimodal Message content +
image_url wire + AgentConfig::with_user_images; PRs #58/#62). Decoupled from
core main's post_call change (th-10eb50) — see Cargo.toml comment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 72c21f9 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
brentrager
enabled auto-merge (squash)
July 4, 2026 21:42
Verified live against llm.smoo.ai: a PDF sent as a data:application/pdf image_url part to gemini-2.5-flash-lite is read natively (answered a question about the file). So PDF 'documents' need no separate ingest — add an AttachmentKind::Pdf that rides the same multimodal/vision path as images (is_model_media). docx/xlsx stay marker-only (docling microVM is the future upgrade). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…th-3be564-multimodal
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Epic th-3be564, Phases 1–3. Big Smooth chat can now take image attachments and the assistant actually sees them.
What you get
attachments, content-address bytes under~/.smooth/attachments/, and build a multimodal turn. Image turns auto-route togemini-2.5-flash-lite(cheap, strong vision, native PDF); text-only turns keep the default model unchanged; an explicitbody.modelstill wins.Message.images→ OpenAIimage_urlparts) +AgentConfig::with_user_images(core PRs pearl th-01c714: bracketed-paste + flatten prompt so multi-line task arrives as one TUI submission #58/pearl th-2b5f63: th cast models — list live model groups on configured provider #62), via a git-rev pin.Wire contract (back-compatible)
POST /api/chat/sessions/{id}/messagesbody →{ content, attachments?: [{ name, mime, data(base64, no prefix) }] }.attachmentsis optional (#[serde(default)]) — existing{content}sends are byte-identical.Scope / follow-ups
[attached document: x.pdf]text marker reaches the model). Real ingest — Gemini-native PDF vs. a docling microVM — is Phase 4 (th-1b14d6).ponytail:ceilings in code.post_callchange, th-10eb50) — documented at Cargo.toml:194.Tests
pick_modelprecedence, back-compat + with-attachments body parse (7). Full bigsmooth suite 322 pass.pnpm build(tsc + vite) clean.cargo build --workspace+ web bundle build clean.🤖 Generated with Claude Code