feat(agent-core): announce image compression and keep originals readable - #1304
Conversation
Every image ingestion point (ReadMediaFile, MCP tool results, clipboard paste, REST upload/inline base64, ACP) now places a <system> caption next to a compressed image stating the original vs. delivered dimensions, byte size, and format, so downsampling is never silent to the model. Originals stay readable: file uploads point at the stored blob, and in-memory images are persisted into the session's media-originals dir (content-addressed, size-capped, removed with the session; temp-dir fallback when no session is known). ReadMediaFile gains region (crop in original-image pixel coordinates, delivered at full fidelity) and full_resolution (skip downscaling, with an explicit error over the per-image byte limit), so the model can zoom into fine detail instead of silently degrading on large images.
🦋 Changeset detectedLatest commit: 89cf617 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
commit: |
The caption announcing an image's compression was inserted before the shared 100K text budget was applied, so a chatty MCP result (page text + screenshot) consumed the budget first and the caption was evicted — or sliced mid-string into an unclosed <system> fragment — while the downsampled image survived, silently reintroducing the exact degradation the caption exists to report, and orphaning the persisted original. Split the size-limit pass in two and reorder the pipeline: the text budget now runs on the tool's own text BEFORE compression inserts captions (exempt by construction), and the per-part binary cap still runs after compression so compressible screenshots are kept.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5a68258c42
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return { | ||
| ok: true, | ||
| data: new Uint8Array(encoded.data), |
There was a problem hiding this comment.
When a requested region is high-entropy enough that encodeWithinBudget cannot get under IMAGE_BYTE_BUDGET, that helper deliberately returns the smallest attempt even if it is still over budget. This branch then returns ok: true without checking encoded.data.length, so ReadMediaFile can send an over-limit crop despite the new contract that crops are fitted or explicitly refused; mirror the skipResize over-budget error before returning success.
Useful? React with 👍 / 👎.
…mantics - cropImageForModel rejects non-finite region coordinates with a clean message instead of surfacing the codec's internal validation dump - the full_resolution and cropped-region over-budget errors now include exact byte counts alongside the rounded sizes, so a file a hair over budget no longer reads "is 3.8 MB, over the 3.8 MB limit" - read-media.md notes that re-reading a file without region or full_resolution reproduces the same downsampled image
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 52aabafdf1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const existing = await stat(path).catch(() => null); | ||
| // Content-addressed: an existing entry with the right size IS this image. | ||
| if (existing === null || existing.size !== bytes.length) { | ||
| await writeFile(path, bytes); |
There was a problem hiding this comment.
When options.dir is absent, originals are stored under the OS temp directory, but this write uses the process-default file mode (and the directory is created just above with the default mode as well). On typical shared systems with umask 022 this makes pasted/MCP image originals world-readable under /tmp/kimi-code-original-images, and the generated captions expose the exact path; create/chmod the fallback cache privately and write files with owner-only permissions.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 89cf6179ea
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| homedir, | ||
| // Session-level, shared across agents: originals persisted for | ||
| // compression captions live with the session, not the agent. | ||
| mediaOriginalsDir: sessionMediaOriginalsDir(this.options.homedir), |
There was a problem hiding this comment.
When a session is created with a non-local Kaos (the SDK accepts caller-supplied Kaos, including SSH-backed implementations), this passes a local session-store path to MCP image persistence while ReadMediaFileTool later reads paths through the agent's kaos.readBytes. A compressed MCP image will therefore get a caption pointing at <local sessionDir>/media-originals/..., but the advertised readback path does not exist in the remote/tool filesystem, so the model cannot actually inspect the original. Persist via the same filesystem the media tool reads from, or omit the readback path unless it is readable there.
Useful? React with 👍 / 👎.
| @@ -0,0 +1,5 @@ | |||
| --- | |||
| "@moonshot-ai/kimi-code": minor | |||
There was a problem hiding this comment.
This PR adds new public exports from @moonshot-ai/kimi-code-sdk (buildImageCompressionCaption, persistOriginalImage, sessionMediaOriginalsDir, and the caption input type), but the changeset only selects the CLI package. The repo's changeset rules require selecting @moonshot-ai/kimi-code-sdk when an internal capability is exposed through the SDK; otherwise the next release will not version/changelog/publish the SDK package containing these new symbols for npm consumers.
Useful? React with 👍 / 👎.
1 participant
Why
Large images are downsampled before reaching the model, but nothing in the context said so. The model would look at a blurry 2000px copy of a 5000px screenshot, fail to read small text, and not know why — silent quality degradation with no way to recover. (Re-reading the original file didn't help either: the same deterministic compression produced the same downsampled copy.)
What
1. Compression is announced next to the image, at every ingestion point.
A shared
<system>caption (buildImageCompressionCaption) states the original vs. delivered dimensions, byte size, and format, plus where the original lives and how to read detail back:ReadMediaFile: the existing<system>block now reports the delivery mode (untouched / downsampled / crop / native resolution)2. Originals stay readable.
media-originals/dir — content-addressed (sha256), size-capped with oldest-first eviction, removed with the session. Temp-dir fallback when no session context exists.3.
ReadMediaFilecan actually recover detail.region {x, y, width, height}(original-image pixel coordinates): crops before the pipeline, so any region within the edge cap arrives at full fidelity; the<system>block explains crop→original coordinate mappingfull_resolution: skips downscaling; fails with an explicit error (suggestingregion) when the file exceeds the per-image byte limit, instead of silently degradingDetail delivered to the model is bounded by pixels-per-request, so cropping is the only mechanism that genuinely increases detail per unit area — the caption steers the model to it.
Notes
BlobStorewire.jsonl offload: that layer externalizes the compressed copy for transcript persistence, transparently;media-originals/holds the pre-compression bytes for model-driven readback. Session-dir placement also means readback paths survive--resume.MAX_IMAGE_EDGE_PX = 2000is unchanged; worth calibrating once we know the vision encoder's effective pixel cap.blobs/+ original inmedia-originals/);exportSessionzips include both.Test plan