feat: offload large base64 media payloads to external blob files - #117
Conversation
Wire.jsonl grows very large when conversations contain images, audio,
or video embedded as data: URIs. This change extracts base64 payloads
exceeding a threshold (default 4KB) into per-agent blob files stored
alongside wire.jsonl, replacing the inline data with a lightweight
blobref URL.
- New blobref.ts handles offload / rehydrate / missing-blob downgrade.
- Blobref format mirrors data URI: blobref:<mime>;<hash>.
- FileSystemAgentRecordPersistence transparently offloads on write.
- KosongLLM rehydrates blobrefs back to data URIs before calling the
LLM provider, so the rest of the stack is unaware of blobrefs.
- Export (zip) naturally includes blob directories via recursive file
collection, no extra changes needed.
Key design decisions:
- Offload scans only known record types (turn.prompt, turn.steer,
context.append_message, context.append_loop_event tool.result) to
avoid accidentally touching unrelated strings.
- Within a ContentPart, media URLs are found by structural pattern
({ xxxUrl: { url } }) rather than hard-coding field names, so new
media types are supported automatically.
- Blob files are named by SHA-256 of the base64 payload, providing
natural deduplication.
- Missing blobs degrade to a text placeholder so the LLM provider
does not receive an invalid media block.
🦋 Changeset detectedLatest commit: d776679 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
commit: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3cd02dffc6
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b4d1dfc607
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (this.blobStore !== undefined) { | ||
| messages = await this.blobStore.rehydrateMessages(params.messages); | ||
| } |
There was a problem hiding this comment.
When a session with offloaded media is resumed, replay restores context.append_message / tool-result content with blobref: URLs in ContextMemory; only normal turn requests pass through this new KosongLLM.chat rehydration path. Full compaction still builds projected history and calls this.agent.generate directly in packages/agent-core/src/agent/compaction/full.ts, so manual/auto compaction after such a resume sends blobref: URLs to the provider instead of the original data URIs, causing invalid media input or summaries that silently omit the media. Rehydrate the compaction messages too, or centralize this at the lower generate seam used by both paths.
Useful? React with 👍 / 👎.
| this.setCache(hash, base64Payload); | ||
| return `${BLOBREF_PROTOCOL}${mimeType};${hash}`; |
There was a problem hiding this comment.
This writes a new on-disk representation into wire.jsonl, but the commit leaves AGENT_WIRE_PROTOCOL_VERSION at 1.2 (checked packages/agent-core/src/agent/records/migration/index.ts). If a session written by this version is later resumed by an older 1.2 reader, it will treat blobref: as a normal media URL and send invalid media to the provider without any version warning or migration gate. Please bump the wire version and add the corresponding migration/compatibility handling when introducing blobrefs.
Useful? React with 👍 / 👎.
| this.blobStore = config.homedir | ||
| ? new BlobStore({ blobsDir: join(config.homedir, 'blobs') }) | ||
| : undefined; |
There was a problem hiding this comment.
When callers use the public persistence injection path with new FileSystemAgentRecordPersistence(..., { blobStore }), this constructor still creates the rehydrating store only from homedir. A standalone or test Agent with custom persistence and no matching homedir can therefore write/replay blobref: records but then send those unresolved URLs on normal LLM turns because KosongLLM receives undefined (or the wrong blob directory). Expose/pass the persistence blob store into Agent instead of deriving it only here.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7639023b2a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| while (this.currentCacheSize + size > this.maxCacheSize && this.cache.size > 0) { | ||
| this.evictLRU(); | ||
| } | ||
| this.currentCacheSize += size; |
There was a problem hiding this comment.
When a single media blob is larger than maxCacheSize, this loop evicts every existing entry but then still inserts the oversized payload and increments currentCacheSize past the cap. This can happen with existing ReadMediaFileTool inputs up to 100MB (about 133MB base64), so the default 50MB cache can retain a huge string after offload/rehydration and defeat the memory-pressure goal; avoid caching entries whose size exceeds the cap.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1edbeeab29
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| content = await readFile(blobPath); | ||
| } catch { | ||
| return c.json({ error: 'blob not found', code: 'NOT_FOUND' }, 404); | ||
| } | ||
| const mimeType = c.req.query('mime') ?? 'application/octet-stream'; | ||
| return new Response(content, { |
There was a problem hiding this comment.
For offloaded media, this route sends the blob file bytes directly with the requested media content type, but BlobStore.writeBlob stores only the base64 payload string in the file. In vis, an image URL like /api/sessions/.../blobs/<hash>?mime=image/png will therefore return ASCII bytes such as iVBOR... as image/png instead of the decoded PNG bytes, so large offloaded images/audio/video still fail to preview. Decode the file contents from base64 before constructing the response, or return a data URL instead.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: de813451ec
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| agentId: string, | ||
| ): void { | ||
| for (const entry of entries) { | ||
| rehydrateRecord(entry.data as Record<string, unknown>, sessionId, agentId); |
There was a problem hiding this comment.
When readAgentWire reads current 1.3 records, it returns the same object for entry.data and entry.raw whenever no migration applies (apps/vis/server/src/lib/wire-reader.ts:87-101). This in-place mutation therefore also rewrites entry.raw, so the wire detail's “copy raw” / “as written on disk” view shows /api/sessions/... URLs instead of the actual blobref: value from wire.jsonl for current sessions with offloaded media. Clone entry.data before resolving blobrefs, or have the reader return a distinct migrated object.
Useful? React with 👍 / 👎.
| const blobPath = join( | ||
| detail.sessionDir, | ||
| 'agents', | ||
| agentId, | ||
| 'blobs', | ||
| hash, | ||
| ); |
There was a problem hiding this comment.
For sessions where an agent's recorded homedir is not exactly <sessionDir>/agents/<id> (for example callers can pass config.homedir through Session.createAgent, and imported/remapped state preserves meta.homedir), the agent writes offloaded media to homedir/blobs, but this route always reads under detail.sessionDir/agents/<id>/blobs. In that context the wire/context endpoints resolve blobrefs to this new route, but every preview 404s even though detail.agents already contains the correct agent.homedir; build the path from that recorded homedir instead.
Useful? React with 👍 / 👎.
| const mimeType = rest.slice(0, semiIdx); | ||
| const hash = rest.slice(semiIdx + 1); | ||
| if (hash.length === 0) return url; | ||
| return `/api/sessions/${encodeURIComponent(sessionId)}/blobs/${encodeURIComponent(hash)}?agent=${encodeURIComponent(agentId)}&mime=${encodeURIComponent(mimeType)}`; |
There was a problem hiding this comment.
Fresh evidence after the vis resolver was added: ImagePreview still only renders URLs starting with data:image/ or http(s):// (apps/vis/web/src/components/shared/ImagePreview.tsx:19), but this resolver returns a root-relative /api/... URL. For any offloaded image shown in the context or wire views, the UI therefore stops at the “unsupported scheme” fallback and never requests the blob route; return an absolute URL or update the preview predicate to accept same-origin paths.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3eac4c4e82
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
|
||
| const blobFiles = await readdir(blobsDir); | ||
| expect(blobFiles).toHaveLength(1); | ||
| expect(await readFile(join(blobsDir, blobFiles[0]!), 'utf8')).toBe(payload); |
There was a problem hiding this comment.
This new assertion expects the blob file to contain the original base64 text, but BlobStore.writeBlob decodes that string with Buffer.from(base64Payload, 'base64') before writing. With the test payload of 'X'.repeat(5000), reading the file as UTF-8 will not equal payload, so the added persistence test fails even though offloading wrote the decoded bytes correctly; compare readFile(...).toString('base64') or the expected decoded buffer instead.
Useful? React with 👍 / 👎.
BlobStore.offload now returns a new AgentRecord with offloaded URLs instead of mutating the input record's ContentPart objects. Tool result parts are shared by reference between persistence records and the in-memory context history; mutating them during the async flush could let blobref: URLs leak into the next LLM request body. drainBatch uses the returned record for JSON serialization. Adds tests for the non-mutating contract and for the previously uncovered context.append_loop_event/tool.result path. Also fixes an unrelated broken assertion in persistence.test.ts that compared a binary blob file against the base64 string.
Fold the read-through cache changeset into the offloading one — both ship together on this branch under the same feature.
1 participant
Problem
wire.jsonl grows very large when conversations contain images, audio, or video embedded as base64 data URIs. During session replay, the entire wire must be traversed, causing high memory pressure as every base64 payload is parsed into memory even when the media is not immediately needed.
What changed
Key design decisions:
Checklist