Skip to content

[codex] Add streaming file APIs#27190

Draft
nornagon-openai wants to merge 8 commits into
mainfrom
nornagon/codex/file-streaming-api
Draft

[codex] Add streaming file APIs#27190
nornagon-openai wants to merge 8 commits into
mainfrom
nornagon/codex/file-streaming-api

Conversation

@nornagon-openai

@nornagon-openai nornagon-openai commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

Why

The existing fs/readFile and fs/writeFile methods materialize an entire file in one request. That is unsuitable for large files and remote app-server clients. Pull-based chunks provide bounded memory use, natural backpressure, cancellation between requests, and optional client-side pipelining without adding a separate push protocol.

Streaming writes should retain the behavior of the existing fs/writeFile API. Replacing the destination with a temporary file would change file identity, break hardlinks, discard executable bits and extended metadata, and potentially weaken custom Windows ACLs. The streamed API therefore opens the destination with ordinary create-or-truncate semantics.

A local benchmark measured strict pull at roughly 270 MiB/s and two-request pipelined pull at roughly 367 MiB/s, matching the temporary push implementation. The push API was therefore omitted.

What changed

Add connection-scoped, client-supplied file handles to app-server v2 and exec-server:

  • fs/readFile/open, read, stat, and close provide bounded positional reads.
  • fs/writeFile/open, write, and close provide bounded sequential writes.
  • Both local and remote exec-server filesystems implement the same handle interface.
  • Read and write handles reject operations intended for the other handle type.
  • Operations for one handle are serialized. Close waits for active work, rejects queued work, and prevents a pending open from becoming live.
  • Remote handles close explicitly and report cleanup errors instead of relying on best-effort Drop requests.
  • Each connection is limited to 64 opening or open file handles.
  • Blocking local opens run on the blocking thread pool, and remote reads preserve the authoritative eof result.
  • App-server schemas, TypeScript bindings, API documentation, and the API design document cover the new surface.

Opening a write handle creates or truncates the destination immediately. Completed chunks are visible immediately. Cancellation, disconnection, or an I/O error can leave a partially written file, matching ordinary truncate-and-write behavior. Existing destination identity and metadata are preserved.

Validation

  • just test -p codex-app-server-protocol (227 tests)
  • just test -p codex-exec-server (204 tests)
  • just test -p codex-app-server (833 tests, 1 skipped)
  • just fix -p codex-file-system -p codex-exec-server -p codex-app-server-protocol -p codex-app-server
  • Regression coverage verifies positional reads, authoritative and exact-boundary EOF, direct truncation, immediate write visibility, hardlink identity, executable permission preservation, bounded handles, open/close races, and close waiting for blocking writes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant