fix(files): stream file uploads from disk instead of buffering in memory#815
Open
jrvb-rl wants to merge 1 commit into
Open
fix(files): stream file uploads from disk instead of buffering in memory#815jrvb-rl wants to merge 1 commit into
jrvb-rl wants to merge 1 commit into
Conversation
upload_file (and any PathLike file input) read the entire file into memory via read_bytes() before the request was built, contradicting the streaming behavior these large-file endpoints imply. Hand httpx an open file handle so its multipart encoder reads lazily in chunks. Tests assert an io.IOBase handle is returned and close it to avoid resource warnings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
upload_file(and anyPathLikefile input) read the entire file into memory viapath.read_bytes()before the request was even built. For the large files these endpoints target, that buffers the whole payload in RAM. This hands httpx an open file handle instead, so its multipart encoder reads the file lazily in chunks as it serializes the body.Split out of #803 as an independent, low-risk change.
Changes
_files.py:_transform_file/_async_transform_filereturnpath.open("rb")(anio.IOBase) instead ofread_bytes().tests/test_files.py: assert anio.IOBasehandle is returned (wasIsBytes()), and close handles in tests that don't issue a request to avoidResourceWarning.Notes
open()); httpx reads from it in chunks during send.Test plan
uv run pytest tests/test_files.py— 14 passedruff check/ruff format --checkclean🤖 Generated with Claude Code