feat(c++/python): support stream deserialization for c++ and python#3307
Merged
chaokunyang merged 21 commits intoapache:mainfrom Feb 26, 2026
Merged
feat(c++/python): support stream deserialization for c++ and python#3307chaokunyang merged 21 commits intoapache:mainfrom
chaokunyang merged 21 commits intoapache:mainfrom
Conversation
This was referenced Feb 6, 2026
Zakir032002
added a commit
to Zakir032002/fory
that referenced
this pull request
Feb 15, 2026
Implements apache#3300 aligned with C++ PR apache#3307 stream model. - Add ForyStreamBuf: growable buffer wrapping dyn Read, no compaction - Make Reader stream-aware: ensure_readable before reads, sync_stream_pos after - Add byte-at-a-time varint fallbacks for stream-backed readers - Fix deserialize_from to transfer stream state via take/restore pattern - Preserve zero-overhead in-memory fast path (branch-light) - Add 12 comprehensive stream tests (primitives, structs, strings, sequential decode, truncated stream errors, Vec, regression) Closes apache#3300
This was referenced Feb 15, 2026
Zakir032002
added a commit
to Zakir032002/fory
that referenced
this pull request
Feb 16, 2026
- Add standalone ForyStreamBuf with growable buffer - Implements fill_buffer for on-demand reading from std::io::Read - Buffer grows monotonically without compaction - No integration with Reader yet (zero impact on existing code) - Includes 4 unit tests for basic functionality Design follows C++ PR apache#3307 and addresses apache#3300. Part 1 of 3-phase implementation.
This was referenced Feb 16, 2026
# Conflicts: # cpp/fory/serialization/serializer.h # cpp/fory/util/buffer_test.cc
urlyy
approved these changes
Feb 26, 2026
pandalee99
approved these changes
Feb 26, 2026
This was referenced Feb 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why?
C++ and Python deserialization currently assumes data is already materialized in memory-backed buffers. This PR adds stream-backed deserialization support so payloads can be read incrementally from input streams while preserving existing serialization behavior and error handling.
What does this PR do?
ForyInputStreamBuf/ForyInputStream) and integrates it withBufferso reads can request more bytes on demand.Fory_PyCreateBufferFromStream) sopyfory.buffer.Buffercan be constructed from Python objects that implementread(size).ensure_sizechecks for header/string/fixed-field reads and falling back from batched varint reads on stream-backed buffers.stream_test.cc, buffer stream tests).python/pyfory/tests/test_stream.py) and buffer stream coverage inpython/pyfory/tests/test_buffer.py.Related issues
N/A
Does this PR introduce any user-facing change?
Benchmark
N/A