zero copy demo#64296
Draft
zclllyybb wants to merge 5 commits into
Draft
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 32052 ms |
Contributor
TPC-DS: Total hot run time: 184045 ms |
zclllyybb
added a commit
to zclllyybb/doris
that referenced
this pull request
Jun 25, 2026
### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#64296 Problem Summary: BE UT and several regression jobs exposed two issues in the zero-copy scan branch. First, insert_many_from() is allowed to be a no-op when length is zero, including the case where position is src.size(), but several concrete column implementations read the source row before checking length. Page-backed spans assert on that one-past-end read. The fix makes the concrete implementations return before reading when length is zero. Second, the new storage read batch enlargement let simple fixed-width scans pass a much larger batch into SegmentIterator, while lower lazy-materialization selector paths still assume the normal session batch contract. That caused regression coredumps in filter_by_selector. The fix restores the OLAP reader batch size to RuntimeState::batch_size() and keeps the no-copy slot-ref projection fast path limited to simple projection scans. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run --filter=ColumnArrayTest.InsertManyFromTest -j 90 - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 31441 ms |
Contributor
TPC-DS: Total hot run time: 173120 ms |
zclllyybb
added a commit
to zclllyybb/doris
that referenced
this pull request
Jun 26, 2026
### What problem does this PR solve? Issue Number: N/A Related PR: apache#64296 Problem Summary: The zero-copy fixed-length column path can keep values in an external page instead of the local PODArray. Lazy materialization calls ColumnVector::filter_by_selector through SegmentIterator::copy_column_data_by_selector, but the inherited selector implementation read the local PODArray directly, so page-backed columns with an empty local data buffer could crash with PODArray::operator[] checks. This commit overrides ColumnVector::filter_by_selector to read through immutable_data(), giving local and external storage the same selector semantics. It also makes non-empty ColumnVector::insert_many_from from an invalid source row throw a Doris exception before materialization, replacing container-specific one-past UB/std::span assertions with an explicit contract, and updates the generic column test accordingly while preserving the historical Array debug-check skip. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run --filter=ColumnIPTest.InsertManyFromTest:ColumnVectorTest.filter_by_selector_from_external_page -j 90 - ./run-be-ut.sh --run --filter='ColumnIPTest.InsertManyFromTest:ColumnVectorTest.*:ColumnArrayTest.InsertManyFromTest:ColumnNullableTest.*:ColumnStringTest.filter_by_selector:ColumnStringTest.insert_many_from:ColumnDictionaryTest.filter_by_selector:PredicateColumnTest.*' -j 90 - ./build.sh --be - git diff --check -- be/src/core/column/column_vector.h be/src/core/column/column_vector.cpp be/test/core/column/common_column_test.h be/test/core/column/column_vector_test.cpp - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary: The zero-copy fixed-width column demo kept a vector of external page spans and exposed a span-iteration helper so consumers could process multi-page scan blocks without materializing. That made ColumnVector carry a more complex segmented immutable-data contract than the execution layer normally expects. This change matches the StarRocks-style single-resource model: ColumnVector may adopt one page-owned fixed-width view when the destination is empty and aligned, while any later append or mutable access materializes the view into Doris-owned storage before continuing. Legacy DATE/DATETIME still use the existing decode-and-convert path because the page bytes are not byte-identical to the in-memory layout. SUM no longer needs a multi-span hot path and reads the normal single immutable view.
### Release note
None
### Check List (For Author)
- Test: Manual test
- PATH=/mnt/disk6/common/ldb_toolchain_toucan/bin:$PATH build-support/clang-format.sh
- git diff --check -- be/src/core/column/column_vector.h be/src/exprs/aggregate/aggregate_function_sum.h
- ./build.sh --be -j 90
- Behavior changed: Yes. Fixed-width zero-copy scan columns now keep at most one page-backed external view and materialize when additional page ranges are appended.
- Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#64296 Problem Summary: BE UT and several regression jobs exposed two issues in the zero-copy scan branch. First, insert_many_from() is allowed to be a no-op when length is zero, including the case where position is src.size(), but several concrete column implementations read the source row before checking length. Page-backed spans assert on that one-past-end read. The fix makes the concrete implementations return before reading when length is zero. Second, the new storage read batch enlargement let simple fixed-width scans pass a much larger batch into SegmentIterator, while lower lazy-materialization selector paths still assume the normal session batch contract. That caused regression coredumps in filter_by_selector. The fix restores the OLAP reader batch size to RuntimeState::batch_size() and keeps the no-copy slot-ref projection fast path limited to simple projection scans. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run --filter=ColumnArrayTest.InsertManyFromTest -j 90 - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: N/A Related PR: apache#64296 Problem Summary: The zero-copy fixed-length column path can keep values in an external page instead of the local PODArray. Lazy materialization calls ColumnVector::filter_by_selector through SegmentIterator::copy_column_data_by_selector, but the inherited selector implementation read the local PODArray directly, so page-backed columns with an empty local data buffer could crash with PODArray::operator[] checks. This commit overrides ColumnVector::filter_by_selector to read through immutable_data(), giving local and external storage the same selector semantics. It also makes non-empty ColumnVector::insert_many_from from an invalid source row throw a Doris exception before materialization, replacing container-specific one-past UB/std::span assertions with an explicit contract, and updates the generic column test accordingly while preserving the historical Array debug-check skip. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run --filter=ColumnIPTest.InsertManyFromTest:ColumnVectorTest.filter_by_selector_from_external_page -j 90 - ./run-be-ut.sh --run --filter='ColumnIPTest.InsertManyFromTest:ColumnVectorTest.*:ColumnArrayTest.InsertManyFromTest:ColumnNullableTest.*:ColumnStringTest.filter_by_selector:ColumnStringTest.insert_many_from:ColumnDictionaryTest.filter_by_selector:PredicateColumnTest.*' -j 90 - ./build.sh --be - git diff --check -- be/src/core/column/column_vector.h be/src/core/column/column_vector.cpp be/test/core/column/common_column_test.h be/test/core/column/column_vector_test.cpp - Behavior changed: No - Does this need documentation: No
Problem Summary: Upstream master added an inline ColumnVector::filter_by_selector implementation, while the zero-copy branch still carried a separate declaration and out-of-line template definition. The merged PR source therefore hit a duplicate member declaration and failed TeamCity compile. Fold the page-backed read path into the upstream inline implementation and remove the duplicate implementation. Release note: None Test Plan: PATH=/mnt/disk6/common/ldb_toolchain_toucan/bin:/mnt/disk3/zhaochangle/.bun/bin:/mnt/disk3/zhaochangle/.opencode/bin:/mnt/disk3/zhaochangle/.local/bin:/mnt/disk6/common/apache-maven-3.9.14/bin:/mnt/disk6/common/ldb_toolchain_028/bin:/mnt/disk6/common/jdk-17.0.16/bin:/mnt/disk6/common/node-v24.14.1-linux-x64/bin:/mnt/disk6/common/apache-maven-3.9.14/bin:/mnt/disk6/common/ldb_toolchain_028/bin:/mnt/disk6/common/jdk-17.0.16/bin:/mnt/disk6/common/node-v24.14.1-linux-x64/bin:/mnt/disk6/common/node-v24.14.1-linux-x64/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/codex-path:/mnt/disk3/zhaochangle/.codex/tmp/arg0/codex-arg03yyoos:/mnt/disk3/zhaochangle/.bun/bin:/mnt/disk3/zhaochangle/.opencode/bin:/mnt/disk3/zhaochangle/.local/bin:/mnt/disk6/common/apache-maven-3.9.14/bin:/mnt/disk6/common/ldb_toolchain_028/bin:/mnt/disk6/common/jdk-17.0.16/bin:/mnt/disk6/common/node-v24.14.1-linux-x64/bin:/mnt/disk6/common/apache-maven-3.9.14/bin:/mnt/disk6/common/ldb_toolchain_028/bin:/mnt/disk6/common/jdk-17.0.16/bin:/mnt/disk6/common/node-v24.14.1-linux-x64/bin:/mnt/disk3/zhaochangle/.opencode/bin:/mnt/disk3/zhaochangle/.local/bin:/mnt/disk3/zhaochangle/.local/bin:/mnt/disk3/zhaochangle/bin:/mnt/disk6/common/apache-maven-3.9.14/bin:/mnt/disk6/common/ldb_toolchain_028/bin:/mnt/disk6/common/jdk-17.0.16/bin:/mnt/disk6/common/node-v24.14.1-linux-x64/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/tableau/tableau_server/packages/customer-bin.20251.25.0520.1026 build-support/clang-format.sh Test Plan: git diff --check -- be/src/core/column/column_vector.h be/src/core/column/column_vector.cpp be/test/core/column/common_column_test.h be/test/core/column/column_vector_test.cpp Test Plan: ./build.sh --be Test Plan: ./run-be-ut.sh --run --filter='ColumnIPTest.InsertManyFromTest:ColumnVectorTest.*:ColumnArrayTest.InsertManyFromTest:ColumnNullableTest.*:ColumnStringTest.filter_by_selector:ColumnStringTest.insert_many_from:ColumnDictionaryTest.filter_by_selector:PredicateColumnTest.*' -j 90
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29005 ms |
Contributor
TPC-DS: Total hot run time: 170322 ms |
Contributor
ClickBench: Total hot run time: 24.53 s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)