Implement largelistview support#303
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
Adds end-to-end support for the Arrow LargeListView logical type across the core library, IPC read/write paths, C Data Interface schema/array interop, and test/integration infrastructure (with the stated limitation that >2GB lists are still not supported).
Changes:
- Introduces
LargeListViewType/LargeListViewArray(+ builder) and theArrowTypeId.LargeListViewenum value. - Extends IPC serialization/deserialization and stream writing to handle LargeListView buffers and value slicing.
- Updates C Data Interface schema import/export and array import, plus JSON integration parsing and test generators/verifiers.
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| test/Apache.Arrow.Tests/TestData.cs | Adds LargeListView to the generated schema and array creation for test matrices. |
| test/Apache.Arrow.Tests/TableTests.cs | Updates expected column counts due to the additional LargeListView field. |
| test/Apache.Arrow.Tests/LargeListViewArrayTests.cs | New unit tests for slicing, overflow checks, and basic properties. |
| test/Apache.Arrow.Tests/LargeListViewArrayBuilderTests.cs | New unit tests for builder append/null/clear and nested string lists. |
| test/Apache.Arrow.Tests/ArrowReaderVerifier.cs | Extends array comparison to support LargeListView arrays. |
| test/Apache.Arrow.Tests/ArrowArrayConcatenatorTests.cs | Adds LargeListView to concatenation test coverage and data generation. |
| test/Apache.Arrow.IntegrationTest/JsonFile.cs | Adds JSON-to-type parsing and array creation support for “largelistview”. |
| src/Apache.Arrow/Types/LargeListViewType.cs | New Arrow type definition for LargeListView. |
| src/Apache.Arrow/Types/IArrowType.cs | Adds ArrowTypeId.LargeListView. |
| src/Apache.Arrow/Ipc/MessageSerializer.cs | Deserializes Flatbuf LargeListView fields into LargeListViewType. |
| src/Apache.Arrow/Ipc/ArrowTypeFlatbufferBuilder.cs | Serializes LargeListViewType into Flatbuf LargeListView. |
| src/Apache.Arrow/Ipc/ArrowStreamWriter.cs | Writes LargeListView buffers and slices values based on min/max offsets. |
| src/Apache.Arrow/Ipc/ArrowReaderImplementation.cs | Accounts for LargeListView’s 3-buffer layout when loading fields. |
| src/Apache.Arrow/C/CArrowSchemaImporter.cs | Imports C schema format +vL as LargeListViewType. |
| src/Apache.Arrow/C/CArrowSchemaExporter.cs | Exports LargeListViewType as C schema format +vL. |
| src/Apache.Arrow/C/CArrowArrayImporter.cs | Imports LargeListView arrays (validity + long offsets + long sizes). |
| src/Apache.Arrow/Arrays/LargeListViewArray.cs | New array implementation + builder and slicing helpers. |
| src/Apache.Arrow/Arrays/ArrowArrayFactory.cs | Instantiates LargeListViewArray for ArrowTypeId.LargeListView. |
| src/Apache.Arrow/Arrays/ArrowArrayBuilderFactory.cs | Builds LargeListViewArray.Builder for ArrowTypeId.LargeListView. |
| src/Apache.Arrow/Arrays/ArrayDataConcatenator.cs | Adds concatenation logic for LargeListView arrays. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
adamreeve
reviewed
Mar 31, 2026
Contributor
adamreeve
left a comment
adamreeve
left a comment
There was a problem hiding this comment.
👍 This looks good to me besides the question about resetting Start in Clear.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's Changed
Support for largelistview has been implemented. This still doesn't support lists which are actually more than 2GB, but at least smaller lists of this type can be loaded.