[core] fix parquet can not read empty row with first column is array.#4711
Closed
Stephen0421 wants to merge 2 commits into
Closed
[core] fix parquet can not read empty row with first column is array.#4711Stephen0421 wants to merge 2 commits into
Stephen0421 wants to merge 2 commits into
Conversation
xccui
suggested changes
Dec 15, 2024
Member
xccui
left a comment
xccui
left a comment
There was a problem hiding this comment.
Thanks for your effort @Stephen0421! I tried the fix locally by reading some data with highly nested schemas. Some tests passed and some failed with the following exception. I'll share the table files with you for debugging.
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 2
at org.apache.paimon.data.columnar.heap.AbstractHeapVector.isNullAt(AbstractHeapVector.java:111)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readRow(NestedColumnReader.java:144)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:102)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readRow(NestedColumnReader.java:124)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:102)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readArray(NestedColumnReader.java:243)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:106)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readRow(NestedColumnReader.java:124)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:102)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readRow(NestedColumnReader.java:124)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:102)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readRow(NestedColumnReader.java:124)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:102)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readArray(NestedColumnReader.java:243)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:106)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readRow(NestedColumnReader.java:124)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:102)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readRow(NestedColumnReader.java:124)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:102)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readRow(NestedColumnReader.java:124)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:102)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readArray(NestedColumnReader.java:243)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:106)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readRow(NestedColumnReader.java:124)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:102)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readRow(NestedColumnReader.java:124)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:102)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readRow(NestedColumnReader.java:124)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:102)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readArray(NestedColumnReader.java:243)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:106)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readRow(NestedColumnReader.java:124)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:102)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readRow(NestedColumnReader.java:124)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:102)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readRow(NestedColumnReader.java:124)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:102)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readArray(NestedColumnReader.java:243)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:106)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readRow(NestedColumnReader.java:124)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readData(NestedColumnReader.java:102)
at org.apache.paimon.format.parquet.reader.NestedColumnReader.readToVector(NestedColumnReader.java:90)
at org.apache.paimon.format.parquet.ParquetReaderFactory$ParquetReader.nextBatch(ParquetReaderFactory.java:406)
| ParquetSchemaConverter.convertToParquetMessageType( | ||
| "paimon-parquet", NESTED_ARRAY_MAP_TYPE); | ||
| String[] candidates = new String[] {"snappy", "zstd", "gzip"}; | ||
| String compress = candidates[new Random().nextInt(3)]; |
Member
There was a problem hiding this comment.
I know this was intended to balance coverage and test running time, but using random in test cases is usually not ideal. Let's use a parameterized test here for these three formats.
Contributor
Author
There was a problem hiding this comment.
Sorry to reply so late, this is refer to the previous pr, i will change it to the parameterized test
| HeapBytesVector phbv = new HeapBytesVector(total, isNull); | ||
| return new ParquetDecimalVector(phbv, total); | ||
| } | ||
| default: |
Member
There was a problem hiding this comment.
I wonder if all the existing types are covered.
Contributor
Author
There was a problem hiding this comment.
yes except the INT32 and INT64, other primitiveType should deserialize as HeapBytesVector
|
|
||
| private boolean isFirstRow = true; | ||
|
|
||
| private boolean cutLevel = false; |
Member
There was a problem hiding this comment.
Could you add a comment for this boolean?
Stephen0421
force-pushed
the
PAIMON-4710
branch
from
fa4cd3a to
e0554db
Compare
…is the same as other child vector.
Contributor
|
Very thanks @Stephen0421 , but we still found some corner cases, we decided to revert old changes. see: #4745 |
Contributor
|
Close this one now. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #4710
Tests
add unit case and test in local.
API and Format
Parquet
Documentation
No