EndOfStreamException - can't read empty data page

[ishepherd]

Version: Parquet.Net latest source (v3.7.7)

Runtime Version: .Net Core v3.0, I presume - I am running the Parquet.Test tests in VS

OS: Windows

Expected behavior

ParquetReader.ReadEntireRowGroup to complete successfully with my test file that was output by Apache Spark 2.4.5. Sorry, cannot share this file.

Python's pandas read_parquet reads it successfully.

Actual behavior

Message: 
    System.IO.EndOfStreamException : Unable to read beyond the end of the stream.
  Stack Trace: 
    BinaryReader.InternalRead(Int32 numBytes)
    BinaryReader.ReadInt32()
    RunLengthBitPackingHybridValuesReader.ReadRleBitpackedHybrid(BinaryReader reader, Int32 bitWidth, Int32 length, Int32[] dest, Int32 offset, Int32 pageSize) line 37
    DataColumnReader.ReadPlainDictionary(BinaryReader reader, Int32 maxReadCount, Int32[] dest, Int32 offset) line 279
    DataColumnReader.ReadColumn(BinaryReader reader, Encoding encoding, Int64 totalValues, Int32 maxReadCount, ColumnRawData cd) line 253
    DataColumnReader.ReadDataPage(PageHeader ph, ColumnRawData cd, Int64 maxValues) line 216
    DataColumnReader.Read() line 88
    ParquetRowGroupReader.ReadColumn(DataField field) line 64
    ParquetReader.ReadEntireRowGroup(Int32 rowGroupIndex) line 141
    ParquetReaderTest.Reads_Exception() line 24

Here is Parquet.Thrift.PageHeader.ToString() for the last page of this column - formatted for readability:

PageHeader(, Type: DATA_PAGE,
    Uncompressed_page_size: 8, Compressed_page_size: 28,
    Crc: -680454176,
    Data_page_header: DataPageHeader(, Num_values: 125, Encoding: PLAIN_DICTIONARY,
        Definition_level_encoding: RLE, Repetition_level_encoding: BIT_PACKED,
        Statistics: Statistics(Null_count: 125)
    )
)

Call graph:

Parquet.File.DataColumnReader
\-> ReadDataPage
    \-> ReadPageData                - ungzips the page, produces 8 bytes
    \-> ReadLevels                  - consumes 7 bytes.
    \-> ReadColumn
        \-> ReadPlainDictionary     - consumes the last byte.
            \-> GetRemainingLength  - returns 0, as expected.
            \-> RunLengthBitPackingHybridValuesReader.ReadRleBitpackedHybrid    <boom>

That method interprets "length == 0" to mean "length is unknown, find it in the stream":

parquet-dotnet/src/Parquet/File/Values/RunLengthBitPackingHybridValuesReader.cs

Lines 35 to 37 in 60e4545

	public static int ReadRleBitpackedHybrid(BinaryReader reader, int bitWidth, int length, int[] dest, int offset, int pageSize)
	{
	if (length == 0) length = reader.ReadInt32();

But the length really is 0 and trying to read from the empty stream causes the boom.

If I just have ReadPlainDictionary skip the ReadRleBitpackedHybrid call in this case, then all is well - The file's remaining columns are read successfully. Is this a sensible solution?

[ishepherd]

This is the 4th page (3rd DATA_PAGE) of the column.

• every remaining value in the column is null
• this page was only emitted to make the numbers add up
  • Num_values: 125. When added to the previous page counts, this is the correct count for the column

I'm just guessing this is a fine and normal occurrence in a Parquet file. Python pandas reads this file without errors

[peteriehl]

I'm running into this problem also. @ishepherd were you able to verify your 2 assumptions?

[ishepherd]

@peteriehl I actually have a change that fixes it, but it's only on a private fork.

I did not raise the PR, because the file with the problem (used in the PR for a unit test) I am not able to share that file.

I think I should raise it anyway, even without a test, maybe you can build on that?

[tomasfalt]

Just to let people know if they run it to this problem. I got this problem when I tried to read a parquet file created by Spark (or at least I got the System.IO.EndOfStreamException : Unable to read beyond the end of the stream. error).

I solved it by using ParquetSharp instead of Parquet.Net and ParquetSharp read the the file perfectly. So until Parquet.Net solve this that might be solution for you @peteriehl.

[peteriehl]

Is the file something you could share so that unit tests can be written? There’s a code change for this, but neither I nor the author of the fix can share the file that proves the fix.

…

Sent from my iPhone

On Jan 15, 2021, at 03:03, tomasfalt ***@***.***> wrote:  Just to let people know if they run it to this problem. I got this problem when I tried to read a parquet file created by Spark (or at least I got the System.IO.EndOfStreamException : Unable to read beyond the end of the stream. error). I solved it by using ParquetSharp instead of Parquet.Net and ParquetSharp read the the file perfectly. So until Parquet.Net solve this that might be solution for you @peteriehl. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[tomasfalt]

Yes, but since it semi confidential I would rather send it to you privately. Please send your email to yewit41475@yutongdt.com and I will send it to you.