Parquet crashing on date column

[maxigit]

Describe the bug
Trying to load a parquet file with some date in it. Get this

ghci> D.readParquet "/home/max/Stats/Parquet/0_debtor_trans.parquet" 
*** Exception: UNKNOWN field ID for TimestampType3
CallStack (from HasCallStack):
  error, called at src/DataFrame/IO/Parquet/Thrift.hs:1151:18 in dataframe-0.4.1.0-49JzeyNWMOsLQT4sWvaYDS:DataFrame.IO.Parquet.Thrift

To Reproduce
Unfortunately, the data comes from production data which I can't give away and I don't really have time now to try to anonymize and/or extract data to reproduce the problem.

If it helps the parquet file has been created dumping a mysql database using python and duckdb. The table schema is

Field	Type	Null	Key	Default	Extra
trans_id	int(11)	NO	PRI	NULL	auto_increment
trans_no	int(11)	NO		0	
stock_id	char(20)	NO	MUL		
type	smallint(6)	NO	MUL	0	
loc_code	char(5)	NO			
tran_date	date	NO		0000-00-00	
person_id	int(11)	YES		NULL	
price	double	NO		0	
reference	char(40)	NO			
qty	double	NO		1	
discount_percent	double	NO		0	
standard_cost	double	NO		0	
visible	tinyint(1)	NO		1	

The issue seems to come from tran_date : date as every table with a date crashes.
readParquet seem to be expecting a timestamp where a simple date (possibly stored as a string) is given.

[daikonradish]

Thank you so much for this. Appreciate the detailed schema description.

[mchav]

Parquet has these things called logical types where even if it's a string it keeps some metadata about what kind of string it is - similar for numbers. The binary format that stores them has a lot of strange parsing rules so this seems like a simple fix.

You're right - the date is read as expected but the metadata read fails. I think I can reproduce this problem - I'll look for some data in the Parquet testing repo covering logical types. Thanks.

[mchav]

@maxigit the PR fixes my best guess of what could be the problem. Would appreciate if you could try it one more time. If your Parquet files are ZSTD compressed they still may fail but the metadata should read fine.

[maxigit]

I doesn't crash anymore but returns a null DataFrame instead

ghci> df <- D.readParquet "/home/max/Stats/Parquet/0_stock_moves.parquet" 
ghci> D.dimensions df
(0,0)

[mchav]

Sigh. Thanks for all this feedback - the parquet implementation clearly need a lot of work. To help my debugging here you could please sync to head and paste the output of:

ghci> import DataFrame.IO.Parquet
ghci> readMetadataFromPath "/home/max/Stats/Parquet/0_stock_moves.parquet" 

Parquet provides a testing repo (https://github.com/apache/parquet-testing) and the implementation word for a fair number of those files. Could potentially take learnings from here and create files with more test coverage.

[maxigit]

Sure

ghci> readMetadataFromPath  "/home/max/Stats/Parquet/0_stock_moves.parquet" 
FileMetaData {version = 2, schema = [SchemaElement {elementName = "schema", elementType = STOP, typeLength = 0, numChildren = 13, fieldId = -1, repetitionType = REQUIRED, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN},SchemaElement {elementName = "trans_id", elementType = I64, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN},SchemaElement {elementName = "trans_no", elementType = I64, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN},SchemaElement {elementName = "stock_id", elementType = STRING, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = STRING_TYPE},SchemaElement {elementName = "type", elementType = I64, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN},SchemaElement {elementName = "loc_code", elementType = STRING, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = STRING_TYPE},SchemaElement {elementName = "tran_date", elementType = I64, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = TimestampType {isAdjustedToUTC = False, unit = TIME_UNIT_UNKNOWN}},SchemaElement {elementName = "", elementType = STOP, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = UNKNOWN_REPETITION_TYPE, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN},SchemaElement {elementName = "person_id", elementType = I64, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN},SchemaElement {elementName = "price", elementType = DOUBLE, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN},SchemaElement {elementName = "reference", elementType = STRING, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = STRING_TYPE},SchemaElement {elementName = "qty", elementType = DOUBLE, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN},SchemaElement {elementName = "discount_percent", elementType = DOUBLE, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN},SchemaElement {elementName = "standard_cost", elementType = DOUBLE, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN}], numRows = 2, rowGroups = [], keyValueMetadata = [], createdBy = Just "visible", columnOrders = [], encryptionAlgorithm = ENCRYPTION_ALGORITHM_UNKNOWN, footerSigningKeyMetadata = ""}

[mchav]

There was an off by one byte reading error in the metadata parse so after reading the timestamp type (and not consuming the stopping bit) the next column was read as SchemaElement {elementName = "", elementType = STOP,.... I added a test with a timestamp type that's similar to yours and confirmed that it works. I'll push a similarly shaped file upstream into the Parquet testing repo.

Your trial is proving immensely helpful in hardening the Parquet implementation. Thanks.

[maxigit]

It's better now but crashing missing another pattern failure, possibly a compression issue.

ghci> readParquet "/home/max/Stats/Parquet/0_stock_moves.parquet" 
*** Exception: user error (Pattern match failure in 'do' block at src/DataFrame/IO/Parquet/Page.hs:42:21-34)

The metadata are

ghci> readMetadataFromPath  "/home/max/Stats/Parquet/0_stock_moves.parquet"
FileMetaData {version = 2, schema = [SchemaElement {elementName = "schema", elementType = STOP, typeLength = 0, numChildren = 13, fieldId = -1, repetitionType = REQUIRED, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN},SchemaElement {elementName = "trans_id", elementType = I64, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN},SchemaElement {elementName = "trans_no", elementType = I64, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN},SchemaElement {elementName = "stock_id", elementType = STRING, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = STRING_TYPE},SchemaElement {elementName = "type", elementType = I64, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN},SchemaElement {elementName = "loc_code", elementType = STRING, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = STRING_TYPE},SchemaElement {elementName = "tran_date", elementType = I64, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = TimestampType {isAdjustedToUTC = False, unit = NANOSECONDS}},SchemaElement {elementName = "person_id", elementType = I64, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN},SchemaElement {elementName = "price", elementType = DOUBLE, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN},SchemaElement {elementName = "reference", elementType = STRING, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = STRING_TYPE},SchemaElement {elementName = "qty", elementType = DOUBLE, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN},SchemaElement {elementName = "discount_percent", elementType = DOUBLE, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN},SchemaElement {elementName = "standard_cost", elementType = DOUBLE, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN},SchemaElement {elementName = "visible", elementType = I64, typeLength = 0, numChildren = 0, fieldId = -1, repetitionType = OPTIONAL, convertedType = 0, scale = 0, precision = 0, logicalType = LOGICAL_TYPE_UNKNOWN}], numRows = 480790, rowGroups = [RowGroup {rowGroupColumns = [ColumnChunk {columnChunkFilePath = "", columnChunkMetadataFileOffset = 0, columnMetaData = ColumnMetaData {columnType = PINT64, columnEncodings = [EPLAIN,ERLE,ERLE_DICTIONARY], columnPathInSchema = ["trans_id"], columnCodec = ZSTD, columnNumValues = 480790, columnTotalUncompressedSize = 4125433, columnTotalCompressedSize = 761696, columnKeyValueMetadata = [], columnDataPageOffset = 133982, columnIndexPageOffset = 0, columnDictionaryPageOffset = 4, columnStatistics = ColumnStatistics {columnMin = "\SOH\NUL\NUL\NUL\NUL\NUL\NUL\NUL", columnMax = "h\134\a\NUL\NUL\NUL\NUL\NUL", columnNullCount = 0, columnDistictCount = 0, columnMinValue = "\SOH\NUL\NUL\NUL\NUL\NUL\NUL\NUL", columnMaxValue = "h\134\a\NUL\NUL\NUL\NUL\NUL", isColumnMaxValueExact = False, isColumnMinValueExact = False}, columnEncodingStats = [PageEncodingStats {pageEncodingPageType = DICTIONARY_PAGE, pageEncoding = EPLAIN, pagesWithEncoding = 1},PageEncodingStats {pageEncodingPageType = DATA_PAGE, pageEncoding = EPLAIN, pagesWithEncoding = 3},PageEncodingStats {pageEncodingPageType = DATA_PAGE, pageEncoding = ERLE_DICTIONARY, pagesWithEncoding = 1}], bloomFilterOffset = 0, bloomFilterLength = 0, columnSizeStatistics = SizeStatisics {unencodedByteArrayDataTypes = 0, repetitionLevelHistogram = [], definitionLevelHistogram = [0,480790]}, columnGeospatialStatistics = GeospatialStatistics {bbox = BoundingBox {xmin = 0.0, xmax = 0.0, ymin = 0.0, ymax = 0.0, zmin = 0.0, zmax = 0.0, mmin = 0.0, mmax = 0.0}, geospatialTypes = []}}, columnChunkOffsetIndexOffset = 0, columnChunkOffsetIndexLength = 0, columnChunkColumnIndexOffset = 0, columnChunkColumnIndexLength = 0, cryptoMetadata = COLUMN_CRYPTO_METADATA_UNKNOWN, encryptedColumnMetadata = ""},ColumnChunk {columnChunkFilePath = "", columnChunkMetadataFileOffset = 0, columnMetaData = ColumnMetaData {columnType = PINT64, columnEncodings = [EPLAIN,ERLE,ERLE_DICTIONARY], columnPathInSchema = ["trans_no"], columnCodec = ZSTD, columnNumValues = 480790, columnTotalUncompressedSize = 669594, columnTotalCompressedSize = 207427, columnKeyValueMetadata = [], columnDataPageOffset = 806003, columnIndexPageOffset = 0, columnDictionaryPageOffset = 761700, columnStatistics = ColumnStatistics {columnMin = "\SOH\NUL\NUL\NUL\NUL\NUL\NUL\NUL", columnMax = "\141\166\NUL\NUL\NUL\NUL\NUL\NUL", columnNullCount = 0, columnDistictCount = 0, columnMinValue = "\SOH\NUL\NUL\NUL\NUL\NUL\NUL\NUL", columnMaxValue = "\141\166\NUL\NUL\NUL\NUL\NUL\NUL", isColumnMaxValueExact = False, isColumnMinValueExact = False}, columnEncodingStats = [PageEncodingStats {pageEncodingPageType = DICTIONARY_PAGE, pageEncoding = EPLAIN, pagesWithEncoding = 1},PageEncodingStats {pageEncodingPageType = DATA_PAGE, pageEncoding = ERLE_DICTIONARY, pagesWithEncoding = 1}], bloomFilterOffset = 0, bloomFilterLength = 0, columnSizeStatistics = SizeStatisics {unencodedByteArrayDataTypes = 0, repetitionLevelHistogram = [], definitionLevelHistogram = [0,480790]}, columnGeospatialStatistics = GeospatialStatistics {bbox = BoundingBox {xmin = 0.0, xmax = 0.0, ymin = 0.0, ymax = 0.0, zmin = 0.0, zmax = 0.0, mmin = 0.0, mmax = 0.0}, geospatialTypes = []}}, columnChunkOffsetIndexOffset = 0, columnChunkOffsetIndexLength = 0, columnChunkColumnIndexOffset = 0, columnChunkColumnIndexLength = 0, cryptoMetadata = COLUMN_CRYPTO_METADATA_UNKNOWN, encryptedColumnMetadata = ""},ColumnChunk {columnChunkFilePath = "", columnChunkMetadataFileOffset = 0, columnMetaData = ColumnMetaData {columnType = PBYTE_ARRAY, columnEncodings = [EPLAIN,ERLE,ERLE_DICTIONARY], columnPathInSchema = ["stock_id"], columnCodec = ZSTD, columnNumValues = 480790, columnTotalUncompressedSize = 991493, columnTotalCompressedSize = 832926, columnKeyValueMetadata = [], columnDataPageOffset = 1001361, columnIndexPageOffset = 0, columnDictionaryPageOffset = 969127, columnStatistics = ColumnStatistics {columnMin = "", columnMax = "", columnNullCount = 0, columnDistictCount = 0, columnMinValue = "AIS", columnMaxValue = "write-off-VAT", isColumnMaxValueExact = False, isColumnMinValueExact = False}, columnEncodingStats = [PageEncodingStats {pageEncodingPageType = DICTIONARY_PAGE, pageEncoding = EPLAIN, pagesWithEncoding = 1},PageEncodingStats {pageEncodingPageType = DATA_PAGE, pageEncoding = ERLE_DICTIONARY, pagesWithEncoding = 1}], bloomFilterOffset = 0, bloomFilterLength = 0, columnSizeStatistics = SizeStatisics {unencodedByteArrayDataTypes = 5751160, repetitionLevelHistogram = [], definitionLevelHistogram = [0,480790]}, columnGeospatialStatistics = GeospatialStatistics {bbox = BoundingBox {xmin = 0.0, xmax = 0.0, ymin = 0.0, ymax = 0.0, zmin = 0.0, zmax = 0.0, mmin = 0.0, mmax = 0.0}, geospatialTypes = []}}, columnChunkOffsetIndexOffset = 0, columnChunkOffsetIndexLength = 0, columnChunkColumnIndexOffset = 0, columnChunkColumnIndexLength = 0, cryptoMetadata = COLUMN_CRYPTO_METADATA_UNKNOWN, encryptedColumnMetadata = ""},ColumnChunk {columnChunkFilePath = "", columnChunkMetadataFileOffset = 0, columnMetaData = ColumnMetaData {columnType = PINT64, columnEncodings = [EPLAIN,ERLE,ERLE_DICTIONARY], columnPathInSchema = ["type"], columnCodec = ZSTD, columnNumValues = 480790, columnTotalUncompressedSize = 32481, columnTotalCompressedSize = 16919, columnKeyValueMetadata = [], columnDataPageOffset = 1802108, columnIndexPageOffset = 0, columnDictionaryPageOffset = 1802053, columnStatistics = ColumnStatistics {columnMin = "\v\NUL\NUL\NUL\NUL\NUL\NUL\NUL", columnMax = "\GS\NUL\NUL\NUL\NUL\NUL\NUL\NUL", columnNullCount = 0, columnDistictCount = 0, columnMinValue = "\v\NUL\NUL\NUL\NUL\NUL\NUL\NUL", columnMaxValue = "\GS\NUL\NUL\NUL\NUL\NUL\NUL\NUL", isColumnMaxValueExact = False, isColumnMinValueExact = False}, columnEncodingStats = [PageEncodingStats {pageEncodingPageType = DICTIONARY_PAGE, pageEncoding = EPLAIN, pagesWithEncoding = 1},PageEncodingStats {pageEncodingPageType = DATA_PAGE, pageEncoding = ERLE_DICTIONARY, pagesWithEncoding = 1}], bloomFilterOffset = 0, bloomFilterLength = 0, columnSizeStatistics = SizeStatisics {unencodedByteArrayDataTypes = 0, repetitionLevelHistogram = [], definitionLevelHistogram = [0,480790]}, columnGeospatialStatistics = GeospatialStatistics {bbox = BoundingBox {xmin = 0.0, xmax = 0.0, ymin = 0.0, ymax = 0.0, zmin = 0.0, zmax = 0.0, mmin = 0.0, mmax = 0.0}, geospatialTypes = []}}, columnChunkOffsetIndexOffset = 0, columnChunkOffsetIndexLength = 0, columnChunkColumnIndexOffset = 0, columnChunkColumnIndexLength = 0, cryptoMetadata = COLUMN_CRYPTO_METADATA_UNKNOWN, encryptedColumnMetadata = ""},ColumnChunk {columnChunkFilePath = "", columnChunkMetadataFileOffset = 0, columnMetaData = ColumnMetaData {columnType = PBYTE_ARRAY, columnEncodings = [EPLAIN,ERLE,ERLE_DICTIONARY], columnPathInSchema = ["loc_code"], columnCodec = ZSTD, columnNumValues = 480790, columnTotalUncompressedSize = 66514, columnTotalCompressedSize = 15167, columnKeyValueMetadata = [], columnDataPageOffset = 1819119, columnIndexPageOffset = 0, columnDictionaryPageOffset = 1818972, columnStatistics = ColumnStatistics {columnMin = "", columnMax = "", columnNullCount = 0, columnDistictCount = 0, columnMinValue = "", columnMaxValue = "WRONG", isColumnMaxValueExact = False, isColumnMinValueExact = False}, columnEncodingStats = [PageEncodingStats {pageEncodingPageType = DICTIONARY_PAGE, pageEncoding = EPLAIN, pagesWithEncoding = 1},PageEncodingStats {pageEncodingPageType = DATA_PAGE, pageEncoding = ERLE_DICTIONARY, pagesWithEncoding = 1}], bloomFilterOffset = 0, bloomFilterLength = 0, columnSizeStatistics = SizeStatisics {unencodedByteArrayDataTypes = 1407070, repetitionLevelHistogram = [], definitionLevelHistogram = [0,480790]}, columnGeospatialStatistics = GeospatialStatistics {bbox = BoundingBox {xmin = 0.0, xmax = 0.0, ymin = 0.0, ymax = 0.0, zmin = 0.0, zmax = 0.0, mmin = 0.0, mmax = 0.0}, geospatialTypes = []}}, columnChunkOffsetIndexOffset = 0, columnChunkOffsetIndexLength = 0, columnChunkColumnIndexOffset = 0, columnChunkColumnIndexLength = 0, cryptoMetadata = COLUMN_CRYPTO_METADATA_UNKNOWN, encryptedColumnMetadata = ""},ColumnChunk {columnChunkFilePath = "", columnChunkMetadataFileOffset = 0, columnMetaData = ColumnMetaData {columnType = PINT64, columnEncodings = [EPLAIN,ERLE,ERLE_DICTIONARY], columnPathInSchema = ["tran_date"], columnCodec = ZSTD, columnNumValues = 480790, columnTotalUncompressedSize = 74936, columnTotalCompressedSize = 59736, columnKeyValueMetadata = [], columnDataPageOffset = 1858902, columnIndexPageOffset = 0, columnDictionaryPageOffset = 1834139, columnStatistics = ColumnStatistics {columnMin = "\NUL\NUL&\205\SOHf\SUB\DC2", columnMax = "\NUL\NUL\207\153\180qg\CAN", columnNullCount = 0, columnDistictCount = 0, columnMinValue = "\NUL\NUL&\205\SOHf\SUB\DC2", columnMaxValue = "\NUL\NUL\207\153\180qg\CAN", isColumnMaxValueExact = False, isColumnMinValueExact = False}, columnEncodingStats = [PageEncodingStats {pageEncodingPageType = DICTIONARY_PAGE, pageEncoding = EPLAIN, pagesWithEncoding = 1},PageEncodingStats {pageEncodingPageType = DATA_PAGE, pageEncoding = ERLE_DICTIONARY, pagesWithEncoding = 1}], bloomFilterOffset = 0, bloomFilterLength = 0, columnSizeStatistics = SizeStatisics {unencodedByteArrayDataTypes = 0, repetitionLevelHistogram = [], definitionLevelHistogram = [0,480790]}, columnGeospatialStatistics = GeospatialStatistics {bbox = BoundingBox {xmin = 0.0, xmax = 0.0, ymin = 0.0, ymax = 0.0, zmin = 0.0, zmax = 0.0, mmin = 0.0, mmax = 0.0}, geospatialTypes = []}}, columnChunkOffsetIndexOffset = 0, columnChunkOffsetIndexLength = 0, columnChunkColumnIndexOffset = 0, columnChunkColumnIndexLength = 0, cryptoMetadata = COLUMN_CRYPTO_METADATA_UNKNOWN, encryptedColumnMetadata = ""},ColumnChunk {columnChunkFilePath = "", columnChunkMetadataFileOffset = 0, columnMetaData = ColumnMetaData {columnType = PINT64, columnEncodings = [EPLAIN,ERLE,ERLE_DICTIONARY], columnPathInSchema = ["person_id"], columnCodec = ZSTD, columnNumValues = 480790, columnTotalUncompressedSize = 35767, columnTotalCompressedSize = 15859, columnKeyValueMetadata = [], columnDataPageOffset = 1894038, columnIndexPageOffset = 0, columnDictionaryPageOffset = 1893875, columnStatistics = ColumnStatistics {columnMin = "\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL", columnMax = "`\NUL\NUL\NUL\NUL\NUL\NUL\NUL", columnNullCount = 0, columnDistictCount = 0, columnMinValue = "\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL", columnMaxValue = "`\NUL\NUL\NUL\NUL\NUL\NUL\NUL", isColumnMaxValueExact = False, isColumnMinValueExact = False}, columnEncodingStats = [PageEncodingStats {pageEncodingPageType = DICTIONARY_PAGE, pageEncoding = EPLAIN, pagesWithEncoding = 1},PageEncodingStats {pageEncodingPageType = DATA_PAGE, pageEncoding = ERLE_DICTIONARY, pagesWithEncoding = 1}], bloomFilterOffset = 0, bloomFilterLength = 0, columnSizeStatistics = SizeStatisics {unencodedByteArrayDataTypes = 0, repetitionLevelHistogram = [], definitionLevelHistogram = [0,480790]}, columnGeospatialStatistics = GeospatialStatistics {bbox = BoundingBox {xmin = 0.0, xmax = 0.0, ymin = 0.0, ymax = 0.0, zmin = 0.0, zmax = 0.0, mmin = 0.0, mmax = 0.0}, geospatialTypes = []}}, columnChunkOffsetIndexOffset = 0, columnChunkOffsetIndexLength = 0, columnChunkColumnIndexOffset = 0, columnChunkColumnIndexLength = 0, cryptoMetadata = COLUMN_CRYPTO_METADATA_UNKNOWN, encryptedColumnMetadata = ""},ColumnChunk {columnChunkFilePath = "", columnChunkMetadataFileOffset = 0, columnMetaData = ColumnMetaData {columnType = PDOUBLE, columnEncodings = [EPLAIN,ERLE,ERLE_DICTIONARY], columnPathInSchema = ["price"], columnCodec = ZSTD, columnNumValues = 480790, columnTotalUncompressedSize = 514476, columnTotalCompressedSize = 303156, columnKeyValueMetadata = [], columnDataPageOffset = 1921611, columnIndexPageOffset = 0, columnDictionaryPageOffset = 1909734, columnStatistics = ColumnStatistics {columnMin = "{\DC4\174G\225ZT\192", columnMax = "\NUL\NUL\NUL\NUL\NUL\136\195@", columnNullCount = 0, columnDistictCount = 0, columnMinValue = "{\DC4\174G\225ZT\192", columnMaxValue = "\NUL\NUL\NUL\NUL\NUL\136\195@", isColumnMaxValueExact = False, isColumnMinValueExact = False}, columnEncodingStats = [PageEncodingStats {pageEncodingPageType = DICTIONARY_PAGE, pageEncoding = EPLAIN, pagesWithEncoding = 1},PageEncodingStats {pageEncodingPageType = DATA_PAGE, pageEncoding = ERLE_DICTIONARY, pagesWithEncoding = 1}], bloomFilterOffset = 0, bloomFilterLength = 0, columnSizeStatistics = SizeStatisics {unencodedByteArrayDataTypes = 0, repetitionLevelHistogram = [], definitionLevelHistogram = [0,480790]}, columnGeospatialStatistics = GeospatialStatistics {bbox = BoundingBox {xmin = 0.0, xmax = 0.0, ymin = 0.0, ymax = 0.0, zmin = 0.0, zmax = 0.0, mmin = 0.0, mmax = 0.0}, geospatialTypes = []}}, columnChunkOffsetIndexOffset = 0, columnChunkOffsetIndexLength = 0, columnChunkColumnIndexOffset = 0, columnChunkColumnIndexLength = 0, cryptoMetadata = COLUMN_CRYPTO_METADATA_UNKNOWN, encryptedColumnMetadata = ""},ColumnChunk {columnChunkFilePath = "", columnChunkMetadataFileOffset = 0, columnMetaData = ColumnMetaData {columnType = PBYTE_ARRAY, columnEncodings = [EPLAIN,ERLE,ERLE_DICTIONARY], columnPathInSchema = ["reference"], columnCodec = ZSTD, columnNumValues = 480790, columnTotalUncompressedSize = 736470, columnTotalCompressedSize = 208449, columnKeyValueMetadata = [], columnDataPageOffset = 2270666, columnIndexPageOffset = 0, columnDictionaryPageOffset = 2212890, columnStatistics = ColumnStatistics {columnMin = "", columnMax = "", columnNullCount = 0, columnDistictCount = 0, columnMinValue = "", columnMaxValue = "r", isColumnMaxValueExact = False, isColumnMinValueExact = False}, columnEncodingStats = [PageEncodingStats {pageEncodingPageType = DICTIONARY_PAGE, pageEncoding = EPLAIN, pagesWithEncoding = 1},PageEncodingStats {pageEncodingPageType = DATA_PAGE, pageEncoding = ERLE_DICTIONARY, pagesWithEncoding = 1}], bloomFilterOffset = 0, bloomFilterLength = 0, columnSizeStatistics = SizeStatisics {unencodedByteArrayDataTypes = 2687354, repetitionLevelHistogram = [], definitionLevelHistogram = [0,480790]}, columnGeospatialStatistics = GeospatialStatistics {bbox = BoundingBox {xmin = 0.0, xmax = 0.0, ymin = 0.0, ymax = 0.0, zmin = 0.0, zmax = 0.0, mmin = 0.0, mmax = 0.0}, geospatialTypes = []}}, columnChunkOffsetIndexOffset = 0, columnChunkOffsetIndexLength = 0, columnChunkColumnIndexOffset = 0, columnChunkColumnIndexLength = 0, cryptoMetadata = COLUMN_CRYPTO_METADATA_UNKNOWN, encryptedColumnMetadata = ""},ColumnChunk {columnChunkFilePath = "", columnChunkMetadataFileOffset = 0, columnMetaData = ColumnMetaData {columnType = PDOUBLE, columnEncodings = [EPLAIN,ERLE,ERLE_DICTIONARY], columnPathInSchema = ["qty"], columnCodec = ZSTD, columnNumValues = 480790, columnTotalUncompressedSize = 432057, columnTotalCompressedSize = 169730, columnKeyValueMetadata = [], columnDataPageOffset = 2422903, columnIndexPageOffset = 0, columnDictionaryPageOffset = 2421339, columnStatistics = ColumnStatistics {columnMin = "\NUL\NUL\NUL\NUL\NUL&\203\192", columnMax = "\NUL\NUL\NUL\NUL\NUL&\203@", columnNullCount = 0, columnDistictCount = 0, columnMinValue = "\NUL\NUL\NUL\NUL\NUL&\203\192", columnMaxValue = "\NUL\NUL\NUL\NUL\NUL&\203@", isColumnMaxValueExact = False, isColumnMinValueExact = False}, columnEncodingStats = [PageEncodingStats {pageEncodingPageType = DICTIONARY_PAGE, pageEncoding = EPLAIN, pagesWithEncoding = 1},PageEncodingStats {pageEncodingPageType = DATA_PAGE, pageEncoding = ERLE_DICTIONARY, pagesWithEncoding = 1}], bloomFilterOffset = 0, bloomFilterLength = 0, columnSizeStatistics = SizeStatisics {unencodedByteArrayDataTypes = 0, repetitionLevelHistogram = [], definitionLevelHistogram = [0,480790]}, columnGeospatialStatistics = GeospatialStatistics {bbox = BoundingBox {xmin = 0.0, xmax = 0.0, ymin = 0.0, ymax = 0.0, zmin = 0.0, zmax = 0.0, mmin = 0.0, mmax = 0.0}, geospatialTypes = []}}, columnChunkOffsetIndexOffset = 0, columnChunkOffsetIndexLength = 0, columnChunkColumnIndexOffset = 0, columnChunkColumnIndexLength = 0, cryptoMetadata = COLUMN_CRYPTO_METADATA_UNKNOWN, encryptedColumnMetadata = ""},ColumnChunk {columnChunkFilePath = "", columnChunkMetadataFileOffset = 0, columnMetaData = ColumnMetaData {columnType = PDOUBLE, columnEncodings = [EPLAIN,ERLE,ERLE_DICTIONARY], columnPathInSchema = ["discount_percent"], columnCodec = ZSTD, columnNumValues = 480790, columnTotalUncompressedSize = 11154, columnTotalCompressedSize = 4516, columnKeyValueMetadata = [], columnDataPageOffset = 2591248, columnIndexPageOffset = 0, columnDictionaryPageOffset = 2591069, columnStatistics = ColumnStatistics {columnMin = "\NUL\NUL\NUL\NUL\NUL\NUL\NUL\128", columnMax = "\NUL\NUL\NUL\NUL\NUL\NUL\240?", columnNullCount = 0, columnDistictCount = 0, columnMinValue = "\NUL\NUL\NUL\NUL\NUL\NUL\NUL\128", columnMaxValue = "\NUL\NUL\NUL\NUL\NUL\NUL\240?", isColumnMaxValueExact = False, isColumnMinValueExact = False}, columnEncodingStats = [PageEncodingStats {pageEncodingPageType = DICTIONARY_PAGE, pageEncoding = EPLAIN, pagesWithEncoding = 1},PageEncodingStats {pageEncodingPageType = DATA_PAGE, pageEncoding = ERLE_DICTIONARY, pagesWithEncoding = 1}], bloomFilterOffset = 0, bloomFilterLength = 0, columnSizeStatistics = SizeStatisics {unencodedByteArrayDataTypes = 0, repetitionLevelHistogram = [], definitionLevelHistogram = [0,480790]}, columnGeospatialStatistics = GeospatialStatistics {bbox = BoundingBox {xmin = 0.0, xmax = 0.0, ymin = 0.0, ymax = 0.0, zmin = 0.0, zmax = 0.0, mmin = 0.0, mmax = 0.0}, geospatialTypes = []}}, columnChunkOffsetIndexOffset = 0, columnChunkOffsetIndexLength = 0, columnChunkColumnIndexOffset = 0, columnChunkColumnIndexLength = 0, cryptoMetadata = COLUMN_CRYPTO_METADATA_UNKNOWN, encryptedColumnMetadata = ""},ColumnChunk {columnChunkFilePath = "", columnChunkMetadataFileOffset = 0, columnMetaData = ColumnMetaData {columnType = PDOUBLE, columnEncodings = [EPLAIN,ERLE,ERLE_DICTIONARY], columnPathInSchema = ["standard_cost"], columnCodec = ZSTD, columnNumValues = 480790, columnTotalUncompressedSize = 900222, columnTotalCompressedSize = 836137, columnKeyValueMetadata = [], columnDataPageOffset = 2726797, columnIndexPageOffset = 0, columnDictionaryPageOffset = 2595585, columnStatistics = ColumnStatistics {columnMin = "\178\157\239\167FC\150\192", columnMax = "q=\n\215\255i\248@", columnNullCount = 0, columnDistictCount = 0, columnMinValue = "\178\157\239\167FC\150\192", columnMaxValue = "q=\n\215\255i\248@", isColumnMaxValueExact = False, isColumnMinValueExact = False}, columnEncodingStats = [PageEncodingStats {pageEncodingPageType = DICTIONARY_PAGE, pageEncoding = EPLAIN, pagesWithEncoding = 1},PageEncodingStats {pageEncodingPageType = DATA_PAGE, pageEncoding = ERLE_DICTIONARY, pagesWithEncoding = 1}], bloomFilterOffset = 0, bloomFilterLength = 0, columnSizeStatistics = SizeStatisics {unencodedByteArrayDataTypes = 0, repetitionLevelHistogram = [], definitionLevelHistogram = [0,480790]}, columnGeospatialStatistics = GeospatialStatistics {bbox = BoundingBox {xmin = 0.0, xmax = 0.0, ymin = 0.0, ymax = 0.0, zmin = 0.0, zmax = 0.0, mmin = 0.0, mmax = 0.0}, geospatialTypes = []}}, columnChunkOffsetIndexOffset = 0, columnChunkOffsetIndexLength = 0, columnChunkColumnIndexOffset = 0, columnChunkColumnIndexLength = 0, cryptoMetadata = COLUMN_CRYPTO_METADATA_UNKNOWN, encryptedColumnMetadata = ""},ColumnChunk {columnChunkFilePath = "", columnChunkMetadataFileOffset = 0, columnMetaData = ColumnMetaData {columnType = PINT64, columnEncodings = [EPLAIN,ERLE,ERLE_DICTIONARY], columnPathInSchema = ["visible"], columnCodec = ZSTD, columnNumValues = 480790, columnTotalUncompressedSize = 5197, columnTotalCompressedSize = 3419, columnKeyValueMetadata = [], columnDataPageOffset = 3431753, columnIndexPageOffset = 0, columnDictionaryPageOffset = 3431722, columnStatistics = ColumnStatistics {columnMin = "\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL", columnMax = "\SOH\NUL\NUL\NUL\NUL\NUL\NUL\NUL", columnNullCount = 0, columnDistictCount = 0, columnMinValue = "\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL", columnMaxValue = "\SOH\NUL\NUL\NUL\NUL\NUL\NUL\NUL", isColumnMaxValueExact = False, isColumnMinValueExact = False}, columnEncodingStats = [PageEncodingStats {pageEncodingPageType = DICTIONARY_PAGE, pageEncoding = EPLAIN, pagesWithEncoding = 1},PageEncodingStats {pageEncodingPageType = DATA_PAGE, pageEncoding = ERLE_DICTIONARY, pagesWithEncoding = 1}], bloomFilterOffset = 0, bloomFilterLength = 0, columnSizeStatistics = SizeStatisics {unencodedByteArrayDataTypes = 0, repetitionLevelHistogram = [], definitionLevelHistogram = [0,480790]}, columnGeospatialStatistics = GeospatialStatistics {bbox = BoundingBox {xmin = 0.0, xmax = 0.0, ymin = 0.0, ymax = 0.0, zmin = 0.0, zmax = 0.0, mmin = 0.0, mmax = 0.0}, geospatialTypes = []}}, columnChunkOffsetIndexOffset = 0, columnChunkOffsetIndexLength = 0, columnChunkColumnIndexOffset = 0, columnChunkColumnIndexLength = 0, cryptoMetadata = COLUMN_CRYPTO_METADATA_UNKNOWN, encryptedColumnMetadata = ""}], totalByteSize = 8595794, rowGroupNumRows = 480790, rowGroupSortingColumns = [], fileOffset = 4, totalCompressedSize = 3435137, ordinal = 0}], keyValueMetadata = [KeyValue {key = "pandas", value = "{\"index_columns\": [{\"kind\": \"range\", \"name\": null, \"start\": 0, \"stop\": 480790, \"step\": 1}], \"column_indexes\": [{\"name\": null, \"field_name\": null, \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": {\"encoding\": \"UTF-8\"}}], \"columns\": [{\"name\": \"trans_id\", \"field_name\": \"trans_id\", \"pandas_type\": \"int64\", \"numpy_type\": \"int64\", \"metadata\": null}, {\"name\": \"trans_no\", \"field_name\": \"trans_no\", \"pandas_type\": \"int64\", \"numpy_type\": \"int64\", \"metadata\": null}, {\"name\": \"stock_id\", \"field_name\": \"stock_id\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"type\", \"field_name\": \"type\", \"pandas_type\": \"int64\", \"numpy_type\": \"int64\", \"metadata\": null}, {\"name\": \"loc_code\", \"field_name\": \"loc_code\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"tran_date\", \"field_name\": \"tran_date\", \"pandas_type\": \"datetime\", \"numpy_type\": \"datetime64[ns]\", \"metadata\": null}, {\"name\": \"person_id\", \"field_name\": \"person_id\", \"pandas_type\": \"int64\", \"numpy_type\": \"int64\", \"metadata\": null}, {\"name\": \"price\", \"field_name\": \"price\", \"pandas_type\": \"float64\", \"numpy_type\": \"float64\", \"metadata\": null}, {\"name\": \"reference\", \"field_name\": \"reference\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"qty\", \"field_name\": \"qty\", \"pandas_type\": \"float64\", \"numpy_type\": \"float64\", \"metadata\": null}, {\"name\": \"discount_percent\", \"field_name\": \"discount_percent\", \"pandas_type\": \"float64\", \"numpy_type\": \"float64\", \"metadata\": null}, {\"name\": \"standard_cost\", \"field_name\": \"standard_cost\", \"pandas_type\": \"float64\", \"numpy_type\": \"float64\", \"metadata\": null}, {\"name\": \"visible\", \"field_name\": \"visible\", \"pandas_type\": \"int64\", \"numpy_type\": \"int64\", \"metadata\": null}], \"creator\": {\"library\": \"pyarrow\", \"version\": \"21.0.0\"}, \"pandas_version\": \"2.3.1\"}"},KeyValue {key = "ARROW:schema", value = "/////2AKAAAQAAAAAAAKAA4ABgAFAAgACgAAAAABBAAQAAAAAAAKAAwAAAAEAAgACgAAAGgHAAAEAAAAAQAAAAwAAAAIAAwABAAIAAgAAABABwAABAAAADAHAAB7ImluZGV4X2NvbHVtbnMiOiBbeyJraW5kIjogInJhbmdlIiwgIm5hbWUiOiBudWxsLCAic3RhcnQiOiAwLCAic3RvcCI6IDQ4MDc5MCwgInN0ZXAiOiAxfV0sICJjb2x1bW5faW5kZXhlcyI6IFt7Im5hbWUiOiBudWxsLCAiZmllbGRfbmFtZSI6IG51bGwsICJwYW5kYXNfdHlwZSI6ICJ1bmljb2RlIiwgIm51bXB5X3R5cGUiOiAib2JqZWN0IiwgIm1ldGFkYXRhIjogeyJlbmNvZGluZyI6ICJVVEYtOCJ9fV0sICJjb2x1bW5zIjogW3sibmFtZSI6ICJ0cmFuc19pZCIsICJmaWVsZF9uYW1lIjogInRyYW5zX2lkIiwgInBhbmRhc190eXBlIjogImludDY0IiwgIm51bXB5X3R5cGUiOiAiaW50NjQiLCAibWV0YWRhdGEiOiBudWxsfSwgeyJuYW1lIjogInRyYW5zX25vIiwgImZpZWxkX25hbWUiOiAidHJhbnNfbm8iLCAicGFuZGFzX3R5cGUiOiAiaW50NjQiLCAibnVtcHlfdHlwZSI6ICJpbnQ2NCIsICJtZXRhZGF0YSI6IG51bGx9LCB7Im5hbWUiOiAic3RvY2tfaWQiLCAiZmllbGRfbmFtZSI6ICJzdG9ja19pZCIsICJwYW5kYXNfdHlwZSI6ICJ1bmljb2RlIiwgIm51bXB5X3R5cGUiOiAib2JqZWN0IiwgIm1ldGFkYXRhIjogbnVsbH0sIHsibmFtZSI6ICJ0eXBlIiwgImZpZWxkX25hbWUiOiAidHlwZSIsICJwYW5kYXNfdHlwZSI6ICJpbnQ2NCIsICJudW1weV90eXBlIjogImludDY0IiwgIm1ldGFkYXRhIjogbnVsbH0sIHsibmFtZSI6ICJsb2NfY29kZSIsICJmaWVsZF9uYW1lIjogImxvY19jb2RlIiwgInBhbmRhc190eXBlIjogInVuaWNvZGUiLCAibnVtcHlfdHlwZSI6ICJvYmplY3QiLCAibWV0YWRhdGEiOiBudWxsfSwgeyJuYW1lIjogInRyYW5fZGF0ZSIsICJmaWVsZF9uYW1lIjogInRyYW5fZGF0ZSIsICJwYW5kYXNfdHlwZSI6ICJkYXRldGltZSIsICJudW1weV90eXBlIjogImRhdGV0aW1lNjRbbnNdIiwgIm1ldGFkYXRhIjogbnVsbH0sIHsibmFtZSI6ICJwZXJzb25faWQiLCAiZmllbGRfbmFtZSI6ICJwZXJzb25faWQiLCAicGFuZGFzX3R5cGUiOiAiaW50NjQiLCAibnVtcHlfdHlwZSI6ICJpbnQ2NCIsICJtZXRhZGF0YSI6IG51bGx9LCB7Im5hbWUiOiAicHJpY2UiLCAiZmllbGRfbmFtZSI6ICJwcmljZSIsICJwYW5kYXNfdHlwZSI6ICJmbG9hdDY0IiwgIm51bXB5X3R5cGUiOiAiZmxvYXQ2NCIsICJtZXRhZGF0YSI6IG51bGx9LCB7Im5hbWUiOiAicmVmZXJlbmNlIiwgImZpZWxkX25hbWUiOiAicmVmZXJlbmNlIiwgInBhbmRhc190eXBlIjogInVuaWNvZGUiLCAibnVtcHlfdHlwZSI6ICJvYmplY3QiLCAibWV0YWRhdGEiOiBudWxsfSwgeyJuYW1lIjogInF0eSIsICJmaWVsZF9uYW1lIjogInF0eSIsICJwYW5kYXNfdHlwZSI6ICJmbG9hdDY0IiwgIm51bXB5X3R5cGUiOiAiZmxvYXQ2NCIsICJtZXRhZGF0YSI6IG51bGx9LCB7Im5hbWUiOiAiZGlzY291bnRfcGVyY2VudCIsICJmaWVsZF9uYW1lIjogImRpc2NvdW50X3BlcmNlbnQiLCAicGFuZGFzX3R5cGUiOiAiZmxvYXQ2NCIsICJudW1weV90eXBlIjogImZsb2F0NjQiLCAibWV0YWRhdGEiOiBudWxsfSwgeyJuYW1lIjogInN0YW5kYXJkX2Nvc3QiLCAiZmllbGRfbmFtZSI6ICJzdGFuZGFyZF9jb3N0IiwgInBhbmRhc190eXBlIjogImZsb2F0NjQiLCAibnVtcHlfdHlwZSI6ICJmbG9hdDY0IiwgIm1ldGFkYXRhIjogbnVsbH0sIHsibmFtZSI6ICJ2aXNpYmxlIiwgImZpZWxkX25hbWUiOiAidmlzaWJsZSIsICJwYW5kYXNfdHlwZSI6ICJpbnQ2NCIsICJudW1weV90eXBlIjogImludDY0IiwgIm1ldGFkYXRhIjogbnVsbH1dLCAiY3JlYXRvciI6IHsibGlicmFyeSI6ICJweWFycm93IiwgInZlcnNpb24iOiAiMjEuMC4wIn0sICJwYW5kYXNfdmVyc2lvbiI6ICIyLjMuMSJ9AAAAAAYAAABwYW5kYXMAAA0AAACIAgAAQAIAAAwCAADYAQAAqAEAAHABAAA4AQAACAEAANgAAACsAAAAcAAAADgAAAAEAAAAvP3//wAAAQIQAAAAGAAAAAQAAAAAAAAABwAAAHZpc2libGUAqP3//wAAAAFAAAAA7P3//wAAAQMQAAAAIAAAAAQAAAAAAAAADQAAAHN0YW5kYXJkX2Nvc3QAAADm/v//AAACACD+//8AAAEDEAAAACQAAAAEAAAAAAAAABAAAABkaXNjb3VudF9wZXJjZW50AAAAAB7///8AAAIAWP7//wAAAQMQAAAAFAAAAAQAAAAAAAAAAwAAAHF0eQBG////AAACAID+//8AAAEFEAAAABwAAAAEAAAAAAAAAAkAAAByZWZlcmVuY2UAAADk/v//rP7//wAAAQMQAAAAGAAAAAQAAAAAAAAABQAAAHByaWNlAAAAnv///wAAAgDY/v//AAABAhAAAAAcAAAABAAAAAAAAAAJAAAAcGVyc29uX2lkAAAAyP7//wAAAAFAAAAADP///wAAAQoQAAAAIAAAAAQAAAAAAAAACQAAAHRyYW5fZGF0ZQAGAAgABgAGAAAAAAADAED///8AAAEFEAAAABwAAAAEAAAAAAAAAAgAAABsb2NfY29kZQAAAACk////bP///wAAAQIQAAAAGAAAAAQAAAAAAAAABAAAAHR5cGUAAAAAWP///wAAAAFAAAAAnP///wAAAQUQAAAAIAAAAAQAAAAAAAAACAAAAHN0b2NrX2lkAAAAAAQABAAEAAAAzP///wAAAQIQAAAAHAAAAAQAAAAAAAAACAAAAHRyYW5zX25vAAAAALz///8AAAABQAAAABAAFAAIAAYABwAMAAAAEAAQAAAAAAABAhAAAAAkAAAABAAAAAAAAAAIAAAAdHJhbnNfaWQAAAAACAAMAAgABwAIAAAAAAAAAUAAAAAAAAAA"}], createdBy = Just "parquet-cpp-arrow version 21.0.0", columnOrders = [TYPE_ORDER,TYPE_ORDER,TYPE_ORDER,TYPE_ORDER,TYPE_ORDER,TYPE_ORDER,TYPE_ORDER,TYPE_ORDER,TYPE_ORDER,TYPE_ORDER,TYPE_ORDER,TYPE_ORDER,TYPE_ORDER], encryptionAlgorithm = ENCRYPTION_ALGORITHM_UNKNOWN, footerSigningKeyMetadata = ""}

[mchav]

Great. Now the only problem is the decompression - similar to the other issue you posted. I think that could either be an error with the record shredding algorithm from dremel or a problem with zstd decompression header. Let me see if I can generate some python DuckDb files that replicate your schema.

[mchav]

Generated a file that approximates your data using the metadata. Fixed the ZSTD streaming issue and a little quirk with logical types - should work for your case now.

That said it's still really slow (3 minutes to read the ~500k rows on my computer when compiled with -O2) for obvious reasons:

• Column chunks are concatenated which means there is A LOT of copying and building columns in N^2.
• The intermediate representation for computation is lists which are terrible for memory locality in various ways.

We've only been working with small, well-behaved Parquet files so it hasn't been a problem. These are issues on our radar leading up to Google Summer of Code where we expect to have someone work on the Parquet writer (see #133). I think this should work for your data now though and I'll ship some optimizations to get that read time down to ~20 seconds by the end of the week.