Skip to content

[SPARK-56896][SQL] Add bulk read paths for timestamp/date Parquet vector updaters #55923

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
iemejia wants to merge 21 commits into from
Closed

[SPARK-56896][SQL] Add bulk read paths for timestamp/date Parquet vector updaters #55923

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
cb2db60
[SPARK-56896][SQL] Add bulk read paths for timestamp/date Parquet vec…
iemejia May 17, 2026
00bee3f
Address review: add LongAsNanos to Group B doc, use production timeTy…
iemejia Jun 10, 2026
d2a6a31
Add bulk read paths for LongAsMicros, LongAsNanos, LongAsMicrosRebase…
iemejia Jun 10, 2026
4069653
Benchmark results for *ParquetVectorUpdaterBenchmark (JDK 17, Scala 2…
iemejia Jun 10, 2026
49848da
Benchmark results for *ParquetVectorUpdaterBenchmark (JDK 21, Scala 2…
iemejia Jun 10, 2026
80a1da2
Benchmark results for *ParquetVectorUpdaterBenchmark (JDK 25, Scala 2…
iemejia Jun 12, 2026
f39a8f2
Benchmark results for *ParquetVectorUpdaterBenchmark (JDK 21, Scala 2…
iemejia Jun 12, 2026
918d571
Benchmark results for *ParquetVectorUpdaterBenchmark (JDK 17, Scala 2…
iemejia Jun 12, 2026
fc8b1e2
Benchmark results for *ParquetVectorUpdaterBenchmark (JDK 25, Scala 2…
iemejia Jun 12, 2026
12e841b
Fix scalastyle: wrap long line in benchmark descriptor
iemejia Jun 12, 2026
db81c2d
Migrate DateToTimestampNTZUpdater to two-pass pattern and remove read…
iemejia Jun 12, 2026
3984f6b
Benchmark results for *ParquetVectorUpdaterBenchmark (JDK 17, Scala 2…
iemejia Jun 12, 2026
6e8d43d
Benchmark results for *ParquetVectorUpdaterBenchmark (JDK 21, Scala 2…
iemejia Jun 12, 2026
9c7d932
Benchmark results for *ParquetVectorUpdaterBenchmark (JDK 25, Scala 2…
iemejia Jun 12, 2026
4a17eb3
Benchmark results for *ParquetVectorUpdaterBenchmark (JDK 17, Scala 2…
iemejia Jun 12, 2026
8e48656
Benchmark results for *ParquetVectorUpdaterBenchmark (JDK 25, Scala 2…
iemejia Jun 12, 2026
8c70c2a
Benchmark results for *ParquetVectorUpdaterBenchmark (JDK 25, Scala 2…
iemejia Jun 12, 2026
15c9703
Benchmark results for *ParquetVectorUpdaterBenchmark (JDK 25, Scala 2…
iemejia Jun 12, 2026
dd406c6
Benchmark results for *ParquetVectorUpdaterBenchmark (JDK 25, Scala 2…
iemejia Jun 12, 2026
b7e2f11
Benchmark results for *ParquetVectorUpdaterBenchmark (JDK 25, Scala 2…
iemejia Jun 12, 2026
88cfb7f
Benchmark results for *ParquetVectorUpdaterBenchmark (JDK 25, Scala 2…
iemejia Jun 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 35 additions & 32 deletions sql/core/benchmarks/ParquetVectorUpdaterBenchmark-jdk21-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,83 +2,86 @@
Identity Updaters
================================================================================================

OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1013-azure
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
AMD EPYC 7763 64-Core Processor
Identity Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
BooleanUpdater 0 0 0 16990.6 0.1 1.0X
ByteUpdater (INT32 -> Byte) 0 0 0 3765.0 0.3 0.2X
ShortUpdater (INT32 -> Short) 1 1 0 1682.9 0.6 0.1X
IntegerUpdater 0 0 0 7756.2 0.1 0.5X
LongUpdater 0 0 0 3870.4 0.3 0.2X
FloatUpdater 0 0 0 7758.5 0.1 0.5X
DoubleUpdater 0 0 0 3875.9 0.3 0.2X
BinaryUpdater 15 15 0 70.4 14.2 0.0X
BooleanUpdater 0 0 0 16982.4 0.1 1.0X
ByteUpdater (INT32 -> Byte) 0 0 0 3744.9 0.3 0.2X
ShortUpdater (INT32 -> Short) 1 1 0 1675.0 0.6 0.1X
IntegerUpdater 0 0 0 10248.0 0.1 0.6X
LongUpdater 0 0 0 5141.4 0.2 0.3X
FloatUpdater 0 0 0 10286.2 0.1 0.6X
DoubleUpdater 0 0 0 5139.3 0.2 0.3X
BinaryUpdater 15 15 0 71.1 14.1 0.0X


================================================================================================
Type-converting Updaters
================================================================================================

OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1013-azure
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
AMD EPYC 7763 64-Core Processor
Type-converting Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
IntegerToLongUpdater 0 0 0 5133.8 0.2 1.0X
IntegerToDoubleUpdater 0 0 0 6090.4 0.2 1.2X
FloatToDoubleUpdater 0 0 0 2527.1 0.4 0.5X
DateToTimestampNTZUpdater 1 1 0 934.8 1.1 0.2X
DowncastLongUpdater (INT64 -> Decimal(9,2)) 0 0 0 5108.5 0.2 1.0X
IntegerToLongUpdater 0 0 0 6158.1 0.2 1.0X
IntegerToDoubleUpdater 0 0 0 6228.1 0.2 1.0X
FloatToDoubleUpdater 0 0 0 2525.4 0.4 0.4X
DateToTimestampNTZUpdater 1 1 0 932.9 1.1 0.2X
LongAsNanosUpdater (TimeType) 1 1 0 1228.5 0.8 0.2X
DowncastLongUpdater (INT64 -> Decimal(9,2)) 0 0 0 5861.5 0.2 1.0X


================================================================================================
Rebase Updaters
================================================================================================

OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1013-azure
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
AMD EPYC 7763 64-Core Processor
Rebase Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------
IntegerWithRebaseUpdater (DATE legacy) 0 0 0 3263.0 0.3 1.0X
LongWithRebaseUpdater (TIMESTAMP_MICROS legacy) 0 0 0 2282.0 0.4 0.7X
LongAsMicrosUpdater (TIMESTAMP_MILLIS) 2 3 0 420.5 2.4 0.1X
Rebase Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------
IntegerWithRebaseUpdater (DATE legacy) 0 0 0 3647.5 0.3 1.0X
LongWithRebaseUpdater (TIMESTAMP_MICROS legacy) 0 0 0 2668.9 0.4 0.7X
LongAsMicrosUpdater (TIMESTAMP_MILLIS) 1 1 0 1228.3 0.8 0.3X
DateToTimestampNTZWithRebaseUpdater (DATE legacy) 1 1 0 797.7 1.3 0.2X
LongAsMicrosRebaseUpdater (TIMESTAMP_MILLIS legacy) 1 1 0 1099.3 0.9 0.3X


================================================================================================
Unsigned Updaters
================================================================================================

OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1013-azure
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
AMD EPYC 7763 64-Core Processor
Unsigned Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------
UnsignedIntegerUpdater (UINT32 -> Long) 0 0 0 5112.2 0.2 1.0X
UnsignedLongUpdater (UINT64 -> Decimal(20,0)) 16 17 0 63.9 15.7 0.0X
UnsignedIntegerUpdater (UINT32 -> Long) 0 0 0 5894.2 0.2 1.0X
UnsignedLongUpdater (UINT64 -> Decimal(20,0)) 17 18 1 60.3 16.6 0.0X


================================================================================================
Decimal Updaters
================================================================================================

OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1013-azure
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
AMD EPYC 7763 64-Core Processor
Decimal Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
IntegerToDecimalUpdater 0 0 0 7750.4 0.1 1.0X
LongToDecimalUpdater 0 0 0 3866.5 0.3 0.5X
FixedLenByteArrayToDecimalUpdater 21 21 0 50.1 19.9 0.0X
IntegerToDecimalUpdater 0 0 0 10291.3 0.1 1.0X
LongToDecimalUpdater 0 0 0 5139.6 0.2 0.5X
FixedLenByteArrayToDecimalUpdater 21 21 0 49.6 20.2 0.0X


================================================================================================
FixedLenByteArray Updaters
================================================================================================

OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1013-azure
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
AMD EPYC 7763 64-Core Processor
FixedLenByteArray Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------------
FixedLenByteArrayUpdater (len=16 -> Binary) 20 20 0 51.7 19.3 1.0X
FixedLenByteArrayAsIntUpdater (len=4 -> Decimal(9,2)) 7 7 1 160.1 6.2 3.1X
FixedLenByteArrayAsLongUpdater (len=8 -> Decimal(18,4)) 8 8 0 133.2 7.5 2.6X
FixedLenByteArrayUpdater (len=16 -> Binary) 20 21 2 51.9 19.3 1.0X
FixedLenByteArrayAsIntUpdater (len=4 -> Decimal(9,2)) 7 7 0 160.2 6.2 3.1X
FixedLenByteArrayAsLongUpdater (len=8 -> Decimal(18,4)) 8 8 0 133.1 7.5 2.6X


Loading