Skip to content

ARROW-10696: [C++] Add SetBitRunReader#8770

Closed
pitrou wants to merge 3 commits into
apache:masterfrom
pitrou:ARROW-10696-set-bit-run-reader
Closed

ARROW-10696: [C++] Add SetBitRunReader#8770
pitrou wants to merge 3 commits into
apache:masterfrom
pitrou:ARROW-10696-set-bit-run-reader

Conversation

@pitrou

@pitrou pitrou commented Nov 25, 2020

Copy link
Copy Markdown
Member

A specialized bitmap reader that yields runs of set bits, for use cases where reset bits (e.g. null bits) don't need any handling.

On some use cases it can be significantly faster than the alternatives.

@github-actions

github-actions Bot commented Nov 25, 2020

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-10696

@pitrou

pitrou commented Nov 25, 2020 •
edited
Loading

Copy link
Copy Markdown
Member Author

ArrayRangeEquals benchmarks:

                                      benchmark            baseline           contender  change %                                                                                                                                                                                counters
22             ArrayRangeEqualsStruct/32768/100   23.863m items/sec  343.067m items/sec  1337.631                {'run_name': 'ArrayRangeEqualsStruct/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 510, 'null_percent': 1.0}
5     ArrayRangeEqualsFixedSizeBinary/32768/100  614.186m items/sec    5.185b items/sec   744.270     {'run_name': 'ArrayRangeEqualsFixedSizeBinary/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 12979, 'null_percent': 1.0}
33        ArrayRangeEqualsListOfInt32/32768/100   26.215m items/sec  215.239m items/sec   721.041           {'run_name': 'ArrayRangeEqualsListOfInt32/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 560, 'null_percent': 1.0}
21          ArrayRangeEqualsBoolean/32768/10000    2.826b items/sec   19.479b items/sec   589.262          {'run_name': 'ArrayRangeEqualsBoolean/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 61059, 'null_percent': 0.01}
51           ArrayRangeEqualsStruct/32768/10000  222.532m items/sec    1.483b items/sec   566.535            {'run_name': 'ArrayRangeEqualsStruct/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 4789, 'null_percent': 0.01}
47             ArrayRangeEqualsString/32768/100  252.039m items/sec    1.350b items/sec   435.816               {'run_name': 'ArrayRangeEqualsString/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 5365, 'null_percent': 1.0}
20              ArrayRangeEqualsStruct/32768/10   12.987m items/sec   61.633m items/sec   374.582                {'run_name': 'ArrayRangeEqualsStruct/32768/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 277, 'null_percent': 10.0}
36            ArrayRangeEqualsFloat32/32768/100    1.096b items/sec    4.474b items/sec   308.366             {'run_name': 'ArrayRangeEqualsFloat32/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 23270, 'null_percent': 1.0}
29              ArrayRangeEqualsInt32/32768/100    1.675b items/sec    6.533b items/sec   290.081               {'run_name': 'ArrayRangeEqualsInt32/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 35608, 'null_percent': 1.0}
46              ArrayRangeEqualsString/32768/10  135.386m items/sec  526.754m items/sec   289.076               {'run_name': 'ArrayRangeEqualsString/32768/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 2853, 'null_percent': 10.0}
38         ArrayRangeEqualsListOfInt32/32768/10   12.637m items/sec   43.741m items/sec   246.142           {'run_name': 'ArrayRangeEqualsListOfInt32/32768/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 269, 'null_percent': 10.0}
28     ArrayRangeEqualsFixedSizeBinary/32768/10  294.337m items/sec  969.941m items/sec   229.534      {'run_name': 'ArrayRangeEqualsFixedSizeBinary/32768/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 6252, 'null_percent': 10.0}
42      ArrayRangeEqualsListOfInt32/32768/10000  202.207m items/sec  656.251m items/sec   224.544       {'run_name': 'ArrayRangeEqualsListOfInt32/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 4360, 'null_percent': 0.01}
13            ArrayRangeEqualsBoolean/32768/100  979.715m items/sec    2.908b items/sec   196.868             {'run_name': 'ArrayRangeEqualsBoolean/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 21042, 'null_percent': 1.0}
40              ArrayRangeEqualsBoolean/32768/1   12.017b items/sec   35.217b items/sec   193.068            {'run_name': 'ArrayRangeEqualsBoolean/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 256448, 'null_percent': 100.0}
45      ArrayRangeEqualsFixedSizeBinary/32768/1   12.204b items/sec   35.234b items/sec   188.703    {'run_name': 'ArrayRangeEqualsFixedSizeBinary/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 269241, 'null_percent': 100.0}
17                ArrayRangeEqualsInt32/32768/1   12.406b items/sec   34.810b items/sec   180.598              {'run_name': 'ArrayRangeEqualsInt32/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 265638, 'null_percent': 100.0}
26          ArrayRangeEqualsListOfInt32/32768/1   12.626b items/sec   35.288b items/sec   179.485        {'run_name': 'ArrayRangeEqualsListOfInt32/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 269942, 'null_percent': 100.0}
44               ArrayRangeEqualsString/32768/1   13.196b items/sec   35.124b items/sec   166.178             {'run_name': 'ArrayRangeEqualsString/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 281694, 'null_percent': 100.0}
35              ArrayRangeEqualsFloat32/32768/1   12.898b items/sec   34.159b items/sec   164.846            {'run_name': 'ArrayRangeEqualsFloat32/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 277754, 'null_percent': 100.0}
53               ArrayRangeEqualsStruct/32768/1   13.618b items/sec   35.687b items/sec   162.059             {'run_name': 'ArrayRangeEqualsStruct/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 291514, 'null_percent': 100.0}
15  ArrayRangeEqualsFixedSizeBinary/32768/10000    4.166b items/sec   10.835b items/sec   160.089  {'run_name': 'ArrayRangeEqualsFixedSizeBinary/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 89321, 'null_percent': 0.01}
16            ArrayRangeEqualsInt32/32768/10000    6.886b items/sec   15.942b items/sec   131.514           {'run_name': 'ArrayRangeEqualsInt32/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 146029, 'null_percent': 0.01}
19               ArrayRangeEqualsStruct/32768/0  748.134m items/sec    1.686b items/sec   125.426                {'run_name': 'ArrayRangeEqualsStruct/32768/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 16023, 'null_percent': 0.0}
32      ArrayRangeEqualsFixedSizeBinary/32768/2  196.551m items/sec  375.563m items/sec    91.077       {'run_name': 'ArrayRangeEqualsFixedSizeBinary/32768/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 4153, 'null_percent': 50.0}
49               ArrayRangeEqualsString/32768/2  139.017m items/sec  239.038m items/sec    71.948                {'run_name': 'ArrayRangeEqualsString/32768/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 2953, 'null_percent': 50.0}
31          ArrayRangeEqualsFloat32/32768/10000    4.639b items/sec    7.537b items/sec    62.473          {'run_name': 'ArrayRangeEqualsFloat32/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 96758, 'null_percent': 0.01}
39          ArrayRangeEqualsListOfInt32/32768/2   12.135m items/sec   19.630m items/sec    61.761            {'run_name': 'ArrayRangeEqualsListOfInt32/32768/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 259, 'null_percent': 50.0}
2                ArrayRangeEqualsStruct/32768/2   21.952m items/sec   34.669m items/sec    57.935                 {'run_name': 'ArrayRangeEqualsStruct/32768/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 465, 'null_percent': 50.0}
1                ArrayRangeEqualsInt32/32768/10  678.038m items/sec    1.057b items/sec    55.821               {'run_name': 'ArrayRangeEqualsInt32/32768/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 14391, 'null_percent': 10.0}
12              ArrayRangeEqualsFloat32/32768/2  271.242m items/sec  394.340m items/sec    45.383               {'run_name': 'ArrayRangeEqualsFloat32/32768/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 5633, 'null_percent': 50.0}
24                ArrayRangeEqualsInt32/32768/2  274.259m items/sec  393.021m items/sec    43.303                 {'run_name': 'ArrayRangeEqualsInt32/32768/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 5764, 'null_percent': 50.0}
50              ArrayRangeEqualsBoolean/32768/2  221.275m items/sec  307.534m items/sec    38.983               {'run_name': 'ArrayRangeEqualsBoolean/32768/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 4708, 'null_percent': 50.0}
48           ArrayRangeEqualsString/32768/10000    1.330b items/sec    1.816b items/sec    36.476           {'run_name': 'ArrayRangeEqualsString/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 28580, 'null_percent': 0.01}
8           ArrayRangeEqualsSparseUnion/32768/0   34.852m items/sec   45.628m items/sec    30.919             {'run_name': 'ArrayRangeEqualsSparseUnion/32768/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 737, 'null_percent': 0.0}
23           ArrayRangeEqualsDenseUnion/32768/0   36.416m items/sec   45.582m items/sec    25.171              {'run_name': 'ArrayRangeEqualsDenseUnion/32768/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 773, 'null_percent': 0.0}
9           ArrayRangeEqualsSparseUnion/32768/1   22.389m items/sec   27.076m items/sec    20.932           {'run_name': 'ArrayRangeEqualsSparseUnion/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 478, 'null_percent': 100.0}
14             ArrayRangeEqualsFloat32/32768/10  569.970m items/sec  660.566m items/sec    15.895             {'run_name': 'ArrayRangeEqualsFloat32/32768/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 11970, 'null_percent': 10.0}
11           ArrayRangeEqualsDenseUnion/32768/1   23.336m items/sec   25.912m items/sec    11.037            {'run_name': 'ArrayRangeEqualsDenseUnion/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 496, 'null_percent': 100.0}
0               ArrayRangeEqualsFloat32/32768/0   10.265b items/sec   11.049b items/sec     7.641              {'run_name': 'ArrayRangeEqualsFloat32/32768/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 218383, 'null_percent': 0.0}
43          ArrayRangeEqualsListOfInt32/32768/0  790.628m items/sec  838.173m items/sec     6.014           {'run_name': 'ArrayRangeEqualsListOfInt32/32768/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 17131, 'null_percent': 0.0}
27              ArrayRangeEqualsBoolean/32768/0   48.625b items/sec   51.207b items/sec     5.309             {'run_name': 'ArrayRangeEqualsBoolean/32768/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 1034591, 'null_percent': 0.0}
6                ArrayRangeEqualsString/32768/0    1.847b items/sec    1.942b items/sec     5.149                {'run_name': 'ArrayRangeEqualsString/32768/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 39452, 'null_percent': 0.0}
10      ArrayRangeEqualsFixedSizeBinary/32768/0   15.947b items/sec   16.438b items/sec     3.075      {'run_name': 'ArrayRangeEqualsFixedSizeBinary/32768/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 337925, 'null_percent': 0.0}
30          ArrayRangeEqualsSparseUnion/32768/2   20.058m items/sec   20.362m items/sec     1.514            {'run_name': 'ArrayRangeEqualsSparseUnion/32768/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 427, 'null_percent': 50.0}
3          ArrayRangeEqualsSparseUnion/32768/10   20.014m items/sec   20.036m items/sec     0.109           {'run_name': 'ArrayRangeEqualsSparseUnion/32768/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 425, 'null_percent': 10.0}
7         ArrayRangeEqualsSparseUnion/32768/100   20.000m items/sec   19.925m items/sec    -0.373           {'run_name': 'ArrayRangeEqualsSparseUnion/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 424, 'null_percent': 1.0}
25      ArrayRangeEqualsSparseUnion/32768/10000   19.848m items/sec   19.658m items/sec    -0.958        {'run_name': 'ArrayRangeEqualsSparseUnion/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 422, 'null_percent': 0.01}
37                ArrayRangeEqualsInt32/32768/0   31.782b items/sec   31.411b items/sec    -1.168                {'run_name': 'ArrayRangeEqualsInt32/32768/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 665046, 'null_percent': 0.0}
4              ArrayRangeEqualsBoolean/32768/10  512.687m items/sec  496.923m items/sec    -3.075             {'run_name': 'ArrayRangeEqualsBoolean/32768/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 10930, 'null_percent': 10.0}
18         ArrayRangeEqualsDenseUnion/32768/100   20.641m items/sec   19.655m items/sec    -4.777            {'run_name': 'ArrayRangeEqualsDenseUnion/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 440, 'null_percent': 1.0}
52       ArrayRangeEqualsDenseUnion/32768/10000   20.665m items/sec   19.609m items/sec    -5.113         {'run_name': 'ArrayRangeEqualsDenseUnion/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 442, 'null_percent': 0.01}
41          ArrayRangeEqualsDenseUnion/32768/10   20.551m items/sec   19.388m items/sec    -5.658            {'run_name': 'ArrayRangeEqualsDenseUnion/32768/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 441, 'null_percent': 10.0}
34           ArrayRangeEqualsDenseUnion/32768/2   20.645m items/sec   19.374m items/sec    -6.156             {'run_name': 'ArrayRangeEqualsDenseUnion/32768/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 440, 'null_percent': 50.0}

@pitrou

pitrou commented Nov 25, 2020 •
edited
Loading

Copy link
Copy Markdown
Member Author

Parquet benchmarks:

                                                                    benchmark            baseline           contender  change %                                                                                                                                                                                       counters
199                                     BM_PlainDecodingSpacedFloat/32768/100       5.506 GiB/sec      18.466 GiB/sec   235.349                {'run_name': 'BM_PlainDecodingSpacedFloat/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 31377, 'null_percent': 1.0}
177                                 BM_PlainEncodingSpacedBoolean/32768/10000      11.377 GiB/sec      33.462 GiB/sec   194.106         {'run_name': 'BM_PlainEncodingSpacedBoolean/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 259382, 'null_percent': 100.0}
179                                   BM_PlainDecodingSpacedBoolean/32768/100       1.358 GiB/sec       3.572 GiB/sec   163.120              {'run_name': 'BM_PlainDecodingSpacedBoolean/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 31133, 'null_percent': 1.0}
282                                     BM_PlainEncodingSpacedFloat/32768/100       6.276 GiB/sec      15.605 GiB/sec   148.621                {'run_name': 'BM_PlainEncodingSpacedFloat/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 35719, 'null_percent': 1.0}
152                                  BM_ArrowBinaryDict/EncodeLowLevel/262144      82.896 MiB/sec     203.718 MiB/sec   145.751                                     {'run_name': 'BM_ArrowBinaryDict/EncodeLowLevel/262144', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 38}
298                                  BM_PlainEncodingSpacedDouble/32768/10000     115.446 GiB/sec     262.819 GiB/sec   127.654          {'run_name': 'BM_PlainEncodingSpacedDouble/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 330346, 'null_percent': 100.0}
247                                    BM_PlainDecodingSpacedDouble/32768/100       9.953 GiB/sec      22.653 GiB/sec   127.594               {'run_name': 'BM_PlainDecodingSpacedDouble/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 28669, 'null_percent': 1.0}
207                                   BM_PlainEncodingSpacedFloat/32768/10000      57.732 GiB/sec     130.097 GiB/sec   125.348           {'run_name': 'BM_PlainEncodingSpacedFloat/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 330727, 'null_percent': 100.0}
203                                       BM_PlainDecodingSpacedFloat/32768/1      21.382 GiB/sec      36.628 GiB/sec    71.298                {'run_name': 'BM_PlainDecodingSpacedFloat/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 122797, 'null_percent': 0.01}
231                                    BM_PlainEncodingSpacedDouble/32768/100      10.467 GiB/sec      17.029 GiB/sec    62.697               {'run_name': 'BM_PlainEncodingSpacedDouble/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 30042, 'null_percent': 1.0}
256                                    BM_PlainEncodingSpacedFloat/32768/5000    1008.863 MiB/sec       1.592 GiB/sec    61.538               {'run_name': 'BM_PlainEncodingSpacedFloat/32768/5000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 5619, 'null_percent': 50.0}
273                                   BM_PlainEncodingSpacedDouble/32768/5000       2.022 GiB/sec       3.201 GiB/sec    58.281              {'run_name': 'BM_PlainEncodingSpacedDouble/32768/5000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 5712, 'null_percent': 50.0}
162                                    BM_PlainDecodingSpacedFloat/32768/5000     981.563 MiB/sec       1.516 GiB/sec    58.168               {'run_name': 'BM_PlainDecodingSpacedFloat/32768/5000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 5473, 'null_percent': 50.0}
173                                   BM_PlainDecodingSpacedDouble/32768/5000       1.891 GiB/sec       2.991 GiB/sec    58.157              {'run_name': 'BM_PlainDecodingSpacedDouble/32768/5000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 5401, 'null_percent': 50.0}
238                                  BM_PlainDecodingSpacedBoolean/32768/5000     250.841 MiB/sec     384.756 MiB/sec    53.386             {'run_name': 'BM_PlainDecodingSpacedBoolean/32768/5000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 5585, 'null_percent': 50.0}
264                                  BM_PlainEncodingSpacedBoolean/32768/5000     234.661 MiB/sec     353.009 MiB/sec    50.433             {'run_name': 'BM_PlainEncodingSpacedBoolean/32768/5000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 5211, 'null_percent': 50.0}
286                                    BM_PlainDecodingSpacedFloat/32768/1000       2.531 GiB/sec       3.739 GiB/sec    47.699              {'run_name': 'BM_PlainDecodingSpacedFloat/32768/1000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 14280, 'null_percent': 10.0}
229                                    BM_PlainEncodingSpacedFloat/32768/1000       2.680 GiB/sec       3.783 GiB/sec    41.159              {'run_name': 'BM_PlainEncodingSpacedFloat/32768/1000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 15210, 'null_percent': 10.0}
251                                      BM_PlainDecodingSpacedDouble/32768/1      25.016 GiB/sec      34.929 GiB/sec    39.627                {'run_name': 'BM_PlainDecodingSpacedDouble/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 71628, 'null_percent': 0.01}
201                                   BM_PlainDecodingSpacedDouble/32768/1000       4.987 GiB/sec       6.893 GiB/sec    38.224             {'run_name': 'BM_PlainDecodingSpacedDouble/32768/1000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 14031, 'null_percent': 10.0}
244                                   BM_PlainEncodingSpacedBoolean/32768/100     664.198 MiB/sec     913.312 MiB/sec    37.506              {'run_name': 'BM_PlainEncodingSpacedBoolean/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 14952, 'null_percent': 1.0}
239                                   BM_PlainEncodingSpacedDouble/32768/1000       5.174 GiB/sec       7.071 GiB/sec    36.652             {'run_name': 'BM_PlainEncodingSpacedDouble/32768/1000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 14708, 'null_percent': 10.0}
230                                  BM_PlainDecodingSpacedBoolean/32768/1000     678.559 MiB/sec     857.465 MiB/sec    26.366            {'run_name': 'BM_PlainDecodingSpacedBoolean/32768/1000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 15333, 'null_percent': 10.0}
102       BM_WriteInt64Column<Repetition::OPTIONAL, Compression::LZ4>/1048576     332.343 MiB/sec     409.541 MiB/sec    23.228         {'run_name': 'BM_WriteInt64Column<Repetition::OPTIONAL, Compression::LZ4>/1048576', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 109}
185                                       BM_PlainEncodingSpacedFloat/32768/1      17.571 GiB/sec      21.578 GiB/sec    22.805                {'run_name': 'BM_PlainEncodingSpacedFloat/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 100736, 'null_percent': 0.01}
176                                  BM_PlainEncodingSpacedBoolean/32768/1000     450.133 MiB/sec     545.271 MiB/sec    21.136             {'run_name': 'BM_PlainEncodingSpacedBoolean/32768/1000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 9948, 'null_percent': 10.0}
187                                     BM_PlainDecodingSpacedBoolean/32768/1       4.496 GiB/sec       5.314 GiB/sec    18.200              {'run_name': 'BM_PlainDecodingSpacedBoolean/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 103077, 'null_percent': 0.01}
25                                            BM_WriteColumn<false,Int64Type>       1.057 GiB/sec       1.220 GiB/sec    15.454                                              {'run_name': 'BM_WriteColumn<false,Int64Type>', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 10}
39                                       BM_ReadColumn<true,BooleanType>/5/10     250.138 MiB/sec     283.639 MiB/sec    13.393                                         {'run_name': 'BM_ReadColumn<true,BooleanType>/5/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 18}
75        BM_WriteInt64Column<Repetition::REQUIRED, Compression::LZ4>/1048576       1.084 GiB/sec       1.216 GiB/sec    12.132         {'run_name': 'BM_WriteInt64Column<Repetition::REQUIRED, Compression::LZ4>/1048576', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 443}
115                                                    BM_RleEncoding/32768/1     724.153 MiB/sec     810.490 MiB/sec    11.923                                                     {'run_name': 'BM_RleEncoding/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 8141}
92                                                      BM_RleEncoding/1024/1     706.002 MiB/sec     788.645 MiB/sec    11.706                                                    {'run_name': 'BM_RleEncoding/1024/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 232581}
106                         BM_WriteInt64Column<Repetition::REPEATED>/1048576     218.593 MiB/sec     243.729 MiB/sec    11.499                            {'run_name': 'BM_WriteInt64Column<Repetition::REPEATED>/1048576', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 77}
109                                                    BM_RleEncoding/65536/1     729.356 MiB/sec     811.227 MiB/sec    11.225                                                     {'run_name': 'BM_RleEncoding/65536/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 4076}
89                                                      BM_RleEncoding/4096/1     725.079 MiB/sec     804.486 MiB/sec    10.952                                                     {'run_name': 'BM_RleEncoding/4096/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 64655}
7                                        BM_ReadColumn<false,Int32Type>/-1/10       1.811 GiB/sec       2.008 GiB/sec    10.851                                         {'run_name': 'BM_ReadColumn<false,Int32Type>/-1/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 32}
58                                        BM_ReadColumn<false,Int32Type>/-1/1       5.426 GiB/sec       5.999 GiB/sec    10.568                                          {'run_name': 'BM_ReadColumn<false,Int32Type>/-1/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 89}
87     BM_ReadInt64Column<Repetition::REQUIRED, Compression::ZSTD>/65536/1024      12.156 GiB/sec      11.951 GiB/sec    -1.682    {'run_name': 'BM_ReadInt64Column<Repetition::REQUIRED, Compression::ZSTD>/65536/1024', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 69528}
[...]
204                                             BM_PlainEncodingBoolean/65536     802.645 MiB/sec     707.974 MiB/sec   -11.795                                              {'run_name': 'BM_PlainEncodingBoolean/65536', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 8937}
132                                                    BM_RleEncoding/65536/8     482.792 MiB/sec     424.143 MiB/sec   -12.148                                                     {'run_name': 'BM_RleEncoding/65536/8', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 2684}
15                                            BM_WriteColumn<true,DoubleType>     622.954 MiB/sec     498.484 MiB/sec   -19.981                                               {'run_name': 'BM_WriteColumn<true,DoubleType>', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 6}
16                                             BM_WriteColumn<true,Int64Type>     668.120 MiB/sec     526.837 MiB/sec   -21.146                                                {'run_name': 'BM_WriteColumn<true,Int64Type>', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 6}
34                                             BM_WriteColumn<true,Int32Type>     362.327 MiB/sec     256.361 MiB/sec   -29.246                                                {'run_name': 'BM_WriteColumn<true,Int32Type>', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 6}
27                                           BM_WriteColumn<true,BooleanType>     101.732 MiB/sec      69.602 MiB/sec   -31.583                                              {'run_name': 'BM_WriteColumn<true,BooleanType>', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 7}
60                                       BM_ReadColumn<true,BooleanType>/-1/1     480.372 MiB/sec     219.376 MiB/sec   -54.332                                         {'run_name': 'BM_ReadColumn<true,BooleanType>/-1/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 32}
289                                 BM_ArrowBinaryDict/EncodeLowLevel/1048576     316.208 MiB/sec     131.127 MiB/sec   -58.531                                    {'run_name': 'BM_ArrowBinaryDict/EncodeLowLevel/1048576', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 37}

@pitrou
pitrou force-pushed the ARROW-10696-set-bit-run-reader branch from fd9d34f to b470888 Compare November 25, 2020 20:05
@pitrou

pitrou commented Nov 25, 2020

Copy link
Copy Markdown
Member Author

Also some speedups on no-null filters:

  • before:
FilterInt64FilterNoNulls/524288/0          19993 ns        19988 ns        34897 bytes_per_second=24.4281G/s data null%=0 mask null%=0 select%=99.9 size=524.288k
FilterInt64FilterNoNulls/524288/1         249210 ns       249153 ns         2805 bytes_per_second=1.95976G/s data null%=0 mask null%=0 select%=50 size=524.288k
FilterInt64FilterNoNulls/524288/2          31896 ns        31889 ns        21867 bytes_per_second=15.3118G/s data null%=0 mask null%=0 select%=1 size=524.288k

FilterStringFilterNoNulls/524288/0         54006 ns        53996 ns        12804 bytes_per_second=9.04295G/s data null%=0 mask null%=0 select%=99.9 size=524.288k
FilterStringFilterNoNulls/524288/1        256494 ns       256438 ns         2721 bytes_per_second=1.90409G/s data null%=0 mask null%=0 select%=50 size=524.288k
FilterStringFilterNoNulls/524288/2         16881 ns        16878 ns        40537 bytes_per_second=28.9302G/s data null%=0 mask null%=0 select%=1 size=524.288k
  • after:
FilterInt64FilterNoNulls/524288/0          14416 ns        14413 ns        49758 bytes_per_second=33.8771G/s data null%=0 mask null%=0 select%=99.9 size=524.288k
FilterInt64FilterNoNulls/524288/1         151899 ns       151870 ns         4514 bytes_per_second=3.21512G/s data null%=0 mask null%=0 select%=50 size=524.288k
FilterInt64FilterNoNulls/524288/2           8578 ns         8577 ns        82823 bytes_per_second=56.931G/s data null%=0 mask null%=0 select%=1 size=524.288k

FilterStringFilterNoNulls/524288/0         45116 ns        45104 ns        15887 bytes_per_second=10.8256G/s data null%=0 mask null%=0 select%=99.9 size=524.288k
FilterStringFilterNoNulls/524288/1        146822 ns       146796 ns         4746 bytes_per_second=3.32625G/s data null%=0 mask null%=0 select%=50 size=524.288k
FilterStringFilterNoNulls/524288/2          5438 ns         5437 ns       126135 bytes_per_second=89.8084G/s data null%=0 mask null%=0 select%=1 size=524.288k

(couldn't run a diff because of ARROW-10738)

@pitrou
pitrou force-pushed the ARROW-10696-set-bit-run-reader branch from b470888 to fc434dc Compare November 25, 2020 20:24
@pitrou
pitrou requested review from bkietz and emkornfield November 26, 2020 09:02
@pitrou
pitrou force-pushed the ARROW-10696-set-bit-run-reader branch from fc434dc to 6ec79cc Compare November 26, 2020 09:25
@pitrou

pitrou commented Nov 26, 2020 •
edited
Loading

Copy link
Copy Markdown
Member Author

Aggregation benchmarks:

                              benchmark          baseline        contender   change %                                                                                                                                                                      counters
108         ModeKernelBoolean/1048576/0    57.487 MiB/sec   37.406 GiB/sec  66530.741            {'run_name': 'ModeKernelBoolean/1048576/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 41, 'null_percent': 0.0}
93          ModeKernelBoolean/1048576/2    28.239 MiB/sec    1.863 GiB/sec   6655.773           {'run_name': 'ModeKernelBoolean/1048576/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 20, 'null_percent': 50.0}
56         ModeKernelBoolean/1048576/10    43.613 MiB/sec    1.799 GiB/sec   4123.421          {'run_name': 'ModeKernelBoolean/1048576/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 31, 'null_percent': 10.0}
29        ModeKernelBoolean/1048576/100    58.888 MiB/sec    1.826 GiB/sec   3075.803          {'run_name': 'ModeKernelBoolean/1048576/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 39, 'null_percent': 1.0}
124     ModeKernelBoolean/1048576/10000    61.189 MiB/sec    1.864 GiB/sec   3018.637       {'run_name': 'ModeKernelBoolean/1048576/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 43, 'null_percent': 0.01}
35             ModeKernelInt8/1048576/0   792.467 MiB/sec    2.473 GiB/sec    219.603             {'run_name': 'ModeKernelInt8/1048576/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 1000, 'null_percent': 0.0}
36        ModeKernelInt16/1048576/10000     1.922 GiB/sec    4.217 GiB/sec    119.409       {'run_name': 'ModeKernelInt16/1048576/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 2138, 'null_percent': 0.01}
100    VarianceKernelDouble/1048576/100     2.445 GiB/sec    4.960 GiB/sec    102.888     {'run_name': 'VarianceKernelDouble/1048576/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 1746, 'null_percent': 1.0}
76            ModeKernelInt32/1048576/0     4.050 GiB/sec    8.075 GiB/sec     99.390            {'run_name': 'ModeKernelInt32/1048576/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 4419, 'null_percent': 0.0}
109     VarianceKernelFloat/1048576/100     1.216 GiB/sec    2.387 GiB/sec     96.341       {'run_name': 'VarianceKernelFloat/1048576/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 871, 'null_percent': 1.0}
20        ModeKernelInt64/1048576/10000     5.547 GiB/sec    9.831 GiB/sec     77.229       {'run_name': 'ModeKernelInt64/1048576/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 6475, 'null_percent': 0.01}
54      VarianceKernelInt64/1048576/100     3.511 GiB/sec    5.748 GiB/sec     63.693      {'run_name': 'VarianceKernelInt64/1048576/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 2519, 'null_percent': 1.0}
28          ModeKernelInt16/1048576/100     1.292 GiB/sec    2.077 GiB/sec     60.697           {'run_name': 'ModeKernelInt16/1048576/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 927, 'null_percent': 1.0}
94          ModeKernelInt32/1048576/100     2.530 GiB/sec    3.969 GiB/sec     56.872          {'run_name': 'ModeKernelInt32/1048576/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 1805, 'null_percent': 1.0}
67             ModeKernelInt8/1048576/2   237.264 MiB/sec  362.684 MiB/sec     52.861             {'run_name': 'ModeKernelInt8/1048576/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 166, 'null_percent': 50.0}
11        VarianceKernelFloat/1048576/2   508.083 MiB/sec  727.822 MiB/sec     43.249        {'run_name': 'VarianceKernelFloat/1048576/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 355, 'null_percent': 50.0}
65       VarianceKernelDouble/1048576/2     1.030 GiB/sec    1.457 GiB/sec     41.439       {'run_name': 'VarianceKernelDouble/1048576/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 734, 'null_percent': 50.0}
84            ModeKernelInt64/1048576/2     1.003 GiB/sec    1.409 GiB/sec     40.411            {'run_name': 'ModeKernelInt64/1048576/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 713, 'null_percent': 50.0}
22            ModeKernelInt16/1048576/2   253.082 MiB/sec  355.311 MiB/sec     40.394            {'run_name': 'ModeKernelInt16/1048576/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 177, 'null_percent': 50.0}
15        VarianceKernelInt32/1048576/2  1011.938 MiB/sec    1.387 GiB/sec     40.305        {'run_name': 'VarianceKernelInt32/1048576/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 701, 'null_percent': 50.0}
98            ModeKernelInt32/1048576/2   502.640 MiB/sec  702.642 MiB/sec     39.790            {'run_name': 'ModeKernelInt32/1048576/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 341, 'null_percent': 50.0}
61        VarianceKernelInt64/1048576/2     1.024 GiB/sec    1.422 GiB/sec     38.871        {'run_name': 'VarianceKernelInt64/1048576/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 726, 'null_percent': 50.0}
81       VarianceKernelFloat/1048576/10   854.282 MiB/sec    1.153 GiB/sec     38.189       {'run_name': 'VarianceKernelFloat/1048576/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 598, 'null_percent': 10.0}
34          ModeKernelInt64/1048576/100     4.766 GiB/sec    6.547 GiB/sec     37.389          {'run_name': 'ModeKernelInt64/1048576/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 3416, 'null_percent': 1.0}
2       VarianceKernelDouble/1048576/10     1.670 GiB/sec    2.290 GiB/sec     37.083     {'run_name': 'VarianceKernelDouble/1048576/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 1195, 'null_percent': 10.0}
111     VarianceKernelInt32/1048576/100     4.159 GiB/sec    5.506 GiB/sec     32.395      {'run_name': 'VarianceKernelInt32/1048576/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 2978, 'null_percent': 1.0}
30           ModeKernelInt8/1048576/100     1.276 GiB/sec    1.634 GiB/sec     28.042            {'run_name': 'ModeKernelInt8/1048576/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 919, 'null_percent': 1.0}
73    VarianceKernelInt32/1048576/10000     6.548 GiB/sec    7.310 GiB/sec     11.634   {'run_name': 'VarianceKernelInt32/1048576/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 4687, 'null_percent': 0.01}
1         VarianceKernelInt32/1048576/0     7.168 GiB/sec    7.952 GiB/sec     10.944        {'run_name': 'VarianceKernelInt32/1048576/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 5116, 'null_percent': 0.0}
[...]
16           ModeKernelInt64/1048576/10     2.376 GiB/sec    2.181 GiB/sec     -8.192          {'run_name': 'ModeKernelInt64/1048576/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 1700, 'null_percent': 10.0}
63        ModeKernelInt32/1048576/10000     6.489 GiB/sec    3.047 GiB/sec    -53.040       {'run_name': 'ModeKernelInt32/1048576/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 4653, 'null_percent': 0.01}
45         ModeKernelInt8/1048576/10000     2.262 GiB/sec    1.002 GiB/sec    -55.719        {'run_name': 'ModeKernelInt8/1048576/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 1617, 'null_percent': 0.01}

@pitrou
pitrou force-pushed the ARROW-10696-set-bit-run-reader branch from 6ec79cc to ad4b49b Compare November 26, 2020 13:59
wesm
wesm reviewed Nov 30, 2020
View reviewed changes

@wesm wesm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense to me -- definitely nice to see the code simplification in various places and the performance improvements. The no-nulls code paths in filtering for example had seemed inelegant to me but this will make it easier to write the "visit all the set bits" code paths easier to write and maintain in the future.

I don't know if the handful of Parquet writing perf regressions are worth investigating but they do not look too serious

Comment thread cpp/src/arrow/compute/kernels/vector_selection.cc Outdated

@wesm wesm Nov 30, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice

@bkietz bkietz Dec 1, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

bkietz
bkietz approved these changes Dec 1, 2020
View reviewed changes

@bkietz bkietz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a few comments

Comment thread cpp/src/arrow/compute/kernels/vector_selection.cc Outdated

@bkietz bkietz Dec 1, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

Comment thread cpp/src/arrow/util/bit_util_test.cc Outdated
Comment thread cpp/src/arrow/util/bit_run_reader.h Outdated

@bkietz bkietz Dec 1, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (num_zeros < 64) {
if (ARROW_PREDICT_FALSE(num_zeros < 64)) {

@bkietz bkietz Dec 1, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO, this is much easier to read as if (num_zeros >= 64) { remaining_ -= 64; continue; } // rest

Comment thread cpp/src/arrow/util/bit_run_reader.h Outdated

@bkietz bkietz Dec 1, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (num_ones < 64) {
if (ARROW_PREDICT_FALSE(num_ones < 64)) {

@pitrou pitrou Dec 2, 2020

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, depending on the distribution of input bits, this may be commonly true (same for the other comment above).

@github-actions github-actions Bot added the needs-rebase A PR that needs to be rebased by the author label Dec 2, 2020
pitrou added 2 commits December 2, 2020 08:40
@pitrou
ARROW-10696: [C++] Add SetBitRunReader
b245c35 
A specialized bitmap reader that yields runs of set bits,
for use cases where reset bits (e.g. null bits) don't need any handling.

On some use cases it can be significantly faster than the alternatives.
@pitrou
Nits
2be8572
@pitrou
pitrou force-pushed the ARROW-10696-set-bit-run-reader branch from ad4b49b to 2be8572 Compare December 2, 2020 07:48
@github-actions github-actions Bot removed the needs-rebase A PR that needs to be rebased by the author label Dec 2, 2020
@pitrou pitrou closed this in b22bae7 Dec 2, 2020
@pitrou
pitrou deleted the ARROW-10696-set-bit-run-reader branch December 2, 2020 13:52
@asfimport asfimport mentioned this pull request Jun 10, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wesm wesm wesm left review comments
@bkietz bkietz bkietz approved these changes
@emkornfield emkornfield Awaiting requested review from emkornfield

Assignees

No one assigned

Labels

Component: C++ Component: Parquet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@pitrou @wesm @bkietz