ARROW-10143: [C++] Rewrite Array(Range)Equals#8703
Closed
pitrou wants to merge 4 commits into
Closed
Conversation
pitrou
force-pushed
the
ARROW-10143-range-equals-rewrite
branch
from
November 18, 2020 10:40
f4b0155 to
e0d17e7
Compare
pitrou
commented
Nov 19, 2020
Member
Author
There was a problem hiding this comment.
@wesm Is constructing a StringArray with null offsets and data supported (it's non-empty but all-nulls)?
Member
There was a problem hiding this comment.
I recall this question coming up in the past and I don't recall the outcome, @xhochy do you remember?
Member
There was a problem hiding this comment.
I can also remember that we had the conversation but I'm unsure about the outcome, too.
pitrou
force-pushed
the
ARROW-10143-range-equals-rewrite
branch
from
November 19, 2020 12:51
e0d17e7 to
5ba6975
Compare
Member
Author
|
Rebased. |
bkietz
requested changes
Nov 20, 2020
Comment on lines
169
to
170
Member
There was a problem hiding this comment.
IIUC BitRunReader yields alternating set/unset BitRuns. Would SetBitRunReader yield only the set BitRuns?
Member
Author
There was a problem hiding this comment.
Yes, it would. That would make its logic much simpler and more easily optimized by the compiler (though, of course, numbers will be the final judge).
Comment on lines
72
to
73
Member
There was a problem hiding this comment.
There's a lot of duplication of logic under the heading of "reading words from bitmaps": VisitWords, BitmapWordReader, BitmapUInt64Reader. In a follow up it'd be handy to consolidate these and ensure they're rigorously benchmarked (or if there are compelling reasons not to consolidate then provide comments indicating when each should be preferred)
Member
Author
There was a problem hiding this comment.
Yeah, I think that would be useful. I should open JIRAs for the various XXX and TODOs here.
Member
There was a problem hiding this comment.
This seems like a generally useful bitmap comparison utility. If it's faster than BitmapsEqual then maybe it could replace that function?
Member
Author
There was a problem hiding this comment.
That will be part of the JIRA followup (investigate replacing BitmapWordReader with BitmapUInt64Reader).
Member
There was a problem hiding this comment.
Interesting, this seems very doable:
- get next type_id of left and right
- early exit if not equal
- find length of run in left_codes
- find length of run in right_codes
- early exit if run lengths are not equal
- compare ranges of children
Member
Author
There was a problem hiding this comment.
Right, though the question is whether this would improve or worsen common cases (I have no idea how unions are used in the wild).
In any case, this could be a separate low-priority JIRA (I don't think optimizing unions is in our current priorities).
Member
There was a problem hiding this comment.
runs of type_id can still be used here, but ranges of children can't be directly compared due to the offsets. It's odd that we "dereference" offsets when comparing dense union arrays but we don't do the same with a dictionary array's indices
Member
Author
There was a problem hiding this comment.
I kept the previous semantics of dictionary array comparison, that is: if the dictionaries themselves are unequal, the arrays are considered unequal even if the indices only refer to equal dictionary elements.
Member
There was a problem hiding this comment.
I see that you kept previous semantics, I was just noting that the difference is odd. I would have expected the two cases to be more symmetric and wondered if you could comment
Member
Author
There was a problem hiding this comment.
I would defer to @wesm for the historical reasons.
ArrayEquals now defers to ArrayRangeEquals under the hood.
ArrayRangeEquals now allows passing an EqualOptions argument.
Also add ArrayRangeApproxEquals.
Comparison speed is massively improved on many input types:
```
benchmark baseline contender change % counters
26 ArrayRangeEqualsStruct/32768/0 6.338m items/sec 797.926m items/sec 12490.248 {'run_name': 'ArrayRangeEqualsStruct/32768/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 135, 'null_percent': 0.0}
16 ArrayRangeEqualsBoolean/32768/0 839.237m items/sec 51.203b items/sec 6001.168 {'run_name': 'ArrayRangeEqualsBoolean/32768/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 17929, 'null_percent': 0.0}
28 ArrayRangeEqualsFixedSizeBinary/32768/0 369.542m items/sec 14.798b items/sec 3904.348 {'run_name': 'ArrayRangeEqualsFixedSizeBinary/32768/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 8130, 'null_percent': 0.0}
24 ArrayRangeEqualsStruct/32768/10000 6.251m items/sec 240.453m items/sec 3746.338 {'run_name': 'ArrayRangeEqualsStruct/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 134, 'null_percent': 0.01}
36 ArrayRangeEqualsStruct/32768/1 412.074m items/sec 13.733b items/sec 3232.616 {'run_name': 'ArrayRangeEqualsStruct/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 8817, 'null_percent': 100.0}
9 ArrayRangeEqualsString/32768/0 67.419m items/sec 1.937b items/sec 2772.931 {'run_name': 'ArrayRangeEqualsString/32768/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 1441, 'null_percent': 0.0}
25 ArrayRangeEqualsListOfInt32/32768/1 524.231m items/sec 13.774b items/sec 2527.447 {'run_name': 'ArrayRangeEqualsListOfInt32/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 11179, 'null_percent': 100.0}
44 ArrayRangeEqualsString/32768/1 577.902m items/sec 13.686b items/sec 2268.185 {'run_name': 'ArrayRangeEqualsString/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 12328, 'null_percent': 100.0}
53 ArrayRangeEqualsFixedSizeBinary/32768/1 596.284m items/sec 13.140b items/sec 2103.587 {'run_name': 'ArrayRangeEqualsFixedSizeBinary/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 12703, 'null_percent': 100.0}
11 ArrayRangeEqualsString/32768/10000 67.501m items/sec 1.382b items/sec 1947.876 {'run_name': 'ArrayRangeEqualsString/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 1442, 'null_percent': 0.01}
46 ArrayRangeEqualsListOfInt32/32768/0 42.696m items/sec 833.958m items/sec 1853.229 {'run_name': 'ArrayRangeEqualsListOfInt32/32768/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 911, 'null_percent': 0.0}
14 ArrayRangeEqualsBoolean/32768/1 698.866m items/sec 13.374b items/sec 1813.652 {'run_name': 'ArrayRangeEqualsBoolean/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 14929, 'null_percent': 100.0}
38 ArrayRangeEqualsInt32/32768/1 835.824m items/sec 13.688b items/sec 1537.720 {'run_name': 'ArrayRangeEqualsInt32/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 17818, 'null_percent': 100.0}
29 ArrayRangeEqualsFixedSizeBinary/32768/10000 278.025m items/sec 4.375b items/sec 1473.475 {'run_name': 'ArrayRangeEqualsFixedSizeBinary/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 5937, 'null_percent': 0.01}
17 ArrayRangeEqualsInt32/32768/0 2.104b items/sec 32.033b items/sec 1422.785 {'run_name': 'ArrayRangeEqualsInt32/32768/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 44741, 'null_percent': 0.0}
18 ArrayRangeEqualsFloat32/32768/1 838.934m items/sec 12.432b items/sec 1381.896 {'run_name': 'ArrayRangeEqualsFloat32/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 17895, 'null_percent': 100.0}
19 ArrayRangeEqualsInt32/32768/10000 757.797m items/sec 6.848b items/sec 803.693 {'run_name': 'ArrayRangeEqualsInt32/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 16177, 'null_percent': 0.01}
15 ArrayRangeEqualsListOfInt32/32768/10000 28.438m items/sec 210.226m items/sec 639.253 {'run_name': 'ArrayRangeEqualsListOfInt32/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 606, 'null_percent': 0.01}
39 ArrayRangeEqualsFloat32/32768/10000 694.965m items/sec 4.706b items/sec 577.178 {'run_name': 'ArrayRangeEqualsFloat32/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 14833, 'null_percent': 0.01}
3 ArrayRangeEqualsBoolean/32768/10000 538.353m items/sec 2.968b items/sec 451.394 {'run_name': 'ArrayRangeEqualsBoolean/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 11475, 'null_percent': 0.01}
41 ArrayRangeEqualsFloat32/32768/0 2.036b items/sec 10.235b items/sec 402.761 {'run_name': 'ArrayRangeEqualsFloat32/32768/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 43978, 'null_percent': 0.0}
23 ArrayRangeEqualsStruct/32768/100 6.298m items/sec 26.042m items/sec 313.515 {'run_name': 'ArrayRangeEqualsStruct/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 133, 'null_percent': 1.0}
1 ArrayRangeEqualsString/32768/100 68.074m items/sec 275.727m items/sec 305.039 {'run_name': 'ArrayRangeEqualsString/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 1453, 'null_percent': 1.0}
42 ArrayRangeEqualsSparseUnion/32768/0 12.649m items/sec 38.360m items/sec 203.256 {'run_name': 'ArrayRangeEqualsSparseUnion/32768/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 274, 'null_percent': 0.0}
45 ArrayRangeEqualsDenseUnion/32768/0 12.995m items/sec 38.503m items/sec 196.287 {'run_name': 'ArrayRangeEqualsDenseUnion/32768/0', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 277, 'null_percent': 0.0}
7 ArrayRangeEqualsFixedSizeBinary/32768/100 274.991m items/sec 633.253m items/sec 130.281 {'run_name': 'ArrayRangeEqualsFixedSizeBinary/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 5864, 'null_percent': 1.0}
13 ArrayRangeEqualsInt32/32768/100 729.634m items/sec 1.644b items/sec 125.256 {'run_name': 'ArrayRangeEqualsInt32/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 15582, 'null_percent': 1.0}
12 ArrayRangeEqualsStruct/32768/10 6.948m items/sec 14.273m items/sec 105.427 {'run_name': 'ArrayRangeEqualsStruct/32768/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 147, 'null_percent': 10.0}
34 ArrayRangeEqualsStruct/32768/2 11.923m items/sec 23.845m items/sec 99.993 {'run_name': 'ArrayRangeEqualsStruct/32768/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 255, 'null_percent': 50.0}
10 ArrayRangeEqualsString/32768/10 73.461m items/sec 146.598m items/sec 99.558 {'run_name': 'ArrayRangeEqualsString/32768/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 1568, 'null_percent': 10.0}
48 ArrayRangeEqualsBoolean/32768/100 525.635m items/sec 1.026b items/sec 95.102 {'run_name': 'ArrayRangeEqualsBoolean/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 11216, 'null_percent': 1.0}
21 ArrayRangeEqualsSparseUnion/32768/1 14.032m items/sec 26.225m items/sec 86.889 {'run_name': 'ArrayRangeEqualsSparseUnion/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 300, 'null_percent': 100.0}
47 ArrayRangeEqualsSparseUnion/32768/2 12.643m items/sec 23.030m items/sec 82.160 {'run_name': 'ArrayRangeEqualsSparseUnion/32768/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 272, 'null_percent': 50.0}
43 ArrayRangeEqualsSparseUnion/32768/10000 12.596m items/sec 22.801m items/sec 81.018 {'run_name': 'ArrayRangeEqualsSparseUnion/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 264, 'null_percent': 0.01}
52 ArrayRangeEqualsSparseUnion/32768/10 12.717m items/sec 22.911m items/sec 80.168 {'run_name': 'ArrayRangeEqualsSparseUnion/32768/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 272, 'null_percent': 10.0}
0 ArrayRangeEqualsSparseUnion/32768/100 12.783m items/sec 22.714m items/sec 77.694 {'run_name': 'ArrayRangeEqualsSparseUnion/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 275, 'null_percent': 1.0}
31 ArrayRangeEqualsDenseUnion/32768/1 14.509m items/sec 25.576m items/sec 76.279 {'run_name': 'ArrayRangeEqualsDenseUnion/32768/1', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 309, 'null_percent': 100.0}
20 ArrayRangeEqualsDenseUnion/32768/10000 13.193m items/sec 22.447m items/sec 70.152 {'run_name': 'ArrayRangeEqualsDenseUnion/32768/10000', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 280, 'null_percent': 0.01}
35 ArrayRangeEqualsDenseUnion/32768/100 13.204m items/sec 22.256m items/sec 68.561 {'run_name': 'ArrayRangeEqualsDenseUnion/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 282, 'null_percent': 1.0}
40 ArrayRangeEqualsDenseUnion/32768/10 13.183m items/sec 22.191m items/sec 68.338 {'run_name': 'ArrayRangeEqualsDenseUnion/32768/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 279, 'null_percent': 10.0}
4 ArrayRangeEqualsDenseUnion/32768/2 13.148m items/sec 21.996m items/sec 67.297 {'run_name': 'ArrayRangeEqualsDenseUnion/32768/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 280, 'null_percent': 50.0}
22 ArrayRangeEqualsFloat32/32768/100 671.171m items/sec 1.097b items/sec 63.414 {'run_name': 'ArrayRangeEqualsFloat32/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 14325, 'null_percent': 1.0}
30 ArrayRangeEqualsString/32768/2 94.060m items/sec 149.539m items/sec 58.983 {'run_name': 'ArrayRangeEqualsString/32768/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 2008, 'null_percent': 50.0}
37 ArrayRangeEqualsFixedSizeBinary/32768/10 250.975m items/sec 305.864m items/sec 21.870 {'run_name': 'ArrayRangeEqualsFixedSizeBinary/32768/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 5341, 'null_percent': 10.0}
32 ArrayRangeEqualsInt32/32768/10 607.891m items/sec 676.139m items/sec 11.227 {'run_name': 'ArrayRangeEqualsInt32/32768/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 12895, 'null_percent': 10.0}
8 ArrayRangeEqualsFixedSizeBinary/32768/2 187.404m items/sec 207.700m items/sec 10.830 {'run_name': 'ArrayRangeEqualsFixedSizeBinary/32768/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 3971, 'null_percent': 50.0}
49 ArrayRangeEqualsBoolean/32768/10 468.989m items/sec 518.521m items/sec 10.561 {'run_name': 'ArrayRangeEqualsBoolean/32768/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 9996, 'null_percent': 10.0}
5 ArrayRangeEqualsInt32/32768/2 259.891m items/sec 270.547m items/sec 4.100 {'run_name': 'ArrayRangeEqualsInt32/32768/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 5505, 'null_percent': 50.0}
33 ArrayRangeEqualsFloat32/32768/2 262.994m items/sec 268.155m items/sec 1.962 {'run_name': 'ArrayRangeEqualsFloat32/32768/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 5580, 'null_percent': 50.0}
50 ArrayRangeEqualsFloat32/32768/10 577.200m items/sec 573.157m items/sec -0.701 {'run_name': 'ArrayRangeEqualsFloat32/32768/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 12270, 'null_percent': 10.0}
2 ArrayRangeEqualsBoolean/32768/2 242.498m items/sec 230.947m items/sec -4.763 {'run_name': 'ArrayRangeEqualsBoolean/32768/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 5152, 'null_percent': 50.0}
27 ArrayRangeEqualsListOfInt32/32768/100 28.080m items/sec 25.874m items/sec -7.855 {'run_name': 'ArrayRangeEqualsListOfInt32/32768/100', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 598, 'null_percent': 1.0}
51 ArrayRangeEqualsListOfInt32/32768/2 18.613m items/sec 12.136m items/sec -34.798 {'run_name': 'ArrayRangeEqualsListOfInt32/32768/2', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 397, 'null_percent': 50.0}
6 ArrayRangeEqualsListOfInt32/32768/10 26.194m items/sec 12.610m items/sec -51.859 {'run_name': 'ArrayRangeEqualsListOfInt32/32768/10', 'run_type': 'iteration', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 560, 'null_percent': 10.0}
```
pitrou
force-pushed
the
ARROW-10143-range-equals-rewrite
branch
from
November 23, 2020 11:09
5ba6975 to
0f367f6
Compare
Member
Author
|
I've addressed review comments. I'd like to merge this if it is ok. |
bkietz
approved these changes
Nov 23, 2020
This was referenced Nov 23, 2020
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ArrayEquals now defers to ArrayRangeEquals under the hood.
ArrayRangeEquals now allows passing an EqualOptions argument.
Also add ArrayRangeApproxEquals.
Comparison speed is massively improved on many input types: