Skip to content

feat: teach RunEnd TrueCount and NullCount#1982

Closed
danking wants to merge 1 commit into
developfrom
dk/run-end-true-count-null-count
Closed

feat: teach RunEnd TrueCount and NullCount#1982
danking wants to merge 1 commit into
developfrom
dk/run-end-true-count-null-count

Conversation

@danking

@danking danking commented Jan 16, 2025

Copy link
Copy Markdown
Contributor

No description provided.

@danking danking added the action/benchmark Trigger full benchmarks to run on this PR label Jan 16, 2025
@github-actions github-actions Bot removed the action/benchmark Trigger full benchmarks to run on this PR label Jan 16, 2025
@github-actions

github-actions Bot commented Jan 16, 2025

Copy link
Copy Markdown
Contributor

Benchmarks: datafusion

Table of Results
name PR 93f52f7 base 9a82b48 ratio (PR/base) unit
arrow/planning 943071 977876 0.964408 ns
arrow/exec 2.00026e+06 2.09722e+06 0.953768 ns
vortex-pushdown-compressed/planning 585104 590999 0.990024 ns
vortex-pushdown-compressed/exec 2.7031e+06 2.9639e+06 0.912009 ns
vortex-pushdown-uncompressed/planning 582603 589403 0.988463 ns
vortex-pushdown-uncompressed/exec 1.54583e+06 1.61218e+06 0.958841 ns
vortex-nopushdown-compressed/planning 945383 969224 0.975402 ns
vortex-nopushdown-compressed/exec 3.1247e+06 4.38223e+06 0.713038 ns
vortex-nopushdown-uncompressed/planning 948166 974427 0.97305 ns
vortex-nopushdown-uncompressed/exec 5.02994e+06 6.1478e+06 0.818169 ns

@github-actions

github-actions Bot commented Jan 16, 2025

Copy link
Copy Markdown
Contributor

Benchmarks: random_access

Table of Results
name PR 93f52f7 base 9a82b48 ratio (PR/base) unit
random-access/vortex-tokio-local-disk 2.54232e+06 3.07203e+06 0.827571 ns
random-access/vortex-local-fs 2.5926e+06 3.15383e+06 0.822049 ns
random-access/parquet-tokio-local-disk 2.20976e+08 2.44489e+08 0.903829 ns

@github-actions

github-actions Bot commented Jan 16, 2025

Copy link
Copy Markdown
Contributor

Benchmarks: Clickbench

Table of Results
name PR 93f52f7 base 9a82b48 ratio (PR/base) unit
clickbench_q00/parquet 1836317 2.03578e+06 0.902023 ns
clickbench_q01/parquet 60647266 6.31307e+07 0.960661 ns
clickbench_q02/parquet 114843571 1.22692e+08 0.936028 ns
clickbench_q03/parquet 83831107 9.10614e+07 0.9206 ns
clickbench_q04/parquet 666313616 7.35969e+08 0.905356 ns
clickbench_q05/parquet 844764393 8.92413e+08 0.946607 ns
clickbench_q06/parquet 1925882 2.05231e+06 0.938398 ns
clickbench_q07/parquet 62706594 6.63296e+07 0.945378 ns
clickbench_q08/parquet 759772475 8.30498e+08 0.914839 ns
clickbench_q09/parquet 1063168351 1.16558e+09 0.91214 ns
clickbench_q10/parquet 254872593 2.76713e+08 0.921073 ns
clickbench_q11/parquet 304301803 3.22978e+08 0.942176 ns
clickbench_q12/parquet 854191533 9.36353e+08 0.912254 ns
clickbench_q13/parquet 1127275268 1.18751e+09 0.949276 ns
clickbench_q14/parquet 859776111 9.14536e+08 0.940123 ns
clickbench_q15/parquet 785060188 8.58396e+08 0.914566 ns
clickbench_q16/parquet 1663305482 1.78858e+09 0.929958 ns
clickbench_q17/parquet 1449183178 1.57027e+09 0.922889 ns
clickbench_q18/parquet 3057156257 3.31019e+09 0.92356 ns
clickbench_q19/parquet 65729497 7.07783e+07 0.928668 ns
clickbench_q20/parquet 1203155200 1.23451e+09 0.974605 ns
clickbench_q21/parquet 1439589235 1.48981e+09 0.966288 ns
clickbench_q22/parquet 2460785490 2.55586e+09 0.962801 ns
clickbench_q23/parquet 8321940897 9.07743e+09 0.916773 ns
clickbench_q24/parquet 535760174 5.557e+08 0.964118 ns
clickbench_q25/parquet 522292076 5.31075e+08 0.983461 ns
clickbench_q26/parquet 605628513 6.20984e+08 0.975273 ns
clickbench_q27/parquet 1646818183 1.75867e+09 0.936399 ns
clickbench_q28/parquet 11396421371 1.17693e+10 0.968315 ns
clickbench_q29/parquet 432015926 4.38639e+08 0.9849 ns
clickbench_q30/parquet 782390576 8.2479e+08 0.948593 ns
clickbench_q31/parquet 812309125 8.66708e+08 0.937235 ns
clickbench_q32/parquet 2795202134 3.05236e+09 0.915751 ns
clickbench_q33/parquet 2885929936 3.25209e+09 0.887407 ns
clickbench_q34/parquet 2802303680 3.09424e+09 0.905652 ns
clickbench_q35/parquet 875189175 9.28427e+08 0.942658 ns
clickbench_q36/parquet 170955299 1.92754e+08 0.886909 ns
clickbench_q37/parquet 87608004 9.45344e+07 0.926732 ns
clickbench_q38/parquet 115553385 1.23059e+08 0.939005 ns
clickbench_q39/parquet 330376628 3.60462e+08 0.916537 ns
clickbench_q40/parquet 50601560 5.57919e+07 0.90697 ns
clickbench_q41/parquet 48877110 5.24323e+07 0.932195 ns
clickbench_q42/parquet 68230220 7.19167e+07 0.948739 ns
clickbench_q00/vortex-file-compressed 2009243 2.35261e+06 0.85405 ns
clickbench_q01/vortex-file-compressed 68806356 7.10084e+07 0.968989 ns
clickbench_q02/vortex-file-compressed 186820019 1.48285e+08 1.25987 ns
clickbench_q03/vortex-file-compressed 100778354 1.06748e+08 0.944077 ns
clickbench_q04/vortex-file-compressed 660063713 7.25367e+08 0.909973 ns
clickbench_q05/vortex-file-compressed 841143697 9.40472e+08 0.894385 ns
clickbench_q06/vortex-file-compressed 2130118 2.33766e+06 0.911218 ns
clickbench_q07/vortex-file-compressed 111771679 1.14751e+08 0.974038 ns
clickbench_q08/vortex-file-compressed 775196355 8.4602e+08 0.916286 ns
clickbench_q09/vortex-file-compressed 1078184294 1.16996e+09 0.921558 ns
clickbench_q10/vortex-file-compressed 342878551 3.60659e+08 0.9507 ns
clickbench_q11/vortex-file-compressed 401144792 4.22865e+08 0.948636 ns
clickbench_q12/vortex-file-compressed 964305907 1.03082e+09 0.935473 ns
clickbench_q13/vortex-file-compressed 1363663278 1.45004e+09 0.940433 ns
clickbench_q14/vortex-file-compressed 951382509 1.02708e+09 0.926299 ns
clickbench_q15/vortex-file-compressed 796466745 8.83415e+08 0.901577 ns
clickbench_q16/vortex-file-compressed 1564804466 1.68538e+09 0.92846 ns
clickbench_q17/vortex-file-compressed 1441272189 1.60405e+09 0.898521 ns
clickbench_q18/vortex-file-compressed 3119538169 3.29675e+09 0.946245 ns
clickbench_q19/vortex-file-compressed 83759823 8.74309e+07 0.958011 ns
clickbench_q20/vortex-file-compressed 796772767 8.23374e+08 0.967692 ns
clickbench_q21/vortex-file-compressed 1026393040 1.05875e+09 0.969441 ns
clickbench_q22/vortex-file-compressed 2103446127 2.18386e+09 0.963179 ns
clickbench_q23/vortex-file-compressed 3613513978 3.90062e+09 0.926395 ns
clickbench_q24/vortex-file-compressed 711983472 7.4354e+08 0.957559 ns
clickbench_q25/vortex-file-compressed 664992654 6.89909e+08 0.963884 ns
clickbench_q26/vortex-file-compressed 760219820 8.01071e+08 0.949005 ns
clickbench_q27/vortex-file-compressed 1647316163 1.72425e+09 0.955381 ns
clickbench_q28/vortex-file-compressed 11349917615 1.17403e+10 0.966745 ns
clickbench_q29/vortex-file-compressed 452847649 4.73523e+08 0.956337 ns
clickbench_q30/vortex-file-compressed 890940018 9.70073e+08 0.918425 ns
clickbench_q31/vortex-file-compressed 1060194374 1.12515e+09 0.94227 ns
clickbench_q32/vortex-file-compressed 3066180139 3.47133e+09 0.883286 ns
clickbench_q33/vortex-file-compressed 2460387619 2.69157e+09 0.914108 ns
clickbench_q34/vortex-file-compressed 2465664760 2.6958e+09 0.914631 ns
clickbench_q35/vortex-file-compressed 945787340 1.04031e+09 0.909138 ns
clickbench_q36/vortex-file-compressed 72680842 7.35297e+07 0.988456 ns
clickbench_q37/vortex-file-compressed 63649405 6.79286e+07 0.937004 ns
clickbench_q38/vortex-file-compressed 58033540 6.20506e+07 0.935262 ns
clickbench_q39/vortex-file-compressed 125136511 1.24028e+08 1.00893 ns
clickbench_q40/vortex-file-compressed 58267338 5.8868e+07 0.989796 ns
clickbench_q41/vortex-file-compressed 55595940 5.61846e+07 0.989522 ns
clickbench_q42/vortex-file-compressed 56247369 5.81545e+07 0.967207 ns

@danking danking mentioned this pull request Jan 16, 2025
Closed
@github-actions

github-actions Bot commented Jan 16, 2025

Copy link
Copy Markdown
Contributor

Benchmarks: compress

Table of Results
name PR 93f52f7 base 9a82b48 ratio (PR/base) unit
compress time/taxi 1.68437e+09 1.65711e+09 1.01645 ns
compress time/taxi throughput 0.279517 0.284116 0.983816 bytes/ns
parquet_rs-zstd compress time/taxi 1.86661e+09 1.80314e+09 1.0352 ns
parquet_rs-zstd compress time/taxi throughput 0.252228 0.261106 0.966 bytes/ns
decompress time/taxi 4.2323e+08 4.10177e+08 1.03182 ns
decompress time/taxi throughput 1.11242 1.14782 0.969159 bytes/ns
parquet_rs-zstd decompress time/taxi 3.1446e+08 3.14782e+08 0.998977 ns
parquet_rs-zstd decompress time/taxi throughput 1.49721 1.49567 1.00102 bytes/ns
compress time/AirlineSentiment 611644 610671 1.00159 ns
compress time/AirlineSentiment throughput 0.00337451 0.00337989 0.99841 bytes/ns
parquet_rs-zstd compress time/AirlineSentiment 57117.5 56928.7 1.00332 ns
parquet_rs-zstd compress time/AirlineSentiment throughput 0.036136 0.0362559 0.996695 bytes/ns
decompress time/AirlineSentiment 103565 101544 1.0199 ns
decompress time/AirlineSentiment throughput 0.0199296 0.0203261 0.980492 bytes/ns
parquet_rs-zstd decompress time/AirlineSentiment 32762.9 33271.2 0.984724 ns
parquet_rs-zstd decompress time/AirlineSentiment throughput 0.062998 0.0620357 1.01551 bytes/ns
compress time/Arade 2.89893e+09 2.8415e+09 1.02021 ns
compress time/Arade throughput 0.271491 0.276979 0.980188 bytes/ns
parquet_rs-zstd compress time/Arade 3.15085e+09 3.03855e+09 1.03696 ns
parquet_rs-zstd compress time/Arade throughput 0.249785 0.259017 0.96436 bytes/ns
decompress time/Arade 5.4543e+08 5.18401e+08 1.05214 ns
decompress time/Arade throughput 1.44296 1.5182 0.950445 bytes/ns
parquet_rs-zstd decompress time/Arade 6.96004e+08 6.87545e+08 1.0123 ns
parquet_rs-zstd decompress time/Arade throughput 1.13079 1.1447 0.987846 bytes/ns
compress time/Bimbo 1.21276e+10 1.20329e+10 1.00787 ns
compress time/Bimbo throughput 0.587203 0.591824 0.992192 bytes/ns
parquet_rs-zstd compress time/Bimbo 2.19917e+10 2.14025e+10 1.02753 ns
parquet_rs-zstd compress time/Bimbo throughput 0.32382 0.332735 0.973205 bytes/ns
decompress time/Bimbo 3.37321e+09 3.25693e+09 1.0357 ns
decompress time/Bimbo throughput 2.11115 2.18652 0.965528 bytes/ns
parquet_rs-zstd decompress time/Bimbo 2.80103e+09 2.76332e+09 1.01365 ns
parquet_rs-zstd decompress time/Bimbo throughput 2.5424 2.5771 0.986539 bytes/ns
compress time/CMSprovider 1.37063e+10 1.39639e+10 0.981548 ns
compress time/CMSprovider throughput 0.375681 0.368749 1.0188 bytes/ns
parquet_rs-zstd compress time/CMSprovider 2.01115e+10 1.9341e+10 1.03984 ns
parquet_rs-zstd compress time/CMSprovider throughput 0.256033 0.266232 0.96169 bytes/ns
decompress time/CMSprovider 3.39359e+09 3.26965e+09 1.03791 ns
decompress time/CMSprovider throughput 1.51733 1.57484 0.963479 bytes/ns
parquet_rs-zstd decompress time/CMSprovider 6.13792e+09 6.0143e+09 1.02055 ns
parquet_rs-zstd decompress time/CMSprovider throughput 0.838916 0.856159 0.979859 bytes/ns
compress time/Euro2016 2.28384e+09 2.21678e+09 1.03025 ns
compress time/Euro2016 throughput 0.17219 0.1774 0.970635 bytes/ns
parquet_rs-zstd compress time/Euro2016 1.61547e+09 1.57815e+09 1.02365 ns
parquet_rs-zstd compress time/Euro2016 throughput 0.243431 0.249188 0.976897 bytes/ns
decompress time/Euro2016 2.48847e+08 2.32465e+08 1.07047 ns
decompress time/Euro2016 throughput 1.58031 1.69168 0.934167 bytes/ns
parquet_rs-zstd decompress time/Euro2016 5.09213e+08 4.95793e+08 1.02707 ns
parquet_rs-zstd decompress time/Euro2016 throughput 0.772281 0.793186 0.973645 bytes/ns
compress time/Food 1.15665e+09 1.13455e+09 1.01948 ns
compress time/Food throughput 0.28766 0.293263 0.980891 bytes/ns
parquet_rs-zstd compress time/Food 1.11264e+09 1.07907e+09 1.03111 ns
parquet_rs-zstd compress time/Food throughput 0.299037 0.30834 0.96983 bytes/ns
decompress time/Food 1.26616e+08 1.24906e+08 1.01369 ns
decompress time/Food throughput 2.6278 2.66378 0.986493 bytes/ns
parquet_rs-zstd decompress time/Food 2.28015e+08 2.26772e+08 1.00548 ns
parquet_rs-zstd decompress time/Food throughput 1.45921 1.46721 0.994547 bytes/ns
compress time/HashTags 2.62863e+09 2.57638e+09 1.02028 ns
compress time/HashTags throughput 0.306054 0.312261 0.980122 bytes/ns
parquet_rs-zstd compress time/HashTags 2.60389e+09 2.52324e+09 1.03196 ns
parquet_rs-zstd compress time/HashTags throughput 0.308962 0.318838 0.969026 bytes/ns
decompress time/HashTags 3.67073e+08 3.58484e+08 1.02396 ns
decompress time/HashTags throughput 2.19167 2.24418 0.976602 bytes/ns
parquet_rs-zstd decompress time/HashTags 8.68171e+08 8.34466e+08 1.04039 ns
parquet_rs-zstd decompress time/HashTags throughput 0.926664 0.964093 0.961178 bytes/ns
compress time/TPC-H l_comment chunked without fsst 3.433e+09 3.17301e+09 1.08194 ns
compress time/TPC-H l_comment chunked without fsst throughput 0.0725959 0.0785441 0.924269 bytes/ns
parquet_rs-zstd compress time/TPC-H l_comment chunked without fsst 9.46744e+08 9.25342e+08 1.02313 ns
parquet_rs-zstd compress time/TPC-H l_comment chunked without fsst throughput 0.263241 0.269329 0.977395 bytes/ns
decompress time/TPC-H l_comment chunked without fsst 5.85624e+07 5.37889e+07 1.08875 ns
decompress time/TPC-H l_comment chunked without fsst throughput 4.25565 4.63332 0.918488 bytes/ns
parquet_rs-zstd decompress time/TPC-H l_comment chunked without fsst 2.54454e+08 2.54356e+08 1.00038 ns
parquet_rs-zstd decompress time/TPC-H l_comment chunked without fsst throughput 0.979438 0.979814 0.999616 bytes/ns
compress time/TPC-H l_comment chunked 1.0527e+09 1.02425e+09 1.02778 ns
compress time/TPC-H l_comment chunked throughput 0.236744 0.24332 0.972973 bytes/ns
parquet_rs-zstd compress time/TPC-H l_comment chunked 9.44818e+08 9.21895e+08 1.02487 ns
parquet_rs-zstd compress time/TPC-H l_comment chunked throughput 0.263777 0.270336 0.975738 bytes/ns
decompress time/TPC-H l_comment chunked 9.45042e+07 9.31146e+07 1.01492 ns
decompress time/TPC-H l_comment chunked throughput 2.63715 2.6765 0.985296 bytes/ns
parquet_rs-zstd decompress time/TPC-H l_comment chunked 2.55539e+08 2.55152e+08 1.00152 ns
parquet_rs-zstd decompress time/TPC-H l_comment chunked throughput 0.975277 0.976757 0.998485 bytes/ns
compress time/TPC-H l_comment canonical 1.04415e+09 1.0255e+09 1.01819 ns
compress time/TPC-H l_comment canonical throughput 0.238682 0.243023 0.982138 bytes/ns
parquet_rs-zstd compress time/TPC-H l_comment canonical 9.46253e+08 9.27755e+08 1.01994 ns
parquet_rs-zstd compress time/TPC-H l_comment canonical throughput 0.263376 0.268628 0.980451 bytes/ns
decompress time/TPC-H l_comment canonical 9.54323e+07 9.30573e+07 1.02552 ns
decompress time/TPC-H l_comment canonical throughput 2.61149 2.67814 0.975113 bytes/ns
parquet_rs-zstd decompress time/TPC-H l_comment canonical 2.54321e+08 2.53266e+08 1.00416 ns
parquet_rs-zstd decompress time/TPC-H l_comment canonical throughput 0.979945 0.984026 0.995853 bytes/ns

@danking danking closed this Jan 17, 2025
@robert3005 robert3005 deleted the dk/run-end-true-count-null-count branch January 29, 2025 22:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@danking