Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: BitPackedArray ptype changed by compute funcs #1724

Merged
merged 8 commits into from
Dec 19, 2024
Merged

fix: BitPackedArray ptype changed by compute funcs #1724

merged 8 commits into from
Dec 19, 2024
+93 −21

Conversation

a10y
Copy link
Contributor

@a10y a10y commented Dec 19, 2024

Still seeing develop clickbench failures related to ptype getting changed by TakeFn. This should fix it. Added a few tests as well.

@a10y a10y added the benchmark Run benchmarks on this branch label Dec 19, 2024
@github-actions github-actions bot removed the benchmark Run benchmarks on this branch label Dec 19, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 19, 2024 •
edited
Loading

Benchmarks: random_access

Table of Results
name PR d3dc038 base fd632a7 ratio (PR/base) unit
random-access/vortex-tokio-local-disk 3.96006e+06 3.81897e+06 1.03695 ns
random-access/vortex-local-fs 3.70785e+06 3.45097e+06 1.07444 ns
random-access/parquet-tokio-local-disk 2.25905e+08 2.27496e+08 0.993008 ns

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 19, 2024 •
edited
Loading

Benchmarks: datafusion

Table of Results
name PR d3dc038 base fd632a7 ratio (PR/base) unit
arrow/planning 808201 799388 1.01102 ns
arrow/exec 1.77867e+06 1.76727e+06 1.00645 ns
vortex-pushdown-compressed/planning 502494 503727 0.997552 ns
vortex-pushdown-compressed/exec 2.6101e+06 2.61266e+06 0.99902 ns
vortex-pushdown-uncompressed/planning 501729 498665 1.00614 ns
vortex-pushdown-uncompressed/exec 1.47414e+06 1.47443e+06 0.999798 ns
vortex-nopushdown-compressed/planning 828653 822033 1.00805 ns
vortex-nopushdown-compressed/exec 3.04097e+06 3.05774e+06 0.994515 ns
vortex-nopushdown-uncompressed/planning 818087 818379 0.999643 ns
vortex-nopushdown-uncompressed/exec 4.90961e+06 5.63634e+06 0.871064 ns

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 19, 2024 •
edited
Loading

Benchmarks: TPC-H

Table of Results
name PR d3dc038 base fd632a7 ratio (PR/base) unit
tpch_q01/arrow 556984266 5.63487e+08 0.988459 ns
tpch_q01/parquet 785217542 7.71247e+08 1.01811 ns
tpch_q01/vortex-file-compressed 464805412 4.63726e+08 1.00233 ns
tpch_q02/arrow 143083288 1.42137e+08 1.00665 ns
tpch_q02/parquet 180424798 1.76651e+08 1.02136 ns
tpch_q02/vortex-file-compressed 148685996 1.46193e+08 1.01705 ns
tpch_q03/arrow 173376661 1.73077e+08 1.00173 ns
tpch_q03/parquet 377999306 3.77074e+08 1.00245 ns
tpch_q03/vortex-file-compressed 211777890 2.13082e+08 0.993881 ns
tpch_q04/arrow 181764864 1.82244e+08 0.99737 ns
tpch_q04/parquet 213881411 2.16018e+08 0.99011 ns
tpch_q04/vortex-file-compressed 162466404 1.59593e+08 1.018 ns
tpch_q05/arrow 325690394 3.31863e+08 0.981399 ns
tpch_q05/parquet 511835471 5.12593e+08 0.998522 ns
tpch_q05/vortex-file-compressed 360049247 3.50613e+08 1.02691 ns
tpch_q06/arrow 26239914 2.59408e+07 1.01153 ns
tpch_q06/parquet 152953889 1.5164e+08 1.00866 ns
tpch_q06/vortex-file-compressed 62029577 5.86231e+07 1.05811 ns
tpch_q07/arrow 631041756 6.31923e+08 0.998605 ns
tpch_q07/parquet 776902781 7.87453e+08 0.986602 ns
tpch_q07/vortex-file-compressed 632253053 6.25704e+08 1.01047 ns
tpch_q08/arrow 260844676 2.65311e+08 0.983167 ns
tpch_q08/parquet 549142303 5.43952e+08 1.00954 ns
tpch_q08/vortex-file-compressed 340518596 3.35866e+08 1.01385 ns
tpch_q09/arrow 471112601 4.87706e+08 0.965977 ns
tpch_q09/parquet 783257529 7.8581e+08 0.996752 ns
tpch_q09/vortex-file-compressed 574184919 5.83328e+08 0.984326 ns
tpch_q10/arrow 274431923 2.71806e+08 1.00966 ns
tpch_q10/parquet 522580606 5.18445e+08 1.00798 ns
tpch_q10/vortex-file-compressed 282515136 2.77571e+08 1.01781 ns
tpch_q11/arrow 142449369 1.4368e+08 0.991437 ns
tpch_q11/parquet 154550310 1.47737e+08 1.04612 ns
tpch_q11/vortex-file-compressed 128963996 1.29169e+08 0.998415 ns
tpch_q12/arrow 182083255 1.82623e+08 0.997045 ns
tpch_q12/parquet 328840568 3.28275e+08 1.00172 ns
tpch_q12/vortex-file-compressed 232536729 2.26925e+08 1.02473 ns
tpch_q13/arrow 174040091 1.88675e+08 0.922433 ns
tpch_q13/parquet 322028316 3.22203e+08 0.999457 ns
tpch_q13/vortex-file-compressed 191420838 1.82173e+08 1.05077 ns
tpch_q14/arrow 39374054 3.84939e+07 1.02287 ns
tpch_q14/parquet 235225953 2.33203e+08 1.00867 ns
tpch_q14/vortex-file-compressed 68593511 6.82069e+07 1.00567 ns
tpch_q15/arrow 69131514 6.87686e+07 1.00528 ns
tpch_q15/parquet 333086098 3.21497e+08 1.03605 ns
tpch_q15/vortex-file-compressed 107754392 1.08359e+08 0.994417 ns
tpch_q16/arrow 104378105 1.04834e+08 0.995655 ns
tpch_q16/parquet 120743171 1.22952e+08 0.982039 ns
tpch_q16/vortex-file-compressed 108791592 1.08719e+08 1.00067 ns
tpch_q17/arrow 617118171 6.31992e+08 0.976466 ns
tpch_q17/parquet 673612161 6.7668e+08 0.995466 ns
tpch_q17/vortex-file-compressed 577177765 5.84652e+08 0.987217 ns
tpch_q18/arrow 1174733982 1.17491e+09 0.999854 ns
tpch_q18/parquet 1391629537 1.36427e+09 1.02006 ns
tpch_q18/vortex-file-compressed 1188565626 1.1894e+09 0.999296 ns
tpch_q19/arrow 153157146 1.53618e+08 0.996998 ns
tpch_q19/parquet 429705120 4.23231e+08 1.0153 ns
tpch_q19/vortex-file-compressed 117725330 1.11904e+08 1.05202 ns
tpch_q20/arrow 219548159 2.27707e+08 0.964168 ns
tpch_q20/parquet 365344553 3.50826e+08 1.04138 ns
tpch_q20/vortex-file-compressed 241828304 2.43572e+08 0.992841 ns
tpch_q21/arrow 992567060 1.00376e+09 0.98885 ns
tpch_q21/parquet 1133380692 1.11396e+09 1.01744 ns
tpch_q21/vortex-file-compressed 949768378 9.31073e+08 1.02008 ns
tpch_q22/arrow 78045757 7.7483e+07 1.00726 ns
tpch_q22/parquet 109278898 1.09567e+08 0.997368 ns
tpch_q22/vortex-file-compressed 83827392 8.24281e+07 1.01698 ns

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 19, 2024 •
edited
Loading

Benchmarks: compress

Table of Results
name PR d3dc038 base fd632a7 ratio (PR/base) unit
compress time/taxi 1.49208e+09 1.46907e+09 1.01566 ns
compress time/taxi throughput 0.31554 0.320481 0.984581 bytes/ns
parquet_rs-zstd compress time/taxi 1.68204e+09 1.68986e+09 0.99537 ns
parquet_rs-zstd compress time/taxi throughput 0.279905 0.278609 1.00465 bytes/ns
decompress time/taxi 4.25991e+08 4.24548e+08 1.0034 ns
decompress time/taxi throughput 1.10521 1.10897 0.996614 bytes/ns
parquet_rs-zstd decompress time/taxi 3.09399e+08 3.13644e+08 0.986465 ns
parquet_rs-zstd decompress time/taxi throughput 1.5217 1.5011 1.01372 bytes/ns
compress time/AirlineSentiment 641234 640659 1.0009 ns
compress time/AirlineSentiment throughput 0.00322191 0.0032248 0.999103 bytes/ns
parquet_rs-zstd compress time/AirlineSentiment 57669.2 56947.2 1.01268 ns
parquet_rs-zstd compress time/AirlineSentiment throughput 0.035825 0.0362792 0.987482 bytes/ns
decompress time/AirlineSentiment 101445 102864 0.986204 ns
decompress time/AirlineSentiment throughput 0.0203657 0.0200848 1.01399 bytes/ns
parquet_rs-zstd decompress time/AirlineSentiment 32370.4 32448.8 0.997584 ns
parquet_rs-zstd decompress time/AirlineSentiment throughput 0.0638237 0.0636695 1.00242 bytes/ns
compress time/Arade 2.65848e+09 2.65451e+09 1.0015 ns
compress time/Arade throughput 0.296047 0.29649 0.998506 bytes/ns
parquet_rs-zstd compress time/Arade 2.87889e+09 2.88405e+09 0.998213 ns
parquet_rs-zstd compress time/Arade throughput 0.273381 0.272893 1.00179 bytes/ns
decompress time/Arade 4.08911e+08 4.11257e+08 0.994295 ns
decompress time/Arade throughput 1.92471 1.91373 1.00574 bytes/ns
parquet_rs-zstd decompress time/Arade 6.9625e+08 7.03258e+08 0.990035 ns
parquet_rs-zstd decompress time/Arade throughput 1.13039 1.11913 1.01007 bytes/ns
compress time/Bimbo 1.08601e+10 1.09058e+10 0.995812 ns
compress time/Bimbo throughput 0.655736 0.65299 1.00421 bytes/ns
parquet_rs-zstd compress time/Bimbo 2.00369e+10 2.06846e+10 0.968688 ns
parquet_rs-zstd compress time/Bimbo throughput 0.355412 0.344283 1.03232 bytes/ns
decompress time/Bimbo 3.70227e+09 3.7227e+09 0.994514 ns
decompress time/Bimbo throughput 1.92351 1.91295 1.00552 bytes/ns
parquet_rs-zstd decompress time/Bimbo 4.50442e+09 4.79456e+09 0.939486 ns
parquet_rs-zstd decompress time/Bimbo throughput 1.58097 1.4853 1.06441 bytes/ns
compress time/CMSprovider 1.29002e+10 1.305e+10 0.988525 ns
compress time/CMSprovider throughput 0.399155 0.394575 1.01161 bytes/ns
parquet_rs-zstd compress time/CMSprovider 1.79456e+10 1.8416e+10 0.974456 ns
parquet_rs-zstd compress time/CMSprovider throughput 0.286934 0.279604 1.02621 bytes/ns
decompress time/CMSprovider 2.96505e+09 2.77197e+09 1.06965 ns
decompress time/CMSprovider throughput 1.73663 1.8576 0.934881 bytes/ns
parquet_rs-zstd decompress time/CMSprovider 5.48186e+09 5.4627e+09 1.00351 ns
parquet_rs-zstd decompress time/CMSprovider throughput 0.939317 0.94261 0.996506 bytes/ns
compress time/Euro2016 2.09217e+09 2.19242e+09 0.954277 ns
compress time/Euro2016 throughput 0.187965 0.179371 1.04791 bytes/ns
parquet_rs-zstd compress time/Euro2016 1.52624e+09 1.52793e+09 0.998899 ns
parquet_rs-zstd compress time/Euro2016 throughput 0.257663 0.257379 1.0011 bytes/ns
decompress time/Euro2016 2.35848e+08 2.67035e+08 0.883209 ns
decompress time/Euro2016 throughput 1.66741 1.47267 1.13224 bytes/ns
parquet_rs-zstd decompress time/Euro2016 4.82497e+08 4.87252e+08 0.990242 ns
parquet_rs-zstd decompress time/Euro2016 throughput 0.815043 0.80709 1.00985 bytes/ns
compress time/Food 1.06142e+09 1.04915e+09 1.01169 ns
compress time/Food throughput 0.313469 0.317134 0.988442 bytes/ns
parquet_rs-zstd compress time/Food 1.02102e+09 1.04076e+09 0.98103 ns
parquet_rs-zstd compress time/Food throughput 0.325872 0.319691 1.01934 bytes/ns
decompress time/Food 1.10295e+08 1.10991e+08 0.993721 ns
decompress time/Food throughput 3.01667 2.99772 1.00632 bytes/ns
parquet_rs-zstd decompress time/Food 2.22259e+08 2.2389e+08 0.992715 ns
parquet_rs-zstd decompress time/Food throughput 1.497 1.48609 1.00734 bytes/ns
compress time/HashTags 2.45359e+09 2.47228e+09 0.992442 ns
compress time/HashTags throughput 0.327888 0.32541 1.00762 bytes/ns
parquet_rs-zstd compress time/HashTags 2.42839e+09 2.41133e+09 1.00707 ns
parquet_rs-zstd compress time/HashTags throughput 0.331291 0.333634 0.992976 bytes/ns
decompress time/HashTags 3.51357e+08 3.538e+08 0.993095 ns
decompress time/HashTags throughput 2.28971 2.27389 1.00695 bytes/ns
parquet_rs-zstd decompress time/HashTags 7.67374e+08 7.666e+08 1.00101 ns
parquet_rs-zstd decompress time/HashTags throughput 1.04838 1.04944 0.998991 bytes/ns
compress time/TPC-H l_comment chunked without fsst 3.06635e+09 3.05018e+09 1.0053 ns
compress time/TPC-H l_comment chunked without fsst throughput 0.0812775 0.0817084 0.994726 bytes/ns
parquet_rs-zstd compress time/TPC-H l_comment chunked without fsst 8.99622e+08 9.04115e+08 0.995031 ns
parquet_rs-zstd compress time/TPC-H l_comment chunked without fsst throughput 0.277033 0.275657 1.00499 bytes/ns
decompress time/TPC-H l_comment chunked without fsst 7.14291e+07 6.97823e+07 1.0236 ns
decompress time/TPC-H l_comment chunked without fsst throughput 3.48912 3.57147 0.976944 bytes/ns
parquet_rs-zstd decompress time/TPC-H l_comment chunked without fsst 2.49665e+08 2.49621e+08 1.00018 ns
parquet_rs-zstd decompress time/TPC-H l_comment chunked without fsst throughput 0.998237 0.998413 0.999824 bytes/ns
compress time/TPC-H l_comment chunked 9.95719e+08 9.8046e+08 1.01556 ns
compress time/TPC-H l_comment chunked throughput 0.250297 0.254192 0.984675 bytes/ns
parquet_rs-zstd compress time/TPC-H l_comment chunked 9.00914e+08 9.02918e+08 0.997781 ns
parquet_rs-zstd compress time/TPC-H l_comment chunked throughput 0.276636 0.276022 1.00222 bytes/ns
decompress time/TPC-H l_comment chunked 9.59169e+07 9.71502e+07 0.987305 ns
decompress time/TPC-H l_comment chunked throughput 2.59834 2.56536 1.01286 bytes/ns
parquet_rs-zstd decompress time/TPC-H l_comment chunked 2.49289e+08 2.50098e+08 0.996764 ns
parquet_rs-zstd decompress time/TPC-H l_comment chunked throughput 0.999744 0.996509 1.00325 bytes/ns
compress time/TPC-H l_comment canonical 9.96957e+08 9.87993e+08 1.00907 ns
compress time/TPC-H l_comment canonical throughput 0.249985 0.252253 0.991008 bytes/ns
parquet_rs-zstd compress time/TPC-H l_comment canonical 9.03501e+08 9.31652e+08 0.969785 ns
parquet_rs-zstd compress time/TPC-H l_comment canonical throughput 0.275843 0.267508 1.03116 bytes/ns
decompress time/TPC-H l_comment canonical 9.57647e+07 1.02721e+08 0.932279 ns
decompress time/TPC-H l_comment canonical throughput 2.60247 2.42623 1.07264 bytes/ns
parquet_rs-zstd decompress time/TPC-H l_comment canonical 2.48175e+08 2.51949e+08 0.985023 ns
parquet_rs-zstd decompress time/TPC-H l_comment canonical throughput 1.00423 0.989186 1.0152 bytes/ns

@a10y a10y added benchmark Run benchmarks on this branch do not merge Pull requests that are not intended to merge labels Dec 19, 2024
@github-actions github-actions bot removed the benchmark Run benchmarks on this branch label Dec 19, 2024
@a10y a10y added the benchmark Run benchmarks on this branch label Dec 19, 2024
@github-actions github-actions bot removed the benchmark Run benchmarks on this branch label Dec 19, 2024
@a10y a10y removed the do not merge Pull requests that are not intended to merge label Dec 19, 2024
@a10y a10y enabled auto-merge (squash) December 19, 2024 15:32
@a10y a10y disabled auto-merge December 19, 2024 15:32
@a10y a10y enabled auto-merge (squash) December 19, 2024 15:33
a10y
a10y commented Dec 19, 2024
View reviewed changes
encodings/fastlanes/src/bitpacking/compute/take.rs
)
.unwrap();

let taken_primitive = super::take_primitive::<u32, u64>(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We usually test blackbox, but had to go whitebox here since the take impl has a configurable UNPACK_THRESHOLD and I didn't want us to stop testing this codepath in case we changed the constant.

This test was able to repro the clickbench error before the fix above to reinterpret_cast before patching

@a10y a10y disabled auto-merge December 19, 2024 15:59
@a10y a10y enabled auto-merge (squash) December 19, 2024 16:08
@a10y a10y merged commit 44a9277 into develop Dec 19, 2024
20 checks passed
@a10y a10y deleted the aduffy/bitpack-compute-changes-ptype branch December 19, 2024 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AdamGS AdamGS AdamGS left review comments

@robert3005 robert3005 robert3005 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@a10y @robert3005 @AdamGS