Skip to content

Commit

Permalink
[SPARK-49753][BUILD] Upgrade ZSTD-JNI to 1.5.6-6
Browse files Browse the repository at this point in the history 
### What changes were proposed in this pull request?
The pr aims to upgrade `zstd-jni` from `1.5.6-5` to `1.5.6-6`.

### Why are the changes needed?
The new version allow including compression level when training a dictionary:  luben/zstd-jni@3ca26ee#diff-bd5c0f62db7cb85cac88c7b6cfad1c0e5e2f433ba45097761654829627b7a31c

All changes in the new version are as follows:
- luben/zstd-jni@v1.5.6-5...v1.5.6-6

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Pass GitHub Actions

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes #48204 from LuciferYang/zstd-jni-1.5.6-6.

Lead-authored-by: yangjie01 <[email protected]>
Co-authored-by: YangJie <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
@LuciferYang @dongjoon-hyun
LuciferYang authored and dongjoon-hyun committed Sep 24, 2024
1 parent 64ea50e commit 438a6e7
Show file tree
Hide file tree
Showing 4 changed files with 58 additions and 58 deletions.
56 changes: 28 additions & 28 deletions core/benchmarks/ZStandardBenchmark-jdk21-results.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,48 +2,48 @@
Benchmark ZStandardCompressionCodec
================================================================================================

OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.5.0-1025-azure
OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.8.0-1014-azure
AMD EPYC 7763 64-Core Processor
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------------
Compression 10000 times at level 1 without buffer pool 657 670 14 0.0 65699.2 1.0X
Compression 10000 times at level 2 without buffer pool 697 697 1 0.0 69673.4 0.9X
Compression 10000 times at level 3 without buffer pool 799 802 3 0.0 79855.2 0.8X
Compression 10000 times at level 1 with buffer pool 593 595 1 0.0 59326.9 1.1X
Compression 10000 times at level 2 with buffer pool 622 624 3 0.0 62194.1 1.1X
Compression 10000 times at level 3 with buffer pool 732 733 1 0.0 73178.6 0.9X
Compression 10000 times at level 1 without buffer pool 659 676 16 0.0 65860.7 1.0X
Compression 10000 times at level 2 without buffer pool 721 723 2 0.0 72135.5 0.9X
Compression 10000 times at level 3 without buffer pool 815 816 1 0.0 81500.6 0.8X
Compression 10000 times at level 1 with buffer pool 608 609 0 0.0 60846.6 1.1X
Compression 10000 times at level 2 with buffer pool 645 647 3 0.0 64476.3 1.0X
Compression 10000 times at level 3 with buffer pool 746 746 1 0.0 74584.0 0.9X

OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.5.0-1025-azure
OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.8.0-1014-azure
AMD EPYC 7763 64-Core Processor
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------------------
Decompression 10000 times from level 1 without buffer pool 813 820 11 0.0 81273.2 1.0X
Decompression 10000 times from level 2 without buffer pool 810 813 3 0.0 80986.2 1.0X
Decompression 10000 times from level 3 without buffer pool 812 813 2 0.0 81183.1 1.0X
Decompression 10000 times from level 1 with buffer pool 746 747 2 0.0 74568.7 1.1X
Decompression 10000 times from level 2 with buffer pool 744 746 2 0.0 74414.5 1.1X
Decompression 10000 times from level 3 with buffer pool 745 746 1 0.0 74538.6 1.1X
Decompression 10000 times from level 1 without buffer pool 828 829 1 0.0 82822.6 1.0X
Decompression 10000 times from level 2 without buffer pool 829 829 1 0.0 82900.7 1.0X
Decompression 10000 times from level 3 without buffer pool 828 833 8 0.0 82784.4 1.0X
Decompression 10000 times from level 1 with buffer pool 758 760 2 0.0 75756.5 1.1X
Decompression 10000 times from level 2 with buffer pool 758 758 1 0.0 75772.3 1.1X
Decompression 10000 times from level 3 with buffer pool 759 759 0 0.0 75852.7 1.1X

OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.5.0-1025-azure
OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.8.0-1014-azure
AMD EPYC 7763 64-Core Processor
Parallel Compression at level 3: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Parallel Compression with 0 workers 48 49 1 0.0 374256.1 1.0X
Parallel Compression with 1 workers 34 36 3 0.0 267557.3 1.4X
Parallel Compression with 2 workers 34 38 2 0.0 263684.3 1.4X
Parallel Compression with 4 workers 37 39 2 0.0 289956.1 1.3X
Parallel Compression with 8 workers 39 41 1 0.0 306975.2 1.2X
Parallel Compression with 16 workers 44 45 1 0.0 340992.0 1.1X
Parallel Compression with 0 workers 58 59 1 0.0 452489.9 1.0X
Parallel Compression with 1 workers 42 45 4 0.0 330066.0 1.4X
Parallel Compression with 2 workers 40 42 1 0.0 312560.3 1.4X
Parallel Compression with 4 workers 40 42 2 0.0 308802.7 1.5X
Parallel Compression with 8 workers 41 45 3 0.0 321331.3 1.4X
Parallel Compression with 16 workers 44 45 1 0.0 343311.5 1.3X

OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.5.0-1025-azure
OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.8.0-1014-azure
AMD EPYC 7763 64-Core Processor
Parallel Compression at level 9: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Parallel Compression with 0 workers 156 158 1 0.0 1220760.5 1.0X
Parallel Compression with 1 workers 191 192 2 0.0 1495168.2 0.8X
Parallel Compression with 2 workers 111 117 5 0.0 864459.9 1.4X
Parallel Compression with 4 workers 106 109 2 0.0 831025.5 1.5X
Parallel Compression with 8 workers 112 115 2 0.0 875732.7 1.4X
Parallel Compression with 16 workers 110 114 2 0.0 858160.9 1.4X
Parallel Compression with 0 workers 158 160 2 0.0 1234257.6 1.0X
Parallel Compression with 1 workers 193 194 1 0.0 1507686.4 0.8X
Parallel Compression with 2 workers 113 127 11 0.0 881068.0 1.4X
Parallel Compression with 4 workers 109 111 2 0.0 849241.3 1.5X
Parallel Compression with 8 workers 111 115 3 0.0 869455.2 1.4X
Parallel Compression with 16 workers 113 116 2 0.0 881832.5 1.4X


56 changes: 28 additions & 28 deletions core/benchmarks/ZStandardBenchmark-results.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,48 +2,48 @@
Benchmark ZStandardCompressionCodec
================================================================================================

OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.8.0-1014-azure
AMD EPYC 7763 64-Core Processor
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------------
Compression 10000 times at level 1 without buffer pool 638 638 0 0.0 63765.0 1.0X
Compression 10000 times at level 2 without buffer pool 675 676 1 0.0 67529.4 0.9X
Compression 10000 times at level 3 without buffer pool 775 783 11 0.0 77531.6 0.8X
Compression 10000 times at level 1 with buffer pool 572 573 1 0.0 57223.2 1.1X
Compression 10000 times at level 2 with buffer pool 603 605 1 0.0 60323.7 1.1X
Compression 10000 times at level 3 with buffer pool 720 727 6 0.0 71980.9 0.9X
Compression 10000 times at level 1 without buffer pool 257 259 2 0.0 25704.2 1.0X
Compression 10000 times at level 2 without buffer pool 674 676 2 0.0 67396.3 0.4X
Compression 10000 times at level 3 without buffer pool 775 787 11 0.0 77497.9 0.3X
Compression 10000 times at level 1 with buffer pool 573 574 0 0.0 57347.3 0.4X
Compression 10000 times at level 2 with buffer pool 602 603 2 0.0 60162.8 0.4X
Compression 10000 times at level 3 with buffer pool 722 725 3 0.0 72247.3 0.4X

OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.8.0-1014-azure
AMD EPYC 7763 64-Core Processor
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------------------
Decompression 10000 times from level 1 without buffer pool 584 585 1 0.0 58381.0 1.0X
Decompression 10000 times from level 2 without buffer pool 585 585 0 0.0 58465.9 1.0X
Decompression 10000 times from level 3 without buffer pool 585 586 1 0.0 58499.5 1.0X
Decompression 10000 times from level 1 with buffer pool 534 534 0 0.0 53375.7 1.1X
Decompression 10000 times from level 2 with buffer pool 533 533 0 0.0 53312.3 1.1X
Decompression 10000 times from level 3 with buffer pool 533 533 1 0.0 53255.1 1.1X
Decompression 10000 times from level 1 without buffer pool 176 177 1 0.1 17641.2 1.0X
Decompression 10000 times from level 2 without buffer pool 176 178 1 0.1 17628.9 1.0X
Decompression 10000 times from level 3 without buffer pool 175 176 0 0.1 17506.1 1.0X
Decompression 10000 times from level 1 with buffer pool 151 152 1 0.1 15051.5 1.2X
Decompression 10000 times from level 2 with buffer pool 150 151 1 0.1 14998.0 1.2X
Decompression 10000 times from level 3 with buffer pool 150 151 0 0.1 15019.4 1.2X

OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.8.0-1014-azure
AMD EPYC 7763 64-Core Processor
Parallel Compression at level 3: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Parallel Compression with 0 workers 46 48 1 0.0 360483.5 1.0X
Parallel Compression with 1 workers 34 36 2 0.0 265816.1 1.4X
Parallel Compression with 2 workers 33 36 2 0.0 254525.8 1.4X
Parallel Compression with 4 workers 34 37 1 0.0 266270.8 1.4X
Parallel Compression with 8 workers 37 39 1 0.0 289289.2 1.2X
Parallel Compression with 16 workers 41 43 1 0.0 320243.3 1.1X
Parallel Compression with 0 workers 57 57 0 0.0 444425.2 1.0X
Parallel Compression with 1 workers 42 44 3 0.0 325107.6 1.4X
Parallel Compression with 2 workers 38 39 2 0.0 294840.0 1.5X
Parallel Compression with 4 workers 36 37 1 0.0 282143.1 1.6X
Parallel Compression with 8 workers 39 40 1 0.0 303793.6 1.5X
Parallel Compression with 16 workers 41 43 1 0.0 324165.5 1.4X

OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.8.0-1014-azure
AMD EPYC 7763 64-Core Processor
Parallel Compression at level 9: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Parallel Compression with 0 workers 154 156 2 0.0 1205934.0 1.0X
Parallel Compression with 1 workers 191 194 4 0.0 1495729.9 0.8X
Parallel Compression with 2 workers 110 114 5 0.0 859158.9 1.4X
Parallel Compression with 4 workers 105 108 3 0.0 822932.2 1.5X
Parallel Compression with 8 workers 109 113 2 0.0 851560.0 1.4X
Parallel Compression with 16 workers 111 115 2 0.0 870695.9 1.4X
Parallel Compression with 0 workers 156 158 1 0.0 1220298.8 1.0X
Parallel Compression with 1 workers 188 189 1 0.0 1467911.4 0.8X
Parallel Compression with 2 workers 111 118 7 0.0 866985.2 1.4X
Parallel Compression with 4 workers 106 109 2 0.0 827592.1 1.5X
Parallel Compression with 8 workers 114 116 2 0.0 888419.5 1.4X
Parallel Compression with 16 workers 111 115 2 0.0 868463.5 1.4X


2 changes: 1 addition & 1 deletion dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Original file line number Diff line number Diff line change
Expand Up @@ -280,4 +280,4 @@ xz/1.10//xz-1.10.jar
zjsonpatch/0.3.0//zjsonpatch-0.3.0.jar
zookeeper-jute/3.9.2//zookeeper-jute-3.9.2.jar
zookeeper/3.9.2//zookeeper-3.9.2.jar
zstd-jni/1.5.6-5//zstd-jni-1.5.6-5.jar
zstd-jni/1.5.6-6//zstd-jni-1.5.6-6.jar
2 changes: 1 addition & 1 deletion pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -835,7 +835,7 @@
<dependency>
<groupId>com.github.luben</groupId>
<artifactId>zstd-jni</artifactId>
<version>1.5.6-5</version>
<version>1.5.6-6</version>
</dependency>
<dependency>
<groupId>com.clearspring.analytics</groupId>
Expand Down

0 comments on commit 438a6e7

Please sign in to comment.