From 3cac0df1f5f818e5e4722a435fb83c44ca155883 Mon Sep 17 00:00:00 2001 From: panbingkun Date: Wed, 1 Nov 2023 23:45:52 +0800 Subject: [PATCH] [SPARK-45327][BUILD] Upgrade zstd-jni to 1.5.5-7 ### What changes were proposed in this pull request? The pr aims to upgrade zstd-jni from 1.5.5-5 to 1.5.5-7. ### Why are the changes needed? 1.Version compare: - v1.5.5-6 VS v1.5.5-7: https://github.com/luben/zstd-jni/compare/v1.5.5-6...v1.5.5-7 - v1.5.5-5 VS v1.5.5-6: https://github.com/luben/zstd-jni/compare/v.1.5.5-5...v1.5.5-6 2.Some changes include the following: - Add new method `getFrameContentSize` that will return also the error. https://github.com/luben/zstd-jni/commit/3f6c55e87a38255bdbeeca3e773f99bf9e8a7b7f - Fix error. https://github.com/luben/zstd-jni/commit/e0c66f1230377d4980ad6abc0767b36c860af538 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #43113 from panbingkun/SPARK-45327. Authored-by: panbingkun Signed-off-by: yangjie01 --- .../ZStandardBenchmark-jdk21-results.txt | 32 +++++++++---------- .../benchmarks/ZStandardBenchmark-results.txt | 28 ++++++++-------- dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +- pom.xml | 6 +++- 4 files changed, 36 insertions(+), 32 deletions(-) diff --git a/core/benchmarks/ZStandardBenchmark-jdk21-results.txt b/core/benchmarks/ZStandardBenchmark-jdk21-results.txt index 0915e73385686..19bd2faa9a5a5 100644 --- a/core/benchmarks/ZStandardBenchmark-jdk21-results.txt +++ b/core/benchmarks/ZStandardBenchmark-jdk21-results.txt @@ -2,26 +2,26 @@ Benchmark ZStandardCompressionCodec ================================================================================================ -OpenJDK 64-Bit Server VM 21+35 on Linux 5.15.0-1046-azure -Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz +OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1050-azure +Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative -------------------------------------------------------------------------------------------------------------------------------------- -Compression 10000 times at level 1 without buffer pool 2649 2653 6 0.0 264919.3 1.0X -Compression 10000 times at level 2 without buffer pool 2785 2785 1 0.0 278455.9 1.0X -Compression 10000 times at level 3 without buffer pool 3078 3082 6 0.0 307845.7 0.9X -Compression 10000 times at level 1 with buffer pool 2353 2378 35 0.0 235340.6 1.1X -Compression 10000 times at level 2 with buffer pool 2462 2466 6 0.0 246194.4 1.1X -Compression 10000 times at level 3 with buffer pool 2761 2765 6 0.0 276095.6 1.0X +Compression 10000 times at level 1 without buffer pool 1817 1819 3 0.0 181709.6 1.0X +Compression 10000 times at level 2 without buffer pool 2081 2083 3 0.0 208053.7 0.9X +Compression 10000 times at level 3 without buffer pool 2288 2290 3 0.0 228795.4 0.8X +Compression 10000 times at level 1 with buffer pool 1997 1998 1 0.0 199686.9 0.9X +Compression 10000 times at level 2 with buffer pool 2062 2063 1 0.0 206209.3 0.9X +Compression 10000 times at level 3 with buffer pool 2243 2243 1 0.0 224271.8 0.8X -OpenJDK 64-Bit Server VM 21+35 on Linux 5.15.0-1046-azure -Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz +OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1050-azure +Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------------------ -Decompression 10000 times from level 1 without buffer pool 2644 2644 0 0.0 264430.7 1.0X -Decompression 10000 times from level 2 without buffer pool 2597 2608 15 0.0 259715.2 1.0X -Decompression 10000 times from level 3 without buffer pool 2586 2600 20 0.0 258633.7 1.0X -Decompression 10000 times from level 1 with buffer pool 2290 2296 9 0.0 228957.6 1.2X -Decompression 10000 times from level 2 with buffer pool 2315 2319 6 0.0 231535.4 1.1X -Decompression 10000 times from level 3 with buffer pool 2283 2302 27 0.0 228308.3 1.2X +Decompression 10000 times from level 1 without buffer pool 1980 1981 1 0.0 197970.8 1.0X +Decompression 10000 times from level 2 without buffer pool 1978 1979 1 0.0 197813.4 1.0X +Decompression 10000 times from level 3 without buffer pool 1981 1983 2 0.0 198141.7 1.0X +Decompression 10000 times from level 1 with buffer pool 1825 1827 3 0.0 182475.2 1.1X +Decompression 10000 times from level 2 with buffer pool 1827 1827 0 0.0 182667.1 1.1X +Decompression 10000 times from level 3 with buffer pool 1826 1826 0 0.0 182579.8 1.1X diff --git a/core/benchmarks/ZStandardBenchmark-results.txt b/core/benchmarks/ZStandardBenchmark-results.txt index 5299a52dc7b8a..6f7efdd6c94c5 100644 --- a/core/benchmarks/ZStandardBenchmark-results.txt +++ b/core/benchmarks/ZStandardBenchmark-results.txt @@ -2,26 +2,26 @@ Benchmark ZStandardCompressionCodec ================================================================================================ -OpenJDK 64-Bit Server VM 17.0.8+7-LTS on Linux 5.15.0-1046-azure +OpenJDK 64-Bit Server VM 17.0.9+8-LTS on Linux 5.15.0-1050-azure Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative -------------------------------------------------------------------------------------------------------------------------------------- -Compression 10000 times at level 1 without buffer pool 2800 2801 2 0.0 279995.2 1.0X -Compression 10000 times at level 2 without buffer pool 2832 2846 19 0.0 283227.4 1.0X -Compression 10000 times at level 3 without buffer pool 2978 3003 35 0.0 297782.6 0.9X -Compression 10000 times at level 1 with buffer pool 2650 2652 2 0.0 265042.4 1.1X -Compression 10000 times at level 2 with buffer pool 2684 2688 5 0.0 268419.4 1.0X -Compression 10000 times at level 3 with buffer pool 2811 2816 8 0.0 281069.3 1.0X +Compression 10000 times at level 1 without buffer pool 2786 2787 2 0.0 278560.1 1.0X +Compression 10000 times at level 2 without buffer pool 2831 2833 3 0.0 283091.0 1.0X +Compression 10000 times at level 3 without buffer pool 2958 2959 2 0.0 295806.3 0.9X +Compression 10000 times at level 1 with buffer pool 211 214 4 0.0 21145.3 13.2X +Compression 10000 times at level 2 with buffer pool 253 255 1 0.0 25328.1 11.0X +Compression 10000 times at level 3 with buffer pool 370 371 1 0.0 37046.1 7.5X -OpenJDK 64-Bit Server VM 17.0.8+7-LTS on Linux 5.15.0-1046-azure +OpenJDK 64-Bit Server VM 17.0.9+8-LTS on Linux 5.15.0-1050-azure Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------------------ -Decompression 10000 times from level 1 without buffer pool 2747 2752 7 0.0 274669.2 1.0X -Decompression 10000 times from level 2 without buffer pool 2743 2748 8 0.0 274265.3 1.0X -Decompression 10000 times from level 3 without buffer pool 2743 2750 10 0.0 274344.4 1.0X -Decompression 10000 times from level 1 with buffer pool 2608 2608 0 0.0 260803.0 1.1X -Decompression 10000 times from level 2 with buffer pool 2608 2608 0 0.0 260804.2 1.1X -Decompression 10000 times from level 3 with buffer pool 2605 2607 3 0.0 260514.9 1.1X +Decompression 10000 times from level 1 without buffer pool 2745 2748 5 0.0 274454.0 1.0X +Decompression 10000 times from level 2 without buffer pool 2744 2745 1 0.0 274438.1 1.0X +Decompression 10000 times from level 3 without buffer pool 2746 2746 1 0.0 274586.0 1.0X +Decompression 10000 times from level 1 with buffer pool 2587 2588 1 0.0 258707.4 1.1X +Decompression 10000 times from level 2 with buffer pool 2586 2586 1 0.0 258566.8 1.1X +Decompression 10000 times from level 3 with buffer pool 2589 2589 0 0.0 258870.6 1.1X diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3 index b1c1ed8d81dae..6364ec48fb664 100644 --- a/dev/deps/spark-deps-hadoop-3-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-3-hive-2.3 @@ -264,4 +264,4 @@ xz/1.9//xz-1.9.jar zjsonpatch/0.3.0//zjsonpatch-0.3.0.jar zookeeper-jute/3.6.3//zookeeper-jute-3.6.3.jar zookeeper/3.6.3//zookeeper-3.6.3.jar -zstd-jni/1.5.5-5//zstd-jni-1.5.5-5.jar +zstd-jni/1.5.5-7//zstd-jni-1.5.5-7.jar diff --git a/pom.xml b/pom.xml index 811d075e203a1..2e0c95516c177 100644 --- a/pom.xml +++ b/pom.xml @@ -799,7 +799,7 @@ com.github.luben zstd-jni - 1.5.5-5 + 1.5.5-7 com.clearspring.analytics @@ -2662,6 +2662,10 @@ javax.annotation javax.annotation-api + + com.github.luben + zstd-jni +