Skip to content

Commit

Permalink
[SPARK-49410][SQL][TESTS] Update collation benchmarks
Browse files Browse the repository at this point in the history 
### What changes were proposed in this pull request?
Updating collation benchmarks to compute relative time, instead of relative speed. Also, re-run all collation benchmarks (after introducing various perf improvements as part of the recent collation effort).

### Why are the changes needed?
Relative speed is displayed in format: "x0.0" for collations that are slower than the referent collation (UTF8_BINARY).

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Existing tests.

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes #47893 from uros-db/update-benchmarks.

Authored-by: Uros Bojanic <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
  • Loading branch information
@uros-db @MaxGekk
uros-db authored and MaxGekk committed Aug 29, 2024
1 parent dad2b76 commit 1e67659
Show file tree
Hide file tree
Showing 6 changed files with 134 additions and 130 deletions.
12 changes: 8 additions & 4 deletions core/src/test/scala/org/apache/spark/benchmark/Benchmark.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -94,9 +94,11 @@ private[spark] class Benchmark(
/**
* Runs the benchmark and outputs the results to stdout. This should be copied and added as
* a comment with the benchmark. Although the results vary from machine to machine, it should
* provide some baseline.
* provide some baseline. If `relativeTime` is set to `true`, the `Relative` column will be
* the relative time of each case relative to the first case (less is better). Otherwise, it
* will be the relative execution speed of each case relative to the first case (more is better).
*/
def run(): Unit = {
def run(relativeTime: Boolean = false): Unit = {
require(benchmarks.nonEmpty)
// scalastyle:off
println("Running benchmark: " + name)
Expand All @@ -112,18 +114,20 @@ private[spark] class Benchmark(
out.println(Benchmark.getJVMOSInfo())
out.println(Benchmark.getProcessorName())
val nameLen = Math.max(40, Math.max(name.length, benchmarks.map(_.name.length).max))
val relativeHeader = if (relativeTime) "Relative time" else "Relative"
out.printf(s"%-${nameLen}s %14s %14s %11s %12s %13s %10s\n",
name + ":", "Best Time(ms)", "Avg Time(ms)", "Stdev(ms)", "Rate(M/s)", "Per Row(ns)", "Relative")
name + ":", "Best Time(ms)", "Avg Time(ms)", "Stdev(ms)", "Rate(M/s)", "Per Row(ns)", relativeHeader)
out.println("-" * (nameLen + 80))
results.zip(benchmarks).foreach { case (result, benchmark) =>
val relative = if (relativeTime) result.bestMs / firstBest else firstBest / result.bestMs
out.printf(s"%-${nameLen}s %14s %14s %11s %12s %13s %10s\n",
benchmark.name,
"%5.0f" format result.bestMs,
"%4.0f" format result.avgMs,
"%5.0f" format result.stdevMs,
"%10.1f" format result.bestRate,
"%6.1f" format (1000 / result.bestRate),
"%3.1fX" format (firstBest / result.bestMs))
"%3.1fX" format relative)
}
out.println()
// scalastyle:on
Expand Down
60 changes: 30 additions & 30 deletions sql/core/benchmarks/CollationBenchmark-jdk21-results.txt
View file
Original file line number Diff line number Diff line change
@@ -1,54 +1,54 @@
OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - equalsFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
collation unit benchmarks - equalsFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
--------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 1344 1345 1 0.1 13438.0 1.0X
UTF8_LCASE 2617 2619 3 0.0 26172.7 0.5X
UNICODE 16947 16976 41 0.0 169465.6 0.1X
UNICODE_CI 16500 16507 10 0.0 164997.5 0.1X
UTF8_BINARY 1353 1353 1 0.1 13526.6 1.0X
UTF8_LCASE 2703 2705 3 0.0 27032.4 2.0X
UNICODE 16848 16894 65 0.0 168482.9 12.5X
UNICODE_CI 16362 16367 8 0.0 163615.6 12.1X

OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - compareFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
collation unit benchmarks - compareFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
---------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 2555 2560 7 0.0 25553.1 1.0X
UTF8_LCASE 3539 3540 0 0.0 35392.8 0.7X
UNICODE 17154 17161 10 0.0 171541.5 0.1X
UNICODE_CI 16915 16926 17 0.0 169146.6 0.2X
UTF8_BINARY 2640 2642 3 0.0 26401.5 1.0X
UTF8_LCASE 3616 3618 2 0.0 36164.8 1.4X
UNICODE 17465 17470 7 0.0 174650.9 6.6X
UNICODE_CI 17251 17264 18 0.0 172510.9 6.5X

OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - hashFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
collation unit benchmarks - hashFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 2771 2771 0 0.0 27708.6 1.0X
UTF8_LCASE 5346 5347 2 0.0 53462.4 0.5X
UNICODE 67678 67692 21 0.0 676775.1 0.0X
UNICODE_CI 57978 57982 6 0.0 579780.7 0.0X
UTF8_BINARY 2843 2844 1 0.0 28427.2 1.0X
UTF8_LCASE 5417 5437 28 0.0 54170.7 1.9X
UNICODE 68601 68619 26 0.0 686010.8 24.1X
UNICODE_CI 56342 56361 26 0.0 563422.2 19.8X

OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - contains: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
collation unit benchmarks - contains: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 8793 8794 1 0.0 87929.3 1.0X
UTF8_LCASE 19382 19394 16 0.0 193824.8 0.5X
UNICODE 363790 363911 171 0.0 3637901.0 0.0X
UNICODE_CI 414597 415090 697 0.0 4145972.5 0.0X
UTF8_BINARY 7674 7674 1 0.0 76735.3 1.0X
UTF8_LCASE 20367 20376 14 0.0 203665.1 2.7X
UNICODE 377133 377909 1098 0.0 3771328.8 49.1X
UNICODE_CI 434710 435099 551 0.0 4347097.2 56.7X

OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - startsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
collation unit benchmarks - startsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 7692 7695 4 0.0 76921.8 1.0X
UTF8_LCASE 16451 16457 9 0.0 164507.8 0.5X
UNICODE 356828 358151 1871 0.0 3568280.7 0.0X
UNICODE_CI 417621 418820 1697 0.0 4176205.0 0.0X
UTF8_BINARY 6956 6959 4 0.0 69561.7 1.0X
UTF8_LCASE 14246 14262 23 0.0 142459.6 2.0X
UNICODE 369940 370072 186 0.0 3699400.9 53.2X
UNICODE_CI 442072 442365 414 0.0 4420718.1 63.6X

OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - endsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
collation unit benchmarks - endsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 7175 7175 0 0.0 71748.8 1.0X
UTF8_LCASE 15267 15291 34 0.0 152674.6 0.5X
UNICODE 371582 371601 26 0.0 3715822.1 0.0X
UNICODE_CI 429637 430336 989 0.0 4296371.0 0.0X
UTF8_BINARY 6927 6927 0 0.0 69265.2 1.0X
UTF8_LCASE 15505 15514 12 0.0 155054.5 2.2X
UNICODE 382361 382426 93 0.0 3823606.6 55.2X
UNICODE_CI 449956 450063 151 0.0 4499562.9 65.0X

60 changes: 30 additions & 30 deletions sql/core/benchmarks/CollationBenchmark-results.txt
View file
Original file line number Diff line number Diff line change
@@ -1,54 +1,54 @@
OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - equalsFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
collation unit benchmarks - equalsFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
--------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 1372 1373 1 0.1 13721.8 1.0X
UTF8_LCASE 3161 3163 2 0.0 31614.5 0.4X
UNICODE 20065 20074 13 0.0 200648.2 0.1X
UNICODE_CI 19950 19952 3 0.0 199497.1 0.1X
UTF8_BINARY 1372 1372 1 0.1 13718.5 1.0X
UTF8_LCASE 3115 3116 1 0.0 31154.4 2.3X
UNICODE 19813 19820 9 0.0 198132.2 14.4X
UNICODE_CI 19669 19686 24 0.0 196694.2 14.3X

OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - compareFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
collation unit benchmarks - compareFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
---------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 1730 1730 0 0.1 17300.7 1.0X
UTF8_LCASE 3181 3183 3 0.0 31807.5 0.5X
UNICODE 18827 18845 26 0.0 188267.7 0.1X
UNICODE_CI 18669 18671 3 0.0 186692.7 0.1X
UTF8_BINARY 1727 1728 1 0.1 17271.3 1.0X
UTF8_LCASE 3034 3035 1 0.0 30337.2 1.8X
UNICODE 19230 19243 18 0.0 192301.2 11.1X
UNICODE_CI 19080 19082 3 0.0 190802.0 11.0X

OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - hashFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
collation unit benchmarks - hashFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 3080 3081 1 0.0 30796.9 1.0X
UTF8_LCASE 9640 9670 42 0.0 96402.5 0.3X
UNICODE 65966 66004 53 0.0 659660.4 0.0X
UNICODE_CI 57631 57813 256 0.0 576314.7 0.1X
UTF8_BINARY 3080 3080 0 0.0 30796.4 1.0X
UTF8_LCASE 6436 6454 25 0.0 64360.0 2.1X
UNICODE 68095 68167 101 0.0 680951.3 22.1X
UNICODE_CI 62122 62123 2 0.0 621215.8 20.2X

OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - contains: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
collation unit benchmarks - contains: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 8792 8794 2 0.0 87923.1 1.0X
UTF8_LCASE 23937 23942 8 0.0 239367.4 0.4X
UNICODE 371949 372083 189 0.0 3719485.9 0.0X
UNICODE_CI 431636 432296 933 0.0 4316361.9 0.0X
UTF8_BINARY 8260 8261 1 0.0 82604.0 1.0X
UTF8_LCASE 23629 23629 0 0.0 236286.4 2.9X
UNICODE 364843 366078 1747 0.0 3648427.9 44.2X
UNICODE_CI 425728 426449 1020 0.0 4257275.1 51.5X

OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - startsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
collation unit benchmarks - startsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 6865 6877 16 0.0 68652.5 1.0X
UTF8_LCASE 22100 22118 26 0.0 220997.8 0.3X
UNICODE 365404 365627 316 0.0 3654037.8 0.0X
UNICODE_CI 431409 432257 1199 0.0 4314085.3 0.0X
UTF8_BINARY 6844 6848 5 0.0 68440.4 1.0X
UTF8_LCASE 21849 21870 30 0.0 218486.3 3.2X
UNICODE 363474 363811 476 0.0 3634738.4 53.1X
UNICODE_CI 427563 428029 659 0.0 4275629.8 62.5X

OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - endsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
collation unit benchmarks - endsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
UTF8_BINARY 6998 6999 2 0.0 69980.7 1.0X
UTF8_LCASE 22332 22358 36 0.0 223323.5 0.3X
UNICODE 382527 382805 393 0.0 3825268.8 0.0X
UNICODE_CI 447621 447949 465 0.0 4476209.7 0.0X
UTF8_BINARY 6904 6907 4 0.0 69039.3 1.0X
UTF8_LCASE 22007 22009 3 0.0 220067.8 3.2X
UNICODE 376402 377858 2060 0.0 3764015.4 54.5X
UNICODE_CI 444485 444809 458 0.0 4444850.8 64.4X

Loading

0 comments on commit 1e67659

Please sign in to comment.