Update benchmark results

JuliaRobotics · Nov 5, 2024 · e3f412b · e3f412b
1 parent 93a5eaf
commit e3f412b
Show file tree

Hide file tree

Showing 3 changed files with 146 additions and 114 deletions.
diff --git a/docs/src/benchmarks.md b/docs/src/benchmarks.md
@@ -1,6 +1,14 @@
 # Benchmarks
 
-To attain maximal performance, it is recommended to pass `-O3`, `--check-bounds=no` as command line flags to `julia`. As of Julia 1.1, maximizing performance for the `dynamics!` algorithm requires either setting the number of BLAS threads to 1 (`using LinearAlgebra; BLAS.set_num_threads(1)`) if using OpenBLAS (the default), or compiling Julia with MKL. See [this issue](https://github.com/JuliaRobotics/RigidBodyDynamics.jl/issues/500) for more information.
+To attain maximal performance, it is recommended to pass `-O3` and `--check-bounds=no` as command line flags to `julia`:
+
+```bash
+cd RigidBodyDynamics.jl
+julia -O3 --check-bounds=no perf/runbenchmarks.jl
+```
+
+> **Warning**
+> For Julia versions previous to `v1.8`, maximizing performance for the `dynamics!` algorithm requires either setting the number of BLAS threads to 1 (`using LinearAlgebra; BLAS.set_num_threads(1)`) if using OpenBLAS (the default), or compiling Julia with MKL. See [this issue](https://github.com/JuliaRobotics/RigidBodyDynamics.jl/issues/500) for more information.
 
 Run `perf/runbenchmarks.jl` to see benchmark results for the Atlas robot (v5). Results below are for the following scenarios:
 
@@ -9,69 +17,62 @@ Run `perf/runbenchmarks.jl` to see benchmark results for the Atlas robot (v5). R
 3. Do inverse dynamics.
 4. Do forward dynamics.
 
-Note that results on CI builds are **not at all** representative because of code coverage. Results on a reasonably fast laptop at commit [870bea6](https://github.com/JuliaRobotics/RigidBodyDynamics.jl/commit/870bea668d5b11ce0555fa0552592d2c3cb15c54):
+> **Note**
+> Results on CI builds are **not at all** representative because of code coverage.
 
-Output of `versioninfo()`:
+Below are the results for **RBD.jl 2.5.0** (commit [`93a5ea`](https://github.com/JuliaRobotics/RigidBodyDynamics.jl/commit/93a5eaf15a5f6714b1ec1ce621b053542dcb721d)) using **Julia 1.11.1** on an **Apple MacBook Air (M2, 2023)** (16GB RAM, 512GB SSD):
 
+Output of `versioninfo()`:
 ```
-Julia Version 1.5.3
-Commit 788b2c77c1 (2020-11-09 13:37 UTC)
+Julia Version 1.11.1
+Commit 8f5b7ca12ad (2024-10-16 10:53 UTC)
+Build Info:
+  Official https://julialang.org/ release
 Platform Info:
-  OS: macOS (x86_64-apple-darwin18.7.0)
-  CPU: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
+  OS: macOS (arm64-apple-darwin22.4.0)
+  CPU: 8 × Apple M2
   WORD_SIZE: 64
-  LIBM: libopenlibm
-  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
+  LLVM: libLLVM-16.0.6 (ORCJIT, apple-m2)
+Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)
 ```
 
-Note that this is a different machine than the one that was used for earlier benchmarks.
-
-Mass matrix:
+> **Note**
+> This is a different machine than the one that was used for earlier benchmarks.
 
+Mass matrix ([`mass_matrix!`](@ref)):
 ```
-  memory estimate:  0 bytes
-  allocs estimate:  0
-  --------------
-  minimum time:     4.415 μs (0.00% GC)
-  median time:      4.579 μs (0.00% GC)
-  mean time:        4.916 μs (0.00% GC)
-  maximum time:     19.794 μs (0.00% GC)
+BenchmarkTools.Trial: 10000 samples with 10 evaluations.
+ Range (min … max):  3.728 μs …  10.024 μs  ┊ GC (min … max): 0.00% … 0.00%
+ Time  (median):     3.874 μs               ┊ GC (median):    0.00%
+ Time  (mean ± σ):   3.903 μs ± 180.208 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%
+ Memory estimate: 0 bytes, allocs estimate: 0.
 ```
 
-Mass matrix and Jacobian from left hand to right foot:
-
+Mass matrix ([`mass_matrix!`](@ref)) and Jacobian ([`geometric_jacobian!`](@ref)) from left hand to right foot:
 ```
-  memory estimate:  0 bytes
-  allocs estimate:  0
-  --------------
-  minimum time:     4.860 μs (0.00% GC)
-  median time:      4.982 μs (0.00% GC)
-  mean time:        5.399 μs (0.00% GC)
-  maximum time:     24.712 μs (0.00% GC)
+BenchmarkTools.Trial: 10000 samples with 10 evaluations.
+ Range (min … max):  3.941 μs …   9.020 μs  ┊ GC (min … max): 0.00% … 0.00%
+ Time  (median):     4.103 μs               ┊ GC (median):    0.00%
+ Time  (mean ± σ):   4.135 μs ± 196.842 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%
+ Memory estimate: 0 bytes, allocs estimate: 0.
 ```
 
 Note the low additional cost of computing a Jacobian when the mass matrix is already computed. This is because RigidBodyDynamics.jl caches intermediate computation results.
 
-Inverse dynamics:
-
+Inverse dynamics ([`inverse_dynamics!`](@ref)):
 ```
-  memory estimate:  0 bytes
-  allocs estimate:  0
-  --------------
-  minimum time:     4.256 μs (0.00% GC)
-  median time:      4.541 μs (0.00% GC)
-  mean time:        4.831 μs (0.00% GC)
-  maximum time:     21.625 μs (0.00% GC)
+BenchmarkTools.Trial: 10000 samples with 10 evaluations.
+ Range (min … max):  2.736 μs …   5.666 μs  ┊ GC (min … max): 0.00% … 0.00%
+ Time  (median):     2.866 μs               ┊ GC (median):    0.00%
+ Time  (mean ± σ):   2.882 μs ± 119.781 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%
+ Memory estimate: 0 bytes, allocs estimate: 0.
 ```
 
-Forward dynamics:
-
+Forward dynamics ([`dynamics!`](@ref)):
 ```
-  memory estimate:  0 bytes
-  allocs estimate:  0
-  --------------
-  minimum time:     13.600 μs (0.00% GC)
-  median time:      14.419 μs (0.00% GC)
-  mean time:        16.071 μs (0.00% GC)
-  maximum time:     55.328 μs (0.00% GC)
+BenchmarkTools.Trial: 10000 samples with 10 evaluations.
+ Range (min … max):  9.791 μs …  13.899 μs  ┊ GC (min … max): 0.00% … 0.00%
+ Time  (median):     9.874 μs               ┊ GC (median):    0.00%
+ Time  (mean ± σ):   9.942 μs ± 292.126 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%
+ Memory estimate: 0 bytes, allocs estimate: 0.
 ```