Review instructions to reproduce the benchmarks

[carlossvg]

No description provided.

[smorita-esol]

I'm trying to reproduce the result below in a humble/Raspberry Pi 4 environment.
https://ros-realtime.github.io/ros2_realtime_benchmarks/benchmark_results/galactic/cyclictest/idle/cyclictest_report.html

But, the result is not as good as the Galatic environment as attached in the tail of this message.
May I confirm if I have to do additional steps except for the items below?

1. Install the Ubuntu 22.04 into Raspberry Pi 4 Model B Rev 1.4 with 4GBytes RAM using the image provided the repository below.
   https://github.com/ros-realtime/ros-realtime-rpi4-image
2. Turn off GUI by the commands below.

   sudo systemctl set-default multi-user.target 
   

Or I'd appreciate it if you could point out another document to setup the benchmark environment.

[My humble env.]

# Histogram
# Total: 018000000 017999945 017999888 017999830
# Min Latencies: 00013 00013 00013 00013
# Avg Latencies: 00016 00017 00016 00016
# Max Latencies: 00117 00090 00100 00094
# Histogram Overflows: 00000 00000 00000 00000
# Histogram Overflow at cycle number:
# Thread 0:
# Thread 1:
# Thread 2:
# Thread 3:

[Galatic Reference ]
https://ros-realtime.github.io/ros2_realtime_benchmarks/benchmark_results/galactic/cyclictest/idle/cyclictest_report.html

# Histogram
# Total: 018000000 017999947 017999896 017999842
# Min Latencies: 00006 00006 00006 00006
# Avg Latencies: 00008 00008 00008 00008
# Max Latencies: 00019 00016 00017 00033
# Histogram Overflows: 00000 00000 00000 00000
# Histogram Overflow at cycle number:
# Thread 0:
# Thread 1:
# Thread 2:
# Thread 3:

[carlossvg]

Hi @smorita-esol

In order to reproduce the results shown in cyclictest_report you can follow the instructions in https://ros-realtime.github.io/ros2_realtime_benchmarks/benchmark_tools/cyclictest.html

The RPI image used for the benchmark is the following one:

• https://github.com/ros-realtime/ros-realtime-rpi4-image/releases/tag/rpi4_ros2_galactic_v5.4.106-rt54-raspi-arm64-lttng

If I remember correctly there are no additional configuration steps, you should be able to get the same results with that image.

Additionally, you could check if you are able to get the same results that are shown in https://github.com/ros-realtime/ros-realtime-rpi4-image/releases plots. These plots can be generated following the instructions from https://github.com/ros-realtime/ros-realtime-rpi4-image/tree/master/latency-analysis.

Let know if this works for you or you need more information.

[smorita-esol]

@carlossvg
Thanks for sharing the information.

I could get almost the same results as the Galatic's reference page, as below(Avg. = 8usec), with the SD card image you pointed out.

# Histogram
# Total: 018000000 017999950 017999900 017999850
# Min Latencies: 00006 00006 00006 00006
# Avg Latencies: 00008 00008 00008 00008
# Max Latencies: 00029 00024 00040 00022
# Histogram Overflows: 00000 00000 00000 00000
# Histogram Overflow at cycle number:
# Thread 0:
# Thread 1:
# Thread 2:
# Thread 3:

And then, I reevaluated the humble's image below again with the same Raspberry Pi 4.

https://github.com/ros-realtime/ros-realtime-rpi4-image/releases/tag/22.04.1_v5.15.39-rt42-raspi_ros2_humble

But, I had almost the same results (Avg. = 16usec) put on the head of this thread.
The difference between the results seems just caused by the kernel versions(5.15.0 => 5.15.39).

Anyway, my initial intention was to confirm if there are environmental problems or not.
So, we will use the Humble's one above to evaluate our thread attribute configuration extension.