Instrument rclpy

[christophebedard]

We've only instrumented rclcpp (and below), since the C++ client library is the obvious choice for applications for which performance is important. However, instrumenting rclpy would still be valuable; it has been requested many many times.

Since part of the ROS 2 instrumentation is in rcl and rmw, we only need to port some of the instrumentation (i.e., what is in rclcpp) to rclpy, so this should not be a lot of work, especially if the rclcpp and rclpy are similar enough.

Some things to figure out:

• Can tracepoints be trivially ported from rclcpp over to rclpy? In other words, does each rclcpp tracepoint have a corresponding location in rclpy? If not, what parts are the same, and what parts are different?
• Do we just change the tracepoint names from ros2:rclcpp_* to ros2:rclpy_*?
• Would the instrumentation only be added to C files, or would some instrumentation need to be added to Python files? Given that the LTTng Python tracing feature is a bit different than C/C++ tracing, this might be harder to integrate into tracetools: https://lttng.org/docs/v2.13/#doc-python-application

[Dekadee]

I've been looking into instrumenting rclpy and managed to get it working for my purposes:

• I am simply reusing the rclcpp tracepoints and adding them to code in rlpcy. The functionality of rclpy is similar enough to make everything work from what i can tell.
• For simplicities sake i have kept the same name, i.e. ros2:rclcpp_*, as i ran into problems overwriting the tracetools package in my workspace. This is more of a temporary workaround, but since the tracepoints are identical for rclpy and rclcpp there is no real need to change the names (it just looks a little wrong).
• The instrumentation works in the pybind11 sections of rlcpy, thus essentially only adding a few extra calls in the Python files, where the data needed for the tracepoints comes from Python. The actual tracepoints are all dispatched from C code.

Check out my fork of rclpy here, i have made my changes on the humble and galactic branches, since i am required to work with those versions.

[christophebedard]

Awesome! I don't really use rclpy, but I'll try to give this a try at some point.

What have you done with the trace data you collected using that rclpy instrumentation? Have you run any kind of analysis on it?

[Dekadee]

The subscription and timer callbacks are picked up by tracetools_analysis just like the rclcpp versions without any changes to that package.

I have also been running some more analysis with an private tool that can extract end-to-end latencies, which can now analyze a mix of rlcpy and rclcpp nodes without any issues so far. This tools uses the data model from tracetools_analysis package, to read the traces so essentially uses the same method.