Concatenate module starts to publish point cloud from the past if the system slowed momentarily

[brkay54]

Checklist

• I've read the contribution guidelines.
• I've searched other issues and no duplicate issues were found.
• I'm convinced that this is not my fault but a bug.

Description

During the use of the Autoware's e2e_simulator to run sensing and perception pipelines, it was observed that the concatenate module behaves unexpectedly under certain conditions.

• The system was configured with separate_concatenate_node_and_time_sync_node. And, the PointCloud2 messages were publised with same header time at the same time to pipeline.

Pipeline latency [ms] statistics of the each lidar when the Autoware starts:

Initially, the concatenate module's output displayed a low delay (~24 ms, visible in the above). However, upon a momentary slow-down of the computer system, the concatenate module's reported pipeline latency increased significantly, jumping from an average delay of 24 ms to 116 ms. This behavior suggests that the module's timing becomes misaligned with its inputs after such a system hiccup.

System hiccup:

As seen above, after the system hiccup, concatenate modules start follow the pointcloud_preprocessor pipelines one step behind. And, this causes additional delay for perception pipeline.

Pipeline latency [ms] statistics of the each lidar after system hiccup:

Expected behavior

The concatenate module is expected to maintain consistent timing relative to its inputs, regardless of system performance fluctuations. Even if the system encounters momentary slow-downs, the concatenate module should recover and realign its timing with the incoming messages from the pointcloud_preprocess pipeline without introducing additional constant delay.

Actual behavior

In the event of a momentary system slow-down, the concatenate module's timing becomes misaligned with the timing of its inputs. This misalignment causes the module to execute its timer and publish messages with a significant delay. Subsequently, the concatenate module continues to publish point cloud messages from the past, resulting in a consistent and increased pipeline latency (as shown in the third image and summarized in the statistics from the first image). This behavior indicates that the timing within the concatenate module does not adequately handle or recover from transient system performance issues.

Steps to reproduce

• Run the perception & sensing pipeline in Autoware. You can use e2e_simulator.launch.xml.
• Plot the "/debug/pipeline_latency_ms" topics in a topic visualizer such as PlotJuggler.

Versions

No response

Possible causes

No response

Additional context

When I run the e2e_simulator.launch.xml with concatenate_and_time_sync_nodelet.cpp, I also observed similar behavior visible below: