Ability to load multiple sequential bag files

[amacneil]

For example, if someone ran rosbag record with --split, they would end up with a bunch of bag files in a folder. It would be nice if we could load them all (or a subset) at once.

From public discussion in ROS discord:

[jtbandes]

Also requested by @jackma.

[valschmidt]

We need this too. We are working through strategies for generating log files that balance file size with the need to illustrate long term trends (over hours, days or more), and the desire to split log topics across various files for better management. So we'd like to be able to load multiple files, regardless of whether they are contain over-lapping or sequential time ranges.

[itzhaolin]

yeah, this is many people need. Many times, The Rosbag files contain overleap or sequential

[luqnaim]

Bump. Noticed this is already possible with webviz where the second bag file topics would have a prefix of "webviz_source_2". Though, the limit for webviz is only 2 bag files at once. Would be cool if foxglove could extend this further to load N number of bag files!

[amacneil]

The specific request in this PR is to be able to load multiple sequential files (e.g. if you split the bag files based on size).

Webviz does not support this, it only allows you to compare two bag files covering the same time range, with a prefix for the second bag file (e.g. if you re-simulated some period of time and wanted to compare outputs). We have that feature request tracked in a separate issue (https://github.com/foxglove/studio/issues/2120).

So @luqnaim are you looking for loading multiple sequential files, or do you want to compare two different files of the same time range (with a topic prefix)?

Also note here that the workaround (for the original request in this PR) is to first use the rosbag CLI tool to merge/split/etc bag files as needed, before loading them into Foxglove Studio. You could also upload the bag files to Foxglove Data Platform, because when we play remote data we combine all topics over the requested time range, regardless of how many separate files you started with.

[luqnaim]

@amacneil ah, thank you for the link. I did not come across that issue during my search. Answering your question, issue https://github.com/foxglove/studio/issues/2120 is exactly what i am looking for. I would like to inspect the behavior of multiple algorithms on the same video sequence and be able to compare them frame by frame. I was able to do this with webviz but was limited to only 2 simultaneous bag files.

[valschmidt]

+1 for this one.

We have a tool, bag_manager, which allows one to select start/stop times and topics to include/exclude from a collection of bag files and writes a new bag file from the result to work around this. But it would be nice if Foxglove could handle this natively - point to a directory rather than a file, index the whole thing, then let you interactively play with everything there.

[itzhaolin]

It complicated to play in foxglove, so I hope foxglove to manager bags.

[jtbandes]

We might want to also think about what happens if the bags are not totally sequential but overlapping (such as if there were two separate bag recorders both using --split, and splitting at different times), and if there is some duplicate message data between bags (such as if two recorders both happened to record one or a handful of small 'core' or 'metadata' topics that users intentionally wanted to be present in all bag files — or even if it happened by accident).

[joegeisz]

I would also love to see this feature!

[adamconkey]

This would be helpful since I think the --split option is quite common for rosbag when you're logging a full system.

[itzhaolin]

I am looking forward to realize this feature very much! Watching...