Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reading can return out-of-order timestamps across file boundary in split bags #857

Closed
emersonknapp opened this issue Aug 27, 2021 · 1 comment
Labels
enhancement New feature or request more-information-needed Further information is required

Comments

@emersonknapp
Copy link
Collaborator

emersonknapp commented Aug 27, 2021

Description

Individual .db3 files within a split bag use the index to return messages in timestamp order when reading (playing). However, a file is played in full before a next file is loaded. That next file could then return messages with timestamps before the end time of the previous bag, if messages were written out-of-timestamp order across a file boundary.

Old title: Open all sqlite3 files simultaneously (if they are local) to eliminate out-of-order timestamps problem. Changed to reflect the problem, rather than solution space.

Related to #842

@emersonknapp emersonknapp added enhancement New feature or request more-information-needed Further information is required labels Aug 27, 2021
@emersonknapp emersonknapp changed the title [Consider] Open all sqlite3 files simultaneously (if they are local) to eliminate out-of-order timestamps problem Reading can return out-of-order timestamps across file boundary in split bags Aug 19, 2022
@MichaelOrlov
Copy link
Contributor

This issue was resolved by using priority queue sorted by timestamp on upper rosbag2_transport::player level.
We have a separate thread which is reading out next message from rosbag2_cpp::reader and pushing it to the priority queue.
Therefore, the messages will be naturally sorted up to the priority queue size, which is by default 1000 messages.
Reader:read_next() taking care about switching to the next bag split if no more elements in the current bag file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request more-information-needed Further information is required
Projects
None yet
Development

No branches or pull requests

2 participants