Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add ability to specify start & end times for reading #31

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Add ability to specify start & end times for reading #31

wants to merge 5 commits into from

Conversation

daniel-feldman-ai
Copy link
Contributor

@daniel-feldman-ai daniel-feldman-ai commented Nov 16, 2021 •
edited
Loading

Description of Changes

Mimics some functionality of ROS Bag views - specifically started from an offset and reading to a specific part of bag on specified topics

Test Plan

Added another test - ran succesfully

==================== Test output for //test:embag_test:
Running main() from gmock_main.cc
[==========] Running 11 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 2 tests from EmbagTest
[ RUN      ] EmbagTest.OpenCloseBag
[       OK ] EmbagTest.OpenCloseBag (0 ms)
[ RUN      ] EmbagTest.OpenCloseOddPaddingBag
[       OK ] EmbagTest.OpenCloseOddPaddingBag (0 ms)
[----------] 2 tests from EmbagTest (0 ms total)

[----------] 4 tests from BagTest
[ RUN      ] BagTest.TopicsInBag
[       OK ] BagTest.TopicsInBag (0 ms)
[ RUN      ] BagTest.TopicInBag
[       OK ] BagTest.TopicInBag (0 ms)
[ RUN      ] BagTest.MsgDefForTopic
[       OK ] BagTest.MsgDefForTopic (2 ms)
[ RUN      ] BagTest.ConnectionsForTopic
[       OK ] BagTest.ConnectionsForTopic (0 ms)
[----------] 4 tests from BagTest (2 ms total)

[----------] 4 tests from ViewTest
[ RUN      ] ViewTest.View
[       OK ] ViewTest.View (0 ms)
[ RUN      ] ViewTest.AllMessages
[       OK ] ViewTest.AllMessages (15 ms)
[ RUN      ] ViewTest.MessagesForTopic
[       OK ] ViewTest.MessagesForTopic (21 ms)
[ RUN      ] ViewTest.MessagesBetweenTimestamps
[       OK ] ViewTest.MessagesBetweenTimestamps (15 ms)
[----------] 4 tests from ViewTest (51 ms total)

[----------] 1 test from StreamTest
[ RUN      ] StreamTest.BagFromStream
[       OK ] StreamTest.BagFromStream (100 ms)
[----------] 1 test from StreamTest (100 ms total)

[----------] Global test environment tear-down
[==========] 11 tests from 4 test suites ran. (153 ms total)
[  PASSED  ] 11 tests.
================================================================================
//test:embag_test                                               (cached) PASSED in 0.2s

Executed 0 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions

@daniel-feldman-ai daniel-feldman-ai changed the title Add set query function to View Add ability to specify start & end times for reading Nov 17, 2021
liambenson
liambenson reviewed Nov 19, 2021
View reviewed changes
lib/view.cc
return getMessages(topics, start_time_ns, end_time_ns);
}

View View::getMessages(const std::vector<std::string> &topics, std::chrono::nanoseconds start_time, std::chrono::nanoseconds end_time) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might make sense to operate on the data with ros_time_t and have that as the argument to the function, but then provide an easy to use constructor for ros_time_t from std::chrono::nanoseconds.

liambenson
liambenson reviewed Nov 19, 2021
View reviewed changes
test/embag_test.cc

int with_time_boundaries_count = 0;
for (const auto &message : view_.getMessages(all_topics, start_time_ns, end_time_ns)) {
with_time_boundaries_count++;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also likely assert that the timestamp of the message is greater than or equal to the start time, and less than the end_time

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liamembark we can't actually - this isn't true. Maybe you can suggest how we can document what is happening:

This compares chunks and not actual messages. So it is entirely possible (and happens in this case) that if i say give me messages between t=5 and t=11, we get messages from t=4 to t=12 because those are the chunks that encompass those messages.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nvm see below

liambenson
liambenson reviewed Nov 19, 2021
View reviewed changes
test/embag_test.cc
TEST_F(ViewTest, MessagesBetweenTimestamps) {
std::chrono::nanoseconds start_time_ns {view_.getStartTime().to_nsec()};
std::chrono::nanoseconds end_time_ns {view_.getEndTime().to_nsec()};
start_time_ns += std::chrono::seconds{1};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also add a end_time_ns?

liambenson
liambenson reviewed Nov 19, 2021
View reviewed changes
lib/view.cc
@@ -185,9 +191,10 @@ View View::getMessages(const std::vector<std::string> &topics) {

for (const auto &connection_record : bag->topic_connection_map_.at(topic)) {
for (const auto &block : connection_record->blocks) {
bag_wrappers_[bag]->chunks_to_parse.emplace(block.into_chunk);
if (block.into_chunk->info.end_time > start_time && block.into_chunk->info.start_time < end_time) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd recommend doing >= and <= to be inclusive.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think you might also want further filtering when yielding the actual messages within the chunk, and not just filtering at the chunk level.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finally, this filter will remove any chunks that straddle the edge of the filter - I don't think thats the behaviour we want. We should be reading any chunks that contain any data within the start and end time.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah we can filter when we yield, that's a good idea.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So save the start/end timestamps to the view, and when yielding do another bounds check?

Copy link
Contributor

@liambenson liambenson Nov 19, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ya exactly - I think that would be the right approach in this case! We definitely still want to filter out chunks for some performance gains, but then a secondary filter at message yield time will do the final checks for us.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liambenson liambenson liambenson left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@daniel-feldman-ai @liambenson