Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to specify start & end times for reading #31

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
+48 −2
Open

Add ability to specify start & end times for reading #31

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
58eea82
Apply cleanly the setquery patch
daniel-feldman-ai Nov 16, 2021
b5aa79b
fix test
daniel-feldman-ai Nov 17, 2021
748c991
remove useless _
daniel-feldman-ai Nov 17, 2021
87f7e9f
Update view.h
daniel-feldman-ai Nov 17, 2021
4530ad9
remove old setquery
daniel-feldman-ai Nov 17, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions lib/ros_value.h
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
#pragma once

#include <chrono>
#include <memory>
#include <cstdint>
#include <cstring>
Expand Down Expand Up @@ -356,6 +357,16 @@ class RosValue {
friend class MessageParser;
};

template <typename Rep, typename Period>
bool operator<(const RosValue::ros_time_t& lhs, const std::chrono::duration<Rep, Period>& rhs) {
return std::chrono::nanoseconds{lhs.to_nsec()} < rhs;
}

template <typename Rep, typename Period>
bool operator>(const RosValue::ros_time_t& lhs, const std::chrono::duration<Rep, Period>& rhs) {
return std::chrono::nanoseconds{lhs.to_nsec()} > rhs;
}

template<>
const std::string RosValue::as<std::string>() const;

Expand Down
11 changes: 9 additions & 2 deletions lib/view.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,12 @@ View View::getMessages(const std::string &topic) {
}

View View::getMessages(const std::vector<std::string> &topics) {
std::chrono::nanoseconds start_time_ns {getStartTime().to_nsec()};
std::chrono::nanoseconds end_time_ns {getEndTime().to_nsec()};
return getMessages(topics, start_time_ns, end_time_ns);
}

View View::getMessages(const std::vector<std::string> &topics, std::chrono::nanoseconds start_time, std::chrono::nanoseconds end_time) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might make sense to operate on the data with ros_time_t and have that as the argument to the function, but then provide an easy to use constructor for ros_time_t from std::chrono::nanoseconds.

bag_wrappers_.clear();

for (const auto& bag : bags_) {
Expand All @@ -185,9 +191,10 @@ View View::getMessages(const std::vector<std::string> &topics) {

for (const auto &connection_record : bag->topic_connection_map_.at(topic)) {
for (const auto &block : connection_record->blocks) {
bag_wrappers_[bag]->chunks_to_parse.emplace(block.into_chunk);
if (block.into_chunk->info.end_time > start_time && block.into_chunk->info.start_time < end_time) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd recommend doing >= and <= to be inclusive.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think you might also want further filtering when yielding the actual messages within the chunk, and not just filtering at the chunk level.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finally, this filter will remove any chunks that straddle the edge of the filter - I don't think thats the behaviour we want. We should be reading any chunks that contain any data within the start and end time.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah we can filter when we yield, that's a good idea.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So save the start/end timestamps to the view, and when yielding do another bounds check?

Copy link
Contributor

@liambenson liambenson Nov 19, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ya exactly - I think that would be the right approach in this case! We definitely still want to filter out chunks for some performance gains, but then a secondary filter at message yield time will do the final checks for us.

bag_wrappers_[bag]->chunks_to_parse.emplace(block.into_chunk);
}
}

bag_wrappers_[bag]->connection_ids.emplace(connection_record->id);
}
}
Expand Down
3 changes: 3 additions & 0 deletions lib/view.h
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
#pragma once

#include <chrono>
#include <set>
#include <unordered_set>
#include <unordered_map>
Expand Down Expand Up @@ -121,6 +122,8 @@ class View {
View getMessages(const std::string &topic);
View getMessages(const std::vector<std::string> &topics);
View getMessages(std::initializer_list<std::string> topics);
View getMessages(const std::vector<std::string> &topics, std::chrono::nanoseconds start_time, std::chrono::nanoseconds end_time);

RosValue::ros_time_t getStartTime();
RosValue::ros_time_t getEndTime();

Expand Down
25 changes: 25 additions & 0 deletions test/embag_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,31 @@ TEST_F(ViewTest, MessagesForTopic) {
}
}

TEST_F(ViewTest, MessagesBetweenTimestamps) {
std::chrono::nanoseconds start_time_ns {view_.getStartTime().to_nsec()};
std::chrono::nanoseconds end_time_ns {view_.getEndTime().to_nsec()};
start_time_ns += std::chrono::seconds{1};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also add a end_time_ns?


std::vector<std::string> all_topics{
"/base_pose_ground_truth",
"/base_scan",
"/luminar_pointcloud",
};

int no_time_boundaries_count = 0;
for (const auto &message : view_.getMessages(all_topics)){
no_time_boundaries_count++;
}

int with_time_boundaries_count = 0;
for (const auto &message : view_.getMessages(all_topics, start_time_ns, end_time_ns)) {
with_time_boundaries_count++;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also likely assert that the timestamp of the message is greater than or equal to the start time, and less than the end_time

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liamembark we can't actually - this isn't true. Maybe you can suggest how we can document what is happening:

This compares chunks and not actual messages. So it is entirely possible (and happens in this case) that if i say give me messages between t=5 and t=11, we get messages from t=4 to t=12 because those are the chunks that encompass those messages.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nvm see below

}

ASSERT_GT(no_time_boundaries_count, with_time_boundaries_count);
}


class StreamTest : public ::testing::Test {
protected:
std::string bag_path_ = "test/test.bag";
Expand Down