Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Image segmenter output messages. #5

Merged
merged 2 commits into from
Sep 5, 2023
Merged

Image segmenter output messages. #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
efa005e
Image segmentation action and messages.
uavster Aug 31, 2023
08b2e9a
Removed Mask2DArray; GetMasks2D action uses Mask2D array directly.
uavster Sep 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion moveit_studio_msgs/moveit_studio_vision_msgs/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,23 +9,33 @@ endif()
find_package(ament_cmake REQUIRED)
find_package(builtin_interfaces REQUIRED)
find_package(geometry_msgs REQUIRED)
find_package(moveit_studio_sdk_msgs REQUIRED)
find_package(rosidl_default_generators REQUIRED)
find_package(sensor_msgs REQUIRED)
find_package(shape_msgs REQUIRED)
find_package(std_msgs REQUIRED)

set(msg_files
"msg/GraspableFace.msg"
"msg/GraspableObject.msg"
"msg/Mask2D.msg"
"msg/ObjectDetection.msg"
"msg/ObjectDetectionArray.msg"
"msg/PointCloudPCD.msg"
)

set(action_files
"action/GetMasks2D.action"
)

rosidl_generate_interfaces(${PROJECT_NAME}
${msg_files}
${msg_files}
${action_files}
DEPENDENCIES
builtin_interfaces
geometry_msgs
moveit_studio_sdk_msgs
sensor_msgs
shape_msgs
std_msgs
)
Expand Down
15 changes: 15 additions & 0 deletions moveit_studio_msgs/moveit_studio_vision_msgs/action/GetMasks2D.action
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Goal

# Image to segment.
sensor_msgs/Image image

# Segmentation parameters. They are stored as key-value pairs because they are
# specific to the segmentation model being run by the action server.
moveit_studio_sdk_msgs/BehaviorParameter[] parameters
uavster marked this conversation as resolved.
Show resolved Hide resolved

---
# Result
moveit_studio_vision_msgs/Mask2D[] masks

---
# Feedback
17 changes: 17 additions & 0 deletions moveit_studio_msgs/moveit_studio_vision_msgs/msg/Mask2D.msg
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Message containing a 2D mask.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this message also contain some string label (for classification) and a numeric confidence score for thresholding?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can decouple segmentation from classification here and have a Mask2DWithClassification message when we have models outputting both. I've seen this in other ROS messages. What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be OK if we want to do it that way, but then you'd need 2 separate behaviors with 2 separate data types in the action clients to do different tasks. So that's where I might prefer one message definition, and if we don't use classification data for certain tasks then so be it -- we ignore that part of the payload.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some doubts about the coupling-cohesion implications. Let's talk about this later. I'll merge for now, and we can add changes later.

#
# This message may be used to represent an instance or semantic mask in the
# output message of an image segmenter.

# The position of the top-left corner of the mask within the segmented image.
# (0, 0) is the top-left corner of the segmented image.
# The segmented image should be referenced in the containing image segmenter
# message.
int32 x
int32 y

# A single plane image with the mask pixels.
# The image pixels can be encoded as:
# - Integer: only non-zero pixels belong to the mask.
# - Floating point: each pixel has a probability of belonging to the mask.
sensor_msgs/Image pixels
2 changes: 2 additions & 0 deletions moveit_studio_msgs/moveit_studio_vision_msgs/package.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@

<depend>builtin_interfaces</depend>
<depend>geometry_msgs</depend>
<depend>moveit_studio_sdk_msgs</depend>
<depend>sensor_msgs</depend>
<depend>shape_msgs</depend>
<depend>std_msgs</depend>

Expand Down