ROS2 image display
In this example, we'll create an application that uses ROS2 to subscribe to two image topics and NiceGUI to display the images in a graphical user interface. We'll also include a "Switch image" button to toggle between the two images.
The source code of this example can be found here: ROS2 image display
Before getting started, make sure you have the following installed:
- ROS2: ROS2 Humble Installation Guide
- Python 3.8 or higher
- NiceGUI: Install NiceGUI using
pip install nicegui
The example consists of two nodes. The sender node is a modified version of the ROS2 publishers example. For this example, we will focus on the nicegui_image_receiver.
import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from cv_bridge import CvBridge
import cv2
from nicegui import app, Client, ui
import threading
from pathlib import Path
from rclpy.executors import ExternalShutdownException
import base64We import the necessary libraries, including ROS2, OpenCV for image processing, NiceGUI for the user interface, and other required modules.
class ImageReceiverNode(Node):
def __init__(self) -> None:
super().__init__('image_receiver_node')
# Create two subscriptions for the two images
self.subscription1 = self.create_subscription(Image, 'sender/im1', self.image_callback1, 10)
self.subscription2 = self.create_subscription(Image, 'sender/im2', self.image_callback2, 10)
# Adding CV bridge for image processing
self.cv_bridge = CvBridge()
# Control variable to switch between the two images
self.show_img1 = TrueWe create a ROS2 node called ImageReceiverNode that subscribes to two image topics, 'sender/im1' and 'sender/im2', and includes a control variable to switch between them.
# Add NiceGUI elements
with globals.index_client:
# Create a row with a width of 40%
with ui.row().style('width: 40%;'):
# Create an empty interactive_image element
self.sub_image = ui.interactive_image()
# Create a button to switch between the images, it calls the switch_image function
ui.button("Switch image", on_click=lambda: self.switch_image())We use NiceGUI to create a graphical user interface. The interface includes an empty image display (interactive_image) and a "Switch image" button that calls the switch_image function.
def switch_image(self) -> None:
# A simple function to trigger the control variable for the image
self.show_img1 = not self.show_img1To switch the image source, we use this switch_image function, that gets called when the button is clicked.
def image_callback1(self, msg) -> None:
# Check if the first image should be shown
if self.show_img1:
# Convert the image to a cv::Mat
image = self.cv_bridge.imgmsg_to_cv2(msg, desired_encoding='bgr8')
# Encode the image to base64
base64_image = self.encode_image_to_base64(image)
# Set the image source to the base64 string to display it
self.sub_image.set_source(f'data:image/png;base64,{base64_image}')
def image_callback2(self, msg) -> None:
#check if the second image should be shown
if not self.show_img1:
#convert the image to a cv::Mat
image = self.cv_bridge.imgmsg_to_cv2(msg, desired_encoding='bgr8')
#encode the image to base64
base64_image = self.encode_image_to_base64(image)
#set the image source to the base64 string to display it
self.sub_image.set_source(f'data:image/png;base64,{base64_image}')
returnThese are the callback functions for the images. After the callback check if the image display trigger is set correctly, they convert the received image to base64. It gets displayed by setting the source of the interactive_image to the new converted base64 string of the incoming image.
def encode_image_to_base64(self, image) -> str:
# Convert image to binary format
_, image_data = cv2.imencode('.png', image)
# Encode the binary image to base64
base64_image = base64.b64encode(image_data).decode('utf-8')
# Return the base64 string
return base64_imageThis is the function that converts the image from a cv::Mat to a base64 string.
def ros_main() -> None:
# Standard ROS2 node initialization
print('Starting ROS2...', flush=True)
rclpy.init()
image_receiver = ImageReceiverNode()
try:
rclpy.spin(image_receiver)
except ExternalShutdownException:
pass
finally:
image_receiver.destroy_node()The ROS2 node itself gets controlled by this function. This is basically the contents of your main() function in a normal ROS2 node.
app.on_startup(lambda: threading.Thread(target=ros_main).start())
# We add reload dirs to watch changes in our package
ui.run(title='Image Display with NiceGUI', uvicorn_reload_dirs=str(Path(__file__).parent.resolve()))The starting of the ROS2 node is handled by NiceGUI. The Node will be started with the app.on_startup() function. The node will be started in its own thread directly, since running NiceGUI and the ROS2 node in the same thread causes one to be blocked by the other.
The second reason we do this is, that the ROS2 node gets restarted every time NiceGUI reloads itself.
The usage is quite simple, just run these run commands in two terminals. Afterward, a browser window should have opened with the GUI running on localhost:8080.
Terminal1 :
python3 receiver.pyIn this version of this tutorial, we start the script directly with python. In ROS2, you can run nodes directly with python3 because the ROS2 initialization uses standard Python APIs, but you must source your ROS2 distribution beforehand to ensure the environment is correctly set.
Terminal2 :
ros2 run image_sender senderIf you want to start the code with ros2 run inside of a package, you have to add the following to the code (example code). This is a workaround to let ros2 start the node, but let NiceGUI run it.
You might want to deactivate reload since there is a bug with uvicorn, that will watch more then just your set folder for reloads. The same goes for reload exclude, which will be ignored.
def main():
pass # NOTE: This is originally used as the ROS entry point, but we give the control of the node to NiceGUI.
ui_run.APP_IMPORT_STRING = f'{__name__}:app'