Zenoh 1.0.0 Porting #276

YuanYuYuan · 2024-09-05T13:17:03Z

Notes

Remove some unused dependencies.
Use z_view_keyexpr_t rather z_owned_keyexpr_t if possible. This keeps the reference only and doesn't need to be dropped.
The attachment type has been removed. Use the plain z_bytes instead.
Attachment is nothing but z_bytes. Use an iterator to get value by key.
z_check is now internal. We check zenoh entities' health by the creation functions' return.

Signed-off-by: Luca Cominardi <[email protected]>

chore: checkout dev/1.0.0

…n the zenoh_c_vendor

ahcorde · 2024-11-14T10:29:01Z

rmw_zenoh_cpp/src/detail/shm_context.hpp

+#ifndef DETAIL__SHM_CONTEXT_HPP_
+#define DETAIL__SHM_CONTEXT_HPP_
+
+


#include <cstddef>

ahcorde · 2024-11-14T10:29:07Z

rmw_zenoh_cpp/src/detail/shm_context.hpp

+namespace rmw_zenoh_cpp
+{
+///=============================================================================
+#ifdef RMW_ZENOH_BUILD_WITH_SHARED_MEMORY


this file is not included in the CMakeLists.txt

ahcorde · 2024-11-14T10:42:49Z

rmw_zenoh_cpp/src/detail/shm_context.hpp

+#ifdef RMW_ZENOH_BUILD_WITH_SHARED_MEMORY
+struct ShmContext
+{
+  z_owned_shm_provider_t shm_provider;


missing header

ahcorde · 2024-11-14T10:43:21Z

rmw_zenoh_cpp/src/detail/shm_context.cpp

+#ifdef RMW_ZENOH_BUILD_WITH_SHARED_MEMORY
+ShmContext::~ShmContext()
+{
+  z_drop(z_move(shm_provider));


I'm getting a segfault here, double free

alireza-moayyedi · 2024-11-15T09:18:31Z

Not sure what the status of the abovementioned issue by @ahcorde is but I am getting the following with the latest commit:

evshary · 2024-11-15T09:26:51Z

Not sure what the status of the abovementioned issue by @ahcorde is but I am getting the following with the latest commit

@alireza-moayyedi Sorry for the inconvenience. There are some errors with the latest commit. Would you mind using the previous one? I've tested with this one.
ZettaScaleLabs@28d917e

We will fix that ASAP.

YuanYuYuan · 2024-11-15T09:44:59Z

Hi @alireza-moayyedi! It has been fixed 😃

alireza-moayyedi · 2024-11-15T09:46:12Z

Not sure what the status of the abovementioned issue by @ahcorde is but I am getting the following with the latest commit

@alireza-moayyedi Sorry for the inconvenience. There are some errors with the latest commit. Would you mind using the previous one? I've tested with this one. ZettaScaleLabs@28d917e

We will fix that ASAP.

hey @evshary ,

Yes indeed I switched to ZettaScaleLabs@28d917e for the time being. And of course, no apologies required. This is what open source is for and I am more than grateful for your contributions ;-)

clalancette

I've done a first review pass (for me), and left some comments. Once this is rebased onto the latest and my comments are addressed, I'll do another review pass.

.github/workflows/style.yaml

.gitignore

rmw_zenoh_cpp/src/detail/attachment_helpers.cpp

rmw_zenoh_cpp/src/detail/graph_cache.hpp

zenoh_c_vendor/CMakeLists.txt

rmw_zenoh_cpp/src/rmw_zenoh.cpp

rmw_zenoh_cpp/src/detail/shm_context.cpp

clalancette · 2024-11-15T16:33:07Z

rmw_zenoh_cpp/src/detail/shm_context.hpp

Let's just remove this file from the pull request for now.

Fixed in 9f78ca4

The file is still present...

clalancette

Besides what I left inline, I'm did a head-to-head comparison between rolling and the dev/1.0.0 branch. Some of this matches what is in #286 (comment) , though some of it is different.

package	test	`rolling` branch (`3d771c8`)	`dev/1.0.0` branch (`9f78ca4`)	note
rcl	test_guard_condition	✔️	❌	particularly under load
rcl_action	test_action_server	✔️	❌
rclcpp	test_client_common	✔️	❌
rclcpp	test_publisher	❌	❌
rclcpp_action	test_server	✔️	❌
rclcpp_lifecycle	test_lifecycle_service_client	❌	✔️	flaky; I was only able to make this fail once
test_rclcpp	test_client_scope_cpp	✔️	❌
test_rclcpp	test_node_name	✔️	❌
test_rclcpp	test_n_nodes	✔️	❌
test_rclcpp	gtest_spin	❌	✔️
test_communication	test_publisher_subscriber_rclcpp_rclpy	✔️	❌
test_communication	test_publisher_subscriber_rclcpp	❌	❌
test_communication	test_requester_replier	✔️	❌
test_quality_of_service	test_deadline	❌	❌
test_quality_of_service	test_liveliness	❌	❌
test_quality_of_service	test_best_available	❌	❌
rclpy	test_action_client	❌	❌
rclpy	test_client	✔️	❌

Total failures: rolling 8, dev/1.0.0 15

rmw_zenoh_cpp/src/detail/rmw_subscription_data.cpp

.github/workflows/style.yaml

rmw_zenoh_cpp/src/detail/graph_cache.hpp

zenoh_c_vendor/CMakeLists.txt

YuanYuYuan · 2024-11-20T14:48:36Z

The failure on test_client_common can be related to this issue ros2/rclcpp#2680.

This reverts commit 5366957.

This reverts commit 2e3ad43.

Yadunund

Re-reviewed 75% of the changes and provided some feedback. My primary concern is the introduction of registering a function with the atexit signal handler. As explained inline, there is still a problem of the static variable being invalidated and 'm confused why we need it with dev/1.0.0 when we do not need it rolling.

Secondly, I think the diff in the PR is way larger than it needs to be. Imo the changes only need reflect changes in the zenoh-c API but there are changes in rmw_zenoh.cpp wrt to checking if entities are valid, and various cleanups to headers.
My recommendation is to take all the changes that remove cleanup headers and target a PR to the current rolling branch. And save other changes unrelated to breaking zenoh-c API to a future PR.

Yadunund · 2024-09-19T18:57:41Z

rmw_zenoh_cpp/src/zenohd/main.cpp

@@ -68,6 +68,9 @@ int main(int argc, char ** argv)
    return 1;
  }

+  // Enable the zenoh built-in logger


Nit

Suggested change

// Enable the zenoh built-in logger

// Enable the zenoh built-in logger.

Yadunund · 2024-11-23T18:24:49Z

rmw_zenoh_cpp/src/detail/rmw_client_data.cpp

@@ -361,28 +351,21 @@ rmw_ret_t ClientData::take_response(
  }

  // Fill in the request_header
-  request_header->request_id.sequence_number =
-    rmw_zenoh_cpp::get_int64_from_attachment(&sample->attachment, "sequence_number");
+  rmw_zenoh_cpp::attachment_data_t attachment(z_sample_attachment(sample));


This API feels awkward and requires you to define the move constructor. Why not just pass const z_loaned_sample_t * and construct the z_sample_attachment internally?

Yadunund · 2024-11-23T18:28:47Z

rmw_zenoh_cpp/src/detail/rmw_client_data.cpp

-  if (zc_liveliness_token_check(&token_)) {
-    zc_liveliness_undeclare_token(z_move(token_));
-  }
-  if (z_check(z_loan(keyexpr_))) {
-    z_drop(z_move(keyexpr_));
-  }
+  zc_liveliness_undeclare_token(z_move(token_));
+
+  z_drop(z_move(keyexpr_));


We previously needed this check since it was possible for token_ and keyexpr_ to be uninitialized when shutdown is called within ClientData::make. Do these functions now check whether the tokens are valid before dropping?

Yadunund · 2024-11-23T18:36:04Z

rmw_zenoh_cpp/src/detail/rmw_context_impl_s.cpp

+// The variable is used to identify whether the process is trying to exit or not.
+// The atexit function we registered will set the flag and prevent us from closing
+// Zenoh Session. Zenoh API can't be used in atexit function, because Tokio context
+// is already destroyed. It will cause panic if we do so.
+static bool is_exiting = false;
+void update_is_exiting()
+{
+  is_exiting = true;
+}
+


Why do we need this logic with 1.0.0 only? How do we ensure two or more threads do not access is_exiting concurrently? Moreover, when the program terminates, static ordering is unknown so there is no guarantee that is_exiting is valid. That's one of the reasons we redid the way we track ClientData in #310
cc @clalancette

Yadunund · 2024-11-23T18:36:41Z

rmw_zenoh_cpp/src/detail/rmw_context_impl_s.cpp

 // This global mapping of raw Data pointers to Data shared pointers allows graph_sub_data_handler()
 // to lookup the pointer, and gain a reference to a shared_ptr if it exists.
 // This guarantees that the Data object will not be destroyed while we are using it.
 static std::mutex data_to_data_shared_ptr_map_mutex;
 static std::unordered_map<rmw_context_impl_s::Data *,
  std::shared_ptr<rmw_context_impl_s::Data>> data_to_data_shared_ptr_map;

-static void graph_sub_data_handler(const z_sample_t * sample, void * data);
+static void graph_sub_data_handler(z_loaned_sample_t * sample, void * data);


Suggested change

static void graph_sub_data_handler(z_loaned_sample_t * sample, void * data);

static void graph_sub_data_handler(const z_loaned_sample_t * sample, void * data);

Yadunund · 2024-11-23T19:26:02Z

rmw_zenoh_cpp/src/rmw_zenoh.cpp

+// TODO(yuyuan): SHM, make this configurable
+#define SHM_BUF_OK_SIZE 2621440
+


This is unused so please remove.

Suggested change

// TODO(yuyuan): SHM, make this configurable

#define SHM_BUF_OK_SIZE 2621440

Yadunund · 2024-11-23T19:26:23Z

rmw_zenoh_cpp/src/rmw_zenoh.cpp

-  if (!context_impl->session_is_valid()) {
-    RMW_SET_ERROR_MSG("zenoh session is invalid");
-    return nullptr;
-  }


Why remove this?

Yadunund · 2024-11-23T19:26:44Z

rmw_zenoh_cpp/src/rmw_zenoh.cpp

+/// Finalize a given publisher handle, reclaim the resources, and deallocate the
+/// publisher handle.


Undo unnecessary change.

Yadunund · 2024-11-23T19:27:14Z

rmw_zenoh_cpp/src/rmw_zenoh.cpp

-  if (!pub_data->liveliness_is_valid()) {
-    return RMW_RET_ERROR;
-  }
-


Why remove this?

Yadunund · 2024-11-23T19:27:40Z

rmw_zenoh_cpp/src/rmw_zenoh.cpp

-  if (!context_impl->session_is_valid()) {
-    RMW_SET_ERROR_MSG("zenoh session is invalid");
-    return nullptr;
-  }


Again, why remove this here and in other functions below?

Mallets and others added 30 commits August 9, 2024 14:32

Update style.yaml

e2fabb6

Signed-off-by: Luca Cominardi <[email protected]>

Update check_logging.yaml

6cd6dbe

Signed-off-by: Luca Cominardi <[email protected]>

Update build.yaml

dcefb68

Signed-off-by: Luca Cominardi <[email protected]>

chore: configure the compiliation

14fe58c

chore: checkout dev/1.0.0

chore: complete the 1st version

301a8ce

fix: memory leak

8ef92b5

wip

06b6fa4

fix: z_error_t -> z_result_t

71e7c5b

Fix scouting/*/autoconnect/* per eclipse-zenoh/zenoh@b31a410 (#3)

a717270

chore: checkout the local zenoh-c

cfa63d9

chore: polish z_open

12dc820

fix: segfault

d331b74

feat: z_bytes_serialize_from_slice without copy

1db4bef

chore: switch back to https://github.com/eclipse-zenoh/zenoh-c

d12f189

Initialize query_ member of ZenohQuery

909329b

refactor: use z_owned_slice_t instead

511c6f6

chore: adapt the latest change of zenoh-c dev/1.0.0

044d25b

chore: use strncmp to avoid copying

d79cf5e

refactor: use z_view_keyexpr_t to avoid copying

6637400

feat: implment the SHM feature

83334c8

chore: adpat the new changes from zenoh-c and fix the bug in liveliness

bf0da8a

fix: segmentation fault due to the unallocated query memory

dfc95ad

fix: workaround the ZID parsing issue

cd071e5

fix Zenoh Config read\check

4f842fd

adopt to recent zenoh-c API changes

af2e344

chore: sync up with the rolling branch

0225a59

chore: update to self-hosted runners

36140fd

fix: adapt the latest change of batching config

080eb15

Merge remote-tracking branch 'origin/rolling' into dev/1.0.0

f60c2a2

build: deprecate the zenohc_debug and incldue the zenohc dependency i…

83a26f2

…n the zenoh_c_vendor

ahcorde requested changes Nov 14, 2024

View reviewed changes

ahcorde reviewed Nov 14, 2024

View reviewed changes

YuanYuYuan added 3 commits November 14, 2024 22:37

fix: use TRUE value to configure the feature flag

14683cd

fix: fix the SHM enabled check

d44002e

fix: inlcude shm_context.cpp in the build

8557523

revert: SHM support

cb87b1e

clalancette requested changes Nov 15, 2024

View reviewed changes

YuanYuYuan added 2 commits November 16, 2024 01:41

Merge branch 'rolling' into wip/merge-rolling

46ec449

fix: adapt the merge and bump up zenoh-c version

2743344

mergify bot mentioned this pull request Nov 18, 2024

Port zenoh_c to zenoh_cpp #311

Draft

YuanYuYuan added 4 commits November 20, 2024 01:12

chore: remove .gitignore

09798c3

fix: correct typo attachement to attachment

b6df66c

Merge branch 'rolling' into wip/merge-rolling

59203df

style: ament_cpplint and ament_uncrustify

9f78ca4

clalancette requested changes Nov 19, 2024

View reviewed changes

rmw_zenoh_cpp/src/detail/rmw_subscription_data.cpp Outdated Show resolved Hide resolved

.github/workflows/style.yaml Show resolved Hide resolved

rmw_zenoh_cpp/src/detail/graph_cache.hpp Show resolved Hide resolved

zenoh_c_vendor/CMakeLists.txt Show resolved Hide resolved

YuanYuYuan added 2 commits November 20, 2024 17:47

refactor: remove the warning of subscriber reliability QoS

7450bb2

fix: add back atexit handler

5ade015

YuanYuYuan added 2 commits November 21, 2024 16:39

merge: integrate rolling into dev/1.0.0

1e705c1

Revert "Don't shutdown contained entities. (ros2#313)"

2e3ad43

This reverts commit 5366957.

Yadunund mentioned this pull request Nov 21, 2024

Add tracing instrumentation using tracetools #294

Open

Reapply "Don't shutdown contained entities. (ros2#313)"

c41ee5a

This reverts commit 2e3ad43.

YuanYuYuan mentioned this pull request Nov 22, 2024

[Feature] Introduce include-what-you-use into the build pipeline #323

Open

YuanYuYuan added 2 commits November 22, 2024 17:16

chore: include the string_array.h

aa19545

fix: include "rcutils/types/string_array.h"

b485a93

Yadunund requested changes Nov 23, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Zenoh 1.0.0 Porting #276

Zenoh 1.0.0 Porting #276

YuanYuYuan commented Sep 5, 2024 •

edited

Loading

ahcorde Nov 14, 2024

ahcorde Nov 14, 2024

ahcorde Nov 14, 2024

ahcorde Nov 14, 2024

alireza-moayyedi commented Nov 15, 2024

evshary commented Nov 15, 2024

YuanYuYuan commented Nov 15, 2024

alireza-moayyedi commented Nov 15, 2024

clalancette left a comment

clalancette Nov 15, 2024

YuanYuYuan Nov 19, 2024

Yadunund Nov 23, 2024

clalancette left a comment

YuanYuYuan commented Nov 20, 2024

Yadunund left a comment •

edited

Loading

Yadunund Sep 19, 2024

Yadunund Nov 23, 2024

Yadunund Nov 23, 2024

Yadunund Nov 23, 2024

Yadunund Nov 23, 2024

Yadunund Nov 23, 2024

Yadunund Nov 23, 2024

Yadunund Nov 23, 2024

Yadunund Nov 23, 2024

Yadunund Nov 23, 2024

		#ifndef DETAIL__SHM_CONTEXT_HPP_
		#define DETAIL__SHM_CONTEXT_HPP_

	// Enable the zenoh built-in logger
	// Enable the zenoh built-in logger.

	static void graph_sub_data_handler(z_loaned_sample_t * sample, void * data);
	static void graph_sub_data_handler(const z_loaned_sample_t * sample, void * data);

		// TODO(yuyuan): SHM, make this configurable
		#define SHM_BUF_OK_SIZE 2621440

		/// Finalize a given publisher handle, reclaim the resources, and deallocate the
		/// publisher handle.

Zenoh 1.0.0 Porting #276

Are you sure you want to change the base?

Zenoh 1.0.0 Porting #276

Conversation

YuanYuYuan commented Sep 5, 2024 • edited Loading

Notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alireza-moayyedi commented Nov 15, 2024

evshary commented Nov 15, 2024

YuanYuYuan commented Nov 15, 2024

alireza-moayyedi commented Nov 15, 2024

clalancette left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clalancette left a comment

Choose a reason for hiding this comment

YuanYuYuan commented Nov 20, 2024

Yadunund left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YuanYuYuan commented Sep 5, 2024 •

edited

Loading

Yadunund left a comment •

edited

Loading