Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to obtain publisher GID and sequence number errors #130

Closed
AlexDayCRL opened this issue Mar 20, 2024 · 13 comments · Fixed by #131
Closed

Unable to obtain publisher GID and sequence number errors #130

AlexDayCRL opened this issue Mar 20, 2024 · 13 comments · Fixed by #131

Comments

@AlexDayCRL
Copy link
Contributor

Hi,

Thanks for all your work on the Zenoh RMW! We are very excited to try it out.

I'm getting the following errors when using data from a rosbag through ros2 bag play. These errors show up in any node that subscribes to topics published over Zenoh. It seems like these errors go away when subscribing to "live" topics rather than those played back through ros2 bag. Do you have any idea what the root cause could be?

[ERROR] [1710953784.261021973] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[ERROR] [1710953784.261040865] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
@clalancette
Copy link
Collaborator

clalancette commented Mar 20, 2024

Thanks for trying it out!

I think this is a bug (or at least, a part I missed) in #123 . In particular, I think that rmw_publish_serialized_message needs to "attach" the publisher GID and sequence number, similar to what rmw_publish does.

Would you be interested in trying that out, and then submitting a pull request to add it? I'm more than happy to review.

@AlexDayCRL
Copy link
Contributor Author

@clalancette sure! I just made a PR (#131).

@ciandonovan
Copy link

This is from a subscriber using the latest rolling branch of rmw_zenoh on a Jazzy distro subscribing live to a camera topic. It works, but deluges the stdout with these messages. Using the shared-memory transport.

[mockup_qml_gui-1] [ERROR] [1719497157.000221366] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.000314464] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.000341074] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.032723472] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.032816011] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.032842760] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.049600756] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.049701398] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.049727169] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.082071154] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.082165510] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.082191840] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.099065772] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.099161315] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.099209017] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.131853878] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.131955986] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.131984621] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.148039035] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.148139258] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.148165378] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.180826513] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.180918982] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.180944405] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.197441264] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.197542394] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.197568585] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.230215471] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.230317649] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.230342862] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.246597162] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.246691099] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.246716311] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.279746836] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.279852297] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.279877509] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.296512026] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.296601632] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.296626356] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.329035363] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.329169458] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.329204728] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.345503378] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.345617918] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.345642781] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.378127077] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.378220874] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.378245598] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.394622330] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.394719549] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.394745810] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.428052207] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.428155293] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.428185814] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.444053473] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.444178000] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.444204540] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.476774671] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.476878734] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.476904995] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.493542863] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.493646927] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.493674095] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.526322937] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.526419598] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.526445369] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.542653435] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.542753727] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.542778660] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.575571375] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.575670550] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.575696252] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.592626546] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.592831460] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.592858419] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.625296969] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.625402150] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.625426944] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.641573130] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.641672723] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.641698215] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.674471445] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.674558188] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.674583470] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.690680139] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.690773168] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.690798869] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.723562950] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.723663102] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.723688874] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.740110444] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.740201517] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.740227218] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.772880042] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.772979565] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.773030061] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.789577345] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.789679034] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.789705714] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.822242950] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.822332905] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.822357699] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.838724094] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.838822431] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.838847574] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.871853305] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.871955902] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.871982162] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.888366228] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.888464494] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.888489777] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.921351216] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.921471343] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.921496626] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.937829498] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.937919453] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.937944805] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.970220695] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.970319451] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.970345921] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.986660494] [rmw_zenoh_cpp]: Unable to obtain publisher GID from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.986756386] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.
[mockup_qml_gui-1] [ERROR] [1719497157.986782158] [rmw_zenoh_cpp]: Unable to obtain sequence number from the attachment.

@clalancette
Copy link
Collaborator

This is from a subscriber using the latest rolling branch of rmw_zenoh on a Jazzy distro subscribing live to a camera topic. It works, but deluges the stdout with these messages. Using the shared-memory transport.

This looks like it is using an older version of rmw_zenoh_cpp along with a newer version. Are you sure you are running the same version of rmw_zenoh_cpp everywhere?

@ciandonovan
Copy link

Both built from source on commit e73df29

@ciandonovan
Copy link

The publisher also spits out the same messages, maybe 50 of them, but not continuously like the subscriber

@ciandonovan
Copy link

Also should mention they're running in separate rootless Podman containers, with /tmp and /dev/shm bind-mounted into each and both using the host's network namespace so the shared-memory transport can work

@clalancette
Copy link
Collaborator

Also should mention they're running in separate rootless Podman containers, with /tmp and /dev/shm bind-mounted into each and both using the host's network namespace so the shared-memory transport can work

It might be the case that shared-memory transport doesn't support attachments. If that is the case, then this can't possibly work; we rely on the attachments to send metadata along with the messages. That said, I really don't know what's happening; I don't see any of this locally.

@ciandonovan
Copy link

Functionally it does work though, I'm seeing the camera stream in the application, and there's no significant loopback traffic, so it must be using the shared-memory transport. The RUST_LOG=trace also indicates this with mentions of zenoh_shm: SharedMemoryBuf::as_slice() as each message is sent

@clalancette
Copy link
Collaborator

Functionally it does work though, I'm seeing the camera stream in the application, and there's no significant loopback traffic, so it must be using the shared-memory transport. The RUST_LOG=trace also indicates this with mentions of zenoh_shm: SharedMemoryBuf::as_slice() as each message is sent

Yeah, it will deliver data, but none of the metadata will be there. So things like timestamps and the such will all be missing and empty. Many applications in ROS don't actually depend on any of that, but some do (like rosbag).

@ciandonovan
Copy link

Is this on the roadmap to be addressed? You mentioned it might not be possible, why is that? Would it require reworking how shared-memory is done in Zenoh itself upstream or is it more fundamental?

@clalancette
Copy link
Collaborator

Is this on the roadmap to be addressed? You mentioned it might not be possible, why is that? Would it require reworking how shared-memory is done in Zenoh itself upstream or is it more fundamental?

No, not on the roadmap at the moment; this is the first I've heard about it. We'll have to do some deeper digging to figure out what is happening. I'm going to suggest you open a new issue, with a reproducer that we can use to do that digging.

@ciandonovan
Copy link

On further investigation it's not an issue exclusive to the shared-memory transport, as the issue also occurs with it disabled and nodes communicating on the loopback interface.

Reproduced with image pub/sub, not sure if it's an issue with that implementation or the fact it's higher-bandwidth.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants