Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Humble/Garden CI failures #116

Closed
Tracked by #81
mabelzhang opened this issue Jan 18, 2023 · 4 comments
Closed
Tracked by #81

Fix Humble/Garden CI failures #116

mabelzhang opened this issue Jan 18, 2023 · 4 comments

Comments

@mabelzhang
Copy link
Collaborator

In #111, @andermi found that when we switched to Humble/Garden, there were intermittent CI failures related to timing caused by something deep in the ROS 2 Humble and/or Gazebo Garden code base. Different tests were failing on CI depending on the run.

However, all the tests pass locally.

We decided to disable CI for the time being so that we can move forward with development in this repo, while those issues are hopefully being resolved upstream.
In the meantime, the team agreed to commit to making sure CI tests pass locally for both the PR opener and the reviewer, before merging a PR.

@mjcarroll Do you have a list of issues/PRs upstream that we can link to here to track the progress on these time-related fixes? Were they in ROS 2 or Gazebo? Alternatively, is it an option at all to backport the Added Mass to Fortress, as Galactic/Fortress was a much more stable combination? Sounds like there's some changes to DART that makes it not an option. Want to have that visible in this ticket so we can refer back to it.

@mjcarroll
Copy link
Collaborator

Alternatively, is it an option at all to backport the Added Mass to Fortress

I don't believe so, as it was a pretty sweeping change across sdf, physics, and sim. We landed the breaking parts before the garden release so that the rest can get in after reviews.

@mjcarroll
Copy link
Collaborator

Do you have a list of issues/PRs upstream that we can link to here to track the progress on these time-related fixes? Were they in ROS 2 or Gazebo?

I believe that it is all in ROS 2, I will track down the issues that I was thinking of.

@mabelzhang
Copy link
Collaborator Author

mabelzhang commented Jan 18, 2023

Thanks! If there's anything that's needed in addition to those upstream issues, I'll leave it to you and @andermi to put a summary here so we can track all of them in one place.

If those issues aren't a priority for ROS 2, let me know...... we might have to put project time on them so that the process here is unblocked. I feel a bit uneasy saying let's disable CI indefinitely 😅

@mabelzhang
Copy link
Collaborator Author

As found in #111 (comment) , the errors were caused by a change of the default DDS in Humble (Cyclone -> FastRTPS). Switching DDS by export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp solved almost all the test failures.

One remaining flaky test is tracked in #117.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants