Add script to get all Debian packages corresponding to a gazebodistro file #4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chapulina
reviewed
May 14, 2020
| echo | ||
|
|
||
| echo "LIST OF PACKAGES:" | ||
| echo "${osrf_pkgs}" |
There was a problem hiding this comment.
When I run bash -e gazebodistro_to_pkgs_for_ros.bash ../collection-citadel.yaml nothing is printed here. Is this expected? I guess I don't fully understand what this script is meant to do.
This is my full output
This tool reads a gazebodistro file and look in the system for
all available pkgs corresponding to that distro file
1. Getting packages using: ignition-cmake2
1. Getting packages using: ignition-common3
1. Getting packages using: ignition-fuel-tools4
1. Getting packages using: ignition-gazebo3
1. Getting packages using: ignition-gui3
1. Getting packages using: ignition-launch2
1. Getting packages using: ignition-math6
1. Getting packages using: ignition-msgs5
1. Getting packages using: ignition-physics2
1. Getting packages using: ignition-plugin1
1. Getting packages using: ignition-rendering3
1. Getting packages using: ignition-sensors3
1. Getting packages using: ignition-tools1
1. Getting packages using: ignition-transport8
1. Getting packages using: sdformat9
2. Get dependencies of: libignition-cmake2-dev
2. Get dependencies of: libignition-common3
2. Get dependencies of: libignition-common3-av
2. Get dependencies of: libignition-common3-av-dev
2. Get dependencies of: libignition-common3-core-dev
2. Get dependencies of: libignition-common3-dev
2. Get dependencies of: libignition-common3-events
2. Get dependencies of: libignition-common3-events-dev
2. Get dependencies of: libignition-common3-graphics
2. Get dependencies of: libignition-common3-graphics-dev
2. Get dependencies of: libignition-common3-profiler
2. Get dependencies of: libignition-common3-profiler-dev
2. Get dependencies of: libignition-fuel-tools4
2. Get dependencies of: libignition-fuel-tools4-dev
2. Get dependencies of: libignition-gazebo3
2. Get dependencies of: libignition-gazebo3-dbg
2. Get dependencies of: libignition-gazebo3-dev
2. Get dependencies of: libignition-gazebo3-plugins
2. Get dependencies of: libignition-gui3
2. Get dependencies of: libignition-gui3-dev
2. Get dependencies of: libignition-launch2
2. Get dependencies of: libignition-launch2-dev
2. Get dependencies of: libignition-math6
2. Get dependencies of: libignition-math6-dbg
2. Get dependencies of: libignition-math6-dev
2. Get dependencies of: libignition-math6-eigen3-dev
2. Get dependencies of: libignition-msgs5
2. Get dependencies of: libignition-msgs5-dev
2. Get dependencies of: libignition-physics2
2. Get dependencies of: libignition-physics2-core-dev
2. Get dependencies of: libignition-physics2-dartsim
2. Get dependencies of: libignition-physics2-dartsim-dev
2. Get dependencies of: libignition-physics2-dev
2. Get dependencies of: libignition-physics2-mesh-dev
2. Get dependencies of: libignition-physics2-sdf-dev
2. Get dependencies of: libignition-plugin
2. Get dependencies of: libignition-plugin-dbg
2. Get dependencies of: libignition-plugin-dev
2. Get dependencies of: libignition-plugin0
2. Get dependencies of: libignition-plugin0-dbg
2. Get dependencies of: libignition-plugin0-dev
2. Get dependencies of: libignition-rendering3
2. Get dependencies of: libignition-rendering3-core-dev
2. Get dependencies of: libignition-rendering3-dev
2. Get dependencies of: libignition-rendering3-ogre1
2. Get dependencies of: libignition-rendering3-ogre1-dev
2. Get dependencies of: libignition-rendering3-ogre2
2. Get dependencies of: libignition-rendering3-ogre2-dev
2. Get dependencies of: libignition-sensors3
2. Get dependencies of: libignition-sensors3-air-pressure
2. Get dependencies of: libignition-sensors3-air-pressure-dev
2. Get dependencies of: libignition-sensors3-altimeter
2. Get dependencies of: libignition-sensors3-altimeter-dev
2. Get dependencies of: libignition-sensors3-camera
2. Get dependencies of: libignition-sensors3-camera-dev
2. Get dependencies of: libignition-sensors3-core-dev
2. Get dependencies of: libignition-sensors3-depth-camera
2. Get dependencies of: libignition-sensors3-depth-camera-dev
2. Get dependencies of: libignition-sensors3-dev
2. Get dependencies of: libignition-sensors3-gpu-lidar
2. Get dependencies of: libignition-sensors3-gpu-lidar-dev
2. Get dependencies of: libignition-sensors3-imu
2. Get dependencies of: libignition-sensors3-imu-dev
2. Get dependencies of: libignition-sensors3-lidar
2. Get dependencies of: libignition-sensors3-lidar-dev
2. Get dependencies of: libignition-sensors3-logical-camera
2. Get dependencies of: libignition-sensors3-logical-camera-dev
2. Get dependencies of: libignition-sensors3-magnetometer
2. Get dependencies of: libignition-sensors3-magnetometer-dev
2. Get dependencies of: libignition-sensors3-rendering
2. Get dependencies of: libignition-sensors3-rendering-dev
2. Get dependencies of: libignition-sensors3-rgbd-camera
2. Get dependencies of: libignition-sensors3-rgbd-camera-dev
2. Get dependencies of: libignition-sensors3-thermal-camera
2. Get dependencies of: libignition-sensors3-thermal-camera-dev
2. Get dependencies of: ignition-tools
2. Get dependencies of: libignition-tools-dev
2. Get dependencies of: libignition-transport8
2. Get dependencies of: libignition-transport8-core-dev
2. Get dependencies of: libignition-transport8-dbg
2. Get dependencies of: libignition-transport8-dev
2. Get dependencies of: libignition-transport8-log
2. Get dependencies of: libignition-transport8-log-dev
2. Get dependencies of: libsdformat9
2. Get dependencies of: libsdformat9-dbg
2. Get dependencies of: libsdformat9-dev
2. Get dependencies of: sdformat9-doc
2. Get dependencies of: sdformat9-sdf
LIST OF PACKAGES:
There was a problem hiding this comment.
uhm weird, there is no 3. step in your execution could you please run it with bash -x and copy the output somewhere please?
There was a problem hiding this comment.
nothing is printed here. Is this expected?
Could be the lack of apt-rdepends package installation.
I guess I don't fully understand what this script is meant to do.
I edited the summary to provide some information.
There was a problem hiding this comment.
Thanks, installing sudo apt install apt-rdepends did the trick. It would be nice to document that.
This works for me now. Just one last question, what's the rationale for putting this script here instead of release-tools?
Co-authored-by: Louise Poubel <[email protected]>
Example of execution on my Ubuntu Bionic system:Example of execution on my Ubuntu Focal docker: |
There was a problem hiding this comment.
I left a few review comments after discovering this PR but started a duplicate effort recently in ros-infrastructure/reprepro-updater#144
I mention a few pros/cons of our two approaches in that discussion and before doing a more thorough review here would like to decide with you which PR to use as a basis for our combined efforts.
| { | ||
| local string=${1} | ||
|
|
||
| sed 's/ /\n/g' <<< $string | sort | uniq | tr '\n' ' ' |
There was a problem hiding this comment.
I'm pretty sure that sort -u is POSIX and that cuts out a pipeline step.
Suggested change
| sed 's/ /\n/g' <<< $string | sort | uniq | tr '\n' ' ' | |
| sed 's/ /\n/g' <<< $string | sort -u | tr '\n' ' ' |
| echo "all available pkgs corresponding to that distro file" | ||
| echo | ||
|
|
||
| BRANCH_NAMES=$(grep version: ${GAZEBO_DISTRO} | awk '{ print $2 }') |
There was a problem hiding this comment.
Eliminate a pipeline step by having awk only print on matching lines.
Suggested change
| BRANCH_NAMES=$(grep version: ${GAZEBO_DISTRO} | awk '{ print $2 }') | |
| BRANCH_NAMES=$(awk '/version:/ { print $2 }' < ${GAZEBO_DISTRO}) |
| mad_output=$(apt-cache madison ${new_pkg}) | ||
| # skip purely virtual | ||
| [[ ${mad_output} == "" ]] && continue | ||
| if [[ -z $(grep 'packages.ros.org\|archive.ubuntu.com' <<< ${mad_output}) ]]; then |
There was a problem hiding this comment.
What is this line indicating?
If we're looking for packages that are sourced from the packages.osrfoundation.org repositories does it make sense to match packages.osrfoundation.org positively rather than assume packages not from packages.ros.org or archive.ubuntu.com are needed?
Since I don't have a coherent Ubuntu workstation with all three repositories configured I can't easily experiment with this during review.
iche033
pushed a commit
that referenced
this pull request
Nov 14, 2022
* Add workflow for macos-latest * Set test env to help find plugins This fixes tests on macOS and windows that were failing to find and/or load a sensor component library. In windows CI and the macOS workflow (which runs `make test` before `make install`) tests were failing with the message "Unable to find sensor plugin path". This is fixed by setting the IGN_PLUGIN_PATH in cmake to the build folder containing the compiled plugins. In the macOS jenkins build (which runs `make test` after `make install`) tests were failing with the message "SDF sensor type does not match template type". It was difficult to track down, but it appears to be caused by a failure to properly dlopen all the shared libraries linked by the component plugins when a test finds an installed component library, rather than one from the build folder. It is fixed by setting DYLD_LIBRARY_PATH to include the location of the installed libraries. Fixes #4. * Remove redundant AddPluginPaths calls from tests They don't work on windows, so just depend on the environment variables set in cmake instead. Signed-off-by: Steve Peters <[email protected]>
iche033
pushed a commit
that referenced
this pull request
Nov 14, 2022
Fix macOS/windows tests that failed to load library (#60) * Add workflow for macos-latest * Set test env to help find plugins This fixes tests on macOS and windows that were failing to find and/or load a sensor component library. In windows CI and the macOS workflow (which runs `make test` before `make install`) tests were failing with the message "Unable to find sensor plugin path". This is fixed by setting the IGN_PLUGIN_PATH in cmake to the build folder containing the compiled plugins. In the macOS jenkins build (which runs `make test` after `make install`) tests were failing with the message "SDF sensor type does not match template type". It was difficult to track down, but it appears to be caused by a failure to properly dlopen all the shared libraries linked by the component plugins when a test finds an installed component library, rather than one from the build folder. It is fixed by setting DYLD_LIBRARY_PATH to include the location of the installed libraries. Fixes #4. * Remove redundant AddPluginPaths calls from tests They don't work on windows, so just depend on the environment variables set in cmake instead. Signed-off-by: Steve Peters <[email protected]>
iche033
pushed a commit
that referenced
this pull request
Nov 14, 2022
* Prepare fo 3.2.0 release (#94) Signed-off-by: Nate Koenig <[email protected]> Co-authored-by: Nate Koenig <[email protected]> * Backport #60 from ign-sensors4 Fix macOS/windows tests that failed to load library (#60) * Add workflow for macos-latest * Set test env to help find plugins This fixes tests on macOS and windows that were failing to find and/or load a sensor component library. In windows CI and the macOS workflow (which runs `make test` before `make install`) tests were failing with the message "Unable to find sensor plugin path". This is fixed by setting the IGN_PLUGIN_PATH in cmake to the build folder containing the compiled plugins. In the macOS jenkins build (which runs `make test` after `make install`) tests were failing with the message "SDF sensor type does not match template type". It was difficult to track down, but it appears to be caused by a failure to properly dlopen all the shared libraries linked by the component plugins when a test finds an installed component library, rather than one from the build folder. It is fixed by setting DYLD_LIBRARY_PATH to include the location of the installed libraries. Fixes #4. * Remove redundant AddPluginPaths calls from tests They don't work on windows, so just depend on the environment variables set in cmake instead. Signed-off-by: Steve Peters <[email protected]> * 👩🌾 Clear Windows warnings (backport #58) (#102) Signed-off-by: Louise Poubel <[email protected]> Co-authored-by: Alejandro Hernández Cordero <[email protected]> * Master branch updates (#106) Signed-off-by: Louise Poubel <[email protected]> Co-authored-by: Nate Koenig <[email protected]> Co-authored-by: Nate Koenig <[email protected]> Co-authored-by: Steve Peters <[email protected]> Co-authored-by: Alejandro Hernández Cordero <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stupid script to generate a debian package list of all packages available in the osrfoundation repo to be uploaded to ROS repositories for a given gazebodistro file.
i.e:
bash -e gazebodistro_to_pkgs_for_ros.bash ../collection-citadel.yamlWhat the script does (note that it must be executed in the desired target platform):
apt-cache search <dep-with-tricks>to find all packages that match the keys (assuming all our binary packages has that pattern in filenames)Using it to generate the package list to be uploaded to ros repositories for new versions using Gazebo11.