Implement TEST=debians #719

mathias-luedtke · 2021-07-22T09:48:44Z

No description provided.

jspricke · 2021-07-22T12:05:28Z

industrial_ci/src/tests/debians.sh

+    cd "$pkg_path"
+    ici_quiet_echo bloom-generate rosdebian --ros-distro="$ROS_DISTRO" --debian-inc="ici~"
+    ici_quiet_echo dpkg-buildpackage -b -uc -us
+    mv ../*.{buildinfo,changes,deb} .


please give reasons why you need this (instead of just using .. in the path for dpkg/apt below.

To only install a single *.deb file, and not all of them every time.

hmkay, maybe add a comment then.

jspricke · 2021-07-22T12:05:55Z

industrial_ci/src/tests/debians.sh

+    ici_quiet_echo bloom-generate rosdebian --ros-distro="$ROS_DISTRO" --debian-inc="ici~"
+    ici_quiet_echo dpkg-buildpackage -b -uc -us
+    mv ../*.{buildinfo,changes,deb} .
+    ici_quiet_echo ici_asroot dpkg -i ./*.deb


Suggested change

ici_quiet_echo ici_asroot dpkg -i ./*.deb

ici_quiet_echo ici_asroot apt-get install ./*.deb

(will resolve dependencies as well)

(will resolve dependencies as well)

It should must resolve dependencies!
All dependencies should already be installed.

you should only install build dependencies to build the package, but installing the build package could bring additional run dependencies.

but installing the build package could bring additional run dependencies.

I just wanted to avoid pulling in the already built packages from the buildfarm.

but then dpkg could fail and thus your CI job (I guess).

but then dpkg could fail and thus your CI job (I guess).

Yes, that's what I wanted to test for..

Just for completeness, I think it could happen that colcons topological sort builds a package A before B but A needs B to get installed. So it could be that a package is not installable after it was build. You could add an install test with all packages at the end, maybe.

jspricke · 2021-07-22T12:11:36Z

industrial_ci/src/tests/debians.sh

+        ici_run "install_${name}_dependencies" ici_quiet ici_install_dependencies "$UNDERLAY" "$ROSDEP_SKIP_KEYS" "$sourcespace"
+
+        while read -r -a pkg; do
+          ici_run "${pkg[0]}_debian" build_and_install_debian "$current/${pkg[1]}"


this would still all be run in the same root filesystem, right?
The point of sbuild is to have a minimal chroot (container) (only essential packages installed) to find missing dependencies. With your setup if you have a repo with multiple ROS packages and package A ab B have a dependency X but only A declares it, packages B would still build if the dependency was installed for A before. That would not happen with sbuild nor work on the ROS build farm (and was the point in #697).

I see. Not sure how to solve this. sbuild is just too complicated to set-up (incl. some manual steps).
The ROS pre-release test builds each package in a clean environment already, so I wonder why it did not fail in the first place.

which pre-release tests do you mean?
http://prerelease.ros.org/ does not, afaik.

Regarding sbuild being too complicated, you could spin up a new docker container for every package

which pre-release tests do you mean?

http://prerelease.ros.org/ does not, afaik.

This one, but with without the need to register it in rosdistro.
I thought that it creates a new container (actually, new containers) for each package, but after digging into the log files this might not be the case :/

Regarding sbuild being too complicated, you could spin up a new docker container for every package

Yes, I will try this next. Together with a shared, local repository.

The latest version does this, at least locally.
I will provide a test case as well later.

industrial_ci/src/tests/debians.sh

jspricke

I think this looks good (but didn't look to deep), Thanks for working on this!

Maybe a test with multiple packages would be a good addition (did you try it with Moveit already?)

jspricke · 2021-07-23T10:36:57Z

industrial_ci/src/tests/debians.sh

+)
+
+function run_debians() {
+    # shellcheck source=industrial_ci/src//isolation/docker.sh


jspricke · 2021-07-23T10:37:50Z

industrial_ci/src/tests/debians.sh

+    export BUILDER=colcon
+    ici_source_builder
+    ici_run "${BUILDER}_setup" ici_quiet builder_setup
+    ici_run "setup_bloom" ici_quiet ici_install_pkgs_for_command bloom-generate python3-bloom debhelper


why do you need debhelper here?

It is not needed anymore, I will remove it!

mathias-luedtke · 2021-07-23T10:47:42Z

Maybe a test with multiple packages would be a good addition

The testcase already builds two packages

did you try it with Moveit already?

I am on it: https://github.com/ipa-mdl/industrial_ci/actions/runs/1059400937

jspricke · 2021-07-23T12:31:03Z

The testcase already builds two packages

Ah, missed the industrial-ci package. Maybe add a dependency between them?

I am on it: https://github.com/ipa-mdl/industrial_ci/actions/runs/1059400937

Awesome, nice result :).

mathias-luedtke · 2021-07-23T14:44:13Z

Awesome, nice result :).

Actually, it failed for a different reason (I believe because of repo mixups)
However, https://github.com/ipa-mdl/industrial_ci/actions/runs/1059855175 fails with:

dpkg-shlibdeps: error: no dependency information found for /opt/ros/galactic/lib/libmoveit_ros_occupancy_map_monitor.so. (used by debian/ros-galactic-moveit-ros-planning/opt/ros/galactic/lib/libmoveit_planning_scene_monitor.so.)
Hint: check if the library actually comes from a package.

Just like the build farm.

Maybe add a dependency between them?

The two packages must be used separately.
But we should add a proper test case. Either an artificial one or a real pepository.

mathias-luedtke mentioned this pull request Jul 22, 2021

pre-release tests that more closely resemble what the buildfarm does for ros2 #697

Closed

mathias-luedtke force-pushed the feature/test-debians branch from 3bf4a63 to a1d128e Compare July 22, 2021 10:02

jspricke suggested changes Jul 22, 2021

View reviewed changes

mathias-luedtke force-pushed the feature/test-debians branch 4 times, most recently from 9c0e6f7 to 5e8f106 Compare July 22, 2021 21:14

mathias-luedtke mentioned this pull request Jul 22, 2021

Do report missing setup.bash if UNDERLAY=/opt/ros/$ROS_DISTRO #721

Merged

mathias-luedtke force-pushed the feature/test-debians branch 8 times, most recently from 00fda10 to 0a581b3 Compare July 22, 2021 23:02

jspricke approved these changes Jul 23, 2021

View reviewed changes

mathias-luedtke force-pushed the feature/test-debians branch 10 times, most recently from 4b4edf0 to cc0be4a Compare July 29, 2021 20:51

mathias-luedtke force-pushed the feature/test-debians branch 2 times, most recently from 3ba1f2b to d8481fb Compare August 2, 2021 08:46

Implement TEST=debians

08c14c6

mathias-luedtke force-pushed the feature/test-debians branch from d8481fb to 08c14c6 Compare August 2, 2021 13:41

mathias-luedtke marked this pull request as ready for review August 2, 2021 13:56

mathias-luedtke merged commit f9b49dd into ros-industrial:master Aug 2, 2021

mathias-luedtke deleted the feature/test-debians branch August 2, 2021 13:57

mathias-luedtke mentioned this pull request Aug 6, 2021

Run tests against the installed version of the package (not the build/devel-time version) #738

Open

tylerjw mentioned this pull request Jan 26, 2022

Test using debians test from industrial_ci moveit/moveit2#1034

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement TEST=debians #719

Implement TEST=debians #719

mathias-luedtke commented Jul 22, 2021

jspricke Jul 22, 2021

mathias-luedtke Jul 22, 2021

jspricke Jul 22, 2021

mathias-luedtke Jul 22, 2021

jspricke Jul 22, 2021

jspricke Jul 22, 2021

mathias-luedtke Jul 22, 2021

jspricke Jul 22, 2021

mathias-luedtke Jul 22, 2021

jspricke Jul 22, 2021

mathias-luedtke Jul 22, 2021

jspricke Jul 23, 2021

jspricke Jul 22, 2021

mathias-luedtke Jul 22, 2021

jspricke Jul 22, 2021

jspricke Jul 22, 2021

mathias-luedtke Jul 22, 2021

mathias-luedtke Jul 22, 2021

jspricke left a comment

jspricke Jul 23, 2021

jspricke Jul 23, 2021

mathias-luedtke Jul 23, 2021

mathias-luedtke commented Jul 23, 2021

jspricke commented Jul 23, 2021

mathias-luedtke commented Jul 23, 2021

	ici_quiet_echo ici_asroot dpkg -i ./*.deb
	ici_quiet_echo ici_asroot apt-get install ./*.deb

Implement TEST=debians #719

Implement TEST=debians #719

Conversation

mathias-luedtke commented Jul 22, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jspricke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mathias-luedtke commented Jul 23, 2021

jspricke commented Jul 23, 2021

mathias-luedtke commented Jul 23, 2021