5.3.1 debbuild failures on 22.04 arm*

[scpeters]

Environment

• OS Version: 22.04 arm64 and armhf
• Source or binary build? debbuild of 5.3.1

Description

• Expected behavior: debbuild should succeed
• Actual behavior: debbuild fails, with complaints about hash mismatches of libignition-common5* packages

Steps to reproduce

1. Rebuild https://build.osrfoundation.org/view/ign-garden/job/gz-common5-debbuilder/885/ and/or https://build.osrfoundation.org/view/ign-garden/job/gz-common5-debbuilder/886/

Output

Get:7 http://packages.osrfoundation.org/gazebo/ubuntu-stable jammy/main armhf libignition-common5-av all 5.3.1-1~jammy [8,836 B]

                         
Err:7 http://packages.osrfoundation.org/gazebo/ubuntu-stable jammy/main armhf libignition-common5-av all 5.3.1-1~jammy
  Hash Sum mismatch
  Hashes of expected file:
   - SHA256:8b17a0e37a42d6ae4b3dd8fa27ccb416c4111adec0a0d11b841c212726750bd5
   - SHA1:a43f8d8cab493ee46c5344aeb574f65fd1db4f91 [weak]
   - MD5Sum:d14d3ddfff2c95974387a623d2d140ba [weak]
   - Filesize:8836 [weak]
   - SHA512:f7e5dcfe137830ae64f39a4f2a7247f6b0e7a91d122b59c5a332edc0ce0517fa3b929d06b240ca03bd1560197cd663a24980b5ff8b40443835e451e3b36415ca
  Hashes of received file:
   - SHA512:cdf6f4d5a4377c8127c657a3f3b42ee6429da902e6818b276e675a2477437d5bd7ab3bda2e5072871c5b657dcf8f6d64192cb1ef598850475e23d2fce52d2c19
   - SHA256:8b17a0e37a42d6ae4b3dd8fa27ccb416c4111adec0a0d11b841c212726750bd5
   - SHA1:a43f8d8cab493ee46c5344aeb574f65fd1db4f91 [weak]
   - MD5Sum:d14d3ddfff2c95974387a623d2d140ba [weak]
   - Filesize:8836 [weak]
  Last modification reported: Thu, 17 Nov 2022 22:24:13 +0000
...
W: Sources disagree on hashes for supposely identical version '5.3.1-1~jammy' of 'libignition-common5-av:armhf'.
E: Failed to fetch http://packages.osrfoundation.org/gazebo/ubuntu-stable/pool/main/g/gz-common5/libignition-common5-av_5.3.1-1%7ejammy_all.deb  Hash Sum mismatch
   Hashes of expected file:
    - SHA256:8b17a0e37a42d6ae4b3dd8fa27ccb416c4111adec0a0d11b841c212726750bd5
    - SHA1:a43f8d8cab493ee46c5344aeb574f65fd1db4f91 [weak]
    - MD5Sum:d14d3ddfff2c95974387a623d2d140ba [weak]
    - Filesize:8836 [weak]
    - SHA512:f7e5dcfe137830ae64f39a4f2a7247f6b0e7a91d122b59c5a332edc0ce0517fa3b929d06b240ca03bd1560197cd663a24980b5ff8b40443835e451e3b36415ca
   Hashes of received file:
    - SHA512:cdf6f4d5a4377c8127c657a3f3b42ee6429da902e6818b276e675a2477437d5bd7ab3bda2e5072871c5b657dcf8f6d64192cb1ef598850475e23d2fce52d2c19
    - SHA256:8b17a0e37a42d6ae4b3dd8fa27ccb416c4111adec0a0d11b841c212726750bd5
    - SHA1:a43f8d8cab493ee46c5344aeb574f65fd1db4f91 [weak]
    - MD5Sum:d14d3ddfff2c95974387a623d2d140ba [weak]
    - Filesize:8836 [weak]
   Last modification reported: Thu, 17 Nov 2022 22:24:13 +0000
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

[j-rivero]

Uh. This is what I think is happening:

• The failure comes from autopkgtest when trying to install the generated packages
• The transitional packages named libignition-common5-* are marked as platform independent (Architecture: all)
• When the armhf build finished the transitional packages from amd64 are already in the repo, so when autopkgtest tries to install them, it found that the package is in the repository with the same name and the hashes don't match. Error.

Why the hashes of both packages that are purely transitional do not match is the key point here. Need to make some changes to get the packages out of the build and compare with the ones in the repo.

As a workaround, we can use a branch to skip the autopkgtest.

[j-rivero]

Umm, binary diff of the .debs. First one is from the build in Jenkins and the second one from the repository:

/tmp ❯ diffoscope libignition-common5-av_5.3.1-1~jammy_all.deb ~/Downloads/libignition-common5-av_5.3.1-1~jammy_all.deb 
--- libignition-common5-av_5.3.1-1~jammy_all.deb
+++ /home/jrivero/Downloads/libignition-common5-av_5.3.1-1~jammy_all.deb
├── file list
│ @@ -1,3 +1,3 @@
│  -rw-r--r--   0        0        0        4 2022-11-18 05:41:44.000000 debian-binary
│ --rw-r--r--   0        0        0      572 2022-11-18 05:41:44.000000 control.tar.xz
│ --rw-r--r--   0        0        0     8072 2022-11-18 05:41:44.000000 data.tar.xz
│ +-rw-r--r--   0        0        0      568 2022-11-18 05:41:44.000000 control.tar.xz
│ +-rw-r--r--   0        0        0     8076 2022-11-18 05:41:44.000000 data.tar.xz
├── control.tar.xz
│ ├── control.tar
│ │ ├── file list
│ │ │ @@ -1,3 +1,3 @@
│ │ │ -drwxr-xr-x   0 root         (0) root         (0)        0 2022-11-17 22:21:12.000000 ./
│ │ │ --rw-r--r--   0 root         (0) root         (0)      355 2022-11-17 22:21:11.000000 ./control
│ │ │ --rw-r--r--   0 root         (0) root         (0)      172 2022-11-17 22:21:12.000000 ./md5sums
│ │ │ +drwxr-xr-x   0 root         (0) root         (0)        0 2022-11-18 05:41:44.000000 ./
│ │ │ +-rw-r--r--   0 root         (0) root         (0)      355 2022-11-18 05:41:44.000000 ./control
│ │ │ +-rw-r--r--   0 root         (0) root         (0)      172 2022-11-18 05:41:44.000000 ./md5sums
├── data.tar.xz
│ ├── data.tar
│ │ ├── file list
│ │ │ @@ -1,7 +1,7 @@
│ │ │ -drwxr-xr-x   0 root         (0) root         (0)        0 2022-11-17 22:21:10.000000 ./
│ │ │ -drwxr-xr-x   0 root         (0) root         (0)        0 2022-11-17 22:21:01.000000 ./usr/
│ │ │ -drwxr-xr-x   0 root         (0) root         (0)        0 2022-11-17 22:21:01.000000 ./usr/share/
│ │ │ -drwxr-xr-x   0 root         (0) root         (0)        0 2022-11-17 22:21:01.000000 ./usr/share/doc/
│ │ │ -drwxr-xr-x   0 root         (0) root         (0)        0 2022-11-17 22:21:03.000000 ./usr/share/doc/libignition-common5-av/
│ │ │ --rw-r--r--   0 root         (0) root         (0)      475 2022-11-17 22:14:08.000000 ./usr/share/doc/libignition-common5-av/changelog.Debian.gz
│ │ │ --rw-r--r--   0 root         (0) root         (0)    24723 2022-11-17 22:14:08.000000 ./usr/share/doc/libignition-common5-av/copyright
│ │ │ +drwxr-xr-x   0 root         (0) root         (0)        0 2022-11-18 05:41:44.000000 ./
│ │ │ +drwxr-xr-x   0 root         (0) root         (0)        0 2022-11-18 05:41:44.000000 ./usr/
│ │ │ +drwxr-xr-x   0 root         (0) root         (0)        0 2022-11-18 05:41:44.000000 ./usr/share/
│ │ │ +drwxr-xr-x   0 root         (0) root         (0)        0 2022-11-18 05:41:44.000000 ./usr/share/doc/
│ │ │ +drwxr-xr-x   0 root         (0) root         (0)        0 2022-11-18 05:41:44.000000 ./usr/share/doc/libignition-common5-av/
│ │ │ +-rw-r--r--   0 root         (0) root         (0)      475 2022-11-18 05:41:44.000000 ./usr/share/doc/libignition-common5-av/changelog.Debian.gz
│ │ │ +-rw-r--r--   0 root         (0) root         (0)    24723 2022-11-18 05:41:44.000000 ./usr/share/doc/libignition-common5-av/copyright

Tough to see without colors:

There are four bytes of difference between control.tar.xz files although the contain the exactly the same files with the same size.

[j-rivero]

I've been looking into this trying to find a workaround for keeping autotestpkg running on ARM but avoiding to fail on problems with the hash sum check but I'm pretty sure that it is impossible to pass an option to apt to ignore that error. We can manually filter the log and make if fail if there are errors that are not the one we expect but probably does not worth the effort. I'm going to disable autopkgtest on ARM, we can rely on the amd64 check.