Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[5.0] integrate reproducible build with CI Build & Test workflow #1710

Merged
merged 6 commits into from
Oct 24, 2023
Merged

[5.0] integrate reproducible build with CI Build & Test workflow #1710

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0d56a95
integrate reproducible build with CI Build & Test workflow
spoonincode Oct 3, 2023
39ae21c
place 'manual' reproducible source dir in same location as CI
spoonincode Oct 4, 2023
f06b2f9
don't change PH workflows for now
spoonincode Oct 19, 2023
e63034b
Merge remote-tracking branch 'origin/release/5.0' into HEAD
spoonincode Oct 23, 2023
121ce77
Merge remote-tracking branch 'origin/release/5.0' into HEAD
spoonincode Oct 24, 2023
4ed36f9
fix tweak missed during merge
spoonincode Oct 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 48 additions & 24 deletions .github/workflows/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,9 @@ jobs:
contents: read
with:
runs-on: '["self-hosted", "enf-x86-beefy"]'
platform-files: .cicd/platforms
platform-files: |
.cicd/platforms
tools/reproducible.Dockerfile:builder

build-base:
name: Run Build Workflow
Expand Down Expand Up @@ -77,13 +79,13 @@ jobs:
echo eos-system-contracts-ref=${{inputs.override-eos-system-contracts}} >> $GITHUB_OUTPUT
fi

dev-package:
name: Build leap-dev package
package:
name: Build deb packages
needs: [platform-cache, build-base]
strategy:
fail-fast: false
matrix:
platform: [ubuntu20, ubuntu22]
platform: [ubuntu20, ubuntu22, reproducible]
runs-on: ubuntu-latest
container: ${{fromJSON(needs.platform-cache.outputs.platforms)[matrix.platform].image}}
steps:
Expand All @@ -94,41 +96,55 @@ jobs:
uses: actions/download-artifact@v3
with:
name: ${{matrix.platform}}-build
- name: Build dev package
- name: Build packages
run: |
zstdcat build.tar.zst | tar x
cd build
cpack
../tools/tweak-deb.sh leap_*.deb
- name: Install dev package
if: matrix.platform != 'reproducible'
run: |
apt-get update && apt-get upgrade -y
apt-get install -y ./build/leap_*.deb ./build/leap-dev*.deb
- name: Test using TestHarness
if: matrix.platform != 'reproducible'
run: |
python3 -c "from TestHarness import Cluster"
- name: Upload dev package
uses: actions/upload-artifact@v3
if: matrix.platform != 'reproducible'
with:
name: leap-dev-${{matrix.platform}}-amd64
path: build/leap-dev*.deb
- name: Upload leap package
uses: actions/upload-artifact@v3
if: matrix.platform == 'reproducible'
with:
name: leap-deb-amd64
path: build/leap_*.deb

tests:
name: Tests
name: Tests (${{matrix.cfg.name}})
needs: [platform-cache, build-base]
strategy:
fail-fast: false
matrix:
platform: [ubuntu20, ubuntu22]
include:
- cfg: {name: 'ubuntu20', base: 'ubuntu20', builddir: 'ubuntu20'}
- cfg: {name: 'ubuntu22', base: 'ubuntu22', builddir: 'ubuntu22'}
- cfg: {name: 'ubuntu20repro', base: 'ubuntu20', builddir: 'reproducible'}
- cfg: {name: 'ubuntu22repro', base: 'ubuntu22', builddir: 'reproducible'}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was not what I was expecting this to end up looking like; I was thinking there would be some sort of JSON file or such that defined the platform hierarchy for building and then testing. But simply ran out of time to explore other approaches. Though, honestly, other then copy pasting it 3 times this isn't too bad. But this is exactly the kind of plumbing this comment chain is discussing some, #1703 (comment)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With care, this kind of copy/paste is OK, avoiding complicating logic.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to figure out a good way to avoid it, but that can come later. The problem I'm having is that any approach to avoid this repetition seems to introduce significant indirection.

For example, maybe I'd have a JSON or YAML file as part of the repo define the relationships between the platforms we're building, what that gets tested on, what runners are used (in case we ever add ARM etc), whether the libtester tests should run (on anything but ubuntu, no), etc. But that would involve a non-trivial chunk of, I guess, javascript at the start of the job to 'unpack' and process that definition in to various rules used in the workflow: complicating indeed.

runs-on: ["self-hosted", "enf-x86-hightier"]
container:
image: ${{fromJSON(needs.platform-cache.outputs.platforms)[matrix.platform].image}}
image: ${{fromJSON(needs.platform-cache.outputs.platforms)[matrix.cfg.base].image}}
options: --security-opt seccomp=unconfined
steps:
- uses: actions/checkout@v3
- name: Download builddir
uses: actions/download-artifact@v3
with:
name: ${{matrix.platform}}-build
name: ${{matrix.cfg.builddir}}-build
- name: Run Parallel Tests
run: |
# https://github.com/actions/runner/issues/2033 -- need this because of full version label test looking at git revs
Expand All @@ -140,66 +156,74 @@ jobs:
run: awk 'BEGIN {err = 1} /bmi2/ && /adx/ {err = 0} END {exit err}' /proc/cpuinfo

np-tests:
name: NP Tests
name: NP Tests (${{matrix.cfg.name}})
needs: [platform-cache, build-base]
strategy:
fail-fast: false
matrix:
platform: [ubuntu20, ubuntu22]
include:
- cfg: {name: 'ubuntu20', base: 'ubuntu20', builddir: 'ubuntu20'}
- cfg: {name: 'ubuntu22', base: 'ubuntu22', builddir: 'ubuntu22'}
- cfg: {name: 'ubuntu20repro', base: 'ubuntu20', builddir: 'reproducible'}
- cfg: {name: 'ubuntu22repro', base: 'ubuntu22', builddir: 'reproducible'}
runs-on: ["self-hosted", "enf-x86-midtier"]
steps:
- uses: actions/checkout@v3
- name: Download builddir
uses: actions/download-artifact@v3
with:
name: ${{matrix.platform}}-build
name: ${{matrix.cfg.builddir}}-build
- name: Run tests in parallel containers
uses: ./.github/actions/parallel-ctest-containers
with:
container: ${{fromJSON(needs.platform-cache.outputs.platforms)[matrix.platform].image}}
container: ${{fromJSON(needs.platform-cache.outputs.platforms)[matrix.cfg.base].image}}
error-log-paths: '["build/etc", "build/var", "build/leap-ignition-wd", "build/TestLogs"]'
log-tarball-prefix: ${{matrix.platform}}
log-tarball-prefix: ${{matrix.cfg.name}}
tests-label: nonparallelizable_tests
test-timeout: 420
- name: Upload logs from failed tests
uses: actions/upload-artifact@v3
if: failure()
with:
name: ${{matrix.platform}}-np-logs
name: ${{matrix.cfg.name}}-np-logs
path: '*-logs.tar.gz'

lr-tests:
name: LR Tests
name: LR Tests (${{matrix.cfg.name}})
needs: [platform-cache, build-base]
strategy:
fail-fast: false
matrix:
platform: [ubuntu20, ubuntu22]
include:
- cfg: {name: 'ubuntu20', base: 'ubuntu20', builddir: 'ubuntu20'}
- cfg: {name: 'ubuntu22', base: 'ubuntu22', builddir: 'ubuntu22'}
- cfg: {name: 'ubuntu20repro', base: 'ubuntu20', builddir: 'reproducible'}
- cfg: {name: 'ubuntu22repro', base: 'ubuntu22', builddir: 'reproducible'}
runs-on: ["self-hosted", "enf-x86-lowtier"]
steps:
- uses: actions/checkout@v3
- name: Download builddir
uses: actions/download-artifact@v3
with:
name: ${{matrix.platform}}-build
name: ${{matrix.cfg.builddir}}-build
- name: Run tests in parallel containers
uses: ./.github/actions/parallel-ctest-containers
with:
container: ${{fromJSON(needs.platform-cache.outputs.platforms)[matrix.platform].image}}
container: ${{fromJSON(needs.platform-cache.outputs.platforms)[matrix.cfg.base].image}}
error-log-paths: '["build/etc", "build/var", "build/leap-ignition-wd", "build/TestLogs"]'
log-tarball-prefix: ${{matrix.platform}}
log-tarball-prefix: ${{matrix.cfg.name}}
tests-label: long_running_tests
test-timeout: 1800
- name: Upload logs from failed tests
uses: actions/upload-artifact@v3
if: failure()
with:
name: ${{matrix.platform}}-lr-logs
name: ${{matrix.cfg.name}}-lr-logs
path: '*-logs.tar.gz'

libtester-tests:
name: libtester tests
needs: [platform-cache, build-base, v, dev-package]
needs: [platform-cache, build-base, v, package]
strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -290,9 +314,9 @@ jobs:

all-passing:
name: All Required Tests Passed
needs: [dev-package, tests, np-tests, libtester-tests]
needs: [tests, np-tests, libtester-tests]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed dev-package here since libtester-tests job(s) already depend on it.

if: always()
runs-on: ubuntu-latest
steps:
- if: needs.dev-package.result != 'success' || needs.tests.result != 'success' || needs.np-tests.result != 'success' || needs.libtester-tests.result != 'success'
- if: needs.tests.result != 'success' || needs.np-tests.result != 'success' || needs.libtester-tests.result != 'success'
run: false
8 changes: 5 additions & 3 deletions tools/reproducible.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,10 +97,12 @@ FROM builder AS build

ARG LEAP_BUILD_JOBS

COPY / /src
RUN cmake -S src -B build -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -GNinja && \
# Yuck: This places the source at the same location as leap's CI (build.yaml, build_base.yaml). Unfortunately this location only matches
# when build.yaml etc are being run from a repository named leap.
COPY / /__w/leap/leap
RUN cmake -S /__w/leap/leap -B build -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -GNinja && \
cmake --build build -t package -- ${LEAP_BUILD_JOBS:+-j$LEAP_BUILD_JOBS} && \
src/tools/tweak-deb.sh build/leap_*.deb
/__w/leap/leap/tools/tweak-deb.sh build/leap_*.deb
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And in the perfect example for failure of,

It's important these two methods remain functionally identical for the produced .deb file to ensure that both CI & manual user builds produce the exact same output

This change was required for the above statement to remain true because in some cases full paths get baked in to the final executable. When built via build-base.yaml the source path is effectively /__w/leap/leap/ while the reproducible.Dockerfile was /src/. I entertained 3 ways to fix this,

  1. Change CI to place source in /src. This involves changes in many locations because we actions/checkout@v3 in many different jobs during the build & tests.
  2. Change CI to make use of something like -ffile-prefix-map=$PWD=/src. This seems to work, but results in more variation of compile flags between CI & manual build.
  3. Change reproducible.Dockerfile's build stage to place source in /__w/leap/leap like CI does.

Given the 11th hour, I opted for option 3 even though it has a negative trade off that CI only places source in /__w/leap/leap if the repository is named leap. If someone were to fork the repo to a different name, they would have divergence between CI & manual reproducible builds. But our current CI workflows are, unfortunately, ENF centric (see #416 for how this could be resolved; though I doubt something like the NP Tests will pass on a free GH runner these days), so this seems like an okay tradeoff for the time being. We can migrate to option 2 in the future is really needed.


FROM scratch AS exporter
COPY --from=build /build/*.deb /build/*.tar.* /