Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

migrate to new platform-cache-workflow to manage Dockerfile caching in CI #1703

Merged
merged 3 commits into from
Oct 3, 2023
Merged

migrate to new platform-cache-workflow to manage Dockerfile caching in CI #1703

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
efc48d6
migrate to new platform-cache-workflow
spoonincode Oct 2, 2023
8445137
remove reference to reproducible.Dockerfile that isn't on this branch…
spoonincode Oct 3, 2023
381b3ed
fix platform-list in PHBC workflow
spoonincode Oct 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 0 additions & 8 deletions .cicd/platforms.json
View file
Open in desktop

This file was deleted.

47 changes: 20 additions & 27 deletions .github/workflows/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,23 +31,23 @@ defaults:
shell: bash

jobs:
platforms:
name: Run Platforms Workflow
uses: ./.github/workflows/platforms.yaml
platform-cache:
name: Platform Cache
uses: AntelopeIO/platform-cache-workflow/.github/workflows/platformcache.yaml@v1
permissions:
packages: write
contents: read
with:
runs-on: '["self-hosted", "enf-x86-beefy"]'
platform-files: .cicd/platforms

build-base:
name: Run Build Workflow
uses: ./.github/workflows/build_base.yaml
needs: [platforms]
needs: [platform-cache]
with:
p: ${{needs.platforms.outputs.p}}
platform-matrix: ${{needs.platforms.outputs.platform-matrix}}
permissions:
packages: write
contents: read
platforms: ${{needs.platform-cache.outputs.platforms}}
platform-list: ${{needs.platform-cache.outputs.platform-list}}

v:
name: Discover Versions
Expand Down Expand Up @@ -79,14 +79,13 @@ jobs:

dev-package:
name: Build leap-dev package
needs: [platforms, build-base]
if: always() && needs.platforms.result == 'success' && needs.build-base.result == 'success'
needs: [platform-cache, build-base]
strategy:
fail-fast: false
matrix:
platform: [ubuntu20, ubuntu22]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a thought, while doing this update should we update the platforms in the matrix: sections throughout our workflows to be dynamic from the platform-cache-workflow like done in ph_backward_compatibility?

platform: ${{fromJSON(needs.platform-cache.outputs.platform-list)}}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like that is the right approach now, but that's only because there is a 1:1 between all the platform files and all the platforms we're going to build leap-dev on. That may not always be true. For example, we may choose not to support building on Ubuntu 20, but still support running the reproducible build on Ubuntu 20. In such a scenario we'd still need a ubuntu20 platform defined in the platforms/platform-list, but its purpose would only be for testing. So using platform-list here wouldn't work.

(I expected to already need to make use of this for the reproducible builds: I expected I'd need a special ubuntu20+cmake3.27 platform to run ctest on since cmake3.27 is what is used to build. Shockingly, that doesn't appear to be the case)

I feel like we probably need some sort of .json file that gets sucked in at the start of the workflow that defines these copy pasted repetitive items like [ubuntu20, ubuntu22]. (It gets more unwieldy with the upcoming reproducible workflow additions)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, hm, I see I did plumb that through elsewhere as platform-list..

leap/.github/workflows/performance_harness_run.yaml

Line 103 in 381b3ed

platform-list: '["${{github.event.inputs.platform-choice}}"]'

leap/.github/workflows/build.yaml

Line 50 in 381b3ed

platform-list: ${{needs.platform-cache.outputs.platform-list}}

In this case here, I guess it could be changed to platform-list for consistency

Copy link
Contributor

@oschwaldp-oci oschwaldp-oci Oct 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually those are from the work I did earlier to use platform-matrix which you renamed to platform-list in this PR with your move to the platform-cache-workflow.

I'm fine with leaving the [ubuntu20, ubuntu22] was just asking since this PR was working in that space. You called out a good example where this won't be 1:1, so that's good enough of a reason to leave it. I just wish there was a way to not have it hardcoded everywhere. It makes updating the workflows a bit tedious and error prone when adopting or dropping a platform.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should come up with something better; but at least let's look at how the reproducible one works before noodling on it more. (and, at the danger of overthinking future proofing, how something like an ARM build or a macOS build would look)

runs-on: ubuntu-latest
container: ${{fromJSON(needs.platforms.outputs.p)[matrix.platform].image}}
container: ${{fromJSON(needs.platform-cache.outputs.platforms)[matrix.platform].image}}
steps:
- uses: actions/checkout@v3
with:
Expand Down Expand Up @@ -115,15 +114,14 @@ jobs:

tests:
name: Tests
needs: [platforms, build-base]
if: always() && needs.platforms.result == 'success' && needs.build-base.result == 'success'
needs: [platform-cache, build-base]
strategy:
fail-fast: false
matrix:
platform: [ubuntu20, ubuntu22]
runs-on: ["self-hosted", "enf-x86-hightier"]
container:
image: ${{fromJSON(needs.platforms.outputs.p)[matrix.platform].image}}
image: ${{fromJSON(needs.platform-cache.outputs.platforms)[matrix.platform].image}}
options: --security-opt seccomp=unconfined
steps:
- uses: actions/checkout@v3
Expand All @@ -143,8 +141,7 @@ jobs:

np-tests:
name: NP Tests
needs: [platforms, build-base]
if: always() && needs.platforms.result == 'success' && needs.build-base.result == 'success'
needs: [platform-cache, build-base]
strategy:
fail-fast: false
matrix:
Expand All @@ -159,7 +156,7 @@ jobs:
- name: Run tests in parallel containers
uses: ./.github/actions/parallel-ctest-containers
with:
container: ${{fromJSON(needs.platforms.outputs.p)[matrix.platform].image}}
container: ${{fromJSON(needs.platform-cache.outputs.platforms)[matrix.platform].image}}
error-log-paths: '["build/etc", "build/var", "build/leap-ignition-wd", "build/TestLogs"]'
log-tarball-prefix: ${{matrix.platform}}
tests-label: nonparallelizable_tests
Expand All @@ -173,8 +170,7 @@ jobs:

lr-tests:
name: LR Tests
needs: [platforms, build-base]
if: always() && needs.platforms.result == 'success' && needs.build-base.result == 'success'
needs: [platform-cache, build-base]
strategy:
fail-fast: false
matrix:
Expand All @@ -189,7 +185,7 @@ jobs:
- name: Run tests in parallel containers
uses: ./.github/actions/parallel-ctest-containers
with:
container: ${{fromJSON(needs.platforms.outputs.p)[matrix.platform].image}}
container: ${{fromJSON(needs.platform-cache.outputs.platforms)[matrix.platform].image}}
error-log-paths: '["build/etc", "build/var", "build/leap-ignition-wd", "build/TestLogs"]'
log-tarball-prefix: ${{matrix.platform}}
tests-label: long_running_tests
Expand All @@ -203,15 +199,14 @@ jobs:

libtester-tests:
name: libtester tests
needs: [platforms, build-base, v, dev-package]
if: always() && needs.platforms.result == 'success' && needs.v.result == 'success' && needs.dev-package.result == 'success'
needs: [platform-cache, build-base, v, dev-package]
strategy:
fail-fast: false
matrix:
platform: [ubuntu20, ubuntu22]
test: [build-tree, make-dev-install, deb-install]
runs-on: ["self-hosted", "enf-x86-midtier"]
container: ${{ matrix.test != 'deb-install' && fromJSON(needs.platforms.outputs.p)[matrix.platform].image || matrix.platform == 'ubuntu20' && 'ubuntu:focal' || 'ubuntu:jammy' }}
container: ${{ matrix.test != 'deb-install' && fromJSON(needs.platform-cache.outputs.platforms)[matrix.platform].image || matrix.platform == 'ubuntu20' && 'ubuntu:focal' || 'ubuntu:jammy' }}
env:
DEBIAN_FRONTEND: noninteractive
TZ: Etc/UTC
Expand Down Expand Up @@ -296,8 +291,6 @@ jobs:
all-passing:
name: All Required Tests Passed
needs: [dev-package, tests, np-tests, libtester-tests]
if: always()
runs-on: ubuntu-latest
steps:
- if: needs.dev-package.result != 'success' || needs.tests.result != 'success' || needs.np-tests.result != 'success' || needs.libtester-tests.result != 'success'
run: false
- run: true
12 changes: 6 additions & 6 deletions .github/workflows/build_base.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@ name: "Build leap"
on:
workflow_call:
inputs:
p:
description: "Discovered Build Platforms"
platforms:
description: "Platforms definitions"
type: string
required: true
platform-matrix:
description: "Platform Matrix"
platform-list:
description: "Array of platforms"
type: string
required: true

Expand All @@ -26,9 +26,9 @@ jobs:
strategy:
fail-fast: false
matrix:
platform: ${{fromJSON(inputs.platform-matrix)}}
platform: ${{fromJSON(inputs.platform-list)}}
runs-on: ["self-hosted", "enf-x86-beefy"]
container: ${{fromJSON(inputs.p)[matrix.platform].image}}
container: ${{fromJSON(inputs.platforms)[matrix.platform].image}}
steps:
- uses: actions/checkout@v3
with:
Expand Down
31 changes: 13 additions & 18 deletions .github/workflows/performance_harness_run.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,21 +58,19 @@ jobs:
echo leap-prerelease=${{inputs.override-leap-prerelease}} >> $GITHUB_OUTPUT
fi

platforms:
name: Run Platforms Workflow
uses: ./.github/workflows/platforms.yaml
with:
override-build-matrix: ${{github.event.inputs.platform-choice}}
platform-cache:
name: Platform Cache
uses: AntelopeIO/platform-cache-workflow/.github/workflows/platformcache.yaml@v1
permissions:
packages: write
contents: read
with:
runs-on: '["self-hosted", "enf-x86-beefy"]'
platform-files: .cicd/platforms

reuse-build:
name: Reuse leap build
needs: [v]
if: |
always() &&
needs.v.result == 'success'
runs-on: ubuntu-latest
outputs:
build-artifact: ${{steps.downloadBuild.outputs.downloaded-file}}
Expand All @@ -97,23 +95,20 @@ jobs:

build-base:
name: Run Build Workflow
needs: [platforms, reuse-build]
if: always() && needs.platforms.result == 'success' && needs.reuse-build.outputs.build-artifact == ''
needs: [platform-cache, reuse-build]
if: needs.reuse-build.outputs.build-artifact == ''
uses: ./.github/workflows/build_base.yaml
with:
p: ${{needs.platforms.outputs.p}}
platform-matrix: ${{needs.platforms.outputs.platform-matrix}}
permissions:
packages: write
contents: read
platforms: ${{needs.platform-cache.outputs.platforms}}
platform-list: '["${{github.event.inputs.platform-choice}}"]'

tests:
name: Tests
needs: [v, platforms, reuse-build, build-base]
if: always() && needs.platforms.result == 'success' && (needs.build-base.result == 'success' || needs.reuse-build.result == 'success')
needs: [v, platform-cache, reuse-build, build-base]
if: always() && (needs.build-base.result == 'success' || needs.reuse-build.result == 'success')
runs-on: ["Leap-Perf-Ubuntu-22-16x64x600"]
container:
image: ${{fromJSON(needs.platforms.outputs.p)[github.event.inputs.platform-choice].image}}
image: ${{fromJSON(needs.platform-cache.outputs.platforms)[github.event.inputs.platform-choice].image}}
steps:
- name: Download builddir
uses: actions/download-artifact@v3
Expand Down
24 changes: 13 additions & 11 deletions .github/workflows/ph_backward_compatibility.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,36 +12,38 @@ defaults:
shell: bash

jobs:
platforms:
name: Run Platforms Workflow
uses: ./.github/workflows/platforms.yaml
platform-cache:
name: Platform Cache
uses: AntelopeIO/platform-cache-workflow/.github/workflows/platformcache.yaml@v1
permissions:
packages: write
contents: read
packages: write
with:
runs-on: '["self-hosted", "enf-x86-beefy"]'
platform-files: .cicd/platforms

build-base:
name: Run Build Workflow
uses: ./.github/workflows/build_base.yaml
needs: [platforms]
needs: [platform-cache]
with:
p: ${{needs.platforms.outputs.p}}
platform-matrix: ${{needs.platforms.outputs.platform-matrix}}
platforms: ${{needs.platform-cache.outputs.platforms}}
platform-list: ${{needs.platform-cache.outputs.platform-list}}
permissions:
packages: write
contents: read

tests:
name: Tests
needs: [platforms, build-base]
if: always() && needs.platforms.result == 'success' && needs.build-base.result == 'success'
needs: [platform-cache, build-base]
strategy:
fail-fast: false
matrix:
platform: ${{fromJSON(needs.platforms.outputs.platform-matrix)}}
platform: ${{fromJSON(needs.platform-cache.outputs.platform-list)}}
release: [3.1, 3.2, 4.0]
runs-on: ["self-hosted", "enf-x86-lowtier"]
container:
image: ${{fromJSON(needs.platforms.outputs.p)[matrix.platform].image}}
image: ${{fromJSON(needs.platform-cache.outputs.platforms)[matrix.platform].image}}
options: --security-opt seccomp=unconfined
steps:
- uses: actions/checkout@v3
Expand Down
88 changes: 0 additions & 88 deletions .github/workflows/platforms.yaml
View file
Open in desktop

This file was deleted.