Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Publish RC container images #7556

Merged
merged 52 commits into from
Aug 11, 2023
Merged

Publish RC container images #7556

merged 52 commits into from
Aug 11, 2023

Conversation

chevdor
Copy link
Contributor

@chevdor chevdor commented Jul 28, 2023

This PR:

  • allows building and publishing container images for RCs
  • Reduce up the list of Dockerfile we have
  • Switch the base images to the new parity/base-bin image
  • Introduce a generic image and script for binary injection (whether those binaries are built from debug, rcs, prs, etc...)

This PR introduces a new workflows that unfortunately cannot be triggered by other worfklow. This is due to the fact that we create new RCs via GH workflows but at the time the RC and its draft are created, there is no binary available to be attached yet.

The binaries are built in our build server, this is where the workflows dispatch event can be triggered, after the artifacts+sha256+sigantures have been uploaded to the release.

The new workflow, checks the artifacts, injects them into an image and publish this image with appropriate tags.

The PR to trigger the workflows disptach can only be finalize once this PR is merged and the new workflow gets an ID.

Todo

  • allow defining mutliple custom tags
  • allow injecting multiple binaries
  • sync with @altaua when everything is ready to adjust the Gitlab jobs
  • Update documentation
  • Use new generic image to build and publish RC binaries (polkadot and staking-miner)
  • [ ] have https://github.com/paritytech/cleanroom/pull/103 ready

Tests:

  • dockerfiles/polkadot/*.Dockerfile builds & run
  • dockerfiles/staking-miner/*.Dockerfile builds & run with generic image
  • dockerfiles/adder-collator/*.Dockerfile builds & run with generic image
  • dockerfiles/malus*.Dockerfile builds & run with generic image

Notes

So far, we are using a broad range of base images where bash is sometimes /usr/bin/bash and sometimes /bin/bash.
Once all images are based of parity/base-bin, everything will need to be updated to /bin/bash.

cc @oleg-plakida

@chevdor chevdor force-pushed the wk-230726-rc-container-image branch from 28d68dc to 6a546da Compare August 2, 2023 09:26
@chevdor chevdor added B0-silent Changes should not be mentioned in any release notes T8-release This PR/Issue is related to topics touching the release notes. labels Aug 2, 2023
@Bullrich Bullrich requested review from Bullrich and removed request for Bullrich August 2, 2023 19:19
scripts/ci/common/lib.sh Outdated Show resolved Hide resolved
doc/docker.md Outdated Show resolved Hide resolved
Copy link
Contributor

@altaua altaua left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, now we just need to address all the TODOs in the pipeline specs to turn ci green.

doc/docker.md Outdated Show resolved Hide resolved
scripts/ci/dockerfiles/build-injected.sh Outdated Show resolved Hide resolved
scripts/ci/dockerfiles/build-injected.sh Outdated Show resolved Hide resolved
scripts/ci/dockerfiles/entrypoint.sh Outdated Show resolved Hide resolved
scripts/ci/dockerfiles/staking-miner/build.sh Show resolved Hide resolved
Copy link
Contributor

@altaua altaua left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changing the status of my review to "request changes"

chevdor and others added 4 commits August 10, 2023 14:33
@chevdor
Copy link
Contributor Author

chevdor commented Aug 10, 2023

@altaua thanks for the feedback, I pushed the related fixes.

@chevdor chevdor requested a review from altaua August 10, 2023 13:14
@altaua altaua requested a review from a team August 10, 2023 15:02
@chevdor chevdor enabled auto-merge (squash) August 11, 2023 12:59
@chevdor chevdor merged commit b420bad into master Aug 11, 2023
3 checks passed
@chevdor chevdor deleted the wk-230726-rc-container-image branch August 11, 2023 13:28
s0me0ne-unkn0wn pushed a commit that referenced this pull request Aug 15, 2023
* WIP

* Add missing checkout

* Add debuggin

* Fix VAR name

* Bug fix

* Rework jobs

* Revert "Rework jobs"

This reverts commit 2bfa79f.

* Add cache

* Add temp default for testing

* Add missing checkout

* Fix patch

* Comment out the GPG check for now

* Rename polkadot_injected_release into a more appropriate polkadot_injected_debian

* Refactoring / renaming

* Introduce a generic image for binary injection

* Flag files to be deleted and changes to be done

* WIP

* Fix multi binaries images

* Add test build scripts

* Remove old file, add polkadot build-injected script

* Fix doc

* Fix tagging

* Add build of the injected container

* Fix for docker

* Remove the need for TTY

* Handling container publishing

* Fix owner and registry

* Fix vars

* Fix repo

* Fix var naming

* Fix case when there is no tag

* Fix case with no tag

* Handle error

* Fix spacings

* Fix tags

* Remove unnecessary grep that may fail

* Add final check

* Clean up and introduce GPG check

* Add doc

* Add doc

* Update doc/docker.md

Co-authored-by: Mira Ressel <[email protected]>

* type

Co-authored-by: Mira Ressel <[email protected]>

* Fix used VAR

* Improve doc

* ci: Update .build-push-image jobs to use the new build-injected.sh

* ci: fix path to build-injected.sh script

* Rename the release artifacts folder to prevent confusion due to a similar folder in the gitlab CI

* ci: check out polkadot repo in .build-push-image

This seems far cleaner than copying the entire scripts/ folder into our
job artifacts.

* feat(build-injected.sh): make PROJECT_ROOT configurable

This lets us avoid a dependency on git in our CI image.

* ci: build injected images with buildah

* ci: pass full image names to zombienet

* Add missing ignore

---------

Co-authored-by: Mira Ressel <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
B0-silent Changes should not be mentioned in any release notes T8-release This PR/Issue is related to topics touching the release notes.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants