How to handle multiple builds/releases for same commit?

[florianmutter]

Our build pipeline roughly works like this:

• build a new docker image with the commit sha as tag
• get the digest from the image
• build manifests for service and set image digest
• push manifests to kuberpult

We now face an issue when a build is maunally retriggered. All steps from above run again but create a new image (image build is not reproducible). The new images is tagged with the commit sha but the manifest in kuberpult is not updated with the new digest (see #1372).

This means an image that is not used is tagged with the commit sha tag and the image that is referenced in the manifest is not tagged with the commit sha anymore. This is misleading and causes issues if we try to delete images that are seemingly not related to any release (e.g. local test builds).

Do you have a suggestion how to handle this? Ideas I had:

• Prevent second build (is this even possible)?
• Make kuberpult update the manifest with the new digest
• Add ${{ github.run_attempt }} to release version (is this possible? Can a release version be 12345-1?)
• Add ${{ github.run_attempt }} to image tag. We would still use the first image but at least the second one would have a tag where we can prevent it from being deleted.

[sven-urbanski-freiheit-com]

Prevent second build (is this even possible)?

Not sure how that would be possible.

Make kuberpult update the manifest with the new digest

The way the /release endpoint is intended is to just create, never update.
If we do recognize that the same thing is released under the same release number, same manifest, etc, then we accept that. But the slightest deviation will result in an /release returning an error.

Add ${{ github.run_attempt }} to release version (is this possible? Can a release version be 12345-1?)

This is not possible, the release version must be a number.

Add ${{ github.run_attempt }} to image tag.

I would not recommend to use the run attempt, because then builds are not really reproducible.

I think the build should be reproducible to solve this properly. We generally recommend to use the commit hash (or some other hash) as part of the docker image. If a build is triggered again, it should create the exact same image with the exact same name.

It's fine if your process creates a new image, but it then should be the same image & tag name. Using digests is an interesting idea that's not fully supported by kuberpult when retrying.

You could use the run_number to figure out, if you should skip the step push manifests to kuberpult. But then you also need to take into consideration if the last run was even getting to attempt this step.

You could also use math to get out of this. E.g. increase the release number by 10 each time (#number of commits times 10), so you have 9 more "retry attempts". I would not recommend this though.

I think the simplest solution is to tag each image, and use the tag instead of the digest.

[florianmutter]

I agree reproducible images would be optimal. We use jib and there seem to be issues with reproducible builds (GoogleContainerTools/jib#4142)

Using tags in the manifest would be a good workaround for now.