Build and push Docker vitess/base and component images to DockerHub…

… from GitHub Actions (#14271)

Signed-off-by: Florent Poinsard <[email protected]>

.github/workflows/docker_build_base.yml

@@ -0,0 +1,261 @@
+name: Docker Build Base
+on:
+  push:
+    branches:
+      - main
+    tags:
+      - '*'
+
+concurrency:
+  group: format('{0}-{1}', ${{ github.ref }}, 'Docker Build Base')
+  cancel-in-progress: true
+
+permissions: read-all
+
+jobs:
+  build_and_push_base:
+    name: Build and push vitess/base Docker images
+    runs-on: gh-hosted-runners-16cores-1
+    if: github.repository == 'vitessio/vitess'
+
+    strategy:
+      fail-fast: true
+      matrix:
+        branch: [ latest, mysql57, percona57, percona80 ]
+
+    steps:
+      - name: Check out code
+        uses: actions/checkout@v3
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v3
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Set Dockerfile path
+        run: |
+          if [[ "${{ matrix.branch }}" == "latest" ]]; then
+            echo "DOCKERFILE=./docker/base/Dockerfile" >> $GITHUB_ENV
+          else
+            echo "DOCKERFILE=./docker/base/Dockerfile.${{ matrix.branch }}" >> $GITHUB_ENV
+          fi
+
+      - name: Build and push on main
+        if: github.ref == 'refs/heads/main'
+        uses: docker/build-push-action@v5
+        with:
+          context: .
+          file: ${{ env.DOCKERFILE }}
+          push: true
+          tags: vitess/base:${{ matrix.branch }}
+
+      ######
+      # All code below only applies to new tags
+      ######
+
+      - name: Get the Git tag
+        if: startsWith(github.ref, 'refs/tags/')
+        run: echo "TAG_NAME=${GITHUB_REF#refs/tags/}" >> $GITHUB_ENV
+
+      - name: Set Docker tag name
+        if: startsWith(github.ref, 'refs/tags/') && matrix.branch == 'latest'
+        run: |
+          if [[ "${{ matrix.branch }}" == "latest" ]]; then
+            echo "DOCKER_TAG=vitess/base:${TAG_NAME}" >> $GITHUB_ENV
+          fi
+
+      - name: Build and push on tags
+        if: startsWith(github.ref, 'refs/tags/') && matrix.branch == 'latest'
+        uses: docker/build-push-action@v5
+        with:
+          context: .
+          file: ${{ env.DOCKERFILE }}
+          push: true
+          tags: ${{ env.DOCKER_TAG }}
+
+  build_and_push_k8s:
+    needs: build_and_push_base
+    name: Build and push vitess/k8s image
+    runs-on: gh-hosted-runners-16cores-1
+    if: github.repository == 'vitessio/vitess'
+
+    strategy:
+      fail-fast: true
+      matrix:
+        debian: [ bullseye, bookworm ]
+
+    steps:
+      - name: Check out code
+        uses: actions/checkout@v3
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v3
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Set Docker context path
+        run: |
+          echo "DOCKER_CTX=./docker/k8s" >> $GITHUB_ENV
+
+      - name: Build and push on main latest tag
+        if: github.ref == 'refs/heads/main' && matrix.debian == 'bookworm'
+        uses: docker/build-push-action@v5
+        with:
+          context: ${{ env.DOCKER_CTX }}
+          push: true
+          tags: vitess/k8s:latest
+          build-args: |
+            VT_BASE_VER=latest
+            DEBIAN_VER=${{ matrix.debian }}-slim
+
+      - name: Build and push on main debian specific tag
+        if: github.ref == 'refs/heads/main'
+        uses: docker/build-push-action@v5
+        with:
+          context: ${{ env.DOCKER_CTX }}
+          push: true
+          tags: vitess/k8s:latest-${{ matrix.debian }}
+          build-args: |
+            VT_BASE_VER=latest
+            DEBIAN_VER=${{ matrix.debian }}-slim
+
+      ######
+      # All code below only applies to new tags
+      ######
+
+      - name: Get the Git tag
+        if: startsWith(github.ref, 'refs/tags/')
+        run: echo "TAG_NAME=${GITHUB_REF#refs/tags/}" >> $GITHUB_ENV
+
+      # We push git-tag-based k8s image to three tags, i.e. for 'v19.0.0' we push to:
+      #
+      #     vitess/k8s:v19.0.0            (DOCKER_TAG_DEFAULT_DEBIAN)
+      #     vitess/k8s:v19.0.0-bookworm   (DOCKER_TAG)
+      #     vitess/k8s:v19.0.0-bullseye   (DOCKER_TAG)
+      #
+      - name: Set Docker tag name
+        if: startsWith(github.ref, 'refs/tags/')
+        run: |
+          echo "DOCKER_TAG_DEFAULT_DEBIAN=vitess/k8s:${TAG_NAME}" >> $GITHUB_ENV
+          echo "DOCKER_TAG=vitess/k8s:${TAG_NAME}-${{ matrix.debian }}" >> $GITHUB_ENV
+
+      # Build and Push component image to DOCKER_TAG, applies to both debian version
+      - name: Build and push on tags using Debian extension
+        if: startsWith(github.ref, 'refs/tags/')
+        uses: docker/build-push-action@v5
+        with:
+          context: ${{ env.DOCKER_CTX }}
+          push: true
+          tags: ${{ env.DOCKER_TAG }}
+          build-args: |
+            VT_BASE_VER=${{ env.TAG_NAME }}
+            DEBIAN_VER=${{ matrix.debian }}-slim
+
+      # Build and Push component image to DOCKER_TAG_DEFAULT_DEBIAN, only applies when building the default Debian version (bookworm)
+      # It is fine to build a second time here when "matrix.debian == 'bookworm'" as we have cached the first build already
+      - name: Build and push on tags without Debian extension
+        if: startsWith(github.ref, 'refs/tags/') && matrix.debian == 'bookworm'
+        uses: docker/build-push-action@v5
+        with:
+          context: ${{ env.DOCKER_CTX }}
+          push: true
+          tags: ${{ env.DOCKER_TAG_DEFAULT_DEBIAN }}
+          build-args: |
+            VT_BASE_VER=${{ env.TAG_NAME }}
+            DEBIAN_VER=${{ matrix.debian }}-slim
+
+
+  build_and_push_components:
+    needs: build_and_push_k8s
+    name: Build and push vitess components Docker images
+    runs-on: gh-hosted-runners-16cores-1
+    if: github.repository == 'vitessio/vitess'
+
+    strategy:
+      fail-fast: true
+      matrix:
+        debian: [ bullseye, bookworm ]
+        component: [ vtadmin, vtorc, vtgate, vttablet, mysqlctld, mysqlctl, vtctl, vtctlclient, vtctld, logrotate, logtail ]
+
+    steps:
+      - name: Check out code
+        uses: actions/checkout@v3
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v3
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Set Docker context path
+        run: |
+          echo "DOCKER_CTX=./docker/k8s/${{ matrix.component }}" >> $GITHUB_ENV
+
+      - name: Build and push on main latest tag
+        if: github.ref == 'refs/heads/main' && matrix.debian == 'bookworm'
+        uses: docker/build-push-action@v5
+        with:
+          context: ${{ env.DOCKER_CTX }}
+          push: true
+          tags: vitess/${{ matrix.component }}:latest
+          build-args: |
+            VT_BASE_VER=latest
+            DEBIAN_VER=${{ matrix.debian }}-slim
+
+      - name: Build and push on main debian specific tag
+        if: github.ref == 'refs/heads/main'
+        uses: docker/build-push-action@v5
+        with:
+          context: ${{ env.DOCKER_CTX }}
+          push: true
+          tags: vitess/${{ matrix.component }}:latest-${{ matrix.debian }}
+          build-args: |
+            VT_BASE_VER=latest
+            DEBIAN_VER=${{ matrix.debian }}-slim
+
+      ######
+      # All code below only applies to new tags
+      ######
+
+      - name: Get the Git tag
+        if: startsWith(github.ref, 'refs/tags/')
+        run: echo "TAG_NAME=${GITHUB_REF#refs/tags/}" >> $GITHUB_ENV
+
+      # We push git-tag-based images to three tags, i.e. for 'v19.0.0' we push to:
+      #
+      #     vitess/${{ matrix.component }}:v19.0.0            (DOCKER_TAG_DEFAULT_DEBIAN)
+      #     vitess/${{ matrix.component }}:v19.0.0-bookworm   (DOCKER_TAG)
+      #     vitess/${{ matrix.component }}:v19.0.0-bullseye   (DOCKER_TAG)
+      #
+      - name: Set Docker tag name
+        if: startsWith(github.ref, 'refs/tags/')
+        run: |
+          echo "DOCKER_TAG_DEFAULT_DEBIAN=vitess/${{ matrix.component }}:${TAG_NAME}" >> $GITHUB_ENV
+          echo "DOCKER_TAG=vitess/${{ matrix.component }}:${TAG_NAME}-${{ matrix.debian }}" >> $GITHUB_ENV
+
+      # Build and Push component image to DOCKER_TAG, applies to both debian version
+      - name: Build and push on tags using Debian extension
+        if: startsWith(github.ref, 'refs/tags/')
+        uses: docker/build-push-action@v5
+        with:
+          context: ${{ env.DOCKER_CTX }}
+          push: true
+          tags: ${{ env.DOCKER_TAG }}
+          build-args: |
+            VT_BASE_VER=${{ env.TAG_NAME }}
+            DEBIAN_VER=${{ matrix.debian }}-slim
+
+      # Build and Push component image to DOCKER_TAG_DEFAULT_DEBIAN, only applies when building the default Debian version (bookworm)
+      # It is fine to build a second time here when "matrix.debian == 'bookworm'" as we have cached the first build already
+      - name: Build and push on tags without Debian extension
+        if: startsWith(github.ref, 'refs/tags/') && matrix.debian == 'bookworm'
+        uses: docker/build-push-action@v5
+        with:
+          context: ${{ env.DOCKER_CTX }}
+          push: true
+          tags: ${{ env.DOCKER_TAG_DEFAULT_DEBIAN }}
+          build-args: |
+            VT_BASE_VER=${{ env.TAG_NAME }}
+            DEBIAN_VER=${{ matrix.debian }}-slim

.github/workflows/docker_build_lite.yml

@@ -63,7 +63,7 @@ jobs:
             echo "DOCKER_TAG=vitess/lite:${TAG_NAME}-${{ matrix.branch }}" >> $GITHUB_ENV
           fi
 
-      - name: Build and push on main
+      - name: Build and push on tags
         if: startsWith(github.ref, 'refs/tags/')
         uses: docker/build-push-action@v5
         with:

doc/internal/release/how-to-release.md

@@ -87,6 +87,11 @@ That includes:
   >  - There are two scripts in the website repository in `./tools/{ga|rc}_release.sh`, use them to update the website documentation. The scripts automate:
   >  - For an RC, we need to create a new entry in the sidebar which represents the next version on `main` and mark the version we are releasing as RC.
   >  - For a GA, we need to mark the version we are releasing as "Stable" and the next one as "Development".
+- **Create a new GitHub Milestone**
+  > - Our GitHub Milestones is a good representation of all our ongoing development cycles. We have a Milestone for `main` and for all release branches. 
+  > - After doing Code Freeze, we can create a new GitHub Milestone that matches the next development cycle.
+  >  - **If we release a major version (v18.0.0-rc1):** we must create a `v19.0.0` Milestone.
+  >  - **If we release a patch release (v17.0.3):** we must create a `v17.0.4` Milestone.
 
 -----
 
@@ -122,10 +127,12 @@ On the release day, there are several things to do:
   > - The benchmarks need to complete before announcing the blog posts or before they get cross-posted.
 - **Go back to dev mode on the release branch.**
   > - The version constants across the codebase must be updated to `SNAPSHOT`.
-- **Build k8s Docker images and publish them.**
-  > - The docker image for `base`, `lite`, etc are built automatically by DockerHub. The k8s images however are dependent on these images and are required to be built manually.
-  > - These images should be built after the `base` image has been built and available on DockerHub.
-  > - To build and publish these images, checkout the new release tag that was just created and run `./release.sh` from the directory `./docker`.
+- **Ensure the k8s images are available on DockerHub.**
+- **Close the current GitHub Milestone**
+  > - Once we are done releasing the current version, we must close its corresponding GitHub Milestone as the development cycle for it is over.
+  > - **This does not apply if we are releasing an RC release.** For instance, if we are releasing `v18.0.0-rc1` we want to keep the `v18.0.0` milestone opened as development is not fully done for `v18.0.0`. 
+  > - For instance, if we release `v18.0.1`, we must close the `v18.0.1` Milestone as the development cycle for `v18.0.1` is over.
+  > - When closing a Milestone, we need to look through all the PRs/Issues in that Milestone and re-assign a newer Milestone to them.
 
 -----
 
@@ -134,6 +141,9 @@ On the release day, there are several things to do:
 Once the release is over, we need to announce it on both Slack and Twitter. We also want to make sure the blog post was cross-posted, if applicable.
 We need to verify that _arewefastyet_ has finished the benchmark too.
 
+Moreover, once the roadmap discussions are over for the next release, we need to update the roadmap presented in the Vitess website (https://vitess.io/docs/resources/roadmap/).
+We must remove everything that is now done in this release and add new items based on the discussions.
+
 -----
 
 ## How to prepare the release of Vitess

tools/back_to_dev_mode.sh

@@ -48,7 +48,6 @@ fi
 function doBackToDevMode () {
   # Preparing the "dev mode" commit
   updateJava $DEV_VERSION
-  updateDockerReleaseScript $DEV_VERSION
   updateVersionGo $DEV_VERSION
 
   git add --all

tools/create_release.sh

@@ -61,7 +61,6 @@ function createRelease () {
   # Preparing the release commit
   updateVitessExamples $RELEASE_VERSION $VTOP_VERSION
   updateJava $RELEASE_VERSION
-  updateDockerReleaseScript $RELEASE_VERSION
   updateVersionGo $RELEASE_VERSION
 
   ## Create the commit for this release and tag it