Remove the bootstrap dependency on the Docker images we ship

[frouioui]

Description

This PR removes the dependency that the vitess/lite and vitess/vttestsever Docker images had on the vitess/bootstrap image. We do not need to build these two images on top of bootstrap as we end-up installing all the required dependencies in the final stage of the build anyway.

The docker/local folder is also removed as a follow up of #16335.

With this PR the whole dependency chain of our images is simplified and any future work on the docker images will be easier. This is also the first step towards simplifying and potentially removing the bootstrap image, the only use right now is for our CI test (in ./test.go and ./docker/test).

I reworked our two github action workflows that enabled us to build and push docker images (vttestserver, lite, and all the binaries). I merged the two workflows into a single one, which will make it easier to know if all images we ship are building correctly. Moreover, for every Pull Requests we will now try to build the Docker Images without pushing them, that way we know if we break a docker build. We often realized this too late in the past, while we were doing a release. Here is what the new workflow looks like:

Note

The new workflow we are adding to CI spawns about 33 jobs that will show up on the workflows list of every Pull Request. Despite having a lot of new jobs, the time needed to complete all workflows should not be impacted by much, we need about 5-8 minutes to complete all 33 jobs of the Build Docker Images workflow.

This workflow should be marked as required once the PR is merged.

Finally, some changes are needed on the vitess-releaser to use the new workflow to ensure GHA has built all the Docker images during the release. (https://github.com/vitessio/vitess-releaser/blob/main/go/releaser/release/docker.go)

This is marked as a draft, we will merge it when the following PRs are merged:

• Fix the install dependencies script in Docker #16340
• Remove docker_local from Makefile #16335
• Remove local-docker-guide from v21 website#1781

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

• Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

• Apply the release notes (needs details) label if users need to know about this change.
• New features should be documented.
• There should be some code comments as to why things are implemented the way they are.
• There should be a comment at the top of each new or modified test to explain what the test does.

New flags

• Is this flag really necessary?
• Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
• vtctl command output order should be stable and awk-able.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 3 lines in your changes missing coverage. Please review.

Project coverage is 68.71%. Comparing base (0db3577) to head (c5ae3ee).

Files	Patch %	Lines
go/tools/go-upgrade/go-upgrade.go	0.00%	3 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #16339   +/-   ##
=======================================
  Coverage   68.71%   68.71%           
=======================================
  Files        1547     1547           
  Lines      198287   198287           
=======================================
+ Hits       136243   136249    +6     
+ Misses      62044    62038    -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[frouioui]

We only have mysql80, since a while.

[frouioui]

I have added these --platform=linux/amd64 directives here and in other places to ensure that the base image we pull is based on linux/amd64, that way, we can build the docker images from any machine. For instance, if we remove this directive and try to build from a Mac, the install dependencies step will fail as there are no Percona package for the Mac's architecture.

[frouioui]

I believe this was wrong to begin with, leading to the error: bash: vtgate: command not found:

$ docker run -it --user=vitess vitess/lite:v20.0.0 bash
vitess@0d7c6ece9106:/$ vtgate
bash: vtgate: command not found
vitess@0d7c6ece9106:/$

With this fix, the vitess binaries are directly in the PATH and people can execute them without specifying the full path to the binary.

[frouioui]

With the fix:

$ docker run -it --user="vitess" docker.io/vitess/lite-test:latest bash
vitess@6a3782014166:/$ vtgate --version 
vtgate version Version: 21.0.0-SNAPSHOT (Git revision 694a0cf8d9e51aa4b130ad1656c4009427eb0e3d branch 'remove-bootstrap-from-lite') built on Thu Jul  4 15:55:37 UTC 2024 by vitess@buildkitsandbox using go1.22.5 linux/amd64
vitess@6a3782014166:/$

[frouioui]

We no longer update the bootstrap version in these path.

[frouioui]

We now have to bump the golang version in more places since we base more of our images on the official go docker image.

[frouioui]

This job will be executed before everyone else. Then build_and_push_vttestserver and build_and_push_lite are executed concurrently. Finally, once build_and_push_lite is done, build_and_push_components will begin building all the binaries concurrently.

[mattlord]

I had a few questions, but perhaps I'm missing or misunderstanding somethings?

[mattlord]

github.repository == 'vitessio/vitess' seems redundant here? And I think it's better to keep the criteria for when to skip within the skip_push step where we already check for this, if possible.

[frouioui]

Fixed via e99c1a7

[mattlord]

Why would we need to login to docker hub if we're NOT pushing?

[frouioui]

That's a really good point, the naming of skip_push was misleading, I renamed it to push via e99c1a7

[mattlord]

Why not explicitly check: if: github.ref == 'refs/heads/main'?

[frouioui]

We could check for this, but that will only build if the PR is based on main. Once we create the release-21.0 branch it will no longer work on that branch.