Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(ci): mount package directory at image build time #13350

Merged
merged 1 commit into from
Jul 10, 2024

Conversation

flrgh
Copy link
Contributor

@flrgh flrgh commented Jul 8, 2024

This updates our release workflow to use docker's build-time bind mount support to supply the package file(s) instead of copying into the image.

The idea behind this is that mounting the package removes a large COPY layer and reduces the image size.

The directory that is mounted is checksum-ed by docker as part of the build context, so build cache is automatically invalidated when the package file changes.

before

$ docker image ls kong/kong-gateway-dev:master \
> --format 'table {{.Repository}}\t{{.Tag}}\t{{.Size}}'
REPOSITORY              TAG       SIZE
kong/kong-gateway-dev   master    538MB
$ docker image history kong/kong-gateway-dev:master \
> --format 'table {{.CreatedAt}}\t{{.CreatedBy}}\t{{.Size}}'
CREATED AT                  CREATED BY                                      SIZE
2024-07-08T01:52:13-07:00   CMD ["kong" "docker-start"]                     0B
2024-07-08T01:52:13-07:00   HEALTHCHECK &{["CMD-SHELL" "kong-health"] "1…   0B
2024-07-08T01:52:13-07:00   STOPSIGNAL SIGQUIT                              0B
2024-07-08T01:52:13-07:00   EXPOSE map[8000/tcp:{} 8001/tcp:{} 8002/tcp:…   0B
2024-07-08T01:52:13-07:00   ENTRYPOINT ["/entrypoint.sh"]                   0B
2024-07-08T01:52:13-07:00   USER kong                                       0B
2024-07-08T01:52:13-07:00   COPY build/dockerfiles/entrypoint.sh /entryp…   2.22kB
2024-07-08T01:52:13-07:00   RUN |6 KONG_VERSION=3.8.0.0 KONG_PREFIX=/usr…   379MB
2024-07-08T01:51:55-07:00   COPY bazel-bin/pkg/kong.amd64.deb /tmp/kong.…   81.3MB
2024-07-08T01:51:55-07:00   ARG KONG_ARTIFACT_PATH=bazel-bin/pkg/           0B
2024-07-08T01:51:55-07:00   ARG KONG_ARTIFACT=kong.amd64.deb                0B
2024-07-08T01:51:55-07:00   ARG TARGETARCH=amd64                            0B
2024-07-08T01:51:55-07:00   ARG EE_PORTS=8002 8445 8003 8446 8004 8447      0B
2024-07-08T01:51:55-07:00   ENV KONG_PREFIX=/usr/local/kong                 0B
2024-07-08T01:51:55-07:00   ARG KONG_PREFIX=/usr/local/kong                 0B
2024-07-08T01:51:55-07:00   ENV KONG_VERSION=3.8.0.0                        0B
2024-07-08T01:51:55-07:00   ARG KONG_VERSION=3.8.0.0                        0B
2024-07-08T01:51:55-07:00   LABEL maintainer=Kong Docker Maintainers <do…   0B
2024-06-27T13:10:12-07:00   /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B
2024-06-27T13:10:12-07:00   /bin/sh -c #(nop) ADD file:d5da92199726e42da…   77.9MB
2024-06-27T13:10:10-07:00   /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B
2024-06-27T13:10:10-07:00   /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B
2024-06-27T13:10:10-07:00   /bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH     0B
2024-06-27T13:10:10-07:00   /bin/sh -c #(nop)  ARG RELEASE                  0B

after

$ docker image ls kong/kong-gateway-dev:b5195bbdb9a613a81359b4c1498220738af99a37 \
> --format 'table {{.Repository}}\t{{.Tag}}\t{{.Size}}'
REPOSITORY              TAG                                        SIZE
kong/kong-gateway-dev   b5195bbdb9a613a81359b4c1498220738af99a37   457MB
$ docker image history kong/kong-gateway-dev:b5195bbdb9a613a81359b4c1498220738af99a37 \
> --format 'table {{.CreatedAt}}\t{{.CreatedBy}}\t{{.Size}}'
CREATED AT                  CREATED BY                                      SIZE
2024-07-08T11:17:32-07:00   CMD ["kong" "docker-start"]                     0B
2024-07-08T11:17:32-07:00   HEALTHCHECK &{["CMD-SHELL" "kong-health"] "1…   0B
2024-07-08T11:17:32-07:00   STOPSIGNAL SIGQUIT                              0B
2024-07-08T11:17:32-07:00   EXPOSE map[8000/tcp:{} 8001/tcp:{} 8002/tcp:…   0B
2024-07-08T11:17:32-07:00   ENTRYPOINT ["/entrypoint.sh"]                   0B
2024-07-08T11:17:32-07:00   USER kong                                       0B
2024-07-08T11:17:32-07:00   COPY build/dockerfiles/entrypoint.sh /entryp…   2.22kB
2024-07-08T11:17:32-07:00   RUN |6 KONG_VERSION=3.8.0.0 KONG_PREFIX=/usr…   379MB
2024-07-08T11:17:32-07:00   ARG KONG_ARTIFACT_PATH=bazel-bin/pkg            0B
2024-07-08T11:17:32-07:00   ARG KONG_ARTIFACT=kong.amd64.deb                0B
2024-07-08T11:17:32-07:00   ARG TARGETARCH=amd64                            0B
2024-07-08T11:17:32-07:00   ARG EE_PORTS=8002 8445 8003 8446 8004 8447      0B
2024-07-08T11:17:32-07:00   ENV KONG_PREFIX=/usr/local/kong                 0B
2024-07-08T11:17:32-07:00   ARG KONG_PREFIX=/usr/local/kong                 0B
2024-07-08T11:17:32-07:00   ENV KONG_VERSION=3.8.0.0                        0B
2024-07-08T11:17:32-07:00   ARG KONG_VERSION=3.8.0.0                        0B
2024-07-08T11:17:32-07:00   LABEL maintainer=Kong Docker Maintainers <do…   0B
2024-06-27T13:10:12-07:00   /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B
2024-06-27T13:10:12-07:00   /bin/sh -c #(nop) ADD file:d5da92199726e42da…   77.9MB
2024-06-27T13:10:10-07:00   /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B
2024-06-27T13:10:10-07:00   /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B
2024-06-27T13:10:10-07:00   /bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH     0B
2024-06-27T13:10:10-07:00   /bin/sh -c #(nop)  ARG RELEASE                  0B

@github-actions github-actions bot added chore Not part of the core functionality of kong, but still needed build/bazel cherry-pick kong-ee schedule this PR for cherry-picking to kong/kong-ee labels Jul 8, 2024
@flrgh
Copy link
Contributor Author

flrgh commented Jul 8, 2024

re: amazonlinux-2 build failure, that's broken in master at the moment (see: https://github.com/Kong/kong-ee/pull/9205)

@flrgh
Copy link
Contributor Author

flrgh commented Jul 8, 2024

Full matrix build log is here. The failure messages on this PR are due to branch protection rules.

@flrgh flrgh force-pushed the ci/docker-mount-package-source branch from 322632f to b9fb86f Compare July 9, 2024 16:15
This updates our release workflow to use docker's build-time
[bind mount](https://docs.docker.com/reference/dockerfile/#run---mounttypebind)
support to supply the package file(s) instead of copying into the image.

The idea behind this is that mounting the package removes a large `COPY`
layer and reduces the image size.

The directory that is mounted is checksum-ed by docker as part of the build
context, so build cache is automatically invalidated when the package file
changes.

```
$ docker image ls kong/kong-gateway-dev:master \
> --format 'table {{.Repository}}\t{{.Tag}}\t{{.Size}}'
REPOSITORY              TAG       SIZE
kong/kong-gateway-dev   master    538MB
```

```
$ docker image history kong/kong-gateway-dev:master \
> --format 'table {{.CreatedAt}}\t{{.CreatedBy}}\t{{.Size}}'
CREATED AT                  CREATED BY                                      SIZE
2024-07-08T01:52:13-07:00   CMD ["kong" "docker-start"]                     0B
2024-07-08T01:52:13-07:00   HEALTHCHECK &{["CMD-SHELL" "kong-health"] "1…   0B
2024-07-08T01:52:13-07:00   STOPSIGNAL SIGQUIT                              0B
2024-07-08T01:52:13-07:00   EXPOSE map[8000/tcp:{} 8001/tcp:{} 8002/tcp:…   0B
2024-07-08T01:52:13-07:00   ENTRYPOINT ["/entrypoint.sh"]                   0B
2024-07-08T01:52:13-07:00   USER kong                                       0B
2024-07-08T01:52:13-07:00   COPY build/dockerfiles/entrypoint.sh /entryp…   2.22kB
2024-07-08T01:52:13-07:00   RUN |6 KONG_VERSION=3.8.0.0 KONG_PREFIX=/usr…   379MB
2024-07-08T01:51:55-07:00   COPY bazel-bin/pkg/kong.amd64.deb /tmp/kong.…   81.3MB
2024-07-08T01:51:55-07:00   ARG KONG_ARTIFACT_PATH=bazel-bin/pkg/           0B
2024-07-08T01:51:55-07:00   ARG KONG_ARTIFACT=kong.amd64.deb                0B
2024-07-08T01:51:55-07:00   ARG TARGETARCH=amd64                            0B
2024-07-08T01:51:55-07:00   ARG EE_PORTS=8002 8445 8003 8446 8004 8447      0B
2024-07-08T01:51:55-07:00   ENV KONG_PREFIX=/usr/local/kong                 0B
2024-07-08T01:51:55-07:00   ARG KONG_PREFIX=/usr/local/kong                 0B
2024-07-08T01:51:55-07:00   ENV KONG_VERSION=3.8.0.0                        0B
2024-07-08T01:51:55-07:00   ARG KONG_VERSION=3.8.0.0                        0B
2024-07-08T01:51:55-07:00   LABEL maintainer=Kong Docker Maintainers <do…   0B
2024-06-27T13:10:12-07:00   /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B
2024-06-27T13:10:12-07:00   /bin/sh -c #(nop) ADD file:d5da92199726e42da…   77.9MB
2024-06-27T13:10:10-07:00   /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B
2024-06-27T13:10:10-07:00   /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B
2024-06-27T13:10:10-07:00   /bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH     0B
2024-06-27T13:10:10-07:00   /bin/sh -c #(nop)  ARG RELEASE                  0B
```

```
$ docker image ls kong/kong-gateway-dev:b5195bbdb9a613a81359b4c1498220738af99a37 \
> --format 'table {{.Repository}}\t{{.Tag}}\t{{.Size}}'
REPOSITORY              TAG                                        SIZE
kong/kong-gateway-dev   b5195bbdb9a613a81359b4c1498220738af99a37   457MB
```

```
$ docker image history kong/kong-gateway-dev:b5195bbdb9a613a81359b4c1498220738af99a37 \
> --format 'table {{.CreatedAt}}\t{{.CreatedBy}}\t{{.Size}}'
CREATED AT                  CREATED BY                                      SIZE
2024-07-08T11:17:32-07:00   CMD ["kong" "docker-start"]                     0B
2024-07-08T11:17:32-07:00   HEALTHCHECK &{["CMD-SHELL" "kong-health"] "1…   0B
2024-07-08T11:17:32-07:00   STOPSIGNAL SIGQUIT                              0B
2024-07-08T11:17:32-07:00   EXPOSE map[8000/tcp:{} 8001/tcp:{} 8002/tcp:…   0B
2024-07-08T11:17:32-07:00   ENTRYPOINT ["/entrypoint.sh"]                   0B
2024-07-08T11:17:32-07:00   USER kong                                       0B
2024-07-08T11:17:32-07:00   COPY build/dockerfiles/entrypoint.sh /entryp…   2.22kB
2024-07-08T11:17:32-07:00   RUN |6 KONG_VERSION=3.8.0.0 KONG_PREFIX=/usr…   379MB
2024-07-08T11:17:32-07:00   ARG KONG_ARTIFACT_PATH=bazel-bin/pkg            0B
2024-07-08T11:17:32-07:00   ARG KONG_ARTIFACT=kong.amd64.deb                0B
2024-07-08T11:17:32-07:00   ARG TARGETARCH=amd64                            0B
2024-07-08T11:17:32-07:00   ARG EE_PORTS=8002 8445 8003 8446 8004 8447      0B
2024-07-08T11:17:32-07:00   ENV KONG_PREFIX=/usr/local/kong                 0B
2024-07-08T11:17:32-07:00   ARG KONG_PREFIX=/usr/local/kong                 0B
2024-07-08T11:17:32-07:00   ENV KONG_VERSION=3.8.0.0                        0B
2024-07-08T11:17:32-07:00   ARG KONG_VERSION=3.8.0.0                        0B
2024-07-08T11:17:32-07:00   LABEL maintainer=Kong Docker Maintainers <do…   0B
2024-06-27T13:10:12-07:00   /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B
2024-06-27T13:10:12-07:00   /bin/sh -c #(nop) ADD file:d5da92199726e42da…   77.9MB
2024-06-27T13:10:10-07:00   /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B
2024-06-27T13:10:10-07:00   /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B
2024-06-27T13:10:10-07:00   /bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH     0B
2024-06-27T13:10:10-07:00   /bin/sh -c #(nop)  ARG RELEASE                  0B
```
@fffonion
Copy link
Contributor

fffonion commented Jul 9, 2024

The approach LGTM, but we may need to add make KONG_ARTIFACT_PATH a sane default (or does source=<empty> also work?)

@flrgh
Copy link
Contributor Author

flrgh commented Jul 9, 2024

The approach LGTM, but we may need to add make KONG_ARTIFACT_PATH a sane default (or does source=<empty> also work?)

@fffonion if not set, the bind mount source will be the build context root, which seems like a sensible default.

We might also just consider mounting the whole build context at some well-known, hardcoded path so that the --mount args are more simple. Let me know if this looks any cleaner to you:

ARG KONG_ARTIFACT_PATH=artifacts # or bazel-bin/pkg

RUN --mount=type=bind,target=/tmp/build \
    [...]
    yum install -y /tmp/build/${KONG_ARTIFACT_PATH}/kong.${RPM_PLATFORM}.${TARGETARCH}.rpm \
    [...]

@fffonion
Copy link
Contributor

If empty default just works, then I'm okay 👍

@fffonion fffonion merged commit 4664b90 into master Jul 10, 2024
24 checks passed
@fffonion fffonion deleted the ci/docker-mount-package-source branch July 10, 2024 03:40
@team-gateway-bot
Copy link
Collaborator

Cherry-pick failed for master, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git remote add upstream https://github.com/kong/kong-ee
git fetch upstream master
git worktree add -d .worktree/cherry-pick-13350-to-master-to-upstream upstream/master
cd .worktree/cherry-pick-13350-to-master-to-upstream
git checkout -b cherry-pick-13350-to-master-to-upstream
ancref=$(git merge-base f9574877485bc93d4d4b28c9ae009a1b90df82ea b9fb86f5652252aa4374bd84f2116661e3523be9)
git cherry-pick -x $ancref..b9fb86f5652252aa4374bd84f2116661e3523be9

@github-actions github-actions bot added the incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention label Jul 10, 2024
@fffonion
Copy link
Contributor

Please take care of EE master cherrypick manually : )

@locao locao removed the incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention label Jul 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build/bazel cherry-pick kong-ee schedule this PR for cherry-picking to kong/kong-ee chore Not part of the core functionality of kong, but still needed size/S skip-changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants