Replace kaniko with buildkit #503

munishchouhan · 2024-05-16T15:23:39Z

This PR will replace the builder image with Buildkit.

Solves #478 and #287.

munishchouhan · 2024-05-16T15:24:18Z

build process is getting stuck at

2024-05-16 17:18:32 time="2024-05-16T15:18:32Z" level=debug msg="auto snapshotter: overlayfs is not available for /var/lib/buildkit, trying fuse-overlayfs: failed to mount overlay: operation not permitted"
2024-05-16 17:18:32 time="2024-05-16T15:18:32Z" level=debug msg="auto snapshotter: fuse-overlayfs is not available for /var/lib/buildkit, falling back to native: fuse-overlayfs not installed: exec: \"fuse-overlayfs\": executable file not found in $PATH"
2024-05-16 17:18:32 time="2024-05-16T15:18:32Z" level=info msg="auto snapshotter: using native"
2024-05-16 17:18:32 time="2024-05-16T15:18:32Z" level=warning msg="using host network as the default"
2024-05-16 17:18:32 time="2024-05-16T15:18:32Z" level=warning msg="failed to prepare cgroup controllers: mkdir /sys/fs/cgroup/init: read-only file system"
2024-05-16 17:18:32 time="2024-05-16T15:18:32Z" level=info msg="found worker \"1txxm5558fckd9ky82z6cfz1z\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:fbef5e0b3043 org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:native], platforms=[linux/arm64 linux/amd64 linux/amd64/v2 linux/riscv64 linux/ppc64le linux/s390x linux/386 linux/mips64le linux/mips64 linux/arm/v7 linux/arm/v6]"
2024-05-16 17:18:32 time="2024-05-16T15:18:32Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
2024-05-16 17:18:32 time="2024-05-16T15:18:32Z" level=info msg="found 1 workers, default=\"1txxm5558fckd9ky82z6cfz1z\""
2024-05-16 17:18:32 time="2024-05-16T15:18:32Z" level=warning msg="currently, only the default worker can be used."
2024-05-16 17:18:32 time="2024-05-16T15:18:32Z" level=info msg="running server on /run/buildkit/buildkitd.sock"

Working on fixing it

pditommaso · 2024-05-16T15:24:58Z

Oh, it was fast

munishchouhan · 2024-05-16T15:37:44Z

@arnaualcazar as you already tried buildkit, plase take a look on this PR and fi you get some idea about why its getting stuck, pleas share

pditommaso · 2024-05-16T16:00:48Z

Have you tried running it just using the docker run cli?

arnaualcazar · 2024-05-16T16:05:51Z

I tested the buildkit with service deployment on a kubernetes cluster: https://github.com/moby/buildkit/blob/master/examples/kubernetes/README.md#deployment--service
After deploying it, you can configure docker build to connect to the buildkit pod to start the build.

pditommaso · 2024-05-16T16:11:20Z

Likely we'll go with rootless job approach

munishchouhan · 2024-05-16T18:00:45Z

Have you tried running it just using the docker run cli?

yes, in local i tried with both k8s and docker

munishchouhan · 2024-05-16T18:15:09Z

with docker run abd debug enabled


/bin/zsh /Users/munish.chouhan/main_ground/wave/master/wave/build-workspace/4170bcb9964acc85_1/docker.sh
munish.chouhan@Munishs-MacBook-Pro 4170bcb9964acc85_1 % /bin/zsh /Users/munish.chouhan/main_ground/wave/master/wave/build-workspace/4170bcb9964acc85_1/docker.sh
time="2024-05-16T18:13:35Z" level=info msg="auto snapshotter: using overlayfs"
time="2024-05-16T18:13:35Z" level=debug msg="running in rootless mode"
time="2024-05-16T18:13:35Z" level=info msg="found worker \"2y6v6ew7j1d5252qdtesy5io5\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:6caea173439a org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/arm64 linux/amd64 linux/amd64/v2 linux/riscv64 linux/ppc64le linux/s390x linux/386 linux/mips64le linux/mips64 linux/arm/v7 linux/arm/v6]"
time="2024-05-16T18:13:35Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
time="2024-05-16T18:13:35Z" level=info msg="found 1 workers, default=\"2y6v6ew7j1d5252qdtesy5io5\""
time="2024-05-16T18:13:35Z" level=warning msg="currently, only the default worker can be used."
time="2024-05-16T18:13:35Z" level=info msg="running server on /run/user/1000/buildkit/buildkitd.sock"
time="2024-05-16T18:13:36Z" level=debug msg="remove snapshot" key=y3arilf1bnoe50kqfcyuswcvu snapshotter=overlayfs
time="2024-05-16T18:13:36Z" level=debug msg="schedule snapshotter cleanup" snapshotter=overlayfs
time="2024-05-16T18:13:36Z" level=debug msg="removed snapshot" key=buildkit/1/y3arilf1bnoe50kqfcyuswcvu snapshotter=overlayfs
time="2024-05-16T18:13:36Z" level=debug msg="snapshot garbage collected" d=4.951125ms snapshotter=overlayfs

munishchouhan · 2024-05-16T20:55:50Z

when i run the same command inside container it works:

munishchouhan · 2024-05-16T21:57:25Z

finally wokring with docker, now will make it work with k8s

munishchouhan · 2024-05-17T14:45:53Z

Now, its working with k8s, but caching is not working in both cases.
I will wokr on that now

munishchouhan · 2024-05-22T14:41:50Z

I am checking with buildkit in slack about how to use repository as cache in it

munishchouhan · 2024-05-23T08:39:58Z

cache is working in k8s setup, I will work on making it work on docker

Signed-off-by: Paolo Di Tommaso <[email protected]>

pditommaso · 2024-06-07T13:30:47Z

Almost there, but something with the oci-mediatypes flag. By default it's true (that's ok) but the resulting container looks like using docker mimetype. For example using

wave -o yaml  --inspect -i community.wave.stage-seqera.io/library/bwa:59eaca2f73380b96

it returns

  manifest:
    annotations: null
    config:
      annotations: null
      digest: sha256:46a700685da518f5dce3ca9667581ee1067de128f2df8aac7b7399b50ec9252d
      mediaType: application/vnd.docker.container.image.v1+json

Instead images built via Kaniko it reports

wave --inspect -i community.wave.seqera.io/library/bwa:0.7.18--324359fbc6e00dba

  manifest:
    annotations: null
    config:
      annotations: null
      digest: sha256:ec339118cfa9667a37d09ad2dfb024028a475d177e33d91fb134ff41803ebb30
      mediaType: application/vnd.oci.image.config.v1+json
      size: 7370

Note application/vnd.oci.image.config.v1+json

pditommaso · 2024-06-07T13:36:57Z

Command looks correct, tho

    Command:
      buildctl-daemonless.sh
    Args:
      build
      --frontend
      dockerfile.v0
      --local
      dockerfile=/efs/wave/build/b5728008c188ab65_1
      --opt
      filename=Containerfile
      --local
      context=/efs/wave/build/b5728008c188ab65_1/context
      --output
      type=image,name=community.wave.stage-seqera.io/library/bwa0.7.18:b5728008c188ab65,push=true
      --opt
      platform=linux/amd64
      --export-cache
      type=registry,image-manifest=true,ref=195996028523.dkr.ecr.eu-west-1.amazonaws.com/wave/build/cache:b5728008c188ab65,mode=max,ignore-error=true,oci-mediatypes=true,compression=gzip,force-compression=false
      --import-cache
      type=registry,ref=195996028523.dkr.ecr.eu-west-1.amazonaws.com/wave/build/cache:b5728008c188ab65

pditommaso · 2024-06-07T13:51:27Z

Maybe the --output type should be oci instead of image

https://github.com/moby/buildkit?tab=readme-ov-file#oci-tarball

Signed-off-by: Paolo Di Tommaso <[email protected]>

munishchouhan · 2024-06-10T07:33:39Z

@pditommaso oci mediatype has been added

"imageName": "wt/abb516651b63/hrma017/dev",
    "manifest": {
      "config": {
        "digest": "sha256:2cc21e44701cfd8bc73a5571f424d5d5036dacb619f3c7688ca5fb9c21176f5e",
        "mediaType": "application/vnd.oci.image.config.v1+json",
        "size": 7801

pditommaso · 2024-06-10T07:43:18Z

Nice. What about using for ociMediatypes config setting to parametrise it as well ?

munishchouhan · 2024-06-10T07:55:11Z

Nice. What about using for ociMediatypes config setting to parametrise it as well ?

yes, it is using the config

wave/src/main/groovy/io/seqera/wave/service/builder/BuildStrategy.groovy

Line 70 in 1ae31a6

    
           << "type=image,name=$req.targetImage,push=true,oci-mediatypes=${buildConfig.ociMediatypes}".toString()

pditommaso · 2024-06-10T08:11:19Z

ok

Signed-off-by: Paolo Di Tommaso <[email protected]>

munishchouhan · 2024-06-10T09:47:18Z

@pditommaso there was an issue with the annotations, but I have fixed it. Please release and test again

pditommaso · 2024-06-10T09:49:20Z

Nice

Signed-off-by: Paolo Di Tommaso <[email protected]>

VERSION

configuration.md

initial commit

bfbfbe7

munishchouhan linked an issue May 16, 2024 that may be closed by this pull request

Use buildkit for container image builds #478

Closed

munishchouhan marked this pull request as draft May 16, 2024 15:34

Merge branch 'master' into 478-use-buildkit-for-container-image-builds

93195b3

added buildkit image

3a5ed17

use rootless image

e574412

munishchouhan added 2 commits May 16, 2024 23:45

added --privileged

f7a794c

removed --privileged added entrypoint

b1a2759

added build job

671d985

munishchouhan self-assigned this May 17, 2024

munishchouhan and others added 4 commits May 20, 2024 13:00

refactored

f6ad952

refactored

8622295

removing unwanted imports

db55e88

Merge branch 'master' into 478-use-buildkit-for-container-image-builds

a7311ef

munishchouhan added 3 commits May 23, 2024 10:40

fixed caching

b10c0f7

fixed docker cache

414b29f

fixed tests

34d4619

munishchouhan and others added 3 commits June 7, 2024 14:16

formatted [ci skip]

bc0993f

Minor change

36d7898

Signed-off-by: Paolo Di Tommaso <[email protected]>

[release] bump version 1.8.0-B2

a55c2ba

Signed-off-by: Paolo Di Tommaso <[email protected]>

pditommaso and others added 4 commits June 7, 2024 16:05

Minor changes [ci skip]

fb239a4

Signed-off-by: Paolo Di Tommaso <[email protected]>

Merge branch 'master' into 478-use-buildkit-for-container-image-builds

b467a66

added mediatype=oci

d36e628

refactored

1ae31a6

pditommaso and others added 4 commits June 10, 2024 10:12

[release] bump version 1.8.0-B3

5051f53

Signed-off-by: Paolo Di Tommaso <[email protected]>

Minor change [release]

35f316d

Signed-off-by: Paolo Di Tommaso <[email protected]>

added docs

a7ff106

refactored and fixed error

b574854

[release] bump 1.8.0-B4

172f872

Signed-off-by: Paolo Di Tommaso <[email protected]>

pditommaso reviewed Jun 10, 2024

View reviewed changes

VERSION Outdated Show resolved Hide resolved

Update VERSION

2ca0dfb

pditommaso reviewed Jun 10, 2024

View reviewed changes

VERSION Outdated Show resolved Hide resolved

Update VERSION

dc33a30

pditommaso reviewed Jun 10, 2024

View reviewed changes

configuration.md Outdated Show resolved Hide resolved

Update configuration.md [ci skip]

fe40427

pditommaso approved these changes Jun 10, 2024

View reviewed changes

pditommaso merged commit a26f5df into master Jun 10, 2024
1 check passed

pditommaso deleted the 478-use-buildkit-for-container-image-builds branch June 10, 2024 10:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace kaniko with buildkit #503

Replace kaniko with buildkit #503

munishchouhan commented May 16, 2024 •

edited by pditommaso

Loading

munishchouhan commented May 16, 2024

pditommaso commented May 16, 2024

munishchouhan commented May 16, 2024

pditommaso commented May 16, 2024

arnaualcazar commented May 16, 2024

pditommaso commented May 16, 2024

munishchouhan commented May 16, 2024

munishchouhan commented May 16, 2024 •

edited

Loading

munishchouhan commented May 16, 2024

munishchouhan commented May 16, 2024

munishchouhan commented May 17, 2024

munishchouhan commented May 22, 2024

munishchouhan commented May 23, 2024

pditommaso commented Jun 7, 2024

pditommaso commented Jun 7, 2024

pditommaso commented Jun 7, 2024

munishchouhan commented Jun 10, 2024

pditommaso commented Jun 10, 2024

munishchouhan commented Jun 10, 2024

pditommaso commented Jun 10, 2024

munishchouhan commented Jun 10, 2024

pditommaso commented Jun 10, 2024

Replace kaniko with buildkit #503

Replace kaniko with buildkit #503

Conversation

munishchouhan commented May 16, 2024 • edited by pditommaso Loading

munishchouhan commented May 16, 2024

pditommaso commented May 16, 2024

munishchouhan commented May 16, 2024

pditommaso commented May 16, 2024

arnaualcazar commented May 16, 2024

pditommaso commented May 16, 2024

munishchouhan commented May 16, 2024

munishchouhan commented May 16, 2024 • edited Loading

munishchouhan commented May 16, 2024

munishchouhan commented May 16, 2024

munishchouhan commented May 17, 2024

munishchouhan commented May 22, 2024

munishchouhan commented May 23, 2024

pditommaso commented Jun 7, 2024

pditommaso commented Jun 7, 2024

pditommaso commented Jun 7, 2024

munishchouhan commented Jun 10, 2024

pditommaso commented Jun 10, 2024

munishchouhan commented Jun 10, 2024

pditommaso commented Jun 10, 2024

munishchouhan commented Jun 10, 2024

pditommaso commented Jun 10, 2024

munishchouhan commented May 16, 2024 •

edited by pditommaso

Loading

munishchouhan commented May 16, 2024 •

edited

Loading