Suggestion: Download a snapshot release of the lifecycle #1305

cmoulliard · 2021-10-07T10:01:30Z

Description

When we would like to use a different lifecycle release from the list of the published releases - see: https://github.com/buildpacks/lifecycle/releases), then we can set the property version
within the builder.toml file to include it within the builder image :

[lifecycle]
version = "0.12.0"

Such a version will then generate a request to download the following artifact "Downloading from https://github.com/buildpacks/lifecycle/releases/download/v0.12.0/lifecycle-v0.12.0+linux.x86-64.tgz"

How would it be possible to use a lifecycle version which is not yet released but where image has been created from the github commit id ?

Example for the commit: 919b8ad

image: https://hub.docker.com/layers/buildpacksio/lifecycle/919b8ad-linux-arm6
sha commit:  https://github.com/buildpacks/lifecycle/commit/919b8add4cb4af630722199a94d4b374f2c5f1eb

I think that we should investigate to propose a solution as by example this is is something that we can do using pack build using this parameter pack build ... --lifecycle-image buildpacksio/lifecycle:919b8ad-linux-arm64 but not if we execute the lifecycle creator part of kubernetes pod

      - name: export
         image: registry.local:5000/redhat-builder/quarkus "LIFECYCLE TO BE USED IS PACKAGED HERE AND CANNOT BE CHANGED
         command:
           - /cnb/lifecycle/exporter
         args:
           - '-log-level=debug'
           - '-layers=/layers'
           - '-app=/workspace'
           - '-group=/layers/group.toml'
           - '-analyzed=/layers/analyzed.toml'
           - '-project-metadata=/layers/project-metadata.toml'
           - '-cache-dir=/cache'
           - '-report=/var/report/report.toml'
           - '-process-type=web'
           - '-run-image=registry.local:5000/redhat-buildpacks/quarkus:run'
           - 'registry.local:5000/quarkus-petclinic'
           - 'registry.local:5000/quarkus-petclinic:b1.20211008'
         env:
           - name: HOME
             value: /builder/home
           - name: CNB_PLATFORM_API
             value: '0.5'
         volumeMounts:
...

The text was updated successfully, but these errors were encountered:

aemengo · 2021-10-07T13:45:45Z

So, the suggestion is that you can create a builder with a lifecycle specified as an image tag? specified just like you can with pack build ... --lifecycle-image?

Or is it, more simply, that you can create a builder with a uri to a custom lifecycle archive? Because that's already possible to today, via the uri: key: https://buildpacks.io/docs/reference/config/builder-config/#lifecycle-_optional_

natalieparellano · 2021-10-07T13:59:42Z

@aemengo do you mind if I move this issue over to pack?

It's been a while since I checked, but I believe that the uri key expects to receive the path to a local (or maybe also remote?) .tgz file. It would be suuuuuper convenient if you could also specify uri: buildpacksio/lifecycle:919b8ad-linux-arm64 and then pack just copies the lifecycle layer out of that image onto the builder it's creating.

Edit: this would avoid the need to compile or separately locate and download the lifecycle artifact for that SHA.

aemengo · 2021-10-07T14:02:42Z

@natalieparellano I don't mind.

@cmoulliard To be clear, we're curious if the uri key is enough to fit your use-case. Or do you require something more.

cmoulliard · 2021-10-07T14:18:48Z

So, the suggestion is that you can create a builder with a lifecycle specified as an image tag?

Yes ;-)

cmoulliard · 2021-10-07T14:21:25Z

Or is it, more simply, that you can create a builder with a uri to a custom lifecycle archive? Because that's already possible to today, via the uri: key: https://buildpacks.io/docs/reference/config/builder-config/#lifecycle-_optional_

The limitation of the current approach is that it is needed to re-build a new builder image if we would like to use, when we will do a build, lifecycle creator v1.0 or v1.1 or v1.0-SNAPSHOT or v1.0-rc1 .... We dont have this problem using pack build client locally as we can pass as parameter the lifecycle image to be used

cmoulliard · 2021-10-07T14:22:56Z

To be clear, we're curious if the uri key is enough to fit your use-case. Or do you require something more.

I'm looking about something more to avoid to have to build a new builder image if I would like to use a different lifecycle creator

aemengo · 2021-10-07T14:40:00Z

@cmoulliard Ok, thank you for clarifying.

So the suggestion is the builder can change the lifecycle in use (via a tag), at runtime? during the build? And this can happen independent of the pack CLI, it can happen on all platforms? say on Kubernetes?

I think this might call for a spec change, perhaps an RFC.

cmoulliard · 2021-10-07T17:00:19Z

Do I miss a step. I built a new lifecycle and created a tgz file but process to create the builder image fails

---> Creating builder for builder quarkus-native
Pulling image redhat/buildpacks-stack-quarkus-run:native
Pulling image redhat/buildpacks-stack-quarkus-build:native
Creating builder redhat/buildpacks-builder-quarkus-native:latest from build-image redhat/buildpacks-stack-quarkus-build:native
Using cached version of https://github.com/buildpacks/lifecycle/releases/download/v0.11.3/lifecycle-v0.11.3+linux.x86-64.tgz
Looking up buildpack ../../buildpacks/quarkus
Downloading buildpack from URI: file:///Users/cmoullia/temp/quarkus-buildpacks/buildpacks/quarkus
Creating builder with the following buildpacks:
-> [email protected]
Adding buildpack [email protected] (diffID=sha256:31e79cc2bea1be709f23e1dc7cdb152ea73b802b221ffe70f055b010430a0695)
Successfully created builder image redhat/buildpacks-builder-quarkus-native:latest
Tip: Run pack build <image-name> --builder redhat/buildpacks-builder-quarkus-native:latest to use this builder
Builder redhat/buildpacks-builder-quarkus-native:latest is already trusted
---> Creating builder for builder quarkus-jvm
Pulling image redhat/buildpacks-stack-quarkus-run:jvm
Pulling image redhat/buildpacks-stack-quarkus-build:jvm
Creating builder redhat/buildpacks-builder-quarkus-jvm:latest from build-image redhat/buildpacks-stack-quarkus-build:jvm
ERROR: failed to create builder: fetch lifecycle: invalid lifecycle: could not find entry path 'lifecycle.toml': not exist

Steps

$HOME/code/cncf/lifecycle
make clean build-linux-amd64
tar -czvf lifecycle.gz out/linux-amd64/lifecycle

edit `builder.toml` file of the builder project
[lifecycle]
uri = "$HOME/code/cncf/lifecycle/lifecycle.gz"

natalieparellano · 2021-10-07T17:17:48Z

Ah, use make clean build package linux-amd64, that will add the lifecycle.toml.

cmoulliard · 2021-10-07T17:55:13Z

Ah, use make clean build package linux-amd64, that will add the lifecycle.toml.

No luck

make clean build package build-linux-amd64
> Cleaning workspace...
rm -rf /Users/cmoullia/code/cncf/lifecycle/out
> Building lifecycle/lifecycle for linux/amd64...
mkdir -p /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle
GOARCH=amd64 CGO_ENABLED=0 go build  -ldflags "-s -w -X 'github.com/buildpacks/lifecycle/cmd.SCMRepository=github.com/buildpacks/lifecycle' -X 'github.com/buildpacks/lifecycle/cmd.SCMCommit=919b8add' -X 'github.com/buildpacks/lifecycle/cmd.Version=0.12.0-rc.1-4+919b8add'" -o /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/lifecycle -a ./cmd/lifecycle
> Creating phase symlinks for linux/amd64...
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/detector
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/analyzer
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/restorer
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/builder
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/exporter
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/rebaser
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/creator
> Building lifecycle/launcher for linux/amd64...
mkdir -p /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle
GOARCH=amd64 CGO_ENABLED=0 go build  -ldflags "-s -w -X 'github.com/buildpacks/lifecycle/cmd.SCMRepository=github.com/buildpacks/lifecycle' -X 'github.com/buildpacks/lifecycle/cmd.SCMCommit=919b8add' -X 'github.com/buildpacks/lifecycle/cmd.Version=0.12.0-rc.1-4+919b8add'" -o /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/launcher -a ./cmd/launcher
test $(du -m /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle/launcher|cut -f 1) -le 3
> Building lifecycle/lifecycle for linux/arm64...
mkdir -p /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle
GOARCH=arm64 CGO_ENABLED=0 go build  -ldflags "-s -w -X 'github.com/buildpacks/lifecycle/cmd.SCMRepository=github.com/buildpacks/lifecycle' -X 'github.com/buildpacks/lifecycle/cmd.SCMCommit=919b8add' -X 'github.com/buildpacks/lifecycle/cmd.Version=0.12.0-rc.1-4+919b8add'" -o /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/lifecycle -a ./cmd/lifecycle
> Creating phase symlinks for linux/arm64...
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/detector
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/analyzer
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/restorer
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/builder
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/exporter
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/rebaser
ln -sf lifecycle /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/creator
> Building lifecycle/launcher for linux/arm64...
mkdir -p /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle
GOARCH=arm64 CGO_ENABLED=0 go build  -ldflags "-s -w -X 'github.com/buildpacks/lifecycle/cmd.SCMRepository=github.com/buildpacks/lifecycle' -X 'github.com/buildpacks/lifecycle/cmd.SCMCommit=919b8add' -X 'github.com/buildpacks/lifecycle/cmd.Version=0.12.0-rc.1-4+919b8add'" -o /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/launcher -a ./cmd/launcher
test $(du -m /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle/launcher|cut -f 1) -le 3
> Building lifecycle/lifecycle for windows/amd64...
go build  -ldflags "-s -w -X 'github.com/buildpacks/lifecycle/cmd.SCMRepository=github.com/buildpacks/lifecycle' -X 'github.com/buildpacks/lifecycle/cmd.SCMCommit=919b8add' -X 'github.com/buildpacks/lifecycle/cmd.Version=0.12.0-rc.1-4+919b8add'" -o /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle/lifecycle.exe -a ./cmd/lifecycle
> Creating phase symlinks for Windows...
ln -sf lifecycle.exe /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle/detector.exe
ln -sf lifecycle.exe /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle/analyzer.exe
ln -sf lifecycle.exe /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle/restorer.exe
ln -sf lifecycle.exe /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle/builder.exe
ln -sf lifecycle.exe /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle/exporter.exe
ln -sf lifecycle.exe /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle/rebaser.exe
ln -sf lifecycle.exe /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle/creator.exe
> Building lifecycle/launcher for windows/amd64...
go build  -ldflags "-s -w -X 'github.com/buildpacks/lifecycle/cmd.SCMRepository=github.com/buildpacks/lifecycle' -X 'github.com/buildpacks/lifecycle/cmd.SCMCommit=919b8add' -X 'github.com/buildpacks/lifecycle/cmd.Version=0.12.0-rc.1-4+919b8add'" -o /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle/launcher.exe -a ./cmd/launcher
> Packaging lifecycle for linux/amd64...
go run ./tools/packager/main.go --inputDir /Users/cmoullia/code/cncf/lifecycle/out/linux-amd64/lifecycle -archivePath /Users/cmoullia/code/cncf/lifecycle/out/lifecycle-v0.12.0-rc.1-4+919b8add+linux.x86-64.tgz -descriptorPath lifecycle.toml -version 0.12.0-rc.1-4+919b8add
> Packaging lifecycle for linux/arm64...
go run ./tools/packager/main.go --inputDir /Users/cmoullia/code/cncf/lifecycle/out/linux-arm64/lifecycle -archivePath /Users/cmoullia/code/cncf/lifecycle/out/lifecycle-v0.12.0-rc.1-4+919b8add+linux.arm64.tgz -descriptorPath lifecycle.toml -version 0.12.0-rc.1-4+919b8add
> Packaging lifecycle for windows/amd64...
go run ./tools/packager/main.go --inputDir /Users/cmoullia/code/cncf/lifecycle/out/windows-amd64/lifecycle -archivePath /Users/cmoullia/code/cncf/lifecycle/out/lifecycle-v0.12.0-rc.1-4+919b8add+windows.x86-64.tgz -descriptorPath lifecycle.toml -version 0.12.0-rc.1-4+919b8add
make: Nothing to be done for `build-linux-amd64'.
 ~/code/cncf/lifecycle   main  ls -la out/linux-amd64/lifecycle 
total 27048
drwxr-xr-x  11 cmoullia  staff       352 Oct  7 19:53 .
drwxr-xr-x   3 cmoullia  staff        96 Oct  7 19:53 ..
lrwxr-xr-x   1 cmoullia  staff         9 Oct  7 19:53 analyzer -> lifecycle
lrwxr-xr-x   1 cmoullia  staff         9 Oct  7 19:53 builder -> lifecycle
lrwxr-xr-x   1 cmoullia  staff         9 Oct  7 19:53 creator -> lifecycle
lrwxr-xr-x   1 cmoullia  staff         9 Oct  7 19:53 detector -> lifecycle
lrwxr-xr-x   1 cmoullia  staff         9 Oct  7 19:53 exporter -> lifecycle
-rwxr-xr-x   1 cmoullia  staff   2478080 Oct  7 19:53 launcher
-rwxr-xr-x   1 cmoullia  staff  11370496 Oct  7 19:53 lifecycle
lrwxr-xr-x   1 cmoullia  staff         9 Oct  7 19:53 rebaser -> lifecycle
lrwxr-xr-x   1 cmoullia  staff         9 Oct  7 19:53 restorer -> lifecycle
 ~/code/cncf/lifecycle   main  ls -la out/linux-amd64/         
total 0
drwxr-xr-x   3 cmoullia  staff   96 Oct  7 19:53 .
drwxr-xr-x  11 cmoullia  staff  352 Oct  7 19:53 ..
drwxr-xr-x  11 cmoullia  staff  352 Oct  7 19:53 lifecycle

natalieparellano · 2021-10-07T18:27:15Z

For whatever reason the lifecycle.toml file is only created in the packaging step, so it will be part of out/lifecycle-v0.12.0-rc.1-6+de98d3f+linux.x86-64.tgz but not out/linux-amd64.

I provided the incorrect command before, make clean build package or make clean build-linux-amd64 package-linux-amd64 should be sufficient.

cmoulliard · 2021-10-07T18:38:42Z

So the suggestion is the builder can change the lifecycle in use (via a tag), at runtime? during the build? And this can happen

Ideally, at runtime, when by example we execute a build using a container, pod and where we would like to override the lifecycle pakaged within the image

aemengo · 2021-10-07T18:44:42Z

@cmoulliard pack won't be able to do what you're asking. If you're daring, next steps would be to create an RFC proposal and we'd likely discuss it at an upcoming community working group meeting

cmoulliard · 2021-10-07T19:48:10Z

@cmoulliard pack won't be able to do what you're asking.

This is not what I'm asking. What I would like is to be able to specify the version of the lifecycle to be used when a pod is doing a build.

       - name: export
         image: registry.local:5000/redhat-builder/quarkus
         command:
           - /cnb/lifecycle/exporter
         args:

That will certainly then need that a pre-step is added (= initContainer) to load the needed go binary before to launch the build ...

aemengo · 2021-10-07T20:40:39Z

Again, it's still not clear what you want. If you:

Want the "builder to download the specified lifecycle" into its container, at runtime, this will require an RFC proposal. It goes against the principles of reproducibility to have core components on a container image be switched out, after the container is built.
Want to change the image used for the export pod in your k8s definition, then you can simply do that:
```
   - name: export
     # image: registry.local:5000/redhat-builder/quarkus
     image: buildpacksio/lifecycle:919b8ad-linux-arm64
     command:
       - /cnb/lifecycle/exporter
     args:
```
This is exactly how the pack behaves when you use the --lifecycle-image flag during pack build. Only in cases when the builder is not trusted, and only for the analyze, restore, export phases
Want to use initContainers to download the lifecycle binary to a volume and so that you can swap out the /cnb/lifecycle/... directories in the following steps, then you can simply do that.

cmoulliard · 2021-10-08T07:40:23Z

For whatever reason the lifecycle.toml file is only created in the packaging step, so it will be part of out/lifecycle-v0.12.0-rc.1-6+de98d3f+linux.x86-64.tgz but not out/linux-amd64.

I provided the incorrect command before, make clean build package or make clean build-linux-amd64 package-linux-amd64 should be sufficient.

Thanks. That did the job :-)

#lifecycle project
make clean build-linux-amd64 package-linux-amd64

# and next creating a new builder
Creating builder redhat/buildpacks-builder-quarkus-native:latest from build-image redhat/buildpacks-stack-quarkus-build:native
Using cached version of https://github.com/buildpacks/lifecycle/releases/download/v0.11.3/lifecycle-v0.11.3+linux.x86-64.tgz
Looking up buildpack ../../buildpacks/quarkus

where `builder.toml`

[lifecycle]
uri = "file:///Users/cmoullia/code/cncf/lifecycle/out/lifecycle-v0.12.0-rc.1-4+919b8add+linux.x86-64.tgz"

WARNING: The message logged using the verbose mode by pack builder gives the impression that the version packaged is Using cached version of https://github.com/buildpacks/lifecycle/releases/download/v0.11.3/lifecycle-v0.11.3+linux.x86-64.tgz instead of what we pass as uri.

That should be improved as the only way now to figure out which version is packaged is to do:

docker run --rm -it redhat/buildpacks-builder-quarkus-jvm /cnb/lifecycle/creator -version
0.12.0-rc.1-4+919b8add

@ameyer-pivotal @natalieparellano

cmoulliard · 2021-10-08T08:43:56Z

@cmoulliard To be clear, we're curious if the uri key is enough to fit your use-case.

uri is enough.

Isue: uri do not work for the lifecycle property if we pass an image

[lifecycle]
uri = "buildpacksio/lifecycle:919b8ad-linux-arm64"

--->

pack builder ...
ERROR: failed to create builder: fetch lifecycle: invalid lifecycle: open lifecycle blob: read blob at path '/Users/cmoullia/temp/quarkus-buildpacks/builders/quarkus-jvm/buildpacksio/lifecycle:919b8ad-linux-arm64': stat /Users/cmoullia/temp/quarkus-buildpacks/builders/quarkus-jvm/buildpacksio/lifecycle:919b8ad-linux-arm64: no such file or directory

@aemengo

aemengo · 2021-10-08T14:00:14Z

@cmoulliard Write RFC

jjbustamante · 2023-08-17T21:22:03Z

Hi @cmoulliard

Checking this issue, I think the issue now is that you find a bug referencing an lifecycle image using the uri field, right? as you mentioned here

In that case, I can remove the status/requires-rfc label and convert this one into a bug

cmoulliard · 2023-08-18T10:19:44Z

I can remove the

Yes

natalieparellano · 2024-07-01T20:24:07Z

Closing as a duplicate of #2076

natalieparellano transferred this issue from buildpacks/lifecycle Oct 7, 2021

jjbustamante added the status/requires-rfc Issue or PR that requires an RFC to be filed. label Jun 7, 2023

jjbustamante added type/bug Issue that reports an unexpected behaviour. status/ready Issue ready to be worked on. and removed status/requires-rfc Issue or PR that requires an RFC to be filed. labels Aug 18, 2023

natalieparellano added help wanted Need some extra hands to get this done. type/enhancement Issue that requests a new feature or improvement. and removed type/bug Issue that reports an unexpected behaviour. labels Aug 18, 2023

natalieparellano added contribfest and removed contribfest labels Mar 20, 2024

natalieparellano closed this as completed Jul 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Suggestion: Download a snapshot release of the lifecycle #1305

Suggestion: Download a snapshot release of the lifecycle #1305

cmoulliard commented Oct 7, 2021

aemengo commented Oct 7, 2021

natalieparellano commented Oct 7, 2021 •

edited

Loading

aemengo commented Oct 7, 2021

cmoulliard commented Oct 7, 2021

cmoulliard commented Oct 7, 2021

cmoulliard commented Oct 7, 2021

aemengo commented Oct 7, 2021

cmoulliard commented Oct 7, 2021

natalieparellano commented Oct 7, 2021

cmoulliard commented Oct 7, 2021

natalieparellano commented Oct 7, 2021

cmoulliard commented Oct 7, 2021

aemengo commented Oct 7, 2021

cmoulliard commented Oct 7, 2021

aemengo commented Oct 7, 2021 •

edited

Loading

cmoulliard commented Oct 8, 2021 •

edited

Loading

cmoulliard commented Oct 8, 2021

aemengo commented Oct 8, 2021

jjbustamante commented Aug 17, 2023

cmoulliard commented Aug 18, 2023

natalieparellano commented Jul 1, 2024

Suggestion: Download a snapshot release of the lifecycle #1305

Suggestion: Download a snapshot release of the lifecycle #1305

Comments

cmoulliard commented Oct 7, 2021

Description

aemengo commented Oct 7, 2021

natalieparellano commented Oct 7, 2021 • edited Loading

aemengo commented Oct 7, 2021

cmoulliard commented Oct 7, 2021

cmoulliard commented Oct 7, 2021

cmoulliard commented Oct 7, 2021

aemengo commented Oct 7, 2021

cmoulliard commented Oct 7, 2021

natalieparellano commented Oct 7, 2021

cmoulliard commented Oct 7, 2021

natalieparellano commented Oct 7, 2021

cmoulliard commented Oct 7, 2021

aemengo commented Oct 7, 2021

cmoulliard commented Oct 7, 2021

aemengo commented Oct 7, 2021 • edited Loading

cmoulliard commented Oct 8, 2021 • edited Loading

cmoulliard commented Oct 8, 2021

aemengo commented Oct 8, 2021

jjbustamante commented Aug 17, 2023

cmoulliard commented Aug 18, 2023

natalieparellano commented Jul 1, 2024

natalieparellano commented Oct 7, 2021 •

edited

Loading

aemengo commented Oct 7, 2021 •

edited

Loading

cmoulliard commented Oct 8, 2021 •

edited

Loading