Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Require Docker 23, Statically link Krane #423

Open
wants to merge 5 commits into
base: develop
Choose a base branch
from

Conversation

cbgbt
Copy link
Contributor

@cbgbt cbgbt commented Dec 23, 2024

Issue number:

Closes #417

Description of changes:


krane: statically link krane into twoliter

This change implements a small FFI library in golang which emulates the
`krane` program from google/go-containerregistry.

Prior to this change, we'd experimented with two approaches to using
`krane` within twoliter, but both approaches had unfortunate
complications:

* Writing `krane` to a sealed anonymous file and executing it -- this
  approach would cause twoliter to be killed by some threat detection
  software. Executing sealed anonymous files is commonly restricted on
  Linux systems
* Writing `krane` to a tempfile and executing it -- it's difficult to
  get Rust to reliably delete named tempfiles, which led to twoliter
  leaking tempfiles of the `krane` binary.

twoliter: require Docker 23

Docker versions older than 23 no longer receive community-supported
security patches.

Additionally, twoliter requires dockerfile syntax 1.4.3. Currently the
tool pulls this image from the network at build time; however, Docker 23
bundles this version internally via buildkit. We can further isolate
twoliter builds from the network by ensuring we are on Docker 23 or
greater.

twoliter: refactor project to reduce module length

The project module is large, making it difficult to organize and read.
This moves much of the vended image/artifact machinery into a new
module. We also add a new trait to help us write functionality over
both Projects that are fully locked and Projects with a locked SDK.

twoliter: fetch build SDK prior to cargo-make

Now that `krane` is run as a static library, we cannot execute it within
Makefile.toml.

This change moves the SDK fetch to occur prior to executing build make
targets.

Testing done:

  • Integration tests
  • Manual testing of signal handler (ctrl-c while downloading SDK)
  • Manual testing of kit publication
  • Test with AWS Codebuild (used by bottlerocket team for release CI)

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@cbgbt cbgbt force-pushed the docker-23 branch 3 times, most recently from 146796b to 4793832 Compare December 23, 2024 22:06
@cbgbt cbgbt marked this pull request as ready for review December 23, 2024 22:25
@cbgbt cbgbt marked this pull request as draft December 23, 2024 22:25
@cbgbt cbgbt force-pushed the docker-23 branch 2 times, most recently from f366a17 to e01e07e Compare December 23, 2024 22:59
Cargo.toml Show resolved Hide resolved
tools/krane/go-src/main.go Show resolved Hide resolved
@cbgbt
Copy link
Contributor Author

cbgbt commented Dec 24, 2024

^ addresses feedback from @yeazelm, as well as experimenting with fixes for build failures on other hosts.

Basically, I had originally generated go.{mod,sum} using go-1.23. After downgrading the required version of go to 1.22, I needed to delete my indirect dependencies and re-run go mod tidy.

This change implements a small FFI library in golang which emulates the
`krane` program from google/go-containerregistry.

Prior to this change, we'd experimented with two approaches to using
`krane` within twoliter, but both approaches had unfortunate
complications:

* Writing `krane` to a sealed anonymous file and executing it -- this
  approach would cause twoliter to be killed by some threat detection
  software. Executing sealed anonymous files is commonly restricted on
  Linux systems
* Writing `krane` to a tempfile and executing it -- it's difficult to
  get Rust to reliably delete named tempfiles, which led to twoliter
  leaking tempfiles of the `krane` binary.
Docker versions older than 23 no longer receive community-supported
security patches.

Additionally, twoliter requires dockerfile syntax 1.4.3. Currently the
tool pulls this image from the network at build time; however, Docker 23
bundles this version internally via buildkit. We can further isolate
twoliter builds from the network by ensuring we are on Docker 23 or
greater.
The project module is large, making it difficult to organize and read.
This moves much of the vended image/artifact machinery into a new
module. We also add a new trait to help us write functionality over
both Projects that are fully locked and Projects with a locked SDK.
Now that `krane` is run as a static library, we cannot execute it within
Makefile.toml.

This change moves the SDK fetch to occur prior to executing build make
targets.
While we do not publish twoliter without pubsys support built in, it can
speed up development compile times to avoid needing to compile pubsys
(which subsequently compiles the AWS SDK for Rust.)
@cbgbt cbgbt marked this pull request as ready for review December 24, 2024 07:54
@cbgbt
Copy link
Contributor Author

cbgbt commented Dec 24, 2024

I made registration of the cleanup signal only fail with a warning. The signal registration fails in AWS CodeBuild for a reason that isn't quite clear to me.

But cleanup on signals is already best-effort, I think proceeding without it is acceptable.

@@ -4195,7 +4231,7 @@ version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf221c93e13a30d793f7645a0e7762c55d169dbb0a49671918a2319d289b10bb"
dependencies = [
"windows-sys 0.59.0",
"windows-sys 0.48.0",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting that it dropped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

krane tempfiles are not properly cleaned up
3 participants