add support or verify support of OCI crypt

[cgwalters]

It'd be great to be sure we support OCI crypt; it'd greatly help use cases of embedding secret data inside a bootable container image. I need to dig into the flow for this.

Clearly for a bootc install style flow we'd need to have the provisioning system (e.g. cloud-init -> AMI or Anaconda) also be configured with decryption keys.

We should definitely still support secrets other ways too; I think support oci-crypt would be also be natural when we extend support for configmaps and particularly secrets.

[cgwalters]

cc @travier

[harche]

@cgwalters I had worked on adding the image encryption support for CRIO as well as skopeo.

I would love to take this up, if no one else has already started working on it.

[cgwalters]

@harche That'd be awesome!

[harche]

Notes. -

Potentially this decryptor can be used to decrypt the image layers.

[vrothberg]

I didn't fully read the code yet, so take my comment with a grain of salt.

Isn't bootc reading the layers directly from the Skopeo socket? So I think OCI crypt should work transparently because skopeo takes care of it.

Nice to see you again, @harche 👋

[harche]

@vrothberg good point, I haven't looked closely the bootc code. I was assuming it was written completely in rust which might require rust implementation of c/image to decrypt the images. But if it uses skopeo socket in the background then the task is much simpler.

[cgwalters]

Right, bootc depends on ostree-ext and that repo has the dependency graph for the architecture.

[harche]

Nice to see you again, @harche 👋

Nice to see you too @vrothberg :)

Right, bootc depends on ostree-ext and that repo has the dependency graph for the architecture.

@cgwalters @vrothberg does it make sense to use rust implementation of c/image instead of relying on the skope socket? If yes, I can create an issue and assign to me to track that work.

[cgwalters]

@harche see https://github.com/containers/containers-image-proxy-rs/#desired-containersimage-features

[cgwalters]

This thread also relates a bit to #128 - basically the direction we probably need to be moving instead is to share more code with c/storage too, which pushes us to extending the proxying between Rust ➡️ Go instead of using a new implementation. Basically we need lockstep features and integration with podman.

[vrothberg]

I very strongly urge to not rewrite the containers/image library in Rust or any other language. There's close to a decade of work behind and it's used in the entire RH ecosystem.

https://github.com/containers/containers-image-proxy-rs/#desired-containersimage-features mentions involving "containers/storage" but I do not understand what that means in detail (or what the problem is). The skopeo proxy (IIRC) happily returns the layers without going through storage. Also with composefs lurking around the corner, using c/storage may be the way forward.

[cgwalters]

I very strongly urge to not rewrite the containers/image library in Rust or any other language.

Well, the confidential computing folks already did - just not all of it. Anyways I think we're all in sync here.

[vrothberg]

confidential computing folks already did - just not all of it

Do you have a pointer? Curious to catch up on that.

[harche]

confidential computing folks already did - just not all of it

Do you have a pointer? Curious to catch up on that.

https://github.com/confidential-containers/guest-components/blob/main/image-rs/docs/design.md#design

IIRC, within Redhat they are in #forum-sandboxed-containers and #wg-confidential-computing in slack.

[harche]

I very strongly urge to not rewrite the containers/image library in Rust or any other language.

Well, the confidential computing folks already did - just not all of it. Anyways I think we're all in sync here.

I agree, we are in sync. Thanks @cgwalters and @vrothberg for your inputs.

[vrothberg]

https://github.com/confidential-containers/guest-components/blob/main/image-rs/docs/design.md#design

IIRC, within Redhat they are in #forum-sandboxed-containers and #wg-confidential-computing in slack.

Thanks for the pointers, @harche! So are Red Hatters contributing to the effort?

[harche]

https://github.com/confidential-containers/guest-components/blob/main/image-rs/docs/design.md#design
IIRC, within Redhat they are in #forum-sandboxed-containers and #wg-confidential-computing in slack.

Thanks for the pointers, @harche! So are Red Hatters contributing to the effort?

I am assuming yes. Look for their updates in aos-devel mailing list (filter it with "confidential-containers")

Some of the recent blog posts by that group,
Confidential containers - based on the confidential containers CNCF project and work done this year with the Azure team
Confidential Computing Attestation - describing various forms of attestation for Confidential Computing use cases
Confidential Virtual Machines - describing the work done in RHEL for supporting confidential virtual machines (the basis for the Azure work)

Now we know they are working in the domain of confidential computing, which uses kata containers, which in turn uses the image-rs (rust implementation of c/image). But I am not sure if they are actively contributing the code to the image-rs code base. Maybe people in those slack channels might be able to answer that.

@bpradipt just checking if we know anyone from the confidential computing group within RH contributing the code to image-rs?

[vrothberg]

Thanks, @harche :)

My intention is not to tell other teams what to do. But I've not heard of that effort yet and want to be aware (and let the runtimes team know).

[harche]

Bootc currently employs the skopeo image proxy for the retrieval of container images. It uses the GetBlob method of the proxy to fetch a specific blob. Subsequently, Bootc independently decompresses the blob using a custom-written Rust logic, instead of leveraging skopeo for this functionality.

This approach introduces a significant challenge when it comes to decrypting the image, as it isolates the decompression step from skopeo's existing decryption features.

In order to advance and overcome this blockade, we must adopt one of the following options:

Option 1: Implement GetLayer in Skopeo Proxy

Introduce a new method named GetLayer into the Skopeo proxy to ensure that each layer returned is already decompressed—and decrypted if required—borrowing the necessary logic from the skopeo copy implementation.

Option 2: Extend new_async_decompressor to Handle Decryption

Refactor the new_async_decompressor function, potentially with a new name, to support both decompression and decryption of encrypted images, possibly resembling the following structure:

fn new_async_decompressor_decrypt<'a>(
    media_type: &oci_image::MediaType,
    src: impl AsyncBufRead + Send + Unpin + 'a,
) -> Result<Box<dyn AsyncBufRead + Send + Unpin + 'a>> {
    match media_type {
        oci_image::MediaType::ImageLayerEncryptedGzip => Ok(Box::new(tokio::io::BufReader::new(
            async_compression::tokio::bufread::GzipDecoder::new(
            async_decrypter::tokio::bufread::Decrypter::new(src, blockChiperOpts),
        ))),
        oci_image::MediaType::ImageLayerGzip => Ok(Box::new(tokio::io::BufReader::new(
            async_compression::tokio::bufread::GzipDecoder::new(src),
        ))),
        oci_image::MediaType::ImageLayer => Ok(Box::new(src)),
        oci_image::MediaType::Other(t) if t.as_str() == DOCKER_TYPE_LAYER_TAR => Ok(Box::new(src)),
        o => Err(anyhow::anyhow!("Unhandled layer type: {}", o))
    }
}

As encrypted image media types are not yet standardized in the OCI spec, with the relevant PR still open, the application will need to define these (such as ImageLayerEncryptedGzip) internally. Meanwhile, libraries that support image encryption/decryption declare that within themselves. e.g. ocicrypt in golang and corresponding rust implementation ocicrypt-rs.

The actual decryption must be handled with reputable existing libraries that implement block cipher to avoid introducing untested cryptography. The only one I know that supports image decryption well with all its features is this implementation that is a part of ocicrypt-rs.

Option 3: Use Existing Rust Libraries for Image Handling

Capitalize by employing established libraries such as image-rs and ocicrypt-rs to manage pulling, decrypting, and decompressing image layers within Rust. This would allow for leveraging native tooling and avoiding extensions to the Skopeo proxy.

/cc @cgwalters @vrothberg

[cgwalters]

Lately I've been thinking that maybe it would help us if we basically switched to depending on podman and doing a podman pull into an alternative storage root, and then doing some postprocessing fixups on that to wire it up with ostree. It may not be really hard actually, and arguably it's what we should have done from the start. (xref #128 )

[vrothberg]

I like @cgwalters idea as I interpret it as reducing complexity and a host dependency. podman image mount may come in handy.

[bpradipt]

https://github.com/confidential-containers/guest-components/blob/main/image-rs/docs/design.md#design
IIRC, within Redhat they are in #forum-sandboxed-containers and #wg-confidential-computing in slack.

Thanks for the pointers, @harche! So are Red Hatters contributing to the effort?

I am assuming yes. Look for their updates in aos-devel mailing list (filter it with "confidential-containers")

Some of the recent blog posts by that group, Confidential containers - based on the confidential containers CNCF project and work done this year with the Azure team Confidential Computing Attestation - describing various forms of attestation for Confidential Computing use cases Confidential Virtual Machines - describing the work done in RHEL for supporting confidential virtual machines (the basis for the Azure work)

Now we know they are working in the domain of confidential computing, which uses kata containers, which in turn uses the image-rs (rust implementation of c/image). But I am not sure if they are actively contributing the code to the image-rs code base. Maybe people in those slack channels might be able to answer that.

@bpradipt just checking if we know anyone from the confidential computing group within RH contributing the code to image-rs?

It's on need basis. For confidential containers (CoCo) based on Kata, image-rs is the component that's responsible for handling the images (download, decryption).

Julien (@littlejawa) will have more details.

[harche]

It's on need basis. For confidential containers (CoCo) based on Kata, image-rs is the component that's responsible for handling the images (download, decryption).

Julien (@littlejawa) will have more details.

Thanks @bpradipt

[harche]

I like @cgwalters idea as I interpret it as reducing complexity and a host dependency. podman image mount may come in handy.

Cool, let's go with that! Thanks @vrothberg and @cgwalters.

[cgwalters]

Lately I've been thinking that maybe it would help us if we basically switched to depending on podman and doing a podman pull into an alternative storage root, and then doing some postprocessing fixups on that to wire it up with ostree.

PR in #215

[cdrage]

Ran into this issue as by default podman pushes zstd, so when using bootc switch you get:

[root@k8s2 ~]# bootc switch docker.io/cdrage/bootc-k3s-node-amd64
layers already present: 0; layers needed: 73 (896.0 MB)
ERROR Switching: Pulling: Importing: Unencapsulating base: Unhandled layer type: application/vnd.oci.image.layer.v1.tar+zstd

The reasoning is that zstd is selected by default when using podman push rather than gzip now.

Requiring me to use: podman push --compression-format gzip --force-compression docker.io/cdrage/bootc-k3s-node-amd64 in order to get it to work correctly.

FYI for others in case they get this error too ^^

[cgwalters]

@cdrage that's different, you need ostreedev/ostree-rs-ext#622 which is in https://github.com/containers/bootc/releases/tag/v0.1.11

[cdrage]

@cdrage that's different, you need ostreedev/ostree-rs-ext#622 which is in https://github.com/containers/bootc/releases/tag/v0.1.11

Doh! Thank you. I'm assuming it'll land in fedora / centos soon. I appreciate it! For now I pushed gzip and installed it that way.