rules_oci@v2beta1 - Image not compatible with AWS lambdas

[juanzolotoochin]

AWS allows to create lambdas using container image, see https://docs.aws.amazon.com/lambda/latest/dg/images-create.html.

We are seeing the following error when trying to run a lambda built with version 2beta1 of rules_oci:

"ImageLayerFailure: UnsupportedImageLayerDetected - Layer Digest sha256:dd2d1b3af2eef813d8cf22b1e17c4500e6c34fd80eddb0f0a78e80632fc872da"

The exact same image works fine when using rules_oci v1.

[thesayyn]

Could be a bug. It would help me a lot debugging if you could provide a reproducer for this.

[juanzolotoochin]

I thought I'd added this to this bug but it was only on the slack channel.

It's hard to give repro instructions because I don't understand the AWS infra enough and this part is buried under a ton of TF code from me.

But after inspecting the two images (rules_oci v1 vs v2) I suspect the problem comes from the missing history entries for the docker layers. The v2 image does not have history entries for any of the tar layers.

  "author": "github.com/chainguard-dev/apko",
  "created": "2024-06-28T05:42:58Z",
  "history": [
    {
      "author": "apko",
      "created": "2024-06-28T05:42:58Z",
      "created_by": "apko",
      "comment": "This is an apko single-layer image"
    },
    {
      "created": "0001-01-01T00:00:00Z"
    },
    {
      "created": "0001-01-01T00:00:00Z"
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:8952da8c429f5b89a61500dc9b5e52ce326da208332a8c16f8e51b0563bf53f7",
      "sha256:dd2d1b3af2eef813d8cf22b1e17c4500e6c34fd80eddb0f0a78e80632fc872da",
      "sha256:45aca4f0d57586f6ab7e6c7b9104204b93f02a4afef6e0f5ffaad4ac003440e8"
    ]
  },

I suspect AWS assumes that len(history) == len(layers) somewhere. Ideally this history would contain useful metadata as sugested by #395 but even if not that, it would be great to have something consistent with rules_oci v1 at least.

[thesayyn]

fixed by #654

[juanzolotoochin]

Sadly this didn't fix it. I have a support case open with AWS trying to figure out what's going on

[juanzolotoochin]

Ok we found the root cause.

It seems that AWS Lambdas are not happy when the layers have different mediaType. The default behavior for rules_oci v2 changed such that the layers were no longer compressed by default. Manually compressing the tars to ensure the same mediatype works around the issue.

This seems to be a bug on the AWS side since it does not seem like mixing media types violates the OCI spec.

[lqc]

Ok we found the root cause.

It seems that AWS Lambdas are not happy when the layers have different mediaType. The default behavior for rules_oci v2 changed such that the layers were no longer compressed by default. Manually compressing the tars to ensure the same mediatype works around the issue.

This seems to be a bug on the AWS side since it does not seem like mixing media types violates the OCI spec.

@juanzolotoochin Is that the only change you made? Could you let me know if you are using pkg_tar() or tar() rule? I'm facing the same issue and the different mediaType was also my first guess, but know they are all "application/vnd.docker.image.rootfs.diff.tar.gzip" and AWS Lambda still rejects the image.

[njlr]

Ok we found the root cause.
It seems that AWS Lambdas are not happy when the layers have different mediaType. The default behavior for rules_oci v2 changed such that the layers were no longer compressed by default. Manually compressing the tars to ensure the same mediatype works around the issue.
This seems to be a bug on the AWS side since it does not seem like mixing media types violates the OCI spec.

@juanzolotoochin Is that the only change you made? Could you let me know if you are using pkg_tar() or tar() rule? I'm facing the same issue and the different mediaType was also my first guess, but know they are all "application/vnd.docker.image.rootfs.diff.tar.gzip" and AWS Lambda still rejects the image.

Here's how I debugged this:

1. Build your tarball using oci_load and filegroup
2. Unpack the .tar so you can inspect the blobs
3. Use file /path/to/unpackted/tar/* to inspect the blobs
4. Identify the blobs that do not match (I decided to prefer gzip compressed data)
5. Modify the targets that produce the mismatched blobs

In my case, I was using pkg_tar and it was not applying compression. I added extension = "tar.gz" and the blob then matched the rest.

[thesayyn]

I have myself had this problem as well, ensuring all the layers have same compression and all the directories have a parent directory in the image tree makes it work.

Unfortunately there is nothing we can do to ensure this for you, it's up to the user.

[thesayyn]

For pkg_tar(extension = "tar.gz") and for tar(compress = "gzip", args = ["--options=gzip:!timestamp"]) makes it work.

[thesayyn]

Okay bazel-contrib/bazel-lib#783 is where we track the upstream issue for tar