Policies Bundle not working with internal ecr repo: getting error "failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: OCI artifact must be a single layer

[chit4]

What steps did you take and what happened:

We are trying to upload all the trivy dependency images, like trivy,trivy-db, trivy-java-db and trivy-checks to internal ecr repo for air gaped environment, i have automated it to fetch image from github and upload it to ecr

While rest all images work fine with our ecr with policy bundles repo we are seeing this following error {"level":"error","ts":"2024-08-09T09:18:39Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: OCI artifact must be a single layer","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).loadPolicies\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:144\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).Hash\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:114\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.(*ResourceController).SetupWithManager.(*ResourceController).reconcileResource.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/resource.go:208\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"} {"level":"error","ts":"2024-08-09T09:18:40Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: OCI artifact must be a single layer","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).loadPolicies\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:144\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).Eval\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:199\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.evaluate\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/helper.go:45\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.(*ResourceController).SetupWithManager.(*ResourceController).reconcileResource.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/resource.go:229\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}

What did you expect to happen:

We wanted it to work seamlessly with internal repo ecr

Anything else you would like to add:

It looks like internal ecr repo is not functioning the way how a ghcr works, need help to fix this

Environment:

• Trivy-Operator version (use trivy-operator version): 0.22.0
• Kubernetes version (use kubectl version): v1.26.15-eks-db838b0
• OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): macos

[github-actions]

This issue is stale because it has been labeled with inactivity.

[gnadaban]

How did you manage to even get this far with ECR?
Are you using IRSA at all? According to #1874 this is not supported, how did you solve the credential expiration?

[badgerspoke]

we're in the same boat - we have the vulnerability DBs in ECR (per this doc) and that's working fine (care of the operator's service account having an associated IAM role - i.e. via IRSA), but I can't get the operator to use ECR for the checks DB.
I'm considering having something populate a cache of the checks and have trivy use that via the custom checks option