trivy-operator error download-policy

[rsantacreu-ust]

What steps did you take and what happened:

Deploying a aquasecurity/trivy-operator helm chart with version

---
apiVersion: v2
name: trivy-operator
description: trivy-operator
type: application
version: 0.22.0
dependencies:
  - name: trivy-operator
    version: 0.24.1
    repository: https://aquasecurity.github.io/helm-charts/

and using the following values file

trivy-operator:
  targetNamespaces: ""
  operator: 
    namespace: "security"
    scanJobsConcurrentLimit: 2
    builtInTrivyServer: false
    builtInServerRegistryInsecure: false
  trivy:
    slow: true
    resources:
      requests:
        cpu: 500m
        memory: 800Mi
      limits:
        memory: 800Mi
        cpu: 1500m
    dbRepositoryInsecure: "true"
    debug: false
  podAnnotations:
    fluentbit.io/exclude: "true"
  resources:
    requests:
      cpu: 100m
      memory: 800Mi
    limits:
      memory: 800Mi
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: topology.kubernetes.io/region
            operator: In
            values:
            - eu-south-2
        topologyKey: "kubernetes.io/hostname"
  policiesBundle:
    insecure: true

We're getting errors in trivy-operator that failed to download policies

{"level":"error","ts":"2024-11-06T10:00:31Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: oci download error: f │ │ ailed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-checks/blobs/sha256:cba49b6781cfcdeb6b063283a711ce0ddb1f36d6e2a5db69ef7d2e3f13998149: TOOMANYREQUESTS: retry-after: 168.77µs, allowed: 44000/minute","stacktrace":"github.com/a │ │ quasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).loadPolicies\n\t/home/runne │ │ r/work/trivy-operator/trivy-operator/pkg/policy/policy.go:144\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).Hash\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:114\ngithub.com/aquasecurity/trivy-operato │ │ r/pkg/configauditreport/controller.(*ResourceController).SetupWithManager.(*ResourceController).reconcileResource.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/resource.go:208\nsigs.k8s.io/controller- │ │ runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sig │ │ s.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/ │ │ controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/cont │ │ roller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}

scan-vulnerabilities pods get the following output

57475ada-1905-46a4-872e-6a76fc6688f2 2024-11-06T10:02:11Z    INFO    Adding schema version to the DB repository for backward compatibility    repository="ghcr.io/aquasecurity/trivy-db:2"                                                           │
57475ada-1905-46a4-872e-6a76fc6688f2 2024-11-06T10:02:11Z    INFO    Need to update DB                                                                                                                                                               
57475ada-1905-46a4-872e-6a76fc6688f2 2024-11-06T10:02:11Z    INFO    Downloading DB...    repository="ghcr.io/aquasecurity/trivy-db:2"

What did you expect to happen:

I expect to not get download policies errors and pods scan-vulnerability finish scaning, the last output that we get is "Downloading DB..."

Anything else you would like to add:

We've added the insecure flag to allow the DB to download if not we get error downloading db with TOOMANYREQUESTS
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Trivy-Operator version (use trivy-operator version): "0.22.0"
• Kubernetes version (use kubectl version): v1.30.4-eks-a737599
• OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): Ubuntu 24.04

[dnikoloski]

I am also unable to deploy trivy-operator.

Warning FailedPullImage 10s kubelet Failed to pull image "ghcr.io/aquasecurity/trivy-operator:0.19.4": rpc error: code = Unknown desc = Error response from daemon: toomanyrequests: retry-after: 22.917µs, allowed: 44000/minut

[pfaelzerchen]

I have the same problem. I was able to solve a part of it with the following settings in values.yaml to use mirrors for querying the images:

global:
  image:
    registry: "public.ecr.aws"

trivy:
  registry:
    mirror:
      "mirror.gcr.io": mirror.gcr.io
      "public.ecr.aws": public.ecr.aws
      "docker.io": docker.io
      "ghcr.io": ghcr.io

  dbRegistry: "public.ecr.aws"
  dbRepository: "aquasecurity/trivy-db"

  javaDbRegistry: "public.ecr.aws"
  javaDbRepository: "aquasecurity/trivy-java-db"

With that, trivy-operator can be deployed and will create report resources.

What remains in the logs are TOOMANYREQUESTS errors for the policiesBundle aquasecurity/trivy-checks. And I was not able to find any mirror for that.

[afdesk]

unfortunately, there is no a mirror for trivy-checks now, but Trivy Team is working hard to resolve it.

[pfaelzerchen]

That is good to hear. I temporarily deactivated the policiesBundle with trivy.useBuiltinRegoPolicies: "false" to get rid of the "useless" errors (and to reduce pressure on the repo).

[afdesk]

That is good to hear. I temporarily deactivated the policiesBundle with trivy.useBuiltinRegoPolicies: "false" to get rid of the "useless" errors (and to reduce pressure on the repo).

good decission, I did the same change. but it seems it doesn't work for image scan

[pfaelzerchen]

I saw by chance that trivy-check is available at Docker Hub. So I gave it a try, but recognized something weird:

trivy:
  useBuiltinRegoPolicies: true

policiesBundle:
  # -- registry of the policies bundle
  registry: docker.io
  # -- repository of the policies bundle
  repository: aquasecurity/trivy-checks

But in the trivy-operator logs it seems that the change in registry is ignored:

{"level":"error","ts":"2024-11-21T14:40:49Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-checks/blobs/sha256:cba49b6781cfcdeb6b063283a711ce0ddb1f36d6e2a5db69ef7d2e3f13998149: TOOMANYREQUESTS: retry-after: 363.439µs, allowed: 44000/minute","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).loadPolicies\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:144\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).Hash\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:114\ngithub.com/aquasecurity/trivy-operator/pkg/operator.(*TTLReportReconciler).applicableForDeletion\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/operator/ttl_report.go:152\ngithub.com/aquasecurity/trivy-operator/pkg/operator.(*TTLReportReconciler).DeleteReportIfExpired\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/operator/ttl_report.go:105\ngithub.com/aquasecurity/trivy-operator/pkg/operator.(*TTLReportReconciler).SetupWithManager.(*TTLReportReconciler).reconcileReport.func5\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/operator/ttl_report.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}
{"level":"error","ts":"2024-11-21T14:40:50Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-checks/blobs/sha256:cba49b6781cfcdeb6b063283a711ce0ddb1f36d6e2a5db69ef7d2e3f13998149: TOOMANYREQUESTS: retry-after: 482.171µs, allowed: 44000/minute","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).loadPolicies\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:144\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).Hash\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:114\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.(*ResourceController).SetupWithManager.(*ResourceController).reconcileResource.func2\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/resource.go:208\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}
{"level":"error","ts":"2024-11-21T14:40:53Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-checks/blobs/sha256:cba49b6781cfcdeb6b063283a711ce0ddb1f36d6e2a5db69ef7d2e3f13998149: TOOMANYREQUESTS: retry-after: 35.881µs, allowed: 44000/minute","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).loadPolicies\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:144\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).Hash\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:114\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.(*ResourceController).SetupWithManager.(*ResourceController).reconcileResource.func2\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/resource.go:208\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}
{"level":"error","ts":"2024-11-21T14:40:54Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-checks/blobs/sha256:cba49b6781cfcdeb6b063283a711ce0ddb1f36d6e2a5db69ef7d2e3f13998149: TOOMANYREQUESTS: retry-after: 406.265µs, allowed: 44000/minute","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).loadPolicies\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:144\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).Hash\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:114\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.(*ResourceController).SetupWithManager.(*ResourceController).reconcileResource.func2\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/resource.go:208\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}
{"level":"error","ts":"2024-11-21T14:40:54Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-checks/blobs/sha256:cba49b6781cfcdeb6b063283a711ce0ddb1f36d6e2a5db69ef7d2e3f13998149: TOOMANYREQUESTS: retry-after: 239.849µs, allowed: 44000/minute","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).loadPolicies\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:144\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).Eval\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:199\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.evaluate\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/helper.go:45\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.(*ResourceController).SetupWithManager.(*ResourceController).reconcileResource.func2\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/resource.go:229\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}

[afdesk]

@pfaelzerchen thanks for the report.
Could you recheck with a correct repository for docker.io instead of aquasecurity/trivy-checks:1?
or better, you can use mirror.gcr.io.

  repository: aquasec/trivy-checks

[pfaelzerchen]

OK, I found a working configuration. When redeploying with helm, useBuiltinRegoPolicies: true will be available to the operator, but not the change in the repo-configuration. After restarting the pod, it will try to load from the new configuration.

What seems to work:

        policiesBundle:
          # -- registry of the policies bundle
          registry: docker.io
          # -- repository of the policies bundle
          repository: aquasec/trivy-checks
          # -- tag version of the policies bundle
          tag: latest

mirror.gcr.io is also working.