Operator endlessly updates LoadBalancer Services object

[bukovjanmic]

When using Tenant CR and exposing services via LoadBalancer type, a MetalLB annotation is added to a service by Openshift noting that an address was allocated. This however conflicts with operator Service object check and the operator tries to reconcile/remove the annotations, after which metallb adds it back.

The Services object is being endlessly updated, which generates unnecessary load for Kubernetes API server.

This is relevant portion of tenant CR:

  exposeServices:
    console: true
    minio: true
  serviceMetadata:
    consoleServiceAnnotations:
      service.beta.openshift.io/serving-cert-secret-name: minio-service-console-cert
    minioServiceAnnotations:
      service.beta.openshift.io/serving-cert-secret-name: minio-service-cert

The generated Service annotations:

kind: Service
apiVersion: v1
metadata:
  annotations:
    metallb.universe.tf/ip-allocated-from-pool: addresspool
    service.beta.openshift.io/serving-cert-secret-name: minio-service-cert

This results in Service object being constantly updated back and forth, in operator logs we see:

I0620 10:14:42.250078 1 minio-services.go:64] MinIO Services don't match: service annotations don't match
I0620 10:14:42.642629 1 console.go:74] Console Service don't match: service annotations don't match. Conciliating

One can argue that metallb should not possibly add annotations to an owned object, however, the logic in service reconciliation could be updated so that extra annotations and possibly labels are ignored during reconciliation.

Another workaround would the to place annotation in the CR, but this seems awkward:

  exposeServices:
    console: true
    minio: true
  serviceMetadata:
    consoleServiceAnnotations:
      service.beta.openshift.io/serving-cert-secret-name: minio-service-console-cert
      metallb.universe.tf/ip-allocated-from-pool: addresspool
    minioServiceAnnotations:
      service.beta.openshift.io/serving-cert-secret-name: minio-service-cert
      metallb.universe.tf/ip-allocated-from-pool: addresspool

[jiuker]

fix in #2129
Not release yet