Supress Vault secret existence checks / "Failed secret modification" K8s events

[awinecki]

Hello, thank you for vault-crd, it's been a really useful component in our stack!

We've run into a slight issue with getting a lot of FailedModification K8s events, which are of type Failure, whereas it doesn't seem anything "wrong" is happening. It's not a big problem, but it obscures other important K8s events in our monitoring.

Repro steps:

1. Create KV2 secret in Vault
2. Deploy Vault K8s resource with corresponding Vault KV2 secret path
3. Remove the Vault KV2 secret, without touching .yaml definitions in K8s (Vault resource stays unchanged)

Not, every now and then, vault-crd will refresh, checking for Vault secret existence and state. If it's not there, it brings up this notification:

vault-crd/src/main/java/de/koudingspawn/vault/kubernetes/scheduler/ScheduledRefresh.java

Line 41 in 35219d5

eventNotification.storeNewEvent(MODIFICATION_FAILED, "Modification of secret failed with exception " + e.getMessage(), resource);

This seems sensible, since the secret is not present. However, I believe our use case is a bit less obvious.

Our use case – default secrets + override

We have multiple development environments of the same apps. For convenience, we share a _default KV2 secret in Vault for each service, which contains all default ENV VARs for the service. Then, if a developer wants to override them or add new, they can create a new Vault secret, specific to the environment. For example:

---
apiVersion: "koudingspawn.de/v1"
kind: Vault
metadata:
  name: "kv-myservice-default-secrets"
spec:
  path: "kv-myservice/_default"
  type: "KEYVALUEV2"
  changeAdjustmentCallback:
    type: deployment
    name: "myservice"
{{ end }}

# Vault secrets for env (override the default-secrets)
---
apiVersion: "koudingspawn.de/v1"
kind: Vault
metadata:
  name: "kv-myservice-env-secrets"
spec:
  path: "kv-myservice/myenvname"
  type: "KEYVALUEV2"
  changeAdjustmentCallback:
    type: deployment
    name: "myservice"

Then in Deployment, we do this:

          envFrom:
            - secretRef:
                name: "kv-myservice-default-secrets"
            - secretRef:
                name: "kv-myservice-env-secrets"
                optional: true
            {{- end }}

—

So in our situation, it seems that having a Vault resource with no correspondent Vault secret is an "acceptable" state (although only for override secrets; the _default should always have a corresponding Vault secret).

Is it possible to somehow supress these events? Perhaps I missed something in the documentation. Otherwise, would you consider extending the resource API to cover such a use case? For example:

apiVersion: "koudingspawn.de/v1"
kind: Vault
metadata:
  name: "kv-myservice-env-secrets"
spec:
  path: "kv-myservice/myenvname"
  checkVaultSecretExistence: false
  type: "KEYVALUEV2"
  changeAdjustmentCallback:
    type: deployment
    name: "myservice"

( checkVaultSecretExistence: false)

It's just an example API I've come up with now, I'm sure there is a better name.

Env info

vault-crd version: 1.6.3
k8s version: 1.24 (EKS)

[DaspawnW]

Hi @awinecki ,

To get it right your expectation is that we don't send out events when a secret is not available and "ignore" them simply?
So more like a "supress warning"?

[awinecki]

@DaspawnW exactly.

I was suggesting the ability to disable the secret existence check. But suppressing the warning/log might be even better.

So something like:

apiVersion: "koudingspawn.de/v1"
kind: Vault
metadata:
  name: "kv-myservice-env-secrets"
spec:
  path: "kv-myservice/myenvname"
  suppressFailedModificationLogs: true
  type: "KEYVALUEV2"
  changeAdjustmentCallback:
    type: deployment
    name: "myservice"