Examples of Prometheus alerts

[MPV]

Anyone who's built PrometheusRule alerts for the metrics in KES yet (and would be up for sharing/collaborating on improvements)?

[EricHorst]

What do you have in mind for collaborating? We only use one alert rule to detect when ExternalSecrets are not sync'ing.

    - alert: SecretSyncError
      annotations:
        summary: SecretSyncError {{ $labels.name }}
        description: |
          Synchronization of the Secret {{ $labels.name }} in namespace {{ $labels.exported_namespace }} is failing.
          Check for errors in the Status field of the associated ExternalSecret object.
          See: https://our_wiki/x/ysnFBg
      expr: increase(sync_calls{exported_namespace=~"cert-manager|ingress-nginx|mci-.+",status="error"}[5m]) > 0
      for: 10m```

The only thing odd in this rule is that my group is only responsible for certain namespaces. Tenants use similar rules tuned for their namespaces.

[MPV]

Thanks for sharing @EricHorst. I find it helpful to getting newcomers (and discussions) started on how to observe KES. 👍

What do you have in mind for collaborating?

My thinking was inspired by the collaboratively created rules in the prometheus operator, for example:
https://github.com/prometheus-operator/kube-prometheus/blob/master/manifests/prometheus-rules.yaml

[MPV]

For anyone else coming here — I added this to get started:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kubernetes-external-secrets
  namespace: kubernetes-external-secrets
spec:
  groups:
    - name: kubernetes-external-secrets
      rules:
        - alert: SecretSyncError
          annotations:
            summary: SecretSyncError {{ $labels.name }}
            description: |
              Synchronization of the Secret `{{ $labels.name }}` in namespace `{{ $labels.exported_namespace }}` is failing.
              Check for errors in the Status field of the associated ExternalSecret object.
          expr: increase(sync_calls{status="error"}[5m]) > 0
          for: 10m