Add monquorum alert

[devopsjonas]

Adds CephMonHighNumberOfLeaderChanges
Fixes CephMonQuorumAtRisk to add label selector. In the case where you have multiple clusters monitored by same Prometheus instance.

Example output:

"groups":
- "name": "quorum-alert.rules"
  "rules":
  - "alert": "CephMonQuorumAtRisk"
    "annotations":
      "description": "Storage cluster quorum is low. Contact Support."
      "message": "Storage quorum at risk"
      "severity_level": "error"
      "storage_type": "ceph"
    "expr": |
      count(ceph_mon_quorum_status{job="rook-ceph-mgr"} == 1) <= ((count(ceph_mon_metadata{job="rook-ceph-mgr"}) % 2) + 1)
    "for": "15m"
    "labels":
      "severity": "critical"
  - "alert": "CephMonHighNumberOfLeaderChanges"
    "annotations":
      "description": "Ceph Monitor \"{{ $labels.job }}\": instance {{ $labels.instance }} has seen {{ $value }} leader changes recently."
      "message": "Storage Cluster has seen many leader changes recently."
      "severity_level": "warning"
      "storage_type": "ceph"
    "expr": |
      rate(ceph_mon_num_elections{job="rook-ceph-mgr"}[15m]) > 3
    "for": "30m"
    "labels":
      "severity": "warning"

[anmolsachan]

Is this change made assuming that there will be multiple exporter jobs working at the same time ?

[devopsjonas]

Yes, I think it makes sense, as if you have 2 clusters monitored with same Prometheus the current version wouldn't work

[devopsjonas]

@anmolsachan fixed. PTAL 🙂

[shtripat]

There is another similar already present at https://github.com/ceph/ceph-mixins/blob/master/alerts/monquorum.libsonnet
How its different from that?

[devopsjonas]

@shtripat it's the same alert I didn't change it, except for adding a exporter selector {%s}

[shtripat]

Ack

[umangachapagain]

looks good to me.
Can you also run make all from project root and add the rule file generated?

[anmolsachan]

@devopsjonas PR looks good to me now. Please attach the generated manifest. You can refer to merged PR's for refenrence.

[devopsjonas]

Generated manifest:

"groups":
- "name": "quorum-alert.rules"
  "rules":
  - "alert": "CephMonQuorumAtRisk"
    "annotations":
      "description": "Storage cluster quorum is low. Contact Support."
      "message": "Storage quorum at risk"
      "severity_level": "error"
      "storage_type": "ceph"
    "expr": |
      count(ceph_mon_quorum_status{job="rook-ceph-mgr"} == 1) <= ((count(ceph_mon_metadata{job="rook-ceph-mgr"}) % 2) + 1)
    "for": "15m"
    "labels":
      "severity": "critical"
  - "alert": "CephMonHighNumberOfLeaderChanges"
    "annotations":
      "description": "Ceph Monitor \"{{ $labels.job }}\": instance {{ $labels.instance }} has seen {{ $value }} leader changes recently."
      "message": "Storage Cluster has seen many leader changes recently."
      "severity_level": "warning"
      "storage_type": "ceph"
    "expr": |
      rate(ceph_mon_num_elections{job="rook-ceph-mgr"}[15m]) > 3
    "for": "30m"
    "labels":
      "severity": "warning"

[devopsjonas]

@umangachapagain I've run:

make all            

jsonnet -S lib/alerts.jsonnet > prometheus_alert_rules.yaml
find . -name 'vendor' -prune -o -name '*.libsonnet' -print -o -name '*.jsonnet' -print | \                                             
        while read f; do \
                jsonnet fmt -n 2 --max-blank-lines 2 --string-style s --comment-style s "$f" | diff -u "$f" -; \                       
        done
promtool check rules prometheus_alert_rules.yaml
Checking prometheus_alert_rules.yaml
  SUCCESS: 11 rules found

git status

On branch add-monquorum-alert
Your branch is up to date with 'origin/add-monquorum-alert'.

nothing to commit, working tree clean

I believe .gitignore is making sure that we don't commit generated alerts.

[shtripat]

@devopsjonas You can run build.sh from extras dir. That will generate a rule file in manifests dir. That file you can check in.

[devopsjonas]

Done, PTAL :)

[shtripat]

LGTM

[shtripat]

@devopsjonas we have got unit test mechanism recently added to the project. I would expect you to add unit tests for the alerting rules as part of these PRs. Otherwise both the PRs looks good to be merged.
Feel free to send another PR for unit tests separately.

[devopsjonas]

@shtripat sure will do 👍

[shtripat]

Merging this now. Add another PR for unit tests.