Skip to content

Commit

Permalink
change the alert for the openshift-image-mirroring failures to only f…
Browse files Browse the repository at this point in the history
…ire when there have been no successful runs for 12 hours, and add the keep_firing_for to 2h so that alert manager won't notify us too frequently when this is the case (openshift#60057)
  • Loading branch information
smg247 authored and yingzhanredhat committed Dec 24, 2024
1 parent 47a81aa commit 4ec8fef
Show file tree
Hide file tree
Showing 2 changed files with 14 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -241,9 +241,15 @@
{
alert: 'openshift-mirroring-failures',
expr: |||
increase(prowjob_state_transitions{job_name="periodic-image-mirroring-openshift",state="failure"}[5m]) > 0
sum by (job_name) (
rate(
prowjob_state_transitions{job="prow-controller-manager",job_name!~"rehearse.*",state="success"}[12h]
)
)
* on (job_name) group_left max by (job_name) (prow_job_labels{job_agent="kubernetes",label_ci_openshift_io_role="image-mirroring",label_ci_openshift_io_area="openshift"}) == 0
|||,
'for': '1m',
'keep_firing_for': '2h',
labels: {
severity: 'critical',
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -292,8 +292,14 @@ spec:
annotations:
message: OpenShift image mirroring jobs have failed. View failed jobs at the <https://prow.ci.openshift.org/?job=periodic-image-mirroring-openshift|overview>.
expr: |
increase(prowjob_state_transitions{job_name="periodic-image-mirroring-openshift",state="failure"}[5m]) > 0
sum by (job_name) (
rate(
prowjob_state_transitions{job="prow-controller-manager",job_name!~"rehearse.*",state="success"}[12h]
)
)
* on (job_name) group_left max by (job_name) (prow_job_labels{job_agent="kubernetes",label_ci_openshift_io_role="image-mirroring",label_ci_openshift_io_area="openshift"}) == 0
for: 1m
keep_firing_for: 2h
labels:
severity: critical
- name: ghproxy
Expand Down

0 comments on commit 4ec8fef

Please sign in to comment.