Add Alertmanager config and templates in Helm chart #188

TheoBrigitte · 2024-12-10T09:05:24Z

This PR does couple of things to get the Alertmanager into a Secret in the Helm chart:

Helm
- Add secret resource, embedding raw and templated alertmanager files
- Expose alertmanager templates values as helm chart values
Alertmanager
- Remove all Mimir related conditions from templates
- Escape template in template
- Split template into url and notification templates, to reduce template in template escaping
- Re-use slack actions, to reduce template in template escaping
- Drop template directive, dynamically set by the operator

How I generated the new Alertmanager config and notification template

Alertmanager config

wget https://raw.githubusercontent.com/giantswarm/prometheus-meta-operator/refs/heads/main/files/templates/alertmanager/alertmanager.yaml
sed -i -e 's/\[\[/{{/g' -e 's/\]\]/}}/g' alertmanager.yaml
patch alertmanager.yaml < <following patch>

13,15d12
< templates:
< - '/etc/alertmanager/config/*.tmpl'
<
24d20
<   {{- if .MimirEnabled }}
34d29
<   {{- end }}
173d167
< {{- if .MimirEnabled }}
187d180
< {{- end }}
206c199
<     actions:
---
>     actions: &slack-actions
209,210c202,203
<       url: '{{ template "__runbookurl" . }}'
<       style: '{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}'
---
>       url: {{`{{ template "__runbookurl" . }}`}}
>       style: {{`{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}`}}
213c206
<       url: '{{ template "__alert_linked_postmortems" . }}'
---
>       url: {{`{{ template "__alert_linked_postmortems" . }}`}}
216c209
<       url: '{{ template "__alerturl" . }}'
---
>       url: {{`{{ template "__alerturl" . }}`}}
219c212
<       url: '{{ template "__dashboardurl" . }}'
---
>       url: {{`{{ template "__dashboardurl" . }}`}}
222,223c215,216
<       url: '{{ template "__alert_silence_link" .}}'
<       style: '{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}'
---
>       url: {{`{{ template "__alert_silence_link" .}}`}}
>       style: {{`{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}`}}
242,259c235
<     actions:
<     - type: button
<       text: ':green_book: OpsRecipe'
<       url: '{{ template "__runbookurl" . }}'
<       style: '{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}'
<     - type: button
<       text: ':coffin: Linked PMs'
<       url: '{{ template "__alert_linked_postmortems" . }}'
<     - type: button
<       text: ':mag: Query'
<       url: '{{ template "__alerturl" . }}'
<     - type: button
<       text: ':grafana: Dashboard'
<       url: '{{ template "__dashboardurl" . }}'
<     - type: button
<       text: ':no_bell: Silence'
<       url: '{{ template "__alert_silence_link" .}}'
<       style: '{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}'
---
>     actions: *slack-actions
278,295c254
<     actions:
<     - type: button
<       text: ':green_book: OpsRecipe'
<       url: '{{ template "__runbookurl" . }}'
<       style: '{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}'
<     - type: button
<       text: ':coffin: Linked PMs'
<       url: '{{ template "__alert_linked_postmortems" . }}'
<     - type: button
<       text: ':mag: Query'
<       url: '{{ template "__alerturl" . }}'
<     - type: button
<       text: ':grafana: Dashboard'
<       url: '{{ template "__dashboardurl" . }}'
<     - type: button
<       text: ':no_bell: Silence'
<       url: '{{ template "__alert_silence_link" . }}'
<       style: '{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}'
---
>     actions: *slack-actions
314,331c273
<     actions:
<     - type: button
<       text: ':green_book: OpsRecipe'
<       url: '{{ template "__runbookurl" . }}'
<       style: '{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}'
<     - type: button
<       text: ':coffin: Linked PMs'
<       url: '{{ template "__alert_linked_postmortems" . }}'
<     - type: button
<       text: ':mag: Query'
<       url: '{{ template "__alerturl" . }}'
<     - type: button
<       text: ':grafana: Dashboard'
<       url: '{{ template "__dashboardurl" . }}'
<     - type: button
<       text: ':no_bell: Silence'
<       url: '{{ template "__alert_silence_link" . }}'
<       style: '{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}'
---
>     actions: *slack-actions
350,367c292
<     actions:
<     - type: button
<       text: ':green_book: OpsRecipe'
<       url: '{{ template "__runbookurl" . }}'
<       style: '{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}'
<     - type: button
<       text: ':coffin: Linked PMs'
<       url: '{{ template "__alert_linked_postmortems" . }}'
<     - type: button
<       text: ':mag: Query'
<       url: '{{ template "__alerturl" . }}'
<     - type: button
<       text: ':grafana: Dashboard'
<       url: '{{ template "__dashboardurl" . }}'
<     - type: button
<       text: ':no_bell: Silence'
<       url: '{{ template "__alert_silence_link" . }}'
<       style: '{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}'
---
>     actions: *slack-actions
382,399c307
<     actions:
<     - type: button
<       text: ':green_book: OpsRecipe'
<       url: '{{ template "__runbookurl" . }}'
<       style: '{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}'
<     - type: button
<       text: ':coffin: Linked PMs'
<       url: '{{ template "__alert_linked_postmortems" . }}'
<     - type: button
<       text: ':mag: Query'
<       url: '{{ template "__alerturl" . }}'
<     - type: button
<       text: ':grafana: Dashboard'
<       url: '{{ template "__dashboardurl" . }}'
<     - type: button
<       text: ':no_bell: Silence'
<       url: '{{ template "__alert_silence_link" .}}'
<       style: '{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}'
---
>     actions: *slack-actions
418,435c326
<     actions:
<     - type: button
<       text: ':green_book: OpsRecipe'
<       url: '{{ template "__runbookurl" . }}'
<       style: '{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}'
<     - type: button
<       text: ':coffin: Linked PMs'
<       url: '{{ template "__alert_linked_postmortems" . }}'
<     - type: button
<       text: ':mag: Query'
<       url: '{{ template "__alerturl" . }}'
<     - type: button
<       text: ':grafana: Dashboard'
<       url: '{{ template "__dashboardurl" . }}'
<     - type: button
<       text: ':no_bell: Silence'
<       url: '{{ template "__alert_silence_link" . }}'
<       style: '{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}'
---
>     actions: *slack-actions
450,467c341
<     actions:
<     - type: button
<       text: ':green_book: OpsRecipe'
<       url: '{{ template "__runbookurl" . }}'
<       style: '{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}'
<     - type: button
<       text: ':coffin: Linked PMs'
<       url: '{{ template "__alert_linked_postmortems" . }}'
<     - type: button
<       text: ':mag: Query'
<       url: '{{ template "__alerturl" . }}'
<     - type: button
<       text: ':grafana: Dashboard'
<       url: '{{ template "__dashboardurl" . }}'
<     - type: button
<       text: ':no_bell: Silence'
<       url: '{{ template "__alert_silence_link" .}}'
<       style: '{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}'
---
>     actions: *slack-actions
482,499c356
<     actions:
<     - type: button
<       text: ':green_book: OpsRecipe'
<       url: '{{ template "__runbookurl" . }}'
<       style: '{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}'
<     - type: button
<       text: ':coffin: Linked PMs'
<       url: '{{ template "__alert_linked_postmortems" . }}'
<     - type: button
<       text: ':mag: Query'
<       url: '{{ template "__alerturl" . }}'
<     - type: button
<       text: ':grafana: Dashboard'
<       url: '{{ template "__dashboardurl" . }}'
<     - type: button
<       text: ':no_bell: Silence'
<       url: '{{ template "__alert_silence_link" .}}'
<       style: '{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}'
---
>     actions: *slack-actions
504c361
<     tags: "{{ (index .Alerts 0).Labels.alertname }},{{ (index .Alerts 0).Labels.cluster_type }},{{ (index .Alerts 0).Labels.severity }},{{ (index .Alerts 0).Labels.team }},{{ (index .Alerts 0).Labels.area }},{{ (index .Alerts 0).Labels.service_priority }},{{ (index .Alerts 0).Labels.provider }},{{ (index .Alerts 0).Labels.installation }},{{ (index .Alerts 0).Labels.pipeline }},{{ (index .Alerts 0).Labels.customer }}"
---
>     tags: {{`{{ (index .Alerts 0).Labels.alertname }},{{ (index .Alerts 0).Labels.cluster_type }},{{ (index .Alerts 0).Labels.severity }},{{ (index .Alerts 0).Labels.team }},{{ (index .Alerts 0).Labels.area }},{{ (index .Alerts 0).Labels.service_priority }},{{ (index .Alerts 0).Labels.provider }},{{ (index .Alerts 0).Labels.installation }},{{ (index .Alerts 0).Labels.pipeline }},{{ (index .Alerts 0).Labels.customer }}`}}

Notification template

wget https://raw.githubusercontent.com/giantswarm/prometheus-meta-operator/refs/heads/main/files/templates/alertmanager/notification-template.tmpl
patch notification-template.tmpl < <following patch>

3,21d2
< {{ define "__alerturl" }}
< [[- if .MimirEnabled -]]
< [[ .GrafanaAddress ]]/alerting/Mimir/{{ .CommonLabels.alertname }}/find
< [[- else -]]
< {{ .ExternalURL }}/#/alerts?receiver={{ .Receiver }}&silenced=false&inhibited=false&active=true&filter=%7Balertname%3D%22{{ .CommonLabels.alertname }}%22%7D
< [[- end -]]
< {{ end }}
<
< {{ define "__dashboardurl" -}}{{ if match "^https://.+" (index .Alerts 0).Annotations.dashboard }}{{ (index .Alerts 0).Annotations.dashboard }}{{ else }}[[ .GrafanaAddress ]]/d/{{ (index .Alerts 0).Annotations.dashboard }}{{ end }}{{- end }}
< {{ define "__runbookurl" -}}https://intranet.giantswarm.io/docs/support-and-ops/ops-recipes/{{ (index .Alerts 0).Annotations.opsrecipe }}{{- end }}
<
< {{ define "__queryurl" }}
< [[- if .MimirEnabled -]]
< [[ .GrafanaAddress ]]/alerting/Mimir/{{ .CommonLabels.alertname }}/find
< [[- else -]]
< {{ (index .Alerts 0).GeneratorURL }}
< [[- end -]]
< {{ end }}
<
59d39
< [[- if .MimirEnabled ]]
61,64d40
< [[- else ]]
< 🔔 Alertmanager {{ template "__alerturl" . }}
< 👀 Query: {{ template "__queryurl" . }}
< [[- end ]]

I would like some opinions before I continue in this direction, because I feel there are a lot of workaround here to get this config into a Secret and it could be easier to have it directly in code. Also does anyone remember what the ProxyURL is used for ? It seems to be OpsGenie related but I gladly ignored it and I have a feeling this could be important.

QuentinBisson · 2024-12-10T10:40:26Z

helm/observability-operator/files/alertmanager/alertmanager.yaml.helm-template

+  http_config:
+    proxy_url:  {{ .alertmanager.proxyURL }}
+{{- end }}
+{{- if .alertmanager.slackAPIToken }}


Is there a CAPA installation where we do not use slackAPIToken?

I think not, but it doesn't hurt to keep it for now, I'd rather be safe.

QuentinBisson · 2024-12-10T10:42:29Z

helm/observability-operator/files/alertmanager/alertmanager.yaml.helm-template

+    send_resolved: true
+    actions: *slack-actions
+
+- name: team_turtles_slack


This maybe should go away

QuentinBisson · 2024-12-10T10:45:15Z

helm/observability-operator/files/alertmanager/url-template.tmpl.helm-template

@@ -0,0 +1,18 @@
+{{`
+{{ define "__alerturl" }}
+`}}{{ .alertmanager.grafanaAddress }}{{`/alerting/Mimir/{{ .CommonLabels.alertname }}/find


I think we should be able to replace this with the grafanaExploreURL or this queryFromGeneratorURL instead https://grafana.com/docs/mimir/latest/references/architecture/components/alertmanager/#templating right?

Yes I'll look into this later

I'm trying to see if we can get rid of this whole file. that would make things simpler imo

helm/observability-operator/templates/alertmanager/_helpers.tpl

helm/observability-operator/files/alertmanager/url-template.tmpl.helm-template

helm/observability-operator/values.yaml

helm/observability-operator/files/alertmanager/notification-template.tmpl

QuentinBisson · 2024-12-10T10:59:38Z

helm/observability-operator/files/alertmanager/notification-template.tmpl

+
+# Link to related PMs
+{{ define "__alert_linked_postmortems" -}}
+https://github.com/giantswarm/giantswarm/issues?q=is%3Aissue+is%3Aopen+label%3Apostmortem+label%3Aalert%2F{{ .CommonLabels.alertname }}


Same remark but that's not that urgent

This is used as a link on Slack

- Add secret resource, embedding raw and templated alertmanager files - Expose alertmanager templates values as helm chart values - Remove all Mimir related conditions - Split template into url and notification templates - Drop template directive, dynamically set by the operator - Escape template in template - Re-use slack actions

…asepath from values inside tpl function

This fixes the infamous: error calling tpl: cannot retrieve Template.Basepath from values inside tpl function It does use .Values in templates to access values and pass $ root context to tpl

Co-authored-by: Quentin Bisson <[email protected]>

helm/observability-operator/Chart.yaml

QuentinBisson · 2024-12-16T13:42:11Z

helm/observability-operator/values.yaml

+alerting:
+  alertmanagerURL: ""
+  grafanaAddress: ""
+  proxyURL: ""


proxy url should be coming from an env variable right?

QuentinBisson · 2024-12-16T13:45:43Z

helm/observability-operator/files/alertmanager/alertmanager.yaml.helm-template

+  resolve_timeout: 5m
+{{- if .Values.alerting.proxyURL }}
+  http_config:
+    proxy_url:  {{ .Values.alerting.proxyURL }}


I think this is not useful anymore with the env variables automatically set on all pods. Do you think you could test without it on goat? Because I'm not sure we can get this value anyway in config as it is set by kyverno policies

QuentinBisson · 2024-12-16T13:46:42Z

Apart from my proxy url questions, I'm fine with this for now but I'd rather we remove the url mumbo jumbo later on

This reverts commit 283c100.

QuentinBisson · 2024-12-17T13:10:36Z

helm/observability-operator/files/alertmanager/notification-template.tmpl

+* Team: {{ (index .Alerts 0).Labels.team }}
+* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}
+* Instances:{{ range .Alerts.Firing }}
+🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}
 {{- end }}

 # This builds the silence URL.  We exclude the alertname in the range


I think we need to remove this silence URL right?

TheoBrigitte requested a review from a team as a code owner December 10, 2024 09:05

TheoBrigitte self-assigned this Dec 10, 2024

TheoBrigitte changed the base branch from main to alertmanager-config December 10, 2024 09:06

QuentinBisson reviewed Dec 10, 2024

View reviewed changes

helm/observability-operator/templates/alertmanager/_helpers.tpl Outdated Show resolved Hide resolved

QuentinBisson reviewed Dec 10, 2024

View reviewed changes

helm/observability-operator/files/alertmanager/url-template.tmpl.helm-template Show resolved Hide resolved

QuentinBisson reviewed Dec 10, 2024

View reviewed changes

helm/observability-operator/values.yaml Outdated Show resolved Hide resolved

QuentinBisson reviewed Dec 10, 2024

View reviewed changes

helm/observability-operator/files/alertmanager/notification-template.tmpl Show resolved Hide resolved

QuentinBisson reviewed Dec 10, 2024

View reviewed changes

TheoBrigitte force-pushed the alertmanager-config-helm branch from 4b4c161 to 11abd5f Compare December 10, 2024 15:18

TheoBrigitte mentioned this pull request Dec 10, 2024

Alertmanager test branch #192

Closed

TheoBrigitte changed the base branch from alertmanager-config to main December 10, 2024 17:45

TheoBrigitte and others added 8 commits December 10, 2024 18:48

use .helm-template instead of .tpl

9eea3f8

re-use existing values

27b62fa

normalize values names

49b5a1a

try to make helm happy: error calling tpl: cannot retrieve Template.B…

2ea1101

…asepath from values inside tpl function

helm 3.10.3 I hate you deeply

801cadd

This fixes the infamous: error calling tpl: cannot retrieve Template.Basepath from values inside tpl function It does use .Values in templates to access values and pass $ root context to tpl

Release v0.10.0 (#191)

8cbdbfb

Co-authored-by: Quentin Bisson <[email protected]>

quote the tmp_string

5d320b9

TheoBrigitte force-pushed the alertmanager-config-helm branch from 9d0c548 to 5d320b9 Compare December 10, 2024 17:50

Add alertmanagerURL

4cfbb20

TheoBrigitte mentioned this pull request Dec 10, 2024

Investigation: Migrate the existing alerts to mimir alertmanager giantswarm/roadmap#3746

Open

3 tasks

QuentinBisson reviewed Dec 10, 2024

View reviewed changes

helm/observability-operator/Chart.yaml Outdated Show resolved Hide resolved

TheoBrigitte added 2 commits December 13, 2024 12:47

Merge remote-tracking branch 'origin/main' into alertmanager-config-helm

492a1d7

rename alertmanager to alerting in values.yaml

9591a92

TheoBrigitte mentioned this pull request Dec 15, 2024

Implement handling of Alertmanager config #180

Merged

QuentinBisson reviewed Dec 16, 2024

View reviewed changes

TheoBrigitte added 9 commits December 16, 2024 19:18

Merge remote-tracking branch 'origin/main' into alertmanager-config-helm

04d6193

set alertmanager controller name

283c100

Revert "set alertmanager controller name"

ec33d05

This reverts commit 283c100.

use mimir url generators;kill the new lines

cd01322

add templates

0b081dc

Use correct datasource, with uid :/

1a42e53

Attempt at making dashboard url a single line

eda8f3a

Align with new notification-template changes

9e09827

fix explore link new line

52901c4

QuentinBisson reviewed Dec 17, 2024

View reviewed changes

TheoBrigitte added 5 commits December 17, 2024 14:47

Merge remote-tracking branch 'origin/main' into alertmanager-config-helm

2b3730c

Add the silence link

253a288

update silence link (again)

51d0f0b

Use proxy_from_environment in Alertmanager

270467c

Merge branch 'main' into alertmanager-config-helm

76d694d

QuentinBisson approved these changes Jan 6, 2025

View reviewed changes

TheoBrigitte added 3 commits January 6, 2025 11:23

remove proxyURL from values

1a5e321

add comments in url-template

cb2a4b3

Merge branch 'main' into alertmanager-config-helm

78b8e4a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Alertmanager config and templates in Helm chart #188

Add Alertmanager config and templates in Helm chart #188

TheoBrigitte commented Dec 10, 2024 •

edited

Loading

QuentinBisson Dec 10, 2024

TheoBrigitte Dec 10, 2024

QuentinBisson Dec 10, 2024

TheoBrigitte Dec 10, 2024

QuentinBisson Dec 10, 2024

TheoBrigitte Dec 10, 2024

QuentinBisson Dec 10, 2024

QuentinBisson Dec 10, 2024

TheoBrigitte Dec 17, 2024

QuentinBisson Dec 16, 2024

QuentinBisson Dec 16, 2024 •

edited

Loading

QuentinBisson commented Dec 16, 2024

QuentinBisson Dec 17, 2024

Add Alertmanager config and templates in Helm chart #188

Are you sure you want to change the base?

Add Alertmanager config and templates in Helm chart #188

Conversation

TheoBrigitte commented Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

QuentinBisson Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

QuentinBisson commented Dec 16, 2024

Choose a reason for hiding this comment

TheoBrigitte commented Dec 10, 2024 •

edited

Loading

QuentinBisson Dec 16, 2024 •

edited

Loading