Skip to content

Commit

Permalink
docs: notification wait for & silences
Browse files Browse the repository at this point in the history
  • Loading branch information
adityathebe authored and moshloop committed Oct 7, 2024
1 parent 97dd788 commit ab947cd
Show file tree
Hide file tree
Showing 4 changed files with 65 additions and 0 deletions.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
28 changes: 28 additions & 0 deletions mission-control/docs/notifications/concepts/silences.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
---
title: Silences
---

Silences prevent notifications from being sent.
Silences are attached to a particular resource (`catalog`, `health check`, or `component`) and are active for a specified duration.

:::note
Notifications that aren't sent due to silence are still visible in the notification history for auditing purposes.
:::

## Use cases

- Planned maintenance or deployments. Eg: You can silence a namespace or a helm release and automatically silence notifications from all of their children.

Check warning on line 14 in mission-control/docs/notifications/concepts/silences.mdx

View workflow job for this annotation

GitHub Actions / vale

[vale] mission-control/docs/notifications/concepts/silences.mdx#L14

[Flanksource.Wordiness] Consider using 'all' instead of 'all of'.
Raw output
{"message": "[Flanksource.Wordiness] Consider using 'all' instead of 'all of'.", "location": {"path": "mission-control/docs/notifications/concepts/silences.mdx", "range": {"start": {"line": 14, "column": 134}}}, "severity": "WARNING"}
- Non-critical resources: Notifications from resources that routinely trigger alerts but are expected and harmless can be silenced.
- Known issues: If there's a known issue that can't be immediately resolved (e.g., due to dependencies or resource constraints), you might silence related alerts until a fix can be implemented.

## Add Silence

Silences can be added from the notification page. Alternatively, if you're using the default slack notification templates, you get a silence button
on each notification.

![Silence Notification form](./silence-notification-form.png)

## Recursive mode

When a silence is recursively applied, it applies to all of its children.

Check warning on line 27 in mission-control/docs/notifications/concepts/silences.mdx

View workflow job for this annotation

GitHub Actions / vale

[vale] mission-control/docs/notifications/concepts/silences.mdx#L27

[Flanksource.Wordiness] Consider using 'all' instead of 'all of'.
Raw output
{"message": "[Flanksource.Wordiness] Consider using 'all' instead of 'all of'.", "location": {"path": "mission-control/docs/notifications/concepts/silences.mdx", "range": {"start": {"line": 27, "column": 54}}}, "severity": "WARNING"}
So silencing a namespace would silence all deployments, statefulsets, pods, etc in that namespace.

Check failure on line 28 in mission-control/docs/notifications/concepts/silences.mdx

View workflow job for this annotation

GitHub Actions / vale

[vale] mission-control/docs/notifications/concepts/silences.mdx#L28

[Flanksource.Spelling] Is 'statefulsets' spelled correctly? Is it missing code formatting?
Raw output
{"message": "[Flanksource.Spelling] Is 'statefulsets' spelled correctly? Is it missing code formatting?", "location": {"path": "mission-control/docs/notifications/concepts/silences.mdx", "range": {"start": {"line": 28, "column": 57}}}, "severity": "ERROR"}
32 changes: 32 additions & 0 deletions mission-control/docs/notifications/concepts/wait-for.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
---
title: Wait For
---

Kubernetes clusters and similar dynamic systems may experience temporary discrepancies between the actual and intended state of resources.
For example, a deployment could momentarily appear unhealthy during a scaling operation.
If alerts are configured for `config.unhealthy` events, these transient state fluctuations might lead to an overwhelming number of unnecessary notifications.

To address this issue, you can utilize the waitFor parameter.
This feature allows you to define a delay before sending notifications for specific events.
After an event occurs, the system will recheck its status following the specified wait period. Only if the undesired state persists will a notification be triggered.

Check warning on line 11 in mission-control/docs/notifications/concepts/wait-for.mdx

View workflow job for this annotation

GitHub Actions / vale

[vale] mission-control/docs/notifications/concepts/wait-for.mdx#L11

[Flanksource.FutureTense] Use present tense verbs, not future tense verbs like 'will'. Say '(event) happens' instead of '(event) will happen'.
Raw output
{"message": "[Flanksource.FutureTense] Use present tense verbs, not future tense verbs like 'will'. Say '(event) happens' instead of '(event) will happen'.", "location": {"path": "mission-control/docs/notifications/concepts/wait-for.mdx", "range": {"start": {"line": 11, "column": 35}}}, "severity": "WARNING"}

Check warning on line 11 in mission-control/docs/notifications/concepts/wait-for.mdx

View workflow job for this annotation

GitHub Actions / vale

[vale] mission-control/docs/notifications/concepts/wait-for.mdx#L11

[Flanksource.ProfanityUnlikely] Be careful with 'period', it's profane in some cases.
Raw output
{"message": "[Flanksource.ProfanityUnlikely] Be careful with 'period', it's profane in some cases.", "location": {"path": "mission-control/docs/notifications/concepts/wait-for.mdx", "range": {"start": {"line": 11, "column": 88}}}, "severity": "WARNING"}

Check warning on line 11 in mission-control/docs/notifications/concepts/wait-for.mdx

View workflow job for this annotation

GitHub Actions / vale

[vale] mission-control/docs/notifications/concepts/wait-for.mdx#L11

[Flanksource.FutureTense] Use present tense verbs, not future tense verbs like 'will'. Say '(event) happens' instead of '(event) will happen'.
Raw output
{"message": "[Flanksource.FutureTense] Use present tense verbs, not future tense verbs like 'will'. Say '(event) happens' instead of '(event) will happen'.", "location": {"path": "mission-control/docs/notifications/concepts/wait-for.mdx", "range": {"start": {"line": 11, "column": 133}}}, "severity": "WARNING"}

:::info
`waitFor` is only applicable on health related events
:::

This approach helps reduce unnecessary notifications caused by transient state changes, ensuring you're alerted only to persistent issues.


```yaml title='notify-unhealthy-deployments.yaml' {8}
apiVersion: mission-control.flanksource.com/v1
kind: Notification
metadata:
name: deployment-unhealthy-alerts
spec:
events:
- config.unhealthy
waitFor: 2m
filter: config.type == 'Kubernetes::Deployment'
to:
email: [email protected]
```
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,11 @@
description: "RepeatGroup allows notifications to be grouped by certain set of keys and only send one per group within the specified repeat interval. Valid group keys: `resource_id` & `source_event`.",
scheme: "`[]string`"
},
{
field: "waitFor",
description: "The duration to delay sending a health-based notification.\n\nAfter this period, the health status is reassessed to confirm it hasn't changed, helping prevent false alarms from transient issues.",

Check warning on line 50 in mission-control/docs/reference/notifications/_notification.mdx

View workflow job for this annotation

GitHub Actions / vale

[vale] mission-control/docs/reference/notifications/_notification.mdx#L50

[Flanksource.ProfanityUnlikely] Be careful with 'period', it's profane in some cases.
Raw output
{"message": "[Flanksource.ProfanityUnlikely] Be careful with 'period', it's profane in some cases.", "location": {"path": "mission-control/docs/reference/notifications/_notification.mdx", "range": {"start": {"line": 50, "column": 92}}}, "severity": "WARNING"}
scheme: "duration"
},
{
field: "title",
description: "Channel dependent e.g. subject for email",
Expand Down

0 comments on commit ab947cd

Please sign in to comment.