-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
- Loading branch information
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
--- | ||
title: Silences | ||
--- | ||
|
||
Silences prevent notifications from being sent. | ||
Silences are attached to a particular resource (`catalog`, `health check`, or `component`) and are active for a specified duration. | ||
|
||
:::note | ||
Notifications that aren't sent due to silence are still visible in the notification history for auditing purposes. | ||
::: | ||
|
||
## Use cases | ||
|
||
- Planned maintenance or deployments. Eg: You can silence a namespace or a helm release and automatically silence notifications from all of their children. | ||
Check warning on line 14 in mission-control/docs/notifications/concepts/silences.mdx GitHub Actions / vale[vale] mission-control/docs/notifications/concepts/silences.mdx#L14
Raw output
|
||
- Non-critical resources: Notifications from resources that routinely trigger alerts but are expected and harmless can be silenced. | ||
- Known issues: If there's a known issue that can't be immediately resolved (e.g., due to dependencies or resource constraints), you might silence related alerts until a fix can be implemented. | ||
|
||
## Add Silence | ||
|
||
Silences can be added from the notification page. Alternatively, if you're using the default slack notification templates, you get a silence button | ||
on each notification. | ||
|
||
![Silence Notification form](./silence-notification-form.png) | ||
|
||
## Recursive mode | ||
|
||
When a silence is recursively applied, it applies to all of its children. | ||
Check warning on line 27 in mission-control/docs/notifications/concepts/silences.mdx GitHub Actions / vale[vale] mission-control/docs/notifications/concepts/silences.mdx#L27
Raw output
|
||
So silencing a namespace would silence all deployments, statefulsets, pods, etc in that namespace. | ||
Check failure on line 28 in mission-control/docs/notifications/concepts/silences.mdx GitHub Actions / vale[vale] mission-control/docs/notifications/concepts/silences.mdx#L28
Raw output
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
--- | ||
title: Wait For | ||
--- | ||
|
||
Kubernetes clusters and similar dynamic systems may experience temporary discrepancies between the actual and intended state of resources. | ||
For example, a deployment could momentarily appear unhealthy during a scaling operation. | ||
If alerts are configured for `config.unhealthy` events, these transient state fluctuations might lead to an overwhelming number of unnecessary notifications. | ||
|
||
To address this issue, you can utilize the waitFor parameter. | ||
This feature allows you to define a delay before sending notifications for specific events. | ||
After an event occurs, the system will recheck its status following the specified wait period. Only if the undesired state persists will a notification be triggered. | ||
Check warning on line 11 in mission-control/docs/notifications/concepts/wait-for.mdx GitHub Actions / vale[vale] mission-control/docs/notifications/concepts/wait-for.mdx#L11
Raw output
Check warning on line 11 in mission-control/docs/notifications/concepts/wait-for.mdx GitHub Actions / vale[vale] mission-control/docs/notifications/concepts/wait-for.mdx#L11
Raw output
Check warning on line 11 in mission-control/docs/notifications/concepts/wait-for.mdx GitHub Actions / vale[vale] mission-control/docs/notifications/concepts/wait-for.mdx#L11
Raw output
|
||
|
||
:::info | ||
`waitFor` is only applicable on health related events | ||
::: | ||
|
||
This approach helps reduce unnecessary notifications caused by transient state changes, ensuring you're alerted only to persistent issues. | ||
|
||
|
||
```yaml title='notify-unhealthy-deployments.yaml' {8} | ||
apiVersion: mission-control.flanksource.com/v1 | ||
kind: Notification | ||
metadata: | ||
name: deployment-unhealthy-alerts | ||
spec: | ||
events: | ||
- config.unhealthy | ||
waitFor: 2m | ||
filter: config.type == 'Kubernetes::Deployment' | ||
to: | ||
email: [email protected] | ||
``` |