Alert creation delay based on user definition #173009

shanisagiv1 · 2023-12-10T09:39:55Z

Describe the feature:
In order to reduce noise for alerting rules with low sensitivity and ensure created alerts will be actionable and reasonable, we want to allow users to define how many rule executions should match before creating the alert.

Describe a specific use case for the feature:

The user should be able to define a new input X. This input should be available for each rule type. This X should be called "Alert Consecutive matches before alert generation" / "Rule query threshold should met X times before alert generations" - TBD about the label..
When determining this input, the rule should run X times and match X times before creating alerts (and triggering its actions). e.g: the rule will run 4 times and has to match the defined threshold 4 times before creating the alert.
the number of consecutive execution should be available on the created alerts (e.g rule runs and was matched 6 times before alert creation)
This input won't affect the alert recovery, which means alert recovery will work as today, when a rule returns no results, life-cycled alerts are recovered automatically.
Out of scope for this task, but might be an extension in the future is "Delayed Recovery", which means we'll have to leverage the same X for delay the recovery based the same logic or to use a separated Y for that. but it's out of scope for this task
This logic should be implemented for life-cycled alerts (stack and o11y) only and be available in the current rule flyout and API.

Related to this: #146220

elasticmachine · 2023-12-10T09:40:49Z

Pinging @elastic/response-ops (Team:ResponseOps)

Erikg346 · 2024-01-05T18:32:21Z

To add an example of why this can be useful:
I would like to run a rule that calculates average CPU usage and only alerts when the average CPU is consistently hitting the threshold 15 times. The reason for this we get spikes in CPU usage but it's ideal to only alert when it's consistent for 15 minutes or 15 times.

@timestamp

) Related to #173009 ## Summary This is the first of two PRs and only focuses on the backend implementation. This PR adds a new `notificationDelay` field to the `Rule` object. With the delay the rule will run X times and has to match the threshold X times before triggering actions. It won't affect the alert recovery, but it can be expanded on easily if we want to include recovered alerts in the future. ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify - Use [Dev Tools](http://localhost:5601/app/dev_tools#/console) to create a rule with the `notificationDelay` ``` POST kbn:/api/alerting/rule { "params": { "searchType": "esQuery", "timeWindowSize": 5, "timeWindowUnit": "m", "threshold": [ -1 ], "thresholdComparator": ">", "size": 100, "esQuery": """{ "query":{ "match_all" : {} } }""", "aggType": "count", "groupBy": "all", "termSize": 5, "excludeHitsFromPreviousRun": false, "sourceFields": [], "index": [ ".kibana-event-log*" ], "timeField": "@timestamp" }, "consumer": "stackAlerts", "schedule": { "interval": "1m" }, "tags": [], "name": "test", "rule_type_id": ".es-query", "actions": [ { "group": "query matched", "id": "${ACTION_ID}", "params": { "level": "info", "message": """Elasticsearch query rule '{{rule.name}}' is active: - Value: {{context.value}} - Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}} - Timestamp: {{context.date}} - Link: {{context.link}}""" }, "frequency": { "notify_when": "onActionGroupChange", "throttle": null, "summary": false } } ], "notification_delay": { "active": 3 } } ``` - Verify that the rule will not trigger actions until it has matched the delay threshold. It might be helpful to look at rule details page and add the Triggered actions column to easily see the action was triggered after X consecutive active alerts <img width="1420" alt="Screen Shot 2024-01-16 at 1 18 52 PM" src="https://github.com/elastic/kibana/assets/109488926/85d8ceef-042c-4a52-950e-24492dc0e79f"> - Verify that the delay does not affect recovered alerts

@timestamp

Resolves #173009 ## Summary This PR: - Changes the field name from `notification_delay` to `alert_delay` - Updates the alerts client and rule registry to index new alert docs on a delay - Updates the framework code to delay the creation of an alert ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify - Use [Dev Tools](http://localhost:5601/app/dev_tools#/console) to create a rule with the `alertDelay` ``` POST kbn:/api/alerting/rule { "params": { "searchType": "esQuery", "timeWindowSize": 5, "timeWindowUnit": "m", "threshold": [ -1 ], "thresholdComparator": ">", "size": 100, "esQuery": """{ "query":{ "match_all" : {} } }""", "aggType": "count", "groupBy": "all", "termSize": 5, "excludeHitsFromPreviousRun": false, "sourceFields": [], "index": [ ".kibana-event-log*" ], "timeField": "@timestamp" }, "consumer": "stackAlerts", "schedule": { "interval": "1m" }, "tags": [], "name": "test", "rule_type_id": ".es-query", "actions": [ { "group": "query matched", "id": "${ACTION_ID}", "params": { "level": "info", "message": """Elasticsearch query rule '{{rule.name}}' is active: - Value: {{context.value}} - Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}} - Timestamp: {{context.date}} - Link: {{context.link}}""" }, "frequency": { "notify_when": "onActionGroupChange", "throttle": null, "summary": false } } ], "alert_delay": { "active": 3 } } ``` - Verify that the alert will not be created until it has matched the delay threshold. - Verify that the delay does not affect recovered alerts

@timestamp

…175851) Resolves elastic#173009 ## Summary This PR: - Changes the field name from `notification_delay` to `alert_delay` - Updates the alerts client and rule registry to index new alert docs on a delay - Updates the framework code to delay the creation of an alert ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify - Use [Dev Tools](http://localhost:5601/app/dev_tools#/console) to create a rule with the `alertDelay` ``` POST kbn:/api/alerting/rule { "params": { "searchType": "esQuery", "timeWindowSize": 5, "timeWindowUnit": "m", "threshold": [ -1 ], "thresholdComparator": ">", "size": 100, "esQuery": """{ "query":{ "match_all" : {} } }""", "aggType": "count", "groupBy": "all", "termSize": 5, "excludeHitsFromPreviousRun": false, "sourceFields": [], "index": [ ".kibana-event-log*" ], "timeField": "@timestamp" }, "consumer": "stackAlerts", "schedule": { "interval": "1m" }, "tags": [], "name": "test", "rule_type_id": ".es-query", "actions": [ { "group": "query matched", "id": "${ACTION_ID}", "params": { "level": "info", "message": """Elasticsearch query rule '{{rule.name}}' is active: - Value: {{context.value}} - Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}} - Timestamp: {{context.date}} - Link: {{context.link}}""" }, "frequency": { "notify_when": "onActionGroupChange", "throttle": null, "summary": false } } ], "alert_delay": { "active": 3 } } ``` - Verify that the alert will not be created until it has matched the delay threshold. - Verify that the delay does not affect recovered alerts

mikecote · 2024-02-08T12:51:19Z

Reopening until #176346 is completed.

@timestamp

…tic#174657) Related to elastic#173009 ## Summary This is the first of two PRs and only focuses on the backend implementation. This PR adds a new `notificationDelay` field to the `Rule` object. With the delay the rule will run X times and has to match the threshold X times before triggering actions. It won't affect the alert recovery, but it can be expanded on easily if we want to include recovered alerts in the future. ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify - Use [Dev Tools](http://localhost:5601/app/dev_tools#/console) to create a rule with the `notificationDelay` ``` POST kbn:/api/alerting/rule { "params": { "searchType": "esQuery", "timeWindowSize": 5, "timeWindowUnit": "m", "threshold": [ -1 ], "thresholdComparator": ">", "size": 100, "esQuery": """{ "query":{ "match_all" : {} } }""", "aggType": "count", "groupBy": "all", "termSize": 5, "excludeHitsFromPreviousRun": false, "sourceFields": [], "index": [ ".kibana-event-log*" ], "timeField": "@timestamp" }, "consumer": "stackAlerts", "schedule": { "interval": "1m" }, "tags": [], "name": "test", "rule_type_id": ".es-query", "actions": [ { "group": "query matched", "id": "${ACTION_ID}", "params": { "level": "info", "message": """Elasticsearch query rule '{{rule.name}}' is active: - Value: {{context.value}} - Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}} - Timestamp: {{context.date}} - Link: {{context.link}}""" }, "frequency": { "notify_when": "onActionGroupChange", "throttle": null, "summary": false } } ], "notification_delay": { "active": 3 } } ``` - Verify that the rule will not trigger actions until it has matched the delay threshold. It might be helpful to look at rule details page and add the Triggered actions column to easily see the action was triggered after X consecutive active alerts <img width="1420" alt="Screen Shot 2024-01-16 at 1 18 52 PM" src="https://github.com/elastic/kibana/assets/109488926/85d8ceef-042c-4a52-950e-24492dc0e79f"> - Verify that the delay does not affect recovered alerts

@timestamp

…175851) Resolves elastic#173009 ## Summary This PR: - Changes the field name from `notification_delay` to `alert_delay` - Updates the alerts client and rule registry to index new alert docs on a delay - Updates the framework code to delay the creation of an alert ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify - Use [Dev Tools](http://localhost:5601/app/dev_tools#/console) to create a rule with the `alertDelay` ``` POST kbn:/api/alerting/rule { "params": { "searchType": "esQuery", "timeWindowSize": 5, "timeWindowUnit": "m", "threshold": [ -1 ], "thresholdComparator": ">", "size": 100, "esQuery": """{ "query":{ "match_all" : {} } }""", "aggType": "count", "groupBy": "all", "termSize": 5, "excludeHitsFromPreviousRun": false, "sourceFields": [], "index": [ ".kibana-event-log*" ], "timeField": "@timestamp" }, "consumer": "stackAlerts", "schedule": { "interval": "1m" }, "tags": [], "name": "test", "rule_type_id": ".es-query", "actions": [ { "group": "query matched", "id": "${ACTION_ID}", "params": { "level": "info", "message": """Elasticsearch query rule '{{rule.name}}' is active: - Value: {{context.value}} - Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}} - Timestamp: {{context.date}} - Link: {{context.link}}""" }, "frequency": { "notify_when": "onActionGroupChange", "throttle": null, "summary": false } } ], "alert_delay": { "active": 3 } } ``` - Verify that the alert will not be created until it has matched the delay threshold. - Verify that the delay does not affect recovered alerts

) Resolves #173009 ## Summary Adds a new input for the user to define the `alertDelay`. This input is available for life-cycled alerts (stack and o11y) rule types. ### Checklist Delete any items that are not applicable to this PR. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify - Using the UI create a rule with the `alertDelay` field set. - Verify that the field is saved properly and that you can edit the `alertDelay` - Verify that you can add the alert delay to existing rules. Create a rule in a different branch and switch to this one. Edit the rule and set the `alertDelay`. Verify that the rule saves and works as expected. --------- Co-authored-by: Lisa Cawley <[email protected]>

…tic#176346) Resolves elastic#173009 ## Summary Adds a new input for the user to define the `alertDelay`. This input is available for life-cycled alerts (stack and o11y) rule types. ### Checklist Delete any items that are not applicable to this PR. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify - Using the UI create a rule with the `alertDelay` field set. - Verify that the field is saved properly and that you can edit the `alertDelay` - Verify that you can add the alert delay to existing rules. Create a rule in a different branch and switch to this one. Edit the rule and set the `alertDelay`. Verify that the rule saves and works as expected. --------- Co-authored-by: Lisa Cawley <[email protected]> (cherry picked from commit 68d6ab2)

#176346) (#177051) # Backport This will backport the following commits from `main` to `8.13`: - [[ResponseOps][FE] Alert creation delay based on user definition (#176346)](#176346)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Alexi Doak <[email protected]>

@timestamp

…175851) Resolves elastic#173009 ## Summary This PR: - Changes the field name from `notification_delay` to `alert_delay` - Updates the alerts client and rule registry to index new alert docs on a delay - Updates the framework code to delay the creation of an alert ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify - Use [Dev Tools](http://localhost:5601/app/dev_tools#/console) to create a rule with the `alertDelay` ``` POST kbn:/api/alerting/rule { "params": { "searchType": "esQuery", "timeWindowSize": 5, "timeWindowUnit": "m", "threshold": [ -1 ], "thresholdComparator": ">", "size": 100, "esQuery": """{ "query":{ "match_all" : {} } }""", "aggType": "count", "groupBy": "all", "termSize": 5, "excludeHitsFromPreviousRun": false, "sourceFields": [], "index": [ ".kibana-event-log*" ], "timeField": "@timestamp" }, "consumer": "stackAlerts", "schedule": { "interval": "1m" }, "tags": [], "name": "test", "rule_type_id": ".es-query", "actions": [ { "group": "query matched", "id": "${ACTION_ID}", "params": { "level": "info", "message": """Elasticsearch query rule '{{rule.name}}' is active: - Value: {{context.value}} - Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}} - Timestamp: {{context.date}} - Link: {{context.link}}""" }, "frequency": { "notify_when": "onActionGroupChange", "throttle": null, "summary": false } } ], "alert_delay": { "active": 3 } } ``` - Verify that the alert will not be created until it has matched the delay threshold. - Verify that the delay does not affect recovered alerts

…tic#176346) Resolves elastic#173009 ## Summary Adds a new input for the user to define the `alertDelay`. This input is available for life-cycled alerts (stack and o11y) rule types. ### Checklist Delete any items that are not applicable to this PR. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify - Using the UI create a rule with the `alertDelay` field set. - Verify that the field is saved properly and that you can edit the `alertDelay` - Verify that you can add the alert delay to existing rules. Create a rule in a different branch and switch to this one. Edit the rule and set the `alertDelay`. Verify that the rule saves and works as expected. --------- Co-authored-by: Lisa Cawley <[email protected]>

botelastic bot added the needs-team Issues missing a team label label Dec 10, 2023

shanisagiv1 added the Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) label Dec 10, 2023

botelastic bot removed the needs-team Issues missing a team label label Dec 10, 2023

shanisagiv1 added this to AppEx: ResponseOps - Rules & Alerts Management Dec 10, 2023

github-project-automation bot moved this to Awaiting Triage in AppEx: ResponseOps - Rules & Alerts Management Dec 10, 2023

shanisagiv1 assigned shanisagiv1 and XavierM Dec 10, 2023

mikecote added this to AppEx: ResponseOps - Execution & Connectors Dec 21, 2023

github-project-automation bot moved this to Awaiting Triage in AppEx: ResponseOps - Execution & Connectors Dec 21, 2023

mikecote moved this from Awaiting Triage to Todo in AppEx: ResponseOps - Execution & Connectors Jan 4, 2024

mikecote assigned doakalexi and unassigned XavierM and shanisagiv1 Jan 4, 2024

doakalexi moved this from Awaiting Triage to In Progress in AppEx: ResponseOps - Rules & Alerts Management Jan 8, 2024

doakalexi moved this from In Progress to Awaiting Triage in AppEx: ResponseOps - Rules & Alerts Management Jan 8, 2024

doakalexi moved this from Todo to In Progress in AppEx: ResponseOps - Execution & Connectors Jan 8, 2024

doakalexi moved this from Awaiting Triage to Todo in AppEx: ResponseOps - Rules & Alerts Management Jan 8, 2024

doakalexi mentioned this issue Jan 16, 2024

[ResponseOps][BE] Alert creation delay based on user definition #174657

Merged

1 task

doakalexi moved this from In Progress to In Review in AppEx: ResponseOps - Execution & Connectors Jan 22, 2024

doakalexi mentioned this issue Jan 23, 2024

[ResponseOps][FE] Alert creation delay based on user definition #175153

Closed

2 tasks

This was referenced Jan 30, 2024

[ResponseOps] Alert creation delay based on user definition #175851

Merged

[ResponseOps] Include alert creation delay in the event log #175941

Closed

[ResponseOps] The count of consecutive active alerts should be available on the alert #175998

Closed

doakalexi closed this as completed in #175851 Feb 6, 2024

github-project-automation bot moved this from Todo to Done in AppEx: ResponseOps - Rules & Alerts Management Feb 6, 2024

github-project-automation bot moved this from In Review to Done in AppEx: ResponseOps - Execution & Connectors Feb 6, 2024

doakalexi mentioned this issue Feb 6, 2024

[ResponseOps][FE] Alert creation delay based on user definition #176346

Merged

2 tasks

mikecote reopened this Feb 8, 2024

mikecote moved this from Done to In Review in AppEx: ResponseOps - Execution & Connectors Feb 8, 2024

lcawl mentioned this issue Feb 14, 2024

[OAS] Add alert_delay alerting rule option for Stack and Observability rules #176941

Merged

doakalexi closed this as completed in #176346 Feb 15, 2024

github-project-automation bot moved this from In Review to Done in AppEx: ResponseOps - Execution & Connectors Feb 15, 2024

This was referenced Mar 11, 2024

[DOCS] Add alert creation delay in Stack rules #178461

Merged

[DOCS] Alert creation delay elastic/observability-docs#3667

Merged

mergify bot mentioned this issue Mar 13, 2024

[8.13](backport #3667) [DOCS] Alert creation delay elastic/observability-docs#3670

Merged

lcawl mentioned this issue Mar 20, 2024

[DOCS] Unhealthy transform rule check updates elastic/elasticsearch#106521

Merged

This was referenced May 22, 2024

[Discuss] Alert only after the metrics threshold is met X times - Customer request #89152

Closed

Delayed alert creation #146220

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Alert creation delay based on user definition #173009

Alert creation delay based on user definition #173009

shanisagiv1 commented Dec 10, 2023 •

edited

Loading

elasticmachine commented Dec 10, 2023

Erikg346 commented Jan 5, 2024 •

edited

Loading

mikecote commented Feb 8, 2024

Alert creation delay based on user definition #173009

Alert creation delay based on user definition #173009

Comments

shanisagiv1 commented Dec 10, 2023 • edited Loading

elasticmachine commented Dec 10, 2023

Erikg346 commented Jan 5, 2024 • edited Loading

mikecote commented Feb 8, 2024

shanisagiv1 commented Dec 10, 2023 •

edited

Loading

Erikg346 commented Jan 5, 2024 •

edited

Loading