Maintenance window scoped query sometimes doesn't work

[kobelb]

Description

While experimenting with the new maintenance window scoped queries, I've seen them not work a few times for suppressing the actions. It's also worked a few times as well, so I think the order of operations may matter.

The first time I saw it not working, I created a maintenance window w/ scoped query, and then created a SLO and a SLO Burn Rate alerting rule. Once I had the SLO Burn Rate rule creating alerts and triggering actions, I noticed that the maintenance window's scoped query was wrong and it wasn't suppressing any alerts. I went back to the maintenance window UI, updated the scoped query to be correct, saved the maintenance window, and continued to see actions triggered for all alerts.

The second time I saw it not working, I created a metric threshold alerting rule with the "group by" option. Once it was creating alerts and triggering actions, I then went and created a maintenance window w/ scoped query that was for kibana.alert.instance.id: "foo-1". After the maintenance window was created, it had no effect and I continued to see actions triggered for the alerts with this field set to this value.

Steps to reproduce:

The following is a recreation for the second time I saw this misbehaving.

1. Create some metrics

Use DevTools to create docs similar to the following, you'll need to update the @timestamp to make them more recent.

POST metrics-foo-default/_doc
{
  "@timestamp": "2023-12-08T13:27:11.488Z",
  "service": "foo-1",
  "success": false
}

POST metrics-foo-default/_doc
{
  "@timestamp": "2023-12-08T13:27:11.488Z",
  "service": "foo-2",
  "success": false
}

POST metrics-foo-default/_doc
{
  "@timestamp": "2023-12-08T13:27:11.488Z",
  "service": "foo-3",
  "success": false
}

POST metrics-foo-default/_doc
{
  "@timestamp": "2023-12-08T13:29:05.692Z",
  "service": "foo-1",
  "success": false
}

POST metrics-foo-default/_doc
{
  "@timestamp": "2023-12-08T13:29:05.692Z",
  "service": "foo-2",
  "success": false
}

POST metrics-foo-default/_doc
{
  "@timestamp": "2023-12-08T13:29:05.692Z",
  "service": "foo-3",
  "success": false
}

2. Create metric threshold alerting rule w/ index document action

3. Create Maintenance Window

Expected behavior:

The notifications index should not have documents with "service: foo-1" any longer being created.

[kobelb]

Thanks for investigating this, @XavierM and @JiaweiWu. The feature is working as designed. The maintenance windows only apply to new alerts, they do not apply to ongoing alerts.