Alerts are not recovered correctly in ICMP monitoring due to 'ping timeout'

[iTiagoCO]

Kibana version: 8.13.2

Elasticsearch version: 8.13.2

Describe the bug:

Alerts configured in the observability rule do not recover correctly when the condition that triggered them is no longer true. This appears to be bug-like behavior.

Steps to reproduce:

Configure a rule in Kibana to monitor ICMP status with condition MATCHING MONITORS ARE DOWN >= 3 times WITHIN last 10 minutes.
Shut down one of the monitored hosts to generate a "ping timeout".

Note that the alert fires correctly but does not recover when the host comes back online.

Expected behavior:

Alerts should automatically recover when the original condition that triggered them is no longer met.

Additional attachment discuss created for the current case that contains more details, screenshots, logs, etc.

https://discuss.elastic.co/t/problem-with-alerts-recovered/362036

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[elasticmachine]

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

[iTiagoCO]

Add error terminal.

[ERROR][plugins.ruleRegistry] ResponseError: {"errors":true,"took":5,"ingest_took":3,"items":[{"create":{"_index":".internal.alerts-observability.uptime.alerts-default-000046","_id":"2b6bc7a5-16d5-40d5-a4dc-a49179f60e5d","status":409,"error":{"type":"version_conflict_engine_exception","reason":"[2b6bc7a5-16d5-40d5-a4dc-a49179f60e5d]: version conflict, document already exists (current version [1])","index_uuid":"LSIoSjsTTRuuGWTTmaz7fQ","shard":"0","index":".internal.alerts-observability.uptime.alerts-default-000046"}}},{"create":{"_index":".internal.alerts-observability.uptime.alerts-default-000046","_id":"831e1226-0373-4d0e-bcfc-81a97bea8b67","status":409,"error":{"type":"version_conflict_engine_exception","reason":"[831e1226-0373-4d0e-bcfc-81a97bea8b67]: version conflict, document already exists (current version [1])","index_uuid":"LSIoSjsTTRuuGWTTmaz7fQ","shard":"0","index":".internal.alerts-observability.uptime.alerts-default-000046"}}},{"create":{"_index":".internal.alert...

[jasonrhodes]

NOTE: Seems like there is a lot of additional detail in the linked discuss thread, it might be a good idea to try to capture more of that here in this issue. We'll try to recreate this and see what we can discover.