[Synthetics] Improve recovery of custom monitor status rules to optionally consider up status

[dominiqueclarke]

Currently, custom monitor status rules recovery exclusively by waiting for the alert condition to no longer be met.

The two current alert conditions we have are:

1. X down checks within the last X checks from X locations
2. X down checks within the last X time range from X locations

When the customer specified condition is no longer met, the alert will recover.

However, in the case of condition 2 in particular, the alert can recover if the monitor stops reporting data and the down checks fall off the specified time range

In the case of condition 1, the condition may no longer be met, if there are now < X down checks, even if the most recent check was still a down check.

The problem is further described, in great detail, here https://github.com/elastic/synthetics-dev/issues/387#issuecomment-2332043258

To address these issues, we'd like to allow the customer a way to specify how they would prefer recovery to happen

For example, you could imagine the option where the user can specify when the recovery should occur:

• The first UP test comes in
• The alert condition is no longer met

The user can then select which of these is appropriate for their case, and they could even specify both, which means that both conditions must be met, before a recovery is sent.

[elasticmachine]

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)