[Response Ops][Alerting] Appendable payload fields

[darnautov]

In Anomaly Detection alerting rule we have only one active alert per rule, as we use anomaly job ID as an alert name. Alert instance is created when the rule matched the condition and remains active if consecutive buckets also satisfy the criteria. With alerts-as-data integration we're able to examine the duration of alerts, which helps in identifying whether an adjustment to the lookback interval is needed.

But we can't keep track of the anomaly score change during the alert duration, as kibana.alert.anomaly_score contains only the latest score before alert is recovered. The user can receive alerts based on interim results, and score can also adjust after the normalization process.
Therefore, it would be beneficial to have the capability to append values to an ADD field, enabling us to preserve changes in the anomaly score over time. This would provide a more comprehensive view of which scores have kept alerts active, compared to the latest results visible in the Anomaly Explorer.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[pmuellr]

We discussed in triage that sending the current alert "doc" into the rule is probably the way we would go. The rule can then modify the existing data or replace completely (which is the only option today).

[mikecote]

@darnautov if the team comes up with a technical approach, would this be a PR you're willing to do? (with our guidance). Not sure what the level of priority is for ML, but this could help unblock you.

[peteharverson]

As discussed with @mikecote and @darnautov , we're happy to work on this - it will allow an enhancement to the work in progress for 8.12, to indicate how the anomaly score changes over the duration of the alert. We'll add it to the tentative plan for 8.13.