[FEATURE] Enrich alert notification message with context. from individual documents' fields in bucket level monitors

[eirsep]

Is your feature request related to a problem?
Bucket level monitors contain only the information of the buckets that are created based on aggregation queries. there is no data from individual documents avaialble.

When alerts are generated and user wants to send out notifications, they don't have enough information to enrich the notification message because search response doesn't return the docs and returns only the aggregated data.

For ex. consider a bucket level monitor configured to monitor CPU of 100 hosts crossing 80% in last 1 hr. Say, each doc in index has fields - cpu, hostname, timestamp, eventid, jvm, If CPU>80 for a certain host they would like to fire alert. They can bucket on hostName and condition cpu >80. If condition matches they will get alert for cpu>80 and they will have information about which hosts are causing alert. But they wouldn't have access to other fields of such document

Possible solution
If alerts are generated i.e. monitor trigger condition is satisfied AND notifications are configured, Alerting plugin should support a post-trigger search request to collect data to enrich notification message.

That way users have additional context to access document's other fields based on bucket values

[eirsep]

Alternatively explore adding an exemplar or a sample document in every bucket level monitor by making size = 10 to get atleast some sample data

[AWSHurneyt]

Related issue #1396