[Logs Overview] Refine change types

[weltenwort]

📓 Summary

The change types currently shown in the log categories data grid are more or less passed through from the change_point aggregation. These types are sometimes hard to interpret as a human or an LLM. Additionally, they are sensitive to small variations in the distribution of the histogram when the document count is low, which leads to confusing flapping between types like spike and step.

• 🔗 related to: [Logs Overview] Enhanced logs component for solution UIs #190848

✔️ Acceptance criteria

• The changes detected by the Elasticsearch aggregation are post-processed into more useful change types:
  • rare: present in fewer than 1/5th of the buckets
    • from stationary by counting buckets
  • new: a detected change point before which all buckets are empty, but after which less than 1/2 of the buckets are empty
    • from step_change, distribution_change, trend_change by counting buckets of the two parts of the histogram
  • disappeared: a detected change point before which less than 1/2 of the buckets are empty, but after which all buckets are empty
    • from step_change, distribution_change, trend_change by counting buckets of the two parts of the histogram
  • decreased: a detected change point before which the average doc count per bucket was larger than after
    • from step_change, distribution_change, trend_change by counting buckets of the two parts of the histogram
    • from non_stationary with a decreasing trend value
  • increased: a detected change point after which the average doc count per bucket was larger than before
    • from step_change, distribution_change, trend_change by counting buckets of the two parts of the histogram
    • from non_stationary with an increasing trend value
  • spike/dip remain as before
  • none: no change was detected
  • unknown: Elasticsearch returned a change type we don't know how to handle (i.e. due to it being a newer version)
• The new change types are more stable with low document counts.

💡 Implementation hints

• in most cases the first and last bucket of the histogram should probably be ignored, because they might show boundary effects

Known change types and their attributes as returned by Elasticsearch

type	has time	has p-value	other attributes
indeterminable	❌	❌	reason
stationary	❌	❌
step_change	✅	✅
distribution_change	✅	✅
non_stationary	❌	✅	rValue, trend
trend_change	✅	✅	rValue
spike	✅	✅
dip	✅	✅

Change point detection algorithm of the change_point aggregation

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)