[Logs UI] Assess "noise" from the /summary endpoint

[Kerry350]

Summary

The /api/log_entries/summary API endpoint frequently results in a 502 response (BACKEND_CLOSED_CONNECTION), most likely timeouts communicating with Elasticsearch. When looking at a specific case of this the cluster was under-resourced, which aligns with that theory. The issue is the frequency of these responses, which is polluting certain statistics.

Whilst the API could possibly use a more accurate status code than 502 the response behaviour isn't necessarily broken.

What we should investigate is why this endpoint differently to others; is there broken retry logic? Or cancellation logic? (etc).

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[weltenwort]

💭 Could this be related to #148670? If the summary was requested often via sync search it could both cause cluster load but also be negatively affected in a feedback loop. Would be interesting to see if the noise subsides with 8.6.1 and 8.7.0. Changing this to use async search could improve it further.

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)