Task Manager health API - workload.value.average_interval_ms

[kobelb]

Problem

Currently, the Task Manager health API returns statistics about Task Manager's configuration, workload, and runtime performance. The workload.value.schedule currently returns the 10 most frequent intervals for the scheduled tasks, but it does not return the intervals for all scheduled tasks, as this would be infeasible to return a "bucket" for every single interval:

As part of the autoscaling Kibana project, we would like to scale Kibana based on the task-capacity vs the scheduled task-load. One of the missing data-points for performing this calculation is the average interval for all scheduled tasks and this can't be inferred from the workload.value.schedule field.

Solution

The task-manager health API should be updated to return the workload.value.average_interval_ms to support this autoscaling calculation.

Currently, each task document has a task.schedule.interval field; however, this is a keyword field and stores the intervals using Elasticsearch's date interval syntax: 10m for 10 minutes, 100ms for 10 milliseconds. As a result, it's not possible to use the Elasticsearch avg aggregation on the task.schedule.interval field. Instead, a task.schedule.interval_ms field should be added so that the Elasticsearch avg aggregation can efficiently run.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[mikecote]

Adding to To-Do, we'll research what it will take to accomplish this to know what lead time we'll need. If the research shows it's easy to do the work, might as well.

[kobelb]

Thanks @mikecote <3 There will be a couple of other smaller issues that I'll be creating a bit later this week that I'd like to get some rough "guesstimates" on for the Autoscaling work. So, I'd recommend holding off until those are created as well because they're all related.

[mikecote]

Thanks, @kobelb! I've moved this into the backlog. Keep me posted when you're ready for us to provide guesstimates and we'll do them all at the same time 🙏