Allow the user to configure a managed index job to execute continuously

[adityaj1107]

Issue by dbbaughe
Tuesday Oct 06, 2020 at 22:40 GMT
Originally opened as opendistro-for-elasticsearch/index-management#301

---

Currently we split every execution of a managed index into individual steps that can be executed in isolation based off the metadata and state of the cluster. But, when you have a state with a couple actions that could all execute immediately it could take a while for processing them.

Let's take an example:

You apply a policy that looks like (quick syntax) to an index that has existed for a while:
(start) -> (hot: rollover(30gb)) -[7d]-> (warm: replicacount(1), allocate()) -[30d]-> (delete: delete())

Perhaps this index is already over 7d and already has 30gb, but you'll wait a while for it to:
Initialize
Attempt to rollover
Wait for rollover
Transition to warm
Set replica count to 1
Allocate to specific node

It would be nice to let it do all of the above without any delay if the user wants. We could introduce a flag on the policy that tells the runner to keep executing the steps as long as it's possible and utilize the renewLease method that is being added to the job scheduler.