[Feature Request] Timeout during execution of the task on cluster manager node

[dhwanilpatel]

Is your feature request related to a problem? Please describe

Many cluster manager tasks generated from user has the timeout like create-index/put-mapping/update-setting/etc. Today this task will timeout only if they are in pending task queue of cluster manager node, i.e. if tasks is waiting in queue for that timeout period then only it will be timed out and throw process_cluster_event_timeout_exception.

If task is picked up by cluster manager node then it never times out even if it take infinite time. This will cause subsequent tasks to be timeout. This can impact overall cluster resiliency as well as cluster manager will not be execute critical task like node-join/left until the task in execution is not completed.

Recently we have faced similar issue where one custom deciders was taking too long time and task execution was taking very long time and it didn't timeout. All subsequent tasks were getting timed out because of these and we were not able to do any admin operation like updating cluster setting/new index creation/deletion/etc.

Describe the solution you'd like

Cluster manager node should timeout the task even if it is in execution phase and not only during waiting period in pending task queue. We can evaluate if we want to create any separate timeout for this or we just use the timeout of particular task during execution phase as well.

Related component

Cluster Manager

Describe alternatives you've considered

No response

Additional context

No response

[peternied]

[Triage - attendees 1 2]
@dhwanilpatel Thanks for filing this issue, please feel free to submit a pull request.