[BUG] [Remote store] Throttling from remote store shouldn't cause leader stepping down and re-elections

[shwetathareja]

Describe the bug

With the remote storage, leader writes the cluster state to both local and remote storage. At times, when the remote storage is experiencing surge in traffic, it can start throttling the requests. This can cause state upload to fail and would cause leader to step down and would result in re-elections. This can go in a loop until throttling subsides.

Expected behavior
Enhance the exception handling where leader need not step down for cases where cluster state updates are failing purely due to remote store throttling.

[kiranprakash154]

Hi, are we on track for this to be released in 2.12 ?

[soosinha]

Removing the 2.12 tag. This will be picked up later