Raft: Dynamic Node State Based on Backend Service Health

[swisstiming-test]

I want to change the state of the local node depending on the health of a backend service. The goal is for the node to relinquish leadership if the service is unhealthy and to be eligible for leadership once the service is healthy again.

I've tried several approaches:

• Temporarily put local node in standby mode (EnableStandbyModeAsync / RevertToNormalModeAsync). Problem: Enabling standby mode fails when the node is currently the leader.
• Temporarily remove the local node from the cluster (RemoveMemberAsync(_cluster.LocalMemberAddress) / AddMemberAsync(_cluster.LocalMemberAddress)). Problem: Removing the member fails when the node is not currently the leader.
• Temporarily stop the local node (StopAsync / StartAsync). Problem: I get an error when I try to start the node again.

Ideally, the node would remain in standby to maintain quorum.

Could you provide guidance on the correct approach or suggest alternative solutions? Thank you for your assistance.

[sakno]

EnableStandbyModeAsync is a dangerous operation. Let's imagine you have 3 nodes. And all of them are turned into Standby mode eventually. Such as cluster operating without a leader and available for read-only operations. I can remove the restriction on EnableStandbyModeAsync method not to be a leader. If the node is a leader, it just stops sending heartbeats. Is it appropriate solution for you?

[swisstiming-test]

Yes, that sounds like a good solution to me. Thank you very much!

[sakno]

Release 5.4.0 has been published, now EnableStandbyModeAsync supports downgrading from Leader state.