Framework to handle missed ops and Handshake

[aravindavk]

When a user initiates the action, if one or more participating nodes are not online then some actions can continue without those nodes. Some actions strictly require all the nodes to be reachable. For example, Volume create. Unless the node is reachable, Moana doesn't know the status of Storage unit availability. So Volume create will fail if all the participating nodes are not online.

Identify the actions that don't strictly require all the nodes to be online. Examples,

• Volume Stop
• Volume Start
• Volume delete
• Volume Option set/reset
• Snapshot operations
• ...

Implement a framework that records the missed ops under each nodes state. When that node comes back online, the first step it should do is sync up with Mgr and get all the missed ops.

• Mgr will not forward any request till the handshake process is complete and the Node agent confirms that it is now ready to take requests.
• Node agent can post-process the Operations to eliminate no-ops. For example, a Volume was stopped and started again. The node agent can skip this step if the final state matches the current state.