[META] Remote Routing Table - v2.15

[himshikha]

Please describe the end goal of this project

Project Meta: #14164

This Meta tracks issues to be targetted for v2.15.

Each shard movement results in a cluster state update which needs to be communicated to all the data nodes to be able to effectively route requests. This results in a scaling problem for a reasonably bigger size having large number of nodes. This can cause inter-node network to get swamped due to bigger states and high volume/frequency of network transfers.

Proposed Solution : Reduce memory and communication overhead for routing table updates using a remote store as an intermediate store and leveraging remote store interactions for data transfers and sparing the node to node network bandwidth

We will move Routing table to remote store. Cluster manager node will be responsible for updating the remote store whenever any updates in routing happen. Since we will have the complete table in storage, we can optimize on what we want to keep in memory on the nodes and use remote store to get the routing information whenever required. Data nodes will only need to keep routings for replica shards whose primary reside on the node.
For reducing communication overhead, cluster state publication will intimate data nodes of the change with updated cluster state term and version rather than complete diff. Data nodes will download the updated routing information from storage. This would make communication from cluster manager faster and each node can individually update their local memory.

Issues

• [Remote Routing Table] Create Service for remote routing table storage #13049
• [Remote Routing Table] Create remote store objects for uploading Routing Table #13960
• [Remote Routing Table] Handle cluster state publication flow for remote routing table #13963
• [Remote Routing Table] Add upload flow for remote routing table #14033
• [Remote Routing Table] Delete stale index routing table files  #14162
• [Remote Routing Table] Add read flow from remote store #14109
• [Remote routing table]Add diff in manifest for effecient read #14108

Related component

ShardManagement:Routing

[peternied]

[Triage - attendees 1 2 3 4 5 6 7 8]
@himshikha Thanks for creating this issue; however, it isn't being accepted due to not having enough detail to justify what is being acomplished, please use the meta template with a number of clear deliverables. Please feel free to open a new issue after addressing the reason.