[Searchable Snapshot] Design a new ShardsAllocator to handle remote shards

[kotwanikunal]

Is your feature request related to a problem? Please describe.

• Searchable snapshots will introduce a new remote searcher role - [Searchable Snapshot] [Design Proposal] Remote Searchable Snapshots #3895
• As a part of this, we would like to design a new shard allocation logic which will assign the searchable snapshot shards to the nodes with this new role
• There is a POC implementation which was done as a part of Implement a POC based on BalancedShardsAllocator for searchable snapshots #4420 and exists at Add support for remote snapshot alliocation kotwanikunal/OpenSearch#4

Describe the solution you'd like

• Design for a new ShardsAllocator which ensures that the remote shards are assigned to the nodes with a remote_searcher role (Ref: Add support for remote snapshot alliocation kotwanikunal/OpenSearch#4)
• Ensures that the hybrid nature of having both the local and the remote shards on the single node are handled appropriately wherein the rebalancing logic for the local shards does not affect the remote shards, remote shards have the appropriate recovery process to handle node failures (Handled by the POC using basic checks, will need a deeper dive)
• The AllocatorDecider(s) are capable of handling both the remote and local shards for the hybrid setup (Ref: Add support for remote snapshot alliocation kotwanikunal/OpenSearch#4)

Additional context

• Implement a POC based on BalancedShardsAllocator for searchable snapshots #4420
• [Searchable Snapshot] Shard allocation for searchable snapshots #4306

[kotwanikunal]

The design was published at #4759