[BUG] Batch async shard fetch holds up significant memory causing OOMs

[Bukhtawar]

Describe the bug

During batch async fetch, it was observed that upto 8k node responses were being cached simultaneously with a whole bunch (4k) of retention leases per store metadata bloating up the over all memory eventually leading to OOM

Related component

Cluster Manager

To Reproduce

1. Create a 200 node cluster with 100k shards, 16G heap on cluster manager nodes
2. Restart all nodes of the cluster
3. Observe leader going OOM

Expected behavior

Cluster shouldn't go out-of-memory as it risks availability

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.