Optimize doc level monitor performance: Batch docs for percolate query searches based on available memory and cpu

[eirsep]

Is your feature request related to a problem?
Doc level monitor execution does the following 2 memory, cpu sensitive operations in search path :

• Fetch docs source for docs (> last read sequence number) for every shard per index
• Accumulate docs source list per index and perform percolate queries this causes heap usage exceeded exception when docs in memory are too high

What solution would you like?
Collect docs from shards up until a threshold which factors heap usage and available memory. Perform percolate query. If more shards remain to be queried repeat the above process until all shards are queried in current execution of doc level monitor
TODO - factor in cpu usage also
What alternatives have you considered?
Perform percolate query per shard - caused CPU regression due to too many requests
Perform percolate query for all docs across all shards, indices being - aggravates heap usage exceed issue as more docs are queried
(Current) Perform percolate query per concrete index - still not determinisitc. may make too many percolate queries (if very few docs in memory from one index) or too less (if too many docs from one index in memory)

[eirsep]

#1331