Percolate query optimization: Fetch fields mentioned in queries instead of entire doc and batch percolate query by heap-based threshold#1331

Draft

eirsep wants to merge 17 commits intoopensearch-project:mainfrom eirsep:percolate_per_shard

+491-136

Commits on Jan 24, 2024

refactor doc level monitor to perform a percolate queru for docs from each shard instead of performing one percolate query on docs from all shards
eirsep
committed
move transformedDocs into for loop in fetchDataAndExecutePercolateQueriesPerShard()
eirsep
committed
rename percolate query execution method
eirsep
committed
fix log messages
eirsep
committed
add setting for percolate query docs size memory threshold to perform percolate query
eirsep
committed
perform percolate query only if threshold setting breached or at the end of collecting data from all indices
eirsep
committed
change terms query clause on indices to list of should clauses as index is a text field in query index mapping
eirsep
committed
add setting and check on num docs in memory to determine if percolate query should be performed immediately
eirsep
committed
use updated index names while process findings
eirsep
committed
register setting in plugin class
eirsep
committed

Commits on Jan 25, 2024

optimize to fetch only relevant fields from source data in doc level monitor to submit to percolate query instead of docs_source
eirsep
committed

Commits on Jan 27, 2024

query shard up to max sequence number instead of just 10000
eirsep
committed

Commits on Feb 2, 2024

add stats
eirsep
committed

Commits on Feb 8, 2024

Commits on Feb 17, 2024