Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize doc level monitors - only query indices with new documents in data streams #1290

Closed
eirsep opened this issue Oct 27, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@eirsep
Copy link
Member

eirsep commented Oct 27, 2023

Doc level monitors intend to track sequence number of indices shards and query only documents ingested since last monitor execution. The problem is that users configure doc level monitors to run against index patterns with rollover or data streams.
More often than not doc level monitors are querying an index patter which resolves to , say 1000 concrete indices, of which 999 are read-only and 1 write index.
We really don't need to query all the 999 read-only indices as they might not have any documents ingested since the previous execution.

we can optimize to query only those indices which are writeable and all indices which have been marked as read-only AFTER the previous monitor execution
for first run of monitor, when creating doc level queries it should only consider write index and the read indices which were created after the enabled_time of monitor

This will drastically reduce the CPU consumption of the doc level monitor and provide a significant optimization for the plugin's resource consumption

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants