optimize doc-level monitor execution workflow for datastreams

[sbcd90]

Issue #, if available:
#1290

Description of changes:
This pr optimizes the doc-level monitor execution workflow for data streams.

The optimization proposed only considers the current write index, any new read-only indices created after last monitor execution run & the last write index of last monitor execution run for current monitor execution run.
It ignores read-only indices that may have been created prior to the last monitor execution run.

The below diagram shows the difference.

Earlier

Now

CheckList:

• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[eirsep]

if I use any alias, there can be multiple writeIndices. why are we assuming there can be only one writeIndex. That applies to datastream only

plz add tests for alias which points to multiple write indices and modify handling as required.

I would explore modifying monitor metadata as required to handle aliases with multiple write indices.

[sbcd90]

An index/datastream alias can have only 1 write index. https://www.elastic.co/guide/en/elasticsearch/reference/current/aliases.html#write-index
Common Problems section - https://opster.com/guides/elasticsearch/glossary/elasticsearch-alias/
An index pattern can have multiple write indices though.

[eirsep]

plz make change to handle indices queried if the monitor metadata section is empty. Only write index and indices created after creation_time timestamp should be considered for querying .

[sbcd90]

plz make change to handle indices queried if the monitor metadata section is empty. Only write index and indices created after creation_time timestamp should be considered for querying .

this is already addressed. Added a test case for it.

alerting/alerting/src/test/kotlin/org/opensearch/alerting/DocumentMonitorRunnerIT.kt

Line 1398 in fc3892b

fun `test document-level monitor ignoring old read-only indices for datastreams`() {

[sbcd90]

integ tests fail for org.opensearch.alerting.util.destinationmigration.DestinationMigrationUtilServiceIT.test migrateData which is unrelated to this pr.
looking into it though.

[eirsep]

if they have 2 data streams a1* and a2* but customer creates index on a* index pattern, there will be multiple write indices for one input in indices section of monitor config. the changes in new PR might break the experience for this scenario
it’s ok if PR doesn’t optimize for that scenario but should not cause new bug

[sbcd90]

if they have 2 data streams a1* and a2* but customer creates index on a* index pattern, there will be multiple write indices for one input in indices section of monitor config. the changes in new PR might break the experience for this scenario it’s ok if PR doesn’t optimize for that scenario but should not cause new bug

the optimization doesnt add a new bug.

alerting/alerting/src/test/kotlin/org/opensearch/alerting/DocumentMonitorRunnerIT.kt

Line 1398 in 510e66a

fun `test document-level monitor when multiple datastreams contain docs across read-only indices that do match query`() {

a follow-up pr need to be created for using this optimization in index patterns.

[opensearch-trigger-bot]

The backport to 2.7 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/alerting/backport-2.7 2.7
# Navigate to the new working tree
pushd ../.worktrees/alerting/backport-2.7
# Create a new branch
git switch --create backport-1302-to-2.7
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 903a6d9f80000cf29065044692c1c698cf0e0089
# Push it to GitHub
git push --set-upstream origin backport-1302-to-2.7
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/alerting/backport-2.7

Then, create a pull request where the base branch is 2.7 and the compare/head branch is backport-1302-to-2.7.

[opensearch-trigger-bot]

The backport to 2.5 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/alerting/backport-2.5 2.5
# Navigate to the new working tree
pushd ../.worktrees/alerting/backport-2.5
# Create a new branch
git switch --create backport-1302-to-2.5
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 903a6d9f80000cf29065044692c1c698cf0e0089
# Push it to GitHub
git push --set-upstream origin backport-1302-to-2.5
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/alerting/backport-2.5

Then, create a pull request where the base branch is 2.5 and the compare/head branch is backport-1302-to-2.5.