Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an _exists_ check to document level monitor queries #1425

Merged
merged 3 commits into from
Mar 7, 2024

Conversation

jowg-amazon
Copy link
Collaborator

@jowg-amazon jowg-amazon commented Feb 13, 2024

Issue #, if available:
#854

Description of changes:
Related to opensearch-project/security-analytics#852

Checks if _exists_ is present in the query. If it is, then replace the value with the field name and the correctly appended index name and monitor id.

CheckList:

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@jowg-amazon jowg-amazon changed the title Add an _exists_ check to queries Add an _exists_ check to document level monitor queries Feb 13, 2024
Copy link
Member

@eirsep eirsep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checks if exists is present in the query.
Can't we do a not null check instead?

@jowg-amazon
Copy link
Collaborator Author

@eirsep Opensearch allows null to be passed in as a value so a not null check would be different than an exists check. We want to check whether or not the field even exists in the document, not if the field is null.

Copy link
Member

@eirsep eirsep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't there be : between exists and ?

@@ -372,7 +379,14 @@ class DocLevelMonitorQueries(private val client: Client, private val clusterServ
var query = it.query
flattenPaths.forEach { fieldPath ->
if (!conflictingPaths.contains(fieldPath.first)) {
query = query.replace("${fieldPath.first}:", "${fieldPath.first}_${sourceIndex}_$monitorId:")
if (query.contains("_exists_")) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we extract code into common method?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extracted the code into a common method

@@ -1674,6 +1674,349 @@ class DocumentMonitorRunnerIT : AlertingRestTestCase() {
assertEquals(1, output.objectMap("trigger_results").values.size)
}

fun `test execute monitor generates alerts and findings with NOT query`() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: rename test to specify we are verifying not + exists

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

renamed the test

Signed-off-by: Joanne Wang <[email protected]>
@jowg-amazon
Copy link
Collaborator Author

jowg-amazon commented Feb 29, 2024

shouldn't there be : between exists and ?

@eirsep Removed the _exists_ flag so it parses to find wherever the query is _exists_: <field name> and transforms the

@AWSHurneyt AWSHurneyt merged commit afa4f5d into opensearch-project:main Mar 7, 2024
15 of 19 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Mar 7, 2024
* clean up and add integ tests

Signed-off-by: Joanne Wang <[email protected]>

* refactored out common method and renamed test

Signed-off-by: Joanne Wang <[email protected]>

* remove _exists_ flag

Signed-off-by: Joanne Wang <[email protected]>

---------

Signed-off-by: Joanne Wang <[email protected]>
(cherry picked from commit afa4f5d)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/alerting/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/alerting/backport-2.x
# Create a new branch
git switch --create backport-1425-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 afa4f5d131a7075593c56efb61ba95d01ad82ebe
# Push it to GitHub
git push --set-upstream origin backport-1425-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/alerting/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport-1425-to-2.x.

jowg-amazon added a commit to jowg-amazon/alerting that referenced this pull request Mar 8, 2024
…roject#1425)

* clean up and add integ tests

Signed-off-by: Joanne Wang <[email protected]>

* refactored out common method and renamed test

Signed-off-by: Joanne Wang <[email protected]>

* remove _exists_ flag

Signed-off-by: Joanne Wang <[email protected]>

---------

Signed-off-by: Joanne Wang <[email protected]>
riysaxen-amzn pushed a commit that referenced this pull request Mar 8, 2024
#1425) (#1456)

* Add an _exists_ check to document level monitor queries (#1425)

* clean up and add integ tests

Signed-off-by: Joanne Wang <[email protected]>

* refactored out common method and renamed test

Signed-off-by: Joanne Wang <[email protected]>

* remove _exists_ flag

Signed-off-by: Joanne Wang <[email protected]>

---------

Signed-off-by: Joanne Wang <[email protected]>

* fix integ test

Signed-off-by: Joanne Wang <[email protected]>

---------

Signed-off-by: Joanne Wang <[email protected]>
sbcd90 pushed a commit to sbcd90/alerting that referenced this pull request Mar 10, 2024
…ject#1307)

* Added 2.11.1 release notes.

* Added 2.11.1 release notes.

---------

(cherry picked from commit 06c1b8a)

Signed-off-by: AWSHurneyt <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

fix workflow security tests in alerting (opensearch-project#1310) (opensearch-project#1311)

Signed-off-by: Subhobrata Dey <[email protected]>

Increment version to 2.12.0-SNAPSHOT (opensearch-project#1239)

Signed-off-by: opensearch-ci-bot <[email protected]>
Co-authored-by: opensearch-ci-bot <[email protected]>

[Backport 2.x] Reference get monitor and search monitor action / request / responses from common-utils (opensearch-project#1315)

* Use get monitor action / req / resp from common-utils

Signed-off-by: Tyler Ohlsen <[email protected]>

* Dummy commit to retrigger

Signed-off-by: Tyler Ohlsen <[email protected]>

---------

Signed-off-by: Tyler Ohlsen <[email protected]>

optimize doc-level monitor execution workflow for datastreams (opensearch-project#1302) (opensearch-project#1322)

Signed-off-by: Subhobrata Dey <[email protected]>

Update to Gradle 8.5 (opensearch-project#1369) (opensearch-project#1371)

Signed-off-by: Andriy Redko <[email protected]>

[Backport 2.x] Inject namedWriteableRegistry during ser/deser of SearchMonitorAction (opensearch-project#1382) (opensearch-project#1384)

* Inject namedWriteableRegistry during ser/deser of SearchMonitorAction (opensearch-project#1382)

Signed-off-by: Tyler Ohlsen <[email protected]>

* remove bin files

Signed-off-by: Tyler Ohlsen <[email protected]>

* remove core bin

Signed-off-by: Tyler Ohlsen <[email protected]>

---------

Signed-off-by: Tyler Ohlsen <[email protected]>

Don't attempt to parse workflow if it doesn't exist (opensearch-project#1346) (opensearch-project#1359)

(cherry picked from commit 733fd4e)

Signed-off-by: Chase Engelbrecht <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Set docData to empty string if actual is null (opensearch-project#1325) (opensearch-project#1334)

(cherry picked from commit 008e076)

Signed-off-by: Chase Engelbrecht <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

removed default admin credentials for alerting (opensearch-project#1399) (opensearch-project#1400)

(cherry picked from commit 3c50f7d)

Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Dennis Toepker <[email protected]>

ipaddress lib upgrade as part of cve fix (opensearch-project#1397) (opensearch-project#1407)

(cherry picked from commit 8d59060)

Signed-off-by: Riya Saxena <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Bulk index findings and sequentially invoke auto-correlations (opensearch-project#1355) (opensearch-project#1410)

* Bulk index findings and sequentially invoke auto-correlations

* Bulk index findings in batches of 10000 and make it configurable

* Addressing review comments

* Add integ tests to test bulk index findings

* Fix ktlint formatting

---------

(cherry picked from commit b561965)

Signed-off-by: Megha Goyal <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Add 2.12 release notes (opensearch-project#1408) (opensearch-project#1413)

* Add 2.12 release notes

* Fix release notes PR

* Add 2 more PRs

---------

(cherry picked from commit b10eaad)

Signed-off-by: Chase Engelbrecht <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

[Backport 2.x] Implemented cross-cluster monitor support (opensearch-project#1404) (opensearch-project#1412)

* Implemented cross-cluster monitor support (opensearch-project#1404)

* Updated alert mappings to accommodate cross-cluster cluster metrics monitors.

Signed-off-by: AWSHurneyt <[email protected]>

* Implemented support for cross-cluster cluster metrics monitors. Implemented GetRemoteIndexes API to populate the frontend UI with details regarding the remote clusters, and indexes.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed a writeable test after changing QueryLevelTriggerRunResult from a data class to an open class for inheritability.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed ktlint errors.

Signed-off-by: AWSHurneyt <[email protected]>

* Removed changes to IndexUtils as they're only needed by doc monitors.

Signed-off-by: AWSHurneyt <[email protected]>

---------

Signed-off-by: AWSHurneyt <[email protected]>

(cherry picked from commit ea36996)
Signed-off-by: AWSHurneyt <[email protected]>

* Fixed a test.

Signed-off-by: AWSHurneyt <[email protected]>

---------

Signed-off-by: AWSHurneyt <[email protected]>

Add publishToMavenLocal in build.sh (opensearch-project#1418) (opensearch-project#1419)

(cherry picked from commit 4cdc1d1)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

fix for MapperException[the [enabled] parameter can't be updated for the object mapping [metadata.source_to_query_index_mapping] (opensearch-project#1432) (opensearch-project#1434)

Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

bacport PRs opensearch-project#1445, opensearch-project#1430, opensearch-project#1441, opensearch-project#1435 to 2.x (opensearch-project#1452)

* Add jvm aware setting and max num docs settings for batching docs for percolate queries (opensearch-project#1435)

* add jvm aware and max docs settings for batching docs for percolate queries

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix stats logging

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add queryfieldnames field in findings mapping

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>

* optimize to fetch only fields relevant to doc level queries in doc level monitor instead of entire _source for each doc (opensearch-project#1441)

* optimize to fetch only fields relevant to doc level queries in doc level monitor

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix test for settings check

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix ktlint

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>

* clean up doc level queries on dry run (opensearch-project#1430)

Signed-off-by: Joanne Wang <[email protected]>

* optimize sequence number calculation and reduce search requests in doc level monitor execution (opensearch-project#1445)

* optimize sequence number calculation and reduce search requests by n where n is number of shards being queried in the executino

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix tests

Signed-off-by: Surya Sashank Nistala <[email protected]>

* optimize check indices and execute to query only write index of aliases and datastreams during monitor creation

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix test

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add javadoc

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add tests to verify seq_no calculation

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>
Signed-off-by: Joanne Wang <[email protected]>
Co-authored-by: Joanne Wang <[email protected]>

[Backport 2.x] Add an _exists_ check to document level monitor queries (opensearch-project#1425) (opensearch-project#1456)

* Add an _exists_ check to document level monitor queries (opensearch-project#1425)

* clean up and add integ tests

Signed-off-by: Joanne Wang <[email protected]>

* refactored out common method and renamed test

Signed-off-by: Joanne Wang <[email protected]>

* remove _exists_ flag

Signed-off-by: Joanne Wang <[email protected]>

---------

Signed-off-by: Joanne Wang <[email protected]>

* fix integ test

Signed-off-by: Joanne Wang <[email protected]>

---------

Signed-off-by: Joanne Wang <[email protected]>

add distributed locking to jobs in alerting (opensearch-project#1403) (opensearch-project#1458)

Signed-off-by: Subhobrata Dey <[email protected]>
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.11 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/alerting/backport-2.11 2.11
# Navigate to the new working tree
pushd ../.worktrees/alerting/backport-2.11
# Create a new branch
git switch --create backport-1425-to-2.11
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 afa4f5d131a7075593c56efb61ba95d01ad82ebe
# Push it to GitHub
git push --set-upstream origin backport-1425-to-2.11
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/alerting/backport-2.11

Then, create a pull request where the base branch is 2.11 and the compare/head branch is backport-1425-to-2.11.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants