get all findings as part of findings API enhancement

[riysaxen-amzn]

Description

• As part of the commit ->

1. The query params detector_id and detectorType have become optional for the findings/_search API. As part of the enhancement, the API can now return all the findings for all the detectors.
2. Added new params severity and detectionType in findings API
3. Added corresponding integration tests for the above functionality

Corresponding Alerting and Common-Utils PRs

Issues Resolved

#795

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 41 lines in your changes are missing coverage. Please review.

Project coverage is 24.76%. Comparing base (75c4429) to head (2ffe265).
Report is 1 commits behind head on main.

❗ Current head 2ffe265 differs from pull request most recent head e967376. Consider uploading reports for the commit e967376 to get more accurate results

Files	Patch %	Lines
...nalytics/transport/TransportGetFindingsAction.java	0.00%	40 Missing ⚠️
...h/securityanalytics/action/GetFindingsRequest.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #803      +/-   ##
============================================
- Coverage     24.97%   24.76%   -0.22%     
+ Complexity     1043     1027      -16     
============================================
  Files           277      277              
  Lines         12771    12722      -49     
  Branches       1391     1402      +11     
============================================
- Hits           3190     3150      -40     
  Misses         9307     9307              
+ Partials        274      265       -9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[eirsep]

i am not sure this design is correct or maybe i have grossly misunderstood the approach.

Can you add a detailed approach of how findings from detectors of all log types are being fetched?

[eirsep]

i am not sure this design is correct or maybe i have grossly misunderstood the approach.

Can you add a detailed approach of how findings from detectors of all log types are being fetched?

[riysaxen-amzn]

i am not sure this design is correct or maybe i have grossly misunderstood the approach.

Can you add a detailed approach of how findings from detectors of all log types are being fetched?

So, the idea is first to get all the detectors using matchAllQuery [TransportGetFindingsAction.java#L190]. Now to get the findings for all these detectors for all the logTypes, we can search query on index-pattern .opensearch-sap-*-findings, this will give us all the findings for all the LogTypes [TransportGetFindingsAction.java#L143]

[eirsep]

how is findings service already handling "*"? I don't see a change in FIndingsService class in this PR

[eirsep]

i am not sure this design is correct or maybe i have grossly misunderstood the approach.
Can you add a detailed approach of how findings from detectors of all log types are being fetched?

So, the idea is first to get all the detectors using matchAllQuery [TransportGetFindingsAction.java#L190]. Now to get the findings for all these detectors for all the logTypes, we can search query on index-pattern .opensearch-sap-*-findings, this will give us all the findings for all the LogTypes [TransportGetFindingsAction.java#L143]

Gotcha! thank you. I wasn't aware that we already are searching on the "all finding indices pattern"

Can you create 2 detectors for different log types and test fetching findings for only one of the log types and verify that behaviour is correct?

[riysaxen-amzn]

i am not sure this design is correct or maybe i have grossly misunderstood the approach.
Can you add a detailed approach of how findings from detectors of all log types are being fetched?

So, the idea is first to get all the detectors using matchAllQuery [TransportGetFindingsAction.java#L190]. Now to get the findings for all these detectors for all the logTypes, we can search query on index-pattern .opensearch-sap-*-findings, this will give us all the findings for all the LogTypes [TransportGetFindingsAction.java#L143]

Gotcha! thank you. I wasn't aware that we already are searching on the "all finding indices pattern"

Can you create 2 detectors for different log types and test fetching findings for only one of the log types and verify that behaviour is correct?

this test is creating detectors of two different logTypes and verifying the behavior -> https://github.com/opensearch-project/security-analytics/pull/803/files#diff-d37f0c96e5ab58c9a80449652179f36f9696044aa275fc7c7f33ce2677fa6793

[engechas]

Minor - could create a private method for this to avoid the duplication with the endTime parsing

[jowg-amazon]

LGTM

[jowg-amazon]

Fix merge conflicts + make sure CI's pass

approved

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-803-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0d1d599fea536ee2afe088f70ee1980a0ec572fe
# Push it to GitHub
git push --set-upstream origin backport/backport-803-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-803-to-2.x.