Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Detection Engine health API: cluster health endpoint #165602

Merged

Conversation

banderror
Copy link
Contributor

@banderror banderror commented Sep 4, 2023

Epic: #125642

Summary

This PR implements the _cluster health endpoint. It returns the same metrics as the _space health endpoint. The difference is that:

  • the _cluster health endpoint calculates its metrics on top of data from all Kibana spaces
  • the _space health endpoint calculates its metrics on top of data from the current Kibana space (the one specified in the URL)

Additionally, it fixes a few bugs in the existing health endpoints related to scoping. This PR ensures that we only aggregate detection rules in the saved objects and the .kibana-event-log-* indices, and not any types of rules.

RBAC

The _cluster health endpoint can be called by any user with at least Read privilege to Security Solution.

Documentation

I also updated the health API's README and added a new document describing what health data we return from what endpoints:

security_solution/common/api/detection_engine/rule_monitoring/detection_engine_health/health_data.md

Checklist

Delete any items that are not applicable to this PR.

  • Documentation was added for features that require explanation or tutorials
    • Added more info to the dev docs
  • Unit or functional tests were updated or added to match the most common scenarios

@banderror banderror added release_note:skip Skip the PR/issue when compiling release notes Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Feature:Rule Monitoring Security Solution Detection Rule Monitoring area Team:Detection Rule Management Security Detection Rule Management Team v8.11.0 labels Sep 4, 2023
@banderror banderror self-assigned this Sep 4, 2023
@banderror banderror force-pushed the detection-engine-health-api-enhancements branch 4 times, most recently from b0ae005 to 9953b1d Compare September 7, 2023 08:27
@banderror banderror removed the skip-ci label Sep 7, 2023
@banderror
Copy link
Contributor Author

buildkite test this

@banderror banderror force-pushed the detection-engine-health-api-enhancements branch 3 times, most recently from 95c29ae to 0ccb716 Compare September 11, 2023 14:14
@banderror banderror changed the title [Security Solution] Detection Engine health API enhancements [Security Solution] Detection Engine health API: cluster health endpoint Sep 11, 2023
@banderror banderror force-pushed the detection-engine-health-api-enhancements branch 6 times, most recently from 19a939d to 4ad9e6b Compare September 19, 2023 11:54
@banderror banderror force-pushed the detection-engine-health-api-enhancements branch 4 times, most recently from e4b5676 to f698b49 Compare October 2, 2023 16:32
@banderror banderror force-pushed the detection-engine-health-api-enhancements branch from f698b49 to b1efca5 Compare October 2, 2023 19:31
@banderror banderror marked this pull request as ready for review October 2, 2023 19:32
@banderror banderror requested a review from a team as a code owner October 2, 2023 19:32
@banderror banderror requested a review from nikitaindik October 2, 2023 19:32
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

Comment on lines +44 to +46
{ term: { [f.EVENT_PROVIDER]: ALERTING_PROVIDER } },
{ term: { [f.EVENT_ACTION]: 'execute' } },
{ term: { [f.EVENT_CATEGORY]: 'siem' } },
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By adding { term: { [f.EVENT_CATEGORY]: 'siem' } } we make sure we don't aggregate those rules that are not detection rules.

Comment on lines +51 to +55
totalExecutions: {
cardinality: {
field: f.RULE_EXECUTION_UUID,
},
},
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving this cardinality aggregation under executeEvents is also important to correctly calculate the number of executions of detection rules.

@banderror banderror force-pushed the detection-engine-health-api-enhancements branch 2 times, most recently from b322ff3 to 48224dc Compare October 3, 2023 12:15
@banderror banderror force-pushed the detection-engine-health-api-enhancements branch from 48224dc to 93a67fb Compare October 3, 2023 14:00
@banderror banderror enabled auto-merge (squash) October 3, 2023 14:18
@nikitaindik
Copy link
Contributor

@elasticmachine merge upstream

@nikitaindik
Copy link
Contributor

@elasticmachine merge upstream

@nikitaindik
Copy link
Contributor

@elasticmachine merge upstream

@nikitaindik
Copy link
Contributor

@elasticmachine merge upstream

Copy link
Contributor

@nikitaindik nikitaindik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked the implementation and tested by calling different methods. LGTM.

@nikitaindik
Copy link
Contributor

@elasticmachine merge upstream

@banderror banderror merged commit 8b6ba3d into elastic:main Oct 3, 2023
@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @banderror

@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Oct 3, 2023
@banderror banderror deleted the detection-engine-health-api-enhancements branch October 3, 2023 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting Feature:Rule Monitoring Security Solution Detection Rule Monitoring area release_note:skip Skip the PR/issue when compiling release notes Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.11.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants