Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RAC][Alerting][Security Solution] Adds Rule Execution UUID #113058

Merged
merged 49 commits into from
Jan 21, 2022

Conversation

spong
Copy link
Member

@spong spong commented Sep 24, 2021

Summary

Resolves: #110135

This PR is for introducing a new UUID (kibana.alert.rule.execution.uuid as defined in the AAD schema) for identifying individual rule executions. This id is introduced as a private readonly member of the alerting server task_manager, and plumbed through the executionHandler and to all appropriate alert event and event-log touch points.

For persistence when writing alerts within the RuleRegistry, kibana.alert.rule.execution.uuid is plumbed through getCommonAlertFields() so it is grouped with like fields and is picked up by both the createPersistenceRuleTypeWrapper used by Security Solution, and createLifecycleExecutor used by Observability rules.

Additionally on the Security Solution side, kibana.alert.rule.execution.uuid was plumbed through the RuleExecutionLog so that all events written to the event-log will now include this id so individual rule status events/metrics can be correlated with specific rule executions.

No UI facing changes were made, however kibana.alert.rule.execution.uuid is now available within the Alerts Table FieldBrowser, and can be toggled and viewed alongside alerts:

As visible when exploring event-log in Discover:

Checklist

Delete any items that are not applicable to this PR.

  • Documentation was added for features that require explanation or tutorials
    • Will need to sync with Doc folks on updates here.
  • Unit or functional tests were updated or added to match the most common scenarios

@elastic elastic deleted a comment from kibanamachine Nov 30, 2021
@spong spong added Feature:RAC label obsolete release_note:feature Makes this part of the condensed release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team v8.1.0 labels Jan 15, 2022
@spong spong marked this pull request as ready for review January 15, 2022 00:35
@spong spong requested review from a team as code owners January 15, 2022 00:35
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@@ -16,7 +16,7 @@ interface EqualCondition {
equal: number;
}

function isEqualConsition(
function isEqualCondition(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

Comment on lines 72 to 83
// Verify unique executionId generated per `action:execute` grouping
const eventExecutionIdSet = new Set();
events.forEach((event) => {
if (event?.event?.action === 'execute') {
eventExecutionIdSet.add(event?.kibana?.alert?.rule?.execution?.uuid);
expect(eventExecutionIdSet.size).to.equal(1);
eventExecutionIdSet.clear();
} else {
eventExecutionIdSet.add(event?.kibana?.alert?.rule?.execution?.uuid);
}
});

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll have to look closer at this - but I believe you are correct. But the assumption "ordering" with the notion of "after". This is all end up based on the timestamp, I believe (from the EL query), so there probably is some kind of race condition that if the rule runs again, before processing the events, that you'd have more of a mish-mash. Seems very unlikely though.

Comment on lines 72 to 83
// Verify unique executionId generated per `action:execute` grouping
const eventExecutionIdSet = new Set();
events.forEach((event) => {
if (event?.event?.action === 'execute') {
eventExecutionIdSet.add(event?.kibana?.alert?.rule?.execution?.uuid);
expect(eventExecutionIdSet.size).to.equal(1);
eventExecutionIdSet.clear();
} else {
eventExecutionIdSet.add(event?.kibana?.alert?.rule?.execution?.uuid);
}
});

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That logic seems fine for now.

It feels like we should add another test case along side this one that validates that the same uuid appears on multiple alerts detected by the same rule execution

Not sure about that - doesn't it already do that? Or it should make sure that there's at least > 1 *-instance events?

Copy link
Member

@pmuellr pmuellr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM; there's some comments in x-pack/test/alerting_api_integration/spaces_only/tests/alerting/event_log_alerts.ts about adding some additional tests, which I think would be simple and nice to have (ensure different execution ids are used for different executions, and make sure all of the different *-instance event types have the execution id in them.

Copy link
Contributor

@xcrzx xcrzx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested the changes locally. Created a rule, waited till it ran a couple of times, and checked that I could query execution events from the event log and correlate them to a particular execution. Excellent work, everything worked as expected 🚀 Thank you, @spong 👍

I think we should also log executionId to default Kibana log along with the other rule execution information. Added a comment on that.

@weltenwort weltenwort self-requested a review January 19, 2022 17:03
Copy link
Member

@weltenwort weltenwort left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

infra plugin changes LGTM

@spong spong enabled auto-merge (squash) January 21, 2022 00:38
@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/rule-data-utils 68 69 +1
alerting 274 275 +1
total +2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
apm 2.8MB 2.8MB +189.0B
cases 314.6KB 314.7KB +103.0B
infra 997.0KB 997.4KB +412.0B
observability 391.1KB 391.4KB +291.0B
total +995.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
apm 31.0KB 31.1KB +103.0B
securitySolution 245.6KB 245.7KB +103.0B
timelines 136.4KB 136.5KB +103.0B
uptime 24.8KB 24.9KB +103.0B
total +412.0B
Unknown metric groups

API count

id before after diff
@kbn/rule-data-utils 71 72 +1
alerting 282 283 +1
total +2

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @spong

@spong spong merged commit 48693d1 into elastic:main Jan 21, 2022
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Jan 21, 2022
@spong spong deleted the add-rule-execution-uuid branch January 21, 2022 02:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting enhancement New value added to drive a business result Feature:Alerting/RulesFramework Issues related to the Alerting Rules Framework Feature:Detection Rules Security Solution rules and Detection Engine Feature:RAC label obsolete release_note:feature Makes this part of the condensed release notes Team:Detection Rule Management Security Detection Rule Management Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.1.0
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

[RAC][Rule Registry] Add unique ID field for identifying all alerts created during a specific Rule execution
8 participants