[Security Solution][Detections] New data model for Rule Execution Log and changes in HTTP APIs #119603
Assignees
Labels
Feature:Rule Management
Security Solution Detection Rule Management area
Feature:Rule Monitoring
Security Solution Detection Rule Monitoring area
Team:Detection Rule Management
Security Detection Rule Management Team
Team:Detections and Resp
Security Detection Response Team
Team: SecuritySolution
Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
technical debt
Improvement of the software architecture and operational architecture
v8.1.0
Comments
banderror
added
technical debt
This was referenced Nov 24, 2021
banderror
added
Feature:Rule Management
28 tasks
banderror
changed the title
banderror
added a commit
that referenced
this issue
Jan 20, 2022
) **Epic:** #118324 **Tickets:** #119603, #119597, #91265, #118511 ## Summary The legacy rule execution logging implementation is replaced by a new one that introduces a new model for execution-related data, a new saved object and a new, cleaner interface and implementation. - [x] The legacy data model is deleted (`IRuleStatusResponseAttributes`, `IRuleStatusSOAttributes`) - [x] The legacy `siem-detection-engine-rule-status` saved object type is deleted and marked as deleted in `src/core` - [x] A new data model is introduced (`x-pack/plugins/security_solution/common/detection_engine/schemas/common/rule_monitoring.ts`). This data model doesn't contain a mixture of successful and failed statuses, which should simplify client-side code (e.g. the code of Rule Management and Monitoring tables, as well as Rule Details page). - [x] A new `siem-detection-engine-rule-execution-info` saved object is introduced (`x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/rule_execution_info/saved_object.ts`). - [x] This SO has 1:1 association with the rule SO, so every rule can have 0 or 1 execution info associated with it. This SO is used in order to 1) update the last execution status and metrics and 2) fetch execution data for N rules more efficiently comparing to the legacy SO. - [x] The logic of creating or updating this SOs is based on the "upsert" approach (planned in #118511). It does not fetch the SO by rule id before updating it anymore. - [x] Rule execution logging logic is rewritten (see `x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log`). The previous rule execution log client is split into two objects: `IRuleExecutionLogClient` for using it from route handlers, and `IRuleExecutionLogger` for writing logs from rule executors. - [x] `IRuleExecutionLogger` instance is scoped to the currently executing rule and space id. There's no need to pass rule id, name, type etc to `.logStatusChange()` every time. - [x] Rule executors and related functions are updated. - [x] API routes are updated, including the rule preview route which uses a special "spy" implementation of `IRuleExecutionLogger`. A rule returned from an API endpoint now has optional `execution_summary` field of type `RuleExecutionSummary`. - [x] UI is updated to use the new data model of `RuleExecutionSummary`: - [x] Rule Management and Monitoring tables - [x] Rule Details page - [x] A new API route is introduced for fetching rule execution events: `/internal/detection_engine/rules/{ruleId}/execution/events`. It is used for rendering the Failure History tab (last 5 failures) and is intended to be used in the coming UI of Rule Execution Log on the Details page. - [x] Rule Details page and Failure History tab are updated to use the new data models and API routes. - [x] I used `react-query` for fetching execution events - [x] See `x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/use_rule_execution_events.tsx` - [x] The lib is updated to the latest version - [x] Tests and fixed and updated according to all the changes - [x] Components related to rule execution statuses are all moved to `x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status`. - [x] I left a lot of `// TODO: https://github.com/elastic/kibana/pull/121644` comments in the code which I'm planning to address and remove in a follow-up PR. Lots of clean up work is needed, but I'd like to unblock the work on Rule Execution Log UI. ## In the next episodes - Address and remove `// TODO: https://github.com/elastic/kibana/pull/121644` comments in the code - Make sure that SO id generation for `siem-detection-engine-rule-execution-info` is safe and future-proof. Sync with the Core team. If there are risks, we will need to choose between risks and performance (reading the SO before updating it). It would be easy to submit a fix if needed. - Add APM integration. Use `withSecuritySpan` in methods of `rule_execution_log` citizens. - Add comments to the code and README. - Add test coverage. - Etc... ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) ### For maintainers - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
28 tasks
|
Implemented in #121644 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Parent ticket: #118324
Summary
Design a new data model for storing current rule execution info. Change all the Detections APIs that return rule monitoring data (statuses and metrics) using the new data model. Leave the current legacy rule status SO under the hood.
The goal is to have a clean HTTP API with the legacy rule status SO under the hood. Next step: #119597
IRuleExecutionLogClientto support the new data model.IRuleExecutionLogClientto return data in the new format.The text was updated successfully, but these errors were encountered: