From becaec81e1d64460dc8881d61f51f72fbce28f03 Mon Sep 17 00:00:00 2001 From: Georgii Gorbachev Date: Mon, 25 Jul 2022 22:09:17 +0200 Subject: [PATCH] [Security Solution][Detections] Extended rule execution logging to Event Log (#126063) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit **Epics:** https://github.com/elastic/kibana/issues/124947, https://github.com/elastic/kibana/issues/118324 **Fixes:** https://github.com/elastic/kibana/issues/131352 ## Summary Console logs written by rule executors can now be "routed" to the Event Log in addition to the console. A new table UI for viewing plain rule execution logs allows the user to look at all status changes, errors, warnings, info and debug messages on the Rule Details page. Screenshot 2022-07-20 at 15 31 54 **This feature is hidden under a feature flag and disabled by default** -- it might not be production-ready yet. We will need to work on product and UX/UI design in the subsequent development cycles to make it ready for release. Until then, we can start using it in development: it should make it easier to troubleshoot issues with rule execution. Add this flag to your Kibana config to enable this feature: ```yaml xpack.securitySolution.enableExperimental: ['extendedRuleExecutionLoggingEnabled'] ``` If the flag is enabled: - Rules will start writing console logs to Event Log as events of a new type `message`, in addition to the existing `status-change` and `execution-metrics` events. - Rule Details page will show a new tab called `Execution events`. This tab will contain a table with plain execution logs. - In Stack Management, you will find two new Kibana Advanced Settings for controlling this extended logging. As for the new Kibana Advanced Settings, by default: - Extended logging is enabled. - The minimum console log level to be written to Event Log is `error`. This only affects the new `message` events. Screenshot 2022-07-20 at 15 41 29 ## Implementation details **Important change**: refactored the folder structure as our first step to **domain-driven architecture** and **splitting the Detection Engine into subdomains**. - Extracted most of the code related to Rule Execution Log and Rule Monitoring in general into a subdomain called `rule_monitoring`. This subdomain now lives in three folders: - `security_solution/common/detection_engine/rule_monitoring` - `security_solution/public/detection_engine/rule_monitoring` - `security_solution/server/lib/detection_engine/rule_monitoring` - Tried to create a developer-friendly and clear folder structure within the subdomain. Other changes: - Changed all rule executors to write console logs via an instance of `IRuleExecutionLogForExecutors` instead of the console `Logger`. - `IRuleExecutionLogForExecutors` is passed to rule executors and downstream functions they call. - `Logger` is not passed anymore. - `buildRuleMessage` and `buildRuleMessageFactory` are deleted. - Added support for writing console logs to Event Log. - Added a new rule execution event type `message` for writing console logs to Event Log. - Every rule execution event now has a `log.level` and `event.severity`. - Improved the format of console logs written by rules. - Created a child logger for console logs of rule executors: `plugins.securitySolution.ruleExecution`. - Added rule static “signature” ID (`rule.rule_id`) as a correlation id to the logs. - Cleaned up the formatting of console logs. - Fixed `ExtMeta` to use interfaces instead of type intersection due to a found [bug](https://github.com/microsoft/TypeScript/issues/47935) in TypeScript that affected this type. - Made changes in the domain model. - Renamed the `AggregateRuleExecutionEvent` into the `RuleExecutionResult`. - The new plain event is called `RuleExecutionEvent`. - Finalized the API endpoint for fetching plain execution logs. - Built a Rule Execution Events Table UI for showing and filtering plain execution logs. - Did some refactoring to extract reusable components/hooks to make development of tables easier in the future. ## Execution events table UI For context, this is how the existing `Execution logs` table looks like when the flag is off (notice the renaming to `Execution results`: Screenshot 2022-07-20 at 15 29 04 This is the new `Execution events` table when the flag is on: Screenshot 2022-07-20 at 15 31 54 Showing only trace and debug events: Screenshot 2022-07-20 at 15 33 48 Showing only status changes: Screenshot 2022-07-20 at 15 35 04 Showing only warning and error `message`s: Screenshot 2022-07-20 at 15 37 11 Expanded rows showing details of the corresponding events: Screenshot 2022-07-20 at 15 39 16 Screenshot 2022-07-20 at 15 39 56 ## Console logs Example: ``` [2022-02-23T17:05:09.901+03:00][DEBUG][plugins.securitySolution.ruleExecution] [+] Starting Signal Rule execution [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] [2022-02-23T17:05:09.907+03:00][DEBUG][plugins.securitySolution.ruleExecution] interval: 5m [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] [2022-02-23T17:05:09.908+03:00][INFO ][plugins.securitySolution.ruleExecution] Changing rule status to "running" [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] [2022-02-23T17:05:10.595+03:00][WARN ][plugins.securitySolution.ruleExecution] This rule is attempting to query data from Elasticsearch indices listed in the "Index pattern" section of the rule definition, however no index matching: ["logs-endpoint.alerts-*"] was found. This warning will continue to appear until a matching index is created or this rule is de-activated. If you have recently enrolled agents enabled with Endpoint Security through Fleet, this warning should stop once an alert is sent from an agent. [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] [2022-02-23T17:05:10.595+03:00][WARN ][plugins.securitySolution.ruleExecution] Changing rule status to "partial failure" [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] [2022-02-23T17:05:11.630+03:00][DEBUG][plugins.securitySolution.ruleExecution] sortIds: undefined [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] [2022-02-23T17:05:11.634+03:00][DEBUG][plugins.securitySolution.ruleExecution] totalHits: 0 [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] [2022-02-23T17:05:11.634+03:00][DEBUG][plugins.securitySolution.ruleExecution] searchResult.hit.hits.length: 0 [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] [2022-02-23T17:05:11.635+03:00][DEBUG][plugins.securitySolution.ruleExecution] totalHits was 0, exiting early [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] [2022-02-23T17:05:11.636+03:00][DEBUG][plugins.securitySolution.ruleExecution] [+] completed bulk index of 0 [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] [2022-02-23T17:05:11.636+03:00][DEBUG][plugins.securitySolution.ruleExecution] [+] Signal Rule execution completed. [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] [2022-02-23T17:05:11.638+03:00][DEBUG][plugins.securitySolution.ruleExecution] [+] Finished indexing 0 signals into .alerts-security.alerts [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] [2022-02-23T17:05:11.639+03:00][DEBUG][plugins.securitySolution.ruleExecution] [+] Finished indexing 0 signals searched between date ranges [ { "to": "2022-02-23T14:05:09.775Z", "from": "2022-02-23T13:55:09.775Z", "maxSignals": 10000 } ] [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] ``` Note that: - The logger name is now `plugins.securitySolution.ruleExecution`, which allows to turn on _only_ rule execution logs in the config (could be useful when debugging). - Every log message has a suffix with correlation ids: `[siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default]` ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] `x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/README.md` - [x] Various JSDoc comments - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) ### For maintainers - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --- .../src/default_csv_array/index.test.ts | 214 ++++++++++++++ .../src/default_csv_array/index.ts | 48 ++++ .../src/default_value/index.test.ts | 106 +++++++ .../src/default_value/index.ts | 31 ++ .../src/index.ts | 2 + .../event_log/server/event_log_client.ts | 1 + .../security_solution/common/constants.ts | 12 +- .../request_schema.test.ts | 168 +++++++++++ .../request_schema.ts | 43 +++ .../response_schema.mock.ts | 25 ++ .../response_schema.ts | 21 ++ .../request_schema.test.ts | 271 ++++++++++++++++++ .../request_schema.ts | 72 +++++ .../response_schema.mock.ts | 21 ++ .../response_schema.ts} | 14 +- .../rule_monitoring/api/urls.ts | 18 ++ .../detection_engine/rule_monitoring/index.ts | 20 ++ .../detection_engine/rule_monitoring/mocks.ts | 13 + .../model/execution_event.mock.ts | 152 ++++++++++ .../rule_monitoring/model/execution_event.ts | 60 ++++ .../model/execution_metrics.ts | 19 ++ .../model/execution_result.mock.ts | 176 ++++++++++++ .../rule_monitoring/model/execution_result.ts | 50 ++++ .../model/execution_settings.ts | 22 ++ .../rule_monitoring/model/execution_status.ts | 80 ++++++ .../model/execution_summary.mock.ts | 43 +++ .../model/execution_summary.ts | 22 ++ .../rule_monitoring/model/log_level.ts | 82 ++++++ .../detection_engine/schemas/common/index.ts | 3 +- .../schemas/common/pagination.ts | 28 ++ .../schemas/common/rule_monitoring.ts | 147 ---------- .../schemas/common/schemas.ts | 24 -- .../schemas/common/sorting.test.ts | 87 ++++++ .../schemas/common/sorting.ts | 46 +++ .../schemas/request/find_rules_schema.ts | 14 +- .../get_rule_execution_events_schema.test.ts | 117 -------- .../get_rule_execution_events_schema.ts | 105 ------- .../detection_engine/schemas/request/index.ts | 4 +- .../schemas/request/rule_schemas.ts | 4 +- .../schemas/response/index.ts | 1 - .../schemas/response/rules_schema.ts | 6 +- .../common/experimental_features.ts | 10 + .../common/utils/enum_from_string.test.ts | 35 +++ .../common/utils/enum_from_string.ts | 37 +++ .../common/utils/to_array.ts | 4 +- .../components/health_truncate_text/index.tsx | 2 +- .../public/detection_engine/jest.config.js | 26 ++ .../api/__mocks__/api_client.ts | 73 +++++ .../rule_monitoring/api/__mocks__/index.ts | 8 + .../rule_monitoring/api/api_client.test.ts | 138 +++++++++ .../rule_monitoring/api/api_client.ts | 85 ++++++ .../api/api_client_interface.ts | 124 ++++++++ .../rule_monitoring/api/index.ts | 9 + .../basic/filters/event_type_filter/index.tsx | 40 +++ .../event_type_filter/translations.ts} | 14 +- .../filters/execution_status_filter/index.tsx | 44 +++ .../execution_status_filter/translations.ts} | 14 +- .../basic/filters/log_level_filter/index.tsx | 40 +++ .../filters/log_level_filter/translations.ts | 15 + .../filters/multiselect_filter/index.tsx | 112 ++++++++ .../indicators/event_type_indicator/index.tsx | 29 ++ .../event_type_indicator/translations.ts | 29 ++ .../indicators/event_type_indicator/utils.ts | 38 +++ .../execution_status_indicator/index.tsx | 41 +++ .../indicators/log_level_indicator/index.tsx | 25 ++ .../indicators/log_level_indicator/utils.ts | 32 +++ .../basic/tables/use_expandable_rows.tsx | 57 ++++ .../components/basic/tables/use_pagination.ts | 60 ++++ .../components/basic/tables/use_sorting.ts | 38 +++ .../components/basic/text/text_block.tsx | 31 ++ .../components/basic/text/truncated_text.tsx | 19 ++ .../execution_events_table.tsx | 117 ++++++++ .../execution_events_table_row_details.tsx | 40 +++ .../execution_events_table/translations.ts | 71 +++++ .../execution_events_table/use_columns.tsx | 100 +++++++ .../use_execution_events.test.tsx | 127 ++++++++ .../use_execution_events.ts | 35 +++ .../execution_events_table/use_filters.ts | 25 ++ .../execution_results_table/translations.ts | 15 + .../use_execution_results.test.tsx} | 27 +- .../use_execution_results.tsx | 34 +++ .../detection_engine/rule_monitoring/index.ts | 15 + .../use_execution_settings.ts | 50 ++++ .../detection_engine/rule_monitoring/mocks.ts | 8 + .../rule_execution_status/rule_status.tsx | 2 +- .../rule_status_badge.test.tsx | 2 +- .../rule_status_badge.tsx | 2 +- .../rule_status_failed_callout.test.tsx | 2 +- .../rule_status_failed_callout.tsx | 2 +- .../rules/rule_execution_status/utils.ts | 2 +- .../detection_engine/rules/__mocks__/api.ts | 44 --- .../detection_engine/rules/api.test.ts | 55 +--- .../containers/detection_engine/rules/api.ts | 70 +---- .../detection_engine/rules/index.ts | 1 - .../containers/detection_engine/rules/mock.ts | 168 ----------- .../detection_engine/rules/translations.ts | 7 - .../detection_engine/rules/types.ts | 6 +- .../rules/use_rule_execution_events.tsx | 80 ------ .../rules/all/use_columns.tsx | 2 +- .../__mocks__/rule_details_context.tsx | 2 +- .../execution_log_search_bar.test.tsx.snap | 74 +---- .../execution_log_columns.tsx | 30 +- .../execution_log_search_bar.test.tsx | 7 +- .../execution_log_search_bar.tsx | 101 ++----- .../execution_log_table.test.tsx | 17 +- .../execution_log_table.tsx | 39 ++- .../execution_log_table/translations.ts | 14 + .../detection_engine/rules/details/index.tsx | 53 +++- .../rules/details/rule_details_context.tsx | 14 +- .../rules/details/translations.ts | 19 +- .../routes/__mocks__/request_context.ts | 2 +- .../routes/__mocks__/request_responses.ts | 108 ------- .../routes/rules/create_rules_route.test.ts | 4 - .../routes/rules/find_rules_route.test.ts | 4 - .../get_rule_execution_events_route.test.ts | 57 ---- .../routes/rules/patch_rules_route.test.ts | 4 - .../routes/rules/preview_rules_route.ts | 4 +- .../routes/rules/read_rules_route.test.ts | 4 - .../routes/rules/update_rules_route.test.ts | 4 - .../detection_engine/routes/rules/utils.ts | 4 +- .../routes/rules/validate.test.ts | 5 +- .../detection_engine/routes/rules/validate.ts | 2 +- .../rule_execution_log/__mocks__/index.ts | 43 --- .../rule_execution_log/client_factories.ts | 56 ---- .../client_for_executors/client.ts | 156 ---------- .../client_for_executors/client_interface.ts | 49 ---- .../client_for_routes/client_interface.ts | 86 ------ .../event_log/event_log_reader.ts | 178 ------------ .../event_log/event_log_writer.ts | 128 --------- .../utils/console_logging.ts | 24 -- .../get_rule_execution_events/route.test.ts | 94 ++++++ .../api/get_rule_execution_events/route.ts | 64 +++++ .../get_rule_execution_results/route.test.ts | 69 +++++ .../api/get_rule_execution_results/route.ts} | 37 ++- .../rule_monitoring/api/register_routes.ts | 15 + .../detection_engine/rule_monitoring/index.ts | 9 + .../logic/rule_execution_log/README.md | 195 +++++++++++++ .../rule_execution_log/__mocks__/index.ts | 79 +++++ .../client_for_executors/client.ts | 242 ++++++++++++++++ .../client_for_executors/client_interface.ts | 119 ++++++++ .../client_for_executors/correlation_ids.ts | 87 ++++++ .../client_for_routes/client.ts | 134 ++++----- .../client_for_routes/client_interface.ts | 107 +++++++ .../rule_execution_log/event_log/constants.ts | 5 - .../event_log/event_log_reader.ts | 243 ++++++++++++++++ .../event_log/event_log_writer.ts | 189 ++++++++++++ .../index.test.ts | 2 +- .../get_execution_event_aggregation/index.ts | 15 +- .../get_execution_event_aggregation/types.ts | 0 .../event_log/register_event_log_provider.ts | 5 +- .../saved_objects_client.ts | 2 +- .../saved_objects_type.ts | 2 +- .../saved_objects_utils.ts | 0 .../fetch_rule_execution_settings.ts | 89 ++++++ .../logic}/rule_execution_log/index.ts | 5 +- .../merge_rule_execution_summary.ts | 10 +- .../logic/rule_execution_log/service.ts | 80 ++++++ .../rule_execution_log/service_interface.ts | 35 +++ .../utils/console_logging.ts | 30 ++ .../rule_execution_log/utils/normalization.ts | 0 .../detection_engine/rule_monitoring/mocks.ts | 8 + .../rule_types/__mocks__/rule_type.ts | 1 - .../create_security_rule_type_wrapper.ts | 150 ++++------ .../rule_types/eql/create_eql_alert_type.ts | 19 +- .../factories/build_rule_message_factory.ts | 28 -- .../factories/bulk_create_factory.ts | 18 +- .../rule_types/factories/index.ts | 1 - .../create_indicator_match_alert_type.ts | 27 +- .../rule_types/ml/create_ml_alert_type.ts | 19 +- .../new_terms/create_new_terms_alert_type.ts | 11 +- .../query/create_query_alert_type.test.ts | 7 +- .../query/create_query_alert_type.ts | 23 +- .../create_saved_query_alert_type.ts | 23 +- .../threshold/create_threshold_alert_type.ts | 16 +- .../lib/detection_engine/rule_types/types.ts | 18 +- .../rules/delete_rules.test.ts | 2 +- .../lib/detection_engine/rules/types.ts | 25 +- .../rules/update_prepacked_rules.test.ts | 2 +- .../rules/update_prepacked_rules.ts | 2 +- .../schemas/rule_converters.ts | 4 +- .../signals/bulk_create_ml_signals.ts | 6 +- .../signals/executors/eql.test.ts | 16 +- .../detection_engine/signals/executors/eql.ts | 12 +- .../signals/executors/ml.test.ts | 44 ++- .../detection_engine/signals/executors/ml.ts | 28 +- .../signals/executors/query.ts | 27 +- .../signals/executors/threat_match.ts | 19 +- .../signals/executors/threshold.test.ts | 33 +-- .../signals/executors/threshold.ts | 17 +- .../create_field_and_set_tuples.test.ts | 41 +-- .../filters/create_field_and_set_tuples.ts | 6 +- .../create_set_to_filter_against.test.ts | 17 +- .../filters/create_set_to_filter_against.ts | 15 +- .../filter_events_against_list.test.ts | 48 ++-- .../filters/filter_events_against_list.ts | 20 +- .../detection_engine/signals/filters/types.ts | 13 +- .../preview/preview_rule_execution_logger.ts | 43 +-- .../signals/rule_messages.test.ts | 61 ---- .../detection_engine/signals/rule_messages.ts | 25 -- .../signals/search_after_bulk_create.test.ts | 64 ++--- .../signals/search_after_bulk_create.ts | 48 ++-- .../signals/send_telemetry_events.ts | 10 +- .../signals/single_search_after.test.ts | 25 +- .../signals/single_search_after.ts | 13 +- .../threat_mapping/build_threat_enrichment.ts | 14 +- .../threat_mapping/create_event_signal.ts | 36 +-- .../threat_mapping/create_threat_signal.ts | 33 +-- .../threat_mapping/create_threat_signals.ts | 54 ++-- .../signals/threat_mapping/get_event_count.ts | 16 +- .../signals/threat_mapping/get_threat_list.ts | 19 +- .../signals/threat_mapping/types.ts | 22 +- .../bulk_create_threshold_signals.ts | 2 - .../threshold/find_threshold_signals.test.ts | 27 +- .../threshold/find_threshold_signals.ts | 15 +- .../lib/detection_engine/signals/types.ts | 7 +- .../detection_engine/signals/utils.test.ts | 62 +--- .../lib/detection_engine/signals/utils.ts | 69 ++--- .../security_solution/server/plugin.ts | 15 +- .../server/request_context_factory.ts | 14 +- .../security_solution/server/routes/index.ts | 4 +- .../security_solution/server/saved_objects.ts | 2 +- .../plugins/security_solution/server/types.ts | 2 +- .../security_solution/server/ui_settings.ts | 100 +++++++ .../queries/utils/get_search_for_all_rules.ts | 2 +- .../utils/get_search_for_custom_rules.ts | 2 +- .../utils/get_search_for_elastic_rules.ts | 2 +- .../translations/translations/fr-FR.json | 2 - .../translations/translations/ja-JP.json | 2 - .../translations/translations/zh-CN.json | 2 - .../group1/check_privileges.ts | 2 +- .../group1/create_new_terms.ts | 2 +- .../group1/create_rules.ts | 2 +- .../group1/create_threat_matching.ts | 4 +- .../group1/generating_signals.ts | 2 +- ...vents.ts => get_rule_execution_results.ts} | 22 +- .../security_and_spaces/group1/index.ts | 2 +- .../security_and_spaces/group1/timestamps.ts | 2 +- .../utils/get_open_signals.ts | 4 +- .../utils/wait_for_rule_success_or_status.ts | 2 +- 239 files changed, 6517 insertions(+), 3092 deletions(-) create mode 100644 packages/kbn-securitysolution-io-ts-types/src/default_csv_array/index.test.ts create mode 100644 packages/kbn-securitysolution-io-ts-types/src/default_csv_array/index.ts create mode 100644 packages/kbn-securitysolution-io-ts-types/src/default_value/index.test.ts create mode 100644 packages/kbn-securitysolution-io-ts-types/src/default_value/index.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_events/request_schema.test.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_events/request_schema.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_events/response_schema.mock.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_events/response_schema.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_results/request_schema.test.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_results/request_schema.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_results/response_schema.mock.ts rename x-pack/plugins/security_solution/common/detection_engine/{schemas/response/get_rule_execution_events_response.ts => rule_monitoring/api/get_rule_execution_results/response_schema.ts} (51%) create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/urls.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/index.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/mocks.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_event.mock.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_event.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_metrics.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_result.mock.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_result.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_settings.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_status.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_summary.mock.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_summary.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/log_level.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/schemas/common/pagination.ts delete mode 100644 x-pack/plugins/security_solution/common/detection_engine/schemas/common/rule_monitoring.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/schemas/common/sorting.test.ts create mode 100644 x-pack/plugins/security_solution/common/detection_engine/schemas/common/sorting.ts delete mode 100644 x-pack/plugins/security_solution/common/detection_engine/schemas/request/get_rule_execution_events_schema.test.ts delete mode 100644 x-pack/plugins/security_solution/common/detection_engine/schemas/request/get_rule_execution_events_schema.ts create mode 100644 x-pack/plugins/security_solution/common/utils/enum_from_string.test.ts create mode 100644 x-pack/plugins/security_solution/common/utils/enum_from_string.ts create mode 100644 x-pack/plugins/security_solution/public/detection_engine/jest.config.js create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/__mocks__/api_client.ts create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/__mocks__/index.ts create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/api_client.test.ts create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/api_client.ts create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/api_client_interface.ts create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/index.ts create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/event_type_filter/index.tsx rename x-pack/plugins/security_solution/{server/lib/detection_engine/signals/rule_messages.mock.ts => public/detection_engine/rule_monitoring/components/basic/filters/event_type_filter/translations.ts} (54%) create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/execution_status_filter/index.tsx rename x-pack/plugins/security_solution/{server/lib/detection_engine/signals/__mocks__/build_rule_message.mock.ts => public/detection_engine/rule_monitoring/components/basic/filters/execution_status_filter/translations.ts} (54%) create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/log_level_filter/index.tsx create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/log_level_filter/translations.ts create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/multiselect_filter/index.tsx create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/event_type_indicator/index.tsx create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/event_type_indicator/translations.ts create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/event_type_indicator/utils.ts create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/execution_status_indicator/index.tsx create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/log_level_indicator/index.tsx create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/log_level_indicator/utils.ts create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/tables/use_expandable_rows.tsx create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/tables/use_pagination.ts create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/tables/use_sorting.ts create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/text/text_block.tsx create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/text/truncated_text.tsx create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/execution_events_table.tsx create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/execution_events_table_row_details.tsx create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/translations.ts create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/use_columns.tsx create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/use_execution_events.test.tsx create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/use_execution_events.ts create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/use_filters.ts create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_results_table/translations.ts rename x-pack/plugins/security_solution/public/{detections/containers/detection_engine/rules/use_rule_execution_events.test.tsx => detection_engine/rule_monitoring/components/execution_results_table/use_execution_results.test.tsx} (86%) create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_results_table/use_execution_results.tsx create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/index.ts create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/logic/execution_settings/use_execution_settings.ts create mode 100644 x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/mocks.ts delete mode 100644 x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/use_rule_execution_events.tsx delete mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/get_rule_execution_events_route.test.ts delete mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/__mocks__/index.ts delete mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/client_factories.ts delete mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/client_for_executors/client.ts delete mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/client_for_executors/client_interface.ts delete mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/client_for_routes/client_interface.ts delete mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/event_log_reader.ts delete mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/event_log_writer.ts delete mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/utils/console_logging.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/get_rule_execution_events/route.test.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/get_rule_execution_events/route.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/get_rule_execution_results/route.test.ts rename x-pack/plugins/security_solution/server/lib/detection_engine/{routes/rules/get_rule_execution_events_route.ts => rule_monitoring/api/get_rule_execution_results/route.ts} (52%) create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/register_routes.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/index.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/README.md create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/__mocks__/index.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_executors/client.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_executors/client_interface.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_executors/correlation_ids.ts rename x-pack/plugins/security_solution/server/lib/detection_engine/{ => rule_monitoring/logic}/rule_execution_log/client_for_routes/client.ts (53%) create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_routes/client_interface.ts rename x-pack/plugins/security_solution/server/lib/detection_engine/{ => rule_monitoring/logic}/rule_execution_log/event_log/constants.ts (75%) create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/event_log_reader.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/event_log_writer.ts rename x-pack/plugins/security_solution/server/lib/detection_engine/{ => rule_monitoring/logic}/rule_execution_log/event_log/get_execution_event_aggregation/index.test.ts (99%) rename x-pack/plugins/security_solution/server/lib/detection_engine/{ => rule_monitoring/logic}/rule_execution_log/event_log/get_execution_event_aggregation/index.ts (97%) rename x-pack/plugins/security_solution/server/lib/detection_engine/{ => rule_monitoring/logic}/rule_execution_log/event_log/get_execution_event_aggregation/types.ts (100%) rename x-pack/plugins/security_solution/server/lib/detection_engine/{ => rule_monitoring/logic}/rule_execution_log/event_log/register_event_log_provider.ts (70%) rename x-pack/plugins/security_solution/server/lib/detection_engine/{ => rule_monitoring/logic}/rule_execution_log/execution_saved_object/saved_objects_client.ts (98%) rename x-pack/plugins/security_solution/server/lib/detection_engine/{ => rule_monitoring/logic}/rule_execution_log/execution_saved_object/saved_objects_type.ts (96%) rename x-pack/plugins/security_solution/server/lib/detection_engine/{ => rule_monitoring/logic}/rule_execution_log/execution_saved_object/saved_objects_utils.ts (100%) create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/execution_settings/fetch_rule_execution_settings.ts rename x-pack/plugins/security_solution/server/lib/detection_engine/{ => rule_monitoring/logic}/rule_execution_log/index.ts (80%) rename x-pack/plugins/security_solution/server/lib/detection_engine/{ => rule_monitoring/logic}/rule_execution_log/merge_rule_execution_summary.ts (76%) create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/service.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/service_interface.ts create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/utils/console_logging.ts rename x-pack/plugins/security_solution/server/lib/detection_engine/{ => rule_monitoring/logic}/rule_execution_log/utils/normalization.ts (100%) create mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/mocks.ts delete mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/factories/build_rule_message_factory.ts delete mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/signals/rule_messages.test.ts delete mode 100644 x-pack/plugins/security_solution/server/lib/detection_engine/signals/rule_messages.ts rename x-pack/test/detection_engine_api_integration/security_and_spaces/group1/{get_rule_execution_events.ts => get_rule_execution_results.ts} (94%) diff --git a/packages/kbn-securitysolution-io-ts-types/src/default_csv_array/index.test.ts b/packages/kbn-securitysolution-io-ts-types/src/default_csv_array/index.test.ts new file mode 100644 index 0000000000000..1f927565ca626 --- /dev/null +++ b/packages/kbn-securitysolution-io-ts-types/src/default_csv_array/index.test.ts @@ -0,0 +1,214 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +import * as t from 'io-ts'; +import { pipe } from 'fp-ts/lib/pipeable'; +import { left } from 'fp-ts/lib/Either'; +import { foldLeftRight, getPaths } from '@kbn/securitysolution-io-ts-utils'; +import { defaultCsvArray } from '.'; + +describe('defaultCsvArray', () => { + describe('Creates a schema of an array that works in the following way:', () => { + type TestType = t.TypeOf; + const TestType = t.union( + [t.literal('foo'), t.literal('bar'), t.literal('42'), t.null, t.undefined], + 'TestType' + ); + + const TestCsvArray = defaultCsvArray(TestType); + + describe('Name of the schema', () => { + it('has a default value', () => { + const CsvArray = defaultCsvArray(TestType); + expect(CsvArray.name).toEqual('DefaultCsvArray'); + }); + + it('can be overriden', () => { + const CsvArray = defaultCsvArray(TestType, 'CustomName'); + expect(CsvArray.name).toEqual('CustomName'); + }); + }); + + describe('Validation succeeds', () => { + describe('when input is a single valid string value', () => { + const cases = [{ input: 'foo' }, { input: 'bar' }, { input: '42' }]; + + cases.forEach(({ input }) => { + it(`${input}`, () => { + const decoded = TestCsvArray.decode(input); + const message = pipe(decoded, foldLeftRight); + const expectedOutput = [input]; // note that it's an array after decode + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual(expectedOutput); + }); + }); + }); + + describe('when input is an array of valid string values', () => { + const cases = [ + { input: ['foo'] }, + { input: ['foo', 'bar'] }, + { input: ['foo', 'bar', '42'] }, + ]; + + cases.forEach(({ input }) => { + it(`${input}`, () => { + const decoded = TestCsvArray.decode(input); + const message = pipe(decoded, foldLeftRight); + const expectedOutput = input; + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual(expectedOutput); + }); + }); + }); + + describe('when input is a string which is a comma-separated array of valid values', () => { + const cases = [ + { + input: 'foo,bar', + expectedOutput: ['foo', 'bar'], + }, + { + input: 'foo,bar,42', + expectedOutput: ['foo', 'bar', '42'], + }, + ]; + + cases.forEach(({ input, expectedOutput }) => { + it(`${input}`, () => { + const decoded = TestCsvArray.decode(input); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual(expectedOutput); + }); + }); + }); + }); + + describe('Validation fails', () => { + describe('when input is a single invalid value', () => { + const cases = [ + { + input: 'val', + expectedErrors: ['Invalid value "val" supplied to "DefaultCsvArray"'], + }, + { + input: '5', + expectedErrors: ['Invalid value "5" supplied to "DefaultCsvArray"'], + }, + { + input: 5, + expectedErrors: ['Invalid value "5" supplied to "DefaultCsvArray"'], + }, + { + input: {}, + expectedErrors: ['Invalid value "{}" supplied to "DefaultCsvArray"'], + }, + ]; + + cases.forEach(({ input, expectedErrors }) => { + it(`${input}`, () => { + const decoded = TestCsvArray.decode(input); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual(expectedErrors); + expect(message.schema).toEqual({}); + }); + }); + }); + + describe('when input is an array of invalid values', () => { + const cases = [ + { + input: ['value 1', 5], + expectedErrors: [ + 'Invalid value "value 1" supplied to "DefaultCsvArray"', + 'Invalid value "5" supplied to "DefaultCsvArray"', + ], + }, + { + input: ['value 1', 'foo'], + expectedErrors: ['Invalid value "value 1" supplied to "DefaultCsvArray"'], + }, + { + input: ['', 5, {}], + expectedErrors: [ + 'Invalid value "" supplied to "DefaultCsvArray"', + 'Invalid value "5" supplied to "DefaultCsvArray"', + 'Invalid value "{}" supplied to "DefaultCsvArray"', + ], + }, + ]; + + cases.forEach(({ input, expectedErrors }) => { + it(`${input}`, () => { + const decoded = TestCsvArray.decode(input); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual(expectedErrors); + expect(message.schema).toEqual({}); + }); + }); + }); + + describe('when input is a string which is a comma-separated array of invalid values', () => { + const cases = [ + { + input: 'value 1,5', + expectedErrors: [ + 'Invalid value "value 1" supplied to "DefaultCsvArray"', + 'Invalid value "5" supplied to "DefaultCsvArray"', + ], + }, + { + input: 'value 1,foo', + expectedErrors: ['Invalid value "value 1" supplied to "DefaultCsvArray"'], + }, + { + input: ',5,{}', + expectedErrors: [ + 'Invalid value "" supplied to "DefaultCsvArray"', + 'Invalid value "5" supplied to "DefaultCsvArray"', + 'Invalid value "{}" supplied to "DefaultCsvArray"', + ], + }, + ]; + + cases.forEach(({ input, expectedErrors }) => { + it(`${input}`, () => { + const decoded = TestCsvArray.decode(input); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual(expectedErrors); + expect(message.schema).toEqual({}); + }); + }); + }); + }); + + describe('Validation returns default value (an empty array)', () => { + describe('when input is', () => { + const cases = [{ input: null }, { input: undefined }, { input: '' }, { input: [] }]; + + cases.forEach(({ input }) => { + it(`${input}`, () => { + const decoded = TestCsvArray.decode(input); + const message = pipe(decoded, foldLeftRight); + const expectedOutput: string[] = []; + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual(expectedOutput); + }); + }); + }); + }); + }); +}); diff --git a/packages/kbn-securitysolution-io-ts-types/src/default_csv_array/index.ts b/packages/kbn-securitysolution-io-ts-types/src/default_csv_array/index.ts new file mode 100644 index 0000000000000..0f9a7b859409b --- /dev/null +++ b/packages/kbn-securitysolution-io-ts-types/src/default_csv_array/index.ts @@ -0,0 +1,48 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +import * as t from 'io-ts'; +import { Either } from 'fp-ts/lib/Either'; + +/** + * Creates a schema of an array that works in the following way: + * - If input is a CSV string, it will be parsed to an array which will be validated. + * - If input is an array, each item is validated to match `itemSchema`. + * - If input is a single string, it is validated to match `itemSchema`. + * - If input is not specified, the result will be set to [] (empty array): + * - null, undefined, empty string, empty array + * + * In all cases when an input is valid, the resulting decoded value will be an array, + * either an empty one or containing valid items. + * + * @param itemSchema Schema of the array's items. + * @param name (Optional) Name of the resulting schema. + */ +export const defaultCsvArray = ( + itemSchema: t.Type, + name?: string +): t.Type => { + return new t.Type( + name ?? `DefaultCsvArray<${itemSchema.name}>`, + t.array(itemSchema).is, + (input, context): Either => { + if (input == null) { + return t.success([]); + } else if (typeof input === 'string') { + if (input === '') { + return t.success([]); + } else { + return t.array(itemSchema).validate(input.split(','), context); + } + } else { + return t.array(itemSchema).validate(input, context); + } + }, + t.identity + ); +}; diff --git a/packages/kbn-securitysolution-io-ts-types/src/default_value/index.test.ts b/packages/kbn-securitysolution-io-ts-types/src/default_value/index.test.ts new file mode 100644 index 0000000000000..0fc5e7584d944 --- /dev/null +++ b/packages/kbn-securitysolution-io-ts-types/src/default_value/index.test.ts @@ -0,0 +1,106 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +import * as t from 'io-ts'; +import { pipe } from 'fp-ts/lib/pipeable'; +import { left } from 'fp-ts/lib/Either'; +import { foldLeftRight, getPaths } from '@kbn/securitysolution-io-ts-utils'; +import { defaultValue } from '.'; + +describe('defaultValue', () => { + describe('Creates a schema that sets a default value if the input value is not specified', () => { + type TestType = t.TypeOf; + const TestType = t.union([t.string, t.number, t.null, t.undefined], 'TestType'); + + const DefaultValue = defaultValue(TestType, 42); + + describe('Name of the schema', () => { + it('has a default value', () => { + expect(defaultValue(TestType, 42).name).toEqual('DefaultValue'); + }); + + it('can be overriden', () => { + expect(defaultValue(TestType, 42, 'CustomName').name).toEqual('CustomName'); + }); + }); + + describe('Validation succeeds', () => { + describe('when input is a valid value', () => { + const cases = [ + { input: 'foo' }, + { input: '42' }, + { input: 42 }, + // including all "falsey" values which are not null or undefined + { input: '' }, + { input: 0 }, + ]; + + cases.forEach(({ input }) => { + it(`${input}`, () => { + const decoded = DefaultValue.decode(input); + const message = pipe(decoded, foldLeftRight); + const expectedOutput = input; + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual(expectedOutput); + }); + }); + }); + }); + + describe('Validation fails', () => { + describe('when input is an invalid value', () => { + const cases = [ + { + input: {}, + expectedErrors: ['Invalid value "{}" supplied to "DefaultValue"'], + }, + { + input: { foo: 42 }, + expectedErrors: ['Invalid value "{"foo":42}" supplied to "DefaultValue"'], + }, + { + input: [], + expectedErrors: ['Invalid value "[]" supplied to "DefaultValue"'], + }, + { + input: ['foo', 42], + expectedErrors: ['Invalid value "["foo",42]" supplied to "DefaultValue"'], + }, + ]; + + cases.forEach(({ input, expectedErrors }) => { + it(`${input}`, () => { + const decoded = DefaultValue.decode(input); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual(expectedErrors); + expect(message.schema).toEqual({}); + }); + }); + }); + }); + + describe('Validation returns specified default value', () => { + describe('when input is', () => { + const cases = [{ input: null }, { input: undefined }]; + + cases.forEach(({ input }) => { + it(`${input}`, () => { + const decoded = DefaultValue.decode(input); + const message = pipe(decoded, foldLeftRight); + const expectedOutput = 42; + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual(expectedOutput); + }); + }); + }); + }); + }); +}); diff --git a/packages/kbn-securitysolution-io-ts-types/src/default_value/index.ts b/packages/kbn-securitysolution-io-ts-types/src/default_value/index.ts new file mode 100644 index 0000000000000..1785225202820 --- /dev/null +++ b/packages/kbn-securitysolution-io-ts-types/src/default_value/index.ts @@ -0,0 +1,31 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +import * as t from 'io-ts'; +import type { Either } from 'fp-ts/lib/Either'; + +/** + * Creates a schema that sets a default value if the input value is not specified. + * + * @param valueSchema Base schema of a value. + * @param value Default value to set. + * @param name (Optional) Name of the resulting schema. + */ +export const defaultValue = ( + valueSchema: t.Type, + value: TValue, + name?: string +): t.Type => { + return new t.Type( + name ?? `DefaultValue<${valueSchema.name}>`, + valueSchema.is, + (input, context): Either => + input == null ? t.success(value) : valueSchema.validate(input, context), + t.identity + ); +}; diff --git a/packages/kbn-securitysolution-io-ts-types/src/index.ts b/packages/kbn-securitysolution-io-ts-types/src/index.ts index 01f9b32ca31af..8d701d6994322 100644 --- a/packages/kbn-securitysolution-io-ts-types/src/index.ts +++ b/packages/kbn-securitysolution-io-ts-types/src/index.ts @@ -9,10 +9,12 @@ export * from './default_array'; export * from './default_boolean_false'; export * from './default_boolean_true'; +export * from './default_csv_array'; export * from './default_empty_string'; export * from './default_string_array'; export * from './default_string_boolean_false'; export * from './default_uuid'; +export * from './default_value'; export * from './default_version_number'; export * from './empty_string_array'; export * from './enumeration'; diff --git a/x-pack/plugins/event_log/server/event_log_client.ts b/x-pack/plugins/event_log/server/event_log_client.ts index 9e4d579c54fc1..13aa2e3eadec0 100644 --- a/x-pack/plugins/event_log/server/event_log_client.ts +++ b/x-pack/plugins/event_log/server/event_log_client.ts @@ -32,6 +32,7 @@ const optionalDateFieldSchema = schema.maybe( const sortSchema = schema.object({ sort_field: schema.oneOf([ schema.literal('@timestamp'), + schema.literal('event.sequence'), // can be used as a tiebreaker for @timestamp schema.literal('event.start'), schema.literal('event.end'), schema.literal('event.provider'), diff --git a/x-pack/plugins/security_solution/common/constants.ts b/x-pack/plugins/security_solution/common/constants.ts index 9c4b9af0bf8f9..63f6b9723e369 100644 --- a/x-pack/plugins/security_solution/common/constants.ts +++ b/x-pack/plugins/security_solution/common/constants.ts @@ -231,6 +231,14 @@ export const IP_REPUTATION_LINKS_SETTING_DEFAULT = `[ export const SHOW_RELATED_INTEGRATIONS_SETTING = 'securitySolution:showRelatedIntegrations' as const; +/** This Kibana Advanced Setting enables extended rule execution logging to Event Log */ +export const EXTENDED_RULE_EXECUTION_LOGGING_ENABLED_SETTING = + 'securitySolution:extendedRuleExecutionLoggingEnabled' as const; + +/** This Kibana Advanced Setting sets minimum log level starting from which execution logs will be written to Event Log */ +export const EXTENDED_RULE_EXECUTION_LOGGING_MIN_LEVEL_SETTING = + 'securitySolution:extendedRuleExecutionLoggingMinLevel' as const; + /** * Id for the notifications alerting type * @deprecated Once we are confident all rules relying on side-car actions SO's have been migrated to SO references we should remove this function @@ -268,10 +276,6 @@ export const DETECTION_ENGINE_RULES_BULK_UPDATE = * Internal detection engine routes */ export const INTERNAL_DETECTION_ENGINE_URL = '/internal/detection_engine' as const; -export const DETECTION_ENGINE_RULE_EXECUTION_EVENTS_URL = - `${INTERNAL_DETECTION_ENGINE_URL}/rules/{ruleId}/execution/events` as const; -export const detectionEngineRuleExecutionEventsUrl = (ruleId: string) => - `${INTERNAL_DETECTION_ENGINE_URL}/rules/${ruleId}/execution/events` as const; export const DETECTION_ENGINE_INSTALLED_INTEGRATIONS_URL = `${INTERNAL_DETECTION_ENGINE_URL}/fleet/integrations/installed` as const; diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_events/request_schema.test.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_events/request_schema.test.ts new file mode 100644 index 0000000000000..c6e02b6f815e3 --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_events/request_schema.test.ts @@ -0,0 +1,168 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { pipe } from 'fp-ts/lib/pipeable'; +import { left } from 'fp-ts/lib/Either'; +import { foldLeftRight, getPaths } from '@kbn/securitysolution-io-ts-utils'; + +import { + GetRuleExecutionEventsRequestParams, + GetRuleExecutionEventsRequestQuery, +} from './request_schema'; + +describe('Request schema of Get rule execution events', () => { + describe('GetRuleExecutionEventsRequestParams', () => { + describe('Validation succeeds', () => { + it('when required parameters are passed', () => { + const input = { + ruleId: 'some id', + }; + + const decoded = GetRuleExecutionEventsRequestParams.decode(input); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual( + expect.objectContaining({ + ruleId: 'some id', + }) + ); + }); + + it('when unknown parameters are passed as well', () => { + const input = { + ruleId: 'some id', + foo: 'bar', // this one is not in the schema and will be stripped + }; + + const decoded = GetRuleExecutionEventsRequestParams.decode(input); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual({ + ruleId: 'some id', + }); + }); + }); + + describe('Validation fails', () => { + const test = (input: unknown) => { + const decoded = GetRuleExecutionEventsRequestParams.decode(input); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors)).length).toBeGreaterThan(0); + expect(message.schema).toEqual({}); + }; + + it('when not all the required parameters are passed', () => { + const input = {}; + test(input); + }); + + it('when ruleId is an empty string', () => { + const input: GetRuleExecutionEventsRequestParams = { + ruleId: '', + }; + + test(input); + }); + }); + }); + + describe('GetRuleExecutionEventsRequestQuery', () => { + describe('Validation succeeds', () => { + it('when valid parameters are passed', () => { + const input = { + event_types: 'message,status-change', + log_levels: 'debug,info,error', + sort_order: 'asc', + page: 42, + per_page: 6, + }; + + const decoded = GetRuleExecutionEventsRequestQuery.decode(input); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual({ + event_types: ['message', 'status-change'], + log_levels: ['debug', 'info', 'error'], + sort_order: 'asc', + page: 42, + per_page: 6, + }); + }); + + it('when unknown parameters are passed as well', () => { + const input = { + event_types: 'message,status-change', + log_levels: 'debug,info,error', + sort_order: 'asc', + page: 42, + per_page: 6, + foo: 'bar', // this one is not in the schema and will be stripped + }; + + const decoded = GetRuleExecutionEventsRequestQuery.decode(input); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual({ + event_types: ['message', 'status-change'], + log_levels: ['debug', 'info', 'error'], + sort_order: 'asc', + page: 42, + per_page: 6, + }); + }); + + it('when no parameters are passed (all are have default values)', () => { + const input = {}; + + const decoded = GetRuleExecutionEventsRequestQuery.decode(input); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual(expect.any(Object)); + }); + }); + + describe('Validation fails', () => { + const test = (input: unknown) => { + const decoded = GetRuleExecutionEventsRequestQuery.decode(input); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors)).length).toBeGreaterThan(0); + expect(message.schema).toEqual({}); + }; + + it('when invalid parameters are passed', () => { + test({ + event_types: 'foo,status-change', + }); + }); + }); + + describe('Validation sets default values', () => { + it('when optional parameters are not passed', () => { + const input = {}; + + const decoded = GetRuleExecutionEventsRequestQuery.decode(input); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual({ + event_types: [], + log_levels: [], + sort_order: 'desc', + page: 1, + per_page: 20, + }); + }); + }); + }); +}); diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_events/request_schema.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_events/request_schema.ts new file mode 100644 index 0000000000000..9ffa2467e0852 --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_events/request_schema.ts @@ -0,0 +1,43 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import * as t from 'io-ts'; + +import { DefaultPerPage, DefaultPage } from '@kbn/securitysolution-io-ts-alerting-types'; +import { defaultCsvArray, NonEmptyString } from '@kbn/securitysolution-io-ts-types'; + +import { DefaultSortOrderDesc } from '../../../schemas/common'; +import { TRuleExecutionEventType } from '../../model/execution_event'; +import { TLogLevel } from '../../model/log_level'; + +/** + * Path parameters of the API route. + */ +export type GetRuleExecutionEventsRequestParams = t.TypeOf< + typeof GetRuleExecutionEventsRequestParams +>; +export const GetRuleExecutionEventsRequestParams = t.exact( + t.type({ + ruleId: NonEmptyString, + }) +); + +/** + * Query string parameters of the API route. + */ +export type GetRuleExecutionEventsRequestQuery = t.TypeOf< + typeof GetRuleExecutionEventsRequestQuery +>; +export const GetRuleExecutionEventsRequestQuery = t.exact( + t.type({ + event_types: defaultCsvArray(TRuleExecutionEventType), + log_levels: defaultCsvArray(TLogLevel), + sort_order: DefaultSortOrderDesc, // defaults to 'desc' + page: DefaultPage, // defaults to 1 + per_page: DefaultPerPage, // defaults to 20 + }) +); diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_events/response_schema.mock.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_events/response_schema.mock.ts new file mode 100644 index 0000000000000..c4c501f2aeac9 --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_events/response_schema.mock.ts @@ -0,0 +1,25 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { ruleExecutionEventMock } from '../../model/execution_event.mock'; +import type { GetRuleExecutionEventsResponse } from './response_schema'; + +const getSomeResponse = (): GetRuleExecutionEventsResponse => { + const events = ruleExecutionEventMock.getSomeEvents(); + return { + events, + pagination: { + page: 1, + per_page: events.length, + total: events.length * 10, + }, + }; +}; + +export const getRuleExecutionEventsResponseMock = { + getSomeResponse, +}; diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_events/response_schema.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_events/response_schema.ts new file mode 100644 index 0000000000000..8637b3b0411a2 --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_events/response_schema.ts @@ -0,0 +1,21 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import * as t from 'io-ts'; +import { PaginationResult } from '../../../schemas/common'; +import { RuleExecutionEvent } from '../../model/execution_event'; + +/** + * Response body of the API route. + */ +export type GetRuleExecutionEventsResponse = t.TypeOf; +export const GetRuleExecutionEventsResponse = t.exact( + t.type({ + events: t.array(RuleExecutionEvent), + pagination: PaginationResult, + }) +); diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_results/request_schema.test.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_results/request_schema.test.ts new file mode 100644 index 0000000000000..8757084a2ec98 --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_results/request_schema.test.ts @@ -0,0 +1,271 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { pipe } from 'fp-ts/lib/pipeable'; +import { left } from 'fp-ts/lib/Either'; +import { foldLeftRight, getPaths } from '@kbn/securitysolution-io-ts-utils'; + +import { RULE_EXECUTION_STATUSES } from '../../model/execution_status'; +import { DefaultSortField, DefaultRuleExecutionStatusCsvArray } from './request_schema'; + +describe('Request schema of Get rule execution results', () => { + describe('DefaultRuleExecutionStatusCsvArray', () => { + describe('Validation succeeds', () => { + describe('when input is a single rule execution status', () => { + const cases = RULE_EXECUTION_STATUSES.map((supportedStatus) => { + return { input: supportedStatus }; + }); + + cases.forEach(({ input }) => { + it(`${input}`, () => { + const decoded = DefaultRuleExecutionStatusCsvArray.decode(input); + const message = pipe(decoded, foldLeftRight); + const expectedOutput = [input]; // note that it's an array after decode + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual(expectedOutput); + }); + }); + }); + + describe('when input is an array of rule execution statuses', () => { + const cases = [ + { input: ['succeeded', 'failed'] }, + { input: ['partial failure', 'going to run', 'running'] }, + ]; + + cases.forEach(({ input }) => { + it(`${input}`, () => { + const decoded = DefaultRuleExecutionStatusCsvArray.decode(input); + const message = pipe(decoded, foldLeftRight); + const expectedOutput = input; + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual(expectedOutput); + }); + }); + }); + + describe('when input is a string which is a comma-separated array of statuses', () => { + const cases = [ + { + input: 'succeeded,failed', + expectedOutput: ['succeeded', 'failed'], + }, + { + input: 'partial failure,going to run,running', + expectedOutput: ['partial failure', 'going to run', 'running'], + }, + ]; + + cases.forEach(({ input, expectedOutput }) => { + it(`${input}`, () => { + const decoded = DefaultRuleExecutionStatusCsvArray.decode(input); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual(expectedOutput); + }); + }); + }); + }); + + describe('Validation fails', () => { + describe('when input is a single invalid value', () => { + const cases = [ + { + input: 'val', + expectedErrors: [ + 'Invalid value "val" supplied to "DefaultCsvArray"', + ], + }, + { + input: '5', + expectedErrors: [ + 'Invalid value "5" supplied to "DefaultCsvArray"', + ], + }, + { + input: 5, + expectedErrors: [ + 'Invalid value "5" supplied to "DefaultCsvArray"', + ], + }, + { + input: {}, + expectedErrors: [ + 'Invalid value "{}" supplied to "DefaultCsvArray"', + ], + }, + ]; + + cases.forEach(({ input, expectedErrors }) => { + it(`${input}`, () => { + const decoded = DefaultRuleExecutionStatusCsvArray.decode(input); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual(expectedErrors); + expect(message.schema).toEqual({}); + }); + }); + }); + + describe('when input is an array of invalid values', () => { + const cases = [ + { + input: ['value 1', 5], + expectedErrors: [ + 'Invalid value "value 1" supplied to "DefaultCsvArray"', + 'Invalid value "5" supplied to "DefaultCsvArray"', + ], + }, + { + input: ['value 1', 'succeeded'], + expectedErrors: [ + 'Invalid value "value 1" supplied to "DefaultCsvArray"', + ], + }, + { + input: ['', 5, {}], + expectedErrors: [ + 'Invalid value "" supplied to "DefaultCsvArray"', + 'Invalid value "5" supplied to "DefaultCsvArray"', + 'Invalid value "{}" supplied to "DefaultCsvArray"', + ], + }, + ]; + + cases.forEach(({ input, expectedErrors }) => { + it(`${input}`, () => { + const decoded = DefaultRuleExecutionStatusCsvArray.decode(input); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual(expectedErrors); + expect(message.schema).toEqual({}); + }); + }); + }); + + describe('when input is a string which is a comma-separated array of invalid values', () => { + const cases = [ + { + input: 'value 1,5', + expectedErrors: [ + 'Invalid value "value 1" supplied to "DefaultCsvArray"', + 'Invalid value "5" supplied to "DefaultCsvArray"', + ], + }, + { + input: 'value 1,succeeded', + expectedErrors: [ + 'Invalid value "value 1" supplied to "DefaultCsvArray"', + ], + }, + { + input: ',5,{}', + expectedErrors: [ + 'Invalid value "" supplied to "DefaultCsvArray"', + 'Invalid value "5" supplied to "DefaultCsvArray"', + 'Invalid value "{}" supplied to "DefaultCsvArray"', + ], + }, + ]; + + cases.forEach(({ input, expectedErrors }) => { + it(`${input}`, () => { + const decoded = DefaultRuleExecutionStatusCsvArray.decode(input); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual(expectedErrors); + expect(message.schema).toEqual({}); + }); + }); + }); + }); + + describe('Validation returns default value (an empty array)', () => { + describe('when input is', () => { + const cases = [{ input: null }, { input: undefined }, { input: '' }, { input: [] }]; + + cases.forEach(({ input }) => { + it(`${input}`, () => { + const decoded = DefaultRuleExecutionStatusCsvArray.decode(input); + const message = pipe(decoded, foldLeftRight); + const expectedOutput: string[] = []; + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual(expectedOutput); + }); + }); + }); + }); + }); + + describe('DefaultSortField', () => { + describe('Validation succeeds', () => { + describe('when input is a valid sort field', () => { + const cases = [ + { input: 'timestamp' }, + { input: 'duration_ms' }, + { input: 'gap_duration_s' }, + { input: 'indexing_duration_ms' }, + { input: 'search_duration_ms' }, + { input: 'schedule_delay_ms' }, + ]; + + cases.forEach(({ input }) => { + it(`${input}`, () => { + const decoded = DefaultSortField.decode(input); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual(input); + }); + }); + }); + }); + + describe('Validation fails', () => { + describe('when input is an invalid sort field', () => { + const cases = [ + { input: 'status' }, + { input: 'message' }, + { input: 'es_search_duration_ms' }, + { input: 'security_status' }, + { input: 'security_message' }, + ]; + + cases.forEach(({ input }) => { + it(`${input}`, () => { + const decoded = DefaultSortField.decode(input); + const message = pipe(decoded, foldLeftRight); + const expectedErrors = [`Invalid value "${input}" supplied to "DefaultSortField"`]; + + expect(getPaths(left(message.errors))).toEqual(expectedErrors); + expect(message.schema).toEqual({}); + }); + }); + }); + }); + + describe('Validation returns the default sort field "timestamp"', () => { + describe('when input is', () => { + const cases = [{ input: null }, { input: undefined }]; + + cases.forEach(({ input }) => { + it(`${input}`, () => { + const decoded = DefaultSortField.decode(input); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual('timestamp'); + }); + }); + }); + }); + }); +}); diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_results/request_schema.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_results/request_schema.ts new file mode 100644 index 0000000000000..33458ab0a875a --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_results/request_schema.ts @@ -0,0 +1,72 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import * as t from 'io-ts'; + +import { DefaultPage, DefaultPerPage } from '@kbn/securitysolution-io-ts-alerting-types'; +import { + defaultCsvArray, + DefaultEmptyString, + defaultValue, + IsoDateString, + NonEmptyString, +} from '@kbn/securitysolution-io-ts-types'; + +import { DefaultSortOrderDesc } from '../../../schemas/common'; +import { SortFieldOfRuleExecutionResult } from '../../model/execution_result'; +import { TRuleExecutionStatus } from '../../model/execution_status'; + +/** + * Types the DefaultRuleExecutionStatusCsvArray as: + * - If not specified, then a default empty array will be set + * - If an array is sent in, then the array will be validated to ensure all elements are a RuleExecutionStatus + * (or that the array is empty) + * - If a CSV string is sent in, then it will be parsed to an array which will be validated + */ +export const DefaultRuleExecutionStatusCsvArray = defaultCsvArray(TRuleExecutionStatus); + +/** + * Types the DefaultSortField as: + * - If undefined, then a default sort field of 'timestamp' will be set + * - If a string is sent in, then the string will be validated to ensure it is as valid sortFields + */ +export const DefaultSortField = defaultValue( + SortFieldOfRuleExecutionResult, + 'timestamp', + 'DefaultSortField' +); + +/** + * Path parameters of the API route. + */ +export type GetRuleExecutionResultsRequestParams = t.TypeOf< + typeof GetRuleExecutionResultsRequestParams +>; +export const GetRuleExecutionResultsRequestParams = t.exact( + t.type({ + ruleId: NonEmptyString, + }) +); + +/** + * Query string parameters of the API route. + */ +export type GetRuleExecutionResultsRequestQuery = t.TypeOf< + typeof GetRuleExecutionResultsRequestQuery +>; +export const GetRuleExecutionResultsRequestQuery = t.exact( + t.type({ + start: IsoDateString, + end: IsoDateString, + query_text: DefaultEmptyString, // defaults to '' + status_filters: DefaultRuleExecutionStatusCsvArray, // defaults to [] + sort_field: DefaultSortField, // defaults to 'timestamp' + sort_order: DefaultSortOrderDesc, // defaults to 'desc' + page: DefaultPage, // defaults to 1 + per_page: DefaultPerPage, // defaults to 20 + }) +); diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_results/response_schema.mock.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_results/response_schema.mock.ts new file mode 100644 index 0000000000000..fb6be3eeb62cc --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_results/response_schema.mock.ts @@ -0,0 +1,21 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { ruleExecutionResultMock } from '../../model/execution_result.mock'; +import type { GetRuleExecutionResultsResponse } from './response_schema'; + +const getSomeResponse = (): GetRuleExecutionResultsResponse => { + const results = ruleExecutionResultMock.getSomeResults(); + return { + events: results, + total: results.length, + }; +}; + +export const getRuleExecutionResultsResponseMock = { + getSomeResponse, +}; diff --git a/x-pack/plugins/security_solution/common/detection_engine/schemas/response/get_rule_execution_events_response.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_results/response_schema.ts similarity index 51% rename from x-pack/plugins/security_solution/common/detection_engine/schemas/response/get_rule_execution_events_response.ts rename to x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_results/response_schema.ts index 10be3a03814a3..7610a21d18181 100644 --- a/x-pack/plugins/security_solution/common/detection_engine/schemas/response/get_rule_execution_events_response.ts +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/get_rule_execution_results/response_schema.ts @@ -6,15 +6,15 @@ */ import * as t from 'io-ts'; -import { aggregateRuleExecutionEvent } from '../common'; +import { RuleExecutionResult } from '../../model/execution_result'; -export const GetAggregateRuleExecutionEventsResponse = t.exact( +/** + * Response body of the API route. + */ +export type GetRuleExecutionResultsResponse = t.TypeOf; +export const GetRuleExecutionResultsResponse = t.exact( t.type({ - events: t.array(aggregateRuleExecutionEvent), + events: t.array(RuleExecutionResult), total: t.number, }) ); - -export type GetAggregateRuleExecutionEventsResponse = t.TypeOf< - typeof GetAggregateRuleExecutionEventsResponse ->; diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/urls.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/urls.ts new file mode 100644 index 0000000000000..595ca6c01d83a --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/api/urls.ts @@ -0,0 +1,18 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { INTERNAL_DETECTION_ENGINE_URL as INTERNAL_URL } from '../../../constants'; + +export const GET_RULE_EXECUTION_EVENTS_URL = + `${INTERNAL_URL}/rules/{ruleId}/execution/events` as const; +export const getRuleExecutionEventsUrl = (ruleId: string) => + `${INTERNAL_URL}/rules/${ruleId}/execution/events` as const; + +export const GET_RULE_EXECUTION_RESULTS_URL = + `${INTERNAL_URL}/rules/{ruleId}/execution/results` as const; +export const getRuleExecutionResultsUrl = (ruleId: string) => + `${INTERNAL_URL}/rules/${ruleId}/execution/results` as const; diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/index.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/index.ts new file mode 100644 index 0000000000000..b7881e2d2f524 --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/index.ts @@ -0,0 +1,20 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +export * from './api/get_rule_execution_events/request_schema'; +export * from './api/get_rule_execution_events/response_schema'; +export * from './api/get_rule_execution_results/request_schema'; +export * from './api/get_rule_execution_results/response_schema'; +export * from './api/urls'; + +export * from './model/execution_event'; +export * from './model/execution_metrics'; +export * from './model/execution_result'; +export * from './model/execution_settings'; +export * from './model/execution_status'; +export * from './model/execution_summary'; +export * from './model/log_level'; diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/mocks.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/mocks.ts new file mode 100644 index 0000000000000..a552a0fc047f6 --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/mocks.ts @@ -0,0 +1,13 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +export * from './api/get_rule_execution_events/response_schema.mock'; +export * from './api/get_rule_execution_results/response_schema.mock'; + +export * from './model/execution_event.mock'; +export * from './model/execution_result.mock'; +export * from './model/execution_summary.mock'; diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_event.mock.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_event.mock.ts new file mode 100644 index 0000000000000..0cb49804b13a6 --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_event.mock.ts @@ -0,0 +1,152 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type { RuleExecutionEvent } from './execution_event'; +import { RuleExecutionEventType } from './execution_event'; +import { LogLevel } from './log_level'; + +const DEFAULT_TIMESTAMP = '2021-12-28T10:10:00.806Z'; +const DEFAULT_SEQUENCE_NUMBER = 0; + +const getMessageEvent = (props: Partial = {}): RuleExecutionEvent => { + return { + // Default values + timestamp: DEFAULT_TIMESTAMP, + sequence: DEFAULT_SEQUENCE_NUMBER, + level: LogLevel.debug, + message: 'Some message', + // Overriden values + ...props, + // Mandatory values for this type of event + type: RuleExecutionEventType.message, + }; +}; + +const getRunningStatusChange = (props: Partial = {}): RuleExecutionEvent => { + return { + // Default values + timestamp: DEFAULT_TIMESTAMP, + sequence: DEFAULT_SEQUENCE_NUMBER, + message: 'Rule changed status to "running"', + // Overriden values + ...props, + // Mandatory values for this type of event + level: LogLevel.info, + type: RuleExecutionEventType['status-change'], + }; +}; + +const getPartialFailureStatusChange = ( + props: Partial = {} +): RuleExecutionEvent => { + return { + // Default values + timestamp: DEFAULT_TIMESTAMP, + sequence: DEFAULT_SEQUENCE_NUMBER, + message: 'Rule changed status to "partial failure". Unknown error', + // Overriden values + ...props, + // Mandatory values for this type of event + level: LogLevel.warn, + type: RuleExecutionEventType['status-change'], + }; +}; + +const getFailedStatusChange = (props: Partial = {}): RuleExecutionEvent => { + return { + // Default values + timestamp: DEFAULT_TIMESTAMP, + sequence: DEFAULT_SEQUENCE_NUMBER, + message: 'Rule changed status to "failed". Unknown error', + // Overriden values + ...props, + // Mandatory values for this type of event + level: LogLevel.error, + type: RuleExecutionEventType['status-change'], + }; +}; + +const getSucceededStatusChange = (props: Partial = {}): RuleExecutionEvent => { + return { + // Default values + timestamp: DEFAULT_TIMESTAMP, + sequence: DEFAULT_SEQUENCE_NUMBER, + message: 'Rule changed status to "succeeded". Rule executed successfully', + // Overriden values + ...props, + // Mandatory values for this type of event + level: LogLevel.info, + type: RuleExecutionEventType['status-change'], + }; +}; + +const getExecutionMetricsEvent = (props: Partial = {}): RuleExecutionEvent => { + return { + // Default values + timestamp: DEFAULT_TIMESTAMP, + sequence: DEFAULT_SEQUENCE_NUMBER, + message: '', + // Overriden values + ...props, + // Mandatory values for this type of event + level: LogLevel.debug, + type: RuleExecutionEventType['execution-metrics'], + }; +}; + +const getSomeEvents = (): RuleExecutionEvent[] => [ + getSucceededStatusChange({ + timestamp: '2021-12-28T10:10:09.806Z', + sequence: 9, + }), + getExecutionMetricsEvent({ + timestamp: '2021-12-28T10:10:08.806Z', + sequence: 8, + }), + getRunningStatusChange({ + timestamp: '2021-12-28T10:10:07.806Z', + sequence: 7, + }), + getMessageEvent({ + timestamp: '2021-12-28T10:10:06.806Z', + sequence: 6, + level: LogLevel.debug, + message: 'Rule execution started', + }), + getFailedStatusChange({ + timestamp: '2021-12-28T10:10:05.806Z', + sequence: 5, + }), + getExecutionMetricsEvent({ + timestamp: '2021-12-28T10:10:04.806Z', + sequence: 4, + }), + getPartialFailureStatusChange({ + timestamp: '2021-12-28T10:10:03.806Z', + sequence: 3, + }), + getMessageEvent({ + timestamp: '2021-12-28T10:10:02.806Z', + sequence: 2, + level: LogLevel.error, + message: 'Some error', + }), + getRunningStatusChange({ + timestamp: '2021-12-28T10:10:01.806Z', + sequence: 1, + }), + getMessageEvent({ + timestamp: '2021-12-28T10:10:00.806Z', + sequence: 0, + level: LogLevel.debug, + message: 'Rule execution started', + }), +]; + +export const ruleExecutionEventMock = { + getSomeEvents, +}; diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_event.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_event.ts new file mode 100644 index 0000000000000..0e269b39ff8f1 --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_event.ts @@ -0,0 +1,60 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import * as t from 'io-ts'; +import { enumeration, IsoDateString } from '@kbn/securitysolution-io-ts-types'; +import { enumFromString } from '../../../utils/enum_from_string'; +import { TLogLevel } from './log_level'; + +/** + * Type of a plain rule execution event. + */ +export enum RuleExecutionEventType { + /** + * Simple log message of some log level, such as debug, info or error. + */ + 'message' = 'message', + + /** + * We log an event of this type each time a rule changes its status during an execution. + */ + 'status-change' = 'status-change', + + /** + * We log an event of this type at the end of a rule execution. It contains various execution + * metrics such as search and indexing durations. + */ + 'execution-metrics' = 'execution-metrics', +} + +export const TRuleExecutionEventType = enumeration( + 'RuleExecutionEventType', + RuleExecutionEventType +); + +/** + * An array of supported types of rule execution events. + */ +export const RULE_EXECUTION_EVENT_TYPES = Object.values(RuleExecutionEventType); + +export const ruleExecutionEventTypeFromString = enumFromString(RuleExecutionEventType); + +/** + * Plain rule execution event. A rule can write many of them during each execution. Events can be + * of different types and log levels. + * + * NOTE: This is a read model of rule execution events and it is pretty generic. It contains only a + * subset of their fields: only those fields that are common to all types of execution events. + */ +export type RuleExecutionEvent = t.TypeOf; +export const RuleExecutionEvent = t.type({ + timestamp: IsoDateString, + sequence: t.number, + level: TLogLevel, + type: TRuleExecutionEventType, + message: t.string, +}); diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_metrics.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_metrics.ts new file mode 100644 index 0000000000000..c6bb71970e599 --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_metrics.ts @@ -0,0 +1,19 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import * as t from 'io-ts'; +import { PositiveInteger } from '@kbn/securitysolution-io-ts-types'; + +export type DurationMetric = t.TypeOf; +export const DurationMetric = PositiveInteger; + +export type RuleExecutionMetrics = t.TypeOf; +export const RuleExecutionMetrics = t.partial({ + total_search_duration_ms: DurationMetric, + total_indexing_duration_ms: DurationMetric, + execution_gap_duration_s: DurationMetric, +}); diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_result.mock.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_result.mock.ts new file mode 100644 index 0000000000000..4a039ca949c82 --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_result.mock.ts @@ -0,0 +1,176 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type { RuleExecutionResult } from './execution_result'; + +const getSomeResults = (): RuleExecutionResult[] => [ + { + execution_uuid: 'dc45a63c-4872-4964-a2d0-bddd8b2e634d', + timestamp: '2022-04-28T21:19:08.047Z', + duration_ms: 3, + status: 'failure', + message: 'siem.queryRule:a6e61cf0-c737-11ec-9e32-e14913ffdd2d: execution failed', + num_active_alerts: 0, + num_new_alerts: 0, + num_recovered_alerts: 0, + num_triggered_actions: 0, + num_succeeded_actions: 0, + num_errored_actions: 0, + total_search_duration_ms: 0, + es_search_duration_ms: 0, + schedule_delay_ms: 2169, + timed_out: false, + indexing_duration_ms: 0, + search_duration_ms: 0, + gap_duration_s: 0, + security_status: 'failed', + security_message: 'Rule failed to execute because rule ran after it was disabled.', + }, + { + execution_uuid: '0fde9271-05d0-4bfb-8ff8-815756d28350', + timestamp: '2022-04-28T21:19:04.973Z', + duration_ms: 1446, + status: 'success', + message: + "rule executed: siem.queryRule:a6e61cf0-c737-11ec-9e32-e14913ffdd2d: 'Click here for hot fresh alerts!'", + num_active_alerts: 0, + num_new_alerts: 0, + num_recovered_alerts: 0, + num_triggered_actions: 0, + num_succeeded_actions: 0, + num_errored_actions: 0, + total_search_duration_ms: 0, + es_search_duration_ms: 0, + schedule_delay_ms: 2089, + timed_out: false, + indexing_duration_ms: 0, + search_duration_ms: 2, + gap_duration_s: 0, + security_status: 'succeeded', + security_message: 'succeeded', + }, + { + execution_uuid: '5daaa259-ded8-4a52-853e-1e7652d325d5', + timestamp: '2022-04-28T21:19:01.976Z', + duration_ms: 1395, + status: 'success', + message: + "rule executed: siem.queryRule:a6e61cf0-c737-11ec-9e32-e14913ffdd2d: 'Click here for hot fresh alerts!'", + num_active_alerts: 0, + num_new_alerts: 0, + num_recovered_alerts: 0, + num_triggered_actions: 0, + num_succeeded_actions: 0, + num_errored_actions: 0, + total_search_duration_ms: 0, + es_search_duration_ms: 1, + schedule_delay_ms: 2637, + timed_out: false, + indexing_duration_ms: 0, + search_duration_ms: 3, + gap_duration_s: 0, + security_status: 'succeeded', + security_message: 'succeeded', + }, + { + execution_uuid: 'c7223e1c-4264-4a27-8697-0d720243fafc', + timestamp: '2022-04-28T21:18:58.431Z', + duration_ms: 1815, + status: 'success', + message: + "rule executed: siem.queryRule:a6e61cf0-c737-11ec-9e32-e14913ffdd2d: 'Click here for hot fresh alerts!'", + num_active_alerts: 0, + num_new_alerts: 0, + num_recovered_alerts: 0, + num_triggered_actions: 0, + num_succeeded_actions: 0, + num_errored_actions: 0, + total_search_duration_ms: 0, + es_search_duration_ms: 1, + schedule_delay_ms: -255429, + timed_out: false, + indexing_duration_ms: 0, + search_duration_ms: 3, + gap_duration_s: 0, + security_status: 'succeeded', + security_message: 'succeeded', + }, + { + execution_uuid: '1f6ba0c1-cc36-4f45-b919-7790b8a8d670', + timestamp: '2022-04-28T21:18:13.954Z', + duration_ms: 2055, + status: 'success', + message: + "rule executed: siem.queryRule:a6e61cf0-c737-11ec-9e32-e14913ffdd2d: 'Click here for hot fresh alerts!'", + num_active_alerts: 0, + num_new_alerts: 0, + num_recovered_alerts: 0, + num_triggered_actions: 0, + num_succeeded_actions: 0, + num_errored_actions: 0, + total_search_duration_ms: 0, + es_search_duration_ms: 0, + schedule_delay_ms: 2027, + timed_out: false, + indexing_duration_ms: 0, + search_duration_ms: 0, + gap_duration_s: 0, + security_status: 'partial failure', + security_message: + 'Check privileges failed to execute ResponseError: index_not_found_exception: [index_not_found_exception] Reason: no such index [yup] name: "Click here for hot fresh alerts!" id: "a6e61cf0-c737-11ec-9e32-e14913ffdd2d" rule id: "34946b12-88d1-49ef-82b7-9cad45972030" execution id: "1f6ba0c1-cc36-4f45-b919-7790b8a8d670" space ID: "default"', + }, + { + execution_uuid: 'b0f65d64-b229-432b-9d39-f4385a7f9368', + timestamp: '2022-04-28T21:15:43.086Z', + duration_ms: 1205, + status: 'success', + message: + "rule executed: siem.queryRule:a6e61cf0-c737-11ec-9e32-e14913ffdd2d: 'Click here for hot fresh alerts!'", + num_active_alerts: 0, + num_new_alerts: 0, + num_recovered_alerts: 0, + num_triggered_actions: 0, + num_succeeded_actions: 0, + num_errored_actions: 0, + total_search_duration_ms: 0, + es_search_duration_ms: 672, + schedule_delay_ms: 3086, + timed_out: false, + indexing_duration_ms: 140, + search_duration_ms: 684, + gap_duration_s: 0, + security_status: 'succeeded', + security_message: 'succeeded', + }, + { + execution_uuid: '7bfd25b9-c0d8-44b1-982c-485169466a8e', + timestamp: '2022-04-28T21:10:40.135Z', + duration_ms: 6321, + status: 'success', + message: + "rule executed: siem.queryRule:a6e61cf0-c737-11ec-9e32-e14913ffdd2d: 'Click here for hot fresh alerts!'", + num_active_alerts: 0, + num_new_alerts: 0, + num_recovered_alerts: 0, + num_triggered_actions: 0, + num_succeeded_actions: 0, + num_errored_actions: 0, + total_search_duration_ms: 0, + es_search_duration_ms: 930, + schedule_delay_ms: 1222, + timed_out: false, + indexing_duration_ms: 2103, + search_duration_ms: 946, + gap_duration_s: 0, + security_status: 'succeeded', + security_message: 'succeeded', + }, +]; + +export const ruleExecutionResultMock = { + getSomeResults, +}; diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_result.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_result.ts new file mode 100644 index 0000000000000..3a15121624b75 --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_result.ts @@ -0,0 +1,50 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import * as t from 'io-ts'; +import { IsoDateString } from '@kbn/securitysolution-io-ts-types'; + +/** + * Rule execution result is an aggregate that groups plain rule execution events by execution UUID. + * It contains such information as execution UUID, date, status and metrics. + */ +export type RuleExecutionResult = t.TypeOf; +export const RuleExecutionResult = t.type({ + execution_uuid: t.string, + timestamp: IsoDateString, + duration_ms: t.number, + status: t.string, + message: t.string, + num_active_alerts: t.number, + num_new_alerts: t.number, + num_recovered_alerts: t.number, + num_triggered_actions: t.number, + num_succeeded_actions: t.number, + num_errored_actions: t.number, + total_search_duration_ms: t.number, + es_search_duration_ms: t.number, + schedule_delay_ms: t.number, + timed_out: t.boolean, + indexing_duration_ms: t.number, + search_duration_ms: t.number, + gap_duration_s: t.number, + security_status: t.string, + security_message: t.string, +}); + +/** + * We support sorting rule execution results by these fields. + */ +export type SortFieldOfRuleExecutionResult = t.TypeOf; +export const SortFieldOfRuleExecutionResult = t.keyof({ + timestamp: IsoDateString, + duration_ms: t.number, + gap_duration_s: t.number, + indexing_duration_ms: t.number, + search_duration_ms: t.number, + schedule_delay_ms: t.number, +}); diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_settings.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_settings.ts new file mode 100644 index 0000000000000..e1ac664f0b537 --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_settings.ts @@ -0,0 +1,22 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +export interface RuleExecutionSettings { + extendedLogging: { + isEnabled: boolean; + minLevel: LogLevelSetting; + }; +} + +export enum LogLevelSetting { + 'trace' = 'trace', + 'debug' = 'debug', + 'info' = 'info', + 'warn' = 'warn', + 'error' = 'error', + 'off' = 'off', +} diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_status.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_status.ts new file mode 100644 index 0000000000000..22e599b18c541 --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_status.ts @@ -0,0 +1,80 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type * as t from 'io-ts'; +import { enumeration, PositiveInteger } from '@kbn/securitysolution-io-ts-types'; +import { assertUnreachable } from '../../../utility_types'; + +/** + * Custom execution status of Security rules that is different from the status + * used in the Alerting Framework. We merge our custom status with the + * Framework's status to determine the resulting status of a rule. + */ +export enum RuleExecutionStatus { + /** + * @deprecated Replaced by the 'running' status but left for backwards compatibility + * with rule execution events already written to Event Log in the prior versions of Kibana. + * Don't use when writing rule status changes. + */ + 'going to run' = 'going to run', + + /** + * Rule execution started but not reached any intermediate or final status. + */ + 'running' = 'running', + + /** + * Rule can partially fail for various reasons either in the middle of an execution + * (in this case we update its status right away) or in the end of it. So currently + * this status can be both intermediate and final at the same time. + * A typical reason for a partial failure: not all the indices that the rule searches + * over actually exist. + */ + 'partial failure' = 'partial failure', + + /** + * Rule failed to execute due to unhandled exception or a reason defined in the + * business logic of its executor function. + */ + 'failed' = 'failed', + + /** + * Rule executed successfully without any issues. Note: this status is just an indication + * of a rule's "health". The rule might or might not generate any alerts despite of it. + */ + 'succeeded' = 'succeeded', +} + +export const TRuleExecutionStatus = enumeration('RuleExecutionStatus', RuleExecutionStatus); + +/** + * An array of supported rule execution statuses. + */ +export const RULE_EXECUTION_STATUSES = Object.values(RuleExecutionStatus); + +export type RuleExecutionStatusOrder = t.TypeOf; +export const RuleExecutionStatusOrder = PositiveInteger; + +export const ruleExecutionStatusToNumber = ( + status: RuleExecutionStatus +): RuleExecutionStatusOrder => { + switch (status) { + case RuleExecutionStatus.succeeded: + return 0; + case RuleExecutionStatus['going to run']: + return 10; + case RuleExecutionStatus.running: + return 15; + case RuleExecutionStatus['partial failure']: + return 20; + case RuleExecutionStatus.failed: + return 30; + default: + assertUnreachable(status); + return 0; + } +}; diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_summary.mock.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_summary.mock.ts new file mode 100644 index 0000000000000..3224b84ba2cef --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_summary.mock.ts @@ -0,0 +1,43 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { RuleExecutionStatus } from './execution_status'; +import type { RuleExecutionSummary } from './execution_summary'; + +const getSummarySucceeded = (): RuleExecutionSummary => ({ + last_execution: { + date: '2020-02-18T15:26:49.783Z', + status: RuleExecutionStatus.succeeded, + status_order: 0, + message: 'succeeded', + metrics: { + total_search_duration_ms: 200, + total_indexing_duration_ms: 800, + execution_gap_duration_s: 500, + }, + }, +}); + +const getSummaryFailed = (): RuleExecutionSummary => ({ + last_execution: { + date: '2020-02-18T15:15:58.806Z', + status: RuleExecutionStatus.failed, + status_order: 30, + message: + 'Signal rule name: "Query with a rule id Number 1", id: "1ea5a820-4da1-4e82-92a1-2b43a7bece08", rule_id: "query-rule-id-1" has a time gap of 5 days (412682928ms), and could be missing signals within that time. Consider increasing your look behind time or adding more Kibana instances.', + metrics: { + total_search_duration_ms: 200, + total_indexing_duration_ms: 800, + execution_gap_duration_s: 500, + }, + }, +}); + +export const ruleExecutionSummaryMock = { + getSummarySucceeded, + getSummaryFailed, +}; diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_summary.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_summary.ts new file mode 100644 index 0000000000000..bbe9e9ad5d7e7 --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/execution_summary.ts @@ -0,0 +1,22 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import * as t from 'io-ts'; +import { IsoDateString } from '@kbn/securitysolution-io-ts-types'; +import { TRuleExecutionStatus, RuleExecutionStatusOrder } from './execution_status'; +import { RuleExecutionMetrics } from './execution_metrics'; + +export type RuleExecutionSummary = t.TypeOf; +export const RuleExecutionSummary = t.type({ + last_execution: t.type({ + date: IsoDateString, + status: TRuleExecutionStatus, + status_order: RuleExecutionStatusOrder, + message: t.string, + metrics: RuleExecutionMetrics, + }), +}); diff --git a/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/log_level.ts b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/log_level.ts new file mode 100644 index 0000000000000..b37ce62ad4891 --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring/model/log_level.ts @@ -0,0 +1,82 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { enumeration } from '@kbn/securitysolution-io-ts-types'; +import { enumFromString } from '../../../utils/enum_from_string'; +import { assertUnreachable } from '../../../utility_types'; +import { RuleExecutionStatus } from './execution_status'; + +export enum LogLevel { + 'trace' = 'trace', + 'debug' = 'debug', + 'info' = 'info', + 'warn' = 'warn', + 'error' = 'error', +} + +export const TLogLevel = enumeration('LogLevel', LogLevel); + +/** + * An array of supported log levels. + */ +export const LOG_LEVELS = Object.values(LogLevel); + +export const logLevelToNumber = (level: keyof typeof LogLevel | null | undefined): number => { + if (!level) { + return 0; + } + + switch (level) { + case 'trace': + return 0; + case 'debug': + return 10; + case 'info': + return 20; + case 'warn': + return 30; + case 'error': + return 40; + default: + assertUnreachable(level); + return 0; + } +}; + +export const logLevelFromNumber = (num: number | null | undefined): LogLevel => { + if (num === null || num === undefined || num < 10) { + return LogLevel.trace; + } + if (num < 20) { + return LogLevel.debug; + } + if (num < 30) { + return LogLevel.info; + } + if (num < 40) { + return LogLevel.warn; + } + return LogLevel.error; +}; + +export const logLevelFromString = enumFromString(LogLevel); + +export const logLevelFromExecutionStatus = (status: RuleExecutionStatus): LogLevel => { + switch (status) { + case RuleExecutionStatus['going to run']: + case RuleExecutionStatus.running: + case RuleExecutionStatus.succeeded: + return LogLevel.info; + case RuleExecutionStatus['partial failure']: + return LogLevel.warn; + case RuleExecutionStatus.failed: + return LogLevel.error; + default: + assertUnreachable(status); + return LogLevel.trace; + } +}; diff --git a/x-pack/plugins/security_solution/common/detection_engine/schemas/common/index.ts b/x-pack/plugins/security_solution/common/detection_engine/schemas/common/index.ts index b7f96bfdafdad..ad8745a8caf21 100644 --- a/x-pack/plugins/security_solution/common/detection_engine/schemas/common/index.ts +++ b/x-pack/plugins/security_solution/common/detection_engine/schemas/common/index.ts @@ -6,6 +6,7 @@ */ export * from './installed_integrations'; -export * from './rule_monitoring'; +export * from './pagination'; export * from './rule_params'; export * from './schemas'; +export * from './sorting'; diff --git a/x-pack/plugins/security_solution/common/detection_engine/schemas/common/pagination.ts b/x-pack/plugins/security_solution/common/detection_engine/schemas/common/pagination.ts new file mode 100644 index 0000000000000..bed2cade86df4 --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/schemas/common/pagination.ts @@ -0,0 +1,28 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import * as t from 'io-ts'; +import { PositiveInteger, PositiveIntegerGreaterThanZero } from '@kbn/securitysolution-io-ts-types'; + +export type Page = t.TypeOf; +export const Page = PositiveIntegerGreaterThanZero; + +export type PageOrUndefined = t.TypeOf; +export const PageOrUndefined = t.union([Page, t.undefined]); + +export type PerPage = t.TypeOf; +export const PerPage = PositiveInteger; + +export type PerPageOrUndefined = t.TypeOf; +export const PerPageOrUndefined = t.union([PerPage, t.undefined]); + +export type PaginationResult = t.TypeOf; +export const PaginationResult = t.type({ + page: Page, + per_page: PerPage, + total: PositiveInteger, +}); diff --git a/x-pack/plugins/security_solution/common/detection_engine/schemas/common/rule_monitoring.ts b/x-pack/plugins/security_solution/common/detection_engine/schemas/common/rule_monitoring.ts deleted file mode 100644 index af005d1b60a8f..0000000000000 --- a/x-pack/plugins/security_solution/common/detection_engine/schemas/common/rule_monitoring.ts +++ /dev/null @@ -1,147 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -import * as t from 'io-ts'; -import { enumeration, IsoDateString, PositiveInteger } from '@kbn/securitysolution-io-ts-types'; - -// ------------------------------------------------------------------------------------------------- -// Rule execution status - -/** - * Custom execution status of Security rules that is different from the status - * used in the Alerting Framework. We merge our custom status with the - * Framework's status to determine the resulting status of a rule. - */ -export enum RuleExecutionStatus { - /** - * @deprecated Replaced by the 'running' status but left for backwards compatibility - * with rule execution events already written to Event Log in the prior versions of Kibana. - * Don't use when writing rule status changes. - */ - 'going to run' = 'going to run', - - /** - * Rule execution started but not reached any intermediate or final status. - */ - 'running' = 'running', - - /** - * Rule can partially fail for various reasons either in the middle of an execution - * (in this case we update its status right away) or in the end of it. So currently - * this status can be both intermediate and final at the same time. - * A typical reason for a partial failure: not all the indices that the rule searches - * over actually exist. - */ - 'partial failure' = 'partial failure', - - /** - * Rule failed to execute due to unhandled exception or a reason defined in the - * business logic of its executor function. - */ - 'failed' = 'failed', - - /** - * Rule executed successfully without any issues. Note: this status is just an indication - * of a rule's "health". The rule might or might not generate any alerts despite of it. - */ - 'succeeded' = 'succeeded', -} - -export const ruleExecutionStatus = enumeration('RuleExecutionStatus', RuleExecutionStatus); - -export const ruleExecutionStatusOrder = PositiveInteger; -export type RuleExecutionStatusOrder = t.TypeOf; - -export const ruleExecutionStatusOrderByStatus: Record< - RuleExecutionStatus, - RuleExecutionStatusOrder -> = { - [RuleExecutionStatus.succeeded]: 0, - [RuleExecutionStatus['going to run']]: 10, - [RuleExecutionStatus.running]: 15, - [RuleExecutionStatus['partial failure']]: 20, - [RuleExecutionStatus.failed]: 30, -}; - -// ------------------------------------------------------------------------------------------------- -// Rule execution metrics - -export const durationMetric = PositiveInteger; -export type DurationMetric = t.TypeOf; - -export const ruleExecutionMetrics = t.partial({ - total_search_duration_ms: durationMetric, - total_indexing_duration_ms: durationMetric, - execution_gap_duration_s: durationMetric, -}); - -export type RuleExecutionMetrics = t.TypeOf; - -// ------------------------------------------------------------------------------------------------- -// Rule execution summary - -export const ruleExecutionSummary = t.type({ - last_execution: t.type({ - date: IsoDateString, - status: ruleExecutionStatus, - status_order: ruleExecutionStatusOrder, - message: t.string, - metrics: ruleExecutionMetrics, - }), -}); - -export type RuleExecutionSummary = t.TypeOf; - -// ------------------------------------------------------------------------------------------------- -// Rule execution events - -export const ruleExecutionEvent = t.type({ - date: IsoDateString, - status: ruleExecutionStatus, - message: t.string, -}); - -export type RuleExecutionEvent = t.TypeOf; - -// ------------------------------------------------------------------------------------------------- -// Aggregate Rule execution events - -export const aggregateRuleExecutionEvent = t.type({ - execution_uuid: t.string, - timestamp: IsoDateString, - duration_ms: t.number, - status: t.string, - message: t.string, - num_active_alerts: t.number, - num_new_alerts: t.number, - num_recovered_alerts: t.number, - num_triggered_actions: t.number, - num_succeeded_actions: t.number, - num_errored_actions: t.number, - total_search_duration_ms: t.number, - es_search_duration_ms: t.number, - schedule_delay_ms: t.number, - timed_out: t.boolean, - indexing_duration_ms: t.number, - search_duration_ms: t.number, - gap_duration_s: t.number, - security_status: t.string, - security_message: t.string, -}); - -export type AggregateRuleExecutionEvent = t.TypeOf; - -export const executionLogTableSortColumns = t.keyof({ - timestamp: IsoDateString, - duration_ms: t.number, - gap_duration_s: t.number, - indexing_duration_ms: t.number, - search_duration_ms: t.number, - schedule_delay_ms: t.number, -}); - -export type ExecutionLogTableSortColumns = t.TypeOf; diff --git a/x-pack/plugins/security_solution/common/detection_engine/schemas/common/schemas.ts b/x-pack/plugins/security_solution/common/detection_engine/schemas/common/schemas.ts index 4f583dba23abf..b0cbd63d8db42 100644 --- a/x-pack/plugins/security_solution/common/detection_engine/schemas/common/schemas.ts +++ b/x-pack/plugins/security_solution/common/detection_engine/schemas/common/schemas.ts @@ -193,36 +193,12 @@ export type QueryFilterOrUndefined = t.TypeOf; export const references = t.array(t.string); export type References = t.TypeOf; -export const per_page = PositiveInteger; -export type PerPage = t.TypeOf; - -export const perPageOrUndefined = t.union([per_page, t.undefined]); -export type PerPageOrUndefined = t.TypeOf; - -export const page = PositiveIntegerGreaterThanZero; -export type Page = t.TypeOf; - -export const pageOrUndefined = t.union([page, t.undefined]); -export type PageOrUndefined = t.TypeOf; - export const signal_ids = t.array(t.string); export type SignalIds = t.TypeOf; // TODO: Can this be more strict or is this is the set of all Elastic Queries? export const signal_status_query = t.object; -export const sort_field = t.string; -export type SortField = t.TypeOf; - -export const sortFieldOrUndefined = t.union([sort_field, t.undefined]); -export type SortFieldOrUndefined = t.TypeOf; - -export const sort_order = t.keyof({ asc: null, desc: null }); -export type SortOrder = t.TypeOf; - -export const sortOrderOrUndefined = t.union([sort_order, t.undefined]); -export type SortOrderOrUndefined = t.TypeOf; - export const tags = t.array(t.string); export type Tags = t.TypeOf; diff --git a/x-pack/plugins/security_solution/common/detection_engine/schemas/common/sorting.test.ts b/x-pack/plugins/security_solution/common/detection_engine/schemas/common/sorting.test.ts new file mode 100644 index 0000000000000..f0d6638740e32 --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/schemas/common/sorting.test.ts @@ -0,0 +1,87 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { pipe } from 'fp-ts/lib/pipeable'; +import { left } from 'fp-ts/lib/Either'; +import { foldLeftRight, getPaths } from '@kbn/securitysolution-io-ts-utils'; +import { DefaultSortOrderAsc, DefaultSortOrderDesc } from './sorting'; + +describe('Common sorting schemas', () => { + describe('DefaultSortOrderAsc', () => { + describe('Validation succeeds', () => { + it('when valid sort order is passed', () => { + const payload = 'desc'; + const decoded = DefaultSortOrderAsc.decode(payload); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual(payload); + }); + }); + + describe('Validation fails', () => { + it('when invalid sort order is passed', () => { + const payload = 'behind_you'; + const decoded = DefaultSortOrderAsc.decode(payload); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual([ + 'Invalid value "behind_you" supplied to "DefaultSortOrderAsc"', + ]); + expect(message.schema).toEqual({}); + }); + }); + + describe('Validation sets the default sort order "asc"', () => { + it('when sort order is not passed', () => { + const payload = undefined; + const decoded = DefaultSortOrderAsc.decode(payload); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual('asc'); + }); + }); + }); + + describe('DefaultSortOrderDesc', () => { + describe('Validation succeeds', () => { + it('when valid sort order is passed', () => { + const payload = 'asc'; + const decoded = DefaultSortOrderDesc.decode(payload); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual(payload); + }); + }); + + describe('Validation fails', () => { + it('when invalid sort order is passed', () => { + const payload = 'behind_you'; + const decoded = DefaultSortOrderDesc.decode(payload); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual([ + 'Invalid value "behind_you" supplied to "DefaultSortOrderDesc"', + ]); + expect(message.schema).toEqual({}); + }); + }); + + describe('Validation sets the default sort order "desc"', () => { + it('when sort order is not passed', () => { + const payload = null; + const decoded = DefaultSortOrderDesc.decode(payload); + const message = pipe(decoded, foldLeftRight); + + expect(getPaths(left(message.errors))).toEqual([]); + expect(message.schema).toEqual('desc'); + }); + }); + }); +}); diff --git a/x-pack/plugins/security_solution/common/detection_engine/schemas/common/sorting.ts b/x-pack/plugins/security_solution/common/detection_engine/schemas/common/sorting.ts new file mode 100644 index 0000000000000..2cf1712e5ffbc --- /dev/null +++ b/x-pack/plugins/security_solution/common/detection_engine/schemas/common/sorting.ts @@ -0,0 +1,46 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import * as t from 'io-ts'; +import type { Either } from 'fp-ts/lib/Either'; +import { capitalize } from 'lodash'; + +export type SortField = t.TypeOf; +export const SortField = t.string; + +export type SortFieldOrUndefined = t.TypeOf; +export const SortFieldOrUndefined = t.union([SortField, t.undefined]); + +export type SortOrder = t.TypeOf; +export const SortOrder = t.keyof({ asc: null, desc: null }); + +export type SortOrderOrUndefined = t.TypeOf; +export const SortOrderOrUndefined = t.union([SortOrder, t.undefined]); + +const defaultSortOrder = (order: SortOrder): t.Type => { + return new t.Type( + `DefaultSortOrder${capitalize(order)}`, + SortOrder.is, + (input, context): Either => + input == null ? t.success(order) : SortOrder.validate(input, context), + t.identity + ); +}; + +/** + * Types the DefaultSortOrderAsc as: + * - If undefined, then a default sort order of 'asc' will be set + * - If a string is sent in, then the string will be validated to ensure it's a valid SortOrder + */ +export const DefaultSortOrderAsc = defaultSortOrder('asc'); + +/** + * Types the DefaultSortOrderDesc as: + * - If undefined, then a default sort order of 'desc' will be set + * - If a string is sent in, then the string will be validated to ensure it's a valid SortOrder + */ +export const DefaultSortOrderDesc = defaultSortOrder('desc'); diff --git a/x-pack/plugins/security_solution/common/detection_engine/schemas/request/find_rules_schema.ts b/x-pack/plugins/security_solution/common/detection_engine/schemas/request/find_rules_schema.ts index 885fe22b1ccb2..39f0105a2a88f 100644 --- a/x-pack/plugins/security_solution/common/detection_engine/schemas/request/find_rules_schema.ts +++ b/x-pack/plugins/security_solution/common/detection_engine/schemas/request/find_rules_schema.ts @@ -8,22 +8,22 @@ import * as t from 'io-ts'; import { DefaultPerPage, DefaultPage } from '@kbn/securitysolution-io-ts-alerting-types'; -import type { PerPage, Page } from '../common/schemas'; -import { queryFilter, fields, sort_field, sort_order } from '../common/schemas'; +import type { PerPage, Page } from '../common'; +import { queryFilter, fields, SortField, SortOrder } from '../common'; export const findRulesSchema = t.exact( t.partial({ fields, filter: queryFilter, - per_page: DefaultPerPage, // defaults to "20" if not sent in during decode - page: DefaultPage, // defaults to "1" if not sent in during decode - sort_field, - sort_order, + sort_field: SortField, + sort_order: SortOrder, + page: DefaultPage, // defaults to 1 + per_page: DefaultPerPage, // defaults to 20 }) ); export type FindRulesSchema = t.TypeOf; export type FindRulesSchemaDecoded = Omit & { - per_page: PerPage; page: Page; + per_page: PerPage; }; diff --git a/x-pack/plugins/security_solution/common/detection_engine/schemas/request/get_rule_execution_events_schema.test.ts b/x-pack/plugins/security_solution/common/detection_engine/schemas/request/get_rule_execution_events_schema.test.ts deleted file mode 100644 index 05a3c6123c256..0000000000000 --- a/x-pack/plugins/security_solution/common/detection_engine/schemas/request/get_rule_execution_events_schema.test.ts +++ /dev/null @@ -1,117 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -import { pipe } from 'fp-ts/lib/pipeable'; -import { left } from 'fp-ts/lib/Either'; -import { foldLeftRight, getPaths } from '@kbn/securitysolution-io-ts-utils'; -import { - DefaultSortField, - DefaultSortOrder, - DefaultStatusFiltersStringArray, -} from './get_rule_execution_events_schema'; - -describe('get_rule_execution_events_schema', () => { - describe('DefaultStatusFiltersStringArray', () => { - test('it should validate a single ruleExecutionStatus', () => { - const payload = 'succeeded'; - const decoded = DefaultStatusFiltersStringArray.decode(payload); - const message = pipe(decoded, foldLeftRight); - - expect(getPaths(left(message.errors))).toEqual([]); - expect(message.schema).toEqual([payload]); - }); - test('it should validate an array of ruleExecutionStatus joined by "\'"', () => { - const payload = ['succeeded', 'failed']; - const decoded = DefaultStatusFiltersStringArray.decode(payload.join(',')); - const message = pipe(decoded, foldLeftRight); - - expect(getPaths(left(message.errors))).toEqual([]); - expect(message.schema).toEqual(payload); - }); - - test('it should not validate an invalid ruleExecutionStatus', () => { - const payload = ['value 1', 5].join(','); - const decoded = DefaultStatusFiltersStringArray.decode(payload); - const message = pipe(decoded, foldLeftRight); - - expect(getPaths(left(message.errors))).toEqual([ - 'Invalid value "value 1" supplied to "DefaultStatusFiltersStringArray"', - 'Invalid value "5" supplied to "DefaultStatusFiltersStringArray"', - ]); - expect(message.schema).toEqual({}); - }); - - test('it should return a default array entry', () => { - const payload = null; - const decoded = DefaultStatusFiltersStringArray.decode(payload); - const message = pipe(decoded, foldLeftRight); - - expect(getPaths(left(message.errors))).toEqual([]); - expect(message.schema).toEqual([]); - }); - }); - describe('DefaultSortField', () => { - test('it should validate a valid sort field', () => { - const payload = 'duration_ms'; - const decoded = DefaultSortField.decode(payload); - const message = pipe(decoded, foldLeftRight); - - expect(getPaths(left(message.errors))).toEqual([]); - expect(message.schema).toEqual(payload); - }); - - test('it should not validate an invalid sort field', () => { - const payload = 'es_search_duration_ms'; - const decoded = DefaultSortField.decode(payload); - const message = pipe(decoded, foldLeftRight); - - expect(getPaths(left(message.errors))).toEqual([ - 'Invalid value "es_search_duration_ms" supplied to "DefaultSortField"', - ]); - expect(message.schema).toEqual({}); - }); - - test('it should return the default sort field "timestamp"', () => { - const payload = null; - const decoded = DefaultSortField.decode(payload); - const message = pipe(decoded, foldLeftRight); - - expect(getPaths(left(message.errors))).toEqual([]); - expect(message.schema).toEqual('timestamp'); - }); - }); - describe('DefaultSortOrder', () => { - test('it should validate a valid sort order', () => { - const payload = 'asc'; - const decoded = DefaultSortOrder.decode(payload); - const message = pipe(decoded, foldLeftRight); - - expect(getPaths(left(message.errors))).toEqual([]); - expect(message.schema).toEqual(payload); - }); - - test('it should not validate an invalid sort order', () => { - const payload = 'behind_you'; - const decoded = DefaultSortOrder.decode(payload); - const message = pipe(decoded, foldLeftRight); - - expect(getPaths(left(message.errors))).toEqual([ - 'Invalid value "behind_you" supplied to "DefaultSortOrder"', - ]); - expect(message.schema).toEqual({}); - }); - - test('it should return the default sort order "desc"', () => { - const payload = null; - const decoded = DefaultSortOrder.decode(payload); - const message = pipe(decoded, foldLeftRight); - - expect(getPaths(left(message.errors))).toEqual([]); - expect(message.schema).toEqual('desc'); - }); - }); -}); diff --git a/x-pack/plugins/security_solution/common/detection_engine/schemas/request/get_rule_execution_events_schema.ts b/x-pack/plugins/security_solution/common/detection_engine/schemas/request/get_rule_execution_events_schema.ts deleted file mode 100644 index 520f4555fb672..0000000000000 --- a/x-pack/plugins/security_solution/common/detection_engine/schemas/request/get_rule_execution_events_schema.ts +++ /dev/null @@ -1,105 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -import * as t from 'io-ts'; - -import { DefaultPerPage, DefaultPage } from '@kbn/securitysolution-io-ts-alerting-types'; -import { DefaultEmptyString, IsoDateString } from '@kbn/securitysolution-io-ts-types'; - -import type { Either } from 'fp-ts/lib/Either'; -import type { ExecutionLogTableSortColumns, RuleExecutionStatus } from '../common'; -import { executionLogTableSortColumns, ruleExecutionStatus } from '../common'; - -/** - * Types the DefaultStatusFiltersStringArray as: - * - If undefined, then a default array will be set - * - If an array is sent in, then the array will be validated to ensure all elements are a ruleExecutionStatus (or that the array is empty) - */ -export const DefaultStatusFiltersStringArray = new t.Type< - RuleExecutionStatus[], - RuleExecutionStatus[], - unknown ->( - 'DefaultStatusFiltersStringArray', - t.array(ruleExecutionStatus).is, - (input, context): Either => { - if (input == null) { - return t.success([]); - } else if (typeof input === 'string') { - if (input === '') { - return t.success([]); - } else { - return t.array(ruleExecutionStatus).validate(input.split(','), context); - } - } else { - return t.array(ruleExecutionStatus).validate(input, context); - } - }, - t.identity -); - -/** - * Types the DefaultSortField as: - * - If undefined, then a default sort field of 'timestamp' will be set - * - If a string is sent in, then the string will be validated to ensure it is as valid sortFields - */ -export const DefaultSortField = new t.Type< - ExecutionLogTableSortColumns, - ExecutionLogTableSortColumns, - unknown ->( - 'DefaultSortField', - executionLogTableSortColumns.is, - (input, context): Either => - input == null ? t.success('timestamp') : executionLogTableSortColumns.validate(input, context), - t.identity -); - -const sortOrder = t.keyof({ asc: null, desc: null }); -type SortOrder = t.TypeOf; - -/** - * Types the DefaultSortOrder as: - * - If undefined, then a default sort order of 'desc' will be set - * - If a string is sent in, then the string will be validated to ensure it is as valid sortOrder - */ -export const DefaultSortOrder = new t.Type( - 'DefaultSortOrder', - sortOrder.is, - (input, context): Either => - input == null ? t.success('desc') : sortOrder.validate(input, context), - t.identity -); - -/** - * Route Request Params - */ -export const GetRuleExecutionEventsRequestParams = t.exact( - t.type({ - ruleId: t.string, - }) -); - -/** - * Route Query Params (as constructed from the above codecs) - */ -export const GetRuleExecutionEventsQueryParams = t.exact( - t.type({ - start: IsoDateString, - end: IsoDateString, - query_text: DefaultEmptyString, // default to "" if not sent in during decode - status_filters: DefaultStatusFiltersStringArray, // defaults to empty array if not sent in during decode - per_page: DefaultPerPage, // defaults to "20" if not sent in during decode - page: DefaultPage, // defaults to "1" if not sent in during decode - sort_field: DefaultSortField, // defaults to "desc" if not sent in during decode - sort_order: DefaultSortOrder, // defaults to "timestamp" if not sent in during decode - }) -); - -export type GetRuleExecutionEventsRequestParams = t.TypeOf< - typeof GetRuleExecutionEventsRequestParams ->; diff --git a/x-pack/plugins/security_solution/common/detection_engine/schemas/request/index.ts b/x-pack/plugins/security_solution/common/detection_engine/schemas/request/index.ts index 7722feb5f080d..9c1a2581b2347 100644 --- a/x-pack/plugins/security_solution/common/detection_engine/schemas/request/index.ts +++ b/x-pack/plugins/security_solution/common/detection_engine/schemas/request/index.ts @@ -12,9 +12,9 @@ export * from './find_rules_schema'; export * from './import_rules_schema'; export * from './patch_rules_bulk_schema'; export * from './patch_rules_schema'; +export * from './perform_bulk_action_schema'; export * from './query_rules_schema'; export * from './query_signals_index_schema'; +export * from './rule_schemas'; export * from './set_signal_status_schema'; export * from './update_rules_bulk_schema'; -export * from './rule_schemas'; -export * from './perform_bulk_action_schema'; diff --git a/x-pack/plugins/security_solution/common/detection_engine/schemas/request/rule_schemas.ts b/x-pack/plugins/security_solution/common/detection_engine/schemas/request/rule_schemas.ts index 95dc6e7f4e65d..fffbbb8078705 100644 --- a/x-pack/plugins/security_solution/common/detection_engine/schemas/request/rule_schemas.ts +++ b/x-pack/plugins/security_solution/common/detection_engine/schemas/request/rule_schemas.ts @@ -29,6 +29,7 @@ import { import { listArray } from '@kbn/securitysolution-io-ts-list-types'; import { version } from '@kbn/securitysolution-io-ts-types'; +import { RuleExecutionSummary } from '../../rule_monitoring'; import { id, index, @@ -70,7 +71,6 @@ import { created_at, created_by, namespace, - ruleExecutionSummary, RelatedIntegrationArray, RequiredFieldArray, SetupGuide, @@ -486,7 +486,7 @@ const responseRequiredFields = { }; const responseOptionalFields = { - execution_summary: ruleExecutionSummary, + execution_summary: RuleExecutionSummary, }; export const fullResponseSchema = t.intersection([ diff --git a/x-pack/plugins/security_solution/common/detection_engine/schemas/response/index.ts b/x-pack/plugins/security_solution/common/detection_engine/schemas/response/index.ts index e88f07faa2684..1b688ce641a7a 100644 --- a/x-pack/plugins/security_solution/common/detection_engine/schemas/response/index.ts +++ b/x-pack/plugins/security_solution/common/detection_engine/schemas/response/index.ts @@ -7,7 +7,6 @@ export * from './error_schema'; export * from './get_installed_integrations_response_schema'; -export * from './get_rule_execution_events_response'; export * from './import_rules_schema'; export * from './prepackaged_rules_schema'; export * from './prepackaged_rules_status_schema'; diff --git a/x-pack/plugins/security_solution/common/detection_engine/schemas/response/rules_schema.ts b/x-pack/plugins/security_solution/common/detection_engine/schemas/response/rules_schema.ts index 1566aa3c23858..794ef71bf0536 100644 --- a/x-pack/plugins/security_solution/common/detection_engine/schemas/response/rules_schema.ts +++ b/x-pack/plugins/security_solution/common/detection_engine/schemas/response/rules_schema.ts @@ -34,10 +34,11 @@ import { max_signals, } from '@kbn/securitysolution-io-ts-alerting-types'; import { DefaultStringArray, version } from '@kbn/securitysolution-io-ts-types'; - import { DefaultListArray } from '@kbn/securitysolution-io-ts-list-types'; + import { isMlRule } from '../../../machine_learning/helpers'; import { isThresholdRule } from '../../utils'; +import { RuleExecutionSummary } from '../../rule_monitoring'; import { anomaly_threshold, data_view_id, @@ -77,7 +78,6 @@ import { rule_name_override, timestamp_override, namespace, - ruleExecutionSummary, RelatedIntegrationArray, RequiredFieldArray, SetupGuide, @@ -189,7 +189,7 @@ export const partialRulesSchema = t.partial({ namespace, note, uuid: id, // Move to 'required' post-migration - execution_summary: ruleExecutionSummary, + execution_summary: RuleExecutionSummary, }); /** diff --git a/x-pack/plugins/security_solution/common/experimental_features.ts b/x-pack/plugins/security_solution/common/experimental_features.ts index e7c0a711100fb..828f790364c32 100644 --- a/x-pack/plugins/security_solution/common/experimental_features.ts +++ b/x-pack/plugins/security_solution/common/experimental_features.ts @@ -37,6 +37,7 @@ export const allowedExperimentalValues = Object.freeze({ * Enables the Endpoint response actions console in various areas of the app */ responseActionsConsoleEnabled: true, + /** * Enables the cloud security posture navigation inside the security solution */ @@ -46,6 +47,15 @@ export const allowedExperimentalValues = Object.freeze({ * Enables the insights module for related alerts by process ancestry */ insightsRelatedAlertsByProcessAncestry: false, + + /** + * Enables extended rule execution logging to Event Log. When this setting is enabled: + * - Rules write their console error, info, debug, and trace messages to Event Log, + * in addition to other events they log there (status changes and execution metrics). + * - We add a Kibana Advanced Setting that controls this behavior (on/off and log level). + * - We show a table with plain execution logs on the Rule Details page. + */ + extendedRuleExecutionLoggingEnabled: false, }); type ExperimentalConfigKeys = Array; diff --git a/x-pack/plugins/security_solution/common/utils/enum_from_string.test.ts b/x-pack/plugins/security_solution/common/utils/enum_from_string.test.ts new file mode 100644 index 0000000000000..cf49c1e4329a1 --- /dev/null +++ b/x-pack/plugins/security_solution/common/utils/enum_from_string.test.ts @@ -0,0 +1,35 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { enumFromString } from './enum_from_string'; + +describe('enumFromString', () => { + enum TestStringEnum { + 'foo' = 'foo', + 'bar' = 'bar', + } + + const testEnumFromString = enumFromString(TestStringEnum); + + it('returns enum if provided with a known value', () => { + expect(testEnumFromString('foo')).toEqual(TestStringEnum.foo); + expect(testEnumFromString('bar')).toEqual(TestStringEnum.bar); + }); + + it('returns null if provided with an unknown value', () => { + expect(testEnumFromString('xyz')).toEqual(null); + expect(testEnumFromString('123')).toEqual(null); + }); + + it('returns null if provided with null', () => { + expect(testEnumFromString(null)).toEqual(null); + }); + + it('returns null if provided with undefined', () => { + expect(testEnumFromString(undefined)).toEqual(null); + }); +}); diff --git a/x-pack/plugins/security_solution/common/utils/enum_from_string.ts b/x-pack/plugins/security_solution/common/utils/enum_from_string.ts new file mode 100644 index 0000000000000..1651fba365bfa --- /dev/null +++ b/x-pack/plugins/security_solution/common/utils/enum_from_string.ts @@ -0,0 +1,37 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +interface Enum { + [s: string]: T; +} + +/** + * WARNING: It works only with string enums. + * https://www.typescriptlang.org/docs/handbook/enums.html#string-enums + * + * Converts a string into a corresponding enum value. + * Returns null if the value is not in the enum. + * + * @param enm Specified enum. + * @returns Enum value or null. + * + * @example + * enum MyEnum { + * 'foo' = 'foo', + * 'bar' = 'bar', + * } + * + * const foo = enumFromString(MyEnum)('foo'); // MyEnum.foo + * const bar = enumFromString(MyEnum)('bar'); // MyEnum.bar + * const unknown = enumFromString(MyEnum)('xyz'); // null + */ +export const enumFromString = (enm: Enum) => { + const supportedEnumValues = Object.values(enm) as unknown as string[]; + return (value: string | null | undefined): T | null => { + return value && supportedEnumValues.includes(value) ? (value as unknown as T) : null; + }; +}; diff --git a/x-pack/plugins/security_solution/common/utils/to_array.ts b/x-pack/plugins/security_solution/common/utils/to_array.ts index fbb2b8d48a250..b6945708ff0db 100644 --- a/x-pack/plugins/security_solution/common/utils/to_array.ts +++ b/x-pack/plugins/security_solution/common/utils/to_array.ts @@ -5,8 +5,9 @@ * 2.0. */ -export const toArray = (value: T | T[] | null): T[] => +export const toArray = (value: T | T[] | null | undefined): T[] => Array.isArray(value) ? value : value == null ? [] : [value]; + export const toStringArray = (value: T | T[] | null): string[] => { if (Array.isArray(value)) { return value.reduce((acc, v) => { @@ -41,6 +42,7 @@ export const toStringArray = (value: T | T[] | null): string[] => { return [`${value}`]; } }; + export const toObjectArrayOfStrings = ( value: T | T[] | null ): Array<{ diff --git a/x-pack/plugins/security_solution/public/common/components/health_truncate_text/index.tsx b/x-pack/plugins/security_solution/public/common/components/health_truncate_text/index.tsx index 830b1776d8fee..57a893cb1eadc 100644 --- a/x-pack/plugins/security_solution/public/common/components/health_truncate_text/index.tsx +++ b/x-pack/plugins/security_solution/public/common/components/health_truncate_text/index.tsx @@ -6,9 +6,9 @@ */ import React from 'react'; +import styled from 'styled-components'; import type { EuiHealthProps } from '@elastic/eui'; import { EuiHealth, EuiToolTip } from '@elastic/eui'; -import styled from 'styled-components'; const StatusTextWrapper = styled.div` width: 100%; diff --git a/x-pack/plugins/security_solution/public/detection_engine/jest.config.js b/x-pack/plugins/security_solution/public/detection_engine/jest.config.js new file mode 100644 index 0000000000000..9f952742833e0 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/jest.config.js @@ -0,0 +1,26 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +module.exports = { + preset: '@kbn/test', + rootDir: '../../../../..', + roots: ['/x-pack/plugins/security_solution/public/detection_engine'], + coverageDirectory: + '/target/kibana-coverage/jest/x-pack/plugins/security_solution/public/detection_engine', + coverageReporters: ['text', 'html'], + collectCoverageFrom: [ + '/x-pack/plugins/security_solution/public/detection_engine/**/*.{ts,tsx}', + ], + // See: https://github.com/elastic/kibana/issues/117255, the moduleNameMapper creates mocks to avoid memory leaks from kibana core. + moduleNameMapper: { + 'core/server$': '/x-pack/plugins/security_solution/server/__mocks__/core.mock.ts', + 'task_manager/server$': + '/x-pack/plugins/security_solution/server/__mocks__/task_manager.mock.ts', + 'alerting/server$': '/x-pack/plugins/security_solution/server/__mocks__/alert.mock.ts', + 'actions/server$': '/x-pack/plugins/security_solution/server/__mocks__/action.mock.ts', + }, +}; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/__mocks__/api_client.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/__mocks__/api_client.ts new file mode 100644 index 0000000000000..9fb1656ad4603 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/__mocks__/api_client.ts @@ -0,0 +1,73 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type { + GetRuleExecutionEventsResponse, + GetRuleExecutionResultsResponse, +} from '../../../../../common/detection_engine/rule_monitoring'; +import { + LogLevel, + RuleExecutionEventType, +} from '../../../../../common/detection_engine/rule_monitoring'; + +import type { + FetchRuleExecutionEventsArgs, + FetchRuleExecutionResultsArgs, + IRuleMonitoringApiClient, +} from '../api_client_interface'; + +export const api: jest.Mocked = { + fetchRuleExecutionEvents: jest + .fn, [FetchRuleExecutionEventsArgs]>() + .mockResolvedValue({ + events: [ + { + timestamp: '2021-12-29T10:42:59.996Z', + sequence: 0, + level: LogLevel.info, + type: RuleExecutionEventType['status-change'], + message: 'Rule changed status to "succeeded". Rule execution completed without errors', + }, + ], + pagination: { + page: 1, + per_page: 20, + total: 1, + }, + }), + + fetchRuleExecutionResults: jest + .fn, [FetchRuleExecutionResultsArgs]>() + .mockResolvedValue({ + events: [ + { + duration_ms: 3866, + es_search_duration_ms: 1236, + execution_uuid: '88d15095-7937-462c-8f21-9763e1387cad', + gap_duration_s: 0, + indexing_duration_ms: 95, + message: + "rule executed: siem.queryRule:fb1fc150-a292-11ec-a2cf-c1b28b0392b0: 'Lots of Execution Events'", + num_active_alerts: 0, + num_errored_actions: 0, + num_new_alerts: 0, + num_recovered_alerts: 0, + num_succeeded_actions: 1, + num_triggered_actions: 1, + schedule_delay_ms: -127535, + search_duration_ms: 1255, + security_message: 'succeeded', + security_status: 'succeeded', + status: 'success', + timed_out: false, + timestamp: '2022-03-13T06:04:05.838Z', + total_search_duration_ms: 0, + }, + ], + total: 1, + }), +}; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/__mocks__/index.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/__mocks__/index.ts new file mode 100644 index 0000000000000..ebfcff4941a99 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/__mocks__/index.ts @@ -0,0 +1,8 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +export * from './api_client'; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/api_client.test.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/api_client.test.ts new file mode 100644 index 0000000000000..9c69b92210146 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/api_client.test.ts @@ -0,0 +1,138 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { KibanaServices } from '../../../common/lib/kibana'; + +import type { + GetRuleExecutionEventsResponse, + GetRuleExecutionResultsResponse, +} from '../../../../common/detection_engine/rule_monitoring'; +import { + LogLevel, + RuleExecutionEventType, +} from '../../../../common/detection_engine/rule_monitoring'; + +import { api } from './api_client'; + +jest.mock('../../../common/lib/kibana'); + +describe('Rule Monitoring API Client', () => { + const fetchMock = jest.fn(); + const mockKibanaServices = KibanaServices.get as jest.Mock; + mockKibanaServices.mockReturnValue({ http: { fetch: fetchMock } }); + + const signal = new AbortController().signal; + + describe('fetchRuleExecutionEvents', () => { + const responseMock: GetRuleExecutionEventsResponse = { + events: [], + pagination: { + page: 1, + per_page: 20, + total: 0, + }, + }; + + beforeEach(() => { + fetchMock.mockClear(); + fetchMock.mockResolvedValue(responseMock); + }); + + it('calls API correctly with only rule id specified', async () => { + await api.fetchRuleExecutionEvents({ ruleId: '42', signal }); + + expect(fetchMock).toHaveBeenCalledWith( + '/internal/detection_engine/rules/42/execution/events', + { + method: 'GET', + query: {}, + signal, + } + ); + }); + + it('calls API correctly with filter and pagination options', async () => { + await api.fetchRuleExecutionEvents({ + ruleId: '42', + eventTypes: [RuleExecutionEventType.message], + logLevels: [LogLevel.warn, LogLevel.error], + sortOrder: 'asc', + page: 42, + perPage: 146, + signal, + }); + + expect(fetchMock).toHaveBeenCalledWith( + '/internal/detection_engine/rules/42/execution/events', + { + method: 'GET', + query: { + event_types: 'message', + log_levels: 'warn,error', + sort_order: 'asc', + page: 42, + per_page: 146, + }, + signal, + } + ); + }); + }); + + describe('fetchRuleExecutionResults', () => { + const responseMock: GetRuleExecutionResultsResponse = { + events: [], + total: 0, + }; + + beforeEach(() => { + fetchMock.mockClear(); + fetchMock.mockResolvedValue(responseMock); + }); + + it('calls API with correct parameters', async () => { + await api.fetchRuleExecutionResults({ + ruleId: '42', + start: '2001-01-01T17:00:00.000Z', + end: '2001-01-02T17:00:00.000Z', + queryText: '', + statusFilters: [], + signal, + }); + + expect(fetchMock).toHaveBeenCalledWith( + '/internal/detection_engine/rules/42/execution/results', + { + method: 'GET', + query: { + end: '2001-01-02T17:00:00.000Z', + page: undefined, + per_page: undefined, + query_text: '', + sort_field: undefined, + sort_order: undefined, + start: '2001-01-01T17:00:00.000Z', + status_filters: '', + }, + signal, + } + ); + }); + + it('returns API response as is', async () => { + const response = await api.fetchRuleExecutionResults({ + ruleId: '42', + start: 'now-30', + end: 'now', + queryText: '', + statusFilters: [], + signal, + }); + expect(response).toEqual(responseMock); + }); + }); +}); diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/api_client.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/api_client.ts new file mode 100644 index 0000000000000..01bc89b7a6be9 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/api_client.ts @@ -0,0 +1,85 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import dateMath from '@kbn/datemath'; + +import { KibanaServices } from '../../../common/lib/kibana'; + +import type { + GetRuleExecutionEventsResponse, + GetRuleExecutionResultsResponse, +} from '../../../../common/detection_engine/rule_monitoring'; +import { + getRuleExecutionEventsUrl, + getRuleExecutionResultsUrl, +} from '../../../../common/detection_engine/rule_monitoring'; + +import type { + FetchRuleExecutionEventsArgs, + FetchRuleExecutionResultsArgs, + IRuleMonitoringApiClient, +} from './api_client_interface'; + +export const api: IRuleMonitoringApiClient = { + fetchRuleExecutionEvents: ( + args: FetchRuleExecutionEventsArgs + ): Promise => { + const { ruleId, eventTypes, logLevels, sortOrder, page, perPage, signal } = args; + + const url = getRuleExecutionEventsUrl(ruleId); + + return http().fetch(url, { + method: 'GET', + query: { + event_types: eventTypes?.join(','), + log_levels: logLevels?.join(','), + sort_order: sortOrder, + page, + per_page: perPage, + }, + signal, + }); + }, + + fetchRuleExecutionResults: ( + args: FetchRuleExecutionResultsArgs + ): Promise => { + const { + ruleId, + start, + end, + queryText, + statusFilters, + page, + perPage, + sortField, + sortOrder, + signal, + } = args; + + const url = getRuleExecutionResultsUrl(ruleId); + const startDate = dateMath.parse(start); + const endDate = dateMath.parse(end, { roundUp: true }); + + return http().fetch(url, { + method: 'GET', + query: { + start: startDate?.utc().toISOString(), + end: endDate?.utc().toISOString(), + query_text: queryText, + status_filters: statusFilters?.sort()?.join(','), + sort_field: sortField, + sort_order: sortOrder, + page, + per_page: perPage, + }, + signal, + }); + }, +}; + +const http = () => KibanaServices.get().http; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/api_client_interface.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/api_client_interface.ts new file mode 100644 index 0000000000000..b6136a15e2366 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/api_client_interface.ts @@ -0,0 +1,124 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type { SortOrder } from '../../../../common/detection_engine/schemas/common'; +import type { + GetRuleExecutionEventsResponse, + GetRuleExecutionResultsResponse, + LogLevel, + RuleExecutionEventType, + RuleExecutionResult, + RuleExecutionStatus, +} from '../../../../common/detection_engine/rule_monitoring'; + +export interface IRuleMonitoringApiClient { + /** + * Fetches plain rule execution events (status changes, metrics, generic events) from Event Log. + * @throws An error if response is not OK. + */ + fetchRuleExecutionEvents( + args: FetchRuleExecutionEventsArgs + ): Promise; + + /** + * Fetches aggregated rule execution results (events grouped by execution UUID) from Event Log. + * @throws An error if response is not OK. + */ + fetchRuleExecutionResults( + args: FetchRuleExecutionResultsArgs + ): Promise; +} + +export interface FetchRuleExecutionEventsArgs { + /** + * Saved Object ID of the rule (`rule.id`, not static `rule.rule_id`). + */ + ruleId: string; + + /** + * Filter by event type. If set, result will include only events matching any of these. + */ + eventTypes?: RuleExecutionEventType[]; + + /** + * Filter by log level. If set, result will include only events matching any of these. + */ + logLevels?: LogLevel[]; + + /** + * What order to sort by (e.g. `asc` or `desc`). + */ + sortOrder?: SortOrder; + + /** + * Current page to fetch. + */ + page?: number; + + /** + * Number of results to fetch per page. + */ + perPage?: number; + + /** + * Optional signal for cancelling the request. + */ + signal?: AbortSignal; +} + +export interface FetchRuleExecutionResultsArgs { + /** + * Saved Object ID of the rule (`rule.id`, not static `rule.rule_id`). + */ + ruleId: string; + + /** + * Start daterange either in UTC ISO8601 or as datemath string (e.g. `2021-12-29T02:44:41.653Z` or `now-30`). + */ + start: string; + + /** + * End daterange either in UTC ISO8601 or as datemath string (e.g. `2021-12-29T02:44:41.653Z` or `now/w`). + */ + end: string; + + /** + * Search string in querystring format, e.g. + * `event.duration > 1000 OR kibana.alert.rule.execution.metrics.execution_gap_duration_s > 100`. + */ + queryText?: string; + + /** + * Array of `statusFilters` (e.g. `succeeded,failed,partial failure`). + */ + statusFilters?: RuleExecutionStatus[]; + + /** + * Keyof AggregateRuleExecutionEvent field to sort by. + */ + sortField?: keyof RuleExecutionResult; + + /** + * What order to sort by (e.g. `asc` or `desc`). + */ + sortOrder?: SortOrder; + + /** + * Current page to fetch. + */ + page?: number; + + /** + * Number of results to fetch per page. + */ + perPage?: number; + + /** + * Optional signal for cancelling the request. + */ + signal?: AbortSignal; +} diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/index.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/index.ts new file mode 100644 index 0000000000000..f20ead0b41939 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/api/index.ts @@ -0,0 +1,9 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +export * from './api_client_interface'; +export * from './api_client'; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/event_type_filter/index.tsx b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/event_type_filter/index.tsx new file mode 100644 index 0000000000000..ba3776fd92983 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/event_type_filter/index.tsx @@ -0,0 +1,40 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import React, { useCallback } from 'react'; + +import type { RuleExecutionEventType } from '../../../../../../../common/detection_engine/rule_monitoring'; +import { RULE_EXECUTION_EVENT_TYPES } from '../../../../../../../common/detection_engine/rule_monitoring'; +import { EventTypeIndicator } from '../../indicators/event_type_indicator'; +import { MultiselectFilter } from '../multiselect_filter'; + +import * as i18n from './translations'; + +interface EventTypeFilterProps { + selectedItems: RuleExecutionEventType[]; + onChange: (selectedItems: RuleExecutionEventType[]) => void; +} + +const EventTypeFilterComponent: React.FC = ({ selectedItems, onChange }) => { + const renderItem = useCallback((item: RuleExecutionEventType) => { + return ; + }, []); + + return ( + + dataTestSubj="eventTypeFilter" + title={i18n.FILTER_TITLE} + items={RULE_EXECUTION_EVENT_TYPES} + selectedItems={selectedItems} + onSelectionChange={onChange} + renderItem={renderItem} + /> + ); +}; + +export const EventTypeFilter = React.memo(EventTypeFilterComponent); +EventTypeFilter.displayName = 'EventTypeFilter'; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/rule_messages.mock.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/event_type_filter/translations.ts similarity index 54% rename from x-pack/plugins/security_solution/server/lib/detection_engine/signals/rule_messages.mock.ts rename to x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/event_type_filter/translations.ts index 3fcd1125ab956..3097b1b789cc8 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/rule_messages.mock.ts +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/event_type_filter/translations.ts @@ -5,11 +5,11 @@ * 2.0. */ -import { buildRuleMessageFactory } from './rule_messages'; +import { i18n } from '@kbn/i18n'; -export const buildRuleMessageMock = buildRuleMessageFactory({ - id: 'fake id', - ruleId: 'fake rule id', - index: 'fakeindex', - name: 'fake name', -}); +export const FILTER_TITLE = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleMonitoring.eventTypeFilter.filterTitle', + { + defaultMessage: 'Event type', + } +); diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/execution_status_filter/index.tsx b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/execution_status_filter/index.tsx new file mode 100644 index 0000000000000..15ee510f8a414 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/execution_status_filter/index.tsx @@ -0,0 +1,44 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import React, { useCallback } from 'react'; + +import type { RuleExecutionStatus } from '../../../../../../../common/detection_engine/rule_monitoring'; +import { ExecutionStatusIndicator } from '../../indicators/execution_status_indicator'; +import { MultiselectFilter } from '../multiselect_filter'; + +import * as i18n from './translations'; + +interface ExecutionStatusFilterProps { + items: RuleExecutionStatus[]; + selectedItems: RuleExecutionStatus[]; + onChange: (selectedItems: RuleExecutionStatus[]) => void; +} + +const ExecutionStatusFilterComponent: React.FC = ({ + items, + selectedItems, + onChange, +}) => { + const renderItem = useCallback((item: RuleExecutionStatus) => { + return ; + }, []); + + return ( + + dataTestSubj="ExecutionStatusFilter" + title={i18n.FILTER_TITLE} + items={items} + selectedItems={selectedItems} + onSelectionChange={onChange} + renderItem={renderItem} + /> + ); +}; + +export const ExecutionStatusFilter = React.memo(ExecutionStatusFilterComponent); +ExecutionStatusFilter.displayName = 'ExecutionStatusFilter'; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/__mocks__/build_rule_message.mock.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/execution_status_filter/translations.ts similarity index 54% rename from x-pack/plugins/security_solution/server/lib/detection_engine/signals/__mocks__/build_rule_message.mock.ts rename to x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/execution_status_filter/translations.ts index f43142d1d0264..003942bd680d5 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/__mocks__/build_rule_message.mock.ts +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/execution_status_filter/translations.ts @@ -5,11 +5,11 @@ * 2.0. */ -import { buildRuleMessageFactory } from '../rule_messages'; +import { i18n } from '@kbn/i18n'; -export const mockBuildRuleMessage = buildRuleMessageFactory({ - id: 'fake id', - ruleId: 'fake rule id', - index: 'fakeindex', - name: 'fake name', -}); +export const FILTER_TITLE = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleMonitoring.executionStatusFilter.filterTitle', + { + defaultMessage: 'Status', + } +); diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/log_level_filter/index.tsx b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/log_level_filter/index.tsx new file mode 100644 index 0000000000000..8aeeee71cd8de --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/log_level_filter/index.tsx @@ -0,0 +1,40 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import React, { useCallback } from 'react'; + +import type { LogLevel } from '../../../../../../../common/detection_engine/rule_monitoring'; +import { LOG_LEVELS } from '../../../../../../../common/detection_engine/rule_monitoring'; +import { LogLevelIndicator } from '../../indicators/log_level_indicator'; +import { MultiselectFilter } from '../multiselect_filter'; + +import * as i18n from './translations'; + +interface LogLevelFilterProps { + selectedItems: LogLevel[]; + onChange: (selectedItems: LogLevel[]) => void; +} + +const LogLevelFilterComponent: React.FC = ({ selectedItems, onChange }) => { + const renderItem = useCallback((item: LogLevel) => { + return ; + }, []); + + return ( + + dataTestSubj="logLevelFilter" + title={i18n.FILTER_TITLE} + items={LOG_LEVELS} + selectedItems={selectedItems} + onSelectionChange={onChange} + renderItem={renderItem} + /> + ); +}; + +export const LogLevelFilter = React.memo(LogLevelFilterComponent); +LogLevelFilter.displayName = 'LogLevelFilter'; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/log_level_filter/translations.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/log_level_filter/translations.ts new file mode 100644 index 0000000000000..0c7f6c7c2f285 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/log_level_filter/translations.ts @@ -0,0 +1,15 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { i18n } from '@kbn/i18n'; + +export const FILTER_TITLE = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleMonitoring.logLevelFilter.filterTitle', + { + defaultMessage: 'Log level', + } +); diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/multiselect_filter/index.tsx b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/multiselect_filter/index.tsx new file mode 100644 index 0000000000000..c666c09561f08 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/filters/multiselect_filter/index.tsx @@ -0,0 +1,112 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import React, { useCallback, useMemo } from 'react'; +import { noop } from 'lodash'; +import { EuiPopover, EuiFilterGroup, EuiFilterButton, EuiFilterSelectItem } from '@elastic/eui'; +import { useBoolState } from '../../../../../../common/hooks/use_bool_state'; + +export interface MultiselectFilterProps { + dataTestSubj?: string; + title: string; + items: T[]; + selectedItems: T[]; + onSelectionChange?: (selectedItems: T[]) => void; + renderItem?: (item: T) => React.ReactChild; + renderLabel?: (item: T) => string; +} + +const MultiselectFilterComponent = (props: MultiselectFilterProps) => { + const { dataTestSubj, title, items, selectedItems, onSelectionChange, renderItem, renderLabel } = + initializeProps(props); + + const [isPopoverOpen, , closePopover, togglePopover] = useBoolState(); + + const handleItemClick = useCallback( + (item: T) => { + const newSelectedItems = selectedItems.includes(item) + ? selectedItems.filter((i) => i !== item) + : [...selectedItems, item]; + onSelectionChange(newSelectedItems); + }, + [selectedItems, onSelectionChange] + ); + + const filterItemElements = useMemo(() => { + return items.map((item, index) => { + const itemLabel = renderLabel(item); + const itemElement = renderItem(item); + return ( + handleItemClick(item)} + > + {itemElement} + + ); + }); + }, [dataTestSubj, items, selectedItems, renderItem, renderLabel, handleItemClick]); + + return ( + + 0} + isSelected={isPopoverOpen} + onClick={togglePopover} + > + {title} + + } + isOpen={isPopoverOpen} + closePopover={closePopover} + panelPaddingSize="none" + repositionOnScroll + > + {filterItemElements} + + + ); +}; + +// We have to wrap it in a function and cast to original type because React.memo +// returns a component type which is not generic. +const enhanceMultiselectFilterComponent = () => { + const Component = React.memo(MultiselectFilterComponent); + Component.displayName = 'MultiselectFilter'; + return Component as typeof MultiselectFilterComponent; +}; + +export const MultiselectFilter = enhanceMultiselectFilterComponent(); + +const initializeProps = ( + props: MultiselectFilterProps +): Required> => { + const onSelectionChange: (selectedItems: T[]) => void = props.onSelectionChange ?? noop; + const renderLabel: (item: T) => string = props.renderLabel ?? String; + const renderItem: (item: T) => React.ReactChild = props.renderItem ?? renderLabel; + + return { + dataTestSubj: props.dataTestSubj ?? 'multiselectFilter', + title: props.title, + items: props.items, + selectedItems: props.selectedItems, + onSelectionChange, + renderLabel, + renderItem, + }; +}; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/event_type_indicator/index.tsx b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/event_type_indicator/index.tsx new file mode 100644 index 0000000000000..506fddad4559f --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/event_type_indicator/index.tsx @@ -0,0 +1,29 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import React from 'react'; +import { EuiBadge } from '@elastic/eui'; +import type { RuleExecutionEventType } from '../../../../../../../common/detection_engine/rule_monitoring'; +import { getBadgeIcon, getBadgeText } from './utils'; + +interface EventTypeIndicatorProps { + type: RuleExecutionEventType; +} + +const EventTypeIndicatorComponent: React.FC = ({ type }) => { + const icon = getBadgeIcon(type); + const text = getBadgeText(type); + + return ( + + {text} + + ); +}; + +export const EventTypeIndicator = React.memo(EventTypeIndicatorComponent); +EventTypeIndicator.displayName = 'EventTypeIndicator'; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/event_type_indicator/translations.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/event_type_indicator/translations.ts new file mode 100644 index 0000000000000..554e1c4f248ca --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/event_type_indicator/translations.ts @@ -0,0 +1,29 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { i18n } from '@kbn/i18n'; + +export const TYPE_MESSAGE = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleMonitoring.eventTypeIndicator.messageText', + { + defaultMessage: 'Message', + } +); + +export const TYPE_STATUS_CHANGE = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleMonitoring.eventTypeIndicator.statusChangeText', + { + defaultMessage: 'Status', + } +); + +export const TYPE_EXECUTION_METRICS = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleMonitoring.eventTypeIndicator.executionMetricsText', + { + defaultMessage: 'Metrics', + } +); diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/event_type_indicator/utils.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/event_type_indicator/utils.ts new file mode 100644 index 0000000000000..9fd383b759e11 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/event_type_indicator/utils.ts @@ -0,0 +1,38 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type { IconType } from '@elastic/eui'; +import { RuleExecutionEventType } from '../../../../../../../common/detection_engine/rule_monitoring'; +import { assertUnreachable } from '../../../../../../../common/utility_types'; + +import * as i18n from './translations'; + +export const getBadgeIcon = (type: RuleExecutionEventType): IconType => { + switch (type) { + case RuleExecutionEventType.message: + return 'console'; + case RuleExecutionEventType['status-change']: + return 'dot'; + case RuleExecutionEventType['execution-metrics']: + return 'gear'; + default: + return assertUnreachable(type, 'Unknown rule execution event type'); + } +}; + +export const getBadgeText = (type: RuleExecutionEventType): string => { + switch (type) { + case RuleExecutionEventType.message: + return i18n.TYPE_MESSAGE; + case RuleExecutionEventType['status-change']: + return i18n.TYPE_STATUS_CHANGE; + case RuleExecutionEventType['execution-metrics']: + return i18n.TYPE_EXECUTION_METRICS; + default: + return assertUnreachable(type, 'Unknown rule execution event type'); + } +}; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/execution_status_indicator/index.tsx b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/execution_status_indicator/index.tsx new file mode 100644 index 0000000000000..ead104c05f969 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/execution_status_indicator/index.tsx @@ -0,0 +1,41 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import React from 'react'; +import { EuiHealth } from '@elastic/eui'; + +import type { RuleExecutionStatus } from '../../../../../../../common/detection_engine/rule_monitoring'; +import { getEmptyTagValue } from '../../../../../../common/components/empty_value'; +import { RuleStatusBadge } from '../../../../../../detections/components/rules/rule_execution_status'; +import { + getCapitalizedStatusText, + getStatusColor, +} from '../../../../../../detections/components/rules/rule_execution_status/utils'; + +const EMPTY_STATUS_TEXT = getEmptyTagValue(); + +interface ExecutionStatusIndicatorProps { + status?: RuleExecutionStatus | null | undefined; + showTooltip?: boolean; +} + +const ExecutionStatusIndicatorComponent: React.FC = ({ + status, + showTooltip = false, +}) => { + const statusText = getCapitalizedStatusText(status) ?? EMPTY_STATUS_TEXT; + const statusColor = getStatusColor(status); + + return showTooltip ? ( + + ) : ( + {statusText} + ); +}; + +export const ExecutionStatusIndicator = React.memo(ExecutionStatusIndicatorComponent); +ExecutionStatusIndicator.displayName = 'ExecutionStatusIndicator'; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/log_level_indicator/index.tsx b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/log_level_indicator/index.tsx new file mode 100644 index 0000000000000..1e825651c9e1a --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/log_level_indicator/index.tsx @@ -0,0 +1,25 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import React from 'react'; +import { EuiBadge } from '@elastic/eui'; +import type { LogLevel } from '../../../../../../../common/detection_engine/rule_monitoring'; +import { getBadgeColor, getBadgeText } from './utils'; + +interface LogLevelIndicatorProps { + logLevel: LogLevel; +} + +const LogLevelIndicatorComponent: React.FC = ({ logLevel }) => { + const color = getBadgeColor(logLevel); + const text = getBadgeText(logLevel); + + return {text}; +}; + +export const LogLevelIndicator = React.memo(LogLevelIndicatorComponent); +LogLevelIndicator.displayName = 'LogLevelIndicator'; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/log_level_indicator/utils.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/log_level_indicator/utils.ts new file mode 100644 index 0000000000000..757d66d99b2a4 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/indicators/log_level_indicator/utils.ts @@ -0,0 +1,32 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { upperCase } from 'lodash'; +import type { IconColor } from '@elastic/eui'; +import { LogLevel } from '../../../../../../../common/detection_engine/rule_monitoring'; +import { assertUnreachable } from '../../../../../../../common/utility_types'; + +export const getBadgeColor = (logLevel: LogLevel): IconColor => { + switch (logLevel) { + case LogLevel.trace: + return 'hollow'; + case LogLevel.debug: + return 'hollow'; + case LogLevel.info: + return 'default'; + case LogLevel.warn: + return 'warning'; + case LogLevel.error: + return 'danger'; + default: + return assertUnreachable(logLevel, 'Unknown log level'); + } +}; + +export const getBadgeText = (logLevel: LogLevel): string => { + return upperCase(logLevel); +}; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/tables/use_expandable_rows.tsx b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/tables/use_expandable_rows.tsx new file mode 100644 index 0000000000000..099359e30a467 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/tables/use_expandable_rows.tsx @@ -0,0 +1,57 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type React from 'react'; +import { useCallback, useMemo, useState } from 'react'; + +type TableItem = Record; +type TableItemId = string; +type TableItemRowMap = Record; + +interface UseExpandableRowsArgs { + getItemId: (item: T) => TableItemId; + renderItem: (item: T) => React.ReactChild; +} + +export const useExpandableRows = (args: UseExpandableRowsArgs) => { + const { getItemId, renderItem } = args; + + const [itemIdToExpandedRowMap, setItemIdToExpandedRowMap] = useState({}); + + const toggleRowExpanded = useCallback( + (item: T) => { + const itemIdToExpandedRowMapValues = { ...itemIdToExpandedRowMap }; + const itemId = getItemId(item); + + if (itemIdToExpandedRowMapValues[itemId]) { + delete itemIdToExpandedRowMapValues[itemId]; + } else { + itemIdToExpandedRowMapValues[itemId] = renderItem(item); + } + + setItemIdToExpandedRowMap(itemIdToExpandedRowMapValues); + }, + [itemIdToExpandedRowMap, getItemId, renderItem] + ); + + const isRowExpanded = useCallback( + (item: T): boolean => { + const itemId = getItemId(item); + return itemIdToExpandedRowMap[itemId] != null; + }, + [itemIdToExpandedRowMap, getItemId] + ); + + return useMemo(() => { + return { + itemIdToExpandedRowMap, + getItemId, + toggleRowExpanded, + isRowExpanded, + }; + }, [itemIdToExpandedRowMap, getItemId, toggleRowExpanded, isRowExpanded]); +}; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/tables/use_pagination.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/tables/use_pagination.ts new file mode 100644 index 0000000000000..d29edc1550128 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/tables/use_pagination.ts @@ -0,0 +1,60 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { useCallback, useMemo, useState } from 'react'; +import type { CriteriaWithPagination } from '@elastic/eui'; + +const DEFAULT_PAGE_SIZE_OPTIONS = [10, 20, 50]; +const DEFAULT_PAGE_NUMBER = 1; + +interface UsePaginationArgs { + pageSizeOptions?: number[]; + pageSizeDefault?: number; + pageNumberDefault?: number; +} + +type TableItem = Record; + +export const usePagination = (args: UsePaginationArgs) => { + const pageSizeOptions = args.pageSizeOptions ?? DEFAULT_PAGE_SIZE_OPTIONS; + const pageSizeDefault = args.pageSizeDefault ?? pageSizeOptions[0]; + const pageNumberDefault = args.pageNumberDefault ?? DEFAULT_PAGE_NUMBER; + + const [pageSize, setPageSize] = useState(pageSizeDefault); + const [pageNumber, setPageNumber] = useState(pageNumberDefault); + const [totalItemCount, setTotalItemCount] = useState(0); + + const state = useMemo(() => { + return { + pageSizeOptions, + pageSize, + pageNumber, + pageIndex: pageNumber - 1, + totalItemCount, + }; + }, [pageSizeOptions, pageSize, pageNumber, totalItemCount]); + + const update = useCallback( + (criteria: CriteriaWithPagination): void => { + setPageNumber(criteria.page.index + 1); + setPageSize(criteria.page.size); + }, + [setPageNumber, setPageSize] + ); + + const updateTotalItemCount = useCallback( + (count: number | null | undefined): void => { + setTotalItemCount(count ?? 0); + }, + [setTotalItemCount] + ); + + return useMemo( + () => ({ state, update, updateTotalItemCount }), + [state, update, updateTotalItemCount] + ); +}; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/tables/use_sorting.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/tables/use_sorting.ts new file mode 100644 index 0000000000000..45a6723fe5a57 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/tables/use_sorting.ts @@ -0,0 +1,38 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { useCallback, useMemo, useState } from 'react'; +import type { CriteriaWithPagination } from '@elastic/eui'; +import type { SortOrder } from '../../../../../../common/detection_engine/schemas/common'; + +type TableItem = Record; + +export const useSorting = (defaultField: keyof T, defaultOrder: SortOrder) => { + const [sortField, setSortField] = useState(defaultField); + const [sortOrder, setSortOrder] = useState(defaultOrder); + + const state = useMemo(() => { + return { + sort: { + field: sortField, + direction: sortOrder, + }, + }; + }, [sortField, sortOrder]); + + const update = useCallback( + (criteria: CriteriaWithPagination): void => { + if (criteria.sort) { + setSortField(criteria.sort.field); + setSortOrder(criteria.sort.direction); + } + }, + [setSortField, setSortOrder] + ); + + return useMemo(() => ({ state, update }), [state, update]); +}; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/text/text_block.tsx b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/text/text_block.tsx new file mode 100644 index 0000000000000..d57619c3589e5 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/text/text_block.tsx @@ -0,0 +1,31 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import React from 'react'; +import { EuiCodeBlock } from '@elastic/eui'; + +const DEFAULT_OVERFLOW_HEIGHT = 320; + +interface TextBlockProps { + text: string | null | undefined; +} + +const TextBlockComponent: React.FC = ({ text }) => { + return ( + + {text ?? ''} + + ); +}; + +export const TextBlock = React.memo(TextBlockComponent); +TextBlock.displayName = 'TextBlock'; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/text/truncated_text.tsx b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/text/truncated_text.tsx new file mode 100644 index 0000000000000..9b9256150a442 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/basic/text/truncated_text.tsx @@ -0,0 +1,19 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import React from 'react'; + +interface TruncatedTextProps { + text: string | null | undefined; +} + +const TruncatedTextComponent: React.FC = ({ text }) => { + return text != null ? {text} : null; +}; + +export const TruncatedText = React.memo(TruncatedTextComponent); +TruncatedText.displayName = 'TruncatedText'; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/execution_events_table.tsx b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/execution_events_table.tsx new file mode 100644 index 0000000000000..852ca23219812 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/execution_events_table.tsx @@ -0,0 +1,117 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import React, { useCallback, useEffect, useMemo } from 'react'; +import type { CriteriaWithPagination } from '@elastic/eui'; +import { EuiBasicTable, EuiFlexGroup, EuiFlexItem, EuiPanel } from '@elastic/eui'; + +import type { RuleExecutionEvent } from '../../../../../common/detection_engine/rule_monitoring'; + +import { HeaderSection } from '../../../../common/components/header_section'; +import { EventTypeFilter } from '../basic/filters/event_type_filter'; +import { LogLevelFilter } from '../basic/filters/log_level_filter'; +import { ExecutionEventsTableRowDetails } from './execution_events_table_row_details'; + +import { useFilters } from './use_filters'; +import { useSorting } from '../basic/tables/use_sorting'; +import { usePagination } from '../basic/tables/use_pagination'; +import { useColumns } from './use_columns'; +import { useExpandableRows } from '../basic/tables/use_expandable_rows'; +import { useExecutionEvents } from './use_execution_events'; + +import * as i18n from './translations'; + +const PAGE_SIZE_OPTIONS = [10, 20, 50, 100, 200]; + +interface ExecutionEventsTableProps { + ruleId: string; +} + +const ExecutionEventsTableComponent: React.FC = ({ ruleId }) => { + const getItemId = useCallback((item: RuleExecutionEvent): string => { + return `${item.timestamp} ${item.sequence}`; + }, []); + + const renderExpandedItem = useCallback((item: RuleExecutionEvent) => { + return ; + }, []); + + const rows = useExpandableRows({ + getItemId, + renderItem: renderExpandedItem, + }); + + const columns = useColumns({ + toggleRowExpanded: rows.toggleRowExpanded, + isRowExpanded: rows.isRowExpanded, + }); + + const filters = useFilters(); + const sorting = useSorting('timestamp', 'desc'); + const pagination = usePagination({ pageSizeOptions: PAGE_SIZE_OPTIONS }); + + const executionEvents = useExecutionEvents({ + ruleId, + eventTypes: filters.state.eventTypes, + logLevels: filters.state.logLevels, + sortOrder: sorting.state.sort.direction, + page: pagination.state.pageNumber, + perPage: pagination.state.pageSize, + }); + + // Each time execution events are fetched + useEffect(() => { + // We need to update total item count for the pagination to work properly + pagination.updateTotalItemCount(executionEvents.data?.pagination.total); + }, [executionEvents, pagination]); + + const items = useMemo(() => executionEvents.data?.events ?? [], [executionEvents.data]); + + const handleTableChange = useCallback( + (criteria: CriteriaWithPagination): void => { + sorting.update(criteria); + pagination.update(criteria); + }, + [sorting, pagination] + ); + + return ( + + {/* Filter bar */} + + + + + + + + + + + + + {/* Table with items */} + + + ); +}; + +export const ExecutionEventsTable = React.memo(ExecutionEventsTableComponent); +ExecutionEventsTable.displayName = 'RuleExecutionEventsTable'; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/execution_events_table_row_details.tsx b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/execution_events_table_row_details.tsx new file mode 100644 index 0000000000000..27c6e0623c2f1 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/execution_events_table_row_details.tsx @@ -0,0 +1,40 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import React from 'react'; +import { EuiDescriptionList } from '@elastic/eui'; +import type { RuleExecutionEvent } from '../../../../../common/detection_engine/rule_monitoring'; +import { TextBlock } from '../basic/text/text_block'; + +import * as i18n from './translations'; + +interface ExecutionEventsTableRowDetailsProps { + item: RuleExecutionEvent; +} + +const ExecutionEventsTableRowDetailsComponent: React.FC = ({ + item, +}) => { + return ( + , + }, + { + title: i18n.ROW_DETAILS_JSON, + description: , + }, + ]} + /> + ); +}; + +export const ExecutionEventsTableRowDetails = React.memo(ExecutionEventsTableRowDetailsComponent); +ExecutionEventsTableRowDetails.displayName = 'ExecutionEventsTableRowDetails'; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/translations.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/translations.ts new file mode 100644 index 0000000000000..0cc671d7b732c --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/translations.ts @@ -0,0 +1,71 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { i18n } from '@kbn/i18n'; + +export const TABLE_TITLE = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleMonitoring.executionEventsTable.tableTitle', + { + defaultMessage: 'Execution log', + } +); + +export const TABLE_SUBTITLE = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleMonitoring.executionEventsTable.tableSubtitle', + { + defaultMessage: 'A detailed log of rule execution events', + } +); + +export const COLUMN_TIMESTAMP = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleMonitoring.executionEventsTable.timestampColumn', + { + defaultMessage: 'Timestamp', + } +); + +export const COLUMN_LOG_LEVEL = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleMonitoring.executionEventsTable.logLevelColumn', + { + defaultMessage: 'Level', + } +); + +export const COLUMN_EVENT_TYPE = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleMonitoring.executionEventsTable.eventTypeColumn', + { + defaultMessage: 'Type', + } +); + +export const COLUMN_MESSAGE = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleMonitoring.executionEventsTable.messageColumn', + { + defaultMessage: 'Message', + } +); + +export const FETCH_ERROR = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleMonitoring.executionEventsTable.fetchErrorDescription', + { + defaultMessage: 'Failed to fetch rule execution events', + } +); + +export const ROW_DETAILS_MESSAGE = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleMonitoring.executionEventsTable.rowDetails.messageTitle', + { + defaultMessage: 'Message', + } +); + +export const ROW_DETAILS_JSON = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleMonitoring.executionEventsTable.rowDetails.jsonTitle', + { + defaultMessage: 'Full JSON', + } +); diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/use_columns.tsx b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/use_columns.tsx new file mode 100644 index 0000000000000..a3d5d95431e0f --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/use_columns.tsx @@ -0,0 +1,100 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import React, { useMemo } from 'react'; +import type { EuiBasicTableColumn } from '@elastic/eui'; +import { EuiButtonIcon, EuiScreenReaderOnly, RIGHT_ALIGNMENT } from '@elastic/eui'; + +import type { + LogLevel, + RuleExecutionEvent, + RuleExecutionEventType, +} from '../../../../../common/detection_engine/rule_monitoring'; + +import { FormattedDate } from '../../../../common/components/formatted_date'; +import { EventTypeIndicator } from '../basic/indicators/event_type_indicator'; +import { LogLevelIndicator } from '../basic/indicators/log_level_indicator'; +import { TruncatedText } from '../basic/text/truncated_text'; + +import * as i18n from './translations'; + +type TableColumn = EuiBasicTableColumn; + +interface UseColumnsArgs { + toggleRowExpanded: (item: RuleExecutionEvent) => void; + isRowExpanded: (item: RuleExecutionEvent) => boolean; +} + +export const useColumns = (args: UseColumnsArgs): TableColumn[] => { + const { toggleRowExpanded, isRowExpanded } = args; + + return useMemo(() => { + return [ + timestampColumn, + logLevelColumn, + eventTypeColumn, + messageColumn, + expanderColumn({ toggleRowExpanded, isRowExpanded }), + ]; + }, [toggleRowExpanded, isRowExpanded]); +}; + +const timestampColumn: TableColumn = { + field: 'timestamp', + name: i18n.COLUMN_TIMESTAMP, + render: (value: string) => , + sortable: true, + truncateText: false, + width: '20%', +}; + +const logLevelColumn: TableColumn = { + field: 'level', + name: i18n.COLUMN_LOG_LEVEL, + render: (value: LogLevel) => , + sortable: false, + truncateText: false, + width: '8%', +}; + +const eventTypeColumn: TableColumn = { + field: 'type', + name: i18n.COLUMN_EVENT_TYPE, + render: (value: RuleExecutionEventType) => , + sortable: false, + truncateText: false, + width: '8%', +}; + +const messageColumn: TableColumn = { + field: 'message', + name: i18n.COLUMN_MESSAGE, + render: (value: string) => , + sortable: false, + truncateText: true, + width: '64%', +}; + +const expanderColumn = ({ toggleRowExpanded, isRowExpanded }: UseColumnsArgs): TableColumn => { + return { + align: RIGHT_ALIGNMENT, + width: '40px', + isExpander: true, + name: ( + + {'Expand rows'} + + ), + render: (item: RuleExecutionEvent) => ( + toggleRowExpanded(item)} + aria-label={isRowExpanded(item) ? 'Collapse' : 'Expand'} + iconType={isRowExpanded(item) ? 'arrowUp' : 'arrowDown'} + /> + ), + }; +}; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/use_execution_events.test.tsx b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/use_execution_events.test.tsx new file mode 100644 index 0000000000000..9a62fd58a0fbc --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/use_execution_events.test.tsx @@ -0,0 +1,127 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import React from 'react'; +import { QueryClient, QueryClientProvider } from 'react-query'; +import { renderHook, cleanup } from '@testing-library/react-hooks'; + +import { + LogLevel, + RuleExecutionEventType, +} from '../../../../../common/detection_engine/rule_monitoring'; + +import { useExecutionEvents } from './use_execution_events'; +import { useToasts } from '../../../../common/lib/kibana'; +import { api } from '../../api'; + +jest.mock('../../../../common/lib/kibana'); +jest.mock('../../api'); + +const SOME_RULE_ID = 'some-rule-id'; + +describe('useExecutionEvents', () => { + beforeEach(() => { + jest.clearAllMocks(); + }); + + afterEach(async () => { + cleanup(); + }); + + const createReactQueryWrapper = () => { + const queryClient = new QueryClient({ + defaultOptions: { + queries: { + // Turn retries off, otherwise we won't be able to test errors + retry: false, + }, + }, + }); + const wrapper: React.FC = ({ children }) => ( + {children} + ); + return wrapper; + }; + + const render = () => + renderHook(() => useExecutionEvents({ ruleId: SOME_RULE_ID }), { + wrapper: createReactQueryWrapper(), + }); + + it('calls the API via fetchRuleExecutionEvents', async () => { + const fetchRuleExecutionEvents = jest.spyOn(api, 'fetchRuleExecutionEvents'); + + const { waitForNextUpdate } = render(); + + await waitForNextUpdate(); + + expect(fetchRuleExecutionEvents).toHaveBeenCalledTimes(1); + expect(fetchRuleExecutionEvents).toHaveBeenLastCalledWith( + expect.objectContaining({ ruleId: SOME_RULE_ID }) + ); + }); + + it('fetches data from the API', async () => { + const { result, waitForNextUpdate } = render(); + + // It starts from a loading state + expect(result.current.isLoading).toEqual(true); + expect(result.current.isSuccess).toEqual(false); + expect(result.current.isError).toEqual(false); + + // When fetchRuleExecutionEvents returns + await waitForNextUpdate(); + + // It switches to a success state + expect(result.current.isLoading).toEqual(false); + expect(result.current.isSuccess).toEqual(true); + expect(result.current.isError).toEqual(false); + expect(result.current.data).toEqual({ + events: [ + { + timestamp: '2021-12-29T10:42:59.996Z', + sequence: 0, + level: LogLevel.info, + type: RuleExecutionEventType['status-change'], + message: 'Rule changed status to "succeeded". Rule execution completed without errors', + }, + ], + pagination: { + page: 1, + per_page: 20, + total: 1, + }, + }); + }); + + it('handles exceptions from the API', async () => { + const exception = new Error('Boom!'); + jest.spyOn(api, 'fetchRuleExecutionEvents').mockRejectedValue(exception); + + const { result, waitForNextUpdate } = render(); + + // It starts from a loading state + expect(result.current.isLoading).toEqual(true); + expect(result.current.isSuccess).toEqual(false); + expect(result.current.isError).toEqual(false); + + // When fetchRuleExecutionEvents throws + await waitForNextUpdate(); + + // It switches to an error state + expect(result.current.isLoading).toEqual(false); + expect(result.current.isSuccess).toEqual(false); + expect(result.current.isError).toEqual(true); + expect(result.current.error).toEqual(exception); + + // And shows a toast with the caught exception + expect(useToasts().addError).toHaveBeenCalledTimes(1); + expect(useToasts().addError).toHaveBeenCalledWith(exception, { + title: 'Failed to fetch rule execution events', + }); + }); +}); diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/use_execution_events.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/use_execution_events.ts new file mode 100644 index 0000000000000..4a37f55dc1d6a --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/use_execution_events.ts @@ -0,0 +1,35 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { useQuery } from 'react-query'; +import { useAppToasts } from '../../../../common/hooks/use_app_toasts'; + +import type { GetRuleExecutionEventsResponse } from '../../../../../common/detection_engine/rule_monitoring'; +import type { FetchRuleExecutionEventsArgs } from '../../api'; +import { api } from '../../api'; + +import * as i18n from './translations'; + +export type UseExecutionEventsArgs = Omit; + +export const useExecutionEvents = (args: UseExecutionEventsArgs) => { + const { addError } = useAppToasts(); + + return useQuery( + ['detectionEngine', 'ruleMonitoring', 'executionEvents', args], + ({ signal }) => { + return api.fetchRuleExecutionEvents({ ...args, signal }); + }, + { + keepPreviousData: true, + refetchInterval: 20000, + onError: (e) => { + addError(e, { title: i18n.FETCH_ERROR }); + }, + } + ); +}; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/use_filters.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/use_filters.ts new file mode 100644 index 0000000000000..96c78328b683b --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_events_table/use_filters.ts @@ -0,0 +1,25 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { useMemo, useState } from 'react'; + +import type { + LogLevel, + RuleExecutionEventType, +} from '../../../../../common/detection_engine/rule_monitoring'; + +export const useFilters = () => { + const [logLevels, setLogLevels] = useState([]); + const [eventTypes, setEventTypes] = useState([]); + + const state = useMemo(() => ({ logLevels, eventTypes }), [logLevels, eventTypes]); + + return useMemo( + () => ({ state, setLogLevels, setEventTypes }), + [state, setLogLevels, setEventTypes] + ); +}; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_results_table/translations.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_results_table/translations.ts new file mode 100644 index 0000000000000..7925059c2b6e6 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_results_table/translations.ts @@ -0,0 +1,15 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { i18n } from '@kbn/i18n'; + +export const FETCH_ERROR = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleMonitoring.executionResultsTable.fetchErrorDescription', + { + defaultMessage: 'Failed to fetch rule execution results', + } +); diff --git a/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/use_rule_execution_events.test.tsx b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_results_table/use_execution_results.test.tsx similarity index 86% rename from x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/use_rule_execution_events.test.tsx rename to x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_results_table/use_execution_results.test.tsx index 5fba7dd7a2ed2..63565f7cfa1b5 100644 --- a/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/use_rule_execution_events.test.tsx +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_results_table/use_execution_results.test.tsx @@ -5,21 +5,20 @@ * 2.0. */ -jest.mock('./api'); -jest.mock('../../../../common/lib/kibana'); - import React from 'react'; import { QueryClient, QueryClientProvider } from 'react-query'; import { renderHook, cleanup } from '@testing-library/react-hooks'; -import { useRuleExecutionEvents } from './use_rule_execution_events'; - -import * as api from './api'; +import { useExecutionResults } from './use_execution_results'; import { useToasts } from '../../../../common/lib/kibana'; +import { api } from '../../api'; + +jest.mock('../../../../common/lib/kibana'); +jest.mock('../../api'); const SOME_RULE_ID = 'some-rule-id'; -describe('useRuleExecutionEvents', () => { +describe('useExecutionResults', () => { beforeEach(() => { jest.clearAllMocks(); }); @@ -46,7 +45,7 @@ describe('useRuleExecutionEvents', () => { const render = () => renderHook( () => - useRuleExecutionEvents({ + useExecutionResults({ ruleId: SOME_RULE_ID, start: 'now-30', end: 'now', @@ -58,15 +57,15 @@ describe('useRuleExecutionEvents', () => { } ); - it('calls the API via fetchRuleExecutionEvents', async () => { - const fetchRuleExecutionEvents = jest.spyOn(api, 'fetchRuleExecutionEvents'); + it('calls the API via fetchRuleExecutionResults', async () => { + const fetchRuleExecutionResults = jest.spyOn(api, 'fetchRuleExecutionResults'); const { waitForNextUpdate } = render(); await waitForNextUpdate(); - expect(fetchRuleExecutionEvents).toHaveBeenCalledTimes(1); - expect(fetchRuleExecutionEvents).toHaveBeenLastCalledWith( + expect(fetchRuleExecutionResults).toHaveBeenCalledTimes(1); + expect(fetchRuleExecutionResults).toHaveBeenLastCalledWith( expect.objectContaining({ ruleId: SOME_RULE_ID }) ); }); @@ -118,7 +117,7 @@ describe('useRuleExecutionEvents', () => { it('handles exceptions from the API', async () => { const exception = new Error('Boom!'); - jest.spyOn(api, 'fetchRuleExecutionEvents').mockRejectedValue(exception); + jest.spyOn(api, 'fetchRuleExecutionResults').mockRejectedValue(exception); const { result, waitForNextUpdate } = render(); @@ -139,7 +138,7 @@ describe('useRuleExecutionEvents', () => { // And shows a toast with the caught exception expect(useToasts().addError).toHaveBeenCalledTimes(1); expect(useToasts().addError).toHaveBeenCalledWith(exception, { - title: 'Failed to fetch rule execution events', + title: 'Failed to fetch rule execution results', }); }); }); diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_results_table/use_execution_results.tsx b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_results_table/use_execution_results.tsx new file mode 100644 index 0000000000000..a07289969af12 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/components/execution_results_table/use_execution_results.tsx @@ -0,0 +1,34 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { useQuery } from 'react-query'; +import { useAppToasts } from '../../../../common/hooks/use_app_toasts'; + +import type { GetRuleExecutionResultsResponse } from '../../../../../common/detection_engine/rule_monitoring'; +import type { FetchRuleExecutionResultsArgs } from '../../api'; +import { api } from '../../api'; + +import * as i18n from './translations'; + +export type UseExecutionResultsArgs = Omit; + +export const useExecutionResults = (args: UseExecutionResultsArgs) => { + const { addError } = useAppToasts(); + + return useQuery( + ['detectionEngine', 'ruleMonitoring', 'executionResults', args], + ({ signal }) => { + return api.fetchRuleExecutionResults({ ...args, signal }); + }, + { + keepPreviousData: true, + onError: (e) => { + addError(e, { title: i18n.FETCH_ERROR }); + }, + } + ); +}; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/index.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/index.ts new file mode 100644 index 0000000000000..b778a4b1034d6 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/index.ts @@ -0,0 +1,15 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +export * from './api'; + +export * from './components/basic/filters/execution_status_filter'; +export * from './components/basic/indicators/execution_status_indicator'; +export * from './components/execution_events_table/execution_events_table'; +export * from './components/execution_results_table/use_execution_results'; + +export * from './logic/execution_settings/use_execution_settings'; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/logic/execution_settings/use_execution_settings.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/logic/execution_settings/use_execution_settings.ts new file mode 100644 index 0000000000000..cc7e7abb5cc89 --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/logic/execution_settings/use_execution_settings.ts @@ -0,0 +1,50 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { useMemo } from 'react'; + +import { useIsExperimentalFeatureEnabled } from '../../../../common/hooks/use_experimental_features'; +import { useUiSetting$ } from '../../../../common/lib/kibana'; + +import { + EXTENDED_RULE_EXECUTION_LOGGING_ENABLED_SETTING, + EXTENDED_RULE_EXECUTION_LOGGING_MIN_LEVEL_SETTING, +} from '../../../../../common/constants'; +import type { RuleExecutionSettings } from '../../../../../common/detection_engine/rule_monitoring'; +import { LogLevelSetting } from '../../../../../common/detection_engine/rule_monitoring'; + +export const useRuleExecutionSettings = (): RuleExecutionSettings => { + const featureFlagEnabled = useIsExperimentalFeatureEnabled('extendedRuleExecutionLoggingEnabled'); + + const advancedSettingEnabled = useAdvancedSettingSafely( + EXTENDED_RULE_EXECUTION_LOGGING_ENABLED_SETTING, + false + ); + const advancedSettingMinLevel = useAdvancedSettingSafely( + EXTENDED_RULE_EXECUTION_LOGGING_MIN_LEVEL_SETTING, + LogLevelSetting.off + ); + + return useMemo(() => { + return { + extendedLogging: { + isEnabled: featureFlagEnabled && advancedSettingEnabled, + minLevel: advancedSettingMinLevel, + }, + }; + }, [featureFlagEnabled, advancedSettingEnabled, advancedSettingMinLevel]); +}; + +const useAdvancedSettingSafely = (key: string, defaultValue: T): T => { + try { + const [value] = useUiSetting$(key); + return value; + } catch (e) { + // It throws when the setting is not registered (when featureFlagEnabled === false). + return defaultValue; + } +}; diff --git a/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/mocks.ts b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/mocks.ts new file mode 100644 index 0000000000000..e944c5663c33a --- /dev/null +++ b/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring/mocks.ts @@ -0,0 +1,8 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +export * from './api/__mocks__'; diff --git a/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status.tsx b/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status.tsx index 3c61a6a944137..f4284381f7a4d 100644 --- a/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status.tsx +++ b/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status.tsx @@ -8,7 +8,7 @@ import React from 'react'; import { EuiFlexItem, EuiHealth, EuiText } from '@elastic/eui'; -import type { RuleExecutionStatus } from '../../../../../common/detection_engine/schemas/common'; +import type { RuleExecutionStatus } from '../../../../../common/detection_engine/rule_monitoring'; import { FormattedDate } from '../../../../common/components/formatted_date'; import { getEmptyTagValue } from '../../../../common/components/empty_value'; diff --git a/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status_badge.test.tsx b/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status_badge.test.tsx index 52a6c723fcbd6..4dd6edcade20f 100644 --- a/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status_badge.test.tsx +++ b/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status_badge.test.tsx @@ -8,7 +8,7 @@ import React from 'react'; import { render, screen } from '@testing-library/react'; -import { RuleExecutionStatus } from '../../../../../common/detection_engine/schemas/common'; +import { RuleExecutionStatus } from '../../../../../common/detection_engine/rule_monitoring'; import { RuleStatusBadge } from './rule_status_badge'; describe('RuleStatusBadge', () => { diff --git a/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status_badge.tsx b/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status_badge.tsx index 6a37d275e3870..bb3f7360892e6 100644 --- a/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status_badge.tsx +++ b/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status_badge.tsx @@ -11,7 +11,7 @@ import { getEmptyTagValue } from '../../../../common/components/empty_value'; import { HealthTruncateText } from '../../../../common/components/health_truncate_text'; import { getCapitalizedStatusText, getStatusColor } from './utils'; -import type { RuleExecutionStatus } from '../../../../../common/detection_engine/schemas/common'; +import type { RuleExecutionStatus } from '../../../../../common/detection_engine/rule_monitoring'; interface RuleStatusBadgeProps { status: RuleExecutionStatus | null | undefined; diff --git a/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status_failed_callout.test.tsx b/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status_failed_callout.test.tsx index 3e4f1f3c43e57..ec6a029ad27bc 100644 --- a/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status_failed_callout.test.tsx +++ b/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status_failed_callout.test.tsx @@ -8,7 +8,7 @@ import React from 'react'; import { render } from '@testing-library/react'; -import { RuleExecutionStatus } from '../../../../../common/detection_engine/schemas/common'; +import { RuleExecutionStatus } from '../../../../../common/detection_engine/rule_monitoring'; import { RuleStatusFailedCallOut } from './rule_status_failed_callout'; jest.mock('../../../../common/lib/kibana'); diff --git a/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status_failed_callout.tsx b/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status_failed_callout.tsx index 5f10150383369..073d88d19181b 100644 --- a/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status_failed_callout.tsx +++ b/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/rule_status_failed_callout.tsx @@ -9,7 +9,7 @@ import React from 'react'; import { EuiCallOut, EuiFlexGroup, EuiFlexItem } from '@elastic/eui'; import { FormattedDate } from '../../../../common/components/formatted_date'; -import { RuleExecutionStatus } from '../../../../../common/detection_engine/schemas/common'; +import { RuleExecutionStatus } from '../../../../../common/detection_engine/rule_monitoring'; import * as i18n from './translations'; diff --git a/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/utils.ts b/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/utils.ts index dc074b157e3d2..c6d5e1ce5af25 100644 --- a/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/utils.ts +++ b/x-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status/utils.ts @@ -8,7 +8,7 @@ import type { IconColor } from '@elastic/eui'; import { capitalize } from 'lodash'; import { assertUnreachable } from '../../../../../common/utility_types'; -import { RuleExecutionStatus } from '../../../../../common/detection_engine/schemas/common'; +import { RuleExecutionStatus } from '../../../../../common/detection_engine/rule_monitoring'; export const getStatusText = (value: RuleExecutionStatus | null | undefined): string | null => { if (value == null) { diff --git a/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/__mocks__/api.ts b/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/__mocks__/api.ts index 346acb249dbb0..69aa2a4502bc0 100644 --- a/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/__mocks__/api.ts +++ b/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/__mocks__/api.ts @@ -6,7 +6,6 @@ */ import type { - GetAggregateRuleExecutionEventsResponse, GetInstalledIntegrationsResponse, RulesSchema, } from '../../../../../../common/detection_engine/schemas/response'; @@ -60,49 +59,6 @@ export const fetchRuleById = jest.fn( export const fetchRules = async (_: FetchRulesProps): Promise => Promise.resolve(rulesMock); -export const fetchRuleExecutionEvents = async ({ - ruleId, - start, - end, - filters, - signal, -}: { - ruleId: string; - start: string; - end: string; - filters?: string; - signal?: AbortSignal; -}): Promise => { - return Promise.resolve({ - events: [ - { - duration_ms: 3866, - es_search_duration_ms: 1236, - execution_uuid: '88d15095-7937-462c-8f21-9763e1387cad', - gap_duration_s: 0, - indexing_duration_ms: 95, - message: - "rule executed: siem.queryRule:fb1fc150-a292-11ec-a2cf-c1b28b0392b0: 'Lots of Execution Events'", - num_active_alerts: 0, - num_errored_actions: 0, - num_new_alerts: 0, - num_recovered_alerts: 0, - num_succeeded_actions: 1, - num_triggered_actions: 1, - schedule_delay_ms: -127535, - search_duration_ms: 1255, - security_message: 'succeeded', - security_status: 'succeeded', - status: 'success', - timed_out: false, - timestamp: '2022-03-13T06:04:05.838Z', - total_search_duration_ms: 0, - }, - ], - total: 1, - }); -}; - export const fetchTags = async ({ signal }: { signal: AbortSignal }): Promise => Promise.resolve(['elastic', 'love', 'quality', 'code']); diff --git a/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/api.test.ts b/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/api.test.ts index 0acaaf7a1fe0f..28d708743419a 100644 --- a/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/api.test.ts +++ b/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/api.test.ts @@ -5,7 +5,9 @@ * 2.0. */ +import { buildEsQuery } from '@kbn/es-query'; import { KibanaServices } from '../../../../common/lib/kibana'; + import { createRule, updateRule, @@ -15,7 +17,6 @@ import { createPrepackagedRules, importRules, exportRules, - fetchRuleExecutionEvents, fetchTags, getPrePackagedRulesStatus, previewRule, @@ -27,7 +28,7 @@ import { } from '../../../../../common/detection_engine/schemas/request/rule_schemas.mock'; import { getPatchRulesSchemaMock } from '../../../../../common/detection_engine/schemas/request/patch_rules_schema.mock'; import { rulesMock } from './mock'; -import { buildEsQuery } from '@kbn/es-query'; + const abortCtrl = new AbortController(); const mockKibanaServices = KibanaServices.get as jest.Mock; jest.mock('../../../../common/lib/kibana'); @@ -617,56 +618,6 @@ describe('Detections Rules API', () => { }); }); - describe('fetchRuleExecutionEvents', () => { - const responseMock = { events: [] }; - - beforeEach(() => { - fetchMock.mockClear(); - fetchMock.mockResolvedValue(responseMock); - }); - - test('calls API with correct parameters', async () => { - await fetchRuleExecutionEvents({ - ruleId: '42', - start: '2001-01-01T17:00:00.000Z', - end: '2001-01-02T17:00:00.000Z', - queryText: '', - statusFilters: [], - signal: abortCtrl.signal, - }); - - expect(fetchMock).toHaveBeenCalledWith( - '/internal/detection_engine/rules/42/execution/events', - { - method: 'GET', - query: { - end: '2001-01-02T17:00:00.000Z', - page: undefined, - per_page: undefined, - query_text: '', - sort_field: undefined, - sort_order: undefined, - start: '2001-01-01T17:00:00.000Z', - status_filters: '', - }, - signal: abortCtrl.signal, - } - ); - }); - - test('returns API response as is', async () => { - const response = await fetchRuleExecutionEvents({ - ruleId: '42', - start: 'now-30', - end: 'now', - queryText: '', - statusFilters: [], - signal: abortCtrl.signal, - }); - expect(response).toEqual(responseMock); - }); - }); - describe('fetchTags', () => { beforeEach(() => { fetchMock.mockClear(); diff --git a/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/api.ts b/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/api.ts index 217810e343970..dfee0f418d2d5 100644 --- a/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/api.ts +++ b/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/api.ts @@ -5,9 +5,7 @@ * 2.0. */ -import type { SortOrder } from '@elastic/elasticsearch/lib/api/typesWithBodyKey'; import { camelCase } from 'lodash'; -import dateMath from '@kbn/datemath'; import type { HttpStart } from '@kbn/core/public'; import { @@ -17,23 +15,17 @@ import { DETECTION_ENGINE_TAGS_URL, DETECTION_ENGINE_RULES_BULK_ACTION, DETECTION_ENGINE_RULES_PREVIEW, - detectionEngineRuleExecutionEventsUrl, DETECTION_ENGINE_INSTALLED_INTEGRATIONS_URL, } from '../../../../../common/constants'; -import type { - AggregateRuleExecutionEvent, - BulkAction, - RuleExecutionStatus, -} from '../../../../../common/detection_engine/schemas/common'; +import type { BulkAction } from '../../../../../common/detection_engine/schemas/common'; import type { FullResponseSchema, PreviewResponse, } from '../../../../../common/detection_engine/schemas/request'; import type { RulesSchema, - GetAggregateRuleExecutionEventsResponse, + GetInstalledIntegrationsResponse, } from '../../../../../common/detection_engine/schemas/response'; -import type { GetInstalledIntegrationsResponse } from '../../../../../common/detection_engine/schemas/response/get_installed_integrations_response_schema'; import type { UpdateRulesProps, @@ -321,64 +313,6 @@ export const exportRules = async ({ }); }; -/** - * Fetch rule execution events (e.g. status changes) from Event Log. - * - * @param ruleId Saved Object ID of the rule (`rule.id`, not static `rule.rule_id`) - * @param start Start daterange either in UTC ISO8601 or as datemath string (e.g. `2021-12-29T02:44:41.653Z` or `now-30`) - * @param end End daterange either in UTC ISO8601 or as datemath string (e.g. `2021-12-29T02:44:41.653Z` or `now/w`) - * @param queryText search string in querystring format (e.g. `event.duration > 1000 OR kibana.alert.rule.execution.metrics.execution_gap_duration_s > 100`) - * @param statusFilters RuleExecutionStatus[] array of `statusFilters` (e.g. `succeeded,failed,partial failure`) - * @param page current page to fetch - * @param perPage number of results to fetch per page - * @param sortField keyof AggregateRuleExecutionEvent field to sort by - * @param sortOrder SortOrder what order to sort by (e.g. `asc` or `desc`) - * @param signal AbortSignal Optional signal for cancelling the request - * - * @throws An error if response is not OK - */ -export const fetchRuleExecutionEvents = async ({ - ruleId, - start, - end, - queryText, - statusFilters, - page, - perPage, - sortField, - sortOrder, - signal, -}: { - ruleId: string; - start: string; - end: string; - queryText?: string; - statusFilters?: RuleExecutionStatus[]; - page?: number; - perPage?: number; - sortField?: keyof AggregateRuleExecutionEvent; - sortOrder?: SortOrder; - signal?: AbortSignal; -}): Promise => { - const url = detectionEngineRuleExecutionEventsUrl(ruleId); - const startDate = dateMath.parse(start); - const endDate = dateMath.parse(end, { roundUp: true }); - return KibanaServices.get().http.fetch(url, { - method: 'GET', - query: { - start: startDate?.utc().toISOString(), - end: endDate?.utc().toISOString(), - query_text: queryText, - status_filters: statusFilters?.sort()?.join(','), - page, - per_page: perPage, - sort_field: sortField, - sort_order: sortOrder, - }, - signal, - }); -}; - /** * Fetch all unique Tags used by Rules * diff --git a/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/index.ts b/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/index.ts index 8f6dbccd1ee57..42c1eff7435bc 100644 --- a/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/index.ts +++ b/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/index.ts @@ -11,4 +11,3 @@ export * from './use_create_rule'; export * from './types'; export * from './use_rule'; export * from './use_pre_packaged_rules'; -export * from './use_rule_execution_events'; diff --git a/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/mock.ts b/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/mock.ts index 99c38b6a62a1d..bdcc8e0ec5e8a 100644 --- a/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/mock.ts +++ b/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/mock.ts @@ -5,7 +5,6 @@ * 2.0. */ -import type { GetAggregateRuleExecutionEventsResponse } from '../../../../../common/detection_engine/schemas/response'; import type { FetchRulesResponse, Rule } from './types'; export const savedRuleMock: Rule = { @@ -136,170 +135,3 @@ export const rulesMock: FetchRulesResponse = { }, ], }; - -export const ruleExecutionEventsMock: GetAggregateRuleExecutionEventsResponse = { - events: [ - { - execution_uuid: 'dc45a63c-4872-4964-a2d0-bddd8b2e634d', - timestamp: '2022-04-28T21:19:08.047Z', - duration_ms: 3, - status: 'failure', - message: 'siem.queryRule:a6e61cf0-c737-11ec-9e32-e14913ffdd2d: execution failed', - num_active_alerts: 0, - num_new_alerts: 0, - num_recovered_alerts: 0, - num_triggered_actions: 0, - num_succeeded_actions: 0, - num_errored_actions: 0, - total_search_duration_ms: 0, - es_search_duration_ms: 0, - schedule_delay_ms: 2169, - timed_out: false, - indexing_duration_ms: 0, - search_duration_ms: 0, - gap_duration_s: 0, - security_status: 'failed', - security_message: 'Rule failed to execute because rule ran after it was disabled.', - }, - { - execution_uuid: '0fde9271-05d0-4bfb-8ff8-815756d28350', - timestamp: '2022-04-28T21:19:04.973Z', - duration_ms: 1446, - status: 'success', - message: - "rule executed: siem.queryRule:a6e61cf0-c737-11ec-9e32-e14913ffdd2d: 'Click here for hot fresh alerts!'", - num_active_alerts: 0, - num_new_alerts: 0, - num_recovered_alerts: 0, - num_triggered_actions: 0, - num_succeeded_actions: 0, - num_errored_actions: 0, - total_search_duration_ms: 0, - es_search_duration_ms: 0, - schedule_delay_ms: 2089, - timed_out: false, - indexing_duration_ms: 0, - search_duration_ms: 2, - gap_duration_s: 0, - security_status: 'succeeded', - security_message: 'succeeded', - }, - { - execution_uuid: '5daaa259-ded8-4a52-853e-1e7652d325d5', - timestamp: '2022-04-28T21:19:01.976Z', - duration_ms: 1395, - status: 'success', - message: - "rule executed: siem.queryRule:a6e61cf0-c737-11ec-9e32-e14913ffdd2d: 'Click here for hot fresh alerts!'", - num_active_alerts: 0, - num_new_alerts: 0, - num_recovered_alerts: 0, - num_triggered_actions: 0, - num_succeeded_actions: 0, - num_errored_actions: 0, - total_search_duration_ms: 0, - es_search_duration_ms: 1, - schedule_delay_ms: 2637, - timed_out: false, - indexing_duration_ms: 0, - search_duration_ms: 3, - gap_duration_s: 0, - security_status: 'succeeded', - security_message: 'succeeded', - }, - { - execution_uuid: 'c7223e1c-4264-4a27-8697-0d720243fafc', - timestamp: '2022-04-28T21:18:58.431Z', - duration_ms: 1815, - status: 'success', - message: - "rule executed: siem.queryRule:a6e61cf0-c737-11ec-9e32-e14913ffdd2d: 'Click here for hot fresh alerts!'", - num_active_alerts: 0, - num_new_alerts: 0, - num_recovered_alerts: 0, - num_triggered_actions: 0, - num_succeeded_actions: 0, - num_errored_actions: 0, - total_search_duration_ms: 0, - es_search_duration_ms: 1, - schedule_delay_ms: -255429, - timed_out: false, - indexing_duration_ms: 0, - search_duration_ms: 3, - gap_duration_s: 0, - security_status: 'succeeded', - security_message: 'succeeded', - }, - { - execution_uuid: '1f6ba0c1-cc36-4f45-b919-7790b8a8d670', - timestamp: '2022-04-28T21:18:13.954Z', - duration_ms: 2055, - status: 'success', - message: - "rule executed: siem.queryRule:a6e61cf0-c737-11ec-9e32-e14913ffdd2d: 'Click here for hot fresh alerts!'", - num_active_alerts: 0, - num_new_alerts: 0, - num_recovered_alerts: 0, - num_triggered_actions: 0, - num_succeeded_actions: 0, - num_errored_actions: 0, - total_search_duration_ms: 0, - es_search_duration_ms: 0, - schedule_delay_ms: 2027, - timed_out: false, - indexing_duration_ms: 0, - search_duration_ms: 0, - gap_duration_s: 0, - security_status: 'partial failure', - security_message: - 'Check privileges failed to execute ResponseError: index_not_found_exception: [index_not_found_exception] Reason: no such index [yup] name: "Click here for hot fresh alerts!" id: "a6e61cf0-c737-11ec-9e32-e14913ffdd2d" rule id: "34946b12-88d1-49ef-82b7-9cad45972030" execution id: "1f6ba0c1-cc36-4f45-b919-7790b8a8d670" space ID: "default"', - }, - { - execution_uuid: 'b0f65d64-b229-432b-9d39-f4385a7f9368', - timestamp: '2022-04-28T21:15:43.086Z', - duration_ms: 1205, - status: 'success', - message: - "rule executed: siem.queryRule:a6e61cf0-c737-11ec-9e32-e14913ffdd2d: 'Click here for hot fresh alerts!'", - num_active_alerts: 0, - num_new_alerts: 0, - num_recovered_alerts: 0, - num_triggered_actions: 0, - num_succeeded_actions: 0, - num_errored_actions: 0, - total_search_duration_ms: 0, - es_search_duration_ms: 672, - schedule_delay_ms: 3086, - timed_out: false, - indexing_duration_ms: 140, - search_duration_ms: 684, - gap_duration_s: 0, - security_status: 'succeeded', - security_message: 'succeeded', - }, - { - execution_uuid: '7bfd25b9-c0d8-44b1-982c-485169466a8e', - timestamp: '2022-04-28T21:10:40.135Z', - duration_ms: 6321, - status: 'success', - message: - "rule executed: siem.queryRule:a6e61cf0-c737-11ec-9e32-e14913ffdd2d: 'Click here for hot fresh alerts!'", - num_active_alerts: 0, - num_new_alerts: 0, - num_recovered_alerts: 0, - num_triggered_actions: 0, - num_succeeded_actions: 0, - num_errored_actions: 0, - total_search_duration_ms: 0, - es_search_duration_ms: 930, - schedule_delay_ms: 1222, - timed_out: false, - indexing_duration_ms: 2103, - search_duration_ms: 946, - gap_duration_s: 0, - security_status: 'succeeded', - security_message: 'succeeded', - }, - ], - total: 7, -}; diff --git a/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/translations.ts b/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/translations.ts index 86107a4019b0a..c1161db6db26f 100644 --- a/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/translations.ts +++ b/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/translations.ts @@ -109,10 +109,3 @@ export const RELOAD_MISSING_PREPACKAGED_RULES_AND_TIMELINES = ( 'Install {missingRules} Elastic prebuilt {missingRules, plural, =1 {rule} other {rules}} and {missingTimelines} Elastic prebuilt {missingTimelines, plural, =1 {timeline} other {timelines}} ', } ); - -export const RULE_EXECUTION_EVENTS_FETCH_FAILURE = i18n.translate( - 'xpack.securitySolution.containers.detectionEngine.ruleExecutionEventsFetchFailDescription', - { - defaultMessage: 'Failed to fetch rule execution events', - } -); diff --git a/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/types.ts b/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/types.ts index 31c0b8da55cbf..ed5ebc233f9ad 100644 --- a/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/types.ts +++ b/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/types.ts @@ -22,6 +22,9 @@ import { severity_mapping, severity, } from '@kbn/securitysolution-io-ts-alerting-types'; + +import { RuleExecutionSummary } from '../../../../../common/detection_engine/rule_monitoring'; + import type { SortOrder, BulkAction, @@ -41,7 +44,6 @@ import { event_category_override, tiebreaker_field, threshold, - ruleExecutionSummary, RelatedIntegrationArray, RequiredFieldArray, SetupGuide, @@ -169,7 +171,7 @@ export const RuleSchema = t.intersection([ exceptions_list: listArray, uuid: t.string, version: t.number, - execution_summary: ruleExecutionSummary, + execution_summary: RuleExecutionSummary, }), ]); diff --git a/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/use_rule_execution_events.tsx b/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/use_rule_execution_events.tsx deleted file mode 100644 index 2aa378379fc14..0000000000000 --- a/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/use_rule_execution_events.tsx +++ /dev/null @@ -1,80 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -import type { SortOrder } from '@elastic/elasticsearch/lib/api/typesWithBodyKey'; -import { useQuery } from 'react-query'; -import type { - AggregateRuleExecutionEvent, - RuleExecutionStatus, -} from '../../../../../common/detection_engine/schemas/common'; -import type { GetAggregateRuleExecutionEventsResponse } from '../../../../../common/detection_engine/schemas/response'; -import { useAppToasts } from '../../../../common/hooks/use_app_toasts'; -import { fetchRuleExecutionEvents } from './api'; -import * as i18n from './translations'; - -export interface UseRuleExecutionEventsArgs { - ruleId: string; - start: string; - end: string; - queryText?: string; - statusFilters?: RuleExecutionStatus[]; - page?: number; - perPage?: number; - sortField?: keyof AggregateRuleExecutionEvent; - sortOrder?: SortOrder; -} - -export const useRuleExecutionEvents = ({ - ruleId, - start, - end, - queryText, - statusFilters, - page, - perPage, - sortField, - sortOrder, -}: UseRuleExecutionEventsArgs) => { - const { addError } = useAppToasts(); - - return useQuery( - [ - 'ruleExecutionEvents', - { - ruleId, - start, - end, - queryText, - statusFilters, - page, - perPage, - sortField, - sortOrder, - }, - ], - async ({ signal }) => { - return fetchRuleExecutionEvents({ - ruleId, - start, - end, - queryText, - statusFilters, - page, - perPage, - sortField, - sortOrder, - signal, - }); - }, - { - keepPreviousData: true, - onError: (e) => { - addError(e, { title: i18n.RULE_EXECUTION_EVENTS_FETCH_FAILURE }); - }, - } - ); -}; diff --git a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/all/use_columns.tsx b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/all/use_columns.tsx index e93c6a4ce0af0..30706752eeffd 100644 --- a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/all/use_columns.tsx +++ b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/all/use_columns.tsx @@ -39,7 +39,7 @@ import { RuleStatusBadge } from '../../../../components/rules/rule_execution_sta import type { DurationMetric, RuleExecutionSummary, -} from '../../../../../../common/detection_engine/schemas/common'; +} from '../../../../../../common/detection_engine/rule_monitoring'; import { useAppToasts } from '../../../../../common/hooks/use_app_toasts'; import { useStartTransaction } from '../../../../../common/lib/apm/use_start_transaction'; import { useInvalidateRules } from '../../../../containers/detection_engine/rules/use_find_rules_query'; diff --git a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/__mocks__/rule_details_context.tsx b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/__mocks__/rule_details_context.tsx index 220a0635f2011..e8cdcebbc938a 100644 --- a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/__mocks__/rule_details_context.tsx +++ b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/__mocks__/rule_details_context.tsx @@ -10,7 +10,7 @@ import type { RuleDetailsContextType } from '../rule_details_context'; export const useRuleDetailsContextMock = { create: (): jest.Mocked => ({ - executionLogs: { + executionResults: { state: { superDatePicker: { recentlyUsedRanges: [], diff --git a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/__snapshots__/execution_log_search_bar.test.tsx.snap b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/__snapshots__/execution_log_search_bar.test.tsx.snap index a4e8e2cf6e9bd..241f363d4885f 100644 --- a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/__snapshots__/execution_log_search_bar.test.tsx.snap +++ b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/__snapshots__/execution_log_search_bar.test.tsx.snap @@ -10,69 +10,17 @@ exports[`ExecutionLogSearchBar snapshots renders correctly against snapshot 1`] - - - Status - - } - closePopover={[Function]} - display="inlineBlock" - hasArrow={true} - isOpen={false} - ownFocus={true} - panelPaddingSize="none" - repositionOnScroll={true} - > - - - Succeeded - - - - - Failed - - - - - Partial failure - - - - + `; diff --git a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_columns.tsx b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_columns.tsx index 8bf9c4895e07a..9d5fd1a974dac 100644 --- a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_columns.tsx +++ b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_columns.tsx @@ -5,26 +5,27 @@ * 2.0. */ +import React from 'react'; import type { EuiBasicTableColumn } from '@elastic/eui'; -import { EuiHealth, EuiLink, EuiText } from '@elastic/eui'; +import { EuiLink, EuiText } from '@elastic/eui'; import type { DocLinksStart } from '@kbn/core/public'; import { FormattedMessage } from '@kbn/i18n-react'; -import { capitalize } from 'lodash'; -import React from 'react'; + import type { - AggregateRuleExecutionEvent, + RuleExecutionResult, RuleExecutionStatus, -} from '../../../../../../../common/detection_engine/schemas/common'; -import { getEmptyTagValue, getEmptyValue } from '../../../../../../common/components/empty_value'; +} from '../../../../../../../common/detection_engine/rule_monitoring'; + +import { getEmptyValue } from '../../../../../../common/components/empty_value'; import { FormattedDate } from '../../../../../../common/components/formatted_date'; -import { getStatusColor } from '../../../../../components/rules/rule_execution_status/utils'; +import { ExecutionStatusIndicator } from '../../../../../../detection_engine/rule_monitoring'; import { PopoverTooltip } from '../../all/popover_tooltip'; import { TableHeaderTooltipCell } from '../../all/table_header_tooltip_cell'; import { RuleDurationFormat } from './rule_duration_format'; import * as i18n from './translations'; -export const EXECUTION_LOG_COLUMNS: Array> = [ +export const EXECUTION_LOG_COLUMNS: Array> = [ { name: ( ), field: 'security_status', - render: (value: RuleExecutionStatus) => - value ? ( - {capitalize(value)} - ) : ( - getEmptyTagValue() - ), + render: (value: RuleExecutionStatus) => ( + + ), sortable: false, truncateText: false, width: '10%', @@ -51,7 +49,7 @@ export const EXECUTION_LOG_COLUMNS: Array ), - render: (value: string) => , + render: (value: string) => , sortable: true, truncateText: false, width: '15%', @@ -88,7 +86,7 @@ export const EXECUTION_LOG_COLUMNS: Array> => [ +): Array> => [ { field: 'gap_duration_s', name: ( diff --git a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_search_bar.test.tsx b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_search_bar.test.tsx index 60b5f7813bbf0..abda12afde763 100644 --- a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_search_bar.test.tsx +++ b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_search_bar.test.tsx @@ -23,7 +23,12 @@ describe('ExecutionLogSearchBar', () => { describe('snapshots', () => { test('renders correctly against snapshot', () => { const wrapper = shallow( - + ); expect(wrapper).toMatchSnapshot(); }); diff --git a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_search_bar.tsx b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_search_bar.tsx index 74804b7c6b557..39a4c3ee159de 100644 --- a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_search_bar.tsx +++ b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_search_bar.tsx @@ -5,20 +5,13 @@ * 2.0. */ -import React, { useCallback, useEffect, useMemo, useState } from 'react'; -import { capitalize, replace } from 'lodash'; -import { - EuiHealth, - EuiFieldSearch, - EuiFlexGroup, - EuiFlexItem, - EuiPopover, - EuiFilterGroup, - EuiFilterButton, - EuiFilterSelectItem, -} from '@elastic/eui'; -import { RuleExecutionStatus } from '../../../../../../../common/detection_engine/schemas/common'; -import { getStatusColor } from '../../../../../components/rules/rule_execution_status/utils'; +import React, { useCallback } from 'react'; +import { replace } from 'lodash'; +import { EuiFieldSearch, EuiFlexGroup, EuiFlexItem } from '@elastic/eui'; + +import { RuleExecutionStatus } from '../../../../../../../common/detection_engine/rule_monitoring'; +import { ExecutionStatusFilter } from '../../../../../../detection_engine/rule_monitoring'; + import * as i18n from './translations'; export const EXECUTION_LOG_SCHEMA_MAPPING = { @@ -42,22 +35,18 @@ export const replaceQueryTextAliases = (queryText: string): string => { ); }; -const statuses = [ +// This only includes statuses which are or can be final +const STATUS_FILTERS = [ RuleExecutionStatus.succeeded, RuleExecutionStatus.failed, RuleExecutionStatus['partial failure'], ]; -const statusFilters = statuses.map((status) => ({ - label: {capitalize(status)}, - selected: false, -})); - interface ExecutionLogTableSearchProps { onlyShowFilters: true; + selectedStatuses: RuleExecutionStatus[]; + onStatusFilterChange: (selectedStatuses: RuleExecutionStatus[]) => void; onSearch: (queryText: string) => void; - onStatusFilterChange: (statusFilters: RuleExecutionStatus[]) => void; - defaultSelectedStatusFilters?: RuleExecutionStatus[]; } /** @@ -68,47 +57,14 @@ interface ExecutionLogTableSearchProps { * Please see this comment for history/details: https://github.com/elastic/kibana/pull/127339/files#r825240516 */ export const ExecutionLogSearchBar = React.memo( - ({ onlyShowFilters, onSearch, onStatusFilterChange, defaultSelectedStatusFilters = [] }) => { - const [isPopoverOpen, setIsPopoverOpen] = useState(false); - const [selectedFilters, setSelectedFilters] = useState( - defaultSelectedStatusFilters - ); - - const onSearchCallback = useCallback( + ({ onlyShowFilters, selectedStatuses, onStatusFilterChange, onSearch }) => { + const handleSearch = useCallback( (queryText: string) => { onSearch(replaceQueryTextAliases(queryText)); }, [onSearch] ); - const onStatusFilterChangeCallback = useCallback( - (filter: RuleExecutionStatus) => { - setSelectedFilters( - selectedFilters.includes(filter) - ? selectedFilters.filter((f) => f !== filter) - : [...selectedFilters, filter] - ); - }, - [selectedFilters] - ); - - const filtersComponent = useMemo(() => { - return statuses.map((filter, index) => ( - onStatusFilterChangeCallback(filter)} - title={filter} - > - {capitalize(filter)} - - )); - }, [onStatusFilterChangeCallback, selectedFilters]); - - useEffect(() => { - onStatusFilterChange(selectedFilters); - }, [onStatusFilterChange, selectedFilters]); - return ( @@ -117,37 +73,18 @@ export const ExecutionLogSearchBar = React.memo( data-test-subj="executionLogSearch" aria-label={i18n.RULE_EXECUTION_LOG_SEARCH_PLACEHOLDER} placeholder={i18n.RULE_EXECUTION_LOG_SEARCH_PLACEHOLDER} - onSearch={onSearchCallback} + onSearch={handleSearch} isClearable={true} fullWidth={true} /> )} - - setIsPopoverOpen(!isPopoverOpen)} - numFilters={statusFilters.length} - isSelected={isPopoverOpen} - hasActiveFilters={selectedFilters.length > 0} - numActiveFilters={selectedFilters.length} - > - {i18n.COLUMN_STATUS} - - } - isOpen={isPopoverOpen} - closePopover={() => setIsPopoverOpen(false)} - panelPaddingSize="none" - repositionOnScroll - > - {filtersComponent} - - + ); diff --git a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_table.test.tsx b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_table.test.tsx index 608853f004eb9..52f85b228ab36 100644 --- a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_table.test.tsx +++ b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_table.test.tsx @@ -5,20 +5,23 @@ * 2.0. */ -import { ruleExecutionEventsMock } from '../../../../../containers/detection_engine/rules/mock'; +import React from 'react'; +import { noop } from 'lodash/fp'; import { render, screen } from '@testing-library/react'; + import { TestProviders } from '../../../../../../common/mock'; import { useRuleDetailsContextMock } from '../__mocks__/rule_details_context'; -import React from 'react'; -import { noop } from 'lodash/fp'; +import { getRuleExecutionResultsResponseMock } from '../../../../../../../common/detection_engine/rule_monitoring/mocks'; -import { useRuleExecutionEvents } from '../../../../../containers/detection_engine/rules'; +import { useExecutionResults } from '../../../../../../detection_engine/rule_monitoring'; import { useSourcererDataView } from '../../../../../../common/containers/sourcerer'; import { useRuleDetailsContext } from '../rule_details_context'; import { ExecutionLogTable } from './execution_log_table'; jest.mock('../../../../../../common/containers/sourcerer'); -jest.mock('../../../../../containers/detection_engine/rules'); +jest.mock( + '../../../../../../detection_engine/rule_monitoring/components/execution_results_table/use_execution_results' +); jest.mock('../rule_details_context'); const mockUseSourcererDataView = useSourcererDataView as jest.Mock; @@ -30,9 +33,9 @@ mockUseSourcererDataView.mockReturnValue({ loading: false, }); -const mockUseRuleExecutionEvents = useRuleExecutionEvents as jest.Mock; +const mockUseRuleExecutionEvents = useExecutionResults as jest.Mock; mockUseRuleExecutionEvents.mockReturnValue({ - data: ruleExecutionEventsMock, + data: getRuleExecutionResultsResponseMock.getSomeResponse(), isLoading: false, isFetching: false, }); diff --git a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_table.tsx b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_table.tsx index fe0ff1b7b555a..e0b798f76b591 100644 --- a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_table.tsx +++ b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/execution_log_table.tsx @@ -5,32 +5,36 @@ * 2.0. */ +import React, { useCallback, useMemo, useRef } from 'react'; import { useDispatch } from 'react-redux'; import styled from 'styled-components'; import moment from 'moment'; -import React, { useCallback, useMemo, useRef } from 'react'; import type { OnTimeChangeProps, OnRefreshProps, OnRefreshChangeProps } from '@elastic/eui'; import { EuiTextColor, EuiFlexGroup, EuiFlexItem, + EuiPanel, EuiSuperDatePicker, EuiSpacer, EuiSwitch, EuiBasicTable, EuiButton, } from '@elastic/eui'; + import type { Filter, Query } from '@kbn/es-query'; import { buildFilter, FILTERS } from '@kbn/es-query'; import { MAX_EXECUTION_EVENTS_DISPLAYED } from '@kbn/securitysolution-rules'; import { mountReactNode } from '@kbn/core/public/utils'; + import { RuleDetailTabs } from '..'; import { RULE_DETAILS_EXECUTION_LOG_TABLE_SHOW_METRIC_COLUMNS_STORAGE_KEY } from '../../../../../../../common/constants'; import type { - AggregateRuleExecutionEvent, + RuleExecutionResult, RuleExecutionStatus, -} from '../../../../../../../common/detection_engine/schemas/common'; +} from '../../../../../../../common/detection_engine/rule_monitoring'; +import { HeaderSection } from '../../../../../../common/components/header_section'; import { UtilityBar, UtilityBarGroup, @@ -56,7 +60,7 @@ import { isRelativeTimeRange, } from '../../../../../../common/store/inputs/model'; import { SourcererScopeName } from '../../../../../../common/store/sourcerer/model'; -import { useRuleExecutionEvents } from '../../../../../containers/detection_engine/rules'; +import { useExecutionResults } from '../../../../../../detection_engine/rule_monitoring'; import { useRuleDetailsContext } from '../rule_details_context'; import * as i18n from './translations'; import { EXECUTION_LOG_COLUMNS, GET_EXECUTION_LOG_METRICS_COLUMNS } from './execution_log_columns'; @@ -97,7 +101,7 @@ const ExecutionLogTableComponent: React.FC = ({ } = useKibana().services; const { - [RuleDetailTabs.executionLogs]: { + [RuleDetailTabs.executionResults]: { state: { superDatePicker: { recentlyUsedRanges, refreshInterval, isPaused, start, end }, queryText, @@ -181,7 +185,7 @@ const ExecutionLogTableComponent: React.FC = ({ isFetching, isLoading, refetch, - } = useRuleExecutionEvents({ + } = useExecutionResults({ ruleId, start, end, @@ -368,7 +372,7 @@ const ExecutionLogTableComponent: React.FC = ({ description: i18n.COLUMN_ACTIONS_TOOLTIP, icon: 'filter', type: 'icon', - onClick: (executionEvent: AggregateRuleExecutionEvent) => { + onClick: (executionEvent: RuleExecutionResult) => { if (executionEvent?.execution_uuid) { onFilterByExecutionIdCallback( executionEvent.execution_uuid, @@ -393,14 +397,18 @@ const ExecutionLogTableComponent: React.FC = ({ ); return ( - <> + + {/* Filter bar */} + + + @@ -418,7 +426,10 @@ const ExecutionLogTableComponent: React.FC = ({ /> + + + {/* Utility bar */} @@ -460,15 +471,17 @@ const ExecutionLogTableComponent: React.FC = ({ + + {/* Table with items */} - + ); }; diff --git a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/translations.ts b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/translations.ts index b161ae3662e0e..114faa77f871e 100644 --- a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/translations.ts +++ b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/execution_log_table/translations.ts @@ -7,6 +7,20 @@ import { i18n } from '@kbn/i18n'; +export const TABLE_TITLE = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleDetails.ruleExecutionLog.tableTitle', + { + defaultMessage: 'Execution log', + } +); + +export const TABLE_SUBTITLE = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleDetails.ruleExecutionLog.tableSubtitle', + { + defaultMessage: 'A log of rule execution results', + } +); + export const SHOWING_EXECUTIONS = (totalItems: number) => i18n.translate( 'xpack.securitySolution.detectionEngine.ruleDetails.ruleExecutionLog.totalExecutionsLabel', diff --git a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/index.tsx b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/index.tsx index 2c9fc4c2f74d0..051e3db28dd7e 100644 --- a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/index.tsx +++ b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/index.tsx @@ -104,10 +104,14 @@ import { RuleStatusFailedCallOut, ruleStatusI18n, } from '../../../../components/rules/rule_execution_status'; +import { + ExecutionEventsTable, + useRuleExecutionSettings, +} from '../../../../../detection_engine/rule_monitoring'; +import { ExecutionLogTable } from './execution_log_table/execution_log_table'; import * as detectionI18n from '../../translations'; import * as ruleI18n from '../translations'; -import { ExecutionLogTable } from './execution_log_table/execution_log_table'; import { RuleDetailsContextProvider } from './rule_details_context'; import * as i18n from './translations'; import { NeedAdminForUpdateRulesCallOut } from '../../../../components/callouts/need_admin_for_update_callout'; @@ -141,8 +145,9 @@ const StyledMinHeightTabContainer = styled.div` export enum RuleDetailTabs { alerts = 'alerts', - executionLogs = 'executionLogs', exceptions = 'exceptions', + executionResults = 'executionResults', + executionEvents = 'executionEvents', } const ruleDetailTabs = [ @@ -159,10 +164,16 @@ const ruleDetailTabs = [ dataTestSubj: 'exceptionsTab', }, { - id: RuleDetailTabs.executionLogs, - name: i18n.RULE_EXECUTION_LOGS, + id: RuleDetailTabs.executionResults, + name: i18n.EXECUTION_RESULTS_TAB, + disabled: false, + dataTestSubj: 'executionResultsTab', + }, + { + id: RuleDetailTabs.executionEvents, + name: i18n.EXECUTION_EVENTS_TAB, disabled: false, - dataTestSubj: 'executionLogsTab', + dataTestSubj: 'executionEventsTab', }, ]; @@ -345,15 +356,23 @@ const RuleDetailsPageComponent: React.FC = ({ return null; }, [rule, spacesApi]); + const ruleExecutionSettings = useRuleExecutionSettings(); + useEffect(() => { + let visibleTabs = ruleDetailTabs; + let currentTab = RuleDetailTabs.alerts; + if (!hasIndexRead) { - setTabs(ruleDetailTabs.filter(({ id }) => id !== RuleDetailTabs.alerts)); - setRuleDetailTab(RuleDetailTabs.exceptions); - } else { - setTabs(ruleDetailTabs); - setRuleDetailTab(RuleDetailTabs.alerts); + visibleTabs = visibleTabs.filter(({ id }) => id !== RuleDetailTabs.alerts); + currentTab = RuleDetailTabs.exceptions; } - }, [hasIndexRead]); + if (!ruleExecutionSettings.extendedLogging.isEnabled) { + visibleTabs = visibleTabs.filter(({ id }) => id !== RuleDetailTabs.executionEvents); + } + + setTabs(visibleTabs); + setRuleDetailTab(currentTab); + }, [hasIndexRead, ruleExecutionSettings]); const showUpdating = useMemo( () => isLoadingIndexPattern || isAlertsLoading || loading, @@ -467,7 +486,12 @@ const RuleDetailsPageComponent: React.FC = ({ setRuleDetailTab(tab.id)} isSelected={tab.id === ruleDetailTab} - disabled={tab.disabled || (tab.id === RuleDetailTabs.executionLogs && !isExistingRule)} + disabled={ + tab.disabled || + ((tab.id === RuleDetailTabs.executionResults || + tab.id === RuleDetailTabs.executionEvents) && + !isExistingRule) + } key={tab.id} data-test-subj={tab.dataTestSubj} > @@ -850,9 +874,12 @@ const RuleDetailsPageComponent: React.FC = ({ onRuleChange={refreshRule} /> )} - {ruleDetailTab === RuleDetailTabs.executionLogs && ( + {ruleDetailTab === RuleDetailTabs.executionResults && ( )} + {ruleDetailTab === RuleDetailTabs.executionEvents && ( + + )} diff --git a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/rule_details_context.tsx b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/rule_details_context.tsx index f7f6f735069d1..58b34eb4a2bee 100644 --- a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/rule_details_context.tsx +++ b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/rule_details_context.tsx @@ -10,9 +10,9 @@ import type { DurationRange } from '@elastic/eui/src/components/date_picker/type import React, { createContext, useContext, useMemo, useState } from 'react'; import { RULE_DETAILS_EXECUTION_LOG_TABLE_SHOW_METRIC_COLUMNS_STORAGE_KEY } from '../../../../../../common/constants'; import type { - AggregateRuleExecutionEvent, + RuleExecutionResult, RuleExecutionStatus, -} from '../../../../../../common/detection_engine/schemas/common'; +} from '../../../../../../common/detection_engine/rule_monitoring'; import { invariant } from '../../../../../../common/utils/invariant'; import { useKibana } from '../../../../../common/lib/kibana'; import { RuleDetailTabs } from '.'; @@ -63,7 +63,7 @@ export interface ExecutionLogTableState { pageSize: number; }; sort: { - sortField: keyof AggregateRuleExecutionEvent; + sortField: keyof RuleExecutionResult; sortDirection: SortOrder; }; } @@ -101,14 +101,14 @@ export interface ExecutionLogTableActions { setShowMetricColumns: React.Dispatch>; setPageIndex: React.Dispatch>; setPageSize: React.Dispatch>; - setSortField: React.Dispatch>; + setSortField: React.Dispatch>; setSortDirection: React.Dispatch>; } export interface RuleDetailsContextType { // TODO: Add section for RuleDetailTabs.exceptions and store query/pagination/etc. // TODO: Let's discuss how to integration with ExceptionsViewerComponent state mgmt - [RuleDetailTabs.executionLogs]: { + [RuleDetailTabs.executionResults]: { state: ExecutionLogTableState; actions: ExecutionLogTableActions; }; @@ -139,13 +139,13 @@ export const RuleDetailsContextProvider = ({ children }: RuleDetailsContextProvi // Pagination state const [pageIndex, setPageIndex] = useState(1); const [pageSize, setPageSize] = useState(5); - const [sortField, setSortField] = useState('timestamp'); + const [sortField, setSortField] = useState('timestamp'); const [sortDirection, setSortDirection] = useState('desc'); // // End Execution Log Table tab const providerValue = useMemo( () => ({ - [RuleDetailTabs.executionLogs]: { + [RuleDetailTabs.executionResults]: { state: { superDatePicker: { recentlyUsedRanges, diff --git a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/translations.ts b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/translations.ts index e0c6f319f6e91..299c355ffa480 100644 --- a/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/translations.ts +++ b/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/translations.ts @@ -35,17 +35,24 @@ export const UNKNOWN = i18n.translate( } ); -export const RULE_EXECUTION_LOGS = i18n.translate( - 'xpack.securitySolution.detectionEngine.ruleDetails.ruleExecutionLogsTab', +export const EXCEPTIONS_TAB = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleDetails.exceptionsTab', { - defaultMessage: 'Rule execution logs ', + defaultMessage: 'Exceptions', } ); -export const EXCEPTIONS_TAB = i18n.translate( - 'xpack.securitySolution.detectionEngine.ruleDetails.exceptionsTab', +export const EXECUTION_RESULTS_TAB = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleDetails.ruleExecutionResultsTab', { - defaultMessage: 'Exceptions', + defaultMessage: 'Execution results', + } +); + +export const EXECUTION_EVENTS_TAB = i18n.translate( + 'xpack.securitySolution.detectionEngine.ruleDetails.ruleExecutionEventsTab', + { + defaultMessage: 'Execution events', } ); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/__mocks__/request_context.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/__mocks__/request_context.ts index 0f20d04fc5f7c..9a43f02ce5fd0 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/__mocks__/request_context.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/__mocks__/request_context.ts @@ -24,7 +24,7 @@ import { ruleRegistryMocks } from '@kbn/rule-registry-plugin/server/mocks'; import { siemMock } from '../../../../mocks'; import { createMockConfig } from '../../../../config.mock'; -import { ruleExecutionLogMock } from '../../rule_execution_log/__mocks__'; +import { ruleExecutionLogMock } from '../../rule_monitoring/mocks'; import { requestMock } from './request'; import { internalFrameworkRequest } from '../../../framework'; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/__mocks__/request_responses.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/__mocks__/request_responses.ts index c8cfeee596500..bbc40dcf4b8d6 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/__mocks__/request_responses.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/__mocks__/request_responses.ts @@ -20,12 +20,10 @@ import { DETECTION_ENGINE_SIGNALS_FINALIZE_MIGRATION_URL, DETECTION_ENGINE_SIGNALS_MIGRATION_STATUS_URL, DETECTION_ENGINE_RULES_BULK_ACTION, - DETECTION_ENGINE_RULE_EXECUTION_EVENTS_URL, DETECTION_ENGINE_RULES_BULK_UPDATE, DETECTION_ENGINE_RULES_BULK_DELETE, DETECTION_ENGINE_RULES_BULK_CREATE, } from '../../../../../common/constants'; -import type { GetAggregateRuleExecutionEventsResponse } from '../../../../../common/detection_engine/schemas/response'; import type { RuleAlertType, HapiReadableStream } from '../../rules/types'; import { requestMock } from './request'; import type { QuerySignalsSchemaDecoded } from '../../../../../common/detection_engine/schemas/request/query_signals_index_schema'; @@ -40,13 +38,10 @@ import { getPerformBulkActionSchemaMock, getPerformBulkActionEditSchemaMock, } from '../../../../../common/detection_engine/schemas/request/perform_bulk_action_schema.mock'; -import type { RuleExecutionSummary } from '../../../../../common/detection_engine/schemas/common'; -import { RuleExecutionStatus } from '../../../../../common/detection_engine/schemas/common'; // eslint-disable-next-line no-restricted-imports import type { LegacyRuleNotificationAlertType } from '../../notifications/legacy_types'; // eslint-disable-next-line no-restricted-imports import type { LegacyIRuleActionsAttributes } from '../../rule_actions/legacy_types'; -import type { RuleExecutionSummariesByRuleId } from '../../rule_execution_log'; export const typicalSetStatusSignalByIdsPayload = (): SetSignalsStatusSchemaDecoded => ({ signal_ids: ['somefakeid1', 'somefakeid2'], @@ -234,19 +229,6 @@ export const getFindResultWithMultiHits = ({ }; }; -export const getRuleExecutionEventsRequest = () => - requestMock.create({ - method: 'get', - path: DETECTION_ENGINE_RULE_EXECUTION_EVENTS_URL, - params: { - ruleId: '04128c15-0d1b-4716-a4c5-46997ac7f3bd', - }, - query: { - start: '2022-03-31T22:02:01.622Z', - end: '2022-03-31T22:02:31.622Z', - }, - }); - export const getImportRulesRequest = (hapiStream?: HapiReadableStream) => requestMock.create({ method: 'post', @@ -445,96 +427,6 @@ export const getEmptySavedObjectsResponse = (): SavedObjectsFindResponse => ({ saved_objects: [], }); -// TODO: https://github.com/elastic/kibana/pull/121644 clean up -export const getRuleExecutionSummarySucceeded = (): RuleExecutionSummary => ({ - last_execution: { - date: '2020-02-18T15:26:49.783Z', - status: RuleExecutionStatus.succeeded, - status_order: 0, - message: 'succeeded', - metrics: { - total_search_duration_ms: 200, - total_indexing_duration_ms: 800, - execution_gap_duration_s: 500, - }, - }, -}); - -// TODO: https://github.com/elastic/kibana/pull/121644 clean up -export const getRuleExecutionSummaryFailed = (): RuleExecutionSummary => ({ - last_execution: { - date: '2020-02-18T15:15:58.806Z', - status: RuleExecutionStatus.failed, - status_order: 30, - message: - 'Signal rule name: "Query with a rule id Number 1", id: "1ea5a820-4da1-4e82-92a1-2b43a7bece08", rule_id: "query-rule-id-1" has a time gap of 5 days (412682928ms), and could be missing signals within that time. Consider increasing your look behind time or adding more Kibana instances.', - metrics: { - total_search_duration_ms: 200, - total_indexing_duration_ms: 800, - execution_gap_duration_s: 500, - }, - }, -}); - -// TODO: https://github.com/elastic/kibana/pull/121644 clean up -export const getRuleExecutionSummaries = (): RuleExecutionSummariesByRuleId => ({ - '04128c15-0d1b-4716-a4c5-46997ac7f3bd': getRuleExecutionSummarySucceeded(), - '1ea5a820-4da1-4e82-92a1-2b43a7bece08': getRuleExecutionSummaryFailed(), -}); - -export const getAggregateExecutionEvents = (): GetAggregateRuleExecutionEventsResponse => ({ - events: [ - { - execution_uuid: '34bab6e0-89b6-4d10-9cbb-cda76d362db6', - timestamp: '2022-03-11T22:04:05.931Z', - duration_ms: 1975, - status: 'success', - message: - "rule executed: siem.queryRule:f78f3550-a186-11ec-89a1-0bce95157aba: 'This Rule Makes Alerts, Actions, AND Moar!'", - num_active_alerts: 0, - num_new_alerts: 0, - num_recovered_alerts: 0, - num_triggered_actions: 0, - num_succeeded_actions: 0, - num_errored_actions: 0, - total_search_duration_ms: 0, - es_search_duration_ms: 538, - schedule_delay_ms: 2091, - timed_out: false, - indexing_duration_ms: 7, - search_duration_ms: 551, - gap_duration_s: 0, - security_status: 'succeeded', - security_message: 'succeeded', - }, - { - execution_uuid: '254d8400-9dc7-43c5-ad4b-227273d1a44b', - timestamp: '2022-03-11T22:02:41.923Z', - duration_ms: 11916, - status: 'success', - message: - "rule executed: siem.queryRule:f78f3550-a186-11ec-89a1-0bce95157aba: 'This Rule Makes Alerts, Actions, AND Moar!'", - num_active_alerts: 0, - num_new_alerts: 0, - num_recovered_alerts: 0, - num_triggered_actions: 1, - num_succeeded_actions: 1, - num_errored_actions: 0, - total_search_duration_ms: 0, - es_search_duration_ms: 1406, - schedule_delay_ms: 1583, - timed_out: false, - indexing_duration_ms: 0, - search_duration_ms: 0, - gap_duration_s: 0, - security_status: 'partial failure', - security_message: - 'Check privileges failed to execute ResponseError: index_not_found_exception: [index_not_found_exception] Reason: no such index [broken-index] name: "This Rule Makes Alerts, Actions, AND Moar!" id: "f78f3550-a186-11ec-89a1-0bce95157aba" rule id: "b64b4540-d035-4826-a1e7-f505bf4b9653" execution id: "254d8400-9dc7-43c5-ad4b-227273d1a44b" space ID: "default"', - }, - ], - total: 2, -}); - export const getBasicEmptySearchResponse = (): estypes.SearchResponse => ({ took: 1, timed_out: false, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/create_rules_route.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/create_rules_route.test.ts index 658461f148cb8..f3c4a698d83d2 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/create_rules_route.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/create_rules_route.test.ts @@ -10,7 +10,6 @@ import { getEmptyFindResult, getRuleMock, getCreateRequest, - getRuleExecutionSummarySucceeded, getFindResultWithSingleHit, createMlRuleRequest, getBasicEmptySearchResponse, @@ -37,9 +36,6 @@ describe('create_rules', () => { clients.rulesClient.find.mockResolvedValue(getEmptyFindResult()); // no current rules clients.rulesClient.create.mockResolvedValue(getRuleMock(getQueryRuleParams())); // creation succeeds - clients.ruleExecutionLog.getExecutionSummary.mockResolvedValue( - getRuleExecutionSummarySucceeded() - ); context.core.elasticsearch.client.asCurrentUser.search.mockResolvedValue( elasticsearchClientMock.createSuccessTransportRequestPromise(getBasicEmptySearchResponse()) diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/find_rules_route.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/find_rules_route.test.ts index b4aada2a60b69..0e26601d64547 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/find_rules_route.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/find_rules_route.test.ts @@ -14,7 +14,6 @@ import { getFindRequest, getFindResultWithSingleHit, getEmptySavedObjectsResponse, - getRuleExecutionSummaries, } from '../__mocks__/request_responses'; import { findRulesRoute } from './find_rules_route'; @@ -31,9 +30,6 @@ describe('find_rules', () => { clients.rulesClient.find.mockResolvedValue(getFindResultWithSingleHit()); clients.rulesClient.get.mockResolvedValue(getRuleMock(getQueryRuleParams())); clients.savedObjectsClient.find.mockResolvedValue(getEmptySavedObjectsResponse()); - clients.ruleExecutionLog.getExecutionSummariesBulk.mockResolvedValue( - getRuleExecutionSummaries() - ); findRulesRoute(server.router, logger); }); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/get_rule_execution_events_route.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/get_rule_execution_events_route.test.ts deleted file mode 100644 index c59a0e4dfe176..0000000000000 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/get_rule_execution_events_route.test.ts +++ /dev/null @@ -1,57 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -import { serverMock, requestContextMock } from '../__mocks__'; -import { - getRuleExecutionEventsRequest, - getAggregateExecutionEvents, -} from '../__mocks__/request_responses'; -import { getRuleExecutionEventsRoute } from './get_rule_execution_events_route'; - -describe('getRuleExecutionEventsRoute', () => { - let server: ReturnType; - let { clients, context } = requestContextMock.createTools(); - - beforeEach(async () => { - server = serverMock.create(); - ({ clients, context } = requestContextMock.createTools()); - - getRuleExecutionEventsRoute(server.router); - }); - - describe('when it finds events in rule execution log', () => { - it('returns 200 response with the events', async () => { - const executionEvents = getAggregateExecutionEvents(); - clients.ruleExecutionLog.getAggregateExecutionEvents.mockResolvedValue(executionEvents); - - const response = await server.inject( - getRuleExecutionEventsRequest(), - requestContextMock.convertContext(context) - ); - - expect(response.status).toEqual(200); - expect(response.body).toEqual(executionEvents); - }); - }); - - describe('when rule execution log client throws an error', () => { - it('returns 500 response with it', async () => { - clients.ruleExecutionLog.getAggregateExecutionEvents.mockRejectedValue(new Error('Boom!')); - - const response = await server.inject( - getRuleExecutionEventsRequest(), - requestContextMock.convertContext(context) - ); - - expect(response.status).toEqual(500); - expect(response.body).toEqual({ - message: 'Boom!', - status_code: 500, - }); - }); - }); -}); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/patch_rules_route.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/patch_rules_route.test.ts index 40a903ba1b61b..743fdefa7947f 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/patch_rules_route.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/patch_rules_route.test.ts @@ -10,7 +10,6 @@ import { mlServicesMock, mlAuthzMock as mockMlAuthzFactory } from '../../../mach import { buildMlAuthz } from '../../../machine_learning/authz'; import { getEmptyFindResult, - getRuleExecutionSummarySucceeded, getRuleMock, getPatchRequest, getFindResultWithSingleHit, @@ -46,9 +45,6 @@ describe('patch_rules', () => { clients.rulesClient.get.mockResolvedValue(getRuleMock(getQueryRuleParams())); // existing rule clients.rulesClient.find.mockResolvedValue(getFindResultWithSingleHit()); // existing rule clients.rulesClient.update.mockResolvedValue(getRuleMock(getQueryRuleParams())); // successful update - clients.ruleExecutionLog.getExecutionSummary.mockResolvedValue( - getRuleExecutionSummarySucceeded() - ); (legacyMigrate as jest.Mock).mockResolvedValue(getRuleMock(getQueryRuleParams())); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/preview_rules_route.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/preview_rules_route.ts index e528db104a486..2a0615b17c1ac 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/preview_rules_route.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/preview_rules_route.ts @@ -36,7 +36,8 @@ import { import { wrapScopedClusterClient } from './utils/wrap_scoped_cluster_client'; import type { RulePreviewLogs } from '../../../../../common/detection_engine/schemas/request'; import { previewRulesSchema } from '../../../../../common/detection_engine/schemas/request'; -import { RuleExecutionStatus } from '../../../../../common/detection_engine/schemas/common'; +import { RuleExecutionStatus } from '../../../../../common/detection_engine/rule_monitoring'; +import type { RuleExecutionContext, StatusChangeArgs } from '../../rule_monitoring'; import type { ConfigType } from '../../../../config'; import { alertInstanceFactoryStub } from '../../signals/preview/alert_instance_factory_stub'; @@ -52,7 +53,6 @@ import { } from '../../rule_types'; import { createSecurityRuleTypeWrapper } from '../../rule_types/create_security_rule_type_wrapper'; import { RULE_PREVIEW_INVOCATION_COUNT } from '../../../../../common/detection_engine/constants'; -import type { RuleExecutionContext, StatusChangeArgs } from '../../rule_execution_log'; import { assertUnreachable } from '../../../../../common/utility_types'; import { wrapSearchSourceClient } from './utils/wrap_search_source_client'; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/read_rules_route.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/read_rules_route.test.ts index 44c2c3a075508..368a2b50cd962 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/read_rules_route.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/read_rules_route.test.ts @@ -16,7 +16,6 @@ import { getFindResultWithSingleHit, nonRuleFindResult, getEmptySavedObjectsResponse, - getRuleExecutionSummarySucceeded, resolveRuleMock, } from '../__mocks__/request_responses'; import { requestMock, requestContextMock, serverMock } from '../__mocks__'; @@ -35,9 +34,6 @@ describe('read_rules', () => { clients.rulesClient.find.mockResolvedValue(getFindResultWithSingleHit()); // rule exists clients.savedObjectsClient.find.mockResolvedValue(getEmptySavedObjectsResponse()); // successful transform - clients.ruleExecutionLog.getExecutionSummary.mockResolvedValue( - getRuleExecutionSummarySucceeded() - ); clients.rulesClient.resolve.mockResolvedValue({ ...resolveRuleMock({ diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/update_rules_route.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/update_rules_route.test.ts index 7f2d5c3bde7e0..ad7a7d1fe5365 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/update_rules_route.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/update_rules_route.test.ts @@ -12,7 +12,6 @@ import { getRuleMock, getUpdateRequest, getFindResultWithSingleHit, - getRuleExecutionSummarySucceeded, nonRuleFindResult, typicalMlRulePayload, } from '../__mocks__/request_responses'; @@ -46,9 +45,6 @@ describe('update_rules', () => { clients.rulesClient.get.mockResolvedValue(getRuleMock(getQueryRuleParams())); // existing rule clients.rulesClient.find.mockResolvedValue(getFindResultWithSingleHit()); // rule exists clients.rulesClient.update.mockResolvedValue(getRuleMock(getQueryRuleParams())); // successful update - clients.ruleExecutionLog.getExecutionSummary.mockResolvedValue( - getRuleExecutionSummarySucceeded() - ); clients.appClient.getSignalsIndex.mockReturnValue('.siem-signals-test-index'); (legacyMigrate as jest.Mock).mockResolvedValue(getRuleMock(getQueryRuleParams())); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/utils.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/utils.ts index 857b0df46ba24..2df4cb712ddd2 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/utils.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/utils.ts @@ -13,7 +13,7 @@ import pMap from 'p-map'; import type { PartialRule, FindResult } from '@kbn/alerting-plugin/server'; import type { ActionsClient, FindActionResult } from '@kbn/actions-plugin/server'; -import type { RuleExecutionSummary } from '../../../../../common/detection_engine/schemas/common'; +import type { RuleExecutionSummary } from '../../../../../common/detection_engine/rule_monitoring'; import type { RulesSchema } from '../../../../../common/detection_engine/schemas/response/rules_schema'; import type { ImportRulesSchema } from '../../../../../common/detection_engine/schemas/request/import_rules_schema'; import type { CreateRulesBulkSchema } from '../../../../../common/detection_engine/schemas/request/create_rules_bulk_schema'; @@ -25,7 +25,7 @@ import { internalRuleToAPIResponse } from '../../schemas/rule_converters'; import type { RuleParams } from '../../schemas/rule_schemas'; // eslint-disable-next-line no-restricted-imports import type { LegacyRulesActionsSavedObject } from '../../rule_actions/legacy_get_rule_actions_saved_object'; -import type { RuleExecutionSummariesByRuleId } from '../../rule_execution_log'; +import type { RuleExecutionSummariesByRuleId } from '../../rule_monitoring'; type PromiseFromStreams = ImportRulesSchema | Error; const MAX_CONCURRENT_SEARCHES = 10; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/validate.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/validate.test.ts index fb038ebabe08e..21db7e52e4f8d 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/validate.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/validate.test.ts @@ -8,7 +8,8 @@ import { transformValidate, transformValidateBulkError } from './validate'; import type { BulkError } from '../utils'; import type { RulesSchema } from '../../../../../common/detection_engine/schemas/response'; -import { getRuleMock, getRuleExecutionSummarySucceeded } from '../__mocks__/request_responses'; +import { getRuleMock } from '../__mocks__/request_responses'; +import { ruleExecutionSummaryMock } from '../../../../../common/detection_engine/rule_monitoring/mocks'; import { getListArrayMock } from '../../../../../common/detection_engine/schemas/types/lists.mock'; import { getThreatMock } from '../../../../../common/detection_engine/schemas/types/threat.mock'; import { getQueryRuleParams } from '../../schemas/rule_schemas.mock'; @@ -111,7 +112,7 @@ describe('validate', () => { test('it should do a validation correctly of a rule id with rule execution summary passed in', () => { const rule = getRuleMock(getQueryRuleParams()); - const ruleExecutionSumary = getRuleExecutionSummarySucceeded(); + const ruleExecutionSumary = ruleExecutionSummaryMock.getSummarySucceeded(); const validatedOrError = transformValidateBulkError('rule-1', rule, ruleExecutionSumary); const expected: RulesSchema = { ...ruleOutput(), diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/validate.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/validate.ts index cdecd3abf5960..4183f217a61fe 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/validate.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/validate.ts @@ -8,7 +8,7 @@ import { validateNonExact } from '@kbn/securitysolution-io-ts-utils'; import type { PartialRule } from '@kbn/alerting-plugin/server'; -import type { RuleExecutionSummary } from '../../../../../common/detection_engine/schemas/common'; +import type { RuleExecutionSummary } from '../../../../../common/detection_engine/rule_monitoring'; import type { FullResponseSchema } from '../../../../../common/detection_engine/schemas/request'; import { fullResponseSchema } from '../../../../../common/detection_engine/schemas/request'; import type { RulesSchema } from '../../../../../common/detection_engine/schemas/response/rules_schema'; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/__mocks__/index.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/__mocks__/index.ts deleted file mode 100644 index 666b35ed93f56..0000000000000 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/__mocks__/index.ts +++ /dev/null @@ -1,43 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -import type { IRuleExecutionLogForRoutes } from '../client_for_routes/client_interface'; -import type { - IRuleExecutionLogForExecutors, - RuleExecutionContext, -} from '../client_for_executors/client_interface'; - -const ruleExecutionLogForRoutesMock = { - create: (): jest.Mocked => ({ - getAggregateExecutionEvents: jest.fn(), - getExecutionSummariesBulk: jest.fn(), - getExecutionSummary: jest.fn(), - clearExecutionSummary: jest.fn(), - getLastFailures: jest.fn(), - }), -}; - -const ruleExecutionLogForExecutorsMock = { - create: ( - context: Partial = {} - ): jest.Mocked => ({ - context: { - executionId: context.executionId ?? 'some execution id', - ruleId: context.ruleId ?? 'some rule id', - ruleName: context.ruleName ?? 'Some rule', - ruleType: context.ruleType ?? 'some rule type', - spaceId: context.spaceId ?? 'some space id', - }, - - logStatusChange: jest.fn(), - }), -}; - -export const ruleExecutionLogMock = { - forRoutes: ruleExecutionLogForRoutesMock, - forExecutors: ruleExecutionLogForExecutorsMock, -}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/client_factories.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/client_factories.ts deleted file mode 100644 index 330010acfbcc4..0000000000000 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/client_factories.ts +++ /dev/null @@ -1,56 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -import type { Logger, SavedObjectsClientContract } from '@kbn/core/server'; -import type { IEventLogClient, IEventLogService } from '@kbn/event-log-plugin/server'; - -import type { IRuleExecutionLogForRoutes } from './client_for_routes/client_interface'; -import { createClientForRoutes } from './client_for_routes/client'; - -import type { - IRuleExecutionLogForExecutors, - RuleExecutionContext, -} from './client_for_executors/client_interface'; -import { createClientForExecutors } from './client_for_executors/client'; - -import { createEventLogReader } from './event_log/event_log_reader'; -import { createEventLogWriter } from './event_log/event_log_writer'; -import { createRuleExecutionSavedObjectsClient } from './execution_saved_object/saved_objects_client'; - -export type RuleExecutionLogForRoutesFactory = ( - savedObjectsClient: SavedObjectsClientContract, - eventLogClient: IEventLogClient, - logger: Logger -) => IRuleExecutionLogForRoutes; - -export const ruleExecutionLogForRoutesFactory: RuleExecutionLogForRoutesFactory = ( - savedObjectsClient, - eventLogClient, - logger -) => { - const soClient = createRuleExecutionSavedObjectsClient(savedObjectsClient, logger); - const eventLogReader = createEventLogReader(eventLogClient); - return createClientForRoutes(soClient, eventLogReader, logger); -}; - -export type RuleExecutionLogForExecutorsFactory = ( - savedObjectsClient: SavedObjectsClientContract, - eventLogService: IEventLogService, - logger: Logger, - context: RuleExecutionContext -) => IRuleExecutionLogForExecutors; - -export const ruleExecutionLogForExecutorsFactory: RuleExecutionLogForExecutorsFactory = ( - savedObjectsClient, - eventLogService, - logger, - context -) => { - const soClient = createRuleExecutionSavedObjectsClient(savedObjectsClient, logger); - const eventLogWriter = createEventLogWriter(eventLogService); - return createClientForExecutors(soClient, eventLogWriter, logger, context); -}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/client_for_executors/client.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/client_for_executors/client.ts deleted file mode 100644 index 46a16d567de7f..0000000000000 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/client_for_executors/client.ts +++ /dev/null @@ -1,156 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -import { sum } from 'lodash'; -import type { Duration } from 'moment'; -import type { Logger } from '@kbn/core/server'; - -import type { - RuleExecutionStatus, - RuleExecutionMetrics, -} from '../../../../../common/detection_engine/schemas/common'; -import { ruleExecutionStatusOrderByStatus } from '../../../../../common/detection_engine/schemas/common'; - -import { withSecuritySpan } from '../../../../utils/with_security_span'; -import type { ExtMeta } from '../utils/console_logging'; -import { truncateValue } from '../utils/normalization'; - -import type { IEventLogWriter } from '../event_log/event_log_writer'; -import type { IRuleExecutionSavedObjectsClient } from '../execution_saved_object/saved_objects_client'; -import type { - IRuleExecutionLogForExecutors, - RuleExecutionContext, - StatusChangeArgs, -} from './client_interface'; - -export const createClientForExecutors = ( - soClient: IRuleExecutionSavedObjectsClient, - eventLog: IEventLogWriter, - logger: Logger, - context: RuleExecutionContext -): IRuleExecutionLogForExecutors => { - const { executionId, ruleId, ruleName, ruleType, spaceId } = context; - - const client: IRuleExecutionLogForExecutors = { - get context() { - return context; - }, - - async logStatusChange(args) { - await withSecuritySpan('IRuleExecutionLogForExecutors.logStatusChange', async () => { - try { - const normalizedArgs = normalizeStatusChangeArgs(args); - await Promise.all([ - writeStatusChangeToSavedObjects(normalizedArgs), - writeStatusChangeToEventLog(normalizedArgs), - ]); - } catch (e) { - const logMessage = 'Error logging rule execution status change'; - const logAttributes = `status: "${args.newStatus}", rule id: "${ruleId}", rule name: "${ruleName}", execution uuid: "${executionId}"`; - const logReason = e instanceof Error ? e.stack ?? e.message : String(e); - const logMeta: ExtMeta = { - rule: { - id: ruleId, - name: ruleName, - type: ruleType, - execution: { - status: args.newStatus, - uuid: executionId, - }, - }, - kibana: { - spaceId, - }, - }; - - logger.error(`${logMessage}; ${logAttributes}; ${logReason}`, logMeta); - } - }); - }, - }; - - // TODO: Add executionId to new status SO? - const writeStatusChangeToSavedObjects = async ( - args: NormalizedStatusChangeArgs - ): Promise => { - const { newStatus, message, metrics } = args; - - await soClient.createOrUpdate(ruleId, { - last_execution: { - date: nowISO(), - status: newStatus, - status_order: ruleExecutionStatusOrderByStatus[newStatus], - message, - metrics: metrics ?? {}, - }, - }); - }; - - const writeStatusChangeToEventLog = (args: NormalizedStatusChangeArgs): void => { - const { newStatus, message, metrics } = args; - - if (metrics) { - eventLog.logExecutionMetrics({ - executionId, - ruleId, - ruleName, - ruleType, - spaceId, - metrics, - }); - } - - eventLog.logStatusChange({ - executionId, - ruleId, - ruleName, - ruleType, - spaceId, - newStatus, - message, - }); - }; - - return client; -}; - -const nowISO = () => new Date().toISOString(); - -interface NormalizedStatusChangeArgs { - newStatus: RuleExecutionStatus; - message: string; - metrics?: RuleExecutionMetrics; -} - -const normalizeStatusChangeArgs = (args: StatusChangeArgs): NormalizedStatusChangeArgs => { - const { newStatus, message, metrics } = args; - - return { - newStatus, - message: truncateValue(message) ?? '', - metrics: metrics - ? { - total_search_duration_ms: normalizeDurations(metrics.searchDurations), - total_indexing_duration_ms: normalizeDurations(metrics.indexingDurations), - execution_gap_duration_s: normalizeGap(metrics.executionGap), - } - : undefined, - }; -}; - -const normalizeDurations = (durations?: string[]): number | undefined => { - if (durations == null) { - return undefined; - } - - const sumAsFloat = sum(durations.map(Number)); - return Math.round(sumAsFloat); -}; - -const normalizeGap = (duration?: Duration): number | undefined => { - return duration ? Math.round(duration.asSeconds()) : undefined; -}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/client_for_executors/client_interface.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/client_for_executors/client_interface.ts deleted file mode 100644 index eca9c0d82c870..0000000000000 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/client_for_executors/client_interface.ts +++ /dev/null @@ -1,49 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -import type { Duration } from 'moment'; -import type { RuleExecutionStatus } from '../../../../../common/detection_engine/schemas/common'; - -/** - * Used from rule executors to log various information about the rule execution: - * - rule status changes - * - rule execution metrics - * - later - generic messages and any kind of info we'd need to log for rule - * monitoring or debugging purposes - */ -export interface IRuleExecutionLogForExecutors { - context: RuleExecutionContext; - - /** - * Writes information about new rule statuses and measured execution metrics: - * 1. To .kibana-* index as a custom `siem-detection-engine-rule-execution-info` saved object. - * This SO is used for fast access to last execution info of a large amount of rules. - * 2. To .kibana-event-log-* index in order to track history of rule executions. - * @param args Information about the status change event. - */ - logStatusChange(args: StatusChangeArgs): Promise; -} - -export interface RuleExecutionContext { - executionId: string; - ruleId: string; - ruleName: string; - ruleType: string; - spaceId: string; -} - -export interface StatusChangeArgs { - newStatus: RuleExecutionStatus; - message?: string; - metrics?: MetricsArgs; -} - -export interface MetricsArgs { - searchDurations?: string[]; - indexingDurations?: string[]; - executionGap?: Duration; -} diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/client_for_routes/client_interface.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/client_for_routes/client_interface.ts deleted file mode 100644 index 0df4edc8ecdf3..0000000000000 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/client_for_routes/client_interface.ts +++ /dev/null @@ -1,86 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -import type * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey'; -import type { - ExecutionLogTableSortColumns, - RuleExecutionEvent, - RuleExecutionStatus, - RuleExecutionSummary, -} from '../../../../../common/detection_engine/schemas/common'; -import type { GetAggregateRuleExecutionEventsResponse } from '../../../../../common/detection_engine/schemas/response'; - -export interface GetAggregateExecutionEventsArgs { - ruleId: string; - start: string; - end: string; - queryText: string; - statusFilters: RuleExecutionStatus[]; - page: number; - perPage: number; - sortField: ExecutionLogTableSortColumns; - sortOrder: estypes.SortOrder; -} - -/** - * Used from route handlers to fetch and manage various information about the rule execution: - * - execution summary of a rule containing such data as the last status and metrics - * - execution events such as recent failures and status changes - */ -export interface IRuleExecutionLogForRoutes { - /** - * Fetches list of execution events aggregated by executionId, combining data from both alerting - * and security-solution event-log documents - * @param ruleId Saved object id of the rule (`rule.id`). - * @param start start of daterange to filter to - * @param end end of daterange to filter to - * @param queryText string of field-based filters, e.g. kibana.alert.rule.execution.status:* - * @param statusFilters array of status filters, e.g. ['succeeded', 'going to run'] - * @param page current page to fetch - * @param perPage number of results to fetch per page - * @param sortField field to sort by - * @param sortOrder what order to sort by (e.g. `asc` or `desc`) - */ - getAggregateExecutionEvents({ - ruleId, - start, - end, - queryText, - statusFilters, - page, - perPage, - sortField, - sortOrder, - }: GetAggregateExecutionEventsArgs): Promise; - - /** - * Fetches a list of current execution summaries of multiple rules. - * @param ruleIds A list of saved object ids of multiple rules (`rule.id`). - */ - getExecutionSummariesBulk(ruleIds: string[]): Promise; - - /** - * Fetches current execution summary of a given rule. - * @param ruleId Saved object id of the rule (`rule.id`). - */ - getExecutionSummary(ruleId: string): Promise; - - /** - * Deletes the current execution summary if it exists. - * @param ruleId Saved object id of the rule (`rule.id`). - */ - clearExecutionSummary(ruleId: string): Promise; - - /** - * Fetches last 5 failures (`RuleExecutionStatus.failed`) of a given rule. - * @param ruleId Saved object id of the rule (`rule.id`). - * @deprecated Will be replaced with a more flexible method for fetching execution events. - */ - getLastFailures(ruleId: string): Promise; -} - -export type RuleExecutionSummariesByRuleId = Record; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/event_log_reader.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/event_log_reader.ts deleted file mode 100644 index f8cc46e9419f9..0000000000000 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/event_log_reader.ts +++ /dev/null @@ -1,178 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -import type * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey'; -import type { IEventLogClient } from '@kbn/event-log-plugin/server'; -import { MAX_EXECUTION_EVENTS_DISPLAYED } from '@kbn/securitysolution-rules'; - -import type { - RuleExecutionEvent, - RuleExecutionStatus, -} from '../../../../../common/detection_engine/schemas/common'; -import type { GetAggregateRuleExecutionEventsResponse } from '../../../../../common/detection_engine/schemas/response'; -import { invariant } from '../../../../../common/utils/invariant'; -import { withSecuritySpan } from '../../../../utils/with_security_span'; -import type { GetAggregateExecutionEventsArgs } from '../client_for_routes/client_interface'; -import { - RULE_EXECUTION_LOG_PROVIDER, - RULE_SAVED_OBJECT_TYPE, - RuleExecutionLogAction, -} from './constants'; -import { - formatExecutionEventResponse, - getExecutionEventAggregation, - mapRuleExecutionStatusToPlatformStatus, -} from './get_execution_event_aggregation'; -import type { ExecutionUuidAggResult } from './get_execution_event_aggregation/types'; -import { EXECUTION_UUID_FIELD } from './get_execution_event_aggregation/types'; - -export interface IEventLogReader { - getAggregateExecutionEvents( - args: GetAggregateExecutionEventsArgs - ): Promise; - - getLastStatusChanges(args: GetLastStatusChangesArgs): Promise; -} - -export interface GetLastStatusChangesArgs { - ruleId: string; - count: number; - includeStatuses?: RuleExecutionStatus[]; -} - -export const createEventLogReader = (eventLog: IEventLogClient): IEventLogReader => { - return { - async getAggregateExecutionEvents( - args: GetAggregateExecutionEventsArgs - ): Promise { - const { ruleId, start, end, statusFilters, page, perPage, sortField, sortOrder } = args; - const soType = RULE_SAVED_OBJECT_TYPE; - const soIds = [ruleId]; - - // Current workaround to support root level filters without missing fields in the aggregate event - // or including events from statuses that aren't selected - // TODO: See: https://github.com/elastic/kibana/pull/127339/files#r825240516 - // First fetch execution uuid's by status filter if provided - let statusIds: string[] = []; - let totalExecutions: number | undefined; - // If 0 or 3 statuses are selected we can search for all statuses and don't need this pre-filter by ID - if (statusFilters.length > 0 && statusFilters.length < 3) { - const outcomes = mapRuleExecutionStatusToPlatformStatus(statusFilters); - const outcomeFilter = outcomes.length ? `OR event.outcome:(${outcomes.join(' OR ')})` : ''; - const statusResults = await eventLog.aggregateEventsBySavedObjectIds(soType, soIds, { - start, - end, - // Also query for `event.outcome` to catch executions that only contain platform events - filter: `kibana.alert.rule.execution.status:(${statusFilters.join( - ' OR ' - )}) ${outcomeFilter}`, - aggs: { - totalExecutions: { - cardinality: { - field: EXECUTION_UUID_FIELD, - }, - }, - filteredExecutionUUIDs: { - terms: { - field: EXECUTION_UUID_FIELD, - order: { executeStartTime: 'desc' }, - size: MAX_EXECUTION_EVENTS_DISPLAYED, - }, - aggs: { - executeStartTime: { - min: { - field: '@timestamp', - }, - }, - }, - }, - }, - }); - const filteredExecutionUUIDs = statusResults.aggregations - ?.filteredExecutionUUIDs as ExecutionUuidAggResult; - statusIds = filteredExecutionUUIDs?.buckets?.map((b) => b.key) ?? []; - totalExecutions = ( - statusResults.aggregations?.totalExecutions as estypes.AggregationsCardinalityAggregate - ).value; - // Early return if no results based on status filter - if (statusIds.length === 0) { - return { - total: 0, - events: [], - }; - } - } - - // Now query for aggregate events, and pass any ID's as filters as determined from the above status/queryText results - const idsFilter = statusIds.length - ? `kibana.alert.rule.execution.uuid:(${statusIds.join(' OR ')})` - : ''; - const results = await eventLog.aggregateEventsBySavedObjectIds(soType, soIds, { - start, - end, - filter: idsFilter, - aggs: getExecutionEventAggregation({ - maxExecutions: MAX_EXECUTION_EVENTS_DISPLAYED, - page, - perPage, - sort: [{ [sortField]: { order: sortOrder } }], - }), - }); - - return formatExecutionEventResponse(results, totalExecutions); - }, - async getLastStatusChanges(args) { - const soType = RULE_SAVED_OBJECT_TYPE; - const soIds = [args.ruleId]; - const count = args.count; - const includeStatuses = (args.includeStatuses ?? []).map((status) => `"${status}"`); - - const filterBy: string[] = [ - `event.provider: ${RULE_EXECUTION_LOG_PROVIDER}`, - 'event.kind: event', - `event.action: ${RuleExecutionLogAction['status-change']}`, - includeStatuses.length > 0 - ? `kibana.alert.rule.execution.status:${includeStatuses.join(' ')}` - : '', - ]; - - const kqlFilter = filterBy - .filter(Boolean) - .map((item) => `(${item})`) - .join(' and '); - - const findResult = await withSecuritySpan('findEventsBySavedObjectIds', () => { - return eventLog.findEventsBySavedObjectIds(soType, soIds, { - page: 1, - per_page: count, - sort: [{ sort_field: '@timestamp', sort_order: 'desc' }], - filter: kqlFilter, - }); - }); - - return findResult.data.map((event) => { - invariant(event, 'Event not found'); - invariant(event['@timestamp'], 'Required "@timestamp" field is not found'); - invariant( - event.kibana?.alert?.rule?.execution?.status, - 'Required "kibana.alert.rule.execution.status" field is not found' - ); - - const date = event['@timestamp']; - const status = event.kibana?.alert?.rule?.execution?.status as RuleExecutionStatus; - const message = event.message ?? ''; - const result: RuleExecutionEvent = { - date, - status, - message, - }; - - return result; - }); - }, - }; -}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/event_log_writer.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/event_log_writer.ts deleted file mode 100644 index be212cd80bd14..0000000000000 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/event_log_writer.ts +++ /dev/null @@ -1,128 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -import { SavedObjectsUtils } from '@kbn/core/server'; -import type { IEventLogService } from '@kbn/event-log-plugin/server'; -import { SAVED_OBJECT_REL_PRIMARY } from '@kbn/event-log-plugin/server'; -import type { - RuleExecutionStatus, - RuleExecutionMetrics, -} from '../../../../../common/detection_engine/schemas/common'; -import { ruleExecutionStatusOrderByStatus } from '../../../../../common/detection_engine/schemas/common'; -import { - RULE_SAVED_OBJECT_TYPE, - RULE_EXECUTION_LOG_PROVIDER, - RuleExecutionLogAction, -} from './constants'; - -export interface IEventLogWriter { - logStatusChange(args: StatusChangeArgs): void; - logExecutionMetrics(args: ExecutionMetricsArgs): void; -} - -export interface BaseArgs { - executionId: string; - ruleId: string; - ruleName: string; - ruleType: string; - spaceId: string; -} - -export interface StatusChangeArgs extends BaseArgs { - newStatus: RuleExecutionStatus; - message?: string; -} - -export interface ExecutionMetricsArgs extends BaseArgs { - metrics: RuleExecutionMetrics; -} - -export const createEventLogWriter = (eventLogService: IEventLogService): IEventLogWriter => { - const eventLogger = eventLogService.getLogger({ - event: { provider: RULE_EXECUTION_LOG_PROVIDER }, - }); - - let sequence = 0; - - return { - logStatusChange({ executionId, ruleId, ruleName, ruleType, spaceId, newStatus, message }) { - eventLogger.logEvent({ - '@timestamp': nowISO(), - message, - rule: { - id: ruleId, - name: ruleName, - category: ruleType, - }, - event: { - kind: 'event', - action: RuleExecutionLogAction['status-change'], - sequence: sequence++, - }, - kibana: { - alert: { - rule: { - execution: { - status: newStatus, - status_order: ruleExecutionStatusOrderByStatus[newStatus], - uuid: executionId, - }, - }, - }, - space_ids: [spaceId], - saved_objects: [ - { - rel: SAVED_OBJECT_REL_PRIMARY, - type: RULE_SAVED_OBJECT_TYPE, - id: ruleId, - namespace: spaceIdToNamespace(spaceId), - }, - ], - }, - }); - }, - - logExecutionMetrics({ executionId, ruleId, ruleName, ruleType, spaceId, metrics }) { - eventLogger.logEvent({ - '@timestamp': nowISO(), - rule: { - id: ruleId, - name: ruleName, - category: ruleType, - }, - event: { - kind: 'metric', - action: RuleExecutionLogAction['execution-metrics'], - sequence: sequence++, - }, - kibana: { - alert: { - rule: { - execution: { - metrics, - uuid: executionId, - }, - }, - }, - space_ids: [spaceId], - saved_objects: [ - { - rel: SAVED_OBJECT_REL_PRIMARY, - type: RULE_SAVED_OBJECT_TYPE, - id: ruleId, - namespace: spaceIdToNamespace(spaceId), - }, - ], - }, - }); - }, - }; -}; - -const nowISO = () => new Date().toISOString(); - -const spaceIdToNamespace = SavedObjectsUtils.namespaceStringToId; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/utils/console_logging.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/utils/console_logging.ts deleted file mode 100644 index 89f776a06dd11..0000000000000 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/utils/console_logging.ts +++ /dev/null @@ -1,24 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -import type { LogMeta } from '@kbn/core/server'; -import type { RuleExecutionStatus } from '../../../../../common/detection_engine/schemas/common'; - -/** - * Custom extended log metadata that rule execution logger can attach to every log record. - */ -export type ExtMeta = LogMeta & { - rule?: LogMeta['rule'] & { - type?: string; - execution?: { - status?: RuleExecutionStatus; - }; - }; - kibana?: { - spaceId?: string; - }; -}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/get_rule_execution_events/route.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/get_rule_execution_events/route.test.ts new file mode 100644 index 0000000000000..519be6d429e9d --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/get_rule_execution_events/route.test.ts @@ -0,0 +1,94 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { serverMock, requestContextMock, requestMock } from '../../../routes/__mocks__'; + +import { + GET_RULE_EXECUTION_EVENTS_URL, + LogLevel, + RuleExecutionEventType, +} from '../../../../../../common/detection_engine/rule_monitoring'; +import { getRuleExecutionEventsResponseMock } from '../../../../../../common/detection_engine/rule_monitoring/mocks'; +import type { GetExecutionEventsArgs } from '../../logic/rule_execution_log'; +import { getRuleExecutionEventsRoute } from './route'; + +describe('getRuleExecutionEventsRoute', () => { + let server: ReturnType; + let { clients, context } = requestContextMock.createTools(); + + beforeEach(async () => { + server = serverMock.create(); + ({ clients, context } = requestContextMock.createTools()); + + getRuleExecutionEventsRoute(server.router); + }); + + const getRuleExecutionEventsRequest = () => + requestMock.create({ + method: 'get', + path: GET_RULE_EXECUTION_EVENTS_URL, + params: { + ruleId: '04128c15-0d1b-4716-a4c5-46997ac7f3bd', + }, + query: { + event_types: `${RuleExecutionEventType['status-change']}`, + log_levels: `${LogLevel.debug},${LogLevel.info}`, + page: 3, + }, + }); + + it('passes request arguments to rule execution log', async () => { + const expectedArgs: GetExecutionEventsArgs = { + ruleId: '04128c15-0d1b-4716-a4c5-46997ac7f3bd', + eventTypes: [RuleExecutionEventType['status-change']], + logLevels: [LogLevel.debug, LogLevel.info], + sortOrder: 'desc', + page: 3, + perPage: 20, + }; + + await server.inject( + getRuleExecutionEventsRequest(), + requestContextMock.convertContext(context) + ); + + expect(clients.ruleExecutionLog.getExecutionEvents).toHaveBeenCalledTimes(1); + expect(clients.ruleExecutionLog.getExecutionEvents).toHaveBeenCalledWith(expectedArgs); + }); + + describe('when it finds events in rule execution log', () => { + it('returns 200 response with the events', async () => { + const events = getRuleExecutionEventsResponseMock.getSomeResponse(); + clients.ruleExecutionLog.getExecutionEvents.mockResolvedValue(events); + + const response = await server.inject( + getRuleExecutionEventsRequest(), + requestContextMock.convertContext(context) + ); + + expect(response.status).toEqual(200); + expect(response.body).toEqual(events); + }); + }); + + describe('when rule execution log client throws an error', () => { + it('returns 500 response with it', async () => { + clients.ruleExecutionLog.getExecutionEvents.mockRejectedValue(new Error('Boom!')); + + const response = await server.inject( + getRuleExecutionEventsRequest(), + requestContextMock.convertContext(context) + ); + + expect(response.status).toEqual(500); + expect(response.body).toEqual({ + message: 'Boom!', + status_code: 500, + }); + }); + }); +}); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/get_rule_execution_events/route.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/get_rule_execution_events/route.ts new file mode 100644 index 0000000000000..109c15409afae --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/get_rule_execution_events/route.ts @@ -0,0 +1,64 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { transformError } from '@kbn/securitysolution-es-utils'; +import { buildRouteValidation } from '../../../../../utils/build_validation/route_validation'; +import { buildSiemResponse } from '../../../routes/utils'; +import type { SecuritySolutionPluginRouter } from '../../../../../types'; + +import type { GetRuleExecutionEventsResponse } from '../../../../../../common/detection_engine/rule_monitoring'; +import { + GET_RULE_EXECUTION_EVENTS_URL, + GetRuleExecutionEventsRequestParams, + GetRuleExecutionEventsRequestQuery, +} from '../../../../../../common/detection_engine/rule_monitoring'; + +/** + * Returns execution events of a given rule (e.g. status changes) from Event Log. + * Accepts rule's saved object ID (`rule.id`) and options for filtering, sorting and pagination. + */ +export const getRuleExecutionEventsRoute = (router: SecuritySolutionPluginRouter) => { + router.get( + { + path: GET_RULE_EXECUTION_EVENTS_URL, + validate: { + params: buildRouteValidation(GetRuleExecutionEventsRequestParams), + query: buildRouteValidation(GetRuleExecutionEventsRequestQuery), + }, + options: { + tags: ['access:securitySolution'], + }, + }, + async (context, request, response) => { + const { params, query } = request; + const siemResponse = buildSiemResponse(response); + + try { + const ctx = await context.resolve(['securitySolution']); + const executionLog = ctx.securitySolution.getRuleExecutionLog(); + const executionEventsResponse = await executionLog.getExecutionEvents({ + ruleId: params.ruleId, + eventTypes: query.event_types, + logLevels: query.log_levels, + sortOrder: query.sort_order, + page: query.page, + perPage: query.per_page, + }); + + const responseBody: GetRuleExecutionEventsResponse = executionEventsResponse; + + return response.ok({ body: responseBody }); + } catch (err) { + const error = transformError(err); + return siemResponse.error({ + body: error.message, + statusCode: error.statusCode, + }); + } + } + ); +}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/get_rule_execution_results/route.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/get_rule_execution_results/route.test.ts new file mode 100644 index 0000000000000..e041670c0b631 --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/get_rule_execution_results/route.test.ts @@ -0,0 +1,69 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { serverMock, requestContextMock, requestMock } from '../../../routes/__mocks__'; + +import { GET_RULE_EXECUTION_RESULTS_URL } from '../../../../../../common/detection_engine/rule_monitoring'; +import { getRuleExecutionResultsResponseMock } from '../../../../../../common/detection_engine/rule_monitoring/mocks'; +import { getRuleExecutionResultsRoute } from './route'; + +describe('getRuleExecutionResultsRoute', () => { + let server: ReturnType; + let { clients, context } = requestContextMock.createTools(); + + const getRuleExecutionResultsRequest = () => + requestMock.create({ + method: 'get', + path: GET_RULE_EXECUTION_RESULTS_URL, + params: { + ruleId: '04128c15-0d1b-4716-a4c5-46997ac7f3bd', + }, + query: { + start: '2022-03-31T22:02:01.622Z', + end: '2022-03-31T22:02:31.622Z', + }, + }); + + beforeEach(async () => { + server = serverMock.create(); + ({ clients, context } = requestContextMock.createTools()); + + getRuleExecutionResultsRoute(server.router); + }); + + describe('when it finds results in rule execution log', () => { + it('returns 200 response with the results', async () => { + const results = getRuleExecutionResultsResponseMock.getSomeResponse(); + clients.ruleExecutionLog.getExecutionResults.mockResolvedValue(results); + + const response = await server.inject( + getRuleExecutionResultsRequest(), + requestContextMock.convertContext(context) + ); + + expect(response.status).toEqual(200); + expect(response.body).toEqual(results); + }); + }); + + describe('when rule execution log client throws an error', () => { + it('returns 500 response with it', async () => { + clients.ruleExecutionLog.getExecutionResults.mockRejectedValue(new Error('Boom!')); + + const response = await server.inject( + getRuleExecutionResultsRequest(), + requestContextMock.convertContext(context) + ); + + expect(response.status).toEqual(500); + expect(response.body).toEqual({ + message: 'Boom!', + status_code: 500, + }); + }); + }); +}); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/get_rule_execution_events_route.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/get_rule_execution_results/route.ts similarity index 52% rename from x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/get_rule_execution_events_route.ts rename to x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/get_rule_execution_results/route.ts index 1cfb7871dbf0f..ff1523502aaea 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/get_rule_execution_events_route.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/get_rule_execution_results/route.ts @@ -6,28 +6,28 @@ */ import { transformError } from '@kbn/securitysolution-es-utils'; -import type { GetAggregateRuleExecutionEventsResponse } from '../../../../../common/detection_engine/schemas/response'; -import { buildRouteValidation } from '../../../../utils/build_validation/route_validation'; -import { buildSiemResponse } from '../utils'; -import type { SecuritySolutionPluginRouter } from '../../../../types'; +import { buildRouteValidation } from '../../../../../utils/build_validation/route_validation'; +import { buildSiemResponse } from '../../../routes/utils'; +import type { SecuritySolutionPluginRouter } from '../../../../../types'; -import { DETECTION_ENGINE_RULE_EXECUTION_EVENTS_URL } from '../../../../../common/constants'; +import type { GetRuleExecutionResultsResponse } from '../../../../../../common/detection_engine/rule_monitoring'; import { - GetRuleExecutionEventsQueryParams, - GetRuleExecutionEventsRequestParams, -} from '../../../../../common/detection_engine/schemas/request/get_rule_execution_events_schema'; + GET_RULE_EXECUTION_RESULTS_URL, + GetRuleExecutionResultsRequestParams, + GetRuleExecutionResultsRequestQuery, +} from '../../../../../../common/detection_engine/rule_monitoring'; /** - * Returns execution events of a given rule (aggregated by executionId) from Event Log. + * Returns execution results of a given rule (aggregated by execution UUID) from Event Log. * Accepts rule's saved object ID (`rule.id`), `start`, `end` and `filters` query params. */ -export const getRuleExecutionEventsRoute = (router: SecuritySolutionPluginRouter) => { +export const getRuleExecutionResultsRoute = (router: SecuritySolutionPluginRouter) => { router.get( { - path: DETECTION_ENGINE_RULE_EXECUTION_EVENTS_URL, + path: GET_RULE_EXECUTION_RESULTS_URL, validate: { - params: buildRouteValidation(GetRuleExecutionEventsRequestParams), - query: buildRouteValidation(GetRuleExecutionEventsQueryParams), + params: buildRouteValidation(GetRuleExecutionResultsRequestParams), + query: buildRouteValidation(GetRuleExecutionResultsRequestQuery), }, options: { tags: ['access:securitySolution'], @@ -45,11 +45,13 @@ export const getRuleExecutionEventsRoute = (router: SecuritySolutionPluginRouter sort_field: sortField, sort_order: sortOrder, } = request.query; + const siemResponse = buildSiemResponse(response); try { - const executionLog = (await context.securitySolution).getRuleExecutionLog(); - const { events, total } = await executionLog.getAggregateExecutionEvents({ + const ctx = await context.resolve(['securitySolution']); + const executionLog = ctx.securitySolution.getRuleExecutionLog(); + const executionResultsResponse = await executionLog.getExecutionResults({ ruleId, start, end, @@ -61,10 +63,7 @@ export const getRuleExecutionEventsRoute = (router: SecuritySolutionPluginRouter sortOrder, }); - const responseBody: GetAggregateRuleExecutionEventsResponse = { - events, - total, - }; + const responseBody: GetRuleExecutionResultsResponse = executionResultsResponse; return response.ok({ body: responseBody }); } catch (err) { diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/register_routes.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/register_routes.ts new file mode 100644 index 0000000000000..c63cf638e7df2 --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/api/register_routes.ts @@ -0,0 +1,15 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type { SecuritySolutionPluginRouter } from '../../../../types'; +import { getRuleExecutionEventsRoute } from './get_rule_execution_events/route'; +import { getRuleExecutionResultsRoute } from './get_rule_execution_results/route'; + +export const registerRuleMonitoringRoutes = (router: SecuritySolutionPluginRouter) => { + getRuleExecutionEventsRoute(router); + getRuleExecutionResultsRoute(router); +}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/index.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/index.ts new file mode 100644 index 0000000000000..ca1b22776c247 --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/index.ts @@ -0,0 +1,9 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +export * from './api/register_routes'; +export * from './logic/rule_execution_log'; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/README.md b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/README.md new file mode 100644 index 0000000000000..01372bf0c4da1 --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/README.md @@ -0,0 +1,195 @@ +# Rule Execution Log + +## Summary + +Rule Execution Log is used to write various execution events to a number of destinations, and then query those +destinations to be able to show plain or aggregated execution data in the app. + +Events we log: + +- Rule execution status changes. See `RuleExecutionEventType['status-change']` and `RuleExecutionStatus`. +- Execution metrics. See `RuleExecutionEventType['execution-metrics']` and `RuleExecutionMetrics`. +- Simple messages. See `RuleExecutionEventType.message`. + +Destinations we write execution logs to: + +- Console Kibana logs. + - Written via an instance of Kibana system `Logger`. +- Event Log (`.kibana-event-log-*` indices). + - Written via an instance of `IEventLogger` from the `event_log` plugin. +- Rule's sidecar saved objects of type `siem-detection-engine-rule-execution-info`. + - Written via an instance of `SavedObjectsClientContract`. + +There are two main interfaces for using Rule Execution Log, these are entrypoints that you can use to start +exploring this implementation: + +- `IRuleExecutionLogForExecutors` - intended to be used from rule executor functions, mainly for the purpose + of writing execution events. +- `IRuleExecutionLogForRoutes` - intended to be used from the API route handlers, mainly for the purpose + of reading (filtering, sorting, searching, aggregating) execution events. + +## Writing status changes + +When we log a rule status change, we do several things: + +- Create or update a `siem-detection-engine-rule-execution-info` sidecar saved object. + Every rule can have exactly 0 or 1 execution info SOs associated with it. + We use it to quickly fetch N execution SOs for N rules to show the rules in a table. +- Write 2 events to Event Log: `execution-metrics` and `status-change`. + These events can be used to show the Rule Execution Log UI on the Rule Details page. +- Write the status change message to console logs (if provided). +- Write the new status itself to console logs. + +This is done by calling the `IRuleExecutionLogForExecutors.logStatusChange` method. + +## Writing console logs + +Console logs from rule executors are written via a logger with the name `plugins.securitySolution.ruleExecution`. +This allows to turn on _only_ rule execution logs in the Kibana config (could be useful when debugging): + +```yaml +logging: + appenders: + custom_console: + type: console + layout: + type: pattern + highlight: true + pattern: "[%date][%level][%logger] %message" + root: + appenders: [custom_console] + level: off + loggers: + - name: plugins.securitySolution.ruleExecution + level: debug # or trace +``` + +Every log message has a suffix with correlation ids: + +```txt +[siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] +``` + +You can also enable printing additional log metadata objects associated with every log record by changing the pattern: + +```yaml +# The metadata object will be printed after the log message +pattern: "[%date][%level][%logger] %message %meta" +``` + +Example of such an object (see `ExtMeta` type for details): + +```txt +{"rule":{"id":"420e1ed0-8f75-11ec-9aaf-c925ad1b24ee","uuid":"9be1325f-7b00-467b-80f1-90d594c22bf4","name":"Test ip range - exceptions with is operator","type":"siem.queryRule","execution":{"uuid":"8d79919b-b09e-4243-ac0c-a4115cd1225f"}},"kibana":{"spaceId":"default"}} +``` + +Example of logs written during a single execution of the "Endpoint Security" rule: + +```txt +[2022-02-23T17:05:09.901+03:00][DEBUG][plugins.securitySolution.ruleExecution] [+] Starting Signal Rule execution [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] +[2022-02-23T17:05:09.907+03:00][DEBUG][plugins.securitySolution.ruleExecution] interval: 5m [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] +[2022-02-23T17:05:09.908+03:00][INFO ][plugins.securitySolution.ruleExecution] Changing rule status to "running" [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] +[2022-02-23T17:05:10.595+03:00][WARN ][plugins.securitySolution.ruleExecution] This rule is attempting to query data from Elasticsearch indices listed in the "Index pattern" section of the rule definition, however no index matching: ["logs-endpoint.alerts-*"] was found. This warning will continue to appear until a matching index is created or this rule is de-activated. If you have recently enrolled agents enabled with Endpoint Security through Fleet, this warning should stop once an alert is sent from an agent. [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] +[2022-02-23T17:05:10.595+03:00][WARN ][plugins.securitySolution.ruleExecution] Changing rule status to "partial failure" [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] +[2022-02-23T17:05:11.630+03:00][DEBUG][plugins.securitySolution.ruleExecution] sortIds: undefined [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] +[2022-02-23T17:05:11.634+03:00][DEBUG][plugins.securitySolution.ruleExecution] totalHits: 0 [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] +[2022-02-23T17:05:11.634+03:00][DEBUG][plugins.securitySolution.ruleExecution] searchResult.hit.hits.length: 0 [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] +[2022-02-23T17:05:11.635+03:00][DEBUG][plugins.securitySolution.ruleExecution] totalHits was 0, exiting early [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] +[2022-02-23T17:05:11.636+03:00][DEBUG][plugins.securitySolution.ruleExecution] [+] completed bulk index of 0 [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] +[2022-02-23T17:05:11.636+03:00][DEBUG][plugins.securitySolution.ruleExecution] [+] Signal Rule execution completed. [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] +[2022-02-23T17:05:11.638+03:00][DEBUG][plugins.securitySolution.ruleExecution] [+] Finished indexing 0 signals into .alerts-security.alerts [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] +[2022-02-23T17:05:11.639+03:00][DEBUG][plugins.securitySolution.ruleExecution] [+] Finished indexing 0 signals searched between date ranges [ + { + "to": "2022-02-23T14:05:09.775Z", + "from": "2022-02-23T13:55:09.775Z", + "maxSignals": 10000 + } +] [siem.queryRule][Endpoint Security][rule id 825b2fab-8b3e-11ec-a4a0-cf820453283c][rule uuid 9a1a2dae-0b5f-4c3d-8305-a268d404c306][exec id ebb7f713-b216-4c90-a456-6c1a6815a065][space default] +``` + +## Finding rule execution data in Elasticsearch + +These are some queries for Kibana Dev Tools that you can use to find execution data associated with a given rule. + +```txt +# Sidecar siem-detection-engine-rule-execution-info saved object +# Rule id: 825b2fab-8b3e-11ec-a4a0-cf820453283c +GET /.kibana/_search +{ + "query": { + "bool": { + "filter": [ + { + "term": { + "type": "siem-detection-engine-rule-execution-info" + } + }, + { + "nested": { + "path": "references", + "query": { + "term": { + "references.id": "825b2fab-8b3e-11ec-a4a0-cf820453283c" + } + } + } + } + ] + } + } +} +``` + +```txt +# Events of type "status-change" written to Event Log +# Rule id: 825b2fab-8b3e-11ec-a4a0-cf820453283c +GET /.kibana-event-log-*/_search +{ + "query": { + "bool": { + "filter": [ + { + "term": { "event.provider": "securitySolution.ruleExecution" } + }, + { + "term": { "event.action": "status-change" } + }, + { + "term": { "rule.id": "825b2fab-8b3e-11ec-a4a0-cf820453283c" } + } + ] + } + }, + "sort": [ + { "@timestamp": { "order": "desc" } }, + { "event.sequence": { "order": "desc" } } + ] +} +``` + +```txt +# Events of type "execution-metrics" written to Event Log +# Rule id: 825b2fab-8b3e-11ec-a4a0-cf820453283c +GET /.kibana-event-log-*/_search +{ + "query": { + "bool": { + "filter": [ + { + "term": { "event.provider": "securitySolution.ruleExecution" } + }, + { + "term": { "event.action": "execution-metrics" } + }, + { + "term": { "rule.id": "825b2fab-8b3e-11ec-a4a0-cf820453283c" } + } + ] + } + }, + "sort": [ + { "@timestamp": { "order": "desc" } }, + { "event.sequence": { "order": "desc" } } + ] +} +``` diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/__mocks__/index.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/__mocks__/index.ts new file mode 100644 index 0000000000000..e43a03c46a906 --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/__mocks__/index.ts @@ -0,0 +1,79 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { + getRuleExecutionEventsResponseMock, + getRuleExecutionResultsResponseMock, + ruleExecutionSummaryMock, +} from '../../../../../../../common/detection_engine/rule_monitoring/mocks'; + +import type { IRuleExecutionLogForRoutes } from '../client_for_routes/client_interface'; +import type { + IRuleExecutionLogForExecutors, + RuleExecutionContext, +} from '../client_for_executors/client_interface'; + +type GetExecutionSummariesBulk = IRuleExecutionLogForRoutes['getExecutionSummariesBulk']; +type GetExecutionSummary = IRuleExecutionLogForRoutes['getExecutionSummary']; +type ClearExecutionSummary = IRuleExecutionLogForRoutes['clearExecutionSummary']; +type GetExecutionEvents = IRuleExecutionLogForRoutes['getExecutionEvents']; +type GetExecutionResults = IRuleExecutionLogForRoutes['getExecutionResults']; + +const ruleExecutionLogForRoutesMock = { + create: (): jest.Mocked => ({ + getExecutionSummariesBulk: jest + .fn, Parameters>() + .mockResolvedValue({ + '04128c15-0d1b-4716-a4c5-46997ac7f3bd': ruleExecutionSummaryMock.getSummarySucceeded(), + '1ea5a820-4da1-4e82-92a1-2b43a7bece08': ruleExecutionSummaryMock.getSummaryFailed(), + }), + + getExecutionSummary: jest + .fn, Parameters>() + .mockResolvedValue(ruleExecutionSummaryMock.getSummarySucceeded()), + + clearExecutionSummary: jest + .fn, Parameters>() + .mockResolvedValue(), + + getExecutionEvents: jest + .fn, Parameters>() + .mockResolvedValue(getRuleExecutionEventsResponseMock.getSomeResponse()), + + getExecutionResults: jest + .fn, Parameters>() + .mockResolvedValue(getRuleExecutionResultsResponseMock.getSomeResponse()), + }), +}; + +const ruleExecutionLogForExecutorsMock = { + create: ( + context: Partial = {} + ): jest.Mocked => ({ + context: { + executionId: context.executionId ?? 'some execution id', + ruleId: context.ruleId ?? 'some rule id', + ruleUuid: context.ruleUuid ?? 'some rule uuid', + ruleName: context.ruleName ?? 'Some rule', + ruleType: context.ruleType ?? 'some rule type', + spaceId: context.spaceId ?? 'some space id', + }, + + trace: jest.fn(), + debug: jest.fn(), + info: jest.fn(), + warn: jest.fn(), + error: jest.fn(), + + logStatusChange: jest.fn(), + }), +}; + +export const ruleExecutionLogMock = { + forRoutes: ruleExecutionLogForRoutesMock, + forExecutors: ruleExecutionLogForExecutorsMock, +}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_executors/client.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_executors/client.ts new file mode 100644 index 0000000000000..4116848b1ffcf --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_executors/client.ts @@ -0,0 +1,242 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { sum } from 'lodash'; +import type { Duration } from 'moment'; +import type { Logger } from '@kbn/core/server'; + +import type { + RuleExecutionMetrics, + RuleExecutionSettings, + RuleExecutionStatus, +} from '../../../../../../../common/detection_engine/rule_monitoring'; +import { + LogLevel, + logLevelFromExecutionStatus, + LogLevelSetting, + logLevelToNumber, + ruleExecutionStatusToNumber, +} from '../../../../../../../common/detection_engine/rule_monitoring'; + +import { assertUnreachable } from '../../../../../../../common/utility_types'; +import { withSecuritySpan } from '../../../../../../utils/with_security_span'; +import { truncateValue } from '../utils/normalization'; +import type { ExtMeta } from '../utils/console_logging'; +import { getCorrelationIds } from './correlation_ids'; + +import type { IEventLogWriter } from '../event_log/event_log_writer'; +import type { IRuleExecutionSavedObjectsClient } from '../execution_saved_object/saved_objects_client'; +import type { + IRuleExecutionLogForExecutors, + RuleExecutionContext, + StatusChangeArgs, +} from './client_interface'; + +export const createClientForExecutors = ( + settings: RuleExecutionSettings, + soClient: IRuleExecutionSavedObjectsClient, + eventLog: IEventLogWriter, + logger: Logger, + context: RuleExecutionContext +): IRuleExecutionLogForExecutors => { + const baseCorrelationIds = getCorrelationIds(context); + const baseLogSuffix = baseCorrelationIds.getLogSuffix(); + const baseLogMeta = baseCorrelationIds.getLogMeta(); + + const { executionId, ruleId, ruleUuid, ruleName, ruleType, spaceId } = context; + + const client: IRuleExecutionLogForExecutors = { + get context() { + return context; + }, + + trace(...messages: string[]): void { + writeMessage(messages, LogLevel.trace); + }, + + debug(...messages: string[]): void { + writeMessage(messages, LogLevel.debug); + }, + + info(...messages: string[]): void { + writeMessage(messages, LogLevel.info); + }, + + warn(...messages: string[]): void { + writeMessage(messages, LogLevel.warn); + }, + + error(...messages: string[]): void { + writeMessage(messages, LogLevel.error); + }, + + async logStatusChange(args: StatusChangeArgs): Promise { + await withSecuritySpan('IRuleExecutionLogForExecutors.logStatusChange', async () => { + const correlationIds = baseCorrelationIds.withStatus(args.newStatus); + const logMeta = correlationIds.getLogMeta(); + + try { + const normalizedArgs = normalizeStatusChangeArgs(args); + + await Promise.all([ + writeStatusChangeToConsole(normalizedArgs, logMeta), + writeStatusChangeToSavedObjects(normalizedArgs), + writeStatusChangeToEventLog(normalizedArgs), + ]); + } catch (e) { + const logMessage = `Error changing rule status to "${args.newStatus}"`; + writeExceptionToConsole(e, logMessage, logMeta); + } + }); + }, + }; + + const writeMessage = (messages: string[], logLevel: LogLevel): void => { + const message = messages.join(' '); + writeMessageToConsole(message, logLevel, baseLogMeta); + writeMessageToEventLog(message, logLevel); + }; + + const writeMessageToConsole = (message: string, logLevel: LogLevel, logMeta: ExtMeta): void => { + switch (logLevel) { + case LogLevel.trace: + logger.trace(`${message} ${baseLogSuffix}`, logMeta); + break; + case LogLevel.debug: + logger.debug(`${message} ${baseLogSuffix}`, logMeta); + break; + case LogLevel.info: + logger.info(`${message} ${baseLogSuffix}`, logMeta); + break; + case LogLevel.warn: + logger.warn(`${message} ${baseLogSuffix}`, logMeta); + break; + case LogLevel.error: + logger.error(`${message} ${baseLogSuffix}`, logMeta); + break; + default: + assertUnreachable(logLevel); + } + }; + + const writeMessageToEventLog = (message: string, logLevel: LogLevel): void => { + const { isEnabled, minLevel } = settings.extendedLogging; + + if (!isEnabled || minLevel === LogLevelSetting.off) { + return; + } + if (logLevelToNumber(logLevel) < logLevelToNumber(minLevel)) { + return; + } + + eventLog.logMessage({ + ruleId, + ruleUuid, + ruleName, + ruleType, + spaceId, + executionId, + message, + logLevel, + }); + }; + + const writeExceptionToConsole = (e: unknown, message: string, logMeta: ExtMeta): void => { + const logReason = e instanceof Error ? e.stack ?? e.message : String(e); + writeMessageToConsole(`${message}. Reason: ${logReason}`, LogLevel.error, logMeta); + }; + + const writeStatusChangeToConsole = (args: NormalizedStatusChangeArgs, logMeta: ExtMeta): void => { + const messageParts: string[] = [`Changing rule status to "${args.newStatus}"`, args.message]; + const logMessage = messageParts.filter(Boolean).join('. '); + const logLevel = logLevelFromExecutionStatus(args.newStatus); + writeMessageToConsole(logMessage, logLevel, logMeta); + }; + + // TODO: Add executionId to new status SO? + const writeStatusChangeToSavedObjects = async ( + args: NormalizedStatusChangeArgs + ): Promise => { + const { newStatus, message, metrics } = args; + + await soClient.createOrUpdate(ruleId, { + last_execution: { + date: nowISO(), + status: newStatus, + status_order: ruleExecutionStatusToNumber(newStatus), + message, + metrics: metrics ?? {}, + }, + }); + }; + + const writeStatusChangeToEventLog = (args: NormalizedStatusChangeArgs): void => { + const { newStatus, message, metrics } = args; + + if (metrics) { + eventLog.logExecutionMetrics({ + ruleId, + ruleUuid, + ruleName, + ruleType, + spaceId, + executionId, + metrics, + }); + } + + eventLog.logStatusChange({ + ruleId, + ruleUuid, + ruleName, + ruleType, + spaceId, + executionId, + newStatus, + message, + }); + }; + + return client; +}; + +const nowISO = () => new Date().toISOString(); + +interface NormalizedStatusChangeArgs { + newStatus: RuleExecutionStatus; + message: string; + metrics?: RuleExecutionMetrics; +} + +const normalizeStatusChangeArgs = (args: StatusChangeArgs): NormalizedStatusChangeArgs => { + const { newStatus, message, metrics } = args; + + return { + newStatus, + message: truncateValue(message) ?? '', + metrics: metrics + ? { + total_search_duration_ms: normalizeDurations(metrics.searchDurations), + total_indexing_duration_ms: normalizeDurations(metrics.indexingDurations), + execution_gap_duration_s: normalizeGap(metrics.executionGap), + } + : undefined, + }; +}; + +const normalizeDurations = (durations?: string[]): number | undefined => { + if (durations == null) { + return undefined; + } + + const sumAsFloat = sum(durations.map(Number)); + return Math.round(sumAsFloat); +}; + +const normalizeGap = (duration?: Duration): number | undefined => { + return duration ? Math.round(duration.asSeconds()) : undefined; +}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_executors/client_interface.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_executors/client_interface.ts new file mode 100644 index 0000000000000..22392e699fcea --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_executors/client_interface.ts @@ -0,0 +1,119 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type { Duration } from 'moment'; +import type { RuleExecutionStatus } from '../../../../../../../common/detection_engine/rule_monitoring'; + +/** + * Used from rule executors to log various information about the rule execution: + * - rule status changes + * - rule execution metrics + * - generic logs with debug/info/warning messages and errors + * + * Write targets: console logs, Event Log, saved objects. + * + * We create a new instance of this interface per each rule execution. + */ +export interface IRuleExecutionLogForExecutors { + /** + * Context with correlation ids and data related to the current rule execution. + */ + context: RuleExecutionContext; + + /** + * Writes a trace message to console logs. + * If enabled, writes it to .kibana-event-log-* index as well. + */ + trace(...messages: string[]): void; + + /** + * Writes a debug message to console logs. + * If enabled, writes it to .kibana-event-log-* index as well. + */ + debug(...messages: string[]): void; + + /** + * Writes an info message to console logs. + * If enabled, writes it to .kibana-event-log-* index as well. + */ + info(...messages: string[]): void; + + /** + * Writes a warning message to console logs. + * If enabled, writes it to .kibana-event-log-* index as well. + */ + warn(...messages: string[]): void; + + /** + * Writes an error message to console logs. + * If enabled, writes it to .kibana-event-log-* index as well. + */ + error(...messages: string[]): void; + + /** + * Writes information about new rule statuses and measured execution metrics: + * 1. To .kibana-* index as a custom `siem-detection-engine-rule-execution-info` saved object. + * This SO is used for fast access to last execution info of a large amount of rules. + * 2. To .kibana-event-log-* index in order to track history of rule executions. + * 3. To console logs. + * @param args Information about the status change event. + */ + logStatusChange(args: StatusChangeArgs): Promise; +} + +/** + * Each time a rule gets executed, we build an instance of rule execution context that + * contains correlation ids and data common to this particular rule execution. + */ +export interface RuleExecutionContext { + /** + * Every execution of a rule executor gets assigned its own UUID at the Alerting Framework + * level. We can use this id to filter all console logs, execution events in Event Log, + * and detection alerts written during a particular rule execution. + */ + executionId: string; + + /** + * Dynamic, saved object id of the rule being executed (rule.id). + */ + ruleId: string; + + /** + * Static, global (or "signature") id of the rule being executed (rule.rule_id). + */ + ruleUuid: string; + + /** + * Name of the rule being executed. + */ + ruleName: string; + + /** + * Alerting Framework's rule type id of the rule being executed. + */ + ruleType: string; + + /** + * Kibana space id of the rule being executed. + */ + spaceId: string; +} + +/** + * Information about the status change event. + */ +export interface StatusChangeArgs { + newStatus: RuleExecutionStatus; + message?: string; + metrics?: MetricsArgs; +} + +export interface MetricsArgs { + searchDurations?: string[]; + indexingDurations?: string[]; + executionGap?: Duration; +} diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_executors/correlation_ids.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_executors/correlation_ids.ts new file mode 100644 index 0000000000000..402635554a0e7 --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_executors/correlation_ids.ts @@ -0,0 +1,87 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type { ExtMeta } from '../utils/console_logging'; +import type { RuleExecutionStatus } from '../../../../../../../common/detection_engine/rule_monitoring'; +import type { RuleExecutionContext } from './client_interface'; + +export interface ICorrelationIds { + withContext(context: RuleExecutionContext): ICorrelationIds; + withStatus(status: RuleExecutionStatus): ICorrelationIds; + + /** + * Returns a string with correlation ids that we append after the log message itself. + */ + getLogSuffix(): string; + + /** + * Returns correlation ids as a metadata object that we include into console log records (structured logs) + */ + getLogMeta(): ExtMeta; +} + +export const getCorrelationIds = (executionContext: RuleExecutionContext): ICorrelationIds => { + return createBuilder({ + context: executionContext, + status: null, + }); +}; + +interface BuilderState { + context: RuleExecutionContext; + status: RuleExecutionStatus | null; +} + +const createBuilder = (state: BuilderState): ICorrelationIds => { + const builder: ICorrelationIds = { + withContext: (context: RuleExecutionContext): ICorrelationIds => { + return createBuilder({ + ...state, + context, + }); + }, + + withStatus: (status: RuleExecutionStatus): ICorrelationIds => { + return createBuilder({ + ...state, + status, + }); + }, + + getLogSuffix: (): string => { + const { executionId, ruleId, ruleUuid, ruleName, ruleType, spaceId } = state.context; + return `[${ruleType}][${ruleName}][rule id ${ruleId}][rule uuid ${ruleUuid}][exec id ${executionId}][space ${spaceId}]`; + }, + + getLogMeta: (): ExtMeta => { + const { context, status } = state; + + const logMeta: ExtMeta = { + rule: { + id: context.ruleId, + uuid: context.ruleUuid, + name: context.ruleName, + type: context.ruleType, + execution: { + uuid: context.executionId, + }, + }, + kibana: { + spaceId: context.spaceId, + }, + }; + + if (status != null && logMeta.rule.execution != null) { + logMeta.rule.execution.status = status; + } + + return logMeta; + }, + }; + + return builder; +}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/client_for_routes/client.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_routes/client.ts similarity index 53% rename from x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/client_for_routes/client.ts rename to x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_routes/client.ts index b8f5b45098e7f..095c77d86a4ce 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/client_for_routes/client.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_routes/client.ts @@ -7,24 +7,27 @@ import { chunk, mapValues } from 'lodash'; import type { Logger } from '@kbn/core/server'; -import type { GetAggregateRuleExecutionEventsResponse } from '../../../../../common/detection_engine/schemas/response'; -import { initPromisePool } from '../../../../utils/promise_pool'; -import { withSecuritySpan } from '../../../../utils/with_security_span'; +import { initPromisePool } from '../../../../../../utils/promise_pool'; +import { withSecuritySpan } from '../../../../../../utils/with_security_span'; +import type { ExtMeta } from '../utils/console_logging'; +import { truncateList } from '../utils/normalization'; -import { RuleExecutionStatus } from '../../../../../common/detection_engine/schemas/common'; +import type { + GetRuleExecutionEventsResponse, + GetRuleExecutionResultsResponse, + RuleExecutionSummary, +} from '../../../../../../../common/detection_engine/rule_monitoring'; import type { IEventLogReader } from '../event_log/event_log_reader'; import type { IRuleExecutionSavedObjectsClient } from '../execution_saved_object/saved_objects_client'; import type { - GetAggregateExecutionEventsArgs, + GetExecutionEventsArgs, + GetExecutionResultsArgs, IRuleExecutionLogForRoutes, + RuleExecutionSummariesByRuleId, } from './client_interface'; -import type { ExtMeta } from '../utils/console_logging'; -import { truncateList } from '../utils/normalization'; - const RULES_PER_CHUNK = 1000; -const MAX_LAST_FAILURES = 5; export const createClientForRoutes = ( soClient: IRuleExecutionSavedObjectsClient, @@ -32,60 +35,11 @@ export const createClientForRoutes = ( logger: Logger ): IRuleExecutionLogForRoutes => { return { - getAggregateExecutionEvents({ - ruleId, - start, - end, - queryText, - statusFilters, - page, - perPage, - sortField, - sortOrder, - }: GetAggregateExecutionEventsArgs): Promise { - return withSecuritySpan( - 'IRuleExecutionLogForRoutes.getAggregateExecutionEvents', - async () => { - try { - return await eventLog.getAggregateExecutionEvents({ - ruleId, - start, - end, - queryText, - statusFilters, - page, - perPage, - sortField, - sortOrder, - }); - } catch (e) { - const logMessage = - 'Error getting last aggregation of execution failures from event log'; - const logAttributes = `rule id: "${ruleId}"`; - const logReason = e instanceof Error ? e.message : String(e); - const logMeta: ExtMeta = { - rule: { id: ruleId }, - }; - - logger.error(`${logMessage}; ${logAttributes}; ${logReason}`, logMeta); - throw e; - } - } - ); - }, - /** - * Get the current rule execution summary for each of the given rule IDs. - * This method splits work into chunks so not to overwhelm Elasticsearch - * when fetching statuses for a big number of rules. - * - * @param ruleIds A list of rule IDs (`rule.id`) to fetch summaries for - * @returns A dict with rule IDs as keys and execution summaries as values - * - * @throws AggregateError if any of the rule status requests fail - */ - getExecutionSummariesBulk(ruleIds) { + getExecutionSummariesBulk: (ruleIds: string[]): Promise => { return withSecuritySpan('IRuleExecutionLogForRoutes.getExecutionSummariesBulk', async () => { try { + // This method splits work into chunks so not to overwhelm Elasticsearch + // when fetching statuses for a big number of rules. const ruleIdsChunks = chunk(ruleIds, RULES_PER_CHUNK); const { results, errors } = await initPromisePool({ @@ -99,10 +53,10 @@ export const createClientForRoutes = ( const ruleIdsString = `[${truncateList(ruleIdsChunk).join(', ')}]`; const logMessage = 'Error fetching a chunk of rule execution saved objects'; - const logAttributes = `num of rules: ${ruleIdsChunk.length}, rule ids: ${ruleIdsString}`; const logReason = e instanceof Error ? e.stack ?? e.message : String(e); + const logSuffix = `[${ruleIdsChunk.length} rules][rule ids: ${ruleIdsString}]`; - logger.error(`${logMessage}; ${logAttributes}; ${logReason}`); + logger.error(`${logMessage}: ${logReason} ${logSuffix}`); throw e; } }, @@ -121,69 +75,87 @@ export const createClientForRoutes = ( const ruleIdsString = `[${truncateList(ruleIds).join(', ')}]`; const logMessage = 'Error bulk getting rule execution summaries'; - const logAttributes = `num of rules: ${ruleIds.length}, rule ids: ${ruleIdsString}`; const logReason = e instanceof Error ? e.message : String(e); + const logSuffix = `[${ruleIds.length} rules][rule ids: ${ruleIdsString}]`; - logger.error(`${logMessage}; ${logAttributes}; ${logReason}`); + logger.error(`${logMessage}: ${logReason} ${logSuffix}`); throw e; } }); }, - getExecutionSummary(ruleId) { + getExecutionSummary: (ruleId: string): Promise => { return withSecuritySpan('IRuleExecutionLogForRoutes.getExecutionSummary', async () => { try { const savedObject = await soClient.getOneByRuleId(ruleId); return savedObject ? savedObject.attributes : null; } catch (e) { const logMessage = 'Error getting rule execution summary'; - const logAttributes = `rule id: "${ruleId}"`; const logReason = e instanceof Error ? e.message : String(e); + const logSuffix = `[rule id ${ruleId}]`; const logMeta: ExtMeta = { rule: { id: ruleId }, }; - logger.error(`${logMessage}; ${logAttributes}; ${logReason}`, logMeta); + logger.error(`${logMessage}: ${logReason} ${logSuffix}`, logMeta); throw e; } }); }, - clearExecutionSummary(ruleId) { + clearExecutionSummary: (ruleId: string): Promise => { return withSecuritySpan('IRuleExecutionLogForRoutes.clearExecutionSummary', async () => { try { await soClient.delete(ruleId); } catch (e) { const logMessage = 'Error clearing rule execution summary'; - const logAttributes = `rule id: "${ruleId}"`; const logReason = e instanceof Error ? e.message : String(e); + const logSuffix = `[rule id ${ruleId}]`; const logMeta: ExtMeta = { rule: { id: ruleId }, }; - logger.error(`${logMessage}; ${logAttributes}; ${logReason}`, logMeta); + logger.error(`${logMessage}: ${logReason} ${logSuffix}`, logMeta); throw e; } }); }, - getLastFailures(ruleId) { - return withSecuritySpan('IRuleExecutionLogForRoutes.getLastFailures', async () => { + getExecutionEvents: (args: GetExecutionEventsArgs): Promise => { + return withSecuritySpan('IRuleExecutionLogForRoutes.getExecutionEvents', async () => { + const { ruleId } = args; try { - return await eventLog.getLastStatusChanges({ - ruleId, - count: MAX_LAST_FAILURES, - includeStatuses: [RuleExecutionStatus.failed], - }); + return await eventLog.getExecutionEvents(args); + } catch (e) { + const logMessage = 'Error getting plain execution events from event log'; + const logReason = e instanceof Error ? e.message : String(e); + const logSuffix = `[rule id ${ruleId}]`; + const logMeta: ExtMeta = { + rule: { id: ruleId }, + }; + + logger.error(`${logMessage}: ${logReason} ${logSuffix}`, logMeta); + throw e; + } + }); + }, + + getExecutionResults: ( + args: GetExecutionResultsArgs + ): Promise => { + return withSecuritySpan('IRuleExecutionLogForRoutes.getExecutionResults', async () => { + const { ruleId } = args; + try { + return await eventLog.getExecutionResults(args); } catch (e) { - const logMessage = 'Error getting last execution failures from event log'; - const logAttributes = `rule id: "${ruleId}"`; + const logMessage = 'Error getting aggregate execution results from event log'; const logReason = e instanceof Error ? e.message : String(e); + const logSuffix = `[rule id ${ruleId}]`; const logMeta: ExtMeta = { rule: { id: ruleId }, }; - logger.error(`${logMessage}; ${logAttributes}; ${logReason}`, logMeta); + logger.error(`${logMessage}: ${logReason} ${logSuffix}`, logMeta); throw e; } }); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_routes/client_interface.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_routes/client_interface.ts new file mode 100644 index 0000000000000..dec3990654b80 --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_routes/client_interface.ts @@ -0,0 +1,107 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type { SortOrder } from '../../../../../../../common/detection_engine/schemas/common'; +import type { + GetRuleExecutionEventsResponse, + GetRuleExecutionResultsResponse, + LogLevel, + RuleExecutionEventType, + RuleExecutionStatus, + RuleExecutionSummary, + SortFieldOfRuleExecutionResult, +} from '../../../../../../../common/detection_engine/rule_monitoring'; + +/** + * Used from route handlers to fetch and manage various information about the rule execution: + * - execution summary of a rule containing such data as the last status and metrics + * - execution events such as recent failures and status changes + */ +export interface IRuleExecutionLogForRoutes { + /** + * Fetches a list of current execution summaries of multiple rules. + * @param ruleIds A list of saved object ids of multiple rules (`rule.id`). + * @returns A dict with rule IDs as keys and execution summaries as values. + * @throws AggregateError if any of the rule status requests fail. + */ + getExecutionSummariesBulk(ruleIds: string[]): Promise; + + /** + * Fetches current execution summary of a given rule. + * @param ruleId Saved object id of the rule (`rule.id`). + */ + getExecutionSummary(ruleId: string): Promise; + + /** + * Deletes the current execution summary if it exists. + * @param ruleId Saved object id of the rule (`rule.id`). + */ + clearExecutionSummary(ruleId: string): Promise; + + /** + * Fetches plain execution events of a given rule from Event Log. This includes debug, info, and + * error messages that executor functions write during a rule execution to the log. + */ + getExecutionEvents(args: GetExecutionEventsArgs): Promise; + + /** + * Fetches execution results aggregated by execution UUID, combining data from both alerting + * and security-solution event-log documents. + */ + getExecutionResults(args: GetExecutionResultsArgs): Promise; +} + +export interface GetExecutionEventsArgs { + /** Saved object id of the rule (`rule.id`). */ + ruleId: string; + + /** Include events of the specified types. If empty, all types of events will be included. */ + eventTypes: RuleExecutionEventType[]; + + /** Include events having these log levels. If empty, events of all levels will be included. */ + logLevels: LogLevel[]; + + /** What order to sort by (e.g. `asc` or `desc`). */ + sortOrder: SortOrder; + + /** Current page to fetch. */ + page: number; + + /** Number of results to fetch per page. */ + perPage: number; +} + +export interface GetExecutionResultsArgs { + /** Saved object id of the rule (`rule.id`). */ + ruleId: string; + + /** Start of daterange to filter to. */ + start: string; + + /** End of daterange to filter to. */ + end: string; + + /** String of field-based filters, e.g. kibana.alert.rule.execution.status:* */ + queryText: string; + + /** Array of status filters, e.g. ['succeeded', 'going to run'] */ + statusFilters: RuleExecutionStatus[]; + + /** Field to sort by. */ + sortField: SortFieldOfRuleExecutionResult; + + /** What order to sort by (e.g. `asc` or `desc`). */ + sortOrder: SortOrder; + + /** Current page to fetch. */ + page: number; + + /** Number of results to fetch per page. */ + perPage: number; +} + +export type RuleExecutionSummariesByRuleId = Record; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/constants.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/constants.ts similarity index 75% rename from x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/constants.ts rename to x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/constants.ts index 08bafa92ac02a..3493e49e88135 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/constants.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/constants.ts @@ -8,8 +8,3 @@ export const RULE_SAVED_OBJECT_TYPE = 'alert'; export const RULE_EXECUTION_LOG_PROVIDER = 'securitySolution.ruleExecution'; - -export enum RuleExecutionLogAction { - 'status-change' = 'status-change', - 'execution-metrics' = 'execution-metrics', -} diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/event_log_reader.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/event_log_reader.ts new file mode 100644 index 0000000000000..2428274165b9c --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/event_log_reader.ts @@ -0,0 +1,243 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey'; +import type { IEventLogClient, IValidatedEvent } from '@kbn/event-log-plugin/server'; +import { MAX_EXECUTION_EVENTS_DISPLAYED } from '@kbn/securitysolution-rules'; + +import { assertUnreachable } from '../../../../../../../common/utility_types'; +import { invariant } from '../../../../../../../common/utils/invariant'; +import { withSecuritySpan } from '../../../../../../utils/with_security_span'; + +import type { + RuleExecutionEvent, + GetRuleExecutionEventsResponse, + GetRuleExecutionResultsResponse, +} from '../../../../../../../common/detection_engine/rule_monitoring'; +import { + LogLevel, + logLevelFromString, + RuleExecutionEventType, + ruleExecutionEventTypeFromString, +} from '../../../../../../../common/detection_engine/rule_monitoring'; +import type { + GetExecutionEventsArgs, + GetExecutionResultsArgs, +} from '../client_for_routes/client_interface'; + +import { RULE_SAVED_OBJECT_TYPE, RULE_EXECUTION_LOG_PROVIDER } from './constants'; +import { + formatExecutionEventResponse, + getExecutionEventAggregation, + mapRuleExecutionStatusToPlatformStatus, +} from './get_execution_event_aggregation'; +import type { ExecutionUuidAggResult } from './get_execution_event_aggregation/types'; +import { EXECUTION_UUID_FIELD } from './get_execution_event_aggregation/types'; + +export interface IEventLogReader { + getExecutionEvents(args: GetExecutionEventsArgs): Promise; + getExecutionResults(args: GetExecutionResultsArgs): Promise; +} + +export const createEventLogReader = (eventLog: IEventLogClient): IEventLogReader => { + return { + async getExecutionEvents( + args: GetExecutionEventsArgs + ): Promise { + const { ruleId, eventTypes, logLevels, sortOrder, page, perPage } = args; + const soType = RULE_SAVED_OBJECT_TYPE; + const soIds = [ruleId]; + + // TODO: include Framework events + const kqlFilter = kqlAnd([ + `event.provider:${RULE_EXECUTION_LOG_PROVIDER}`, + eventTypes.length > 0 ? `event.action:(${kqlOr(eventTypes)})` : '', + logLevels.length > 0 ? `log.level:(${kqlOr(logLevels)})` : '', + ]); + + const findResult = await withSecuritySpan('findEventsBySavedObjectIds', () => { + return eventLog.findEventsBySavedObjectIds(soType, soIds, { + filter: kqlFilter, + sort: [ + { sort_field: '@timestamp', sort_order: sortOrder }, + { sort_field: 'event.sequence', sort_order: sortOrder }, + ], + page, + per_page: perPage, + }); + }); + + return { + events: findResult.data.map((event) => normalizeEvent(event)), + pagination: { + page: findResult.page, + per_page: findResult.per_page, + total: findResult.total, + }, + }; + }, + + async getExecutionResults( + args: GetExecutionResultsArgs + ): Promise { + const { ruleId, start, end, statusFilters, page, perPage, sortField, sortOrder } = args; + const soType = RULE_SAVED_OBJECT_TYPE; + const soIds = [ruleId]; + + // Current workaround to support root level filters without missing fields in the aggregate event + // or including events from statuses that aren't selected + // TODO: See: https://github.com/elastic/kibana/pull/127339/files#r825240516 + // First fetch execution uuid's by status filter if provided + let statusIds: string[] = []; + let totalExecutions: number | undefined; + // If 0 or 3 statuses are selected we can search for all statuses and don't need this pre-filter by ID + if (statusFilters.length > 0 && statusFilters.length < 3) { + const outcomes = mapRuleExecutionStatusToPlatformStatus(statusFilters); + const outcomeFilter = outcomes.length ? `OR event.outcome:(${outcomes.join(' OR ')})` : ''; + const statusResults = await eventLog.aggregateEventsBySavedObjectIds(soType, soIds, { + start, + end, + // Also query for `event.outcome` to catch executions that only contain platform events + filter: `kibana.alert.rule.execution.status:(${statusFilters.join( + ' OR ' + )}) ${outcomeFilter}`, + aggs: { + totalExecutions: { + cardinality: { + field: EXECUTION_UUID_FIELD, + }, + }, + filteredExecutionUUIDs: { + terms: { + field: EXECUTION_UUID_FIELD, + order: { executeStartTime: 'desc' }, + size: MAX_EXECUTION_EVENTS_DISPLAYED, + }, + aggs: { + executeStartTime: { + min: { + field: '@timestamp', + }, + }, + }, + }, + }, + }); + const filteredExecutionUUIDs = statusResults.aggregations + ?.filteredExecutionUUIDs as ExecutionUuidAggResult; + statusIds = filteredExecutionUUIDs?.buckets?.map((b) => b.key) ?? []; + totalExecutions = ( + statusResults.aggregations?.totalExecutions as estypes.AggregationsCardinalityAggregate + ).value; + // Early return if no results based on status filter + if (statusIds.length === 0) { + return { + total: 0, + events: [], + }; + } + } + + // Now query for aggregate events, and pass any ID's as filters as determined from the above status/queryText results + const idsFilter = statusIds.length + ? `kibana.alert.rule.execution.uuid:(${statusIds.join(' OR ')})` + : ''; + const results = await eventLog.aggregateEventsBySavedObjectIds(soType, soIds, { + start, + end, + filter: idsFilter, + aggs: getExecutionEventAggregation({ + maxExecutions: MAX_EXECUTION_EVENTS_DISPLAYED, + page, + perPage, + sort: [{ [sortField]: { order: sortOrder } }], + }), + }); + + return formatExecutionEventResponse(results, totalExecutions); + }, + }; +}; + +const kqlAnd = (items: T[]): string => { + return items.filter(Boolean).map(String).join(' and '); +}; + +const kqlOr = (items: T[]): string => { + return items.filter(Boolean).map(String).join(' or '); +}; + +const normalizeEvent = (rawEvent: IValidatedEvent): RuleExecutionEvent => { + invariant(rawEvent, 'Event not found'); + + const timestamp = normalizeEventTimestamp(rawEvent); + const sequence = normalizeEventSequence(rawEvent); + const level = normalizeLogLevel(rawEvent); + const type = normalizeEventType(rawEvent); + const message = normalizeEventMessage(rawEvent, type); + + return { timestamp, sequence, level, type, message }; +}; + +type RawEvent = NonNullable; + +const normalizeEventTimestamp = (event: RawEvent): string => { + invariant(event['@timestamp'], 'Required "@timestamp" field is not found'); + return event['@timestamp']; +}; + +const normalizeEventSequence = (event: RawEvent): number => { + const value = event.event?.sequence; + if (typeof value === 'number') { + return value; + } + if (typeof value === 'string') { + return Number(value); + } + return 0; +}; + +const normalizeLogLevel = (event: RawEvent): LogLevel => { + const value = event.log?.level; + if (!value) { + return LogLevel.debug; + } + + return logLevelFromString(value) ?? LogLevel.trace; +}; + +const normalizeEventType = (event: RawEvent): RuleExecutionEventType => { + const value = event.event?.action; + invariant(value, 'Required "event.action" field is not found'); + + return ruleExecutionEventTypeFromString(value) ?? RuleExecutionEventType.message; +}; + +const normalizeEventMessage = (event: RawEvent, type: RuleExecutionEventType): string => { + if (type === RuleExecutionEventType.message) { + return event.message || ''; + } + + if (type === RuleExecutionEventType['status-change']) { + invariant( + event.kibana?.alert?.rule?.execution?.status, + 'Required "kibana.alert.rule.execution.status" field is not found' + ); + + const status = event.kibana?.alert?.rule?.execution?.status; + const message = event.message || ''; + + return `Rule changed status to "${status}". ${message}`; + } + + if (type === RuleExecutionEventType['execution-metrics']) { + return ''; + } + + assertUnreachable(type); + return ''; +}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/event_log_writer.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/event_log_writer.ts new file mode 100644 index 0000000000000..aa1fcf36aba68 --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/event_log_writer.ts @@ -0,0 +1,189 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { SavedObjectsUtils } from '@kbn/core/server'; +import type { IEventLogService } from '@kbn/event-log-plugin/server'; +import { SAVED_OBJECT_REL_PRIMARY } from '@kbn/event-log-plugin/server'; +import type { + RuleExecutionMetrics, + RuleExecutionStatus, +} from '../../../../../../../common/detection_engine/rule_monitoring'; +import { + LogLevel, + logLevelFromExecutionStatus, + logLevelToNumber, + RuleExecutionEventType, + ruleExecutionStatusToNumber, +} from '../../../../../../../common/detection_engine/rule_monitoring'; +import { RULE_SAVED_OBJECT_TYPE, RULE_EXECUTION_LOG_PROVIDER } from './constants'; + +export interface IEventLogWriter { + logMessage(args: MessageArgs): void; + logStatusChange(args: StatusChangeArgs): void; + logExecutionMetrics(args: ExecutionMetricsArgs): void; +} + +export interface BaseArgs { + ruleId: string; + ruleUuid: string; + ruleName: string; + ruleType: string; + spaceId: string; + executionId: string; +} + +export interface MessageArgs extends BaseArgs { + logLevel: LogLevel; + message: string; +} + +export interface StatusChangeArgs extends BaseArgs { + newStatus: RuleExecutionStatus; + message?: string; +} + +export interface ExecutionMetricsArgs extends BaseArgs { + metrics: RuleExecutionMetrics; +} + +export const createEventLogWriter = (eventLogService: IEventLogService): IEventLogWriter => { + const eventLogger = eventLogService.getLogger({ + event: { provider: RULE_EXECUTION_LOG_PROVIDER }, + }); + + let sequence = 0; + + return { + logMessage: (args: MessageArgs): void => { + eventLogger.logEvent({ + '@timestamp': nowISO(), + message: args.message, + rule: { + id: args.ruleId, + uuid: args.ruleUuid, + name: args.ruleName, + category: args.ruleType, + }, + event: { + kind: 'event', + action: RuleExecutionEventType.message, + sequence: sequence++, + severity: logLevelToNumber(args.logLevel), + }, + log: { + level: args.logLevel, + }, + kibana: { + alert: { + rule: { + execution: { + uuid: args.executionId, + }, + }, + }, + space_ids: [args.spaceId], + saved_objects: [ + { + rel: SAVED_OBJECT_REL_PRIMARY, + type: RULE_SAVED_OBJECT_TYPE, + id: args.ruleId, + namespace: spaceIdToNamespace(args.spaceId), + }, + ], + }, + }); + }, + + logStatusChange: (args: StatusChangeArgs): void => { + const logLevel = logLevelFromExecutionStatus(args.newStatus); + eventLogger.logEvent({ + '@timestamp': nowISO(), + message: args.message, + rule: { + id: args.ruleId, + uuid: args.ruleUuid, + name: args.ruleName, + category: args.ruleType, + }, + event: { + kind: 'event', + action: RuleExecutionEventType['status-change'], + sequence: sequence++, + severity: logLevelToNumber(logLevel), + }, + log: { + level: logLevel, + }, + kibana: { + alert: { + rule: { + execution: { + uuid: args.executionId, + status: args.newStatus, + status_order: ruleExecutionStatusToNumber(args.newStatus), + }, + }, + }, + space_ids: [args.spaceId], + saved_objects: [ + { + rel: SAVED_OBJECT_REL_PRIMARY, + type: RULE_SAVED_OBJECT_TYPE, + id: args.ruleId, + namespace: spaceIdToNamespace(args.spaceId), + }, + ], + }, + }); + }, + + logExecutionMetrics: (args: ExecutionMetricsArgs): void => { + const logLevel = LogLevel.debug; + eventLogger.logEvent({ + '@timestamp': nowISO(), + rule: { + id: args.ruleId, + uuid: args.ruleUuid, + name: args.ruleName, + category: args.ruleType, + }, + event: { + kind: 'metric', + action: RuleExecutionEventType['execution-metrics'], + sequence: sequence++, + severity: logLevelToNumber(logLevel), + }, + log: { + level: logLevel, + }, + kibana: { + alert: { + rule: { + execution: { + uuid: args.executionId, + metrics: args.metrics, + }, + }, + }, + space_ids: [args.spaceId], + saved_objects: [ + { + rel: SAVED_OBJECT_REL_PRIMARY, + type: RULE_SAVED_OBJECT_TYPE, + id: args.ruleId, + namespace: spaceIdToNamespace(args.spaceId), + }, + ], + }, + }); + }, + }; +}; + +const nowISO = () => new Date().toISOString(); + +const spaceIdToNamespace = SavedObjectsUtils.namespaceStringToId; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/get_execution_event_aggregation/index.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/get_execution_event_aggregation/index.test.ts similarity index 99% rename from x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/get_execution_event_aggregation/index.test.ts rename to x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/get_execution_event_aggregation/index.test.ts index dcd592d7a70fc..83bf237747dda 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/get_execution_event_aggregation/index.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/get_execution_event_aggregation/index.test.ts @@ -12,8 +12,8 @@ * 2.0. */ -import { RuleExecutionStatus } from '../../../../../../common/detection_engine/schemas/common'; import { MAX_EXECUTION_EVENTS_DISPLAYED } from '@kbn/securitysolution-rules'; +import { RuleExecutionStatus } from '../../../../../../../../common/detection_engine/rule_monitoring'; import { formatExecutionEventResponse, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/get_execution_event_aggregation/index.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/get_execution_event_aggregation/index.ts similarity index 97% rename from x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/get_execution_event_aggregation/index.ts rename to x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/get_execution_event_aggregation/index.ts index dc415bbc200b9..c1ebb5e77f98a 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/get_execution_event_aggregation/index.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/get_execution_event_aggregation/index.ts @@ -5,14 +5,17 @@ * 2.0. */ +import { flatMap, get } from 'lodash'; import type * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey'; import { BadRequestError } from '@kbn/securitysolution-es-utils'; -import { flatMap, get } from 'lodash'; import { MAX_EXECUTION_EVENTS_DISPLAYED } from '@kbn/securitysolution-rules'; import type { AggregateEventsBySavedObjectResult } from '@kbn/event-log-plugin/server'; -import type { AggregateRuleExecutionEvent } from '../../../../../../common/detection_engine/schemas/common'; -import { RuleExecutionStatus } from '../../../../../../common/detection_engine/schemas/common'; -import type { GetAggregateRuleExecutionEventsResponse } from '../../../../../../common/detection_engine/schemas/response'; + +import type { + RuleExecutionResult, + GetRuleExecutionResultsResponse, +} from '../../../../../../../../common/detection_engine/rule_monitoring'; +import { RuleExecutionStatus } from '../../../../../../../../common/detection_engine/rule_monitoring'; import type { ExecutionEventAggregationOptions, ExecutionUuidAggResult, @@ -267,7 +270,7 @@ export const getProviderAndActionFilter = (provider: string, action: string) => */ export const formatAggExecutionEventFromBucket = ( bucket: ExecutionUuidAggBucket -): AggregateRuleExecutionEvent => { +): RuleExecutionResult => { const durationUs = bucket?.ruleExecution?.executionDuration?.value ?? 0; const scheduleDelayUs = bucket?.ruleExecution?.scheduleDelay?.value ?? 0; const timedOut = (bucket?.timeoutMessage?.doc_count ?? 0) > 0; @@ -318,7 +321,7 @@ export const formatAggExecutionEventFromBucket = ( export const formatExecutionEventResponse = ( results: AggregateEventsBySavedObjectResult, totalExecutions?: number -): GetAggregateRuleExecutionEventsResponse => { +): GetRuleExecutionResultsResponse => { const { aggregations } = results; if (!aggregations) { diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/get_execution_event_aggregation/types.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/get_execution_event_aggregation/types.ts similarity index 100% rename from x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/get_execution_event_aggregation/types.ts rename to x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/get_execution_event_aggregation/types.ts diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/register_event_log_provider.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/register_event_log_provider.ts similarity index 70% rename from x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/register_event_log_provider.ts rename to x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/register_event_log_provider.ts index c05724198e5b2..94542e913d454 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/event_log/register_event_log_provider.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/register_event_log_provider.ts @@ -6,11 +6,12 @@ */ import type { IEventLogService } from '@kbn/event-log-plugin/server'; -import { RuleExecutionLogAction, RULE_EXECUTION_LOG_PROVIDER } from './constants'; +import { RuleExecutionEventType } from '../../../../../../../common/detection_engine/rule_monitoring'; +import { RULE_EXECUTION_LOG_PROVIDER } from './constants'; export const registerEventLogProvider = (eventLogService: IEventLogService) => { eventLogService.registerProviderActions( RULE_EXECUTION_LOG_PROVIDER, - Object.keys(RuleExecutionLogAction) + Object.keys(RuleExecutionEventType) ); }; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/execution_saved_object/saved_objects_client.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/execution_saved_object/saved_objects_client.ts similarity index 98% rename from x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/execution_saved_object/saved_objects_client.ts rename to x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/execution_saved_object/saved_objects_client.ts index 3333d87c7e8ef..08a7c49f739c0 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/execution_saved_object/saved_objects_client.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/execution_saved_object/saved_objects_client.ts @@ -8,7 +8,7 @@ import type { Logger, SavedObjectsClientContract } from '@kbn/core/server'; import { SavedObjectsErrorHelpers } from '@kbn/core/server'; -import { withSecuritySpan } from '../../../../utils/with_security_span'; +import { withSecuritySpan } from '../../../../../../utils/with_security_span'; import type { RuleExecutionSavedObject, RuleExecutionAttributes } from './saved_objects_type'; import { RULE_EXECUTION_SO_TYPE } from './saved_objects_type'; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/execution_saved_object/saved_objects_type.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/execution_saved_object/saved_objects_type.ts similarity index 96% rename from x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/execution_saved_object/saved_objects_type.ts rename to x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/execution_saved_object/saved_objects_type.ts index 788eb26c3c5ae..ac3b28e87e0d9 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/execution_saved_object/saved_objects_type.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/execution_saved_object/saved_objects_type.ts @@ -10,7 +10,7 @@ import type { RuleExecutionMetrics, RuleExecutionStatus, RuleExecutionStatusOrder, -} from '../../../../../common/detection_engine/schemas/common'; +} from '../../../../../../../common/detection_engine/rule_monitoring'; export const RULE_EXECUTION_SO_TYPE = 'siem-detection-engine-rule-execution-info'; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/execution_saved_object/saved_objects_utils.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/execution_saved_object/saved_objects_utils.ts similarity index 100% rename from x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/execution_saved_object/saved_objects_utils.ts rename to x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/execution_saved_object/saved_objects_utils.ts diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/execution_settings/fetch_rule_execution_settings.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/execution_settings/fetch_rule_execution_settings.ts new file mode 100644 index 0000000000000..a73dba17b6b9f --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/execution_settings/fetch_rule_execution_settings.ts @@ -0,0 +1,89 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type { Logger, SavedObjectsClientContract } from '@kbn/core/server'; +import type { ConfigType } from '../../../../../../config'; +import { withSecuritySpan } from '../../../../../../utils/with_security_span'; +import type { SecuritySolutionPluginCoreSetupDependencies } from '../../../../../../plugin_contract'; + +import { + EXTENDED_RULE_EXECUTION_LOGGING_ENABLED_SETTING, + EXTENDED_RULE_EXECUTION_LOGGING_MIN_LEVEL_SETTING, +} from '../../../../../../../common/constants'; +import type { RuleExecutionSettings } from '../../../../../../../common/detection_engine/rule_monitoring'; +import { LogLevelSetting } from '../../../../../../../common/detection_engine/rule_monitoring'; + +export const fetchRuleExecutionSettings = async ( + config: ConfigType, + logger: Logger, + core: SecuritySolutionPluginCoreSetupDependencies, + savedObjectsClient: SavedObjectsClientContract +): Promise => { + try { + const ruleExecutionSettings = await withSecuritySpan('fetchRuleExecutionSettings', async () => { + const [coreStart] = await withSecuritySpan('getCoreStartServices', () => + core.getStartServices() + ); + + const kibanaAdvancedSettings = await withSecuritySpan('getKibanaAdvancedSettings', () => { + const settingsClient = coreStart.uiSettings.asScopedToClient(savedObjectsClient); + return settingsClient.getAll(); + }); + + return getRuleExecutionSettingsFrom(config, kibanaAdvancedSettings); + }); + + return ruleExecutionSettings; + } catch (e) { + const logMessage = 'Error fetching rule execution settings'; + const logReason = e instanceof Error ? e.stack ?? e.message : String(e); + logger.error(`${logMessage}: ${logReason}`); + + return getRuleExecutionSettingsDefault(config); + } +}; + +const getRuleExecutionSettingsFrom = ( + config: ConfigType, + advancedSettings: Record +): RuleExecutionSettings => { + const featureFlagEnabled = config.experimentalFeatures.extendedRuleExecutionLoggingEnabled; + + const advancedSettingEnabled = getSetting( + advancedSettings, + EXTENDED_RULE_EXECUTION_LOGGING_ENABLED_SETTING, + false + ); + const advancedSettingMinLevel = getSetting( + advancedSettings, + EXTENDED_RULE_EXECUTION_LOGGING_MIN_LEVEL_SETTING, + LogLevelSetting.off + ); + + return { + extendedLogging: { + isEnabled: featureFlagEnabled && advancedSettingEnabled, + minLevel: advancedSettingMinLevel, + }, + }; +}; + +const getRuleExecutionSettingsDefault = (config: ConfigType): RuleExecutionSettings => { + const featureFlagEnabled = config.experimentalFeatures.extendedRuleExecutionLoggingEnabled; + + return { + extendedLogging: { + isEnabled: featureFlagEnabled, + minLevel: featureFlagEnabled ? LogLevelSetting.error : LogLevelSetting.off, + }, + }; +}; + +const getSetting = (settings: Record, key: string, defaultValue: T): T => { + const setting = settings[key]; + return setting != null ? (setting as T) : defaultValue; +}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/index.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/index.ts similarity index 80% rename from x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/index.ts rename to x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/index.ts index c479e95037769..1cc6be1e0c6c4 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/index.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/index.ts @@ -7,10 +7,11 @@ export * from './client_for_executors/client_interface'; export * from './client_for_routes/client_interface'; -export * from './client_factories'; +export * from './service_interface'; +export * from './service'; export { ruleExecutionType } from './execution_saved_object/saved_objects_type'; -export { registerEventLogProvider } from './event_log/register_event_log_provider'; +export { RULE_EXECUTION_LOG_PROVIDER } from './event_log/constants'; export { mergeRuleExecutionSummary } from './merge_rule_execution_summary'; export * from './utils/normalization'; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/merge_rule_execution_summary.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/merge_rule_execution_summary.ts similarity index 76% rename from x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/merge_rule_execution_summary.ts rename to x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/merge_rule_execution_summary.ts index 2e2ac74e94cc5..2b017d27bb971 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/merge_rule_execution_summary.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/merge_rule_execution_summary.ts @@ -5,12 +5,12 @@ * 2.0. */ -import type { RuleExecutionSummary } from '../../../../common/detection_engine/schemas/common'; +import type { RuleExecutionSummary } from '../../../../../../common/detection_engine/rule_monitoring'; import { RuleExecutionStatus, - ruleExecutionStatusOrderByStatus, -} from '../../../../common/detection_engine/schemas/common'; -import type { RuleAlertType } from '../rules/types'; + ruleExecutionStatusToNumber, +} from '../../../../../../common/detection_engine/rule_monitoring'; +import type { RuleAlertType } from '../../../rules/types'; export const mergeRuleExecutionSummary = ( rule: RuleAlertType, @@ -32,7 +32,7 @@ export const mergeRuleExecutionSummary = ( last_execution: { date: frameworkStatus.lastExecutionDate.toISOString(), status: RuleExecutionStatus.failed, - status_order: ruleExecutionStatusOrderByStatus[RuleExecutionStatus.failed], + status_order: ruleExecutionStatusToNumber(RuleExecutionStatus.failed), message: `Reason: ${frameworkStatus.error?.reason} Message: ${frameworkStatus.error?.message}`, metrics: customStatus.metrics, }, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/service.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/service.ts new file mode 100644 index 0000000000000..db5aefdaf7370 --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/service.ts @@ -0,0 +1,80 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type { Logger } from '@kbn/core/server'; +import type { ConfigType } from '../../../../../config'; +import { withSecuritySpan } from '../../../../../utils/with_security_span'; +import type { + SecuritySolutionPluginCoreSetupDependencies, + SecuritySolutionPluginSetupDependencies, +} from '../../../../../plugin_contract'; + +import type { IRuleExecutionLogForRoutes } from './client_for_routes/client_interface'; +import { createClientForRoutes } from './client_for_routes/client'; +import type { IRuleExecutionLogForExecutors } from './client_for_executors/client_interface'; +import { createClientForExecutors } from './client_for_executors/client'; + +import { registerEventLogProvider } from './event_log/register_event_log_provider'; +import { createEventLogReader } from './event_log/event_log_reader'; +import { createEventLogWriter } from './event_log/event_log_writer'; +import { createRuleExecutionSavedObjectsClient } from './execution_saved_object/saved_objects_client'; +import { fetchRuleExecutionSettings } from './execution_settings/fetch_rule_execution_settings'; +import type { + ClientForExecutorsParams, + ClientForRoutesParams, + IRuleExecutionLogService, +} from './service_interface'; + +export const createRuleExecutionLogService = ( + config: ConfigType, + logger: Logger, + core: SecuritySolutionPluginCoreSetupDependencies, + plugins: SecuritySolutionPluginSetupDependencies +): IRuleExecutionLogService => { + return { + registerEventLogProvider: () => { + registerEventLogProvider(plugins.eventLog); + }, + + createClientForRoutes: (params: ClientForRoutesParams): IRuleExecutionLogForRoutes => { + const { savedObjectsClient, eventLogClient } = params; + + const soClient = createRuleExecutionSavedObjectsClient(savedObjectsClient, logger); + const eventLogReader = createEventLogReader(eventLogClient); + + return createClientForRoutes(soClient, eventLogReader, logger); + }, + + createClientForExecutors: ( + params: ClientForExecutorsParams + ): Promise => { + return withSecuritySpan('IRuleExecutionLogService.createClientForExecutors', async () => { + const { savedObjectsClient, context } = params; + + const childLogger = logger.get('ruleExecution'); + + const ruleExecutionSettings = await fetchRuleExecutionSettings( + config, + childLogger, + core, + savedObjectsClient + ); + + const soClient = createRuleExecutionSavedObjectsClient(savedObjectsClient, childLogger); + const eventLogWriter = createEventLogWriter(plugins.eventLog); + + return createClientForExecutors( + ruleExecutionSettings, + soClient, + eventLogWriter, + childLogger, + context + ); + }); + }, + }; +}; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/service_interface.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/service_interface.ts new file mode 100644 index 0000000000000..27207ea2afde0 --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/service_interface.ts @@ -0,0 +1,35 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type { SavedObjectsClientContract } from '@kbn/core/server'; +import type { IEventLogClient } from '@kbn/event-log-plugin/server'; + +import type { IRuleExecutionLogForRoutes } from './client_for_routes/client_interface'; +import type { + IRuleExecutionLogForExecutors, + RuleExecutionContext, +} from './client_for_executors/client_interface'; + +export interface IRuleExecutionLogService { + registerEventLogProvider(): void; + + createClientForRoutes(params: ClientForRoutesParams): IRuleExecutionLogForRoutes; + + createClientForExecutors( + params: ClientForExecutorsParams + ): Promise; +} + +export interface ClientForRoutesParams { + savedObjectsClient: SavedObjectsClientContract; + eventLogClient: IEventLogClient; +} + +export interface ClientForExecutorsParams { + savedObjectsClient: SavedObjectsClientContract; + context: RuleExecutionContext; +} diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/utils/console_logging.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/utils/console_logging.ts new file mode 100644 index 0000000000000..d45c5ee7c65d0 --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/utils/console_logging.ts @@ -0,0 +1,30 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type { LogMeta } from '@kbn/core/server'; +import type { RuleExecutionStatus } from '../../../../../../../common/detection_engine/rule_monitoring'; + +/** + * Extended metadata that rule execution logger can attach to every console log record. + */ +export interface ExtMeta extends LogMeta { + rule: ExtRule; + kibana?: ExtKibana; +} + +interface ExtRule extends NonNullable { + id: string; + type?: string; + execution?: { + uuid: string; + status?: RuleExecutionStatus; + }; +} + +interface ExtKibana { + spaceId?: string; +} diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/utils/normalization.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/utils/normalization.ts similarity index 100% rename from x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/utils/normalization.ts rename to x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/utils/normalization.ts diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/mocks.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/mocks.ts new file mode 100644 index 0000000000000..73288c05c3a71 --- /dev/null +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/mocks.ts @@ -0,0 +1,8 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +export * from './logic/rule_execution_log/__mocks__'; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/__mocks__/rule_type.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/__mocks__/rule_type.ts index 2f636934cbc0c..e0e985ff23865 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/__mocks__/rule_type.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/__mocks__/rule_type.ts @@ -88,7 +88,6 @@ export const createRuleTypeMocks = ( return { dependencies: { alerting, - buildRuleMessage: jest.fn(), config$: mockedConfig$, lists: listMock.createSetup(), logger: loggerMock, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/create_security_rule_type_wrapper.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/create_security_rule_type_wrapper.ts index 25b64601db141..13a8592ca1bd0 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/create_security_rule_type_wrapper.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/create_security_rule_type_wrapper.ts @@ -6,15 +6,13 @@ */ import { isEmpty } from 'lodash'; - -import { parseScheduleDates } from '@kbn/securitysolution-io-ts-utils'; import agent from 'elastic-apm-node'; +import type * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey'; import { TIMESTAMP } from '@kbn/rule-data-utils'; import { createPersistenceRuleTypeWrapper } from '@kbn/rule-registry-plugin/server'; -import type * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey'; +import { parseScheduleDates } from '@kbn/securitysolution-io-ts-utils'; -import { buildRuleMessageFactory } from './factories/build_rule_message_factory'; import { checkPrivilegesFromEsClient, getExceptions, @@ -31,8 +29,8 @@ import { scheduleNotificationActions } from '../notifications/schedule_notificat import { getNotificationResultsLink } from '../notifications/utils'; import { createResultObject } from './utils'; import { bulkCreateFactory, wrapHitsFactory, wrapSequencesFactory } from './factories'; -import { truncateList } from '../rule_execution_log'; -import { RuleExecutionStatus } from '../../../../common/detection_engine/schemas/common'; +import { RuleExecutionStatus } from '../../../../common/detection_engine/rule_monitoring'; +import { truncateList } from '../rule_monitoring'; import { scheduleThrottledNotificationActions } from '../notifications/schedule_throttle_notification_actions'; import aadFieldConversion from '../routes/index/signal_aad_mapping.json'; import { extractReferences, injectReferences } from '../signals/saved_object_references'; @@ -41,15 +39,7 @@ import { getInputIndex, DataViewError } from '../signals/get_input_output_index' /* eslint-disable complexity */ export const createSecurityRuleTypeWrapper: CreateSecurityRuleTypeWrapper = - ({ - lists, - logger, - config, - ruleDataClient, - eventLogService, - ruleExecutionLoggerFactory, - version, - }) => + ({ lists, logger, config, ruleDataClient, ruleExecutionLoggerFactory, version }) => (type) => { const { alertIgnoreFields: ignoreFields, alertMergeStrategy: mergeStrategy } = config; const persistenceRuleType = createPersistenceRuleTypeWrapper({ ruleDataClient, logger }); @@ -63,7 +53,7 @@ export const createSecurityRuleTypeWrapper: CreateSecurityRuleTypeWrapper = }, async executor(options) { agent.setTransactionName(`${options.rule.ruleTypeId} execution`); - return withSecuritySpan('scurityRuleTypeExecutor', async () => { + return withSecuritySpan('securityRuleTypeExecutor', async () => { const { alertId, executionId, @@ -77,7 +67,6 @@ export const createSecurityRuleTypeWrapper: CreateSecurityRuleTypeWrapper = rule, } = options; let runState = state; - let hasError = false; let inputIndex: string[] = []; let runtimeMappings: estypes.MappingRuntimeFields | undefined; const { @@ -99,18 +88,17 @@ export const createSecurityRuleTypeWrapper: CreateSecurityRuleTypeWrapper = const esClient = scopedClusterClient.asCurrentUser; - const ruleExecutionLogger = ruleExecutionLoggerFactory( + const ruleExecutionLogger = await ruleExecutionLoggerFactory({ savedObjectsClient, - eventLogService, - logger, - { + context: { executionId, ruleId: alertId, + ruleUuid: params.ruleId, ruleName: rule.name, ruleType: rule.ruleTypeId, spaceId, - } - ); + }, + }); const completeRule = { ruleConfig: rule, @@ -126,24 +114,16 @@ export const createSecurityRuleTypeWrapper: CreateSecurityRuleTypeWrapper = const refresh = actions.length ? 'wait_for' : false; - const buildRuleMessage = buildRuleMessageFactory({ - id: alertId, - executionId, - ruleId, - name, - index: spaceId, - }); - - logger.debug(buildRuleMessage('[+] Starting Signal Rule execution')); - logger.debug(buildRuleMessage(`interval: ${interval}`)); - - let wroteWarningStatus = false; + ruleExecutionLogger.debug('[+] Starting Signal Rule execution'); + ruleExecutionLogger.debug(`interval: ${interval}`); await ruleExecutionLogger.logStatusChange({ newStatus: RuleExecutionStatus.running, }); let result = createResultObject(state); + let wroteWarningStatus = false; + let hasError = false; const notificationRuleParams: NotificationRuleTypeParams = { ...params, @@ -179,14 +159,11 @@ export const createSecurityRuleTypeWrapper: CreateSecurityRuleTypeWrapper = inputIndex = index ?? []; runtimeMappings = dataViewRuntimeMappings; } catch (exc) { - let errorMessage; - if (exc instanceof DataViewError) { - errorMessage = buildRuleMessage(`Data View not found ${exc}`); - } else { - errorMessage = buildRuleMessage(`Check for indices to search failed ${exc}`); - } + const errorMessage = + exc instanceof DataViewError + ? `Data View not found ${exc}` + : `Check for indices to search failed ${exc}`; - logger.error(errorMessage); await ruleExecutionLogger.logStatusChange({ newStatus: RuleExecutionStatus.failed, message: errorMessage, @@ -205,8 +182,6 @@ export const createSecurityRuleTypeWrapper: CreateSecurityRuleTypeWrapper = wroteWarningStatus = await hasReadIndexPrivileges({ privileges, - logger, - buildRuleMessage, ruleExecutionLogger, uiSettingsClient, }); @@ -231,42 +206,35 @@ export const createSecurityRuleTypeWrapper: CreateSecurityRuleTypeWrapper = timestampFieldCapsResponse: timestampFieldCaps, inputIndices: inputIndex, ruleExecutionLogger, - logger, - buildRuleMessage, }); } } } catch (exc) { - const errorMessage = buildRuleMessage(`Check privileges failed to execute ${exc}`); - logger.warn(errorMessage); await ruleExecutionLogger.logStatusChange({ newStatus: RuleExecutionStatus['partial failure'], - message: errorMessage, + message: `Check privileges failed to execute ${exc}`, }); wroteWarningStatus = true; } + const { tuples, remainingGap } = getRuleRangeTuples({ - logger, + startedAt, previousStartedAt, from, to, interval, maxSignals: maxSignals ?? DEFAULT_MAX_SIGNALS, - buildRuleMessage, - startedAt, + ruleExecutionLogger, }); if (remainingGap.asMilliseconds() > 0) { - const gapString = remainingGap.humanize(); - const gapMessage = buildRuleMessage( - `${gapString} (${remainingGap.asMilliseconds()}ms) were not queried between this rule execution and the last execution, so signals may have been missed.`, - 'Consider increasing your look behind time or adding more Kibana instances.' - ); - logger.warn(gapMessage); hasError = true; + + const gapDuration = `${remainingGap.humanize()} (${remainingGap.asMilliseconds()}ms)`; + await ruleExecutionLogger.logStatusChange({ newStatus: RuleExecutionStatus.failed, - message: gapMessage, + message: `${gapDuration} were not queried between this rule execution and the last execution, so signals may have been missed. Consider increasing your look behind time or adding more Kibana instances`, metrics: { executionGap: remainingGap }, }); } @@ -286,10 +254,9 @@ export const createSecurityRuleTypeWrapper: CreateSecurityRuleTypeWrapper = }); const bulkCreate = bulkCreateFactory( - logger, alertWithPersistence, - buildRuleMessage, - refresh + refresh, + ruleExecutionLogger ); const legacySignalFields: string[] = Object.keys(aadFieldConversion); @@ -316,21 +283,21 @@ export const createSecurityRuleTypeWrapper: CreateSecurityRuleTypeWrapper = services, state: runState, runOpts: { + completeRule, inputIndex, - runtimeMappings, - buildRuleMessage, - bulkCreate, exceptionItems, - listClient, - completeRule, + runtimeMappings, searchAfterSize, tuple, + bulkCreate, wrapHits, wrapSequences, + listClient, ruleDataReader: ruleDataClient.getReader({ namespace: options.spaceId }), mergeStrategy, primaryTimestamp, secondaryTimestamp, + ruleExecutionLogger, }, }); @@ -352,10 +319,9 @@ export const createSecurityRuleTypeWrapper: CreateSecurityRuleTypeWrapper = } if (result.warningMessages.length) { - const warningMessage = buildRuleMessage(truncateList(result.warningMessages).join()); await ruleExecutionLogger.logStatusChange({ newStatus: RuleExecutionStatus['partial failure'], - message: warningMessage, + message: truncateList(result.warningMessages).join(), }); } @@ -372,9 +338,7 @@ export const createSecurityRuleTypeWrapper: CreateSecurityRuleTypeWrapper = ?.kibana_siem_app_url, }); - logger.debug( - buildRuleMessage(`Found ${createdSignalsCount} signals for notification.`) - ); + ruleExecutionLogger.debug(`Found ${createdSignalsCount} signals for notification.`); if (completeRule.ruleConfig.throttle != null) { // NOTE: Since this is throttled we have to call it even on an error condition, otherwise it will "reset" the throttle and fire early @@ -405,19 +369,17 @@ export const createSecurityRuleTypeWrapper: CreateSecurityRuleTypeWrapper = } if (result.success) { - logger.debug(buildRuleMessage('[+] Signal Rule execution completed.')); - logger.debug( - buildRuleMessage( - `[+] Finished indexing ${createdSignalsCount} signals into ${ruleDataClient.indexNameWithNamespace( - spaceId - )}` - ) + ruleExecutionLogger.debug('[+] Signal Rule execution completed.'); + ruleExecutionLogger.debug( + `[+] Finished indexing ${createdSignalsCount} signals into ${ruleDataClient.indexNameWithNamespace( + spaceId + )}` ); if (!hasError && !wroteWarningStatus && !result.warning) { await ruleExecutionLogger.logStatusChange({ newStatus: RuleExecutionStatus.succeeded, - message: 'succeeded', + message: 'Rule execution completed successfully', metrics: { searchDurations: result.searchAfterTimes, indexingDurations: result.bulkCreateTimes, @@ -425,24 +387,17 @@ export const createSecurityRuleTypeWrapper: CreateSecurityRuleTypeWrapper = }); } - logger.debug( - buildRuleMessage( - `[+] Finished indexing ${createdSignalsCount} ${ - !isEmpty(tuples) - ? `signals searched between date ranges ${JSON.stringify(tuples, null, 2)}` - : '' - }` - ) + ruleExecutionLogger.debug( + `[+] Finished indexing ${createdSignalsCount} ${ + !isEmpty(tuples) + ? `signals searched between date ranges ${JSON.stringify(tuples, null, 2)}` + : '' + }` ); } else { - const errorMessage = buildRuleMessage( - 'Bulk Indexing of signals failed:', - truncateList(result.errors).join() - ); - logger.error(errorMessage); await ruleExecutionLogger.logStatusChange({ newStatus: RuleExecutionStatus.failed, - message: errorMessage, + message: `Bulk Indexing of signals failed: ${truncateList(result.errors).join()}`, metrics: { searchDurations: result.searchAfterTimes, indexingDurations: result.bulkCreateTimes, @@ -451,15 +406,10 @@ export const createSecurityRuleTypeWrapper: CreateSecurityRuleTypeWrapper = } } catch (error) { const errorMessage = error.message ?? '(no error message given)'; - const message = buildRuleMessage( - 'An error occurred during rule execution:', - `message: "${errorMessage}"` - ); - logger.error(message); await ruleExecutionLogger.logStatusChange({ newStatus: RuleExecutionStatus.failed, - message, + message: `An error occurred during rule execution: message: "${errorMessage}"`, metrics: { searchDurations: result.searchAfterTimes, indexingDurations: result.bulkCreateTimes, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/eql/create_eql_alert_type.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/eql/create_eql_alert_type.ts index 22676f6d120c3..abfdc5fe491a7 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/eql/create_eql_alert_type.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/eql/create_eql_alert_type.ts @@ -14,10 +14,11 @@ import { eqlRuleParams } from '../../schemas/rule_schemas'; import { eqlExecutor } from '../../signals/executors/eql'; import type { CreateRuleOptions, SecurityAlertType } from '../types'; import { validateImmutable, validateIndexPatterns } from '../utils'; + export const createEqlAlertType = ( createOptions: CreateRuleOptions ): SecurityAlertType => { - const { experimentalFeatures, logger, version } = createOptions; + const { version } = createOptions; return { id: EQL_RULE_TYPE_ID, name: 'Event Correlation Rule', @@ -63,12 +64,13 @@ export const createEqlAlertType = ( async executor(execOptions) { const { runOpts: { + completeRule, + tuple, inputIndex, runtimeMappings, - bulkCreate, exceptionItems, - completeRule, - tuple, + ruleExecutionLogger, + bulkCreate, wrapHits, wrapSequences, primaryTimestamp, @@ -79,16 +81,15 @@ export const createEqlAlertType = ( } = execOptions; const result = await eqlExecutor({ + completeRule, + tuple, inputIndex, runtimeMappings, - bulkCreate, exceptionItems, - experimentalFeatures, - logger, - completeRule, + ruleExecutionLogger, services, - tuple, version, + bulkCreate, wrapHits, wrapSequences, primaryTimestamp, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/factories/build_rule_message_factory.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/factories/build_rule_message_factory.ts deleted file mode 100644 index bac112bb3cab1..0000000000000 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/factories/build_rule_message_factory.ts +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -export type BuildRuleMessage = (...messages: string[]) => string; -export interface BuildRuleMessageFactoryParams { - executionId: string; - name: string; - id: string; - ruleId: string | null | undefined; - index: string; -} - -// TODO: change `index` param to `spaceId` -export const buildRuleMessageFactory = - ({ executionId, id, ruleId, index, name }: BuildRuleMessageFactoryParams): BuildRuleMessage => - (...messages) => - [ - ...messages, - `name: "${name}"`, - `id: "${id}"`, - `rule id: "${ruleId ?? '(unknown rule id)'}"`, - `execution id: "${executionId}"`, - `space ID: "${index}"`, - ].join(' '); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/factories/bulk_create_factory.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/factories/bulk_create_factory.ts index bb02aba492ed5..fadba855baa32 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/factories/bulk_create_factory.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/factories/bulk_create_factory.ts @@ -8,10 +8,9 @@ import { performance } from 'perf_hooks'; import { isEmpty } from 'lodash'; -import type { Logger } from '@kbn/core/server'; import type { PersistenceAlertService } from '@kbn/rule-registry-plugin/server'; import type { AlertWithCommonFieldsLatest } from '@kbn/rule-registry-plugin/common/schemas'; -import type { BuildRuleMessage } from '../../signals/rule_messages'; +import type { IRuleExecutionLogForExecutors } from '../../rule_monitoring'; import { makeFloatString } from '../../signals/utils'; import type { RefreshTypes } from '../../types'; import type { @@ -30,10 +29,9 @@ export interface GenericBulkCreateResponse { export const bulkCreateFactory = ( - logger: Logger, alertWithPersistence: PersistenceAlertService, - buildRuleMessage: BuildRuleMessage, - refreshForBulkCreate: RefreshTypes + refreshForBulkCreate: RefreshTypes, + ruleExecutionLogger: IRuleExecutionLogForExecutors ) => async ( wrappedDocs: Array>, @@ -64,15 +62,13 @@ export const bulkCreateFactory = const end = performance.now(); - logger.debug( - buildRuleMessage( - `individual bulk process time took: ${makeFloatString(end - start)} milliseconds` - ) + ruleExecutionLogger.debug( + `individual bulk process time took: ${makeFloatString(end - start)} milliseconds` ); if (!isEmpty(errors)) { - logger.debug( - buildRuleMessage(`[-] bulkResponse had errors with responses of: ${JSON.stringify(errors)}`) + ruleExecutionLogger.debug( + `[-] bulkResponse had errors with responses of: ${JSON.stringify(errors)}` ); return { errors: Object.keys(errors), diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/factories/index.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/factories/index.ts index 1c1f6fab6322b..4d93238974487 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/factories/index.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/factories/index.ts @@ -5,7 +5,6 @@ * 2.0. */ -export * from './build_rule_message_factory'; export * from './bulk_create_factory'; export * from './wrap_hits_factory'; export * from './wrap_sequences_factory'; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/indicator_match/create_indicator_match_alert_type.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/indicator_match/create_indicator_match_alert_type.ts index 70cf31d8ae7b5..94fc6d78965bb 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/indicator_match/create_indicator_match_alert_type.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/indicator_match/create_indicator_match_alert_type.ts @@ -14,10 +14,11 @@ import { threatRuleParams } from '../../schemas/rule_schemas'; import { threatMatchExecutor } from '../../signals/executors/threat_match'; import type { CreateRuleOptions, SecurityAlertType } from '../types'; import { validateImmutable, validateIndexPatterns } from '../utils'; + export const createIndicatorMatchAlertType = ( createOptions: CreateRuleOptions ): SecurityAlertType => { - const { eventsTelemetry, experimentalFeatures, logger, version } = createOptions; + const { eventsTelemetry, version } = createOptions; return { id: INDICATOR_RULE_TYPE_ID, name: 'Indicator Match Rule', @@ -66,13 +67,13 @@ export const createIndicatorMatchAlertType = ( runOpts: { inputIndex, runtimeMappings, - buildRuleMessage, - bulkCreate, + completeRule, + tuple, exceptionItems, listClient, - completeRule, + ruleExecutionLogger, searchAfterSize, - tuple, + bulkCreate, wrapHits, primaryTimestamp, secondaryTimestamp, @@ -84,18 +85,16 @@ export const createIndicatorMatchAlertType = ( const result = await threatMatchExecutor({ inputIndex, runtimeMappings, - buildRuleMessage, - bulkCreate, - exceptionItems, - experimentalFeatures, - eventsTelemetry, - listClient, - logger, completeRule, - searchAfterSize, - services, tuple, + listClient, + exceptionItems, + services, version, + searchAfterSize, + ruleExecutionLogger, + eventsTelemetry, + bulkCreate, wrapHits, primaryTimestamp, secondaryTimestamp, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/ml/create_ml_alert_type.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/ml/create_ml_alert_type.ts index abfc279a6e08f..926615fc8d176 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/ml/create_ml_alert_type.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/ml/create_ml_alert_type.ts @@ -18,7 +18,7 @@ import { validateImmutable } from '../utils'; export const createMlAlertType = ( createOptions: CreateRuleOptions ): SecurityAlertType => { - const { logger, ml } = createOptions; + const { ml } = createOptions; return { id: ML_RULE_TYPE_ID, name: 'Machine Learning Rule', @@ -63,11 +63,11 @@ export const createMlAlertType = ( async executor(execOptions) { const { runOpts: { - buildRuleMessage, bulkCreate, + completeRule, exceptionItems, listClient, - completeRule, + ruleExecutionLogger, tuple, wrapHits, }, @@ -76,15 +76,14 @@ export const createMlAlertType = ( } = execOptions; const result = await mlExecutor({ - buildRuleMessage, - bulkCreate, - exceptionItems, - listClient, - logger, - ml, completeRule, - services, tuple, + ml, + listClient, + exceptionItems, + services, + ruleExecutionLogger, + bulkCreate, wrapHits, }); return { ...result, state }; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/new_terms/create_new_terms_alert_type.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/new_terms/create_new_terms_alert_type.ts index 9bea2eb495b26..e02bcc6251f40 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/new_terms/create_new_terms_alert_type.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/new_terms/create_new_terms_alert_type.ts @@ -100,7 +100,7 @@ export const createNewTermsAlertType = ( async executor(execOptions) { const { runOpts: { - buildRuleMessage, + ruleExecutionLogger, bulkCreate, completeRule, exceptionItems, @@ -185,12 +185,11 @@ export const createNewTermsAlertType = ( from: tuple.from.toISOString(), to: tuple.to.toISOString(), services, + ruleExecutionLogger, filter, - logger, pageSize: 0, primaryTimestamp, secondaryTimestamp, - buildRuleMessage, runtimeMappings, }); const searchResultWithAggs = searchResult as RecentTermsAggResult; @@ -239,12 +238,11 @@ export const createNewTermsAlertType = ( from: parsedHistoryWindowSize.toISOString(), to: tuple.to.toISOString(), services, + ruleExecutionLogger, filter, - logger, pageSize: 0, primaryTimestamp, secondaryTimestamp, - buildRuleMessage, }); searchAfterResults.searchDurations.push(pageSearchDuration); searchAfterResults.searchErrors.push(...pageSearchErrors); @@ -285,12 +283,11 @@ export const createNewTermsAlertType = ( from: tuple.from.toISOString(), to: tuple.to.toISOString(), services, + ruleExecutionLogger, filter, - logger, pageSize: 0, primaryTimestamp, secondaryTimestamp, - buildRuleMessage, }); searchAfterResults.searchDurations.push(docFetchSearchDuration); searchAfterResults.searchErrors.push(...docFetchSearchErrors); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/query/create_query_alert_type.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/query/create_query_alert_type.test.ts index 4a6b6b3c9d867..c7328055e1a7d 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/query/create_query_alert_type.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/query/create_query_alert_type.test.ts @@ -13,7 +13,7 @@ import { createRuleTypeMocks } from '../__mocks__/rule_type'; import { createSecurityRuleTypeWrapper } from '../create_security_rule_type_wrapper'; import { createMockConfig } from '../../routes/__mocks__'; import { createMockTelemetryEventsSender } from '../../../telemetry/__mocks__'; -import { ruleExecutionLogMock } from '../../rule_execution_log/__mocks__'; +import { ruleExecutionLogMock } from '../../rule_monitoring/mocks'; import { sampleDocNoSortId } from '../../signals/__mocks__/es_results'; import { getQueryRuleParams } from '../../schemas/rule_schemas.mock'; @@ -32,14 +32,13 @@ jest.mock('../utils/get_list_client', () => ({ describe('Custom Query Alerts', () => { const mocks = createRuleTypeMocks(); const { dependencies, executor, services } = mocks; - const { alerting, eventLogService, lists, logger, ruleDataClient } = dependencies; + const { alerting, lists, logger, ruleDataClient } = dependencies; const securityRuleTypeWrapper = createSecurityRuleTypeWrapper({ lists, logger, config: createMockConfig(), ruleDataClient, - eventLogService, - ruleExecutionLoggerFactory: () => ruleExecutionLogMock.forExecutors.create(), + ruleExecutionLoggerFactory: () => Promise.resolve(ruleExecutionLogMock.forExecutors.create()), version: '8.3', }); const eventsTelemetry = createMockTelemetryEventsSender(true); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/query/create_query_alert_type.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/query/create_query_alert_type.ts index b9a6c17bfc261..14e309a83c959 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/query/create_query_alert_type.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/query/create_query_alert_type.ts @@ -17,7 +17,7 @@ import { validateImmutable, validateIndexPatterns } from '../utils'; export const createQueryAlertType = ( createOptions: CreateRuleOptions ): SecurityAlertType => { - const { eventsTelemetry, experimentalFeatures, logger, version } = createOptions; + const { eventsTelemetry, experimentalFeatures, version } = createOptions; return { id: QUERY_RULE_TYPE_ID, name: 'Custom Query Rule', @@ -65,13 +65,13 @@ export const createQueryAlertType = ( runOpts: { inputIndex, runtimeMappings, - buildRuleMessage, - bulkCreate, + completeRule, + tuple, exceptionItems, listClient, - completeRule, + ruleExecutionLogger, searchAfterSize, - tuple, + bulkCreate, wrapHits, primaryTimestamp, secondaryTimestamp, @@ -81,18 +81,17 @@ export const createQueryAlertType = ( } = execOptions; const result = await queryExecutor({ - buildRuleMessage, - bulkCreate, + completeRule, + tuple, exceptionItems, + listClient, experimentalFeatures, + ruleExecutionLogger, eventsTelemetry, - listClient, - logger, - completeRule, - searchAfterSize, services, - tuple, version, + searchAfterSize, + bulkCreate, wrapHits, inputIndex, runtimeMappings, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/saved_query/create_saved_query_alert_type.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/saved_query/create_saved_query_alert_type.ts index fe9c377aa04f6..f8009220581e1 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/saved_query/create_saved_query_alert_type.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/saved_query/create_saved_query_alert_type.ts @@ -17,7 +17,7 @@ import { validateImmutable, validateIndexPatterns } from '../utils'; export const createSavedQueryAlertType = ( createOptions: CreateRuleOptions ): SecurityAlertType => { - const { experimentalFeatures, logger, version } = createOptions; + const { experimentalFeatures, version } = createOptions; return { id: SAVED_QUERY_RULE_TYPE_ID, name: 'Saved Query Rule', @@ -65,13 +65,13 @@ export const createSavedQueryAlertType = ( runOpts: { inputIndex, runtimeMappings, - buildRuleMessage, - bulkCreate, + completeRule, + tuple, exceptionItems, listClient, - completeRule, + ruleExecutionLogger, searchAfterSize, - tuple, + bulkCreate, wrapHits, primaryTimestamp, secondaryTimestamp, @@ -83,18 +83,17 @@ export const createSavedQueryAlertType = ( const result = await queryExecutor({ inputIndex, runtimeMappings, - buildRuleMessage, - bulkCreate, + completeRule: completeRule as CompleteRule, + tuple, exceptionItems, experimentalFeatures, - eventsTelemetry: undefined, listClient, - logger, - completeRule: completeRule as CompleteRule, - searchAfterSize, + ruleExecutionLogger, + eventsTelemetry: undefined, services, - tuple, version, + searchAfterSize, + bulkCreate, wrapHits, primaryTimestamp, secondaryTimestamp, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/threshold/create_threshold_alert_type.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/threshold/create_threshold_alert_type.ts index 43e57057bcbc1..5c8426e194f0d 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/threshold/create_threshold_alert_type.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/threshold/create_threshold_alert_type.ts @@ -19,7 +19,7 @@ import { validateImmutable, validateIndexPatterns } from '../utils'; export const createThresholdAlertType = ( createOptions: CreateRuleOptions ): SecurityAlertType => { - const { experimentalFeatures, logger, version } = createOptions; + const { version } = createOptions; return { id: THRESHOLD_RULE_TYPE_ID, name: 'Threshold Rule', @@ -65,7 +65,6 @@ export const createThresholdAlertType = ( async executor(execOptions) { const { runOpts: { - buildRuleMessage, bulkCreate, exceptionItems, completeRule, @@ -76,6 +75,7 @@ export const createThresholdAlertType = ( runtimeMappings, primaryTimestamp, secondaryTimestamp, + ruleExecutionLogger, }, services, startedAt, @@ -83,17 +83,15 @@ export const createThresholdAlertType = ( } = execOptions; const result = await thresholdExecutor({ - buildRuleMessage, - bulkCreate, - exceptionItems, - experimentalFeatures, - logger, completeRule, + tuple, + exceptionItems, + ruleExecutionLogger, services, + version, startedAt, state, - tuple, - version, + bulkCreate, wrapHits, ruleDataReader, inputIndex, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/types.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/types.ts index 3212fee6a1c1d..d2ed6965a547a 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/types.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/types.ts @@ -24,11 +24,10 @@ import type { IRuleDataClient, IRuleDataReader, } from '@kbn/rule-registry-plugin/server'; -import type { IEventLogService } from '@kbn/event-log-plugin/server'; + import type { ConfigType } from '../../../config'; import type { SetupPlugins } from '../../../plugin'; import type { CompleteRule, RuleParams } from '../schemas/rule_schemas'; -import type { BuildRuleMessage } from '../signals/rule_messages'; import type { BulkCreate, SearchAfterAndBulkCreateReturnType, @@ -37,7 +36,7 @@ import type { } from '../signals/types'; import type { ExperimentalFeatures } from '../../../../common/experimental_features'; import type { ITelemetryEventsSender } from '../../telemetry/sender'; -import type { RuleExecutionLogForExecutorsFactory } from '../rule_execution_log'; +import type { IRuleExecutionLogForExecutors, IRuleExecutionLogService } from '../rule_monitoring'; export interface SecurityAlertTypeReturnValue { bulkCreateTimes: string[]; @@ -53,17 +52,17 @@ export interface SecurityAlertTypeReturnValue { } export interface RunOpts { - buildRuleMessage: BuildRuleMessage; - bulkCreate: BulkCreate; - exceptionItems: ExceptionListItemSchema[]; - listClient: ListClient; completeRule: CompleteRule; - searchAfterSize: number; tuple: { to: Moment; from: Moment; maxSignals: number; }; + exceptionItems: ExceptionListItemSchema[]; + ruleExecutionLogger: IRuleExecutionLogForExecutors; + listClient: ListClient; + searchAfterSize: number; + bulkCreate: BulkCreate; wrapHits: WrapHits; wrapSequences: WrapSequences; ruleDataReader: IRuleDataReader; @@ -102,8 +101,7 @@ export interface CreateSecurityRuleTypeWrapperProps { logger: Logger; config: ConfigType; ruleDataClient: IRuleDataClient; - eventLogService: IEventLogService; - ruleExecutionLoggerFactory: RuleExecutionLogForExecutorsFactory; + ruleExecutionLoggerFactory: IRuleExecutionLogService['createClientForExecutors']; version: string; } diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rules/delete_rules.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rules/delete_rules.test.ts index 7c0daefc6c6c3..0c7edae7022c7 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rules/delete_rules.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rules/delete_rules.test.ts @@ -6,7 +6,7 @@ */ import { rulesClientMock } from '@kbn/alerting-plugin/server/mocks'; -import { ruleExecutionLogMock } from '../rule_execution_log/__mocks__'; +import { ruleExecutionLogMock } from '../rule_monitoring/mocks'; import { deleteRules } from './delete_rules'; import type { DeleteRuleOptions } from './types'; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rules/types.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rules/types.ts index 569b253dd9089..6e3b1a3dea2cb 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rules/types.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rules/types.ts @@ -8,27 +8,28 @@ import type { Readable } from 'stream'; import type { SavedObjectAttributes, SavedObjectsClientContract } from '@kbn/core/server'; +import type { SanitizedRule } from '@kbn/alerting-plugin/common'; +import type { RulesClient, PartialRule } from '@kbn/alerting-plugin/server'; import { ruleTypeMappings } from '@kbn/securitysolution-rules'; -import type { RulesClient, PartialRule } from '@kbn/alerting-plugin/server'; -import type { SanitizedRule } from '@kbn/alerting-plugin/common'; -import type { UpdateRulesSchema } from '../../../../common/detection_engine/schemas/request'; import type { + FieldsOrUndefined, Id, IdOrUndefined, - RuleIdOrUndefined, - PerPageOrUndefined, PageOrUndefined, - SortFieldOrUndefined, + PerPageOrUndefined, QueryFilterOrUndefined, - FieldsOrUndefined, + RuleIdOrUndefined, + SortFieldOrUndefined, SortOrderOrUndefined, } from '../../../../common/detection_engine/schemas/common'; -import type { RuleParams } from '../schemas/rule_schemas'; -import type { IRuleExecutionLogForRoutes } from '../rule_execution_log'; import type { CreateRulesSchema } from '../../../../common/detection_engine/schemas/request/rule_schemas'; import type { PatchRulesSchema } from '../../../../common/detection_engine/schemas/request/patch_rules_schema'; +import type { UpdateRulesSchema } from '../../../../common/detection_engine/schemas/request'; + +import type { RuleParams } from '../schemas/rule_schemas'; +import type { IRuleExecutionLogForRoutes } from '../rule_monitoring'; export type RuleAlertType = SanitizedRule; @@ -96,12 +97,12 @@ export interface DeleteRuleOptions { export interface FindRuleOptions { rulesClient: RulesClient; - perPage: PerPageOrUndefined; - page: PageOrUndefined; - sortField: SortFieldOrUndefined; filter: QueryFilterOrUndefined; fields: FieldsOrUndefined; + sortField: SortFieldOrUndefined; sortOrder: SortOrderOrUndefined; + page: PageOrUndefined; + perPage: PerPageOrUndefined; } export interface LegacyMigrateParams { diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rules/update_prepacked_rules.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rules/update_prepacked_rules.test.ts index dc28f2184323c..d71274c7f1540 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rules/update_prepacked_rules.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rules/update_prepacked_rules.test.ts @@ -14,7 +14,7 @@ import { getAddPrepackagedRulesSchemaMock, getAddPrepackagedThreatMatchRulesSchemaMock, } from '../../../../common/detection_engine/schemas/request/add_prepackaged_rules_schema.mock'; -import { ruleExecutionLogMock } from '../rule_execution_log/__mocks__'; +import { ruleExecutionLogMock } from '../rule_monitoring/mocks'; import { legacyMigrate } from './utils'; import { getQueryRuleParams, getThreatRuleParams } from '../schemas/rule_schemas.mock'; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rules/update_prepacked_rules.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rules/update_prepacked_rules.ts index b6b528c307b38..1487aa79e4874 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/rules/update_prepacked_rules.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rules/update_prepacked_rules.ts @@ -16,7 +16,7 @@ import type { RuleParams } from '../schemas/rule_schemas'; import { legacyMigrate } from './utils'; import { deleteRules } from './delete_rules'; import { PrepackagedRulesError } from '../routes/rules/add_prepackaged_rules_route'; -import type { IRuleExecutionLogForRoutes } from '../rule_execution_log'; +import type { IRuleExecutionLogForRoutes } from '../rule_monitoring'; import { createRules } from './create_rules'; import { transformAlertToRuleAction } from '../../../../common/detection_engine/transform_actions'; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/schemas/rule_converters.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/schemas/rule_converters.ts index 0056606f17f15..74d55c262d17b 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/schemas/rule_converters.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/schemas/rule_converters.ts @@ -41,9 +41,9 @@ import { assertUnreachable } from '../../../../common/utility_types'; import type { RelatedIntegrationArray, RequiredFieldArray, - RuleExecutionSummary, SetupGuide, } from '../../../../common/detection_engine/schemas/common'; +import type { RuleExecutionSummary } from '../../../../common/detection_engine/rule_monitoring'; import { eqlPatchParams, machineLearningPatchParams, @@ -81,7 +81,7 @@ import { } from '../rules/utils'; // eslint-disable-next-line no-restricted-imports import type { LegacyRuleActions } from '../rule_actions/legacy_types'; -import { mergeRuleExecutionSummary } from '../rule_execution_log'; +import { mergeRuleExecutionSummary } from '../rule_monitoring'; // These functions provide conversions from the request API schema to the internal rule schema and from the internal rule schema // to the response API schema. This provides static type-check assurances that the internal schema is in sync with the API schema for diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/bulk_create_ml_signals.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/bulk_create_ml_signals.ts index ef8662960d0a5..11b7a035d8a2f 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/bulk_create_ml_signals.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/bulk_create_ml_signals.ts @@ -8,7 +8,6 @@ import type * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey'; import { flow, omit } from 'lodash/fp'; import set from 'set-value'; -import type { Logger } from '@kbn/core/server'; import type { AlertInstanceContext, AlertInstanceState, @@ -16,20 +15,19 @@ import type { } from '@kbn/alerting-plugin/server'; import type { GenericBulkCreateResponse } from '../rule_types/factories'; import type { Anomaly } from '../../machine_learning'; -import type { BuildRuleMessage } from './rule_messages'; import type { BulkCreate, WrapHits } from './types'; import type { CompleteRule, MachineLearningRuleParams } from '../schemas/rule_schemas'; import { buildReasonMessageForMlAlert } from './reason_formatters'; import type { BaseFieldsLatest } from '../../../../common/detection_engine/schemas/alerts'; +import type { IRuleExecutionLogForExecutors } from '../rule_monitoring'; interface BulkCreateMlSignalsParams { anomalyHits: Array>; completeRule: CompleteRule; services: RuleExecutorServices; - logger: Logger; + ruleExecutionLogger: IRuleExecutionLogForExecutors; id: string; signalsIndex: string; - buildRuleMessage: BuildRuleMessage; bulkCreate: BulkCreate; wrapHits: WrapHits; } diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/eql.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/eql.test.ts index 3f52dd1fcd4e8..ab52fedf60e41 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/eql.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/eql.test.ts @@ -6,24 +6,23 @@ */ import dateMath from '@kbn/datemath'; -import { loggingSystemMock } from '@kbn/core/server/mocks'; import type { RuleExecutorServicesMock } from '@kbn/alerting-plugin/server/mocks'; import { alertsMock } from '@kbn/alerting-plugin/server/mocks'; -import { eqlExecutor } from './eql'; import { getExceptionListItemSchemaMock } from '@kbn/lists-plugin/common/schemas/response/exception_list_item_schema.mock'; import { getEntryListMock } from '@kbn/lists-plugin/common/schemas/types/entry_list.mock'; -import { getCompleteRuleMock, getEqlRuleParams } from '../../schemas/rule_schemas.mock'; +import { DEFAULT_INDEX_PATTERN } from '../../../../../common/constants'; import { getIndexVersion } from '../../routes/index/get_index_version'; import { SIGNALS_TEMPLATE_VERSION } from '../../routes/index/get_signals_template'; -import { allowedExperimentalValues } from '../../../../../common/experimental_features'; import type { EqlRuleParams } from '../../schemas/rule_schemas'; -import { DEFAULT_INDEX_PATTERN } from '../../../../../common/constants'; +import { getCompleteRuleMock, getEqlRuleParams } from '../../schemas/rule_schemas.mock'; +import { ruleExecutionLogMock } from '../../rule_monitoring/mocks'; +import { eqlExecutor } from './eql'; jest.mock('../../routes/index/get_index_version'); describe('eql_executor', () => { const version = '8.0.0'; - let logger: ReturnType; + const ruleExecutionLogger = ruleExecutionLogMock.forExecutors.create(); let alertServices: RuleExecutorServicesMock; (getIndexVersion as jest.Mock).mockReturnValue(SIGNALS_TEMPLATE_VERSION); const params = getEqlRuleParams(); @@ -35,8 +34,8 @@ describe('eql_executor', () => { }; beforeEach(() => { + jest.clearAllMocks(); alertServices = alertsMock.createRuleExecutorServices(); - logger = loggingSystemMock.createLogger(); alertServices.scopedClusterClient.asCurrentUser.eql.search.mockResolvedValue({ hits: { total: { relation: 'eq', value: 10 }, @@ -54,10 +53,9 @@ describe('eql_executor', () => { completeRule: eqlCompleteRule, tuple, exceptionItems, - experimentalFeatures: allowedExperimentalValues, + ruleExecutionLogger, services: alertServices, version, - logger, bulkCreate: jest.fn(), wrapHits: jest.fn(), wrapSequences: jest.fn(), diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/eql.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/eql.ts index 5e54515be1056..c1de5fc5f18d9 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/eql.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/eql.ts @@ -7,7 +7,6 @@ import { performance } from 'perf_hooks'; import type { ExceptionListItemSchema } from '@kbn/securitysolution-io-ts-list-types'; -import type { Logger } from '@kbn/core/server'; import type { AlertInstanceContext, AlertInstanceState, @@ -27,7 +26,6 @@ import type { SignalSource, } from '../types'; import { createSearchAfterReturnType, makeFloatString } from '../utils'; -import type { ExperimentalFeatures } from '../../../../../common/experimental_features'; import { buildReasonMessageForEqlAlert } from '../reason_formatters'; import type { CompleteRule, EqlRuleParams } from '../../schemas/rule_schemas'; import { withSecuritySpan } from '../../../../utils/with_security_span'; @@ -35,6 +33,7 @@ import type { BaseFieldsLatest, WrappedFieldsLatest, } from '../../../../../common/detection_engine/schemas/alerts'; +import type { IRuleExecutionLogForExecutors } from '../../rule_monitoring'; export const eqlExecutor = async ({ inputIndex, @@ -42,10 +41,9 @@ export const eqlExecutor = async ({ completeRule, tuple, exceptionItems, - experimentalFeatures, + ruleExecutionLogger, services, version, - logger, bulkCreate, wrapHits, wrapSequences, @@ -57,10 +55,9 @@ export const eqlExecutor = async ({ completeRule: CompleteRule; tuple: RuleRangeTuple; exceptionItems: ExceptionListItemSchema[]; - experimentalFeatures: ExperimentalFeatures; + ruleExecutionLogger: IRuleExecutionLogForExecutors; services: RuleExecutorServices; version: string; - logger: Logger; bulkCreate: BulkCreate; wrapHits: WrapHits; wrapSequences: WrapSequences; @@ -94,8 +91,9 @@ export const eqlExecutor = async ({ tiebreakerField: ruleParams.tiebreakerField, }); + ruleExecutionLogger.debug(`EQL query request: ${JSON.stringify(request)}`); + const eqlSignalSearchStart = performance.now(); - logger.debug(`EQL query request: ${JSON.stringify(request)}`); const response = await services.scopedClusterClient.asCurrentUser.eql.search( request diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/ml.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/ml.test.ts index c838f3243fc33..58a693c71bc41 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/ml.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/ml.test.ts @@ -6,18 +6,17 @@ */ import dateMath from '@kbn/datemath'; -import { loggingSystemMock } from '@kbn/core/server/mocks'; import type { RuleExecutorServicesMock } from '@kbn/alerting-plugin/server/mocks'; import { alertsMock } from '@kbn/alerting-plugin/server/mocks'; import { mlExecutor } from './ml'; import { getExceptionListItemSchemaMock } from '@kbn/lists-plugin/common/schemas/response/exception_list_item_schema.mock'; import { getCompleteRuleMock, getMlRuleParams } from '../../schemas/rule_schemas.mock'; -import { buildRuleMessageFactory } from '../rule_messages'; import { getListClientMock } from '@kbn/lists-plugin/server/services/lists/list_client.mock'; import { findMlSignals } from '../find_ml_signals'; import { bulkCreateMlSignals } from '../bulk_create_ml_signals'; import { mlPluginServerMock } from '@kbn/ml-plugin/server/mocks'; import type { MachineLearningRuleParams } from '../../schemas/rule_schemas'; +import { ruleExecutionLogMock } from '../../rule_monitoring/mocks'; jest.mock('../find_ml_signals'); jest.mock('../bulk_create_ml_signals'); @@ -25,32 +24,30 @@ jest.mock('../bulk_create_ml_signals'); describe('ml_executor', () => { let jobsSummaryMock: jest.Mock; let mlMock: ReturnType; - const exceptionItems = [getExceptionListItemSchemaMock()]; - let logger: ReturnType; let alertServices: RuleExecutorServicesMock; + let ruleExecutionLogger: ReturnType; const params = getMlRuleParams(); const mlCompleteRule = getCompleteRuleMock(params); - + const exceptionItems = [getExceptionListItemSchemaMock()]; const tuple = { from: dateMath.parse(params.from)!, to: dateMath.parse(params.to)!, maxSignals: params.maxSignals, }; - const buildRuleMessage = buildRuleMessageFactory({ - id: mlCompleteRule.alertId, - ruleId: mlCompleteRule.ruleParams.ruleId, - name: mlCompleteRule.ruleConfig.name, - index: mlCompleteRule.ruleParams.outputIndex, - }); beforeEach(() => { jobsSummaryMock = jest.fn(); - alertServices = alertsMock.createRuleExecutorServices(); - logger = loggingSystemMock.createLogger(); mlMock = mlPluginServerMock.createSetupContract(); mlMock.jobServiceProvider.mockReturnValue({ jobsSummary: jobsSummaryMock, }); + alertServices = alertsMock.createRuleExecutorServices(); + ruleExecutionLogger = ruleExecutionLogMock.forExecutors.create({ + ruleId: mlCompleteRule.alertId, + ruleUuid: mlCompleteRule.ruleParams.ruleId, + ruleName: mlCompleteRule.ruleConfig.name, + ruleType: mlCompleteRule.ruleConfig.ruleTypeId, + }); (findMlSignals as jest.Mock).mockResolvedValue({ _shards: {}, hits: { @@ -73,8 +70,7 @@ describe('ml_executor', () => { ml: undefined, exceptionItems, services: alertServices, - logger, - buildRuleMessage, + ruleExecutionLogger, listClient: getListClientMock(), bulkCreate: jest.fn(), wrapHits: jest.fn(), @@ -90,14 +86,15 @@ describe('ml_executor', () => { ml: mlMock, exceptionItems, services: alertServices, - logger, - buildRuleMessage, + ruleExecutionLogger, listClient: getListClientMock(), bulkCreate: jest.fn(), wrapHits: jest.fn(), }); - expect(logger.warn).toHaveBeenCalled(); - expect(logger.warn.mock.calls[0][0]).toContain('Machine learning job(s) are not started'); + expect(ruleExecutionLogger.warn).toHaveBeenCalled(); + expect(ruleExecutionLogger.warn.mock.calls[0][0]).toContain( + 'Machine learning job(s) are not started' + ); expect(response.warningMessages.length).toEqual(1); }); @@ -116,14 +113,15 @@ describe('ml_executor', () => { ml: mlMock, exceptionItems, services: alertServices, - logger, - buildRuleMessage, + ruleExecutionLogger, listClient: getListClientMock(), bulkCreate: jest.fn(), wrapHits: jest.fn(), }); - expect(logger.warn).toHaveBeenCalled(); - expect(logger.warn.mock.calls[0][0]).toContain('Machine learning job(s) are not started'); + expect(ruleExecutionLogger.warn).toHaveBeenCalled(); + expect(ruleExecutionLogger.warn.mock.calls[0][0]).toContain( + 'Machine learning job(s) are not started' + ); expect(response.warningMessages.length).toEqual(1); }); }); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/ml.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/ml.ts index 4e568128e9f03..0c6c9b8181512 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/ml.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/ml.ts @@ -5,7 +5,7 @@ * 2.0. */ -import type { KibanaRequest, Logger } from '@kbn/core/server'; +import type { KibanaRequest } from '@kbn/core/server'; import type { ExceptionListItemSchema } from '@kbn/securitysolution-io-ts-list-types'; import type { AlertInstanceContext, @@ -18,11 +18,11 @@ import type { CompleteRule, MachineLearningRuleParams } from '../../schemas/rule import { bulkCreateMlSignals } from '../bulk_create_ml_signals'; import { filterEventsAgainstList } from '../filters/filter_events_against_list'; import { findMlSignals } from '../find_ml_signals'; -import type { BuildRuleMessage } from '../rule_messages'; import type { BulkCreate, RuleRangeTuple, WrapHits } from '../types'; import { createErrorsFromShard, createSearchAfterReturnType, mergeReturns } from '../utils'; import type { SetupPlugins } from '../../../../plugin'; import { withSecuritySpan } from '../../../../utils/with_security_span'; +import type { IRuleExecutionLogForExecutors } from '../../rule_monitoring'; export const mlExecutor = async ({ completeRule, @@ -31,8 +31,7 @@ export const mlExecutor = async ({ listClient, exceptionItems, services, - logger, - buildRuleMessage, + ruleExecutionLogger, bulkCreate, wrapHits, }: { @@ -42,8 +41,7 @@ export const mlExecutor = async ({ listClient: ListClient; exceptionItems: ExceptionListItemSchema[]; services: RuleExecutorServices; - logger: Logger; - buildRuleMessage: BuildRuleMessage; + ruleExecutionLogger: IRuleExecutionLogForExecutors; bulkCreate: BulkCreate; wrapHits: WrapHits; }) => { @@ -69,7 +67,7 @@ export const mlExecutor = async ({ jobSummaries.length < 1 || jobSummaries.some((job) => !isJobStarted(job.jobState, job.datafeedState)) ) { - const warningMessage = buildRuleMessage( + const warningMessage = [ 'Machine learning job(s) are not started:', ...jobSummaries.map((job) => [ @@ -77,10 +75,11 @@ export const mlExecutor = async ({ `job status: "${job.jobState}"`, `datafeed status: "${job.datafeedState}"`, ].join(', ') - ) - ); + ), + ].join(' '); + result.warningMessages.push(warningMessage); - logger.warn(warningMessage); + ruleExecutionLogger.warn(warningMessage); result.warning = true; } @@ -100,24 +99,23 @@ export const mlExecutor = async ({ const [filteredAnomalyHits, _] = await filterEventsAgainstList({ listClient, exceptionsList: exceptionItems, - logger, + ruleExecutionLogger, events: anomalyResults.hits.hits, - buildRuleMessage, }); const anomalyCount = filteredAnomalyHits.length; if (anomalyCount) { - logger.debug(buildRuleMessage(`Found ${anomalyCount} signals from ML anomalies.`)); + ruleExecutionLogger.debug(`Found ${anomalyCount} signals from ML anomalies`); } + const { success, errors, bulkCreateDuration, createdItemsCount, createdItems } = await bulkCreateMlSignals({ anomalyHits: filteredAnomalyHits, completeRule, services, - logger, + ruleExecutionLogger, id: completeRule.alertId, signalsIndex: ruleParams.outputIndex, - buildRuleMessage, bulkCreate, wrapHits, }); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/query.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/query.ts index fcfa61dcf3624..48e979472c4c7 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/query.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/query.ts @@ -5,7 +5,6 @@ * 2.0. */ -import type { Logger } from '@kbn/core/server'; import type { ExceptionListItemSchema } from '@kbn/securitysolution-io-ts-list-types'; import type { AlertInstanceContext, @@ -19,7 +18,6 @@ import { getFilter } from '../get_filter'; import { searchAfterAndBulkCreate } from '../search_after_bulk_create'; import type { RuleRangeTuple, BulkCreate, WrapHits } from '../types'; import type { ITelemetryEventsSender } from '../../../telemetry/sender'; -import type { BuildRuleMessage } from '../rule_messages'; import type { CompleteRule, SavedQueryRuleParams, @@ -28,21 +26,21 @@ import type { import type { ExperimentalFeatures } from '../../../../../common/experimental_features'; import { buildReasonMessageForQueryAlert } from '../reason_formatters'; import { withSecuritySpan } from '../../../../utils/with_security_span'; +import type { IRuleExecutionLogForExecutors } from '../../rule_monitoring'; export const queryExecutor = async ({ inputIndex, runtimeMappings, completeRule, tuple, - listClient, exceptionItems, + listClient, experimentalFeatures, + ruleExecutionLogger, + eventsTelemetry, services, version, searchAfterSize, - logger, - eventsTelemetry, - buildRuleMessage, bulkCreate, wrapHits, primaryTimestamp, @@ -52,15 +50,14 @@ export const queryExecutor = async ({ runtimeMappings: estypes.MappingRuntimeFields | undefined; completeRule: CompleteRule | CompleteRule; tuple: RuleRangeTuple; - listClient: ListClient; exceptionItems: ExceptionListItemSchema[]; + listClient: ListClient; experimentalFeatures: ExperimentalFeatures; + ruleExecutionLogger: IRuleExecutionLogForExecutors; + eventsTelemetry: ITelemetryEventsSender | undefined; services: RuleExecutorServices; version: string; searchAfterSize: number; - logger: Logger; - eventsTelemetry: ITelemetryEventsSender | undefined; - buildRuleMessage: BuildRuleMessage; bulkCreate: BulkCreate; wrapHits: WrapHits; primaryTimestamp: string; @@ -82,18 +79,16 @@ export const queryExecutor = async ({ return searchAfterAndBulkCreate({ tuple, - listClient, - exceptionsList: exceptionItems, completeRule, services, - logger, + listClient, + exceptionsList: exceptionItems, + ruleExecutionLogger, eventsTelemetry, - id: completeRule.alertId, inputIndexPattern: inputIndex, - filter: esFilter, pageSize: searchAfterSize, + filter: esFilter, buildReasonMessage: buildReasonMessageForQueryAlert, - buildRuleMessage, bulkCreate, wrapHits, runtimeMappings, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/threat_match.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/threat_match.ts index a25561c52e271..2d3cd8e078242 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/threat_match.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/threat_match.ts @@ -5,7 +5,6 @@ * 2.0. */ -import type { Logger } from '@kbn/core/server'; import type { ExceptionListItemSchema } from '@kbn/securitysolution-io-ts-list-types'; import type * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey'; @@ -17,27 +16,24 @@ import type { import type { ListClient } from '@kbn/lists-plugin/server'; import type { RuleRangeTuple, BulkCreate, WrapHits } from '../types'; import type { ITelemetryEventsSender } from '../../../telemetry/sender'; -import type { BuildRuleMessage } from '../rule_messages'; import { createThreatSignals } from '../threat_mapping/create_threat_signals'; import type { CompleteRule, ThreatRuleParams } from '../../schemas/rule_schemas'; -import type { ExperimentalFeatures } from '../../../../../common/experimental_features'; import { withSecuritySpan } from '../../../../utils/with_security_span'; import { DEFAULT_INDICATOR_SOURCE_PATH } from '../../../../../common/constants'; +import type { IRuleExecutionLogForExecutors } from '../../rule_monitoring'; export const threatMatchExecutor = async ({ inputIndex, runtimeMappings, completeRule, tuple, - listClient, exceptionItems, + listClient, services, version, searchAfterSize, - logger, + ruleExecutionLogger, eventsTelemetry, - experimentalFeatures, - buildRuleMessage, bulkCreate, wrapHits, primaryTimestamp, @@ -47,15 +43,13 @@ export const threatMatchExecutor = async ({ runtimeMappings: estypes.MappingRuntimeFields | undefined; completeRule: CompleteRule; tuple: RuleRangeTuple; - listClient: ListClient; exceptionItems: ExceptionListItemSchema[]; + listClient: ListClient; services: RuleExecutorServices; version: string; searchAfterSize: number; - logger: Logger; + ruleExecutionLogger: IRuleExecutionLogForExecutors; eventsTelemetry: ITelemetryEventsSender | undefined; - experimentalFeatures: ExperimentalFeatures; - buildRuleMessage: BuildRuleMessage; bulkCreate: BulkCreate; wrapHits: WrapHits; primaryTimestamp: string; @@ -66,7 +60,6 @@ export const threatMatchExecutor = async ({ return withSecuritySpan('threatMatchExecutor', async () => { return createThreatSignals({ alertId: completeRule.alertId, - buildRuleMessage, bulkCreate, completeRule, concurrentSearches: ruleParams.concurrentSearches ?? 1, @@ -77,9 +70,9 @@ export const threatMatchExecutor = async ({ itemsPerSearch: ruleParams.itemsPerSearch ?? 9000, language: ruleParams.language, listClient, - logger, outputIndex: ruleParams.outputIndex, query: ruleParams.query, + ruleExecutionLogger, savedId: ruleParams.savedId, searchAfterSize, services, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/threshold.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/threshold.test.ts index 7b57d3c24b8da..dd51f7aaef25d 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/threshold.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/threshold.test.ts @@ -6,7 +6,6 @@ */ import dateMath from '@kbn/datemath'; -import { loggingSystemMock } from '@kbn/core/server/mocks'; import { elasticsearchClientMock } from '@kbn/core-elasticsearch-client-server-mocks'; import type { RuleExecutorServicesMock } from '@kbn/alerting-plugin/server/mocks'; import { alertsMock } from '@kbn/alerting-plugin/server/mocks'; @@ -14,33 +13,26 @@ import { thresholdExecutor } from './threshold'; import { getExceptionListItemSchemaMock } from '@kbn/lists-plugin/common/schemas/response/exception_list_item_schema.mock'; import { getEntryListMock } from '@kbn/lists-plugin/common/schemas/types/entry_list.mock'; import { getThresholdRuleParams, getCompleteRuleMock } from '../../schemas/rule_schemas.mock'; -import { buildRuleMessageFactory } from '../rule_messages'; import { sampleEmptyAggsSearchResults } from '../__mocks__/es_results'; import { getThresholdTermsHash } from '../utils'; -import { allowedExperimentalValues } from '../../../../../common/experimental_features'; import type { ThresholdRuleParams } from '../../schemas/rule_schemas'; import { createRuleDataClientMock } from '@kbn/rule-registry-plugin/server/rule_data_client/rule_data_client.mock'; import { TIMESTAMP } from '@kbn/rule-data-utils'; +import type { IRuleExecutionLogForExecutors } from '../../rule_monitoring'; +import { ruleExecutionLogMock } from '../../rule_monitoring/mocks'; describe('threshold_executor', () => { - const version = '8.0.0'; - let logger: ReturnType; let alertServices: RuleExecutorServicesMock; - const params = getThresholdRuleParams(); + let ruleExecutionLogger: IRuleExecutionLogForExecutors; + const version = '8.0.0'; + const params = getThresholdRuleParams(); const thresholdCompleteRule = getCompleteRuleMock(params); - const tuple = { from: dateMath.parse(params.from)!, to: dateMath.parse(params.to)!, maxSignals: params.maxSignals, }; - const buildRuleMessage = buildRuleMessageFactory({ - id: thresholdCompleteRule.alertId, - ruleId: thresholdCompleteRule.ruleParams.ruleId, - name: thresholdCompleteRule.ruleConfig.name, - index: thresholdCompleteRule.ruleParams.outputIndex, - }); beforeEach(() => { alertServices = alertsMock.createRuleExecutorServices(); @@ -52,7 +44,12 @@ describe('threshold_executor', () => { }, }) ); - logger = loggingSystemMock.createLogger(); + ruleExecutionLogger = ruleExecutionLogMock.forExecutors.create({ + ruleId: thresholdCompleteRule.alertId, + ruleUuid: thresholdCompleteRule.ruleParams.ruleId, + ruleName: thresholdCompleteRule.ruleConfig.name, + ruleType: thresholdCompleteRule.ruleConfig.ruleTypeId, + }); }); describe('thresholdExecutor', () => { @@ -63,12 +60,10 @@ describe('threshold_executor', () => { completeRule: thresholdCompleteRule, tuple, exceptionItems, - experimentalFeatures: allowedExperimentalValues, services: alertServices, state: { initialized: true, signalHistory: {} }, version, - logger, - buildRuleMessage, + ruleExecutionLogger, startedAt: new Date(), bulkCreate: jest.fn().mockImplementation((hits) => ({ errors: [], @@ -119,12 +114,10 @@ describe('threshold_executor', () => { completeRule: thresholdCompleteRule, tuple, exceptionItems: [], - experimentalFeatures: allowedExperimentalValues, services: alertServices, state, version, - logger, - buildRuleMessage, + ruleExecutionLogger, startedAt: new Date(), bulkCreate: jest.fn().mockImplementation((hits) => ({ errors: [], diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/threshold.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/threshold.ts index 2cce95a5a9160..b5bf0cdc337a3 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/threshold.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/executors/threshold.ts @@ -9,8 +9,6 @@ import type { SearchHit } from '@elastic/elasticsearch/lib/api/typesWithBodyKey' import type * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey'; import type { ExceptionListItemSchema } from '@kbn/securitysolution-io-ts-list-types'; -import type { Logger } from '@kbn/core/server'; - import type { AlertInstanceContext, AlertInstanceState, @@ -34,10 +32,9 @@ import type { WrapHits, } from '../types'; import { createSearchAfterReturnType } from '../utils'; -import type { BuildRuleMessage } from '../rule_messages'; -import type { ExperimentalFeatures } from '../../../../../common/experimental_features'; import { withSecuritySpan } from '../../../../utils/with_security_span'; import { buildThresholdSignalHistory } from '../threshold/build_signal_history'; +import type { IRuleExecutionLogForExecutors } from '../../rule_monitoring'; export const thresholdExecutor = async ({ inputIndex, @@ -45,11 +42,9 @@ export const thresholdExecutor = async ({ completeRule, tuple, exceptionItems, - experimentalFeatures, + ruleExecutionLogger, services, version, - logger, - buildRuleMessage, startedAt, state, bulkCreate, @@ -63,11 +58,9 @@ export const thresholdExecutor = async ({ completeRule: CompleteRule; tuple: RuleRangeTuple; exceptionItems: ExceptionListItemSchema[]; - experimentalFeatures: ExperimentalFeatures; services: RuleExecutorServices; + ruleExecutionLogger: IRuleExecutionLogForExecutors; version: string; - logger: Logger; - buildRuleMessage: BuildRuleMessage; startedAt: Date; state: ThresholdAlertState; bulkCreate: BulkCreate; @@ -136,10 +129,9 @@ export const thresholdExecutor = async ({ to: tuple.to.toISOString(), maxSignals: tuple.maxSignals, services, - logger, + ruleExecutionLogger, filter: esFilter, threshold: ruleParams.threshold, - buildRuleMessage, runtimeMappings, primaryTimestamp, secondaryTimestamp, @@ -152,7 +144,6 @@ export const thresholdExecutor = async ({ completeRule, filter: esFilter, services, - logger, inputIndexPattern: inputIndex, signalsIndex: ruleParams.outputIndex, startedAt, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/create_field_and_set_tuples.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/create_field_and_set_tuples.test.ts index 47725bbeefcc2..d6b452216be92 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/create_field_and_set_tuples.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/create_field_and_set_tuples.test.ts @@ -6,18 +6,19 @@ */ import { createFieldAndSetTuples } from './create_field_and_set_tuples'; -import { mockLogger, sampleDocWithSortId } from '../__mocks__/es_results'; +import { sampleDocWithSortId } from '../__mocks__/es_results'; import { getExceptionListItemSchemaMock } from '@kbn/lists-plugin/common/schemas/response/exception_list_item_schema.mock'; import { listMock } from '@kbn/lists-plugin/server/mocks'; import { getSearchListItemResponseMock } from '@kbn/lists-plugin/common/schemas/response/search_list_item_schema.mock'; import type { EntryList } from '@kbn/securitysolution-io-ts-list-types'; -import { buildRuleMessageMock as buildRuleMessage } from '../rule_messages.mock'; +import { ruleExecutionLogMock } from '../../rule_monitoring/mocks'; describe('filterEventsAgainstList', () => { let listClient = listMock.getListClient(); let exceptionItem = getExceptionListItemSchemaMock(); let events = [sampleDocWithSortId('123', undefined, '1.1.1.1')]; + const ruleExecutionLogger = ruleExecutionLogMock.forExecutors.create(); beforeEach(() => { jest.clearAllMocks(); @@ -55,10 +56,9 @@ describe('filterEventsAgainstList', () => { exceptionItem.entries = []; const field = await createFieldAndSetTuples({ listClient, - logger: mockLogger, + ruleExecutionLogger, events, exceptionItem, - buildRuleMessage, }); expect(field).toEqual([]); }); @@ -66,10 +66,9 @@ describe('filterEventsAgainstList', () => { test('it returns a single field and set tuple if entries has a single item', async () => { const field = await createFieldAndSetTuples({ listClient, - logger: mockLogger, + ruleExecutionLogger, events, exceptionItem, - buildRuleMessage, }); expect(field.length).toEqual(1); }); @@ -78,10 +77,9 @@ describe('filterEventsAgainstList', () => { (exceptionItem.entries[0] as EntryList).operator = 'included'; const [{ operator }] = await createFieldAndSetTuples({ listClient, - logger: mockLogger, + ruleExecutionLogger, events, exceptionItem, - buildRuleMessage, }); expect(operator).toEqual('included'); }); @@ -90,10 +88,9 @@ describe('filterEventsAgainstList', () => { (exceptionItem.entries[0] as EntryList).operator = 'excluded'; const [{ operator }] = await createFieldAndSetTuples({ listClient, - logger: mockLogger, + ruleExecutionLogger, events, exceptionItem, - buildRuleMessage, }); expect(operator).toEqual('excluded'); }); @@ -102,10 +99,9 @@ describe('filterEventsAgainstList', () => { (exceptionItem.entries[0] as EntryList).field = 'source.ip'; const [{ field }] = await createFieldAndSetTuples({ listClient, - logger: mockLogger, + ruleExecutionLogger, events, exceptionItem, - buildRuleMessage, }); expect(field).toEqual('source.ip'); }); @@ -115,10 +111,9 @@ describe('filterEventsAgainstList', () => { (exceptionItem.entries[0] as EntryList).field = 'source.ip'; const [{ matchedSet }] = await createFieldAndSetTuples({ listClient, - logger: mockLogger, + ruleExecutionLogger, events, exceptionItem, - buildRuleMessage, }); expect([...matchedSet]).toEqual([JSON.stringify(['1.1.1.1'])]); }); @@ -131,10 +126,9 @@ describe('filterEventsAgainstList', () => { (exceptionItem.entries[0] as EntryList).field = 'source.ip'; const [{ matchedSet }] = await createFieldAndSetTuples({ listClient, - logger: mockLogger, + ruleExecutionLogger, events, exceptionItem, - buildRuleMessage, }); expect([...matchedSet]).toEqual([JSON.stringify(['1.1.1.1']), JSON.stringify(['2.2.2.2'])]); }); @@ -144,10 +138,9 @@ describe('filterEventsAgainstList', () => { (exceptionItem.entries[0] as EntryList).field = 'source.ip'; const [{ matchedSet }] = await createFieldAndSetTuples({ listClient, - logger: mockLogger, + ruleExecutionLogger, events, exceptionItem, - buildRuleMessage, }); expect([...matchedSet]).toEqual([JSON.stringify(['1.1.1.1', '2.2.2.2'])]); }); @@ -179,10 +172,9 @@ describe('filterEventsAgainstList', () => { ]; const fields = await createFieldAndSetTuples({ listClient, - logger: mockLogger, + ruleExecutionLogger, events, exceptionItem, - buildRuleMessage, }); expect(fields.length).toEqual(2); }); @@ -214,10 +206,9 @@ describe('filterEventsAgainstList', () => { ]; const [{ operator: operator1 }, { operator: operator2 }] = await createFieldAndSetTuples({ listClient, - logger: mockLogger, + ruleExecutionLogger, events, exceptionItem, - buildRuleMessage, }); expect(operator1).toEqual('included'); expect(operator2).toEqual('excluded'); @@ -250,10 +241,9 @@ describe('filterEventsAgainstList', () => { ]; const [{ field: field1 }, { field: field2 }] = await createFieldAndSetTuples({ listClient, - logger: mockLogger, + ruleExecutionLogger, events, exceptionItem, - buildRuleMessage, }); expect(field1).toEqual('source.ip'); expect(field2).toEqual('destination.ip'); @@ -287,10 +277,9 @@ describe('filterEventsAgainstList', () => { const [{ matchedSet: matchedSet1 }, { matchedSet: matchedSet2 }] = await createFieldAndSetTuples({ listClient, - logger: mockLogger, + ruleExecutionLogger, events, exceptionItem, - buildRuleMessage, }); expect([...matchedSet1]).toEqual([JSON.stringify(['1.1.1.1']), JSON.stringify(['2.2.2.2'])]); expect([...matchedSet2]).toEqual([JSON.stringify(['3.3.3.3']), JSON.stringify(['5.5.5.5'])]); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/create_field_and_set_tuples.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/create_field_and_set_tuples.ts index 0d58a4f7be078..6c04d2985f8d6 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/create_field_and_set_tuples.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/create_field_and_set_tuples.ts @@ -14,8 +14,7 @@ export const createFieldAndSetTuples = async ({ events, exceptionItem, listClient, - logger, - buildRuleMessage, + ruleExecutionLogger, }: CreateFieldAndSetTuplesOptions): Promise => { const typedEntries = exceptionItem.entries.filter((entry): entry is EntryList => entriesList.is(entry) @@ -30,8 +29,7 @@ export const createFieldAndSetTuples = async ({ listId: id, listType: type, listClient, - logger, - buildRuleMessage, + ruleExecutionLogger, }); return { field, operator, matchedSet }; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/create_set_to_filter_against.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/create_set_to_filter_against.test.ts index c6061d37f4279..d28bc2a39418d 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/create_set_to_filter_against.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/create_set_to_filter_against.test.ts @@ -5,16 +5,17 @@ * 2.0. */ -import { mockLogger, sampleDocWithSortId } from '../__mocks__/es_results'; +import { sampleDocWithSortId } from '../__mocks__/es_results'; import { listMock } from '@kbn/lists-plugin/server/mocks'; import { getSearchListItemResponseMock } from '@kbn/lists-plugin/common/schemas/response/search_list_item_schema.mock'; import { createSetToFilterAgainst } from './create_set_to_filter_against'; -import { buildRuleMessageMock as buildRuleMessage } from '../rule_messages.mock'; +import { ruleExecutionLogMock } from '../../rule_monitoring/mocks'; describe('createSetToFilterAgainst', () => { let listClient = listMock.getListClient(); let events = [sampleDocWithSortId('123', undefined, '1.1.1.1')]; + const ruleExecutionLogger = ruleExecutionLogMock.forExecutors.create(); beforeEach(() => { jest.clearAllMocks(); @@ -42,8 +43,7 @@ describe('createSetToFilterAgainst', () => { listId: 'list-123', listType: 'ip', listClient, - logger: mockLogger, - buildRuleMessage, + ruleExecutionLogger, }); expect([...field]).toEqual([]); }); @@ -56,8 +56,7 @@ describe('createSetToFilterAgainst', () => { listId: 'list-123', listType: 'ip', listClient, - logger: mockLogger, - buildRuleMessage, + ruleExecutionLogger, }); expect(listClient.searchListItemByValues).toHaveBeenCalledWith({ listId: 'list-123', @@ -78,8 +77,7 @@ describe('createSetToFilterAgainst', () => { listId: 'list-123', listType: 'ip', listClient, - logger: mockLogger, - buildRuleMessage, + ruleExecutionLogger, }); expect(listClient.searchListItemByValues).toHaveBeenCalledWith({ listId: 'list-123', @@ -100,8 +98,7 @@ describe('createSetToFilterAgainst', () => { listId: 'list-123', listType: 'ip', listClient, - logger: mockLogger, - buildRuleMessage, + ruleExecutionLogger, }); expect(listClient.searchListItemByValues).toHaveBeenCalledWith({ listId: 'list-123', diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/create_set_to_filter_against.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/create_set_to_filter_against.ts index 97369f269a7a0..e419d13589a57 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/create_set_to_filter_against.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/create_set_to_filter_against.ts @@ -26,8 +26,7 @@ export const createSetToFilterAgainst = async ({ listId, listType, listClient, - logger, - buildRuleMessage, + ruleExecutionLogger, }: CreateSetToFilterAgainstOptions): Promise> => { const valuesFromSearchResultField = events.reduce((acc, searchResultItem) => { const valueField = searchResultItem.fields ? searchResultItem.fields[field] : undefined; @@ -37,10 +36,8 @@ export const createSetToFilterAgainst = async ({ return acc; }, new Set()); - logger.debug( - buildRuleMessage( - `number of distinct values from ${field}: ${[...valuesFromSearchResultField].length}` - ) + ruleExecutionLogger.debug( + `number of distinct values from ${field}: ${[...valuesFromSearchResultField].length}` ); const matchedListItems = await listClient.searchListItemByValues({ @@ -49,10 +46,8 @@ export const createSetToFilterAgainst = async ({ value: [...valuesFromSearchResultField], }); - logger.debug( - buildRuleMessage( - `number of matched items from list with id ${listId}: ${matchedListItems.length}` - ) + ruleExecutionLogger.debug( + `number of matched items from list with id ${listId}: ${matchedListItems.length}` ); return new Set( diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/filter_events_against_list.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/filter_events_against_list.test.ts index 22dc5136fcded..b702b3ac63acc 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/filter_events_against_list.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/filter_events_against_list.test.ts @@ -6,18 +6,20 @@ */ import uuid from 'uuid'; -import { filterEventsAgainstList } from './filter_events_against_list'; -import { buildRuleMessageMock as buildRuleMessage } from '../rule_messages.mock'; -import { mockLogger, repeatedHitsWithSortId } from '../__mocks__/es_results'; import { getExceptionListItemSchemaMock } from '@kbn/lists-plugin/common/schemas/response/exception_list_item_schema.mock'; -import { listMock } from '@kbn/lists-plugin/server/mocks'; import { getSearchListItemResponseMock } from '@kbn/lists-plugin/common/schemas/response/search_list_item_schema.mock'; +import { listMock } from '@kbn/lists-plugin/server/mocks'; + +import { filterEventsAgainstList } from './filter_events_against_list'; +import { repeatedHitsWithSortId } from '../__mocks__/es_results'; +import { ruleExecutionLogMock } from '../../rule_monitoring/mocks'; const someGuids = Array.from({ length: 13 }).map((x) => uuid.v4()); describe('filterEventsAgainstList', () => { let listClient = listMock.getListClient(); + const ruleExecutionLogger = ruleExecutionLogMock.forExecutors.create(); beforeEach(() => { jest.clearAllMocks(); @@ -31,7 +33,7 @@ describe('filterEventsAgainstList', () => { it('should respond with eventSearchResult if exceptionList is empty array', async () => { const [included, excluded] = await filterEventsAgainstList({ - logger: mockLogger, + ruleExecutionLogger, listClient, exceptionsList: [], events: repeatedHitsWithSortId(4, someGuids.slice(0, 3), [ @@ -40,7 +42,6 @@ describe('filterEventsAgainstList', () => { '3.3.3.3', '7.7.7.7', ]), - buildRuleMessage, }); expect(included.length).toEqual(4); expect(excluded.length).toEqual(0); @@ -48,7 +49,7 @@ describe('filterEventsAgainstList', () => { it('should respond with eventSearchResult if exceptionList does not contain value list exceptions', async () => { const [included, excluded] = await filterEventsAgainstList({ - logger: mockLogger, + ruleExecutionLogger, listClient, exceptionsList: [getExceptionListItemSchemaMock()], events: repeatedHitsWithSortId(4, someGuids.slice(0, 3), [ @@ -57,11 +58,10 @@ describe('filterEventsAgainstList', () => { '3.3.3.3', '7.7.7.7', ]), - buildRuleMessage, }); expect(included.length).toEqual(4); expect(excluded.length).toEqual(0); - expect((mockLogger.debug as unknown as jest.Mock).mock.calls[0][0]).toContain( + expect(ruleExecutionLogger.debug.mock.calls[0][0]).toContain( 'no exception items of type list found - returning original search result' ); }); @@ -82,11 +82,10 @@ describe('filterEventsAgainstList', () => { ]; const [included, excluded] = await filterEventsAgainstList({ - logger: mockLogger, + ruleExecutionLogger, listClient, exceptionsList: [exceptionItem], events: repeatedHitsWithSortId(4, someGuids.slice(0, 3)), - buildRuleMessage, }); expect(included.length).toEqual(4); expect(excluded.length).toEqual(0); @@ -114,7 +113,7 @@ describe('filterEventsAgainstList', () => { ) ); const [included, excluded] = await filterEventsAgainstList({ - logger: mockLogger, + ruleExecutionLogger, listClient, exceptionsList: [exceptionItem], events: repeatedHitsWithSortId(4, someGuids.slice(0, 3), [ @@ -123,7 +122,6 @@ describe('filterEventsAgainstList', () => { '3.3.3.3', '7.7.7.7', ]), - buildRuleMessage, }); expect((listClient.searchListItemByValues as jest.Mock).mock.calls[0][0].type).toEqual('ip'); expect((listClient.searchListItemByValues as jest.Mock).mock.calls[0][0].listId).toEqual( @@ -175,7 +173,7 @@ describe('filterEventsAgainstList', () => { ]); const [included, excluded] = await filterEventsAgainstList({ - logger: mockLogger, + ruleExecutionLogger, listClient, exceptionsList: [exceptionItem, exceptionItemAgain], events: repeatedHitsWithSortId(9, someGuids.slice(0, 9), [ @@ -189,7 +187,6 @@ describe('filterEventsAgainstList', () => { '8.8.8.8', '9.9.9.9', ]), - buildRuleMessage, }); expect(listClient.searchListItemByValues as jest.Mock).toHaveBeenCalledTimes(2); expect(included.length).toEqual(6); @@ -237,7 +234,7 @@ describe('filterEventsAgainstList', () => { ]); const [included, excluded] = await filterEventsAgainstList({ - logger: mockLogger, + ruleExecutionLogger, listClient, exceptionsList: [exceptionItem, exceptionItemAgain], events: repeatedHitsWithSortId(9, someGuids.slice(0, 9), [ @@ -251,7 +248,6 @@ describe('filterEventsAgainstList', () => { '8.8.8.8', '9.9.9.9', ]), - buildRuleMessage, }); expect(listClient.searchListItemByValues as jest.Mock).toHaveBeenCalledTimes(2); // @ts-expect-error @@ -297,7 +293,7 @@ describe('filterEventsAgainstList', () => { ]); const [included, excluded] = await filterEventsAgainstList({ - logger: mockLogger, + ruleExecutionLogger, listClient, exceptionsList: [exceptionItem], events: repeatedHitsWithSortId( @@ -326,7 +322,6 @@ describe('filterEventsAgainstList', () => { '2.2.2.2', ] ), - buildRuleMessage, }); expect(listClient.searchListItemByValues as jest.Mock).toHaveBeenCalledTimes(2); expect(included.length).toEqual(8); @@ -375,7 +370,7 @@ describe('filterEventsAgainstList', () => { ]); const [included, excluded] = await filterEventsAgainstList({ - logger: mockLogger, + ruleExecutionLogger, listClient, exceptionsList: [exceptionItem], events: repeatedHitsWithSortId(9, someGuids.slice(0, 9), [ @@ -389,7 +384,6 @@ describe('filterEventsAgainstList', () => { '8.8.8.8', '9.9.9.9', ]), - buildRuleMessage, }); expect(listClient.searchListItemByValues as jest.Mock).toHaveBeenCalledTimes(2); expect(included.length).toEqual(9); @@ -443,7 +437,7 @@ describe('filterEventsAgainstList', () => { ]); const [included, excluded] = await filterEventsAgainstList({ - logger: mockLogger, + ruleExecutionLogger, listClient, exceptionsList: [exceptionItem], events: repeatedHitsWithSortId( @@ -460,7 +454,6 @@ describe('filterEventsAgainstList', () => { ['3.3.3.3', '4.4.4.4'], ] ), - buildRuleMessage, }); expect(listClient.searchListItemByValues as jest.Mock).toHaveBeenCalledTimes(2); expect((listClient.searchListItemByValues as jest.Mock).mock.calls[0][0].value).toEqual([ @@ -505,11 +498,10 @@ describe('filterEventsAgainstList', () => { }, ]; const [included, excluded] = await filterEventsAgainstList({ - logger: mockLogger, + ruleExecutionLogger, listClient, exceptionsList: [exceptionItem], events: repeatedHitsWithSortId(4, someGuids.slice(0, 3)), - buildRuleMessage, }); expect(included.length).toEqual(0); expect(excluded.length).toEqual(4); @@ -537,7 +529,7 @@ describe('filterEventsAgainstList', () => { ) ); const [included, excluded] = await filterEventsAgainstList({ - logger: mockLogger, + ruleExecutionLogger, listClient, exceptionsList: [exceptionItem], events: repeatedHitsWithSortId(4, someGuids.slice(0, 3), [ @@ -546,7 +538,6 @@ describe('filterEventsAgainstList', () => { '3.3.3.3', '7.7.7.7', ]), - buildRuleMessage, }); expect((listClient.searchListItemByValues as jest.Mock).mock.calls[0][0].type).toEqual('ip'); expect((listClient.searchListItemByValues as jest.Mock).mock.calls[0][0].listId).toEqual( @@ -592,7 +583,7 @@ describe('filterEventsAgainstList', () => { ]); const [included, excluded] = await filterEventsAgainstList({ - logger: mockLogger, + ruleExecutionLogger, listClient, exceptionsList: [exceptionItem], events: repeatedHitsWithSortId( @@ -609,7 +600,6 @@ describe('filterEventsAgainstList', () => { ['3.3.3.3', '4.4.4.4'], ] ), - buildRuleMessage, }); expect(listClient.searchListItemByValues as jest.Mock).toHaveBeenCalledTimes(2); expect((listClient.searchListItemByValues as jest.Mock).mock.calls[0][0].value).toEqual([ diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/filter_events_against_list.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/filter_events_against_list.ts index 91fa599e9e207..1093742d76d6e 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/filter_events_against_list.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/filter_events_against_list.ts @@ -32,15 +32,14 @@ import { createFieldAndSetTuples } from './create_field_and_set_tuples'; * * @param listClient The list client to use for queries * @param exceptionsList The exception list - * @param logger Logger for messages - * @param eventSearchResult The current events from the search + * @param ruleExecutionLogger Logger for messages + * @param events The current events from the search */ export const filterEventsAgainstList = async ({ listClient, exceptionsList, - logger, + ruleExecutionLogger, events, - buildRuleMessage, }: FilterEventsAgainstListOptions): Promise> => { try { const atLeastOneLargeValueList = exceptionsList.some(({ entries }) => @@ -48,8 +47,8 @@ export const filterEventsAgainstList = async ({ ); if (!atLeastOneLargeValueList) { - logger.debug( - buildRuleMessage('no exception items of type list found - returning original search result') + ruleExecutionLogger.debug( + 'no exception items of type list found - returning original search result' ); return [events, []]; } @@ -70,17 +69,14 @@ export const filterEventsAgainstList = async ({ events: includedEvents, exceptionItem, listClient, - logger, - buildRuleMessage, + ruleExecutionLogger, }); const [nextIncludedEvents, nextExcludedEvents] = partitionEvents({ events: includedEvents, fieldAndSetTuples, }); - logger.debug( - buildRuleMessage( - `Exception with id ${exceptionItem.id} filtered out ${nextExcludedEvents.length} events` - ) + ruleExecutionLogger.debug( + `Exception with id ${exceptionItem.id} filtered out ${nextExcludedEvents.length} events` ); return [nextIncludedEvents, [...excludedEvents, ...nextExcludedEvents]]; }, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/types.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/types.ts index 114dd5ee43f1c..f4f8aaf91f969 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/types.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/filters/types.ts @@ -5,18 +5,15 @@ * 2.0. */ import type * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey'; -import type { Logger } from '@kbn/core/server'; - import type { Type, ExceptionListItemSchema } from '@kbn/securitysolution-io-ts-list-types'; import type { ListClient } from '@kbn/lists-plugin/server'; -import type { BuildRuleMessage } from '../rule_messages'; +import type { IRuleExecutionLogForExecutors } from '../../rule_monitoring'; export interface FilterEventsAgainstListOptions { listClient: ListClient; exceptionsList: ExceptionListItemSchema[]; - logger: Logger; + ruleExecutionLogger: IRuleExecutionLogForExecutors; events: Array>; - buildRuleMessage: BuildRuleMessage; } export type FilterEventsAgainstListReturn = [ @@ -30,8 +27,7 @@ export interface CreateSetToFilterAgainstOptions { listId: string; listType: Type; listClient: ListClient; - logger: Logger; - buildRuleMessage: BuildRuleMessage; + ruleExecutionLogger: IRuleExecutionLogForExecutors; } export interface FilterEventsOptions { @@ -43,8 +39,7 @@ export interface CreateFieldAndSetTuplesOptions { events: Array>; exceptionItem: ExceptionListItemSchema; listClient: ListClient; - logger: Logger; - buildRuleMessage: BuildRuleMessage; + ruleExecutionLogger: IRuleExecutionLogForExecutors; } export interface FieldSet { diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/preview/preview_rule_execution_logger.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/preview/preview_rule_execution_logger.ts index d1d5209328344..978a43a29a878 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/preview/preview_rule_execution_logger.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/preview/preview_rule_execution_logger.ts @@ -6,29 +6,36 @@ */ import type { - RuleExecutionLogForExecutorsFactory, + IRuleExecutionLogService, RuleExecutionContext, StatusChangeArgs, -} from '../../rule_execution_log'; +} from '../../rule_monitoring'; + +export interface IPreviewRuleExecutionLogger { + factory: IRuleExecutionLogService['createClientForExecutors']; +} export const createPreviewRuleExecutionLogger = ( loggedStatusChanges: Array -) => { - const factory: RuleExecutionLogForExecutorsFactory = ( - savedObjectsClient, - eventLogService, - logger, - context - ) => { - return { - context, +): IPreviewRuleExecutionLogger => { + return { + factory: ({ context }) => { + const spyLogger = { + context, - logStatusChange(args: StatusChangeArgs): Promise { - loggedStatusChanges.push({ ...context, ...args }); - return Promise.resolve(); - }, - }; - }; + trace: () => {}, + debug: () => {}, + info: () => {}, + warn: () => {}, + error: () => {}, - return { factory }; + logStatusChange: (args: StatusChangeArgs): Promise => { + loggedStatusChanges.push({ ...context, ...args }); + return Promise.resolve(); + }, + }; + + return Promise.resolve(spyLogger); + }, + }; }; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/rule_messages.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/rule_messages.test.ts deleted file mode 100644 index 35cefcaad8189..0000000000000 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/rule_messages.test.ts +++ /dev/null @@ -1,61 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -import type { BuildRuleMessageFactoryParams } from './rule_messages'; -import { buildRuleMessageFactory } from './rule_messages'; - -describe('buildRuleMessageFactory', () => { - let factoryParams: BuildRuleMessageFactoryParams; - beforeEach(() => { - factoryParams = { - name: 'name', - id: 'id', - ruleId: 'ruleId', - index: 'index', - }; - }); - - it('appends rule attributes to the provided message', () => { - const buildMessage = buildRuleMessageFactory(factoryParams); - - const message = buildMessage('my message'); - expect(message).toEqual(expect.stringContaining('my message')); - expect(message).toEqual(expect.stringContaining('name: "name"')); - expect(message).toEqual(expect.stringContaining('id: "id"')); - expect(message).toEqual(expect.stringContaining('rule id: "ruleId"')); - expect(message).toEqual(expect.stringContaining('signals index: "index"')); - }); - - it('joins message parts with spaces', () => { - const buildMessage = buildRuleMessageFactory(factoryParams); - - const message = buildMessage('my message'); - expect(message).toEqual(expect.stringContaining('my message ')); - expect(message).toEqual(expect.stringContaining(' name: "name" ')); - expect(message).toEqual(expect.stringContaining(' id: "id" ')); - expect(message).toEqual(expect.stringContaining(' rule id: "ruleId" ')); - expect(message).toEqual(expect.stringContaining(' signals index: "index"')); - }); - - it('joins multiple arguments with spaces', () => { - const buildMessage = buildRuleMessageFactory(factoryParams); - - const message = buildMessage('my message', 'here is more'); - expect(message).toEqual(expect.stringContaining('my message ')); - expect(message).toEqual(expect.stringContaining(' here is more')); - }); - - it('defaults the rule ID if not provided ', () => { - const buildMessage = buildRuleMessageFactory({ - ...factoryParams, - ruleId: undefined, - }); - - const message = buildMessage('my message', 'here is more'); - expect(message).toEqual(expect.stringContaining('rule id: "(unknown rule id)"')); - }); -}); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/rule_messages.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/rule_messages.ts deleted file mode 100644 index 5f30220e71402..0000000000000 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/rule_messages.ts +++ /dev/null @@ -1,25 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -export type BuildRuleMessage = (...messages: string[]) => string; -export interface BuildRuleMessageFactoryParams { - name: string; - id: string; - ruleId: string | null | undefined; - index: string; -} - -export const buildRuleMessageFactory = - ({ id, ruleId, index, name }: BuildRuleMessageFactoryParams): BuildRuleMessage => - (...messages) => - [ - ...messages, - `name: "${name}"`, - `id: "${id}"`, - `rule id: "${ruleId ?? '(unknown rule id)'}"`, - `signals index: "${index}"`, - ].join(' '); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/search_after_bulk_create.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/search_after_bulk_create.test.ts index 51e98d7575444..9561d19fe4378 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/search_after_bulk_create.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/search_after_bulk_create.test.ts @@ -7,8 +7,6 @@ import { sampleEmptyDocSearchResults, - sampleRuleGuid, - mockLogger, repeatedSearchResultsWithSortId, repeatedSearchResultsWithNoSortId, sampleDocSearchResultsNoSortIdNoHits, @@ -28,7 +26,7 @@ import { elasticsearchClientMock } from '@kbn/core-elasticsearch-client-server-m import { getCompleteRuleMock, getQueryRuleParams } from '../schemas/rule_schemas.mock'; import { bulkCreateFactory } from '../rule_types/factories/bulk_create_factory'; import { wrapHitsFactory } from '../rule_types/factories/wrap_hits_factory'; -import { mockBuildRuleMessage } from './__mocks__/build_rule_message.mock'; +import { ruleExecutionLogMock } from '../rule_monitoring/mocks'; import type { BuildReasonMessage } from './reason_formatters'; import type { QueryRuleParams } from '../schemas/rule_schemas'; import { createPersistenceServicesMock } from '@kbn/rule-registry-plugin/server/utils/create_persistence_rule_type_wrapper.mock'; @@ -48,8 +46,6 @@ import { import { SERVER_APP_ID } from '../../../../common/constants'; import type { CommonAlertFieldsLatest } from '@kbn/rule-registry-plugin/common/schemas'; -const buildRuleMessage = mockBuildRuleMessage; - describe('searchAfterAndBulkCreate', () => { let mockService: RuleExecutorServicesMock; let mockPersistenceServices: jest.Mocked; @@ -58,6 +54,7 @@ describe('searchAfterAndBulkCreate', () => { let wrapHits: WrapHits; let inputIndexPattern: string[] = []; let listClient = listMock.getListClient(); + const ruleExecutionLogger = ruleExecutionLogMock.forExecutors.create(); const someGuids = Array.from({ length: 13 }).map(() => uuid.v4()); const sampleParams = getQueryRuleParams(); const queryCompleteRule = getCompleteRuleMock(sampleParams); @@ -78,6 +75,7 @@ describe('searchAfterAndBulkCreate', () => { }; sampleParams.maxSignals = 30; let tuple: RuleRangeTuple; + beforeEach(() => { jest.clearAllMocks(); buildReasonMessage = jest.fn().mockResolvedValue('some alert reason message'); @@ -86,21 +84,19 @@ describe('searchAfterAndBulkCreate', () => { inputIndexPattern = ['auditbeat-*']; mockService = alertsMock.createRuleExecutorServices(); tuple = getRuleRangeTuples({ - logger: mockLogger, previousStartedAt: new Date(), startedAt: new Date(), from: sampleParams.from, to: sampleParams.to, interval: '5m', maxSignals: sampleParams.maxSignals, - buildRuleMessage, + ruleExecutionLogger, }).tuples[0]; mockPersistenceServices = createPersistenceServicesMock(); bulkCreate = bulkCreateFactory( - mockLogger, mockPersistenceServices.alertWithPersistence, - buildRuleMessage, - false + false, + ruleExecutionLogger ); wrapHits = wrapHitsFactory({ completeRule: queryCompleteRule, @@ -209,14 +205,12 @@ describe('searchAfterAndBulkCreate', () => { listClient, exceptionsList: [exceptionItem], services: mockService, - logger: mockLogger, + ruleExecutionLogger, eventsTelemetry: undefined, - id: sampleRuleGuid, inputIndexPattern, pageSize: 1, filter: defaultFilter, buildReasonMessage, - buildRuleMessage, bulkCreate, wrapHits, runtimeMappings: undefined, @@ -306,14 +300,12 @@ describe('searchAfterAndBulkCreate', () => { listClient, exceptionsList: [exceptionItem], services: mockService, - logger: mockLogger, + ruleExecutionLogger, eventsTelemetry: undefined, - id: sampleRuleGuid, inputIndexPattern, pageSize: 1, filter: defaultFilter, buildReasonMessage, - buildRuleMessage, bulkCreate, wrapHits, runtimeMappings: undefined, @@ -383,14 +375,12 @@ describe('searchAfterAndBulkCreate', () => { listClient, exceptionsList: [exceptionItem], services: mockService, - logger: mockLogger, + ruleExecutionLogger, eventsTelemetry: undefined, - id: sampleRuleGuid, inputIndexPattern, pageSize: 1, filter: defaultFilter, buildReasonMessage, - buildRuleMessage, bulkCreate, wrapHits, runtimeMappings: undefined, @@ -444,14 +434,12 @@ describe('searchAfterAndBulkCreate', () => { listClient, exceptionsList: [exceptionItem], services: mockService, - logger: mockLogger, + ruleExecutionLogger, eventsTelemetry: undefined, - id: sampleRuleGuid, inputIndexPattern, pageSize: 1, filter: defaultFilter, buildReasonMessage, - buildRuleMessage, bulkCreate, wrapHits, runtimeMappings: undefined, @@ -515,14 +503,12 @@ describe('searchAfterAndBulkCreate', () => { listClient, exceptionsList: [], services: mockService, - logger: mockLogger, + ruleExecutionLogger, eventsTelemetry: undefined, - id: sampleRuleGuid, inputIndexPattern, pageSize: 1, filter: defaultFilter, buildReasonMessage, - buildRuleMessage, bulkCreate, wrapHits, runtimeMappings: undefined, @@ -572,14 +558,12 @@ describe('searchAfterAndBulkCreate', () => { listClient, exceptionsList: [exceptionItem], services: mockService, - logger: mockLogger, + ruleExecutionLogger, eventsTelemetry: undefined, - id: sampleRuleGuid, inputIndexPattern, pageSize: 1, filter: defaultFilter, buildReasonMessage, - buildRuleMessage, bulkCreate, wrapHits, runtimeMappings: undefined, @@ -643,14 +627,12 @@ describe('searchAfterAndBulkCreate', () => { listClient, exceptionsList: [exceptionItem], services: mockService, - logger: mockLogger, + ruleExecutionLogger, eventsTelemetry: undefined, - id: sampleRuleGuid, inputIndexPattern, pageSize: 1, filter: defaultFilter, buildReasonMessage, - buildRuleMessage, bulkCreate, wrapHits, runtimeMappings: undefined, @@ -716,14 +698,12 @@ describe('searchAfterAndBulkCreate', () => { listClient, exceptionsList: [], services: mockService, - logger: mockLogger, + ruleExecutionLogger, eventsTelemetry: undefined, - id: sampleRuleGuid, inputIndexPattern, pageSize: 1, filter: defaultFilter, buildReasonMessage, - buildRuleMessage, bulkCreate, wrapHits, runtimeMappings: undefined, @@ -765,14 +745,12 @@ describe('searchAfterAndBulkCreate', () => { tuple, completeRule: queryCompleteRule, services: mockService, - logger: mockLogger, + ruleExecutionLogger, eventsTelemetry: undefined, - id: sampleRuleGuid, inputIndexPattern, pageSize: 1, filter: defaultFilter, buildReasonMessage, - buildRuleMessage, bulkCreate, wrapHits, runtimeMappings: undefined, @@ -813,14 +791,12 @@ describe('searchAfterAndBulkCreate', () => { tuple, completeRule: queryCompleteRule, services: mockService, - logger: mockLogger, + ruleExecutionLogger, eventsTelemetry: undefined, - id: sampleRuleGuid, inputIndexPattern, pageSize: 1, filter: defaultFilter, buildReasonMessage, - buildRuleMessage, bulkCreate, wrapHits, runtimeMappings: undefined, @@ -943,14 +919,12 @@ describe('searchAfterAndBulkCreate', () => { listClient, exceptionsList: [], services: mockService, - logger: mockLogger, + ruleExecutionLogger, eventsTelemetry: undefined, - id: sampleRuleGuid, inputIndexPattern, pageSize: 1, filter: defaultFilter, buildReasonMessage, - buildRuleMessage, bulkCreate, wrapHits, runtimeMappings: undefined, @@ -1032,14 +1006,12 @@ describe('searchAfterAndBulkCreate', () => { listClient, exceptionsList: [], services: mockService, - logger: mockLogger, + ruleExecutionLogger, eventsTelemetry: undefined, - id: sampleRuleGuid, inputIndexPattern, pageSize: 1, filter: defaultFilter, buildReasonMessage, - buildRuleMessage, bulkCreate, wrapHits, runtimeMappings: undefined, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/search_after_bulk_create.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/search_after_bulk_create.ts index afd2bb32ed8ae..6b6dedc302c0f 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/search_after_bulk_create.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/search_after_bulk_create.ts @@ -25,7 +25,6 @@ import { withSecuritySpan } from '../../../utils/with_security_span'; // search_after through documents and re-index using bulk endpoint. export const searchAfterAndBulkCreate = async ({ buildReasonMessage, - buildRuleMessage, bulkCreate, completeRule, enrichment = identity, @@ -34,8 +33,8 @@ export const searchAfterAndBulkCreate = async ({ filter, inputIndexPattern, listClient, - logger, pageSize, + ruleExecutionLogger, services, sortOrder, trackTotalHits, @@ -57,7 +56,7 @@ export const searchAfterAndBulkCreate = async ({ let signalsCreatedCount = 0; if (tuple == null || tuple.to == null || tuple.from == null) { - logger.error(buildRuleMessage(`[-] malformed date tuple`)); + ruleExecutionLogger.error(`[-] malformed date tuple`); return createSearchAfterReturnType({ success: false, errors: ['malformed date tuple'], @@ -68,17 +67,17 @@ export const searchAfterAndBulkCreate = async ({ while (signalsCreatedCount < tuple.maxSignals) { try { let mergedSearchResults = createSearchResultReturnType(); - logger.debug(buildRuleMessage(`sortIds: ${sortIds}`)); + ruleExecutionLogger.debug(`sortIds: ${sortIds}`); + if (hasSortId) { const { searchResult, searchDuration, searchErrors } = await singleSearchAfter({ - buildRuleMessage, searchAfterSortIds: sortIds, index: inputIndexPattern, runtimeMappings, from: tuple.from.toISOString(), to: tuple.to.toISOString(), services, - logger, + ruleExecutionLogger, filter, pageSize: Math.ceil(Math.min(tuple.maxSignals, pageSize)), primaryTimestamp, @@ -112,18 +111,16 @@ export const searchAfterAndBulkCreate = async ({ // determine if there are any candidate signals to be processed const totalHits = getTotalHitsValue(mergedSearchResults.hits.total); - logger.debug(buildRuleMessage(`totalHits: ${totalHits}`)); - logger.debug( - buildRuleMessage(`searchResult.hit.hits.length: ${mergedSearchResults.hits.hits.length}`) + ruleExecutionLogger.debug(`totalHits: ${totalHits}`); + ruleExecutionLogger.debug( + `searchResult.hit.hits.length: ${mergedSearchResults.hits.hits.length}` ); if (totalHits === 0 || mergedSearchResults.hits.hits.length === 0) { - logger.debug( - buildRuleMessage( - `${ - totalHits === 0 ? 'totalHits' : 'searchResult.hits.hits.length' - } was 0, exiting early` - ) + ruleExecutionLogger.debug( + `${ + totalHits === 0 ? 'totalHits' : 'searchResult.hits.hits.length' + } was 0, exiting early` ); break; } @@ -134,9 +131,8 @@ export const searchAfterAndBulkCreate = async ({ const [includedEvents, _] = await filterEventsAgainstList({ listClient, exceptionsList, - logger, + ruleExecutionLogger, events: mergedSearchResults.hits.hits, - buildRuleMessage, }); // only bulk create if there are filteredEvents leftover @@ -168,25 +164,25 @@ export const searchAfterAndBulkCreate = async ({ }), ]); signalsCreatedCount += createdCount; - logger.debug(buildRuleMessage(`created ${createdCount} signals`)); - logger.debug(buildRuleMessage(`signalsCreatedCount: ${signalsCreatedCount}`)); - logger.debug(buildRuleMessage(`enrichedEvents.hits.hits: ${enrichedEvents.length}`)); + + ruleExecutionLogger.debug(`created ${createdCount} signals`); + ruleExecutionLogger.debug(`signalsCreatedCount: ${signalsCreatedCount}`); + ruleExecutionLogger.debug(`enrichedEvents.hits.hits: ${enrichedEvents.length}`); sendAlertTelemetryEvents( - logger, - eventsTelemetry, enrichedEvents, createdItems, - buildRuleMessage + eventsTelemetry, + ruleExecutionLogger ); } if (!hasSortId) { - logger.debug(buildRuleMessage('ran out of sort ids to sort on')); + ruleExecutionLogger.debug('ran out of sort ids to sort on'); break; } } catch (exc: unknown) { - logger.error(buildRuleMessage(`[-] search_after_bulk_create threw an error ${exc}`)); + ruleExecutionLogger.error(`[-] search_after_bulk_create threw an error ${exc}`); return mergeReturns([ toReturn, createSearchAfterReturnType({ @@ -196,7 +192,7 @@ export const searchAfterAndBulkCreate = async ({ ]); } } - logger.debug(buildRuleMessage(`[+] completed bulk index of ${toReturn.createdSignalsCount}`)); + ruleExecutionLogger.debug(`[+] completed bulk index of ${toReturn.createdSignalsCount}`); return toReturn; }); }; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/send_telemetry_events.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/send_telemetry_events.ts index a8ded1ea3f063..65742d5145110 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/send_telemetry_events.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/send_telemetry_events.ts @@ -5,10 +5,9 @@ * 2.0. */ -import type { Logger } from '@kbn/core/server'; import type { ITelemetryEventsSender } from '../../telemetry/sender'; import type { TelemetryEvent } from '../../telemetry/types'; -import type { BuildRuleMessage } from './rule_messages'; +import type { IRuleExecutionLogForExecutors } from '../rule_monitoring'; import type { SignalSource, SignalSourceHit } from './types'; interface SearchResultSource { @@ -44,11 +43,10 @@ export function enrichEndpointAlertsSignalID( } export function sendAlertTelemetryEvents( - logger: Logger, - eventsTelemetry: ITelemetryEventsSender | undefined, filteredEvents: SignalSourceHit[], createdEvents: SignalSource[], - buildRuleMessage: BuildRuleMessage + eventsTelemetry: ITelemetryEventsSender | undefined, + ruleExecutionLogger: IRuleExecutionLogForExecutors ) { if (eventsTelemetry === undefined) { return; @@ -74,6 +72,6 @@ export function sendAlertTelemetryEvents( try { eventsTelemetry.queueTelemetryEvents(selectedEvents); } catch (exc) { - logger.error(buildRuleMessage(`[-] queing telemetry events failed ${exc}`)); + ruleExecutionLogger.error(`[-] queing telemetry events failed ${exc}`); } } diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/single_search_after.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/single_search_after.test.ts index b75eb81eb4163..28b00e45dd5a3 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/single_search_after.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/single_search_after.test.ts @@ -7,23 +7,17 @@ import type * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey'; import { sampleDocSearchResultsNoSortId, - mockLogger, sampleDocSearchResultsWithSortId, } from './__mocks__/es_results'; import { singleSearchAfter } from './single_search_after'; import type { RuleExecutorServicesMock } from '@kbn/alerting-plugin/server/mocks'; import { alertsMock } from '@kbn/alerting-plugin/server/mocks'; -import { buildRuleMessageFactory } from './rule_messages'; import { elasticsearchClientMock } from '@kbn/core-elasticsearch-client-server-mocks'; +import { ruleExecutionLogMock } from '../rule_monitoring/mocks'; -const buildRuleMessage = buildRuleMessageFactory({ - id: 'fake id', - ruleId: 'fake rule id', - index: 'fakeindex', - name: 'fake name', -}); describe('singleSearchAfter', () => { const mockService: RuleExecutorServicesMock = alertsMock.createRuleExecutorServices(); + const ruleExecutionLogger = ruleExecutionLogMock.forExecutors.create(); beforeEach(() => { jest.clearAllMocks(); @@ -39,12 +33,11 @@ describe('singleSearchAfter', () => { from: 'now-360s', to: 'now', services: mockService, - logger: mockLogger, + ruleExecutionLogger, pageSize: 1, filter: {}, primaryTimestamp: '@timestamp', secondaryTimestamp: undefined, - buildRuleMessage, runtimeMappings: undefined, }); expect(searchResult).toEqual(sampleDocSearchResultsNoSortId()); @@ -59,12 +52,11 @@ describe('singleSearchAfter', () => { from: 'now-360s', to: 'now', services: mockService, - logger: mockLogger, + ruleExecutionLogger, pageSize: 1, filter: {}, primaryTimestamp: '@timestamp', secondaryTimestamp: undefined, - buildRuleMessage, runtimeMappings: undefined, }); expect(searchErrors).toEqual([]); @@ -111,12 +103,11 @@ describe('singleSearchAfter', () => { from: 'now-360s', to: 'now', services: mockService, - logger: mockLogger, + ruleExecutionLogger, pageSize: 1, filter: {}, primaryTimestamp: '@timestamp', secondaryTimestamp: undefined, - buildRuleMessage, runtimeMappings: undefined, }); expect(searchErrors).toEqual([ @@ -136,12 +127,11 @@ describe('singleSearchAfter', () => { from: 'now-360s', to: 'now', services: mockService, - logger: mockLogger, + ruleExecutionLogger, pageSize: 1, filter: {}, primaryTimestamp: '@timestamp', secondaryTimestamp: undefined, - buildRuleMessage, runtimeMappings: undefined, }); expect(searchResult).toEqual(sampleDocSearchResultsWithSortId()); @@ -158,12 +148,11 @@ describe('singleSearchAfter', () => { from: 'now-360s', to: 'now', services: mockService, - logger: mockLogger, + ruleExecutionLogger, pageSize: 1, filter: {}, primaryTimestamp: '@timestamp', secondaryTimestamp: undefined, - buildRuleMessage, runtimeMappings: undefined, }) ).rejects.toThrow('Fake Error'); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/single_search_after.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/single_search_after.ts index ae5b4824d282e..0a0534e887c5e 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/single_search_after.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/single_search_after.ts @@ -11,9 +11,7 @@ import type { AlertInstanceState, RuleExecutorServices, } from '@kbn/alerting-plugin/server'; -import type { Logger } from '@kbn/core/server'; import type { SignalSearchResponse, SignalSource } from './types'; -import type { BuildRuleMessage } from './rule_messages'; import { buildEventsSearchQuery } from './build_events_query'; import { createErrorsFromShard, makeFloatString } from './utils'; import type { @@ -21,6 +19,7 @@ import type { TimestampOverrideOrUndefined, } from '../../../../common/detection_engine/schemas/common/schemas'; import { withSecuritySpan } from '../../../utils/with_security_span'; +import type { IRuleExecutionLogForExecutors } from '../rule_monitoring'; interface SingleSearchAfterParams { aggregations?: Record; @@ -29,13 +28,12 @@ interface SingleSearchAfterParams { from: string; to: string; services: RuleExecutorServices; - logger: Logger; + ruleExecutionLogger: IRuleExecutionLogForExecutors; pageSize: number; sortOrder?: estypes.SortOrder; filter: estypes.QueryDslQueryContainer; primaryTimestamp: TimestampOverride; secondaryTimestamp: TimestampOverrideOrUndefined; - buildRuleMessage: BuildRuleMessage; trackTotalHits?: boolean; runtimeMappings: estypes.MappingRuntimeFields | undefined; } @@ -52,12 +50,11 @@ export const singleSearchAfter = async < to, services, filter, - logger, + ruleExecutionLogger, pageSize, sortOrder, primaryTimestamp, secondaryTimestamp, - buildRuleMessage, trackTotalHits, }: SingleSearchAfterParams): Promise<{ searchResult: SignalSearchResponse; @@ -99,13 +96,13 @@ export const singleSearchAfter = async < searchErrors, }; } catch (exc) { - logger.error(buildRuleMessage(`[-] nextSearchAfter threw an error ${exc}`)); + ruleExecutionLogger.error(`[-] nextSearchAfter threw an error ${exc}`); if ( exc.message.includes(`No mapping found for [${primaryTimestamp}] in order to sort on`) || (secondaryTimestamp && exc.message.includes(`No mapping found for [${secondaryTimestamp}] in order to sort on`)) ) { - logger.error(buildRuleMessage(`[-] failure reason: ${exc.message}`)); + ruleExecutionLogger.error(`[-] failure reason: ${exc.message}`); const searchRes: SignalSearchResponse = { took: 0, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/build_threat_enrichment.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/build_threat_enrichment.ts index ec431a23e0e54..fc5ec50c6bc6f 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/build_threat_enrichment.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/build_threat_enrichment.ts @@ -11,9 +11,8 @@ import type { BuildThreatEnrichmentOptions, GetMatchedThreats } from './types'; import { getThreatList } from './get_threat_list'; export const buildThreatEnrichment = ({ - buildRuleMessage, exceptionItems, - logger, + ruleExecutionLogger, services, threatFilters, threatIndex, @@ -37,14 +36,13 @@ export const buildThreatEnrichment = ({ const threatResponse = await getThreatList({ esClient: services.scopedClusterClient.asCurrentUser, exceptionItems, - threatFilters: [...threatFilters, matchedThreatsFilter], - query: threatQuery, - language: threatLanguage, index: threatIndex, - searchAfter: undefined, - logger, - buildRuleMessage, + language: threatLanguage, perPage: undefined, + query: threatQuery, + ruleExecutionLogger, + searchAfter: undefined, + threatFilters: [...threatFilters, matchedThreatsFilter], threatListConfig: { _source: [`${threatIndicatorPath}.*`, 'threat.feed.*'], fields: undefined, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/create_event_signal.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/create_event_signal.ts index 67e39e71421df..a1a63b2e2493c 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/create_event_signal.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/create_event_signal.ts @@ -19,7 +19,6 @@ import { export const createEventSignal = async ({ alertId, - buildRuleMessage, bulkCreate, completeRule, currentResult, @@ -30,9 +29,9 @@ export const createEventSignal = async ({ inputIndex, language, listClient, - logger, outputIndex, query, + ruleExecutionLogger, savedId, searchAfterSize, services, @@ -60,10 +59,8 @@ export const createEventSignal = async ({ if (!threatFilter.query || threatFilter.query?.bool.should.length === 0) { // empty event list and we do not want to return everything as being // a hit so opt to return the existing result. - logger.debug( - buildRuleMessage( - 'Indicator items are empty after filtering for missing data, returning without attempting a match' - ) + ruleExecutionLogger.debug( + 'Indicator items are empty after filtering for missing data, returning without attempting a match' ); return currentResult; } else { @@ -74,8 +71,7 @@ export const createEventSignal = async ({ query: threatQuery, language: threatLanguage, index: threatIndex, - logger, - buildRuleMessage, + ruleExecutionLogger, threatListConfig: { _source: [`${threatIndicatorPath}.*`, 'threat.feed.*'], fields: undefined, @@ -111,10 +107,8 @@ export const createEventSignal = async ({ lists: exceptionItems, }); - logger.debug( - buildRuleMessage( - `${ids?.length} matched signals found from ${threatListHits.length} indicators` - ) + ruleExecutionLogger.debug( + `${ids?.length} matched signals found from ${threatListHits.length} indicators` ); const threatEnrichment = (signals: SignalSourceHit[]): Promise => @@ -127,18 +121,16 @@ export const createEventSignal = async ({ const result = await searchAfterAndBulkCreate({ buildReasonMessage: buildReasonMessageForThreatMatchAlert, - buildRuleMessage, bulkCreate, completeRule, enrichment: threatEnrichment, eventsTelemetry, exceptionsList: exceptionItems, filter: esFilter, - id: alertId, inputIndexPattern: inputIndex, listClient, - logger, pageSize: searchAfterSize, + ruleExecutionLogger, services, sortOrder: 'desc', trackTotalHits: false, @@ -149,14 +141,12 @@ export const createEventSignal = async ({ secondaryTimestamp, }); - logger.debug( - buildRuleMessage( - `${ - threatFilter.query?.bool.should.length - } items have completed match checks and the total times to search were ${ - result.searchAfterTimes.length !== 0 ? result.searchAfterTimes : '(unknown) ' - }ms` - ) + ruleExecutionLogger.debug( + `${ + threatFilter.query?.bool.should.length + } items have completed match checks and the total times to search were ${ + result.searchAfterTimes.length !== 0 ? result.searchAfterTimes : '(unknown) ' + }ms` ); return result; } diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/create_threat_signal.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/create_threat_signal.ts index faf61c09ca79b..b063ef87761bc 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/create_threat_signal.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/create_threat_signal.ts @@ -15,7 +15,6 @@ import type { SearchAfterAndBulkCreateReturnType } from '../types'; export const createThreatSignal = async ({ alertId, - buildRuleMessage, bulkCreate, completeRule, currentResult, @@ -26,9 +25,9 @@ export const createThreatSignal = async ({ inputIndex, language, listClient, - logger, outputIndex, query, + ruleExecutionLogger, savedId, searchAfterSize, services, @@ -50,10 +49,8 @@ export const createThreatSignal = async ({ if (!threatFilter.query || threatFilter.query?.bool.should.length === 0) { // empty threat list and we do not want to return everything as being // a hit so opt to return the existing result. - logger.debug( - buildRuleMessage( - 'Indicator items are empty after filtering for missing data, returning without attempting a match' - ) + ruleExecutionLogger.debug( + 'Indicator items are empty after filtering for missing data, returning without attempting a match' ); return currentResult; } else { @@ -68,26 +65,22 @@ export const createThreatSignal = async ({ lists: exceptionItems, }); - logger.debug( - buildRuleMessage( - `${threatFilter.query?.bool.should.length} indicator items are being checked for existence of matches` - ) + ruleExecutionLogger.debug( + `${threatFilter.query?.bool.should.length} indicator items are being checked for existence of matches` ); const result = await searchAfterAndBulkCreate({ buildReasonMessage: buildReasonMessageForThreatMatchAlert, - buildRuleMessage, bulkCreate, completeRule, enrichment: threatEnrichment, eventsTelemetry, exceptionsList: exceptionItems, filter: esFilter, - id: alertId, inputIndexPattern: inputIndex, listClient, - logger, pageSize: searchAfterSize, + ruleExecutionLogger, services, sortOrder: 'desc', trackTotalHits: false, @@ -98,14 +91,12 @@ export const createThreatSignal = async ({ secondaryTimestamp, }); - logger.debug( - buildRuleMessage( - `${ - threatFilter.query?.bool.should.length - } items have completed match checks and the total times to search were ${ - result.searchAfterTimes.length !== 0 ? result.searchAfterTimes : '(unknown) ' - }ms` - ) + ruleExecutionLogger.debug( + `${ + threatFilter.query?.bool.should.length + } items have completed match checks and the total times to search were ${ + result.searchAfterTimes.length !== 0 ? result.searchAfterTimes : '(unknown) ' + }ms` ); return result; } diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/create_threat_signals.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/create_threat_signals.ts index 579bf1ca8859c..f0f303e7c837f 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/create_threat_signals.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/create_threat_signals.ts @@ -25,7 +25,6 @@ import { THREAT_PIT_KEEP_ALIVE } from '../../../../../common/cti/constants'; export const createThreatSignals = async ({ alertId, - buildRuleMessage, bulkCreate, completeRule, concurrentSearches, @@ -36,9 +35,9 @@ export const createThreatSignals = async ({ itemsPerSearch, language, listClient, - logger, outputIndex, query, + ruleExecutionLogger, savedId, searchAfterSize, services, @@ -56,7 +55,7 @@ export const createThreatSignals = async ({ secondaryTimestamp, }: CreateThreatSignalsOptions): Promise => { const params = completeRule.ruleParams; - logger.debug(buildRuleMessage('Indicator matching rule starting')); + ruleExecutionLogger.debug('Indicator matching rule starting'); const perPage = concurrentSearches * itemsPerSearch; const verifyExecutionCanProceed = buildExecutionIntervalValidator( completeRule.ruleConfig.schedule.interval @@ -90,10 +89,10 @@ export const createThreatSignals = async ({ secondaryTimestamp, }); - logger.debug(`Total event count: ${eventCount}`); + ruleExecutionLogger.debug(`Total event count: ${eventCount}`); // if (eventCount === 0) { - // logger.debug(buildRuleMessage('Indicator matching rule has completed')); + // ruleExecutionLogger.debug('Indicator matching rule has completed'); // return results; // } @@ -116,7 +115,7 @@ export const createThreatSignals = async ({ index: threatIndex, }); - logger.debug(buildRuleMessage(`Total indicator items: ${threatListCount}`)); + ruleExecutionLogger.debug(`Total indicator items: ${threatListCount}`); const threatListConfig = { fields: threatMapping.map((mapping) => mapping.entries.map((item) => item.value)).flat(), @@ -124,9 +123,8 @@ export const createThreatSignals = async ({ }; const threatEnrichment = buildThreatEnrichment({ - buildRuleMessage, exceptionItems, - logger, + ruleExecutionLogger, services, threatFilters: allThreatFilters, threatIndex, @@ -153,31 +151,25 @@ export const createThreatSignals = async ({ while (list.hits.hits.length !== 0) { verifyExecutionCanProceed(); const chunks = chunk(itemsPerSearch, list.hits.hits); - logger.debug( - buildRuleMessage(`${chunks.length} concurrent indicator searches are starting.`) - ); + ruleExecutionLogger.debug(`${chunks.length} concurrent indicator searches are starting.`); const concurrentSearchesPerformed = chunks.map>(createSignal); const searchesPerformed = await Promise.all(concurrentSearchesPerformed); results = combineConcurrentResults(results, searchesPerformed); documentCount -= list.hits.hits.length; - logger.debug( - buildRuleMessage( - `Concurrent indicator match searches completed with ${results.createdSignalsCount} signals found`, - `search times of ${results.searchAfterTimes}ms,`, - `bulk create times ${results.bulkCreateTimes}ms,`, - `all successes are ${results.success}` - ) + ruleExecutionLogger.debug( + `Concurrent indicator match searches completed with ${results.createdSignalsCount} signals found`, + `search times of ${results.searchAfterTimes}ms,`, + `bulk create times ${results.bulkCreateTimes}ms,`, + `all successes are ${results.success}` ); if (results.createdSignalsCount >= params.maxSignals) { - logger.debug( - buildRuleMessage( - `Indicator match has reached its max signals count ${params.maxSignals}. Additional documents not checked are ${documentCount}` - ) + ruleExecutionLogger.debug( + `Indicator match has reached its max signals count ${params.maxSignals}. Additional documents not checked are ${documentCount}` ); break; } - logger.debug(buildRuleMessage(`Documents items left to check are ${documentCount}`)); + ruleExecutionLogger.debug(`Documents items left to check are ${documentCount}`); list = await getDocumentList({ searchAfter: list.hits.hits[list.hits.hits.length - 1].sort, @@ -191,14 +183,13 @@ export const createThreatSignals = async ({ getDocumentList: async ({ searchAfter }) => getEventList({ services, + ruleExecutionLogger, exceptionItems, filters: allEventFilters, query, language, index: inputIndex, searchAfter, - logger, - buildRuleMessage, perPage, tuple, runtimeMappings, @@ -209,7 +200,6 @@ export const createThreatSignals = async ({ createSignal: (slicedChunk) => createEventSignal({ alertId, - buildRuleMessage, bulkCreate, completeRule, currentEventList: slicedChunk, @@ -220,10 +210,10 @@ export const createThreatSignals = async ({ inputIndex, language, listClient, - logger, outputIndex, query, reassignThreatPitId, + ruleExecutionLogger, savedId, searchAfterSize, services, @@ -255,8 +245,7 @@ export const createThreatSignals = async ({ language: threatLanguage, index: threatIndex, searchAfter, - logger, - buildRuleMessage, + ruleExecutionLogger, perPage, threatListConfig, pitId: threatPitId, @@ -268,7 +257,6 @@ export const createThreatSignals = async ({ createSignal: (slicedChunk) => createThreatSignal({ alertId, - buildRuleMessage, bulkCreate, completeRule, currentResult: results, @@ -279,9 +267,9 @@ export const createThreatSignals = async ({ inputIndex, language, listClient, - logger, outputIndex, query, + ruleExecutionLogger, savedId, searchAfterSize, services, @@ -301,11 +289,11 @@ export const createThreatSignals = async ({ await services.scopedClusterClient.asCurrentUser.closePointInTime({ id: threatPitId }); } catch (error) { // Don't fail due to a bad point in time closure. We have seen failures in e2e tests during nominal operations. - logger.warn( + ruleExecutionLogger.warn( `Error trying to close point in time: "${threatPitId}", it will expire within "${THREAT_PIT_KEEP_ALIVE}". Error is: "${error}"` ); } - logger.debug(buildRuleMessage('Indicator matching rule has completed')); + ruleExecutionLogger.debug('Indicator matching rule has completed'); return results; }; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/get_event_count.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/get_event_count.ts index f01722612559a..347b8fa4c5d63 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/get_event_count.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/get_event_count.ts @@ -15,6 +15,7 @@ export const MAX_PER_PAGE = 9000; export const getEventList = async ({ services, + ruleExecutionLogger, query, language, index, @@ -22,8 +23,6 @@ export const getEventList = async ({ searchAfter, exceptionItems, filters, - buildRuleMessage, - logger, tuple, primaryTimestamp, secondaryTimestamp, @@ -34,22 +33,19 @@ export const getEventList = async ({ throw new TypeError('perPage cannot exceed the size of 10000'); } - logger.debug( - buildRuleMessage( - `Querying the events items from the index: "${index}" with searchAfter: "${searchAfter}" for up to ${calculatedPerPage} indicator items` - ) + ruleExecutionLogger.debug( + `Querying the events items from the index: "${index}" with searchAfter: "${searchAfter}" for up to ${calculatedPerPage} indicator items` ); const filter = getQueryFilter(query, language ?? 'kuery', filters, index, exceptionItems); const { searchResult } = await singleSearchAfter({ - buildRuleMessage, searchAfterSortIds: searchAfter, index, from: tuple.from.toISOString(), to: tuple.to.toISOString(), services, - logger, + ruleExecutionLogger, filter, pageSize: Math.ceil(Math.min(tuple.maxSignals, calculatedPerPage)), primaryTimestamp, @@ -59,9 +55,7 @@ export const getEventList = async ({ runtimeMappings, }); - logger.debug( - buildRuleMessage(`Retrieved events items of size: ${searchResult.hits.hits.length}`) - ); + ruleExecutionLogger.debug(`Retrieved events items of size: ${searchResult.hits.hits.length}`); return searchResult; }; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/get_threat_list.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/get_threat_list.ts index 7ca91748fa567..27d8359453d71 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/get_threat_list.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/get_threat_list.ts @@ -22,18 +22,17 @@ export const INDICATOR_PER_PAGE = 1000; export const getThreatList = async ({ esClient, - query, - language, + exceptionItems, index, + language, + perPage, + query, + ruleExecutionLogger, searchAfter, - exceptionItems, threatFilters, - buildRuleMessage, - logger, threatListConfig, pitId, reassignPitId, - perPage, runtimeMappings, listClient, }: GetThreatListOptions): Promise> => { @@ -49,10 +48,8 @@ export const getThreatList = async ({ exceptionItems ); - logger.debug( - buildRuleMessage( - `Querying the indicator items from the index: "${index}" with searchAfter: "${searchAfter}" for up to ${calculatedPerPage} indicator items` - ) + ruleExecutionLogger.debug( + `Querying the indicator items from the index: "${index}" with searchAfter: "${searchAfter}" for up to ${calculatedPerPage} indicator items` ); const response = await esClient.search< @@ -74,7 +71,7 @@ export const getThreatList = async ({ pit: { id: pitId }, }); - logger.debug(buildRuleMessage(`Retrieved indicator items of size: ${response.hits.hits.length}`)); + ruleExecutionLogger.debug(`Retrieved indicator items of size: ${response.hits.hits.length}`); reassignPitId(response.pit_id); diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/types.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/types.ts index 62f3e9fc7e69e..b4c5d217a3a61 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/types.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threat_mapping/types.ts @@ -25,9 +25,8 @@ import type { AlertInstanceState, RuleExecutorServices, } from '@kbn/alerting-plugin/server'; -import type { ElasticsearchClient, Logger } from '@kbn/core/server'; +import type { ElasticsearchClient } from '@kbn/core/server'; import type { ITelemetryEventsSender } from '../../../telemetry/sender'; -import type { BuildRuleMessage } from '../rule_messages'; import type { BulkCreate, RuleRangeTuple, @@ -36,12 +35,12 @@ import type { WrapHits, } from '../types'; import type { CompleteRule, ThreatRuleParams } from '../../schemas/rule_schemas'; +import type { IRuleExecutionLogForExecutors } from '../../rule_monitoring'; export type SortOrderOrUndefined = 'asc' | 'desc' | undefined; export interface CreateThreatSignalsOptions { alertId: string; - buildRuleMessage: BuildRuleMessage; bulkCreate: BulkCreate; completeRule: CompleteRule; concurrentSearches: ConcurrentSearches; @@ -52,9 +51,9 @@ export interface CreateThreatSignalsOptions { itemsPerSearch: ItemsPerSearch; language: LanguageOrUndefined; listClient: ListClient; - logger: Logger; outputIndex: string; query: string; + ruleExecutionLogger: IRuleExecutionLogForExecutors; savedId: string | undefined; searchAfterSize: number; services: RuleExecutorServices; @@ -74,7 +73,6 @@ export interface CreateThreatSignalsOptions { export interface CreateThreatSignalOptions { alertId: string; - buildRuleMessage: BuildRuleMessage; bulkCreate: BulkCreate; completeRule: CompleteRule; currentResult: SearchAfterAndBulkCreateReturnType; @@ -85,9 +83,9 @@ export interface CreateThreatSignalOptions { inputIndex: string[]; language: LanguageOrUndefined; listClient: ListClient; - logger: Logger; outputIndex: string; query: string; + ruleExecutionLogger: IRuleExecutionLogForExecutors; savedId: string | undefined; searchAfterSize: number; services: RuleExecutorServices; @@ -103,7 +101,6 @@ export interface CreateThreatSignalOptions { export interface CreateEventSignalOptions { alertId: string; - buildRuleMessage: BuildRuleMessage; bulkCreate: BulkCreate; completeRule: CompleteRule; currentResult: SearchAfterAndBulkCreateReturnType; @@ -114,9 +111,9 @@ export interface CreateEventSignalOptions { inputIndex: string[]; language: LanguageOrUndefined; listClient: ListClient; - logger: Logger; outputIndex: string; query: string; + ruleExecutionLogger: IRuleExecutionLogForExecutors; savedId: string | undefined; searchAfterSize: number; services: RuleExecutorServices; @@ -186,14 +183,13 @@ interface ThreatListConfig { } export interface GetThreatListOptions { - buildRuleMessage: BuildRuleMessage; esClient: ElasticsearchClient; exceptionItems: ExceptionListItemSchema[]; index: string[]; language: ThreatLanguageOrUndefined; - logger: Logger; perPage?: number; query: string; + ruleExecutionLogger: IRuleExecutionLogForExecutors; searchAfter: estypes.SortResults | undefined; threatFilters: unknown[]; threatListConfig: ThreatListConfig; @@ -238,9 +234,8 @@ export interface ThreatMatchNamedQuery { export type GetMatchedThreats = (ids: string[]) => Promise; export interface BuildThreatEnrichmentOptions { - buildRuleMessage: BuildRuleMessage; exceptionItems: ExceptionListItemSchema[]; - logger: Logger; + ruleExecutionLogger: IRuleExecutionLogForExecutors; services: RuleExecutorServices; threatFilters: unknown[]; threatIndex: ThreatIndex; @@ -254,14 +249,13 @@ export interface BuildThreatEnrichmentOptions { export interface EventsOptions { services: RuleExecutorServices; + ruleExecutionLogger: IRuleExecutionLogForExecutors; query: string; - buildRuleMessage: BuildRuleMessage; language: ThreatLanguageOrUndefined; exceptionItems: ExceptionListItemSchema[]; index: string[]; searchAfter: estypes.SortResults | undefined; perPage?: number; - logger: Logger; filters: unknown[]; primaryTimestamp: string; secondaryTimestamp?: string; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threshold/bulk_create_threshold_signals.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threshold/bulk_create_threshold_signals.ts index b02aa91adf90c..a4b7eb9cdcedb 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threshold/bulk_create_threshold_signals.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threshold/bulk_create_threshold_signals.ts @@ -7,7 +7,6 @@ import { TIMESTAMP } from '@kbn/rule-data-utils'; -import type { Logger } from '@kbn/core/server'; import type { AlertInstanceContext, AlertInstanceState, @@ -27,7 +26,6 @@ interface BulkCreateThresholdSignalsParams { completeRule: CompleteRule; services: RuleExecutorServices; inputIndexPattern: string[]; - logger: Logger; filter: unknown; signalsIndex: string; startedAt: Date; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threshold/find_threshold_signals.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threshold/find_threshold_signals.test.ts index 0668afad39431..1d17d1ed63966 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threshold/find_threshold_signals.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threshold/find_threshold_signals.test.ts @@ -8,18 +8,11 @@ import type { RuleExecutorServicesMock } from '@kbn/alerting-plugin/server/mocks'; import { alertsMock } from '@kbn/alerting-plugin/server/mocks'; import { getQueryFilter } from '../../../../../common/detection_engine/get_query_filter'; -import { mockLogger, sampleEmptyDocSearchResults } from '../__mocks__/es_results'; -import { buildRuleMessageFactory } from '../rule_messages'; +import { sampleEmptyDocSearchResults } from '../__mocks__/es_results'; import * as single_search_after from '../single_search_after'; import { findThresholdSignals } from './find_threshold_signals'; import { TIMESTAMP } from '@kbn/rule-data-utils'; - -const buildRuleMessage = buildRuleMessageFactory({ - id: 'fake id', - ruleId: 'fake rule id', - index: 'fakeindex', - name: 'fake name', -}); +import { ruleExecutionLogMock } from '../../rule_monitoring/mocks'; const queryFilter = getQueryFilter('', 'kuery', [], ['*'], []); const mockSingleSearchAfter = jest.fn(async () => ({ @@ -37,6 +30,7 @@ const mockSingleSearchAfter = jest.fn(async () => ({ describe('findThresholdSignals', () => { let mockService: RuleExecutorServicesMock; + const ruleExecutionLogger = ruleExecutionLogMock.forExecutors.create(); beforeEach(() => { jest.clearAllMocks(); @@ -51,13 +45,12 @@ describe('findThresholdSignals', () => { maxSignals: 100, inputIndexPattern: ['*'], services: mockService, - logger: mockLogger, + ruleExecutionLogger, filter: queryFilter, threshold: { field: [], value: 100, }, - buildRuleMessage, runtimeMappings: undefined, primaryTimestamp: TIMESTAMP, secondaryTimestamp: undefined, @@ -87,13 +80,12 @@ describe('findThresholdSignals', () => { maxSignals: 100, inputIndexPattern: ['*'], services: mockService, - logger: mockLogger, + ruleExecutionLogger, filter: queryFilter, threshold: { field: ['host.name'], value: 100, }, - buildRuleMessage, runtimeMappings: undefined, primaryTimestamp: TIMESTAMP, secondaryTimestamp: undefined, @@ -148,14 +140,13 @@ describe('findThresholdSignals', () => { maxSignals: 100, inputIndexPattern: ['*'], services: mockService, - logger: mockLogger, + ruleExecutionLogger, filter: queryFilter, threshold: { field: ['host.name', 'user.name'], value: 100, cardinality: [], }, - buildRuleMessage, runtimeMappings: undefined, primaryTimestamp: TIMESTAMP, secondaryTimestamp: undefined, @@ -217,7 +208,7 @@ describe('findThresholdSignals', () => { maxSignals: 100, inputIndexPattern: ['*'], services: mockService, - logger: mockLogger, + ruleExecutionLogger, filter: queryFilter, threshold: { field: ['host.name', 'user.name'], @@ -229,7 +220,6 @@ describe('findThresholdSignals', () => { }, ], }, - buildRuleMessage, runtimeMappings: undefined, primaryTimestamp: TIMESTAMP, secondaryTimestamp: undefined, @@ -304,7 +294,7 @@ describe('findThresholdSignals', () => { maxSignals: 100, inputIndexPattern: ['*'], services: mockService, - logger: mockLogger, + ruleExecutionLogger, filter: queryFilter, threshold: { cardinality: [ @@ -316,7 +306,6 @@ describe('findThresholdSignals', () => { field: [], value: 200, }, - buildRuleMessage, runtimeMappings: undefined, primaryTimestamp: TIMESTAMP, secondaryTimestamp: undefined, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threshold/find_threshold_signals.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threshold/find_threshold_signals.ts index bd87df8b2a4c8..6f13f495027e3 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threshold/find_threshold_signals.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/threshold/find_threshold_signals.ts @@ -12,7 +12,6 @@ import type { AlertInstanceState, RuleExecutorServices, } from '@kbn/alerting-plugin/server'; -import type { Logger } from '@kbn/core/server'; import type { ESBoolQuery } from '../../../../../common/typed_json'; import type { @@ -20,7 +19,6 @@ import type { TimestampOverride, TimestampOverrideOrUndefined, } from '../../../../../common/detection_engine/schemas/common/schemas'; -import type { BuildRuleMessage } from '../rule_messages'; import { singleSearchAfter } from '../single_search_after'; import { buildThresholdMultiBucketAggregation, @@ -32,6 +30,7 @@ import type { ThresholdSingleBucketAggregationResult, } from './types'; import { shouldFilterByCardinality } from './utils'; +import type { IRuleExecutionLogForExecutors } from '../../rule_monitoring'; interface FindThresholdSignalsParams { from: string; @@ -39,10 +38,9 @@ interface FindThresholdSignalsParams { maxSignals: number; inputIndexPattern: string[]; services: RuleExecutorServices; - logger: Logger; + ruleExecutionLogger: IRuleExecutionLogForExecutors; filter: ESBoolQuery; threshold: ThresholdNormalized; - buildRuleMessage: BuildRuleMessage; runtimeMappings: estypes.MappingRuntimeFields | undefined; primaryTimestamp: TimestampOverride; secondaryTimestamp: TimestampOverrideOrUndefined; @@ -61,10 +59,9 @@ export const findThresholdSignals = async ({ maxSignals, inputIndexPattern, services, - logger, + ruleExecutionLogger, filter, threshold, - buildRuleMessage, runtimeMappings, primaryTimestamp, secondaryTimestamp, @@ -96,11 +93,10 @@ export const findThresholdSignals = async ({ from, to, services, - logger, + ruleExecutionLogger, filter, pageSize: 0, sortOrder: 'desc', - buildRuleMessage, runtimeMappings, primaryTimestamp, secondaryTimestamp, @@ -132,11 +128,10 @@ export const findThresholdSignals = async ({ from, to, services, - logger, + ruleExecutionLogger, filter, pageSize: 0, sortOrder: 'desc', - buildRuleMessage, trackTotalHits: true, runtimeMappings, primaryTimestamp, diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/types.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/types.ts index a8c26ad14dd33..db88284bc8881 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/types.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/types.ts @@ -16,7 +16,6 @@ import type { RuleExecutorServices, } from '@kbn/alerting-plugin/server'; import type { ListClient } from '@kbn/lists-plugin/server'; -import type { Logger } from '@kbn/core/server'; import type { EcsFieldMap } from '@kbn/rule-registry-plugin/common/assets/field_maps/ecs_field_map'; import type { TypeOfFieldMap } from '@kbn/rule-registry-plugin/common/field_map'; import type { Status } from '../../../../common/detection_engine/schemas/common/schemas'; @@ -27,7 +26,6 @@ import type { SearchTypes, EqlSequence, } from '../../../../common/detection_engine/types'; -import type { BuildRuleMessage } from './rule_messages'; import type { ITelemetryEventsSender } from '../../telemetry/sender'; import type { CompleteRule, @@ -43,6 +41,7 @@ import type { DetectionAlert, WrappedFieldsLatest, } from '../../../../common/detection_engine/schemas/alerts'; +import type { IRuleExecutionLogForExecutors } from '../rule_monitoring'; export interface ThresholdResult { terms?: Array<{ @@ -270,13 +269,11 @@ export interface SearchAfterAndBulkCreateParams { services: RuleExecutorServices; listClient: ListClient; exceptionsList: ExceptionListItemSchema[]; - logger: Logger; + ruleExecutionLogger: IRuleExecutionLogForExecutors; eventsTelemetry: ITelemetryEventsSender | undefined; - id: string; inputIndexPattern: string[]; pageSize: number; filter: estypes.QueryDslQueryContainer; - buildRuleMessage: BuildRuleMessage; buildReasonMessage: BuildReasonMessage; enrichment?: SignalsEnrichment; bulkCreate: BulkCreate; diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/utils.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/utils.test.ts index 0f56cc0238d0b..8bb1673fb4d41 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/utils.test.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/utils.test.ts @@ -13,9 +13,8 @@ import { ALERT_REASON, ALERT_RULE_PARAMETERS, ALERT_UUID, TIMESTAMP } from '@kbn import type { RuleExecutorServicesMock } from '@kbn/alerting-plugin/server/mocks'; import { alertsMock } from '@kbn/alerting-plugin/server/mocks'; import { listMock } from '@kbn/lists-plugin/server/mocks'; -import { buildRuleMessageFactory } from './rule_messages'; import type { ExceptionListClient } from '@kbn/lists-plugin/server'; -import { RuleExecutionStatus } from '../../../../common/detection_engine/schemas/common'; +import { RuleExecutionStatus } from '../../../../common/detection_engine/rule_monitoring'; import { getListArrayMock } from '../../../../common/detection_engine/schemas/types/lists.mock'; import { getExceptionListItemSchemaMock } from '@kbn/lists-plugin/common/schemas/response/exception_list_item_schema.mock'; @@ -52,7 +51,6 @@ import { sampleEmptyBulkResponse, sampleBulkError, sampleBulkErrorItem, - mockLogger, sampleSignalHit, sampleDocSearchResultsWithSortId, sampleEmptyDocSearchResults, @@ -63,24 +61,19 @@ import { sampleAlertDocAADNoSortIdWithTimestamp, } from './__mocks__/es_results'; import type { ShardError } from '../../types'; -import { ruleExecutionLogMock } from '../rule_execution_log/__mocks__'; - -const buildRuleMessage = buildRuleMessageFactory({ - id: 'fake id', - ruleId: 'fake rule id', - index: 'fakeindex', - name: 'fake name', -}); +import { ruleExecutionLogMock } from '../rule_monitoring/mocks'; describe('utils', () => { const anchor = '2020-01-01T06:06:06.666Z'; const unix = moment(anchor).valueOf(); let nowDate = moment('2020-01-01T00:00:00.000Z'); let clock: sinon.SinonFakeTimers; + let ruleExecutionLogger: ReturnType; beforeEach(() => { nowDate = moment('2020-01-01T00:00:00.000Z'); clock = sinon.useFakeTimers(unix); + ruleExecutionLogger = ruleExecutionLogMock.forExecutors.create(); }); afterEach(() => { @@ -449,14 +442,13 @@ describe('utils', () => { describe('getRuleRangeTuples', () => { test('should return a single tuple if no gap', () => { const { tuples, remainingGap } = getRuleRangeTuples({ - logger: mockLogger, previousStartedAt: moment().subtract(30, 's').toDate(), startedAt: moment().subtract(30, 's').toDate(), interval: '30s', from: 'now-30s', to: 'now', maxSignals: 20, - buildRuleMessage, + ruleExecutionLogger, }); const someTuple = tuples[0]; expect(moment(someTuple.to).diff(moment(someTuple.from), 's')).toEqual(30); @@ -466,14 +458,13 @@ describe('utils', () => { test('should return a single tuple if malformed interval prevents gap calculation', () => { const { tuples, remainingGap } = getRuleRangeTuples({ - logger: mockLogger, previousStartedAt: moment().subtract(30, 's').toDate(), startedAt: moment().subtract(30, 's').toDate(), interval: 'invalid', from: 'now-30s', to: 'now', maxSignals: 20, - buildRuleMessage, + ruleExecutionLogger, }); const someTuple = tuples[0]; expect(moment(someTuple.to).diff(moment(someTuple.from), 's')).toEqual(30); @@ -483,14 +474,13 @@ describe('utils', () => { test('should return two tuples if gap and previouslyStartedAt', () => { const { tuples, remainingGap } = getRuleRangeTuples({ - logger: mockLogger, previousStartedAt: moment().subtract(65, 's').toDate(), startedAt: moment().toDate(), interval: '50s', from: 'now-55s', to: 'now', maxSignals: 20, - buildRuleMessage, + ruleExecutionLogger, }); const someTuple = tuples[1]; expect(moment(someTuple.to).diff(moment(someTuple.from), 's')).toEqual(55); @@ -499,14 +489,13 @@ describe('utils', () => { test('should return five tuples when give long gap', () => { const { tuples, remainingGap } = getRuleRangeTuples({ - logger: mockLogger, previousStartedAt: moment().subtract(65, 's').toDate(), // 64 is 5 times the interval + lookback, which will trigger max lookback startedAt: moment().toDate(), interval: '10s', from: 'now-13s', to: 'now', maxSignals: 20, - buildRuleMessage, + ruleExecutionLogger, }); expect(tuples.length).toEqual(5); tuples.forEach((item, index) => { @@ -522,14 +511,13 @@ describe('utils', () => { test('should return a single tuple when give a negative gap (rule ran sooner than expected)', () => { const { tuples, remainingGap } = getRuleRangeTuples({ - logger: mockLogger, previousStartedAt: moment().subtract(-15, 's').toDate(), startedAt: moment().subtract(-15, 's').toDate(), interval: '10s', from: 'now-13s', to: 'now', maxSignals: 20, - buildRuleMessage, + ruleExecutionLogger, }); expect(tuples.length).toEqual(1); const someTuple = tuples[0]; @@ -651,8 +639,6 @@ describe('utils', () => { }, }, }; - const ruleExecutionLogger = ruleExecutionLogMock.forExecutors.create(); - mockLogger.warn.mockClear(); const res = await hasTimestampFields({ timestampField, @@ -662,14 +648,9 @@ describe('utils', () => { >, inputIndices: ['myfa*'], ruleExecutionLogger, - logger: mockLogger, - buildRuleMessage, }); expect(res).toBeTruthy(); - expect(mockLogger.warn).toHaveBeenCalledWith( - 'The following indices are missing the timestamp override field "event.ingested": ["myfakeindex-1","myfakeindex-2"] name: "fake name" id: "fake id" rule id: "fake rule id" signals index: "fakeindex"' - ); expect(ruleExecutionLogger.logStatusChange).toHaveBeenCalledWith({ newStatus: RuleExecutionStatus['partial failure'], message: @@ -702,9 +683,6 @@ describe('utils', () => { }, }; - const ruleExecutionLogger = ruleExecutionLogMock.forExecutors.create(); - mockLogger.warn.mockClear(); - const res = await hasTimestampFields({ timestampField, timestampFieldCapsResponse: timestampFieldCapsResponse as TransportResult< @@ -713,14 +691,9 @@ describe('utils', () => { >, inputIndices: ['myfa*'], ruleExecutionLogger, - logger: mockLogger, - buildRuleMessage, }); expect(res).toBeTruthy(); - expect(mockLogger.warn).toHaveBeenCalledWith( - 'The following indices are missing the timestamp field "@timestamp": ["myfakeindex-1","myfakeindex-2"] name: "fake name" id: "fake id" rule id: "fake rule id" signals index: "fakeindex"' - ); expect(ruleExecutionLogger.logStatusChange).toHaveBeenCalledWith({ newStatus: RuleExecutionStatus['partial failure'], message: @@ -738,10 +711,9 @@ describe('utils', () => { }, }; - const ruleExecutionLogger = ruleExecutionLogMock.forExecutors.create({ + ruleExecutionLogger = ruleExecutionLogMock.forExecutors.create({ ruleName: 'Endpoint Security', }); - mockLogger.warn.mockClear(); const res = await hasTimestampFields({ timestampField, @@ -751,14 +723,9 @@ describe('utils', () => { >, inputIndices: ['logs-endpoint.alerts-*'], ruleExecutionLogger, - logger: mockLogger, - buildRuleMessage, }); expect(res).toBeTruthy(); - expect(mockLogger.warn).toHaveBeenCalledWith( - 'This rule is attempting to query data from Elasticsearch indices listed in the "Index pattern" section of the rule definition, however no index matching: ["logs-endpoint.alerts-*"] was found. This warning will continue to appear until a matching index is created or this rule is disabled. If you have recently enrolled agents enabled with Endpoint Security through Fleet, this warning should stop once an alert is sent from an agent. name: "fake name" id: "fake id" rule id: "fake rule id" signals index: "fakeindex"' - ); expect(ruleExecutionLogger.logStatusChange).toHaveBeenCalledWith({ newStatus: RuleExecutionStatus['partial failure'], message: @@ -777,12 +744,10 @@ describe('utils', () => { }; // SUT uses rule execution logger's context to check the rule name - const ruleExecutionLogger = ruleExecutionLogMock.forExecutors.create({ + ruleExecutionLogger = ruleExecutionLogMock.forExecutors.create({ ruleName: 'NOT Endpoint Security', }); - mockLogger.warn.mockClear(); - const res = await hasTimestampFields({ timestampField, timestampFieldCapsResponse: timestampFieldCapsResponse as TransportResult< @@ -791,14 +756,9 @@ describe('utils', () => { >, inputIndices: ['logs-endpoint.alerts-*'], ruleExecutionLogger, - logger: mockLogger, - buildRuleMessage, }); expect(res).toBeTruthy(); - expect(mockLogger.warn).toHaveBeenCalledWith( - 'This rule is attempting to query data from Elasticsearch indices listed in the "Index pattern" section of the rule definition, however no index matching: ["logs-endpoint.alerts-*"] was found. This warning will continue to appear until a matching index is created or this rule is disabled. name: "fake name" id: "fake id" rule id: "fake rule id" signals index: "fakeindex"' - ); expect(ruleExecutionLogger.logStatusChange).toHaveBeenCalledWith({ newStatus: RuleExecutionStatus['partial failure'], message: diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/utils.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/utils.ts index c75c8f1a1c125..6fc3708db6b98 100644 --- a/x-pack/plugins/security_solution/server/lib/detection_engine/signals/utils.ts +++ b/x-pack/plugins/security_solution/server/lib/detection_engine/signals/utils.ts @@ -22,7 +22,6 @@ import type { import type { ElasticsearchClient, IUiSettingsClient, - Logger, SavedObjectsClientContract, } from '@kbn/core/server'; import type { @@ -36,7 +35,7 @@ import type { TimestampOverride, Privilege, } from '../../../../common/detection_engine/schemas/common'; -import { RuleExecutionStatus } from '../../../../common/detection_engine/schemas/common'; +import { RuleExecutionStatus } from '../../../../common/detection_engine/rule_monitoring'; import type { BulkResponseErrorAggregation, SignalHit, @@ -50,7 +49,6 @@ import type { SimpleHit, WrappedEventHit, } from './types'; -import type { BuildRuleMessage } from './rule_messages'; import type { ShardError } from '../../types'; import type { EqlRuleParams, @@ -61,7 +59,7 @@ import type { ThresholdRuleParams, } from '../schemas/rule_schemas'; import type { BaseHit, SearchTypes } from '../../../../common/detection_engine/types'; -import type { IRuleExecutionLogForExecutors } from '../rule_execution_log'; +import type { IRuleExecutionLogForExecutors } from '../rule_monitoring'; import { withSecuritySpan } from '../../../utils/with_security_span'; import type { DetectionAlert } from '../../../../common/detection_engine/schemas/alerts'; import { ENABLE_CCS_READ_WARNING_SETTING } from '../../../../common/constants'; @@ -70,12 +68,10 @@ export const MAX_RULE_GAP_RATIO = 4; export const hasReadIndexPrivileges = async (args: { privileges: Privilege; - logger: Logger; - buildRuleMessage: BuildRuleMessage; ruleExecutionLogger: IRuleExecutionLogForExecutors; uiSettingsClient: IUiSettingsClient; }): Promise => { - const { privileges, logger, buildRuleMessage, ruleExecutionLogger, uiSettingsClient } = args; + const { privileges, ruleExecutionLogger, uiSettingsClient } = args; const isCcsPermissionWarningEnabled = await uiSettingsClient.get(ENABLE_CCS_READ_WARNING_SETTING); @@ -89,19 +85,16 @@ export const hasReadIndexPrivileges = async (args: { (indexName) => privileges.index[indexName].read ); + // Some indices have read privileges others do not. if (indexesWithNoReadPrivileges.length > 0) { - // some indices have read privileges others do not. - // set a warning status - const errorString = `This rule may not have the required read privileges to the following indices/index patterns: ${JSON.stringify( - indexesWithNoReadPrivileges - )}`; - logger.warn(buildRuleMessage(errorString)); + const indexesString = JSON.stringify(indexesWithNoReadPrivileges); await ruleExecutionLogger.logStatusChange({ newStatus: RuleExecutionStatus['partial failure'], - message: errorString, + message: `This rule may not have the required read privileges to the following indices/index patterns: ${indexesString}`, }); return true; } + return false; }; @@ -113,18 +106,8 @@ export const hasTimestampFields = async (args: { timestampFieldCapsResponse: TransportResult, unknown>; inputIndices: string[]; ruleExecutionLogger: IRuleExecutionLogForExecutors; - logger: Logger; - buildRuleMessage: BuildRuleMessage; }): Promise => { - const { - timestampField, - timestampFieldCapsResponse, - inputIndices, - ruleExecutionLogger, - logger, - buildRuleMessage, - } = args; - + const { timestampField, timestampFieldCapsResponse, inputIndices, ruleExecutionLogger } = args; const { ruleName } = ruleExecutionLogger.context; if (isEmpty(timestampFieldCapsResponse.body.indices)) { @@ -135,11 +118,12 @@ export const hasTimestampFields = async (args: { ? 'If you have recently enrolled agents enabled with Endpoint Security through Fleet, this warning should stop once an alert is sent from an agent.' : '' }`; - logger.warn(buildRuleMessage(errorString.trimEnd())); + await ruleExecutionLogger.logStatusChange({ newStatus: RuleExecutionStatus['partial failure'], message: errorString.trimEnd(), }); + return true; } else if ( isEmpty(timestampFieldCapsResponse.body.fields) || @@ -159,7 +143,6 @@ export const hasTimestampFields = async (args: { : timestampFieldCapsResponse.body.fields[timestampField]?.unmapped?.indices )}`; - logger.warn(buildRuleMessage(errorString)); await ruleExecutionLogger.logStatusChange({ newStatus: RuleExecutionStatus['partial failure'], message: errorString, @@ -167,6 +150,7 @@ export const hasTimestampFields = async (args: { return true; } + return false; }; @@ -413,29 +397,28 @@ export const errorAggregator = ( }; export const getRuleRangeTuples = ({ - logger, + startedAt, previousStartedAt, from, to, interval, maxSignals, - buildRuleMessage, - startedAt, + ruleExecutionLogger, }: { - logger: Logger; + startedAt: Date; previousStartedAt: Date | null | undefined; from: string; to: string; interval: string; maxSignals: number; - buildRuleMessage: BuildRuleMessage; - startedAt: Date; + ruleExecutionLogger: IRuleExecutionLogForExecutors; }) => { - const originalTo = dateMath.parse(to, { forceNow: startedAt }); const originalFrom = dateMath.parse(from, { forceNow: startedAt }); - if (originalTo == null || originalFrom == null) { - throw new Error(buildRuleMessage('dateMath parse failed')); + const originalTo = dateMath.parse(to, { forceNow: startedAt }); + if (originalFrom == null || originalTo == null) { + throw new Error('Failed to parse date math of rule.from or rule.to'); } + const tuples = [ { to: originalTo, @@ -443,11 +426,15 @@ export const getRuleRangeTuples = ({ maxSignals, }, ]; + const intervalDuration = parseInterval(interval); if (intervalDuration == null) { - logger.error(`Failed to compute gap between rule runs: could not parse rule interval`); + ruleExecutionLogger.error( + 'Failed to compute gap between rule runs: could not parse rule interval' + ); return { tuples, remainingGap: moment.duration(0) }; } + const gap = getGapBetweenRuns({ previousStartedAt, originalTo, @@ -465,13 +452,19 @@ export const getRuleRangeTuples = ({ catchup, intervalDuration, }); + tuples.push(...catchupTuples); + // Each extra tuple adds one extra intervalDuration to the time range this rule will cover. const remainingGapMilliseconds = Math.max( gap.asMilliseconds() - catchup * intervalDuration.asMilliseconds(), 0 ); - return { tuples: tuples.reverse(), remainingGap: moment.duration(remainingGapMilliseconds) }; + + return { + tuples: tuples.reverse(), + remainingGap: moment.duration(remainingGapMilliseconds), + }; }; /** diff --git a/x-pack/plugins/security_solution/server/plugin.ts b/x-pack/plugins/security_solution/server/plugin.ts index 0fe1adc22f880..b8a70d5b338ef 100644 --- a/x-pack/plugins/security_solution/server/plugin.ts +++ b/x-pack/plugins/security_solution/server/plugin.ts @@ -72,10 +72,7 @@ import { PolicyWatcher } from './endpoint/lib/policy/license_watch'; import { migrateArtifactsToFleet } from './endpoint/lib/artifacts/migrate_artifacts_to_fleet'; import aadFieldConversion from './lib/detection_engine/routes/index/signal_aad_mapping.json'; import previewPolicy from './lib/detection_engine/routes/index/preview_policy.json'; -import { - registerEventLogProvider, - ruleExecutionLogForExecutorsFactory, -} from './lib/detection_engine/rule_execution_log'; +import { createRuleExecutionLogService } from './lib/detection_engine/rule_monitoring'; import { getKibanaPrivilegesFeaturePrivileges, getCasesKibanaFeature } from './features'; import { EndpointMetadataService } from './endpoint/services/metadata'; import type { CreateRuleOptions } from './lib/detection_engine/rule_types/types'; @@ -150,8 +147,8 @@ export class Plugin implements ISecuritySolutionPlugin { initSavedObjects(core.savedObjects); initUiSettings(core.uiSettings, experimentalFeatures); - const eventLogService = plugins.eventLog; - registerEventLogProvider(eventLogService); + const ruleExecutionLogService = createRuleExecutionLogService(config, logger, core, plugins); + ruleExecutionLogService.registerEventLogProvider(); const requestContextFactory = new RequestContextFactory({ config, @@ -159,6 +156,7 @@ export class Plugin implements ISecuritySolutionPlugin { core, plugins, endpointAppContextService: this.endpointAppContextService, + ruleExecutionLogService, }); const router = core.http.createRouter(); @@ -180,7 +178,7 @@ export class Plugin implements ISecuritySolutionPlugin { initUsageCollectors({ core, - eventLogIndex: eventLogService.getIndexPattern(), + eventLogIndex: plugins.eventLog.getIndexPattern(), signalsIndex: DEFAULT_ALERTS_INDEX, ml: plugins.ml, usageCollection: plugins.usageCollection, @@ -242,8 +240,7 @@ export class Plugin implements ISecuritySolutionPlugin { logger: this.logger, config: this.config, ruleDataClient, - eventLogService, - ruleExecutionLoggerFactory: ruleExecutionLogForExecutorsFactory, + ruleExecutionLoggerFactory: ruleExecutionLogService.createClientForExecutors, version: pluginContext.env.packageInfo.version, }; diff --git a/x-pack/plugins/security_solution/server/request_context_factory.ts b/x-pack/plugins/security_solution/server/request_context_factory.ts index 72050aba3f9db..7812dd4d7c040 100644 --- a/x-pack/plugins/security_solution/server/request_context_factory.ts +++ b/x-pack/plugins/security_solution/server/request_context_factory.ts @@ -13,7 +13,7 @@ import type { FleetAuthz } from '@kbn/fleet-plugin/common'; import { DEFAULT_SPACE_ID } from '../common/constants'; import { AppClientFactory } from './client'; import type { ConfigType } from './config'; -import { ruleExecutionLogForRoutesFactory } from './lib/detection_engine/rule_execution_log'; +import type { IRuleExecutionLogService } from './lib/detection_engine/rule_monitoring'; import { buildFrameworkRequest } from './lib/timeline/utils/common'; import type { SecuritySolutionPluginCoreSetupDependencies, @@ -45,6 +45,7 @@ interface ConstructorOptions { core: SecuritySolutionPluginCoreSetupDependencies; plugins: SecuritySolutionPluginSetupDependencies; endpointAppContextService: EndpointAppContextService; + ruleExecutionLogService: IRuleExecutionLogService; } export class RequestContextFactory implements IRequestContextFactory { @@ -59,7 +60,7 @@ export class RequestContextFactory implements IRequestContextFactory { request: KibanaRequest ): Promise { const { options, appClientFactory } = this; - const { config, logger, core, plugins, endpointAppContextService } = options; + const { config, core, plugins, endpointAppContextService, ruleExecutionLogService } = options; const { lists, ruleRegistry, security } = plugins; const [, startPlugins] = await core.getStartServices(); @@ -109,11 +110,10 @@ export class RequestContextFactory implements IRequestContextFactory { getRuleDataService: () => ruleRegistry.ruleDataService, getRuleExecutionLog: memoize(() => - ruleExecutionLogForRoutesFactory( - coreContext.savedObjects.client, - startPlugins.eventLog.getClient(request), - logger - ) + ruleExecutionLogService.createClientForRoutes({ + savedObjectsClient: coreContext.savedObjects.client, + eventLogClient: startPlugins.eventLog.getClient(request), + }) ), getExceptionListClient: () => { diff --git a/x-pack/plugins/security_solution/server/routes/index.ts b/x-pack/plugins/security_solution/server/routes/index.ts index 129f611e4aa96..98b6b5ec5933f 100644 --- a/x-pack/plugins/security_solution/server/routes/index.ts +++ b/x-pack/plugins/security_solution/server/routes/index.ts @@ -35,7 +35,7 @@ import { deleteRulesBulkRoute } from '../lib/detection_engine/routes/rules/delet import { performBulkActionRoute } from '../lib/detection_engine/routes/rules/perform_bulk_action_route'; import { importRulesRoute } from '../lib/detection_engine/routes/rules/import_rules_route'; import { exportRulesRoute } from '../lib/detection_engine/routes/rules/export_rules_route'; -import { getRuleExecutionEventsRoute } from '../lib/detection_engine/routes/rules/get_rule_execution_events_route'; +import { registerRuleMonitoringRoutes } from '../lib/detection_engine/rule_monitoring'; import { getPrepackagedRulesStatusRoute } from '../lib/detection_engine/routes/rules/get_prepackaged_rules_status_route'; import { createTimelinesRoute, @@ -117,7 +117,7 @@ export const initRoutes = ( deleteRulesBulkRoute(router, logger); performBulkActionRoute(router, ml, logger); - getRuleExecutionEventsRoute(router); + registerRuleMonitoringRoutes(router); getInstalledIntegrationsRoute(router, logger); diff --git a/x-pack/plugins/security_solution/server/saved_objects.ts b/x-pack/plugins/security_solution/server/saved_objects.ts index e8943c66b3ad7..225c0ad453a3d 100644 --- a/x-pack/plugins/security_solution/server/saved_objects.ts +++ b/x-pack/plugins/security_solution/server/saved_objects.ts @@ -10,7 +10,7 @@ import type { CoreSetup } from '@kbn/core/server'; import { noteType, pinnedEventType, timelineType } from './lib/timeline/saved_object_mappings'; // eslint-disable-next-line no-restricted-imports import { legacyType as legacyRuleActionsType } from './lib/detection_engine/rule_actions/legacy_saved_object_mappings'; -import { ruleExecutionType } from './lib/detection_engine/rule_execution_log'; +import { ruleExecutionType } from './lib/detection_engine/rule_monitoring'; import { ruleAssetType } from './lib/detection_engine/rules/rule_asset/rule_asset_saved_object_mappings'; import { type as signalsMigrationType } from './lib/detection_engine/migrations/saved_objects'; import { diff --git a/x-pack/plugins/security_solution/server/types.ts b/x-pack/plugins/security_solution/server/types.ts index 7e9a29b4dd0b5..73b5216ae5e9a 100644 --- a/x-pack/plugins/security_solution/server/types.ts +++ b/x-pack/plugins/security_solution/server/types.ts @@ -20,7 +20,7 @@ import type { IRuleDataService } from '@kbn/rule-registry-plugin/server'; import { AppClient } from './client'; import type { ConfigType } from './config'; -import type { IRuleExecutionLogForRoutes } from './lib/detection_engine/rule_execution_log'; +import type { IRuleExecutionLogForRoutes } from './lib/detection_engine/rule_monitoring'; import type { FrameworkRequest } from './lib/framework'; import type { EndpointAuthz } from '../common/endpoint/types/authz'; import type { diff --git a/x-pack/plugins/security_solution/server/ui_settings.ts b/x-pack/plugins/security_solution/server/ui_settings.ts index ddc41ae3e2926..4b8fe1be50ec0 100644 --- a/x-pack/plugins/security_solution/server/ui_settings.ts +++ b/x-pack/plugins/security_solution/server/ui_settings.ts @@ -33,8 +33,11 @@ import { NEWS_FEED_URL_SETTING_DEFAULT, ENABLE_CCS_READ_WARNING_SETTING, SHOW_RELATED_INTEGRATIONS_SETTING, + EXTENDED_RULE_EXECUTION_LOGGING_ENABLED_SETTING, + EXTENDED_RULE_EXECUTION_LOGGING_MIN_LEVEL_SETTING, } from '../common/constants'; import type { ExperimentalFeatures } from '../common/experimental_features'; +import { LogLevelSetting } from '../common/detection_engine/rule_monitoring'; type SettingsConfig = Record>; @@ -264,6 +267,103 @@ export const initUiSettings = ( requiresPageReload: true, schema: schema.boolean(), }, + ...(experimentalFeatures.extendedRuleExecutionLoggingEnabled + ? { + [EXTENDED_RULE_EXECUTION_LOGGING_ENABLED_SETTING]: { + name: i18n.translate( + 'xpack.securitySolution.uiSettings.extendedRuleExecutionLoggingEnabledLabel', + { + defaultMessage: 'Extended rule execution logging', + } + ), + description: i18n.translate( + 'xpack.securitySolution.uiSettings.extendedRuleExecutionLoggingEnabledDescription', + { + defaultMessage: + '

Enables extended rule execution logging to .kibana-event-log-* indices. Shows plain execution events on the Rule Details page.

', + } + ), + type: 'boolean', + schema: schema.boolean(), + value: true, + category: [APP_ID], + requiresPageReload: false, + }, + [EXTENDED_RULE_EXECUTION_LOGGING_MIN_LEVEL_SETTING]: { + name: i18n.translate( + 'xpack.securitySolution.uiSettings.extendedRuleExecutionLoggingMinLevelLabel', + { + defaultMessage: 'Extended rule execution logging: min level', + } + ), + description: i18n.translate( + 'xpack.securitySolution.uiSettings.extendedRuleExecutionLoggingMinLevelDescription', + { + defaultMessage: + '

Sets minimum log level starting from which rules will write extended logs to .kibana-event-log-* indices. This affects only events of type Message, other events are being written to .kibana-event-log-* regardless of this setting and their log level.

', + } + ), + type: 'select', + schema: schema.oneOf([ + schema.literal(LogLevelSetting.off), + schema.literal(LogLevelSetting.error), + schema.literal(LogLevelSetting.warn), + schema.literal(LogLevelSetting.info), + schema.literal(LogLevelSetting.debug), + schema.literal(LogLevelSetting.trace), + ]), + value: LogLevelSetting.error, + options: [ + LogLevelSetting.off, + LogLevelSetting.error, + LogLevelSetting.warn, + LogLevelSetting.info, + LogLevelSetting.debug, + LogLevelSetting.trace, + ], + optionLabels: { + [LogLevelSetting.off]: i18n.translate( + 'xpack.securitySolution.uiSettings.extendedRuleExecutionLoggingMinLevelOff', + { + defaultMessage: 'Off', + } + ), + [LogLevelSetting.error]: i18n.translate( + 'xpack.securitySolution.uiSettings.extendedRuleExecutionLoggingMinLevelError', + { + defaultMessage: 'Error', + } + ), + [LogLevelSetting.warn]: i18n.translate( + 'xpack.securitySolution.uiSettings.extendedRuleExecutionLoggingMinLevelWarn', + { + defaultMessage: 'Warn', + } + ), + [LogLevelSetting.info]: i18n.translate( + 'xpack.securitySolution.uiSettings.extendedRuleExecutionLoggingMinLevelInfo', + { + defaultMessage: 'Info', + } + ), + [LogLevelSetting.debug]: i18n.translate( + 'xpack.securitySolution.uiSettings.extendedRuleExecutionLoggingMinLevelDebug', + { + defaultMessage: 'Debug', + } + ), + [LogLevelSetting.trace]: i18n.translate( + 'xpack.securitySolution.uiSettings.extendedRuleExecutionLoggingMinLevelTrace', + { + defaultMessage: 'Trace', + } + ), + }, + category: [APP_ID], + requiresPageReload: false, + }, + } + : {}), }; uiSettings.register(orderSettings(securityUiSettings)); diff --git a/x-pack/plugins/security_solution/server/usage/queries/utils/get_search_for_all_rules.ts b/x-pack/plugins/security_solution/server/usage/queries/utils/get_search_for_all_rules.ts index c74ce1a2262c3..acd2a7aaa7ba7 100644 --- a/x-pack/plugins/security_solution/server/usage/queries/utils/get_search_for_all_rules.ts +++ b/x-pack/plugins/security_solution/server/usage/queries/utils/get_search_for_all_rules.ts @@ -7,7 +7,7 @@ import type { SearchRequest } from '@elastic/elasticsearch/lib/api/types'; import type { AggregationsAggregationContainer } from '@elastic/elasticsearch/lib/api/typesWithBodyKey'; -import { RULE_EXECUTION_LOG_PROVIDER } from '../../../lib/detection_engine/rule_execution_log/event_log/constants'; +import { RULE_EXECUTION_LOG_PROVIDER } from '../../../lib/detection_engine/rule_monitoring'; /** * Given an aggregation of "aggs" this will return a search for all rules within 24 hours. diff --git a/x-pack/plugins/security_solution/server/usage/queries/utils/get_search_for_custom_rules.ts b/x-pack/plugins/security_solution/server/usage/queries/utils/get_search_for_custom_rules.ts index 88f70469e57ee..88470ee2f3cae 100644 --- a/x-pack/plugins/security_solution/server/usage/queries/utils/get_search_for_custom_rules.ts +++ b/x-pack/plugins/security_solution/server/usage/queries/utils/get_search_for_custom_rules.ts @@ -9,7 +9,7 @@ import type { AggregationsAggregationContainer, SearchRequest, } from '@elastic/elasticsearch/lib/api/types'; -import { RULE_EXECUTION_LOG_PROVIDER } from '../../../lib/detection_engine/rule_execution_log/event_log/constants'; +import { RULE_EXECUTION_LOG_PROVIDER } from '../../../lib/detection_engine/rule_monitoring'; /** * Given an aggregation of "aggs" this will return a search for rules that are NOT elastic diff --git a/x-pack/plugins/security_solution/server/usage/queries/utils/get_search_for_elastic_rules.ts b/x-pack/plugins/security_solution/server/usage/queries/utils/get_search_for_elastic_rules.ts index c2624fedb6e60..4cd8bb0ae17a8 100644 --- a/x-pack/plugins/security_solution/server/usage/queries/utils/get_search_for_elastic_rules.ts +++ b/x-pack/plugins/security_solution/server/usage/queries/utils/get_search_for_elastic_rules.ts @@ -9,7 +9,7 @@ import type { AggregationsAggregationContainer, SearchRequest, } from '@elastic/elasticsearch/lib/api/types'; -import { RULE_EXECUTION_LOG_PROVIDER } from '../../../lib/detection_engine/rule_execution_log/event_log/constants'; +import { RULE_EXECUTION_LOG_PROVIDER } from '../../../lib/detection_engine/rule_monitoring'; /** * Given an aggregation of "aggs" this will return a search for rules that are elastic diff --git a/x-pack/plugins/translations/translations/fr-FR.json b/x-pack/plugins/translations/translations/fr-FR.json index 8faa1714dbcef..939336dbd4d2b 100644 --- a/x-pack/plugins/translations/translations/fr-FR.json +++ b/x-pack/plugins/translations/translations/fr-FR.json @@ -24470,7 +24470,6 @@ "xpack.securitySolution.containers.detectionEngine.createPrePackagedRuleAndTimelineSuccesDescription": "Installation effectuée des règles et modèles de chronologies prépackagés à partir d'Elastic", "xpack.securitySolution.containers.detectionEngine.createPrePackagedRuleSuccesDescription": "Installation effectuée des règles prépackagées à partir d'Elastic", "xpack.securitySolution.containers.detectionEngine.createPrePackagedTimelineSuccesDescription": "Installation effectuée des modèles de chronologies prépackagées à partir d'Elastic", - "xpack.securitySolution.containers.detectionEngine.ruleExecutionEventsFetchFailDescription": "Impossible de récupérer les événements d'exécution de règle", "xpack.securitySolution.containers.detectionEngine.rulesAndTimelines": "Impossible de récupérer les règles et les chronologies", "xpack.securitySolution.containers.detectionEngine.tagFetchFailDescription": "Impossible de récupérer les balises", "xpack.securitySolution.containers.errors.stopJobFailureTitle": "Échec d'arrêt de la tâche", @@ -25422,7 +25421,6 @@ "xpack.securitySolution.detectionEngine.ruleDetails.ruleExecutionLog.timestampColumn": "Horodatage", "xpack.securitySolution.detectionEngine.ruleDetails.ruleExecutionLog.timestampColumnTooltip": "Date et heures auxquelles l'exécution de la règle a été lancée.", "xpack.securitySolution.detectionEngine.ruleDetails.ruleExecutionLog.totalExecutionsLabel": "Affichage de {totalItems} {totalItems, plural, =1 {exécution de règle} other {exécutions de règle}}", - "xpack.securitySolution.detectionEngine.ruleDetails.ruleExecutionLogsTab": "Logs d'exécution de règle ", "xpack.securitySolution.detectionEngine.ruleDetails.ruleUpdateDescription": "Mis à jour par : {by} le {date}", "xpack.securitySolution.detectionEngine.ruleDetails.unknownDescription": "Inconnu", "xpack.securitySolution.detectionEngine.rules.aboutRuleTitle": "À propos de la règle", diff --git a/x-pack/plugins/translations/translations/ja-JP.json b/x-pack/plugins/translations/translations/ja-JP.json index b07d488574717..efdd5e2b0d1e4 100644 --- a/x-pack/plugins/translations/translations/ja-JP.json +++ b/x-pack/plugins/translations/translations/ja-JP.json @@ -24550,7 +24550,6 @@ "xpack.securitySolution.containers.detectionEngine.createPrePackagedRuleAndTimelineSuccesDescription": "Elasticから事前にパッケージ化されているルールとタイムラインテンプレートをインストールしました", "xpack.securitySolution.containers.detectionEngine.createPrePackagedRuleSuccesDescription": "Elastic から事前にパッケージ化されているルールをインストールしました", "xpack.securitySolution.containers.detectionEngine.createPrePackagedTimelineSuccesDescription": "Elasticから事前にパッケージ化されているタイムラインテンプレートをインストールしました", - "xpack.securitySolution.containers.detectionEngine.ruleExecutionEventsFetchFailDescription": "ルール実行イベントを取得できませんでした", "xpack.securitySolution.containers.detectionEngine.rulesAndTimelines": "ルールとタイムラインを取得できませんでした", "xpack.securitySolution.containers.detectionEngine.tagFetchFailDescription": "タグを取得できませんでした", "xpack.securitySolution.containers.errors.stopJobFailureTitle": "ジョブ停止エラー", @@ -25502,7 +25501,6 @@ "xpack.securitySolution.detectionEngine.ruleDetails.ruleExecutionLog.timestampColumn": "タイムスタンプ", "xpack.securitySolution.detectionEngine.ruleDetails.ruleExecutionLog.timestampColumnTooltip": "日時ルール実行が開始されました。", "xpack.securitySolution.detectionEngine.ruleDetails.ruleExecutionLog.totalExecutionsLabel": "{totalItems} {totalItems, plural, other {個のルール例外}}を表示しています", - "xpack.securitySolution.detectionEngine.ruleDetails.ruleExecutionLogsTab": "ルール実行ログ ", "xpack.securitySolution.detectionEngine.ruleDetails.ruleUpdateDescription": "更新者：{by} 日付：{date}", "xpack.securitySolution.detectionEngine.ruleDetails.unknownDescription": "不明", "xpack.securitySolution.detectionEngine.rules.aboutRuleTitle": "ルールについて", diff --git a/x-pack/plugins/translations/translations/zh-CN.json b/x-pack/plugins/translations/translations/zh-CN.json index 761b6b11c3672..b755fce328f6b 100644 --- a/x-pack/plugins/translations/translations/zh-CN.json +++ b/x-pack/plugins/translations/translations/zh-CN.json @@ -24576,7 +24576,6 @@ "xpack.securitySolution.containers.detectionEngine.createPrePackagedRuleAndTimelineSuccesDescription": "已安装 Elastic 预先打包的规则和时间线模板", "xpack.securitySolution.containers.detectionEngine.createPrePackagedRuleSuccesDescription": "已安装 Elastic 的预打包规则", "xpack.securitySolution.containers.detectionEngine.createPrePackagedTimelineSuccesDescription": "安装 Elastic 预先打包的时间线模板", - "xpack.securitySolution.containers.detectionEngine.ruleExecutionEventsFetchFailDescription": "无法提取规则执行事件", "xpack.securitySolution.containers.detectionEngine.rulesAndTimelines": "无法提取规则和时间线", "xpack.securitySolution.containers.detectionEngine.tagFetchFailDescription": "无法提取标签", "xpack.securitySolution.containers.errors.stopJobFailureTitle": "停止作业失败", @@ -25528,7 +25527,6 @@ "xpack.securitySolution.detectionEngine.ruleDetails.ruleExecutionLog.timestampColumn": "时间戳", "xpack.securitySolution.detectionEngine.ruleDetails.ruleExecutionLog.timestampColumnTooltip": "已发起日期时间规则执行。", "xpack.securitySolution.detectionEngine.ruleDetails.ruleExecutionLog.totalExecutionsLabel": "正在显示 {totalItems} 个{totalItems, plural, other {规则执行}}", - "xpack.securitySolution.detectionEngine.ruleDetails.ruleExecutionLogsTab": "规则执行日志 ", "xpack.securitySolution.detectionEngine.ruleDetails.ruleUpdateDescription": "由 {by} 于 {date}更新", "xpack.securitySolution.detectionEngine.ruleDetails.unknownDescription": "未知", "xpack.securitySolution.detectionEngine.rules.aboutRuleTitle": "关于规则", diff --git a/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/check_privileges.ts b/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/check_privileges.ts index 467b87b6c64b9..d208b522a1d44 100644 --- a/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/check_privileges.ts +++ b/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/check_privileges.ts @@ -8,7 +8,7 @@ import expect from '@kbn/expect'; import { DETECTION_ENGINE_RULES_URL } from '@kbn/security-solution-plugin/common/constants'; import { ROLES } from '@kbn/security-solution-plugin/common/test'; -import { RuleExecutionStatus } from '@kbn/security-solution-plugin/common/detection_engine/schemas/common'; +import { RuleExecutionStatus } from '@kbn/security-solution-plugin/common/detection_engine/rule_monitoring'; import { ThresholdCreateSchema } from '@kbn/security-solution-plugin/common/detection_engine/schemas/request'; import { FtrProviderContext } from '../../common/ftr_provider_context'; import { diff --git a/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/create_new_terms.ts b/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/create_new_terms.ts index 7fb6411b8b885..2f0fa70754e33 100644 --- a/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/create_new_terms.ts +++ b/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/create_new_terms.ts @@ -8,7 +8,7 @@ import { orderBy } from 'lodash'; import expect from '@kbn/expect'; -import { RuleExecutionStatus } from '@kbn/security-solution-plugin/common/detection_engine/schemas/common'; +import { RuleExecutionStatus } from '@kbn/security-solution-plugin/common/detection_engine/rule_monitoring'; import { NewTermsCreateSchema } from '@kbn/security-solution-plugin/common/detection_engine/schemas/request'; import { DETECTION_ENGINE_RULES_URL } from '@kbn/security-solution-plugin/common/constants'; import { getCreateNewTermsRulesSchemaMock } from '@kbn/security-solution-plugin/common/detection_engine/schemas/request/rule_schemas.mock'; diff --git a/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/create_rules.ts b/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/create_rules.ts index babf1aedb64a3..5b9ebc3483e95 100644 --- a/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/create_rules.ts +++ b/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/create_rules.ts @@ -8,7 +8,7 @@ import expect from '@kbn/expect'; import { DETECTION_ENGINE_RULES_URL } from '@kbn/security-solution-plugin/common/constants'; -import { RuleExecutionStatus } from '@kbn/security-solution-plugin/common/detection_engine/schemas/common'; +import { RuleExecutionStatus } from '@kbn/security-solution-plugin/common/detection_engine/rule_monitoring'; import { CreateRulesSchema } from '@kbn/security-solution-plugin/common/detection_engine/schemas/request'; import { ROLES } from '@kbn/security-solution-plugin/common/test'; diff --git a/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/create_threat_matching.ts b/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/create_threat_matching.ts index b38b9b09bbe96..a472e75582481 100644 --- a/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/create_threat_matching.ts +++ b/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/create_threat_matching.ts @@ -20,9 +20,9 @@ import { } from '@kbn/rule-data-utils'; import { flattenWithPrefix } from '@kbn/securitysolution-rules'; -import { RuleExecutionStatus } from '@kbn/security-solution-plugin/common/detection_engine/schemas/common'; -import { CreateRulesSchema } from '@kbn/security-solution-plugin/common/detection_engine/schemas/request'; import { DETECTION_ENGINE_RULES_URL } from '@kbn/security-solution-plugin/common/constants'; +import { RuleExecutionStatus } from '@kbn/security-solution-plugin/common/detection_engine/rule_monitoring'; +import { CreateRulesSchema } from '@kbn/security-solution-plugin/common/detection_engine/schemas/request'; import { getCreateThreatMatchRulesSchemaMock } from '@kbn/security-solution-plugin/common/detection_engine/schemas/request/rule_schemas.mock'; import { getThreatMatchingSchemaPartialMock } from '@kbn/security-solution-plugin/common/detection_engine/schemas/response/rules_schema.mocks'; diff --git a/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/generating_signals.ts b/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/generating_signals.ts index 1e953795887f2..916d846264cbb 100644 --- a/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/generating_signals.ts +++ b/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/generating_signals.ts @@ -23,7 +23,7 @@ import { flattenWithPrefix } from '@kbn/securitysolution-rules'; import { orderBy, get } from 'lodash'; -import { RuleExecutionStatus } from '@kbn/security-solution-plugin/common/detection_engine/schemas/common'; +import { RuleExecutionStatus } from '@kbn/security-solution-plugin/common/detection_engine/rule_monitoring'; import { EqlCreateSchema, QueryCreateSchema, diff --git a/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/get_rule_execution_events.ts b/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/get_rule_execution_results.ts similarity index 94% rename from x-pack/test/detection_engine_api_integration/security_and_spaces/group1/get_rule_execution_events.ts rename to x-pack/test/detection_engine_api_integration/security_and_spaces/group1/get_rule_execution_results.ts index b5743935a3688..31506b5a066c0 100644 --- a/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/get_rule_execution_events.ts +++ b/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/get_rule_execution_results.ts @@ -11,8 +11,10 @@ import expect from '@kbn/expect'; import moment from 'moment'; import { set } from '@elastic/safer-lodash-set'; import uuid from 'uuid'; -import { detectionEngineRuleExecutionEventsUrl } from '@kbn/security-solution-plugin/common/constants'; -import { RuleExecutionStatus } from '@kbn/security-solution-plugin/common/detection_engine/schemas/common'; +import { + getRuleExecutionResultsUrl, + RuleExecutionStatus, +} from '@kbn/security-solution-plugin/common/detection_engine/rule_monitoring'; import { FtrProviderContext } from '../../common/ftr_provider_context'; import { @@ -39,7 +41,7 @@ export default ({ getService }: FtrProviderContext) => { const es = getService('es'); const log = getService('log'); - describe('Get Rule Execution Log Events', () => { + describe('Get Rule Execution Results', () => { before(async () => { await esArchiver.load('x-pack/test/functional/es_archives/auditbeat/hosts'); await esArchiver.load('x-pack/test/functional/es_archives/security_solution/alias'); @@ -61,7 +63,7 @@ export default ({ getService }: FtrProviderContext) => { const start = dateMath.parse('now-24h')?.utc().toISOString(); const end = dateMath.parse('now', { roundUp: true })?.utc().toISOString(); const response = await supertest - .get(detectionEngineRuleExecutionEventsUrl('1')) + .get(getRuleExecutionResultsUrl('1')) .set('kbn-xsrf', 'true') .query({ start, end }); @@ -80,7 +82,7 @@ export default ({ getService }: FtrProviderContext) => { const start = dateMath.parse('now-24h')?.utc().toISOString(); const end = dateMath.parse('now', { roundUp: true })?.utc().toISOString(); const response = await supertest - .get(detectionEngineRuleExecutionEventsUrl(id)) + .get(getRuleExecutionResultsUrl(id)) .set('kbn-xsrf', 'true') .query({ start, end }); @@ -92,7 +94,9 @@ export default ({ getService }: FtrProviderContext) => { expect(response.body.events[0].indexing_duration_ms).to.greaterThan(0); expect(response.body.events[0].gap_duration_s).to.eql(0); expect(response.body.events[0].security_status).to.eql('succeeded'); - expect(response.body.events[0].security_message).to.eql('succeeded'); + expect(response.body.events[0].security_message).to.eql( + 'Rule execution completed successfully' + ); }); it('should return execution events for a rule that has executed in a warning state', async () => { @@ -104,7 +108,7 @@ export default ({ getService }: FtrProviderContext) => { const start = dateMath.parse('now-24h')?.utc().toISOString(); const end = dateMath.parse('now', { roundUp: true })?.utc().toISOString(); const response = await supertest - .get(detectionEngineRuleExecutionEventsUrl(id)) + .get(getRuleExecutionResultsUrl(id)) .set('kbn-xsrf', 'true') .query({ start, end }); @@ -151,7 +155,7 @@ export default ({ getService }: FtrProviderContext) => { await waitForEventLogExecuteComplete(es, log, id); const response = await supertest - .get(detectionEngineRuleExecutionEventsUrl(id)) + .get(getRuleExecutionResultsUrl(id)) .set('kbn-xsrf', 'true') .query({ start, end }); @@ -222,7 +226,7 @@ export default ({ getService }: FtrProviderContext) => { // Be sure to provide between 1-2 filters so that the server must prefetch events const response = await supertest - .get(detectionEngineRuleExecutionEventsUrl(id)) + .get(getRuleExecutionResultsUrl(id)) .set('kbn-xsrf', 'true') .query({ start, end, status_filters: 'failed,succeeded' }); diff --git a/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/index.ts b/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/index.ts index 37a60223498b4..11b1e5e5deda0 100644 --- a/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/index.ts +++ b/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/index.ts @@ -32,7 +32,7 @@ export default ({ loadTestFile }: FtrProviderContext): void => { loadTestFile(require.resolve('./find_rules')); loadTestFile(require.resolve('./generating_signals')); loadTestFile(require.resolve('./get_prepackaged_rules_status')); - loadTestFile(require.resolve('./get_rule_execution_events')); + loadTestFile(require.resolve('./get_rule_execution_results')); loadTestFile(require.resolve('./import_rules')); loadTestFile(require.resolve('./import_export_rules')); loadTestFile(require.resolve('./legacy_actions_migrations')); diff --git a/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/timestamps.ts b/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/timestamps.ts index 0b055017524ff..db7ed86b97f71 100644 --- a/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/timestamps.ts +++ b/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/timestamps.ts @@ -7,7 +7,7 @@ import expect from '@kbn/expect'; import { orderBy } from 'lodash'; -import { RuleExecutionStatus } from '@kbn/security-solution-plugin/common/detection_engine/schemas/common'; +import { RuleExecutionStatus } from '@kbn/security-solution-plugin/common/detection_engine/rule_monitoring'; import { EqlCreateSchema, QueryCreateSchema, diff --git a/x-pack/test/detection_engine_api_integration/utils/get_open_signals.ts b/x-pack/test/detection_engine_api_integration/utils/get_open_signals.ts index 6ce1a61c6bda7..f5d880cb3433e 100644 --- a/x-pack/test/detection_engine_api_integration/utils/get_open_signals.ts +++ b/x-pack/test/detection_engine_api_integration/utils/get_open_signals.ts @@ -5,10 +5,10 @@ * 2.0. */ -import type { ToolingLog } from '@kbn/tooling-log'; import type SuperTest from 'supertest'; import type { Client } from '@elastic/elasticsearch'; -import { RuleExecutionStatus } from '@kbn/security-solution-plugin/common/detection_engine/schemas/common'; +import type { ToolingLog } from '@kbn/tooling-log'; +import { RuleExecutionStatus } from '@kbn/security-solution-plugin/common/detection_engine/rule_monitoring'; import type { FullResponseSchema } from '@kbn/security-solution-plugin/common/detection_engine/schemas/request'; import { waitForRuleSuccessOrStatus } from './wait_for_rule_success_or_status'; diff --git a/x-pack/test/detection_engine_api_integration/utils/wait_for_rule_success_or_status.ts b/x-pack/test/detection_engine_api_integration/utils/wait_for_rule_success_or_status.ts index 0ab4385a54e66..aed1ca76b6a78 100644 --- a/x-pack/test/detection_engine_api_integration/utils/wait_for_rule_success_or_status.ts +++ b/x-pack/test/detection_engine_api_integration/utils/wait_for_rule_success_or_status.ts @@ -8,8 +8,8 @@ import type { ToolingLog } from '@kbn/tooling-log'; import type SuperTest from 'supertest'; -import { RuleExecutionStatus } from '@kbn/security-solution-plugin/common/detection_engine/schemas/common'; import { DETECTION_ENGINE_RULES_URL } from '@kbn/security-solution-plugin/common/constants'; +import { RuleExecutionStatus } from '@kbn/security-solution-plugin/common/detection_engine/rule_monitoring'; import { waitFor } from './wait_for'; /**