From 302b1c0bd5cc54fbb89e17fb82b875a352f761cc Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Thu, 12 Dec 2024 12:40:27 -0500 Subject: [PATCH] [8.17] Document impact of using logsDB for security users (#6272) * First draft * Removes serverless content * Minor change * Removes unnecessary words * Fixes casing for all mentions of logsdb ESS and Serverless * Update docs/detections/detections-logsdb-impact.asciidoc Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> * Update docs/detections/detections-logsdb-impact.asciidoc Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> * Update docs/serverless/rules/detections-logsdb-impact.asciidoc Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> * Update docs/serverless/rules/detections-logsdb-impact.asciidoc Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> * Adds licensing info --------- Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> (cherry picked from commit 61b1f1b142c5369c708ec796ce77e22d59f62888) # Conflicts: # docs/serverless/rules/detection-engine-overview.asciidoc # docs/serverless/rules/detections-logsdb-impact.asciidoc --- .../detection-engine-intro.asciidoc | 6 + docs/detections/detections-index.asciidoc | 2 + .../detections-logsdb-impact.asciidoc | 65 ++++++++ .../rules/detection-engine-overview.asciidoc | 149 ++++++++++++++++++ .../rules/detections-logsdb-impact.asciidoc | 63 ++++++++ 5 files changed, 285 insertions(+) create mode 100644 docs/detections/detections-logsdb-impact.asciidoc create mode 100644 docs/serverless/rules/detection-engine-overview.asciidoc create mode 100644 docs/serverless/rules/detections-logsdb-impact.asciidoc diff --git a/docs/detections/detection-engine-intro.asciidoc b/docs/detections/detection-engine-intro.asciidoc index b522d3f4bd..105e2ade3e 100644 --- a/docs/detections/detection-engine-intro.asciidoc +++ b/docs/detections/detection-engine-intro.asciidoc @@ -167,3 +167,9 @@ and you should contact your {kib} administrator. NOTE: For *self-managed* {stack} deployments only, this message may be displayed when the <> setting is not enabled in the `elasticsearch.yml` file. For more information, refer to <>. + +[discrete] +[[detections-logsdb-index-mode]] +== Using logsdb index mode + +To learn how your rules and alerts are affected by using the {ref}/logs-data-stream.html[logsdb index mode], refer to <>. \ No newline at end of file diff --git a/docs/detections/detections-index.asciidoc b/docs/detections/detections-index.asciidoc index 2d1cab74d6..ff45aa827b 100644 --- a/docs/detections/detections-index.asciidoc +++ b/docs/detections/detections-index.asciidoc @@ -2,6 +2,8 @@ include::detection-engine-intro.asciidoc[] include::detections-req.asciidoc[leveloffset=+1] +include::detections-logsdb-impact.asciidoc[leveloffset=+1] + include::about-rules.asciidoc[] diff --git a/docs/detections/detections-logsdb-impact.asciidoc b/docs/detections/detections-logsdb-impact.asciidoc new file mode 100644 index 0000000000..1b304c7f1e --- /dev/null +++ b/docs/detections/detections-logsdb-impact.asciidoc @@ -0,0 +1,65 @@ +[[detections-logsdb-index-mode-impact]] += Using logsdb index mode with {elastic-sec} + +NOTE: To use the {ref}/mapping-source-field.html#synthetic-source[synthetic `_source`] feature, you must have the appropriate subscription. Refer to the subscription page for https://www.elastic.co/subscriptions/cloud[Elastic Cloud] and {subscriptions}[Elastic Stack/self-managed] for the breakdown of available features and their associated subscription tiers. + +This topic explains the impact of using logsdb index mode with {elastic-sec}. + +With logsdb index mode, the original `_source` field is not stored in the index but can be reconstructed using {ref}/mapping-source-field.html#synthetic-source[synthetic `_source`]. + +When the `_source` is reconstructed, {ref}/mapping-source-field.html#synthetic-source-modifications[modifications] are possible. Therefore, there could be a mismatch between users' expectations and how fields are formatted. + +Continue reading to find out how this affects specific {elastic-sec} components. + +[discrete] +[[logsdb-alerts]] +== Alerts + +When alerts are generated, the `_source` event is copied into the alert to retain the original data. When the logsdb index mode is applied, the `_source` event stored in the alert is reconstructed using synthetic `_source`. + +If you're switching to use logsdb index mode, the `_source` field stored in the alert might look different in certain situations: + +* {ref}/mapping-source-field.html#synthetic-source-modifications-leaf-arrays[Arrays can be reconstructed differently or deduplicated] +* {ref}/mapping-source-field.html#synthetic-source-modifications-field-names[Field names] +* `geo_point` data fields (refer to {ref}/mapping-source-field.html#synthetic-source-modifications-ranges[Representation of ranges] and {ref}/mapping-source-field.html#synthetic-source-precision-loss-for-point-types[Reduced precision of `geo_point` values] for more information) + +Alerts generated by the following rule types could be affected: + +* Custom query +* Event correlation (non-sequence only) +* Non-aggregate rule types (for example, {esql} rules that use non-aggregating queries) + +Alerts that are generated by threshold, {ml}, and event correlation sequence rules are not affected since they do not contain copies of the original source. + +[discrete] +[[logsdb-rule-actions]] +== Rule actions + +While we do not recommend using `_source` for actions, in cases where the action relies on the `_source`, the same limitations and changes apply. + +If you send alert notifications by enabling {kibana-ref}/alerting-getting-started.html#alerting-concepts-actions[actions] to the external systems that have workflows or automations based on fields formatted from the original source, they may be affected. In particular, this can happen when the fields used are arrays of objects. + +We recommend checking and adjusting the rule actions using `_source` before switching to logsdb index mode. + +[discrete] +[[logsdb-runtime-fields]] +== Runtime fields + +Runtime fields that reference `_source` may be affected. Some runtime fields might not work and need to be adjusted. For example, if an event was indexed with the value of `agent.name` in the dot-notation form, it will be returned in the nested form and might not work. + +The following is an example of accessing `_source` that works with the logsdb index mode enabled: + +[source,console] +---- +"source": """ emit(params._source.agent.name + "_____" + doc['agent.name'].value ); """ +"source": """ emit(params._source['agent']['name'] + "_____" + doc['agent.name'].value ); """ +"source": """ emit(field('agent.name').get(null) + "_____" + doc['agent.name'].value ); """ +"source": """ emit($('agent.name', null) + "_____" + doc['agent.name'].value ); """ +---- + +The following will not work with synthetic source (logsdb index mode enabled): + +[source,console] +---- +"source": """ emit(params._source['agent.name'] + "_____" + doc['agent.name'].value ); """ +---- \ No newline at end of file diff --git a/docs/serverless/rules/detection-engine-overview.asciidoc b/docs/serverless/rules/detection-engine-overview.asciidoc new file mode 100644 index 0000000000..605a843dc8 --- /dev/null +++ b/docs/serverless/rules/detection-engine-overview.asciidoc @@ -0,0 +1,149 @@ +[[security-detection-engine-overview]] += Detection engine overview + +// :description: Learn about the detection engine and its features. +// :keywords: serverless, security, overview + + +Use the detection engine to create and manage rules and view the alerts +these rules create. Rules periodically search indices (such as `logs-*` and +`filebeat-*`) for suspicious source events and create alerts when a rule's +conditions are met. When an alert is created, its status is `Open`. To help +track investigations, an alert's <> can be set as +`Open`, `Acknowledged`, or `Closed`. + +[role="screenshot"] +image::images/detection-engine-overview/-detections-alert-page.png[Alerts page] + +In addition to creating <>, enable +<> to immediately start detecting +suspicious activity. For detailed information on all the prebuilt rules, see the <>. Once the prebuilt rules are loaded and +running, <> and <> explain +how to modify the rules to reduce false positives and get a better set of +actionable alerts. You can also use exceptions and value lists when creating or +modifying your own rules. + +There are two special prebuilt rules you need to know about: + +// Links to prebuilt rule pages temporarily removed for initial serverless docs. + +* **Endpoint Security**: +Automatically creates an alert from all incoming Elastic Endpoint alerts. To +receive Elastic Endpoint alerts, you must install the Endpoint agent on your +hosts (see <>). ++ +When this rule is enabled, the following Endpoint events are displayed as +detection alerts: ++ +** Malware Prevention Alert +** Malware Detection Alert ++ +[NOTE] +==== +When you load the prebuilt rules, this is the only rule that is enabled +by default. +==== + +// Links to prebuilt rule pages temporarily removed for initial serverless docs. + +* **External Alerts**: Automatically creates an alert for +all incoming third-party system alerts (for example, Suricata alerts). + +If you want to receive notifications via external systems, such as Slack or +email, when alerts are created, use the {kibana-ref}/alerting-getting-started.html[Alerting and Actions] framework. + +After rules have started running, you can monitor their executions to verify +they are functioning correctly, as well as view, manage, and troubleshoot +alerts (see <> and <>). + +You can create and manage rules and alerts via the UI or the {security-guide}/rule-api-overview.html[Detections API]. + +// Link to classic docs until serverless API docs are available. + +[IMPORTANT] +==== +To make sure you can access Detections and manage rules, see +<>. +==== + +[discrete] +[[support-indicator-rules]] +== Limited support for indicator match rules + +Indicator match rules provide a powerful capability to search your security data; however, their queries can consume significant deployment resources. When creating an <>, we recommend limiting the time range of the indicator index query to the minimum period necessary for the desired rule coverage. For example, the default indicator index query `@timestamp > "now-30d/d"` searches specified indicator indices for indicators ingested during the past 30 days and rounds the query start time down to the nearest day (resolves to UTC `00:00:00`). Without this limitation, the rule will include all of the indicators in your indicator indices, which may extend the time it takes for the indicator index query to complete. + +In addition, indicator match rules with an additional look-back time value greater than 24 hours are not supported. + +[discrete] +[[detections-permissions]] +== Detections configuration and prerequisites + +<> provides detailed information on all the +permissions required to initiate and use the Detections feature. + +[discrete] +[[malware-prevention]] +== Malware prevention + +Malware, short for malicious software, is any software program designed to damage or execute unauthorized actions on a +computer system. Examples of malware include viruses, worms, Trojan horses, adware, scareware, and spyware. Some +malware, such as viruses, can severely damage a computer's hard drive by deleting files or directory information. Other +malware, such as spyware, can obtain user data without their knowledge. + +Malware may be stealthy and appear as legitimate executable code, scripts, active content, and other software. It is also +often embedded in non-malicious files, non-suspicious websites, and standard programs — sometimes making the root +source difficult to identify. If infected and not resolved promptly, malware can cause irreparable damage to a computer +network. + +For information on how to enable malware protection on your host, see <>. + +[discrete] +[[machine-learning-model]] +=== Machine learning model + +To determine if a file is malicious or benign, a machine learning model looks for static attributes of files (without executing +the file) that include file structure, layout, and content. This includes information such as file header data, imports, exports, +section names, and file size. These attributes are extracted from millions of benign and malicious file samples, which then +are passed to a machine-learning algorithm that distinguishes a benign file from a malicious one. The machine learning +model is updated as new data is procured and analyzed. + +[discrete] +[[security-detection-engine-overview-threshold]] +=== Threshold + +A malware threshold determines the action the agent should take if malware is detected. The Elastic Agent uses a recommended threshold level that generates a balanced number of alerts with a low probability of undetected malware. This threshold also minimizes the number of false positive alerts. + +[discrete] +[[ransomware-prevention]] +== Ransomware prevention + +Ransomware is computer malware that installs discreetly on a user's computer and encrypts data until a specified amount of money (ransom) is paid. Ransomware is usually similar to other malware in its delivery and execution, infecting systems +through spear-phishing or drive-by downloads. If not resolved immediately, ransomware can cause irreparable damage to an entire computer network. + +Behavioral ransomware prevention on the Elastic Endpoint detects and stops ransomware attacks on Windows systems by analyzing data from low-level system processes, and is effective across an array of widespread ransomware families — including those targeting the system’s master boot record. + +For information on how to enable ransomware protection on your host, see <>. + +[discrete] +[[security-detection-engine-overview-resolve-ui-error-messages]] +=== Resolve UI error messages + +Depending on your user role privileges and whether detection system indices have already been created, you might get one of these error messages when you +open the **Alerts** or **Rules** page: + +* **`Let’s set up your detection engine`** ++ +If you get this message, a user with specific privileges must visit the +**Alerts** or **Rules** page before you can view detection alerts and rules. +Refer to <> for a list of all the requirements. +* **`Detection engine permissions required`** ++ +If you get this message, you do not have the +<> to view the **Detections** feature, +and you should contact your project administrator. + +[discrete] +[[detections-logsdb-index-mode]] +== Using logsdb index mode + +logsdb is enabled by default for Elastic serverless. Refer to <> to learn more. \ No newline at end of file diff --git a/docs/serverless/rules/detections-logsdb-impact.asciidoc b/docs/serverless/rules/detections-logsdb-impact.asciidoc new file mode 100644 index 0000000000..dbbceda14c --- /dev/null +++ b/docs/serverless/rules/detections-logsdb-impact.asciidoc @@ -0,0 +1,63 @@ +[[detections-logsdb-index-mode-impact]] += Using logsdb index mode with {sec-serverless} + +Logsdb is enabled by default for {serverless-full}. This topic explains the impact of using logsdb index mode with {sec-serverless}. + +With logsdb index mode, the original `_source` field is not stored in the index but can be reconstructed using {ref}/mapping-source-field.html#synthetic-source[synthetic `_source`]. + +When the `_source` is reconstructed, {ref}/mapping-source-field.html#synthetic-source-modifications[modifications] are possible. Therefore, there could be a mismatch between users' expectations and how fields are formatted. + +Continue reading to find out how this affects specific {sec-serverless} components. + +[discrete] +[[logsdb-alerts]] +== Alerts + +When alerts are generated, the `_source` event is copied into the alert to retain the original data. When the logsdb index mode is applied, the `_source` event stored in the alert is reconstructed using synthetic `_source`. + +If you're switching to use logsdb index mode, the `_source` field stored in the alert might look different in certain situations: + +* {ref}/mapping-source-field.html#synthetic-source-modifications-leaf-arrays[Arrays can be reconstructed differently or deduplicated] +* {ref}/mapping-source-field.html#synthetic-source-modifications-field-names[Field names] +* `geo_point` data fields (refer to {ref}/mapping-source-field.html#synthetic-source-modifications-ranges[Representation of ranges] and {ref}/mapping-source-field.html#synthetic-source-precision-loss-for-point-types[Reduced precision of `geo_point` values] for more information) + +Alerts generated by the following rule types could be affected: + +* Custom query +* Event correlation (non-sequence only) +* Non-aggregate rule types (for example, {esql} rules that use non-aggregating queries) + +Alerts that are generated by threshold, {ml}, and event correlation sequence rules are not affected since they do not contain copies of the original source. + +[discrete] +[[logsdb-rule-actions]] +== Rule actions + +While we do not recommend using `_source` for actions, in cases where the action relies on the `_source`, the same limitations and changes apply. + +If you send alert notifications by enabling {kibana-ref}/alerting-getting-started.html#alerting-concepts-actions[actions] to the external systems that have workflows or automations based on fields formatted from the original source, they may be affected. In particular, this can happen when the fields used are arrays of objects. + +We recommend checking and adjusting the rule actions using `_source` before switching to logsdb index mode. + +[discrete] +[[logsdb-runtime-fields]] +== Runtime fields + +Runtime fields that reference `_source` may be affected. Some runtime fields might not work and need to be adjusted. For example, if an event was indexed with the value of `agent.name` in the dot-notation form, it will be returned in the nested form and might not work. + +The following is an example of accessing `_source` that works with the logsdb index mode enabled: + +[source,console] +---- +"source": """ emit(params._source.agent.name + "_____" + doc['agent.name'].value ); """ +"source": """ emit(params._source['agent']['name'] + "_____" + doc['agent.name'].value ); """ +"source": """ emit(field('agent.name').get(null) + "_____" + doc['agent.name'].value ); """ +"source": """ emit($('agent.name', null) + "_____" + doc['agent.name'].value ); """ +---- + +The following will not work with synthetic source (logsdb index mode enabled): + +[source,console] +---- +"source": """ emit(params._source['agent.name'] + "_____" + doc['agent.name'].value ); """ +---- \ No newline at end of file