From 180eab840afabcf7f59e7e12995fbd4d854d4246 Mon Sep 17 00:00:00 2001 From: Vitalii Dmyterko <92328789+vitaliidm@users.noreply.github.com> Date: Thu, 26 Oct 2023 15:34:39 +0100 Subject: [PATCH] [Detection Engine] update API docs for new ES|QL rule (#4052) * initial commit * updates doc * update API * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> * Update docs/detections/api/rules/rules-api-update.asciidoc Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> * CR feedback * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> * CR * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> * Update docs/detections/api/rules/rules-api-update.asciidoc Co-authored-by: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> * Combining tables and adding note to update rule * Update docs/detections/api/rules/rules-api-create.asciidoc * Removing extra term * Combining tables in create rule * Fixing tag rendering * Adding ref to UI docs * littile cleanup and fixes * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> --------- Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> Co-authored-by: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Co-authored-by: nastasha.solomon (cherry picked from commit 5f33c461b4d0476145e5546e24df21f58994c2ab) --- .../api/rules/rules-api-create.asciidoc | 141 ++++++++++++++---- .../api/rules/rules-api-update.asciidoc | 54 ++++--- 2 files changed, 143 insertions(+), 52 deletions(-) diff --git a/docs/detections/api/rules/rules-api-create.asciidoc b/docs/detections/api/rules/rules-api-create.asciidoc index 86f43ba4fd..71ade8249b 100644 --- a/docs/detections/api/rules/rules-api-create.asciidoc +++ b/docs/detections/api/rules/rules-api-create.asciidoc @@ -31,9 +31,12 @@ specified {ref}/indices-create-index.html[{es} index]. For example, you can create an index for IP addresses and use this index to create an alert whenever an event's `destination.ip` equals a value in the index. The index's field mappings should be {ecs-ref}[ECS-compliant]. +* *New terms*: Generates an alert for each new term detected in source documents within a specified time range. +* *ES|QL*: Uses {ref}/esql.html[Elasticsearch Query Language (ES|QL)] to find events and aggregate search results. ++ +preview::[] * *{ml-cap} rules*: Creates an alert when a {ml} job discovers an anomaly above the defined threshold (see <>). -* *New terms*: Generates an alert for each new term detected in source documents within a specified time range. IMPORTANT: To create {ml} rules, you must have the https://www.elastic.co/subscriptions[appropriate license] or use a @@ -102,8 +105,8 @@ A JSON object that defines the rule's values: * <> * <> * <> +* <> * <> -* <> * <> * <> * <> @@ -140,6 +143,7 @@ occurred |type |String a|Data type on which the rule is based: * `eql`: EQL query (see {ref}/eql.html[Event Query Language]). +* `esql`: ES\|QL query (see {ref}/esql.html[Elasticsearch Query Language]). * `query`: query with or without additional filters. * `saved_query`: saved search, identified in the `saved_id` field. * `machine_learning`: rule based on a {ml} job's anomaly scores. @@ -152,15 +156,21 @@ specified field. |============================================== [[req-fields-query-threshold]] -===== Required field for query, indicator match, threshold, and new terms rules +===== Required fields for query, indicator match, threshold, new terms, event correlation, and ES|QL rules [width="100%",options="header"] |============================================== |Name |Type |Description |query |String a|{kibana-ref}/search.html[Query] used by the rule to create -alerts. For indicator match rules, only the query's results are used to determine -whether an alert is generated. +alerts. + +[NOTE] +====== +* For indicator match rules, only the query's results are used to determine +whether an alert is generated. +* ES\|QL rules have additional query requirements. Refer to <> for more information. +====== |============================================== @@ -209,6 +219,17 @@ generated. |============================================== +[[req-fields-esql]] +===== Required field for ES|QL rules + +[width="100%",options="header"] +|============================================== +|Name |Type |Description + +|language |String |Must be `esql`. + +|============================================== + [[req-fields-ml]] ===== Required fields for machine learning rules @@ -354,27 +375,6 @@ Required when `actions` are used to send notifications. |============================================== -[[opt-fields-query-eql]] -===== Optional fields for query, indicator match, event correlation, and new terms rules - -[width="100%",options="header"] -|============================================== -|Name |Type |Description - -|exceptions_list |Object[] a|Array of -<>, which define -exceptions that prevent the rule from generating alerts even when its other -criteria are met. The object has these fields: - -* `id` (string, required): ID of the exception container. -* `list_id` (string, required): List ID of the exception container. -* `namespace_type` (string required): Determines whether the exceptions are -valid in only the rule's {kib} space (`single`) or in all {kib} spaces -(`agnostic`). -* `type` (string, required): The exception type, which must be either -a detection rule exception (`detection`) or an endpoint exception (`endpoint`). -|============================================== - [[opt-fields-threat-match]] ===== Optional fields for indicator match rules @@ -402,21 +402,25 @@ documents from the {es} index containing the threat values. |============================================== [[opt-fields-eql-query-threshold]] -===== Optional fields for event correlation, query, threshold, and new terms rules +===== Optional fields for event correlation, query, threshold, indicator match, new terms, and ES|QL rules [width="100%",options="header"] |============================================== |Name |Type |Description -|filters |Object[] |The {ref}/query-filter-context.html[query and filter +|filters |Object[] a|The {ref}/query-filter-context.html[query and filter context] array used to define the conditions for when alerts are created from events. Defaults to an empty array. -|index |String[] |Indices on which the rule functions. Defaults to the +NOTE: This field is not supported for ES\|QL rules. + +|index |String[] a|Indices on which the rule functions. Defaults to the Security Solution indices defined on the {kib} Advanced Settings page (*Kibana* → *Stack Management* → *Advanced Settings* → `securitySolution:defaultIndex`). +NOTE: This field is not supported for ES\|QL rules. + |risk_score_mapping |Object[] a|Overrides generated alerts' `risk_score` with a value from the source event: @@ -444,6 +448,19 @@ values from the source event: When unspecified, rules query the `@timestamp` field. The source field must be an {es} date data type. +|exceptions_list |Object[] a|Array of +<>, which define +exceptions that prevent the rule from generating alerts even when its other +criteria are met. The object has these fields: + +* `id` (string, required): ID of the exception container. +* `list_id` (string, required): List ID of the exception container. +* `namespace_type` (string required): Determines whether the exceptions are +valid in only the rule's {kib} space (`single`) or in all {kib} spaces +(`agnostic`). +* `type` (string, required): The exception type, which must be either +a detection rule exception (`detection`) or an endpoint exception (`endpoint`). + |============================================== [[opt-fields-eql-create]] @@ -969,6 +986,29 @@ POST api/detection_engine/rules `200`:: Indicates a successful call. +*Example 7* + +ES|QL rule that creates alerts from events that match an Excel parent process: + +[source,json] +-------------------------------------------------- +POST api/detection_engine/rules +{ + "type": "esql", + "language": "esql", + "query": "from auditbeat-8.10.2 [metadata _id, _version, _index] | where process.parent.name == \"EXCEL.EXE\"", + "name": "Find Excel events", + "description": "Find Excel events", + "tags": [], + "interval": "5m", + "from": "now-360s", + "to": "now", + "enabled": false, + "risk_score": 21, + "severity": "low" +} +-------------------------------------------------- + ==== Response payload A JSON object that includes a unique ID, the time the rule was created, and its @@ -1327,3 +1367,46 @@ Example response for a new terms rule: } -------------------------------------------------- <1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`. + +Example response for an ES|QL rule: + +[source,json] +-------------------------------------------------- +{ + "name": "Find Excel events", + "description": "Find Excel events", + "risk_score": 21, + "severity": "low", + "output_index": "", + "tags": [], + "interval": "5m", + "enabled": false, + "author": [], + "false_positives": [], + "from": "now-360s", + "max_signals": 100, + "risk_score_mapping": [], + "severity_mapping": [], + "threat": [], + "to": "now", + "references": [], + "version": 1, + "exceptions_list": [], + "actions": [], + "id": "d0f20490-6da4-11ee-b85e-09e9b661f2e2", + "updated_at": "2023-10-18T10:55:14.269Z", + "updated_by": "elastic", + "created_at": "2023-10-18T10:55:14.269Z", + "created_by": "elastic", + "revision": 0, + "rule_id": "e4b53a89-debd-4a0d-a3e3-20606952e589", + "immutable": false, + "related_integrations": [], <1> + "required_fields": [], <1> + "setup": "", <1> + "type": "esql", + "language": "esql", + "query": "from auditbeat-8.10.2 [metadata _id] | where process.parent.name == \"EXCEL.EXE\"" +} +-------------------------------------------------- +<1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`. diff --git a/docs/detections/api/rules/rules-api-update.asciidoc b/docs/detections/api/rules/rules-api-update.asciidoc index 8552edf606..8c9cb35bb6 100644 --- a/docs/detections/api/rules/rules-api-update.asciidoc +++ b/docs/detections/api/rules/rules-api-update.asciidoc @@ -60,6 +60,7 @@ occurred |type |String a|Data type on which the rule is based: * `eql`: EQL query (see {ref}/eql.html[Event Query Language]). +* `esql`: ES\|QL query (refer to {ref}/esql.html[Elasticsearch Query Language]). * `query`: query with or without additional filters. * `saved_query`: saved search, identified in the `saved_id` field. * `machine_learning`: rule based on a {ml} job's anomaly scores. @@ -125,6 +126,16 @@ generated. |============================================== +===== Field required for ES|QL rules `PUT` calls + +[width="100%",options="header"] +|============================================== +|Name |Type |Description + +|language |String |Must be `esql`. + +|============================================== + ===== Fields required for machine-learning rules `PUT` calls [width="100%",options="header"] @@ -260,26 +271,6 @@ number. |============================================== -===== Optional fields for query, threat-match, EQL, and new terms rules - -[width="100%",options="header"] -|============================================== -|Name |Type |Description - -|exceptions_list |Object[] a|Array of -<>, which define -exceptions that prevent the rule from generating alerts even when its other -criteria are met. The object has these fields: - -* `id` (string, required): ID of the exception container. -* `list_id` (string, required): List ID of the exception container. -* `namespace_type` (string required): Determines whether the exceptions are -valid in only the rule's {kib} space (`single`) or in all {kib} spaces -(`agnostic`). -* `type` (string, required): The exception type, which must be either -a detection rule exception (`detection`) or an endpoint exception (`endpoint`). -|============================================== - ===== Optional fields for threat-match rules [width="100%",options="header"] @@ -304,21 +295,25 @@ documents from the {es} index containing the threat values. `kuery` or `lucene`. Defaults to `kuery`. |============================================== -===== Optional fields for EQL, query, threshold, and new terms rules +===== Optional fields for EQL, query, threshold, indicator match, new terms rules, and ES|QL rules [width="100%",options="header"] |============================================== |Name |Type |Description -|filters |Object[] |The {ref}/query-filter-context.html[query and filter +|filters |Object[] a|The {ref}/query-filter-context.html[query and filter context] array used to define the conditions for when alerts are created from events. Defaults to an empty array. -|index |String[] |Indices on which the rule functions. Defaults to the +NOTE: This field is not supported for ES\|QL rules. + +|index |String[] a|Indices on which the rule functions. Defaults to the Security Solution indices defined on the {kib} Advanced Settings page (*Kibana* → *Stack Management* → *Advanced Settings* → `securitySolution:defaultIndex`). +NOTE: This field is not supported for ES\|QL rules. + |risk_score_mapping |Object[] a|Overrides generated alerts' `risk_score` with a value from the source event: @@ -346,6 +341,19 @@ values from the source event: When unspecified, rules query the `@timestamp` field. The source field must be an {es} date data type. +|exceptions_list |Object[] a|Array of +<>, which define +exceptions that prevent the rule from generating alerts even when its other +criteria are met. The object has these fields: + +* `id` (string, required): ID of the exception container. +* `list_id` (string, required): List ID of the exception container. +* `namespace_type` (string required): Determines whether the exceptions are +valid in only the rule's {kib} space (`single`) or in all {kib} spaces +(`agnostic`). +* `type` (string, required): The exception type, which must be either +a detection rule exception (`detection`) or an endpoint exception (`endpoint`). + |============================================== [[opt-fields-eql-update]]