[8.11] ES|QL rule docs (backport #4062) (#4196)

Co-authored-by: Benjamin Ironside Goldstein <[email protected]> Co-authored-by: Abdon Pijpelink <[email protected]> Co-authored-by: Nastasha Solomon <[email protected]>
elastic · Nov 6, 2023 · f6afb0b · f6afb0b
1 parent 3a297e3
commit f6afb0b
Show file tree

Hide file tree

Showing 5 changed files with 104 additions and 28 deletions.
diff --git a/docs/detections/about-rules.asciidoc b/docs/detections/about-rules.asciidoc
@@ -42,6 +42,10 @@ TIP: You can also use value lists as the indicator match index. See <<indicator-
 
 * <<create-new-terms-rule, *New terms*>>: Generates an alert for each new term detected in source documents within a specified time range. You can also detect a combination of up to three new terms (for example, a `host.ip` and `host.id` that have never been observed together before).
 
+* <<create-esql-rule, *ES|QL*>>: Searches the defined indices and creates an alert when results match an {ref}/esql.html[Elasticsearch Query Language (ES|QL)] query.
++
+preview::[]
+
 [role="screenshot"]
 image::images/all-rules.png[Shows the Rules page]
 

diff --git a/docs/detections/api/rules/rules-api-create.asciidoc b/docs/detections/api/rules/rules-api-create.asciidoc
@@ -37,7 +37,7 @@ create an index for IP addresses and use this index to create an alert whenever
 an event's `destination.ip` equals a value in the index. The index's field
 mappings should be {ecs-ref}[ECS-compliant].
 * *New terms*: Generates an alert for each new term detected in source documents within a specified time range.
-* *ES|QL*: Uses {ref}/esql.html[Elasticsearch Query Language (ES|QL)] to find events and aggregate search results.
+* *{esql}*: Uses {ref}/esql.html[Elasticsearch Query Language ({esql})] to find events and aggregate search results.
 +
 preview::[]
 * *{ml-cap} rules*: Creates an alert when a {ml} job discovers an anomaly above
@@ -161,7 +161,7 @@ specified field.
 |==============================================
 
 [[req-fields-query-threshold]]
-===== Required fields for query, indicator match, threshold, new terms, event correlation, and ES|QL rules
+===== Required fields for query, indicator match, threshold, new terms, event correlation, and {esql} rules
 
 [width="100%",options="header"]
 |==============================================
@@ -225,7 +225,7 @@ generated.
 |==============================================
 
 [[req-fields-esql]]
-===== Required field for ES|QL rules
+===== Required field for {esql} rules
 
 [width="100%",options="header"]
 |==============================================
@@ -407,7 +407,7 @@ documents from the {es} index containing the threat values.
 |==============================================
 
 [[opt-fields-eql-query-threshold]]
-===== Optional fields for event correlation, query, threshold, indicator match, new terms, and ES|QL rules
+===== Optional fields for event correlation, query, threshold, indicator match, new terms, and {esql} rules
 
 [width="100%",options="header"]
 |==============================================
@@ -993,7 +993,7 @@ POST api/detection_engine/rules
 
 *Example 7*
 
-ES|QL rule that creates alerts from events that match an Excel parent process:
+{esql} rule that creates alerts from events that match an Excel parent process:
 
 [source,json]
 --------------------------------------------------
@@ -1373,7 +1373,7 @@ Example response for a new terms rule:
 --------------------------------------------------
 <1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`.
 
-Example response for an ES|QL rule:
+Example response for an {esql} rule:
 
 [source,json]
 --------------------------------------------------

diff --git a/docs/detections/api/rules/rules-api-update.asciidoc b/docs/detections/api/rules/rules-api-update.asciidoc
@@ -131,7 +131,7 @@ generated.
 
 |==============================================
 
-===== Field required for ES|QL rules `PUT` calls
+===== Field required for {esql} rules `PUT` calls
 
 [width="100%",options="header"]
 |==============================================
@@ -300,7 +300,7 @@ documents from the {es} index containing the threat values.
 `kuery` or `lucene`. Defaults to `kuery`.
 |==============================================
 
-===== Optional fields for EQL, query, threshold, indicator match, new terms rules, and ES|QL rules 
+===== Optional fields for EQL, query, threshold, indicator match, new terms rules, and {esql} rules 
 
 [width="100%",options="header"]
 |==============================================

diff --git a/docs/detections/images/esql-ref-button.png b/docs/detections/images/esql-ref-button.png
diff --git a/docs/detections/rules-ui-create.asciidoc b/docs/detections/rules-ui-create.asciidoc
@@ -261,37 +261,109 @@ For example, if a rule has an interval of 5 minutes, no additional look-back tim
 
 [discrete]
 [[create-esql-rule]]
-=== Create an ES|QL rule
+=== Create an {esql} rule
 
-IMPORTANT: This is a placeholder for future documentation. The following content is incomplete.
+preview::[]
 
-. Go to *Rules* -> *Detection rules (SIEM)* -> *Create new rule*. The *Create new rule* page displays.
-. To create a rule that uses ES|QL, select **ES|QL**, 
-then write a query. There are two types of ES|QL queries:
+Use {ref}/esql.html[{esql}] to query your source events and aggregate event data. Query results are returned in a table with rows and columns. Each row becomes an alert.
+
+To create an {esql} rule:
+
+. Go to *Rules* -> *Detection rules (SIEM)* -> *Create new rule*. The *Create new rule* page appears.
+. Select **{esql}**, then write a <<esql-rule-query-types,query>>. 
++
+TIP: Click the help icon (image:images/esql-ref-button.png[Click the ES|QL help icon,20,20]) to open the in-product reference documentation for all {esql} commands and functions.
++
+[NOTE] 
+====
+To avoid conflicts between the {ref}/esql-commands.html#esql-limit[`LIMIT`] value (which defines how many rows are returned in the results table) and the <<opt-fields-all,`max_signals`>> value (which determines the maximum number of alerts a rule can generate per rule execution), make sure the `max_signals` value does not exceed the `LIMIT` value. 
 
-.. **Aggregating query**: This is a query that uses the `STATS...BY` grouping commands. Query results cannot be matched with a particular document in Elasticsearch. For example:
+If you do not define a `LIMIT` value, the rule automatically uses the `max_signals` value. 
+====
 +
-[esql]
+. Click *Continue* to <<rule-ui-basic-params, configure basic rule settings>>.
+
+[float]
+[[esql-rule-query-types]]
+==== {esql} query types 
+
+{esql} rule queries are loosely categorized into two types: aggregating and non-aggregating. 
+
+[float]
+[[esql-agg-query]]
+===== Aggregating query 
+
+Aggregating queries use {ref}/esql-functions-operators.html#esql-agg-functions[`STATS...BY`] functions to aggregate source event data. Alerts generated by an {esql} rule with an aggregating query only contain the fields returned by the query.
+
+Here is an example aggregating query:
+
+[source,esql]
+----
+FROM logs-*
+| STATS host_count = COUNT(host.name) BY host.name
+| SORT host_count DESC
+| WHERE host_count > 20
+----
+
+- This query starts by searching logs from indices that match the pattern `logs-*`. 
+- The query then aggregates the count of events by `host.name`.
+- Next, it sorts the result by `host_count` in descending order.
+- Then, it filters for events where the `host_count` field appears more than 20 times during the specified rule interval.
+
+NOTE: Rules that use aggregating queries might create duplicate alerts. This can happen  when events that occur in the additional look-back time are aggregated both in the current rule execution and in a previous rule execution.
+
+[float]
+[[esql-non-agg-query]]
+===== Non-aggregating query 
+Non-aggregating queries doesn't use `STATS...BY` functions and doesn't aggregate source event data. Alerts generated by an {esql} rule with a non-aggregating query only contain the fields returned by the query.
+
+Here is an example non-aggregating query:
+[source,esql]
 -----
-FROM logs*
-| STATS count = COUNT(host.name) BY host.name
-| SORT host.name
+FROM logs-* [metadata _id, _index, _version]
+| WHERE event.category == "process"  AND event.id == "8a4f500d"
+| LIMIT 10
 -----
+- This query starts by querying logs from indices that match the pattern `logs-*`. The `[metadata _id, _index, _version]` operator allows <<esql-non-agg-query-dedupe,alert deduplication>>.  
+- Next, the query filters events where the `event.category` is a process and the `event.id` is `8a4f500d`. 
+- Then, it limits the output to the top 10 results.
 
-.. **Non-aggregating query**: This is a query that _does not_ use the `STATS...BY` grouping commands. Each row in the query results can be tracked to a source document in Elasticsearch. 
-+
-For this type of query, use the operator `[metadata _id, _index, _version]` after defining the index source. This will allow for alerts to be deduplicated and linked to the source documents. For example:
-+
-[esql]
+[float]
+[[esql-non-agg-query-dedupe]]
+===== Turn on alert deduplication for rules using non-aggregating queries
+
+To deduplicate alerts, a query needs access to the `_id`, `_index`, and `_version` metadata fields of the queried source event documents. You can allow this by adding the `[metadata _id, _index, _version]` operator after the `FROM` source command, for example:
+
+[source,esql]
 -----
-FROM logs* [metadata _id, _index, _version]
-| WHERE event.id == "test"
+FROM logs-* [metadata _id, _index, _version]
+| WHERE event.category == "process"  AND event.id == "8a4f500d"
 | LIMIT 10
 -----
-+
-Ensure, metadata properties `id`, `_index`, `_version` are carried over through pipe operators.
 
-. Click *Continue* to <<rule-ui-basic-params, configure basic rule settings>>.
+When those metadata fields are provided, unique alert IDs are created for each alert generated by the query.
+
+When developing the query, make sure you don't {ref}/esql-commands.html#esql-drop[`DROP`] or filter out the `_id`, `_index`, or `_version` metadata fields. 
+
+Here is an example of a query that fails to deduplicate alerts. It uses the `DROP` command to omit the `_id` property from the results table:
+
+[source,esql]
+-----
+FROM logs-* [metadata _id, _index, _version]
+| WHERE event.category == "process"  AND event.id == "8a4f500d"
+| DROP _id
+| LIMIT 10
+-----
+
+Here is another example of an invalid query that uses the `KEEP` command to only return `event.*` fields in the results table: 
+
+[source,esql]
+-----
+FROM logs-* [metadata _id, _index, _version]
+| WHERE event.category == "process"  AND event.id == "8a4f500d"
+| KEEP event.*
+| LIMIT 10
+-----
 
 [float]
 [[rule-ui-basic-params]]