diff --git a/docs/detections/about-rules.asciidoc b/docs/detections/about-rules.asciidoc index d02ef77d7c..96e131c745 100644 --- a/docs/detections/about-rules.asciidoc +++ b/docs/detections/about-rules.asciidoc @@ -42,6 +42,10 @@ TIP: You can also use value lists as the indicator match index. See <>: Generates an alert for each new term detected in source documents within a specified time range. You can also detect a combination of up to three new terms (for example, a `host.ip` and `host.id` that have never been observed together before). +* <>: Searches the defined indices and creates an alert when results match an {ref}/esql.html[Elasticsearch Query Language (ES|QL)] query. ++ +preview::[] + [role="screenshot"] image::images/all-rules.png[Shows the Rules page] diff --git a/docs/detections/api/rules/rules-api-create.asciidoc b/docs/detections/api/rules/rules-api-create.asciidoc index 3603168eb6..3a58fcf0ee 100644 --- a/docs/detections/api/rules/rules-api-create.asciidoc +++ b/docs/detections/api/rules/rules-api-create.asciidoc @@ -37,7 +37,7 @@ create an index for IP addresses and use this index to create an alert whenever an event's `destination.ip` equals a value in the index. The index's field mappings should be {ecs-ref}[ECS-compliant]. * *New terms*: Generates an alert for each new term detected in source documents within a specified time range. -* *ES|QL*: Uses {ref}/esql.html[Elasticsearch Query Language (ES|QL)] to find events and aggregate search results. +* *{esql}*: Uses {ref}/esql.html[Elasticsearch Query Language ({esql})] to find events and aggregate search results. + preview::[] * *{ml-cap} rules*: Creates an alert when a {ml} job discovers an anomaly above @@ -161,7 +161,7 @@ specified field. |============================================== [[req-fields-query-threshold]] -===== Required fields for query, indicator match, threshold, new terms, event correlation, and ES|QL rules +===== Required fields for query, indicator match, threshold, new terms, event correlation, and {esql} rules [width="100%",options="header"] |============================================== @@ -225,7 +225,7 @@ generated. |============================================== [[req-fields-esql]] -===== Required field for ES|QL rules +===== Required field for {esql} rules [width="100%",options="header"] |============================================== @@ -407,7 +407,7 @@ documents from the {es} index containing the threat values. |============================================== [[opt-fields-eql-query-threshold]] -===== Optional fields for event correlation, query, threshold, indicator match, new terms, and ES|QL rules +===== Optional fields for event correlation, query, threshold, indicator match, new terms, and {esql} rules [width="100%",options="header"] |============================================== @@ -993,7 +993,7 @@ POST api/detection_engine/rules *Example 7* -ES|QL rule that creates alerts from events that match an Excel parent process: +{esql} rule that creates alerts from events that match an Excel parent process: [source,json] -------------------------------------------------- @@ -1373,7 +1373,7 @@ Example response for a new terms rule: -------------------------------------------------- <1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`. -Example response for an ES|QL rule: +Example response for an {esql} rule: [source,json] -------------------------------------------------- diff --git a/docs/detections/api/rules/rules-api-update.asciidoc b/docs/detections/api/rules/rules-api-update.asciidoc index b6b9129cc2..eb0b156e9c 100644 --- a/docs/detections/api/rules/rules-api-update.asciidoc +++ b/docs/detections/api/rules/rules-api-update.asciidoc @@ -131,7 +131,7 @@ generated. |============================================== -===== Field required for ES|QL rules `PUT` calls +===== Field required for {esql} rules `PUT` calls [width="100%",options="header"] |============================================== @@ -300,7 +300,7 @@ documents from the {es} index containing the threat values. `kuery` or `lucene`. Defaults to `kuery`. |============================================== -===== Optional fields for EQL, query, threshold, indicator match, new terms rules, and ES|QL rules +===== Optional fields for EQL, query, threshold, indicator match, new terms rules, and {esql} rules [width="100%",options="header"] |============================================== diff --git a/docs/detections/images/esql-ref-button.png b/docs/detections/images/esql-ref-button.png new file mode 100644 index 0000000000..0907eb6441 Binary files /dev/null and b/docs/detections/images/esql-ref-button.png differ diff --git a/docs/detections/rules-ui-create.asciidoc b/docs/detections/rules-ui-create.asciidoc index 680b8221c2..eb37316c4e 100644 --- a/docs/detections/rules-ui-create.asciidoc +++ b/docs/detections/rules-ui-create.asciidoc @@ -261,37 +261,109 @@ For example, if a rule has an interval of 5 minutes, no additional look-back tim [discrete] [[create-esql-rule]] -=== Create an ES|QL rule +=== Create an {esql} rule -IMPORTANT: This is a placeholder for future documentation. The following content is incomplete. +preview::[] -. Go to *Rules* -> *Detection rules (SIEM)* -> *Create new rule*. The *Create new rule* page displays. -. To create a rule that uses ES|QL, select **ES|QL**, -then write a query. There are two types of ES|QL queries: +Use {ref}/esql.html[{esql}] to query your source events and aggregate event data. Query results are returned in a table with rows and columns. Each row becomes an alert. + +To create an {esql} rule: + +. Go to *Rules* -> *Detection rules (SIEM)* -> *Create new rule*. The *Create new rule* page appears. +. Select **{esql}**, then write a <>. ++ +TIP: Click the help icon (image:images/esql-ref-button.png[Click the ES|QL help icon,20,20]) to open the in-product reference documentation for all {esql} commands and functions. ++ +[NOTE] +==== +To avoid conflicts between the {ref}/esql-commands.html#esql-limit[`LIMIT`] value (which defines how many rows are returned in the results table) and the <> value (which determines the maximum number of alerts a rule can generate per rule execution), make sure the `max_signals` value does not exceed the `LIMIT` value. -.. **Aggregating query**: This is a query that uses the `STATS...BY` grouping commands. Query results cannot be matched with a particular document in Elasticsearch. For example: +If you do not define a `LIMIT` value, the rule automatically uses the `max_signals` value. +==== + -[esql] +. Click *Continue* to <>. + +[float] +[[esql-rule-query-types]] +==== {esql} query types + +{esql} rule queries are loosely categorized into two types: aggregating and non-aggregating. + +[float] +[[esql-agg-query]] +===== Aggregating query + +Aggregating queries use {ref}/esql-functions-operators.html#esql-agg-functions[`STATS...BY`] functions to aggregate source event data. Alerts generated by an {esql} rule with an aggregating query only contain the fields returned by the query. + +Here is an example aggregating query: + +[source,esql] +---- +FROM logs-* +| STATS host_count = COUNT(host.name) BY host.name +| SORT host_count DESC +| WHERE host_count > 20 +---- + +- This query starts by searching logs from indices that match the pattern `logs-*`. +- The query then aggregates the count of events by `host.name`. +- Next, it sorts the result by `host_count` in descending order. +- Then, it filters for events where the `host_count` field appears more than 20 times during the specified rule interval. + +NOTE: Rules that use aggregating queries might create duplicate alerts. This can happen when events that occur in the additional look-back time are aggregated both in the current rule execution and in a previous rule execution. + +[float] +[[esql-non-agg-query]] +===== Non-aggregating query +Non-aggregating queries doesn't use `STATS...BY` functions and doesn't aggregate source event data. Alerts generated by an {esql} rule with a non-aggregating query only contain the fields returned by the query. + +Here is an example non-aggregating query: +[source,esql] ----- -FROM logs* -| STATS count = COUNT(host.name) BY host.name -| SORT host.name +FROM logs-* [metadata _id, _index, _version] +| WHERE event.category == "process" AND event.id == "8a4f500d" +| LIMIT 10 ----- +- This query starts by querying logs from indices that match the pattern `logs-*`. The `[metadata _id, _index, _version]` operator allows <>. +- Next, the query filters events where the `event.category` is a process and the `event.id` is `8a4f500d`. +- Then, it limits the output to the top 10 results. -.. **Non-aggregating query**: This is a query that _does not_ use the `STATS...BY` grouping commands. Each row in the query results can be tracked to a source document in Elasticsearch. -+ -For this type of query, use the operator `[metadata _id, _index, _version]` after defining the index source. This will allow for alerts to be deduplicated and linked to the source documents. For example: -+ -[esql] +[float] +[[esql-non-agg-query-dedupe]] +===== Turn on alert deduplication for rules using non-aggregating queries + +To deduplicate alerts, a query needs access to the `_id`, `_index`, and `_version` metadata fields of the queried source event documents. You can allow this by adding the `[metadata _id, _index, _version]` operator after the `FROM` source command, for example: + +[source,esql] ----- -FROM logs* [metadata _id, _index, _version] -| WHERE event.id == "test" +FROM logs-* [metadata _id, _index, _version] +| WHERE event.category == "process" AND event.id == "8a4f500d" | LIMIT 10 ----- -+ -Ensure, metadata properties `id`, `_index`, `_version` are carried over through pipe operators. -. Click *Continue* to <>. +When those metadata fields are provided, unique alert IDs are created for each alert generated by the query. + +When developing the query, make sure you don't {ref}/esql-commands.html#esql-drop[`DROP`] or filter out the `_id`, `_index`, or `_version` metadata fields. + +Here is an example of a query that fails to deduplicate alerts. It uses the `DROP` command to omit the `_id` property from the results table: + +[source,esql] +----- +FROM logs-* [metadata _id, _index, _version] +| WHERE event.category == "process" AND event.id == "8a4f500d" +| DROP _id +| LIMIT 10 +----- + +Here is another example of an invalid query that uses the `KEEP` command to only return `event.*` fields in the results table: + +[source,esql] +----- +FROM logs-* [metadata _id, _index, _version] +| WHERE event.category == "process" AND event.id == "8a4f500d" +| KEEP event.* +| LIMIT 10 +----- [float] [[rule-ui-basic-params]]