Skip to content

Commit

Permalink
[8.14] [Request][ESS] Update ES|QL rule type metadata operator syntax…
Browse files Browse the repository at this point in the history
… (backport #5167) (#5211)

* First draft

* update metadata for API create

* Nat's edits

---------

Co-authored-by: Vitalii Dmyterko <[email protected]>
(cherry picked from commit ff26101)

Co-authored-by: Nastasha Solomon <[email protected]>
  • Loading branch information
mergify[bot] and nastasha-solomon authored May 14, 2024
1 parent bc526ca commit 5f24c10
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 11 deletions.
4 changes: 2 additions & 2 deletions docs/detections/api/rules/rules-api-create.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -1064,7 +1064,7 @@ POST api/detection_engine/rules
{
"type": "esql",
"language": "esql",
"query": "from auditbeat-8.10.2 [metadata _id, _version, _index] | where process.parent.name == \"EXCEL.EXE\"",
"query": "from auditbeat-8.10.2 METADATA _id, _version, _index | where process.parent.name == \"EXCEL.EXE\"",
"name": "Find Excel events",
"description": "Find Excel events",
"tags": [],
Expand Down Expand Up @@ -1525,7 +1525,7 @@ Example response for an {esql} rule:
"setup": "",
"type": "esql",
"language": "esql",
"query": "from auditbeat-8.10.2 [metadata _id] | where process.parent.name == \"EXCEL.EXE\""
"query": "from auditbeat-8.10.2 METADATA _id | where process.parent.name == \"EXCEL.EXE\""
}
--------------------------------------------------
<1> dev:[] These fields are under development and their usage may change: `related_integrations` and `required_fields`.
18 changes: 9 additions & 9 deletions docs/detections/rules-ui-create.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -311,28 +311,28 @@ NOTE: Rules that use aggregating queries might create duplicate alerts. This can
[float]
[[esql-non-agg-query]]
===== Non-aggregating query
Non-aggregating queries doesn't use `STATS...BY` functions and doesn't aggregate source event data. Alerts generated by an {esql} rule with a non-aggregating query only contain the fields returned by the query.
Non-aggregating queries don't use `STATS...BY` functions and don't aggregate source event data. Alerts generated by an {esql} rule with a non-aggregating query only contain the fields returned by the query.

Here is an example non-aggregating query:
[source,esql]
-----
FROM logs-* [metadata _id, _index, _version]
FROM logs-* METADATA _id, _index, _version
| WHERE event.category == "process" AND event.id == "8a4f500d"
| LIMIT 10
-----
- This query starts by querying logs from indices that match the pattern `logs-*`. The `[metadata _id, _index, _version]` operator allows <<esql-non-agg-query-dedupe,alert deduplication>>.
- This query starts by querying logs from indices that match the pattern `logs-*`. The `METADATA _id, _index, _version` operator allows <<esql-non-agg-query-dedupe,alert deduplication>>.
- Next, the query filters events where the `event.category` is a process and the `event.id` is `8a4f500d`.
- Then, it limits the output to the top 10 results.

[float]
[[esql-non-agg-query-dedupe]]
===== Turn on alert deduplication for rules using non-aggregating queries

To deduplicate alerts, a query needs access to the `_id`, `_index`, and `_version` metadata fields of the queried source event documents. You can allow this by adding the `[metadata _id, _index, _version]` operator after the `FROM` source command, for example:
To deduplicate alerts, a query needs access to the `_id`, `_index`, and `_version` metadata fields of the queried source event documents. You can allow this by adding the `METADATA _id, _index, _version` operator after the `FROM` source command, for example:

[source,esql]
-----
FROM logs-* [metadata _id, _index, _version]
FROM logs-* METADATA _id, _index, _version
| WHERE event.category == "process" AND event.id == "8a4f500d"
| LIMIT 10
-----
Expand All @@ -345,7 +345,7 @@ Here is an example of a query that fails to deduplicate alerts. It uses the `DRO

[source,esql]
-----
FROM logs-* [metadata _id, _index, _version]
FROM logs-* METADATA _id, _index, _version
| WHERE event.category == "process" AND event.id == "8a4f500d"
| DROP _id
| LIMIT 10
Expand All @@ -355,7 +355,7 @@ Here is another example of an invalid query that uses the `KEEP` command to only

[source,esql]
-----
FROM logs-* [metadata _id, _index, _version]
FROM logs-* METADATA _id, _index, _version
| WHERE event.category == "process" AND event.id == "8a4f500d"
| KEEP event.*
| LIMIT 10
Expand All @@ -367,7 +367,7 @@ FROM logs-* [metadata _id, _index, _version]

When writing your query, consider the following:

- The {ref}/esql-commands.html#esql-limit[`LIMIT`] command specifies the number of rows an {esql} query returns and the number of alerts created per rule execution. Similarly, a detection rule's <<opt-fields-all,`max_signals`>> setting specifies the maximum number of alerts it can create every time it runs.
- The {ref}/esql-commands.html#esql-limit[`LIMIT`] command specifies the maximum number of rows an {esql} query returns and the maximum number of alerts created per rule execution. Similarly, a detection rule's <<opt-fields-all,`max_signals`>> setting specifies the maximum number of alerts it can create every time it runs.
+
If the `LIMIT` value is lower than the `max_signals` value, the rule uses the `LIMIT` value to determine the maximum number of alerts the rule generates. If the `LIMIT` value is higher than the `max_signals` value, the rule uses the `max_signals` value.
+
Expand All @@ -382,7 +382,7 @@ NOTE: The `max_signals` default value is 100. You can modify it using the <<rule

The {esql} rule has the following limitations:

- If your {esql} query creates new fields that aren’t part of the ECS schema, they won’t be mapped to the alerts index and you can't search or filter for them from the Alerts table. As a workaround, create <<runtime-fields,runtime fields>>.
- If your {esql} query creates new fields that aren’t part of the ECS schema, they won’t be mapped to the alerts index, and you can't search or filter for them from the Alerts table. As a workaround, create <<runtime-fields,runtime fields>>.
- If your {esql} query creates new fields that aren’t in the query’s source index, they can’t be added to the rule’s <<rule-ui-advanced-params,custom highlighted fields>>.

[float]
Expand Down

0 comments on commit 5f24c10

Please sign in to comment.