Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Request][ESS] Update ES|QL rule type metadata operator syntax #5167

Merged
merged 5 commits into from
May 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/detections/api/rules/rules-api-create.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -1064,7 +1064,7 @@ POST api/detection_engine/rules
{
"type": "esql",
"language": "esql",
"query": "from auditbeat-8.10.2 [metadata _id, _version, _index] | where process.parent.name == \"EXCEL.EXE\"",
"query": "from auditbeat-8.10.2 METADATA _id, _version, _index | where process.parent.name == \"EXCEL.EXE\"",
"name": "Find Excel events",
"description": "Find Excel events",
"tags": [],
Expand Down Expand Up @@ -1525,7 +1525,7 @@ Example response for an {esql} rule:
"setup": "",
"type": "esql",
"language": "esql",
"query": "from auditbeat-8.10.2 [metadata _id] | where process.parent.name == \"EXCEL.EXE\""
"query": "from auditbeat-8.10.2 METADATA _id | where process.parent.name == \"EXCEL.EXE\""
}
--------------------------------------------------
<1> dev:[] These fields are under development and their usage may change: `related_integrations` and `required_fields`.
18 changes: 9 additions & 9 deletions docs/detections/rules-ui-create.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -311,28 +311,28 @@ NOTE: Rules that use aggregating queries might create duplicate alerts. This can
[float]
[[esql-non-agg-query]]
===== Non-aggregating query
Non-aggregating queries doesn't use `STATS...BY` functions and doesn't aggregate source event data. Alerts generated by an {esql} rule with a non-aggregating query only contain the fields returned by the query.
Non-aggregating queries don't use `STATS...BY` functions and don't aggregate source event data. Alerts generated by an {esql} rule with a non-aggregating query only contain the fields returned by the query.

Here is an example non-aggregating query:
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved
[source,esql]
-----
FROM logs-* [metadata _id, _index, _version]
FROM logs-* METADATA _id, _index, _version
| WHERE event.category == "process" AND event.id == "8a4f500d"
| LIMIT 10
-----
- This query starts by querying logs from indices that match the pattern `logs-*`. The `[metadata _id, _index, _version]` operator allows <<esql-non-agg-query-dedupe,alert deduplication>>.
- This query starts by querying logs from indices that match the pattern `logs-*`. The `METADATA _id, _index, _version` operator allows <<esql-non-agg-query-dedupe,alert deduplication>>.
- Next, the query filters events where the `event.category` is a process and the `event.id` is `8a4f500d`.
- Then, it limits the output to the top 10 results.

[float]
[[esql-non-agg-query-dedupe]]
===== Turn on alert deduplication for rules using non-aggregating queries

To deduplicate alerts, a query needs access to the `_id`, `_index`, and `_version` metadata fields of the queried source event documents. You can allow this by adding the `[metadata _id, _index, _version]` operator after the `FROM` source command, for example:
To deduplicate alerts, a query needs access to the `_id`, `_index`, and `_version` metadata fields of the queried source event documents. You can allow this by adding the `METADATA _id, _index, _version` operator after the `FROM` source command, for example:

[source,esql]
-----
FROM logs-* [metadata _id, _index, _version]
FROM logs-* METADATA _id, _index, _version
| WHERE event.category == "process" AND event.id == "8a4f500d"
| LIMIT 10
-----
Expand All @@ -345,7 +345,7 @@ Here is an example of a query that fails to deduplicate alerts. It uses the `DRO

[source,esql]
-----
FROM logs-* [metadata _id, _index, _version]
FROM logs-* METADATA _id, _index, _version
| WHERE event.category == "process" AND event.id == "8a4f500d"
| DROP _id
| LIMIT 10
Expand All @@ -355,7 +355,7 @@ Here is another example of an invalid query that uses the `KEEP` command to only

[source,esql]
-----
FROM logs-* [metadata _id, _index, _version]
FROM logs-* METADATA _id, _index, _version
| WHERE event.category == "process" AND event.id == "8a4f500d"
| KEEP event.*
| LIMIT 10
Expand All @@ -367,7 +367,7 @@ FROM logs-* [metadata _id, _index, _version]

When writing your query, consider the following:

- The {ref}/esql-commands.html#esql-limit[`LIMIT`] command specifies the number of rows an {esql} query returns and the number of alerts created per rule execution. Similarly, a detection rule's <<opt-fields-all,`max_signals`>> setting specifies the maximum number of alerts it can create every time it runs.
- The {ref}/esql-commands.html#esql-limit[`LIMIT`] command specifies the maximum number of rows an {esql} query returns and the maximum number of alerts created per rule execution. Similarly, a detection rule's <<opt-fields-all,`max_signals`>> setting specifies the maximum number of alerts it can create every time it runs.
+
If the `LIMIT` value is lower than the `max_signals` value, the rule uses the `LIMIT` value to determine the maximum number of alerts the rule generates. If the `LIMIT` value is higher than the `max_signals` value, the rule uses the `max_signals` value.
+
Expand All @@ -382,7 +382,7 @@ NOTE: The `max_signals` default value is 100. You can modify it using the <<rule

The {esql} rule has the following limitations:

- If your {esql} query creates new fields that aren’t part of the ECS schema, they won’t be mapped to the alerts index and you can't search or filter for them from the Alerts table. As a workaround, create <<runtime-fields,runtime fields>>.
- If your {esql} query creates new fields that aren’t part of the ECS schema, they won’t be mapped to the alerts index, and you can't search or filter for them from the Alerts table. As a workaround, create <<runtime-fields,runtime fields>>.
- If your {esql} query creates new fields that aren’t in the query’s source index, they can’t be added to the rule’s <<rule-ui-advanced-params,custom highlighted fields>>.

[float]
Expand Down