Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Request][ESS] Update ES|QL rule type metadata operator syntax #5167

Merged
merged 5 commits into from
May 14, 2024
Merged

[Request][ESS] Update ES|QL rule type metadata operator syntax #5167

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
036ee54
First draft
nastasha-solomon May 6, 2024
9ca2d58
update metadata for API create
vitaliidm May 7, 2024
5fb8233
Merge branch 'main' into issue-5150-esql-metadata-syntax-fix
nastasha-solomon May 10, 2024
0195f62
Nat's edits
nastasha-solomon May 10, 2024
4e470d8
Merge branch 'main' into issue-5150-esql-metadata-syntax-fix
nastasha-solomon May 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/detections/api/rules/rules-api-create.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1064,7 +1064,7 @@ POST api/detection_engine/rules
{
"type": "esql",
"language": "esql",
"query": "from auditbeat-8.10.2 [metadata _id, _version, _index] | where process.parent.name == \"EXCEL.EXE\"",
"query": "from auditbeat-8.10.2 METADATA _id, _version, _index | where process.parent.name == \"EXCEL.EXE\"",
"name": "Find Excel events",
"description": "Find Excel events",
"tags": [],
Expand Down Expand Up @@ -1525,7 +1525,7 @@ Example response for an {esql} rule:
"setup": "",
"type": "esql",
"language": "esql",
"query": "from auditbeat-8.10.2 [metadata _id] | where process.parent.name == \"EXCEL.EXE\""
"query": "from auditbeat-8.10.2 METADATA _id | where process.parent.name == \"EXCEL.EXE\""
}
--------------------------------------------------
<1> dev:[] These fields are under development and their usage may change: `related_integrations` and `required_fields`.
12 changes: 6 additions & 6 deletions docs/detections/rules-ui-create.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -316,23 +316,23 @@ Non-aggregating queries doesn't use `STATS...BY` functions and doesn't aggregate
Here is an example non-aggregating query:
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved
[source,esql]
-----
FROM logs-* [metadata _id, _index, _version]
FROM logs-* METADATA _id, _index, _version
| WHERE event.category == "process" AND event.id == "8a4f500d"
| LIMIT 10
-----
- This query starts by querying logs from indices that match the pattern `logs-*`. The `[metadata _id, _index, _version]` operator allows <<esql-non-agg-query-dedupe,alert deduplication>>.
- This query starts by querying logs from indices that match the pattern `logs-*`. The `METADATA _id, _index, _version` operator allows <<esql-non-agg-query-dedupe,alert deduplication>>.
- Next, the query filters events where the `event.category` is a process and the `event.id` is `8a4f500d`.
- Then, it limits the output to the top 10 results.

[float]
[[esql-non-agg-query-dedupe]]
===== Turn on alert deduplication for rules using non-aggregating queries

To deduplicate alerts, a query needs access to the `_id`, `_index`, and `_version` metadata fields of the queried source event documents. You can allow this by adding the `[metadata _id, _index, _version]` operator after the `FROM` source command, for example:
To deduplicate alerts, a query needs access to the `_id`, `_index`, and `_version` metadata fields of the queried source event documents. You can allow this by adding the `METADATA _id, _index, _version` operator after the `FROM` source command, for example:

[source,esql]
-----
FROM logs-* [metadata _id, _index, _version]
FROM logs-* METADATA _id, _index, _version
| WHERE event.category == "process" AND event.id == "8a4f500d"
| LIMIT 10
-----
Expand All @@ -345,7 +345,7 @@ Here is an example of a query that fails to deduplicate alerts. It uses the `DRO

[source,esql]
-----
FROM logs-* [metadata _id, _index, _version]
FROM logs-* METADATA _id, _index, _version
| WHERE event.category == "process" AND event.id == "8a4f500d"
| DROP _id
| LIMIT 10
Expand All @@ -355,7 +355,7 @@ Here is another example of an invalid query that uses the `KEEP` command to only

[source,esql]
-----
FROM logs-* [metadata _id, _index, _version]
FROM logs-* METADATA _id, _index, _version
| WHERE event.category == "process" AND event.id == "8a4f500d"
| KEEP event.*
| LIMIT 10
Expand Down