Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.11] [Detection Engine] update API docs for new ES|QL rule (backport #4052) #4106

Merged
merged 1 commit into from
Oct 26, 2023
Merged

[8.11] [Detection Engine] update API docs for new ES|QL rule (backport #4052) #4106

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
141 changes: 112 additions & 29 deletions docs/detections/api/rules/rules-api-create.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,12 @@ specified {ref}/indices-create-index.html[{es} index]. For example, you can
create an index for IP addresses and use this index to create an alert whenever
an event's `destination.ip` equals a value in the index. The index's field
mappings should be {ecs-ref}[ECS-compliant].
* *New terms*: Generates an alert for each new term detected in source documents within a specified time range.
* *ES|QL*: Uses {ref}/esql.html[Elasticsearch Query Language (ES|QL)] to find events and aggregate search results.
+
preview::[]
* *{ml-cap} rules*: Creates an alert when a {ml} job discovers an anomaly above
the defined threshold (see <<machine-learning>>).
* *New terms*: Generates an alert for each new term detected in source documents within a specified time range.

IMPORTANT: To create {ml} rules, you must have the
https://www.elastic.co/subscriptions[appropriate license] or use a
Expand Down Expand Up @@ -102,8 +105,8 @@ A JSON object that defines the rule's values:
* <<req-fields-ml>>
* <<req-fields-threat-match>>
* <<req-fields-new-terms>>
* <<req-fields-esql>>
* <<opt-fields-all>>
* <<opt-fields-query-eql>>
* <<opt-fields-threat-match>>
* <<opt-fields-query-threshold>>
* <<opt-fields-eql-query-threshold>>
Expand Down Expand Up @@ -140,6 +143,7 @@ occurred
|type |String a|Data type on which the rule is based:

* `eql`: EQL query (see {ref}/eql.html[Event Query Language]).
* `esql`: ES\|QL query (see {ref}/esql.html[Elasticsearch Query Language]).
* `query`: query with or without additional filters.
* `saved_query`: saved search, identified in the `saved_id` field.
* `machine_learning`: rule based on a {ml} job's anomaly scores.
Expand All @@ -152,15 +156,21 @@ specified field.
|==============================================

[[req-fields-query-threshold]]
===== Required field for query, indicator match, threshold, and new terms rules
===== Required fields for query, indicator match, threshold, new terms, event correlation, and ES|QL rules

[width="100%",options="header"]
|==============================================
|Name |Type |Description

|query |String a|{kibana-ref}/search.html[Query] used by the rule to create
alerts. For indicator match rules, only the query's results are used to determine
whether an alert is generated.
alerts.

[NOTE]
======
* For indicator match rules, only the query's results are used to determine
whether an alert is generated.
* ES\|QL rules have additional query requirements. Refer to <<create-esql-rule,Create ES\|QL rules>> for more information.
======

|==============================================

Expand Down Expand Up @@ -209,6 +219,17 @@ generated.

|==============================================

[[req-fields-esql]]
===== Required field for ES|QL rules

[width="100%",options="header"]
|==============================================
|Name |Type |Description

|language |String |Must be `esql`.

|==============================================

[[req-fields-ml]]
===== Required fields for machine learning rules

Expand Down Expand Up @@ -354,27 +375,6 @@ Required when `actions` are used to send notifications.

|==============================================

[[opt-fields-query-eql]]
===== Optional fields for query, indicator match, event correlation, and new terms rules

[width="100%",options="header"]
|==============================================
|Name |Type |Description

|exceptions_list |Object[] a|Array of
<<exceptions-api-overview, exception containers>>, which define
exceptions that prevent the rule from generating alerts even when its other
criteria are met. The object has these fields:

* `id` (string, required): ID of the exception container.
* `list_id` (string, required): List ID of the exception container.
* `namespace_type` (string required): Determines whether the exceptions are
valid in only the rule's {kib} space (`single`) or in all {kib} spaces
(`agnostic`).
* `type` (string, required): The exception type, which must be either
a detection rule exception (`detection`) or an endpoint exception (`endpoint`).
|==============================================

[[opt-fields-threat-match]]
===== Optional fields for indicator match rules

Expand Down Expand Up @@ -402,21 +402,25 @@ documents from the {es} index containing the threat values.
|==============================================

[[opt-fields-eql-query-threshold]]
===== Optional fields for event correlation, query, threshold, and new terms rules
===== Optional fields for event correlation, query, threshold, indicator match, new terms, and ES|QL rules

[width="100%",options="header"]
|==============================================
|Name |Type |Description

|filters |Object[] |The {ref}/query-filter-context.html[query and filter
|filters |Object[] a|The {ref}/query-filter-context.html[query and filter
context] array used to define the conditions for when alerts are created from
events. Defaults to an empty array.

|index |String[] |Indices on which the rule functions. Defaults to the
NOTE: This field is not supported for ES\|QL rules.

|index |String[] a|Indices on which the rule functions. Defaults to the
Security Solution indices defined on the {kib} Advanced Settings page
(*Kibana* → *Stack Management* → *Advanced Settings* →
`securitySolution:defaultIndex`).

NOTE: This field is not supported for ES\|QL rules.

|risk_score_mapping |Object[] a|Overrides generated alerts' `risk_score` with
a value from the source event:

Expand Down Expand Up @@ -444,6 +448,19 @@ values from the source event:
When unspecified, rules query the `@timestamp` field. The source field
must be an {es} date data type.

|exceptions_list |Object[] a|Array of
<<exceptions-api-overview, exception containers>>, which define
exceptions that prevent the rule from generating alerts even when its other
criteria are met. The object has these fields:

* `id` (string, required): ID of the exception container.
* `list_id` (string, required): List ID of the exception container.
* `namespace_type` (string required): Determines whether the exceptions are
valid in only the rule's {kib} space (`single`) or in all {kib} spaces
(`agnostic`).
* `type` (string, required): The exception type, which must be either
a detection rule exception (`detection`) or an endpoint exception (`endpoint`).

|==============================================

[[opt-fields-eql-create]]
Expand Down Expand Up @@ -969,6 +986,29 @@ POST api/detection_engine/rules
`200`::
Indicates a successful call.

*Example 7*

ES|QL rule that creates alerts from events that match an Excel parent process:

[source,json]
--------------------------------------------------
POST api/detection_engine/rules
{
"type": "esql",
"language": "esql",
"query": "from auditbeat-8.10.2 [metadata _id, _version, _index] | where process.parent.name == \"EXCEL.EXE\"",
"name": "Find Excel events",
"description": "Find Excel events",
"tags": [],
"interval": "5m",
"from": "now-360s",
"to": "now",
"enabled": false,
"risk_score": 21,
"severity": "low"
}
--------------------------------------------------

==== Response payload

A JSON object that includes a unique ID, the time the rule was created, and its
Expand Down Expand Up @@ -1327,3 +1367,46 @@ Example response for a new terms rule:
}
--------------------------------------------------
<1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`.

Example response for an ES|QL rule:

[source,json]
--------------------------------------------------
{
"name": "Find Excel events",
"description": "Find Excel events",
"risk_score": 21,
"severity": "low",
"output_index": "",
"tags": [],
"interval": "5m",
"enabled": false,
"author": [],
"false_positives": [],
"from": "now-360s",
"max_signals": 100,
"risk_score_mapping": [],
"severity_mapping": [],
"threat": [],
"to": "now",
"references": [],
"version": 1,
"exceptions_list": [],
"actions": [],
"id": "d0f20490-6da4-11ee-b85e-09e9b661f2e2",
"updated_at": "2023-10-18T10:55:14.269Z",
"updated_by": "elastic",
"created_at": "2023-10-18T10:55:14.269Z",
"created_by": "elastic",
"revision": 0,
"rule_id": "e4b53a89-debd-4a0d-a3e3-20606952e589",
"immutable": false,
"related_integrations": [], <1>
"required_fields": [], <1>
"setup": "", <1>
"type": "esql",
"language": "esql",
"query": "from auditbeat-8.10.2 [metadata _id] | where process.parent.name == \"EXCEL.EXE\""
}
--------------------------------------------------
<1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`.
54 changes: 31 additions & 23 deletions docs/detections/api/rules/rules-api-update.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ occurred
|type |String a|Data type on which the rule is based:

* `eql`: EQL query (see {ref}/eql.html[Event Query Language]).
* `esql`: ES\|QL query (refer to {ref}/esql.html[Elasticsearch Query Language]).
* `query`: query with or without additional filters.
* `saved_query`: saved search, identified in the `saved_id` field.
* `machine_learning`: rule based on a {ml} job's anomaly scores.
Expand Down Expand Up @@ -125,6 +126,16 @@ generated.

|==============================================

===== Field required for ES|QL rules `PUT` calls

[width="100%",options="header"]
|==============================================
|Name |Type |Description

|language |String |Must be `esql`.

|==============================================

===== Fields required for machine-learning rules `PUT` calls

[width="100%",options="header"]
Expand Down Expand Up @@ -260,26 +271,6 @@ number.

|==============================================

===== Optional fields for query, threat-match, EQL, and new terms rules

[width="100%",options="header"]
|==============================================
|Name |Type |Description

|exceptions_list |Object[] a|Array of
<<exceptions-api-overview, exception containers>>, which define
exceptions that prevent the rule from generating alerts even when its other
criteria are met. The object has these fields:

* `id` (string, required): ID of the exception container.
* `list_id` (string, required): List ID of the exception container.
* `namespace_type` (string required): Determines whether the exceptions are
valid in only the rule's {kib} space (`single`) or in all {kib} spaces
(`agnostic`).
* `type` (string, required): The exception type, which must be either
a detection rule exception (`detection`) or an endpoint exception (`endpoint`).
|==============================================

===== Optional fields for threat-match rules

[width="100%",options="header"]
Expand All @@ -304,21 +295,25 @@ documents from the {es} index containing the threat values.
`kuery` or `lucene`. Defaults to `kuery`.
|==============================================

===== Optional fields for EQL, query, threshold, and new terms rules
===== Optional fields for EQL, query, threshold, indicator match, new terms rules, and ES|QL rules

[width="100%",options="header"]
|==============================================
|Name |Type |Description

|filters |Object[] |The {ref}/query-filter-context.html[query and filter
|filters |Object[] a|The {ref}/query-filter-context.html[query and filter
context] array used to define the conditions for when alerts are created from
events. Defaults to an empty array.

|index |String[] |Indices on which the rule functions. Defaults to the
NOTE: This field is not supported for ES\|QL rules.

|index |String[] a|Indices on which the rule functions. Defaults to the
Security Solution indices defined on the {kib} Advanced Settings page
(*Kibana* → *Stack Management* → *Advanced Settings* →
`securitySolution:defaultIndex`).

NOTE: This field is not supported for ES\|QL rules.

|risk_score_mapping |Object[] a|Overrides generated alerts' `risk_score` with
a value from the source event:

Expand Down Expand Up @@ -346,6 +341,19 @@ values from the source event:
When unspecified, rules query the `@timestamp` field. The source field
must be an {es} date data type.

|exceptions_list |Object[] a|Array of
<<exceptions-api-overview, exception containers>>, which define
exceptions that prevent the rule from generating alerts even when its other
criteria are met. The object has these fields:

* `id` (string, required): ID of the exception container.
* `list_id` (string, required): List ID of the exception container.
* `namespace_type` (string required): Determines whether the exceptions are
valid in only the rule's {kib} space (`single`) or in all {kib} spaces
(`agnostic`).
* `type` (string, required): The exception type, which must be either
a detection rule exception (`detection`) or an endpoint exception (`endpoint`).

|==============================================

[[opt-fields-eql-update]]
Expand Down