[Detection Engine] update API docs for new ES|QL rule (#4052)

* initial commit * updates doc * update API * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: natasha-moore-elastic <[email protected]> * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: natasha-moore-elastic <[email protected]> * Update docs/detections/api/rules/rules-api-update.asciidoc Co-authored-by: natasha-moore-elastic <[email protected]> * CR feedback * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: natasha-moore-elastic <[email protected]> * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: natasha-moore-elastic <[email protected]> * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: Nastasha Solomon <[email protected]> * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: Nastasha Solomon <[email protected]> * CR * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: Nastasha Solomon <[email protected]> * Update docs/detections/api/rules/rules-api-update.asciidoc Co-authored-by: Nastasha Solomon <[email protected]> * Combining tables and adding note to update rule * Update docs/detections/api/rules/rules-api-create.asciidoc * Removing extra term * Combining tables in create rule * Fixing tag rendering * Adding ref to UI docs * littile cleanup and fixes * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: natasha-moore-elastic <[email protected]> * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: natasha-moore-elastic <[email protected]> * Update docs/detections/api/rules/rules-api-create.asciidoc Co-authored-by: Nastasha Solomon <[email protected]> --------- Co-authored-by: natasha-moore-elastic <[email protected]> Co-authored-by: Nastasha Solomon <[email protected]> Co-authored-by: nastasha.solomon <[email protected]> (cherry picked from commit 5f33c46)
elastic · Oct 26, 2023 · 180eab8 · 180eab8
1 parent e303797
commit 180eab8
Show file tree

Hide file tree

Showing 2 changed files with 143 additions and 52 deletions.
diff --git a/docs/detections/api/rules/rules-api-create.asciidoc b/docs/detections/api/rules/rules-api-create.asciidoc
@@ -31,9 +31,12 @@ specified {ref}/indices-create-index.html[{es} index]. For example, you can
 create an index for IP addresses and use this index to create an alert whenever
 an event's `destination.ip` equals a value in the index. The index's field
 mappings should be {ecs-ref}[ECS-compliant].
+* *New terms*: Generates an alert for each new term detected in source documents within a specified time range.
+* *ES|QL*: Uses {ref}/esql.html[Elasticsearch Query Language (ES|QL)] to find events and aggregate search results.
++
+preview::[]
 * *{ml-cap} rules*: Creates an alert when a {ml} job discovers an anomaly above
 the defined threshold (see <<machine-learning>>).
-* *New terms*: Generates an alert for each new term detected in source documents within a specified time range.
 
 IMPORTANT: To create {ml} rules, you must have the
 https://www.elastic.co/subscriptions[appropriate license] or use a
@@ -102,8 +105,8 @@ A JSON object that defines the rule's values:
 * <<req-fields-ml>>
 * <<req-fields-threat-match>>
 * <<req-fields-new-terms>>
+* <<req-fields-esql>>
 * <<opt-fields-all>>
-* <<opt-fields-query-eql>>
 * <<opt-fields-threat-match>>
 * <<opt-fields-query-threshold>>
 * <<opt-fields-eql-query-threshold>>
@@ -140,6 +143,7 @@ occurred
 |type |String a|Data type on which the rule is based:
 
 * `eql`: EQL query (see {ref}/eql.html[Event Query Language]).
+* `esql`: ES\|QL query (see {ref}/esql.html[Elasticsearch Query Language]).
 * `query`: query with or without additional filters.
 * `saved_query`: saved search, identified in the `saved_id` field.
 * `machine_learning`: rule based on a {ml} job's anomaly scores.
@@ -152,15 +156,21 @@ specified field.
 |==============================================
 
 [[req-fields-query-threshold]]
-===== Required field for query, indicator match, threshold, and new terms rules
+===== Required fields for query, indicator match, threshold, new terms, event correlation, and ES|QL rules
 
 [width="100%",options="header"]
 |==============================================
 |Name |Type |Description
 
 |query |String a|{kibana-ref}/search.html[Query] used by the rule to create
-alerts. For indicator match rules, only the query's results are used to determine
-whether an alert is generated.
+alerts. 
+
+[NOTE]
+======
+* For indicator match rules, only the query's results are used to determine
+whether an alert is generated. 
+* ES\|QL rules have additional query requirements. Refer to <<create-esql-rule,Create ES\|QL rules>> for more information. 
+======
 
 |==============================================
 
@@ -209,6 +219,17 @@ generated.
 
 |==============================================
 
+[[req-fields-esql]]
+===== Required field for ES|QL rules
+
+[width="100%",options="header"]
+|==============================================
+|Name |Type |Description
+
+|language |String |Must be `esql`.
+
+|==============================================
+
 [[req-fields-ml]]
 ===== Required fields for machine learning rules
 
@@ -354,27 +375,6 @@ Required when `actions` are used to send notifications.
 
 |==============================================
 
-[[opt-fields-query-eql]]
-===== Optional fields for query, indicator match, event correlation, and new terms rules
-
-[width="100%",options="header"]
-|==============================================
-|Name |Type |Description
-
-|exceptions_list |Object[] a|Array of
-<<exceptions-api-overview, exception containers>>, which define
-exceptions that prevent the rule from generating alerts even when its other
-criteria are met. The object has these fields:
-
-* `id` (string, required): ID of the exception container.
-* `list_id` (string, required): List ID of the exception container.
-* `namespace_type` (string required): Determines whether the exceptions are
-valid in only the rule's {kib} space (`single`) or in all {kib} spaces
-(`agnostic`).
-* `type` (string, required): The exception type, which must be either
-a detection rule exception (`detection`) or an endpoint exception (`endpoint`).
-|==============================================
-
 [[opt-fields-threat-match]]
 ===== Optional fields for indicator match rules
 
@@ -402,21 +402,25 @@ documents from the {es} index containing the threat values.
 |==============================================
 
 [[opt-fields-eql-query-threshold]]
-===== Optional fields for event correlation, query, threshold, and new terms rules
+===== Optional fields for event correlation, query, threshold, indicator match, new terms, and ES|QL rules
 
 [width="100%",options="header"]
 |==============================================
 |Name |Type |Description
 
-|filters |Object[] |The {ref}/query-filter-context.html[query and filter
+|filters |Object[] a|The {ref}/query-filter-context.html[query and filter
 context] array used to define the conditions for when alerts are created from
 events. Defaults to an empty array.
 
-|index |String[] |Indices on which the rule functions. Defaults to the
+NOTE: This field is not supported for ES\|QL rules.
+
+|index |String[] a|Indices on which the rule functions. Defaults to the
 Security Solution indices defined on the {kib} Advanced Settings page
 (*Kibana* → *Stack Management* → *Advanced Settings* →
 `securitySolution:defaultIndex`).
 
+NOTE: This field is not supported for ES\|QL rules.
+
 |risk_score_mapping |Object[] a|Overrides generated alerts' `risk_score` with
 a value from the source event:
 
@@ -444,6 +448,19 @@ values from the source event:
 When unspecified, rules query the `@timestamp` field. The source field
 must be an {es} date data type.
 
+|exceptions_list |Object[] a|Array of
+<<exceptions-api-overview, exception containers>>, which define
+exceptions that prevent the rule from generating alerts even when its other
+criteria are met. The object has these fields:
+
+* `id` (string, required): ID of the exception container.
+* `list_id` (string, required): List ID of the exception container.
+* `namespace_type` (string required): Determines whether the exceptions are
+valid in only the rule's {kib} space (`single`) or in all {kib} spaces
+(`agnostic`).
+* `type` (string, required): The exception type, which must be either
+a detection rule exception (`detection`) or an endpoint exception (`endpoint`).
+
 |==============================================
 
 [[opt-fields-eql-create]]
@@ -969,6 +986,29 @@ POST api/detection_engine/rules
 `200`::
     Indicates a successful call.
 
+*Example 7*
+
+ES|QL rule that creates alerts from events that match an Excel parent process:
+
+[source,json]
+--------------------------------------------------
+POST api/detection_engine/rules
+{
+  "type": "esql",
+  "language": "esql",
+  "query": "from auditbeat-8.10.2 [metadata _id, _version, _index] | where process.parent.name == \"EXCEL.EXE\"",
+  "name": "Find Excel events",
+  "description": "Find Excel events",
+  "tags": [],
+  "interval": "5m",
+  "from": "now-360s",
+  "to": "now",
+  "enabled": false,
+  "risk_score": 21,
+  "severity": "low"
+}
+--------------------------------------------------
+
 ==== Response payload
 
 A JSON object that includes a unique ID, the time the rule was created, and its
@@ -1327,3 +1367,46 @@ Example response for a new terms rule:
 }
 --------------------------------------------------
 <1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`.
+
+Example response for an ES|QL rule:
+
+[source,json]
+--------------------------------------------------
+{
+  "name": "Find Excel events",
+  "description": "Find Excel events",
+  "risk_score": 21,
+  "severity": "low",
+  "output_index": "",
+  "tags": [],
+  "interval": "5m",
+  "enabled": false,
+  "author": [],
+  "false_positives": [],
+  "from": "now-360s",
+  "max_signals": 100,
+  "risk_score_mapping": [],
+  "severity_mapping": [],
+  "threat": [],
+  "to": "now",
+  "references": [],
+  "version": 1,
+  "exceptions_list": [],
+  "actions": [],
+  "id": "d0f20490-6da4-11ee-b85e-09e9b661f2e2",
+  "updated_at": "2023-10-18T10:55:14.269Z",
+  "updated_by": "elastic",
+  "created_at": "2023-10-18T10:55:14.269Z",
+  "created_by": "elastic",
+  "revision": 0,
+  "rule_id": "e4b53a89-debd-4a0d-a3e3-20606952e589",
+  "immutable": false,
+  "related_integrations": [],     <1>
+  "required_fields": [],          <1>
+  "setup": "",                    <1>
+  "type": "esql",
+  "language": "esql",
+  "query": "from auditbeat-8.10.2 [metadata _id] | where process.parent.name == \"EXCEL.EXE\""
+}
+--------------------------------------------------
+<1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`.
diff --git a/docs/detections/api/rules/rules-api-update.asciidoc b/docs/detections/api/rules/rules-api-update.asciidoc
@@ -60,6 +60,7 @@ occurred
 |type |String a|Data type on which the rule is based:
 
 * `eql`: EQL query (see {ref}/eql.html[Event Query Language]).
+* `esql`: ES\|QL query (refer to {ref}/esql.html[Elasticsearch Query Language]).
 * `query`: query with or without additional filters.
 * `saved_query`: saved search, identified in the `saved_id` field.
 * `machine_learning`: rule based on a {ml} job's anomaly scores.
@@ -125,6 +126,16 @@ generated.
 
 |==============================================
 
+===== Field required for ES|QL rules `PUT` calls
+
+[width="100%",options="header"]
+|==============================================
+|Name |Type |Description
+
+|language |String |Must be `esql`.
+
+|==============================================
+
 ===== Fields required for machine-learning rules `PUT` calls
 
 [width="100%",options="header"]
@@ -260,26 +271,6 @@ number.
 
 |==============================================
 
-===== Optional fields for query, threat-match, EQL, and new terms rules
-
-[width="100%",options="header"]
-|==============================================
-|Name |Type |Description
-
-|exceptions_list |Object[] a|Array of
-<<exceptions-api-overview, exception containers>>, which define
-exceptions that prevent the rule from generating alerts even when its other
-criteria are met. The object has these fields:
-
-* `id` (string, required): ID of the exception container.
-* `list_id` (string, required): List ID of the exception container.
-* `namespace_type` (string required): Determines whether the exceptions are
-valid in only the rule's {kib} space (`single`) or in all {kib} spaces
-(`agnostic`).
-* `type` (string, required): The exception type, which must be either
-a detection rule exception (`detection`) or an endpoint exception (`endpoint`).
-|==============================================
-
 ===== Optional fields for threat-match rules
 
 [width="100%",options="header"]
@@ -304,21 +295,25 @@ documents from the {es} index containing the threat values.
 `kuery` or `lucene`. Defaults to `kuery`.
 |==============================================
 
-===== Optional fields for EQL, query, threshold, and new terms rules
+===== Optional fields for EQL, query, threshold, indicator match, new terms rules, and ES|QL rules 
 
 [width="100%",options="header"]
 |==============================================
 |Name |Type |Description
 
-|filters |Object[] |The {ref}/query-filter-context.html[query and filter
+|filters |Object[] a|The {ref}/query-filter-context.html[query and filter
 context] array used to define the conditions for when alerts are created from
 events. Defaults to an empty array.
 
-|index |String[] |Indices on which the rule functions. Defaults to the
+NOTE: This field is not supported for ES\|QL rules.
+
+|index |String[] a|Indices on which the rule functions. Defaults to the
 Security Solution indices defined on the {kib} Advanced Settings page
 (*Kibana* → *Stack Management* → *Advanced Settings* →
 `securitySolution:defaultIndex`).
 
+NOTE: This field is not supported for ES\|QL rules.
+
 |risk_score_mapping |Object[] a|Overrides generated alerts' `risk_score` with
 a value from the source event:
 
@@ -346,6 +341,19 @@ values from the source event:
 When unspecified, rules query the `@timestamp` field. The source field
 must be an {es} date data type.
 
+|exceptions_list |Object[] a|Array of
+<<exceptions-api-overview, exception containers>>, which define
+exceptions that prevent the rule from generating alerts even when its other
+criteria are met. The object has these fields:
+
+* `id` (string, required): ID of the exception container.
+* `list_id` (string, required): List ID of the exception container.
+* `namespace_type` (string required): Determines whether the exceptions are
+valid in only the rule's {kib} space (`single`) or in all {kib} spaces
+(`agnostic`).
+* `type` (string, required): The exception type, which must be either
+a detection rule exception (`detection`) or an endpoint exception (`endpoint`).
+
 |==============================================
 
 [[opt-fields-eql-update]]