Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Detection Engine] update API docs for new ES|QL rule #4052

Merged
merged 28 commits into from
Oct 26, 2023
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
3dc8d32
initial commit
vitaliidm Oct 17, 2023
1ef78b7
updates doc
vitaliidm Oct 17, 2023
b05ceb6
update API
vitaliidm Oct 17, 2023
92577dc
Update docs/detections/api/rules/rules-api-create.asciidoc
vitaliidm Oct 18, 2023
d05aa8b
Update docs/detections/api/rules/rules-api-create.asciidoc
vitaliidm Oct 18, 2023
71de0eb
Update docs/detections/api/rules/rules-api-update.asciidoc
vitaliidm Oct 18, 2023
79423b9
CR feedback
vitaliidm Oct 18, 2023
122e781
Merge branch 'esql-api' of ssh://github.com/vitaliidm/security-docs i…
vitaliidm Oct 18, 2023
11c615d
Update docs/detections/api/rules/rules-api-create.asciidoc
vitaliidm Oct 18, 2023
ef14010
Update docs/detections/api/rules/rules-api-create.asciidoc
vitaliidm Oct 18, 2023
5f67351
Update docs/detections/api/rules/rules-api-create.asciidoc
vitaliidm Oct 18, 2023
b2dad40
Update docs/detections/api/rules/rules-api-create.asciidoc
vitaliidm Oct 18, 2023
3bad833
CR
vitaliidm Oct 18, 2023
faaf521
Update docs/detections/api/rules/rules-api-create.asciidoc
vitaliidm Oct 18, 2023
a8023c6
Update docs/detections/api/rules/rules-api-update.asciidoc
vitaliidm Oct 18, 2023
73dff75
Combining tables and adding note to update rule
nastasha-solomon Oct 18, 2023
8aaaec9
Update docs/detections/api/rules/rules-api-create.asciidoc
vitaliidm Oct 18, 2023
a13764c
Merge branch 'main' into esql-api
nastasha-solomon Oct 18, 2023
3184ad9
Removing extra term
nastasha-solomon Oct 18, 2023
5d69fe4
Combining tables in create rule
nastasha-solomon Oct 18, 2023
2eda287
Fixing tag rendering
nastasha-solomon Oct 18, 2023
576067e
Adding ref to UI docs
nastasha-solomon Oct 18, 2023
0ef8d4e
littile cleanup and fixes
vitaliidm Oct 19, 2023
01d1c7f
Update docs/detections/api/rules/rules-api-create.asciidoc
vitaliidm Oct 23, 2023
4b04630
Update docs/detections/api/rules/rules-api-create.asciidoc
vitaliidm Oct 23, 2023
4b8a2a2
Update docs/detections/api/rules/rules-api-create.asciidoc
vitaliidm Oct 24, 2023
476d532
Merge branch 'main' into esql-api
vitaliidm Oct 25, 2023
afc4ab5
Merge branch 'main' into esql-api
vitaliidm Oct 26, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 93 additions & 1 deletion docs/detections/api/rules/rules-api-create.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ mappings should be {ecs-ref}[ECS-compliant].
* *{ml-cap} rules*: Creates an alert when a {ml} job discovers an anomaly above
the defined threshold (see <<machine-learning>>).
* *New terms*: Generates an alert for each new term detected in source documents within a specified time range.
* *ES|QL*: Uses the Elasticsearch Query Language (ES|QL) rule to find events and aggregate search results.
vitaliidm marked this conversation as resolved.
Show resolved Hide resolved

NOTE: ES|QL rule is in technical preview stage
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved

IMPORTANT: To create {ml} rules, you must have the
https://www.elastic.co/subscriptions[appropriate license] or use a
Expand Down Expand Up @@ -102,11 +105,13 @@ A JSON object that defines the rule's values:
* <<req-fields-ml>>
* <<req-fields-threat-match>>
* <<req-fields-new-terms>>
* <<req-fields-esql>>
* <<opt-fields-all>>
* <<opt-fields-query-eql>>
* <<opt-fields-threat-match>>
* <<opt-fields-query-threshold>>
* <<opt-fields-eql-query-threshold>>
* <<opt-fields-eql-query-threshold-new_terms-esql>>

[[ref-fields-all]]
===== Required fields for all rule types
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved
Expand Down Expand Up @@ -140,6 +145,7 @@ occurred
|type |String a|Data type on which the rule is based:

* `eql`: EQL query (see {ref}/eql.html[Event Query Language]).
* `esql`: ES\|QL query (see {ref}/esql.html[Elasticsearch Query Language]).
* `query`: query with or without additional filters.
* `saved_query`: saved search, identified in the `saved_id` field.
* `machine_learning`: rule based on a {ml} job's anomaly scores.
Expand All @@ -152,7 +158,7 @@ specified field.
|==============================================

[[req-fields-query-threshold]]
===== Required field for query, indicator match, threshold, and new terms rules
===== Required field for query, indicator match, threshold, new terms, event correlation, and ES|QL rules
vitaliidm marked this conversation as resolved.
Show resolved Hide resolved

[width="100%",options="header"]
|==============================================
Expand Down Expand Up @@ -209,6 +215,17 @@ generated.

|==============================================

[[req-fields-esql]]
===== Required field for ES|QL rules

[width="100%",options="header"]
|==============================================
|Name |Type |Description

|language |String |Must be `esql`.

|==============================================

[[req-fields-ml]]
===== Required fields for machine learning rules

Expand Down Expand Up @@ -417,6 +434,15 @@ Security Solution indices defined on the {kib} Advanced Settings page
(*Kibana* → *Stack Management* → *Advanced Settings* →
`securitySolution:defaultIndex`).

|==============================================

[[opt-fields-eql-query-threshold-new_terms-esql]]
===== Optional fields for event correlation, query, threshold, new terms, and ES|QL rules
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved

[width="100%",options="header"]
|==============================================
|Name |Type |Description

|risk_score_mapping |Object[] a|Overrides generated alerts' `risk_score` with
a value from the source event:

Expand Down Expand Up @@ -969,6 +995,29 @@ POST api/detection_engine/rules
`200`::
Indicates a successful call.

*Example 7*

ES|QL rule that creates alerts from events that match an Excel parent process:

[source,json]
--------------------------------------------------
POST api/detection_engine/rules
{
"type":"esql",
"language":"esql",
"query":"from auditbeat-8.10.2 [metadata _id] | where process.parent.name == \"EXCEL.EXE\"",
vitaliidm marked this conversation as resolved.
Show resolved Hide resolved
"name":"Find Excel events",
"description":"Find Excel events",
"tags":[],
"interval":"5m",
"from":"now-360s",
"to":"now",
"enabled":false,
"risk_score": 21,
"severity": "low"
}
--------------------------------------------------

==== Response payload

A JSON object that includes a unique ID, the time the rule was created, and its
Expand Down Expand Up @@ -1327,3 +1376,46 @@ Example response for a new terms rule:
}
--------------------------------------------------
<1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`.

Example response for an ES|QL rule:

[source,json]
--------------------------------------------------
{
"name": "Find Excel events",
"description": "Find Excel events",
"risk_score": 21,
"severity": "low",
"output_index": "",
"tags": [],
"interval": "5m",
"enabled": false,
"author": [],
"false_positives": [],
"from": "now-360s",
"max_signals": 100,
"risk_score_mapping": [],
"severity_mapping": [],
"threat": [],
"to": "now",
"references": [],
"version": 1,
"exceptions_list": [],
"actions": [],
"id": "d0f20490-6da4-11ee-b85e-09e9b661f2e2",
"updated_at": "2023-10-18T10:55:14.269Z",
"updated_by": "elastic",
"created_at": "2023-10-18T10:55:14.269Z",
"created_by": "elastic",
"revision": 0,
"rule_id": "e4b53a89-debd-4a0d-a3e3-20606952e589",
"immutable": false,
"related_integrations": [], <1>
"required_fields": [], <1>
"setup": "", <1>
"type": "esql",
"language": "esql",
"query": "from auditbeat-8.10.2 [metadata _id] | where process.parent.name == \"EXCEL.EXE\""
vitaliidm marked this conversation as resolved.
Show resolved Hide resolved
}
--------------------------------------------------
<1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`.
19 changes: 19 additions & 0 deletions docs/detections/api/rules/rules-api-update.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ occurred
|type |String a|Data type on which the rule is based:

* `eql`: EQL query (see {ref}/eql.html[Event Query Language]).
* `esql`: ES\|QL query (see {ref}/esql.html[Elasticsearch Query Language]).
vitaliidm marked this conversation as resolved.
Show resolved Hide resolved
* `query`: query with or without additional filters.
* `saved_query`: saved search, identified in the `saved_id` field.
* `machine_learning`: rule based on a {ml} job's anomaly scores.
Expand Down Expand Up @@ -125,6 +126,16 @@ generated.

|==============================================

===== Field required for ES|QL rules `PUT` calls

[width="100%",options="header"]
|==============================================
|Name |Type |Description

|language |String |Must be `esql`.

|==============================================

===== Fields required for machine-learning rules `PUT` calls

[width="100%",options="header"]
Expand Down Expand Up @@ -319,6 +330,14 @@ Security Solution indices defined on the {kib} Advanced Settings page
(*Kibana* → *Stack Management* → *Advanced Settings* →
`securitySolution:defaultIndex`).

|==============================================

===== Optional fields for EQL, query, threshold, new terms, and ES|QL rules
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved

[width="100%",options="header"]
|==============================================
|Name |Type |Description

|risk_score_mapping |Object[] a|Overrides generated alerts' `risk_score` with
a value from the source event:

Expand Down