Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new terms rule type to Detections API create/update rule docs #3914

Merged
merged 7 commits into from
Sep 20, 2023
Merged

Add new terms rule type to Detections API create/update rule docs #3914

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
2e75cf0
Add new terms rule type to Detections API create/update rule docs
marshallmain Sep 11, 2023
a01fda6
Apply suggestions from code review
marshallmain Sep 13, 2023
443db43
Update copy for history_window_start recommended format
marshallmain Sep 14, 2023
d169fa7
Apply suggestions from code review
marshallmain Sep 18, 2023
c662b40
Add example request and response for new terms rule
marshallmain Sep 18, 2023
e8ddd20
Fix copy paste errors
marshallmain Sep 19, 2023
79e9a5d
Merge branch 'main' into new-terms-api-docs
marshallmain Sep 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 90 additions & 4 deletions docs/detections/api/rules/rules-api-create.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ an event's `destination.ip` equals a value in the index. The index's field
mappings should be {ecs-ref}[ECS-compliant].
* *{ml-cap} rules*: Creates an alert when a {ml} job discovers an anomaly above
the defined threshold (see <<machine-learning>>).
* *New terms*: Generates an alert for each new term detected in source documents within a specified time range.

IMPORTANT: To create {ml} rules, you must have the
https://www.elastic.co/subscriptions[appropriate license] or use a
Expand Down Expand Up @@ -100,6 +101,7 @@ A JSON object that defines the rule's values:
* <<req-fields-eql>>
* <<req-fields-ml>>
* <<req-fields-threat-match>>
* <<req-fields-new-terms>>
* <<opt-fields-all>>
* <<opt-fields-query-eql>>
* <<opt-fields-threat-match>>
Expand Down Expand Up @@ -145,11 +147,12 @@ occurred
{es} index.
* `threshold`: rule based on the number of times a `query` matches the
specified field.
* `new_terms`: rule that alerts on values that have not been seen before

|==============================================

[[req-fields-query-threshold]]
===== Required field for query, indicator match and threshold rules
===== Required field for query, indicator match, threshold, and new terms rules

[width="100%",options="header"]
|==============================================
Expand Down Expand Up @@ -250,6 +253,22 @@ uses both `and` and `or` logic.

|==============================================

[[req-fields-new-terms]]
===== Required fields for new terms rules

[width="100%",options="header"]
|==============================================
|Name |Type |Description

|new_terms_fields |String[] |Fields to monitor for new values. Must contain 1–3 field names.

|history_window_start |String |Start date to use when checking if a term has been seen before.
Supports relative dates – for example, `now-30d` will search the last 30 days of data when checking if a term
is new. We do not recommend using absolute dates, which can cause issues with rule performance
due to querying increasing amounts of data over time.

|==============================================

[[opt-fields-all]]
===== Optional fields for all rule types

Expand Down Expand Up @@ -336,7 +355,7 @@ Required when `actions` are used to send notifications.
|==============================================

[[opt-fields-query-eql]]
===== Optional fields for query, indicator match, and event correlation rules
===== Optional fields for query, indicator match, event correlation, and new terms rules

[width="100%",options="header"]
|==============================================
Expand Down Expand Up @@ -372,7 +391,7 @@ documents from the {es} index containing the threat values.
|==============================================

[[opt-fields-query-threshold]]
===== Optional fields for query, indicator match, and threshold rules
===== Optional fields for query, indicator match, threshold, and new terms rules

[width="100%",options="header"]
|==============================================
Expand All @@ -383,7 +402,7 @@ documents from the {es} index containing the threat values.
|==============================================

[[opt-fields-eql-query-threshold]]
===== Optional fields for event correlation, query, and threshold rules
===== Optional fields for event correlation, query, threshold, and new terms rules

[width="100%",options="header"]
|==============================================
Expand Down Expand Up @@ -923,6 +942,28 @@ logic. In this example, both the event's `destination.ip` and
<4> Sibling `entries` are evaluated using `or` logic. An alert is generated when
at least one `entries` object evaluates to `true`.

*Example 6*

New terms rule that creates alerts a new IP address is detected for a user:

[source,console]
--------------------------------------------------
POST api/detection_engine/rules
{
"risk_score": 21,
"description": "Detects a user associated with a new IP address",
"name": "New User IP Detected",
"severity": "medium",
natasha-moore-elastic marked this conversation as resolved.
Show resolved Hide resolved
"type": "new_terms",
"language": "kuery",
"query": "*",
"new_terms_fields": ["user.id", "source.ip"],
"history_window_start": "now-30d",
"index": ["auditbeat*"]
}
--------------------------------------------------
// KIBANA

==== Response code

`200`::
Expand Down Expand Up @@ -1241,3 +1282,48 @@ Example response for an indicator match rule:
}
--------------------------------------------------
<1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`.

Example response for a new terms rule:

[source,json]
--------------------------------------------------
{
"author": [],
"created_at": "2020-10-06T07:07:58.227Z",
"updated_at": "2020-10-06T07:07:58.237Z",
"created_by": "elastic",
"description": "Detects a user associated with a new IP address",
"enabled": true,
"false_positives": [],
"from": "now-6m",
"id": "eb7225c0-566b-11ee-8b4f-bbf3afdeb9f4",
"immutable": false,
"interval": "5m",
"rule_id": "c6f5d0bc-7be9-47d4-b2f3-073d22641e30",
"max_signals": 100,
"risk_score": 21,
"risk_score_mapping": [],
"name": "New User IP Detected",
"references": [],
"severity": "medium",
"severity_mapping": [],
"updated_by": "elastic",
"tags": [],
"to": "now",
"type": "new_terms",
"threat": [],
"version": 1,
"exceptions_list": [],
"index": [
"auditbeat*"
],
"query": "*",
"language": "kuery",
"new_terms_fields": ["user.id", "source.ip"],
"history_window_start": "now-30d",
"related_integrations": [], <1>
"required_fields": [], <1>
"setup": "" <1>
}
--------------------------------------------------
<1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`.
24 changes: 20 additions & 4 deletions docs/detections/api/rules/rules-api-update.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,10 +67,11 @@ occurred
{es} index.
* `threshold`: rule based on the number of times a `query` matches the
specified field.
* `new_terms`: rule that alerts on values that have not been seen before

|==============================================

===== Field required for query, threat-match and threshold rules `PUT` calls
===== Field required for query, threat-match, threshold, and new terms rules `PUT` calls

[width="100%",options="header"]
|==============================================
Expand Down Expand Up @@ -166,6 +167,21 @@ uses both `and` and `or` logic.

|==============================================

===== Fields required for new terms rules `PUT` calls

[width="100%",options="header"]
|==============================================
|Name |Type |Description

|new_terms_fields |String[] |Fields to monitor for new values. Must contain 1–3 field names.

|history_window_start |String |Start date to use when checking if a term has been seen before.
Supports relative dates – for example, `now-30d` will search the last 30 days of data when checking if a term
is new. We do not recommend using absolute dates, which can cause issues with rule performance
due to querying increasing amounts of data over time.

|==============================================

===== Optional fields for all rule types

[width="100%",options="header"]
Expand Down Expand Up @@ -244,7 +260,7 @@ number.

|==============================================

===== Optional fields for query, threat-match and EQL rules
===== Optional fields for query, threat-match, EQL, and new terms rules

[width="100%",options="header"]
|==============================================
Expand Down Expand Up @@ -278,7 +294,7 @@ documents from the {es} index containing the threat values.
|Much like an ingest processor, users can use this field to define where their threat indicator can be found on their indicator documents. Defaults to `threatintel.indicator`.
|==============================================

===== Optional fields for query, threat-match and threshold rules
===== Optional fields for query, threat-match, threshold, and new terms rules

[width="100%",options="header"]
|==============================================
Expand All @@ -288,7 +304,7 @@ documents from the {es} index containing the threat values.
`kuery` or `lucene`. Defaults to `kuery`.
|==============================================

===== Optional fields for EQL, query and threshold rules
===== Optional fields for EQL, query, threshold, and new terms rules

[width="100%",options="header"]
|==============================================
Expand Down