Add new terms rule type to Detections API create/update rule docs (#3914

) (#3975) * Add new terms rule type to Detections API create/update rule docs * Apply suggestions from code review Co-authored-by: natasha-moore-elastic <[email protected]> * Update copy for history_window_start recommended format * Apply suggestions from code review Co-authored-by: natasha-moore-elastic <[email protected]> * Add example request and response for new terms rule * Fix copy paste errors --------- Co-authored-by: natasha-moore-elastic <[email protected]> (cherry picked from commit cbd0155) Co-authored-by: Marshall Main <[email protected]>
elastic · Sep 26, 2023 · 7b6382d · 7b6382d
1 parent 99e58c7
commit 7b6382d
Show file tree

Hide file tree

Showing 2 changed files with 110 additions and 8 deletions.
diff --git a/docs/detections/api/rules/rules-api-create.asciidoc b/docs/detections/api/rules/rules-api-create.asciidoc
@@ -33,6 +33,7 @@ an event's `destination.ip` equals a value in the index. The index's field
 mappings should be {ecs-ref}[ECS-compliant].
 * *{ml-cap} rules*: Creates an alert when a {ml} job discovers an anomaly above
 the defined threshold (see <<machine-learning>>).
+* *New terms*: Generates an alert for each new term detected in source documents within a specified time range.
 
 IMPORTANT: To create {ml} rules, you must have the
 https://www.elastic.co/subscriptions[appropriate license] or use a
@@ -100,6 +101,7 @@ A JSON object that defines the rule's values:
 * <<req-fields-eql>>
 * <<req-fields-ml>>
 * <<req-fields-threat-match>>
+* <<req-fields-new-terms>>
 * <<opt-fields-all>>
 * <<opt-fields-query-eql>>
 * <<opt-fields-threat-match>>
@@ -145,11 +147,12 @@ occurred
 {es} index.
 * `threshold`: rule based on the number of times a `query` matches the
 specified field.
+* `new_terms`: rule that alerts on values that have not been seen before
 
 |==============================================
 
 [[req-fields-query-threshold]]
-===== Required field for query, indicator match and threshold rules
+===== Required field for query, indicator match, threshold, and new terms rules
 
 [width="100%",options="header"]
 |==============================================
@@ -250,6 +253,22 @@ uses both `and` and `or` logic.
 
 |==============================================
 
+[[req-fields-new-terms]]
+===== Required fields for new terms rules
+
+[width="100%",options="header"]
+|==============================================
+|Name |Type |Description
+
+|new_terms_fields |String[] |Fields to monitor for new values. Must contain 1–3 field names.
+
+|history_window_start |String |Start date to use when checking if a term has been seen before.
+Supports relative dates – for example, `now-30d` will search the last 30 days of data when checking if a term
+is new. We do not recommend using absolute dates, which can cause issues with rule performance
+due to querying increasing amounts of data over time.
+
+|==============================================
+
 [[opt-fields-all]]
 ===== Optional fields for all rule types
 
@@ -336,7 +355,7 @@ Required when `actions` are used to send notifications.
 |==============================================
 
 [[opt-fields-query-eql]]
-===== Optional fields for query, indicator match, and event correlation rules
+===== Optional fields for query, indicator match, event correlation, and new terms rules
 
 [width="100%",options="header"]
 |==============================================
@@ -372,7 +391,7 @@ documents from the {es} index containing the threat values.
 |==============================================
 
 [[opt-fields-query-threshold]]
-===== Optional fields for query, indicator match, and threshold rules
+===== Optional fields for query, indicator match, threshold, and new terms rules
 
 [width="100%",options="header"]
 |==============================================
@@ -383,7 +402,7 @@ documents from the {es} index containing the threat values.
 |==============================================
 
 [[opt-fields-eql-query-threshold]]
-===== Optional fields for event correlation, query, and threshold rules
+===== Optional fields for event correlation, query, threshold, and new terms rules
 
 [width="100%",options="header"]
 |==============================================
@@ -923,6 +942,28 @@ logic. In this example, both the event's `destination.ip` and
 <4> Sibling `entries` are evaluated using `or` logic. An alert is generated when
 at least one `entries` object evaluates to `true`.
 
+*Example 6*
+
+New terms rule that creates alerts a new IP address is detected for a user:
+
+[source,console]
+--------------------------------------------------
+POST api/detection_engine/rules
+{
+  "risk_score": 21,
+  "description": "Detects a user associated with a new IP address",
+  "name": "New User IP Detected",
+  "severity": "medium",
+  "type": "new_terms",
+  "language": "kuery",
+  "query": "*",
+  "new_terms_fields": ["user.id", "source.ip"],
+  "history_window_start": "now-30d",
+  "index": ["auditbeat*"]
+}
+--------------------------------------------------
+// KIBANA
+
 ==== Response code
 
 `200`::
@@ -1241,3 +1282,48 @@ Example response for an indicator match rule:
 }
 --------------------------------------------------
 <1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`.
+
+Example response for a new terms rule:
+
+[source,json]
+--------------------------------------------------
+{
+  "author": [],
+  "created_at": "2020-10-06T07:07:58.227Z",
+  "updated_at": "2020-10-06T07:07:58.237Z",
+  "created_by": "elastic",
+  "description": "Detects a user associated with a new IP address",
+  "enabled": true,
+  "false_positives": [],
+  "from": "now-6m",
+  "id": "eb7225c0-566b-11ee-8b4f-bbf3afdeb9f4",
+  "immutable": false,
+  "interval": "5m",
+  "rule_id": "c6f5d0bc-7be9-47d4-b2f3-073d22641e30",
+  "max_signals": 100,
+  "risk_score": 21,
+  "risk_score_mapping": [],
+  "name": "New User IP Detected",
+  "references": [],
+  "severity": "medium",
+  "severity_mapping": [],
+  "updated_by": "elastic",
+  "tags": [],
+  "to": "now",
+  "type": "new_terms",
+  "threat": [],
+  "version": 1,
+  "exceptions_list": [],
+  "index": [
+    "auditbeat*"
+  ],
+  "query": "*",
+  "language": "kuery",
+  "new_terms_fields": ["user.id", "source.ip"],
+  "history_window_start": "now-30d",
+  "related_integrations": [],       <1>
+  "required_fields": [],            <1>
+  "setup": ""                       <1>
+}
+--------------------------------------------------
+<1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`.
diff --git a/docs/detections/api/rules/rules-api-update.asciidoc b/docs/detections/api/rules/rules-api-update.asciidoc
@@ -67,10 +67,11 @@ occurred
 {es} index.
 * `threshold`: rule based on the number of times a `query` matches the
 specified field.
+* `new_terms`: rule that alerts on values that have not been seen before
 
 |==============================================
 
-===== Field required for query, threat-match and threshold rules `PUT` calls
+===== Field required for query, threat-match, threshold, and new terms rules `PUT` calls
 
 [width="100%",options="header"]
 |==============================================
@@ -166,6 +167,21 @@ uses both `and` and `or` logic.
 
 |==============================================
 
+===== Fields required for new terms rules `PUT` calls
+
+[width="100%",options="header"]
+|==============================================
+|Name |Type |Description
+
+|new_terms_fields |String[] |Fields to monitor for new values. Must contain 1–3 field names.
+
+|history_window_start |String |Start date to use when checking if a term has been seen before.
+Supports relative dates – for example, `now-30d` will search the last 30 days of data when checking if a term
+is new. We do not recommend using absolute dates, which can cause issues with rule performance
+due to querying increasing amounts of data over time.
+
+|==============================================
+
 ===== Optional fields for all rule types
 
 [width="100%",options="header"]
@@ -244,7 +260,7 @@ number.
 
 |==============================================
 
-===== Optional fields for query, threat-match and EQL rules
+===== Optional fields for query, threat-match, EQL, and new terms rules
 
 [width="100%",options="header"]
 |==============================================
@@ -278,7 +294,7 @@ documents from the {es} index containing the threat values.
 |Much like an ingest processor, users can use this field to define where their threat indicator can be found on their indicator documents. Defaults to `threatintel.indicator`.
 |==============================================
 
-===== Optional fields for query, threat-match and threshold rules
+===== Optional fields for query, threat-match, threshold, and new terms rules
 
 [width="100%",options="header"]
 |==============================================
@@ -288,7 +304,7 @@ documents from the {es} index containing the threat values.
 `kuery` or `lucene`. Defaults to `kuery`.
 |==============================================
 
-===== Optional fields for EQL, query and threshold rules
+===== Optional fields for EQL, query, threshold, and new terms rules
 
 [width="100%",options="header"]
 |==============================================