From df6f68a28d3f9a2733cdb5f38c27c0da280e6c1c Mon Sep 17 00:00:00 2001 From: "mergify[bot]" <37929162+mergify[bot]@users.noreply.github.com> Date: Tue, 26 Sep 2023 07:05:07 -0700 Subject: [PATCH] Add new terms rule type to Detections API create/update rule docs (#3914) (#3978) * Add new terms rule type to Detections API create/update rule docs * Apply suggestions from code review Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> * Update copy for history_window_start recommended format * Apply suggestions from code review Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> * Add example request and response for new terms rule * Fix copy paste errors --------- Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> (cherry picked from commit 67448e06725d004f6412b2ce27d79f85238a3d6d) Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com> --- .../api/rules/rules-api-create.asciidoc | 94 ++++++++++++++++++- .../api/rules/rules-api-update.asciidoc | 24 ++++- 2 files changed, 110 insertions(+), 8 deletions(-) diff --git a/docs/detections/api/rules/rules-api-create.asciidoc b/docs/detections/api/rules/rules-api-create.asciidoc index 0492af7d93..572aed55fb 100644 --- a/docs/detections/api/rules/rules-api-create.asciidoc +++ b/docs/detections/api/rules/rules-api-create.asciidoc @@ -28,6 +28,7 @@ an event's `destination.ip` equals a value in the index. The index's field mappings should be {ecs-ref}[ECS-compliant]. * *{ml-cap} rules*: Creates an alert when a {ml} job discovers an anomaly above the defined threshold (see <>). +* *New terms*: Generates an alert for each new term detected in source documents within a specified time range. IMPORTANT: To create {ml} rules, you must have the https://www.elastic.co/subscriptions[appropriate license] or use a @@ -95,6 +96,7 @@ A JSON object that defines the rule's values: * <> * <> * <> +* <> * <> * <> * <> @@ -140,11 +142,12 @@ occurred {es} index. * `threshold`: rule based on the number of times a `query` matches the specified field. +* `new_terms`: rule that alerts on values that have not been seen before |============================================== [[req-fields-query-threshold]] -===== Required field for query, indicator match and threshold rules +===== Required field for query, indicator match, threshold, and new terms rules [width="100%",options="header"] |============================================== @@ -245,6 +248,22 @@ uses both `and` and `or` logic. |============================================== +[[req-fields-new-terms]] +===== Required fields for new terms rules + +[width="100%",options="header"] +|============================================== +|Name |Type |Description + +|new_terms_fields |String[] |Fields to monitor for new values. Must contain 1–3 field names. + +|history_window_start |String |Start date to use when checking if a term has been seen before. +Supports relative dates – for example, `now-30d` will search the last 30 days of data when checking if a term +is new. We do not recommend using absolute dates, which can cause issues with rule performance +due to querying increasing amounts of data over time. + +|============================================== + [[opt-fields-all]] ===== Optional fields for all rule types @@ -318,7 +337,7 @@ Required when `actions` are used to send notifications. |============================================== [[opt-fields-query-eql]] -===== Optional fields for query, indicator match, and event correlation rules +===== Optional fields for query, indicator match, event correlation, and new terms rules [width="100%",options="header"] |============================================== @@ -354,7 +373,7 @@ documents from the {es} index containing the threat values. |============================================== [[opt-fields-query-threshold]] -===== Optional fields for query, indicator match, and threshold rules +===== Optional fields for query, indicator match, threshold, and new terms rules [width="100%",options="header"] |============================================== @@ -365,7 +384,7 @@ documents from the {es} index containing the threat values. |============================================== [[opt-fields-eql-query-threshold]] -===== Optional fields for event correlation, query, and threshold rules +===== Optional fields for event correlation, query, threshold, and new terms rules [width="100%",options="header"] |============================================== @@ -799,6 +818,28 @@ logic. In this example, both the event's `destination.ip` and <4> Sibling `entries` are evaluated using `or` logic. An alert is generated when at least one `entries` object evaluates to `true`. +*Example 6* + +New terms rule that creates alerts a new IP address is detected for a user: + +[source,console] +-------------------------------------------------- +POST api/detection_engine/rules +{ + "risk_score": 21, + "description": "Detects a user associated with a new IP address", + "name": "New User IP Detected", + "severity": "medium", + "type": "new_terms", + "language": "kuery", + "query": "*", + "new_terms_fields": ["user.id", "source.ip"], + "history_window_start": "now-30d", + "index": ["auditbeat*"] +} +-------------------------------------------------- +// KIBANA + ==== Response code `200`:: @@ -1118,3 +1159,48 @@ Example response for an indicator match rule: } -------------------------------------------------- <1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`. + +Example response for a new terms rule: + +[source,json] +-------------------------------------------------- +{ + "author": [], + "created_at": "2020-10-06T07:07:58.227Z", + "updated_at": "2020-10-06T07:07:58.237Z", + "created_by": "elastic", + "description": "Detects a user associated with a new IP address", + "enabled": true, + "false_positives": [], + "from": "now-6m", + "id": "eb7225c0-566b-11ee-8b4f-bbf3afdeb9f4", + "immutable": false, + "interval": "5m", + "rule_id": "c6f5d0bc-7be9-47d4-b2f3-073d22641e30", + "max_signals": 100, + "risk_score": 21, + "risk_score_mapping": [], + "name": "New User IP Detected", + "references": [], + "severity": "medium", + "severity_mapping": [], + "updated_by": "elastic", + "tags": [], + "to": "now", + "type": "new_terms", + "threat": [], + "version": 1, + "exceptions_list": [], + "index": [ + "auditbeat*" + ], + "query": "*", + "language": "kuery", + "new_terms_fields": ["user.id", "source.ip"], + "history_window_start": "now-30d", + "related_integrations": [], <1> + "required_fields": [], <1> + "setup": "" <1> +} +-------------------------------------------------- +<1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`. diff --git a/docs/detections/api/rules/rules-api-update.asciidoc b/docs/detections/api/rules/rules-api-update.asciidoc index 6abdff0d0e..0b2b47d266 100644 --- a/docs/detections/api/rules/rules-api-update.asciidoc +++ b/docs/detections/api/rules/rules-api-update.asciidoc @@ -67,10 +67,11 @@ occurred {es} index. * `threshold`: rule based on the number of times a `query` matches the specified field. +* `new_terms`: rule that alerts on values that have not been seen before |============================================== -===== Field required for query, threat-match and threshold rules `PUT` calls +===== Field required for query, threat-match, threshold, and new terms rules `PUT` calls [width="100%",options="header"] |============================================== @@ -166,6 +167,21 @@ uses both `and` and `or` logic. |============================================== +===== Fields required for new terms rules `PUT` calls + +[width="100%",options="header"] +|============================================== +|Name |Type |Description + +|new_terms_fields |String[] |Fields to monitor for new values. Must contain 1–3 field names. + +|history_window_start |String |Start date to use when checking if a term has been seen before. +Supports relative dates – for example, `now-30d` will search the last 30 days of data when checking if a term +is new. We do not recommend using absolute dates, which can cause issues with rule performance +due to querying increasing amounts of data over time. + +|============================================== + ===== Optional fields for all rule types [width="100%",options="header"] @@ -237,7 +253,7 @@ number. |============================================== -===== Optional fields for query, threat-match and EQL rules +===== Optional fields for query, threat-match, EQL, and new terms rules [width="100%",options="header"] |============================================== @@ -271,7 +287,7 @@ documents from the {es} index containing the threat values. |Much like an ingest processor, users can use this field to define where their threat indicator can be found on their indicator documents. Defaults to `threatintel.indicator`. |============================================== -===== Optional fields for query, threat-match and threshold rules +===== Optional fields for query, threat-match, threshold, and new terms rules [width="100%",options="header"] |============================================== @@ -281,7 +297,7 @@ documents from the {es} index containing the threat values. `kuery` or `lucene`. Defaults to `kuery`. |============================================== -===== Optional fields for EQL, query and threshold rules +===== Optional fields for EQL, query, threshold, and new terms rules [width="100%",options="header"] |==============================================