From f6afb0b6ad0a625e55eeddceeb64b1f67f559509 Mon Sep 17 00:00:00 2001 From: "mergify[bot]" <37929162+mergify[bot]@users.noreply.github.com> Date: Mon, 6 Nov 2023 12:02:36 -0500 Subject: [PATCH] [8.11] ES|QL rule docs (backport #4062) (#4196) Co-authored-by: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> Co-authored-by: Abdon Pijpelink Co-authored-by: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> --- docs/detections/about-rules.asciidoc | 4 + .../api/rules/rules-api-create.asciidoc | 12 +- .../api/rules/rules-api-update.asciidoc | 4 +- docs/detections/images/esql-ref-button.png | Bin 0 -> 3453 bytes docs/detections/rules-ui-create.asciidoc | 112 ++++++++++++++---- 5 files changed, 104 insertions(+), 28 deletions(-) create mode 100644 docs/detections/images/esql-ref-button.png diff --git a/docs/detections/about-rules.asciidoc b/docs/detections/about-rules.asciidoc index d02ef77d7c..96e131c745 100644 --- a/docs/detections/about-rules.asciidoc +++ b/docs/detections/about-rules.asciidoc @@ -42,6 +42,10 @@ TIP: You can also use value lists as the indicator match index. See <>: Generates an alert for each new term detected in source documents within a specified time range. You can also detect a combination of up to three new terms (for example, a `host.ip` and `host.id` that have never been observed together before). +* <>: Searches the defined indices and creates an alert when results match an {ref}/esql.html[Elasticsearch Query Language (ES|QL)] query. ++ +preview::[] + [role="screenshot"] image::images/all-rules.png[Shows the Rules page] diff --git a/docs/detections/api/rules/rules-api-create.asciidoc b/docs/detections/api/rules/rules-api-create.asciidoc index 3603168eb6..3a58fcf0ee 100644 --- a/docs/detections/api/rules/rules-api-create.asciidoc +++ b/docs/detections/api/rules/rules-api-create.asciidoc @@ -37,7 +37,7 @@ create an index for IP addresses and use this index to create an alert whenever an event's `destination.ip` equals a value in the index. The index's field mappings should be {ecs-ref}[ECS-compliant]. * *New terms*: Generates an alert for each new term detected in source documents within a specified time range. -* *ES|QL*: Uses {ref}/esql.html[Elasticsearch Query Language (ES|QL)] to find events and aggregate search results. +* *{esql}*: Uses {ref}/esql.html[Elasticsearch Query Language ({esql})] to find events and aggregate search results. + preview::[] * *{ml-cap} rules*: Creates an alert when a {ml} job discovers an anomaly above @@ -161,7 +161,7 @@ specified field. |============================================== [[req-fields-query-threshold]] -===== Required fields for query, indicator match, threshold, new terms, event correlation, and ES|QL rules +===== Required fields for query, indicator match, threshold, new terms, event correlation, and {esql} rules [width="100%",options="header"] |============================================== @@ -225,7 +225,7 @@ generated. |============================================== [[req-fields-esql]] -===== Required field for ES|QL rules +===== Required field for {esql} rules [width="100%",options="header"] |============================================== @@ -407,7 +407,7 @@ documents from the {es} index containing the threat values. |============================================== [[opt-fields-eql-query-threshold]] -===== Optional fields for event correlation, query, threshold, indicator match, new terms, and ES|QL rules +===== Optional fields for event correlation, query, threshold, indicator match, new terms, and {esql} rules [width="100%",options="header"] |============================================== @@ -993,7 +993,7 @@ POST api/detection_engine/rules *Example 7* -ES|QL rule that creates alerts from events that match an Excel parent process: +{esql} rule that creates alerts from events that match an Excel parent process: [source,json] -------------------------------------------------- @@ -1373,7 +1373,7 @@ Example response for a new terms rule: -------------------------------------------------- <1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`. -Example response for an ES|QL rule: +Example response for an {esql} rule: [source,json] -------------------------------------------------- diff --git a/docs/detections/api/rules/rules-api-update.asciidoc b/docs/detections/api/rules/rules-api-update.asciidoc index b6b9129cc2..eb0b156e9c 100644 --- a/docs/detections/api/rules/rules-api-update.asciidoc +++ b/docs/detections/api/rules/rules-api-update.asciidoc @@ -131,7 +131,7 @@ generated. |============================================== -===== Field required for ES|QL rules `PUT` calls +===== Field required for {esql} rules `PUT` calls [width="100%",options="header"] |============================================== @@ -300,7 +300,7 @@ documents from the {es} index containing the threat values. `kuery` or `lucene`. Defaults to `kuery`. |============================================== -===== Optional fields for EQL, query, threshold, indicator match, new terms rules, and ES|QL rules +===== Optional fields for EQL, query, threshold, indicator match, new terms rules, and {esql} rules [width="100%",options="header"] |============================================== diff --git a/docs/detections/images/esql-ref-button.png b/docs/detections/images/esql-ref-button.png new file mode 100644 index 0000000000000000000000000000000000000000..0907eb64419048cc8d1b383dd5896621fab8b62f GIT binary patch literal 3453 zcmV-@4TAECP)4Tx07!|Imj_f+=@Nkdd(%TgONfA!(3|uYKuQQLfP!?e5t0x>i7_OA*iczq z!HS9qmK9wUQLs_OhKOQY6cn&8$RePuiv>|pc^5bYci(yMoj2#qnJ<(1XXd}>|L>U# z0GY}Wi&J4`0MbNKNtnMkJtj7ej_U(NppGaRm~y!3VrFP)FhFF={#`zH0BGFaw;a#) z@BjZ-QQ`5^xd1?+2q*B;xoHTWMRbTOmPi3W9YXvDnNsmM-$0luK^jE(={O4}d1#yy zCK!#Lh=~YeA*>Dnp2Xou1OSi>5kEbHD?sv)d=Rz~@q{9Tk>9{0Eo})8VWciFMPkGV zkLPD3{E$cRx10&`oE(lo&XaqY5WJ5tU7X5U_J2N(|1D{$vZ)%zh?0^d0bz(m`hp&& zED4gcC}D2M6bpG1@e`aR3y7RzZaOP&ig_HLAo<$VxxrJMDD-2?^GMkdQ_N5I4WDAk zk}!F#LOVTR6GF0 z>S>?pLI9Ne0Jw31E0bhQ`o^ylz#wk|B|rrlKnEBAQ(y_~fD>>7UceUwf)Ee^V!w zt9ON20mtrRY`Yjp!2eQFINu34I^kjUGaO$B;2v7z>OmhK-5A2r!wLJWLVhFs2%F z4bzV4#eBqKu{5j+))^asjm0Komt!|#_hPHC4cJy}4|W)b!)f3waPGJe90w=G<>7YY zj^i%l+Hk$NPk05qF5V9BhmXUH@N4l!_+$7>_%?hWew3h0Fe11TLI^ym}55Z#Fp#AMHq0l1)g08K z)Uwn{)h?=a(NHvF8k?3%+d?}{>!5v^p*@2+gFj=#jLI2pGe*>Bs(Y#P)$`S>)Z5j+ zXwWr$HBvOTY1C>w(?n~UYtGZm)ZC|eO>=N2b*9@)?#%p|r)NIag0;-G!nBra9n`v` zHLR_p?XNA?-mQIAdr*g_S~?kl<)ok>rn7t@>QLwZ_z z0eUjM19~lbqxvTLk@{=(tMs245Dgdxi3U3i8V!aFbq(hjt}v`Hd~8H8Vi+YE?J{aM z8ZkC8jy7IzTx;BKqHe-A$u_Ald1^{F^)y{-T4wsd3~R5ZR>5{*_qpM?Mm$)*pu!3>{r{@*}rwLaNs-ab9m@Tah&70-m%f~ zo0GGX#HrG0z}e6_-g%F6CxgleWo%^JaKX5EyR3G(;PTnk$yMrl+Vw9tOScraV{ZMk zO=k0FADZ3cuJ6uqKj7ZwLHCIF_{F2kQ_qv*S?2k|%fO50b=a$qX~s-u9%sJsw(*vD zS9_1JTv;nw4L&Fzf1iAx7GG80NZ%6Q=YINr0>28sA%92z<^Bx;n1G;wZ2_HZZ8n!( z&K?SM4qO@76hsON4=M?I5o{hT39g?5&k3Bfea_RlMsr1TYePT?J7jyvvrv=JrJ?8N zVdsU;E1mZ;%rESP#aPFz zjJX}F87qvfi^Ii5#T||tp6@$<=ltFUjtlY@bSyMkn6>ccBF#mqi!Lr!TFhB|CLR+X z9e*tTD<_0g#u-jvC+tZW;$k|oNzv&?+ihGl)(e%Ze+M=$3t zZ(O0bV$F)~mCTg~Rza&0RyD3RSe?7NH^)Ec=o;dhlr_JtwOL!Z_QSg9b@jR0xjDH# zKl%Suk*Am^$?IJ2wtoN5=%0l@w`{QAuzSPzd|v*o0-J)Og0CC78*gp0-Bi43Y;)q~ zmMuaU<=DF=?p-N$PVbAtC+iP~vcjWK*uygUwn?(*q`*#s{$#!)W2N&1v zHr%~+_t%o7lJ-*X(yBdLdp7JD*~{D8w$E!{<$mq`1^d7JBK+my0sjLvWyWPi2eAh; z4!%4Tb*TAQ#;@gvH4hgY{&qxkr0Zzd(WY|e^73O^$F@|!6|#zf;|q@8JK=MpuF|sd zKoza3;3POHJNfEV{Hc!9L8lwfFwRt-H9EVeno^x#1Jx|68LCaH?W&8dYdsfuuCd;& zzWTh?`6CzfE|gwWySU{N@lx)kvCG+)ha1EVgI5GsdKwouKDiom^+D6Trj~0#*KS<* zyMDEq*?jqi`;7}XU2mSd#kf^_+v#@A9mhM>zd8O^-Qv_zbJzK9-949k^{um8FSU8L zUAgab|M~;=gWK(M+gm##J05l}=bltN(|zYf_=_h!+@9Ber2p}?H}@s^Wl^7QUwOY>|Ahg+fqSp!zv_K0emy#v z_ovFAd)}D7Is2E_Uw4LLhI-#h-j2Q7_+I1vkq-_Z8izxNpMOmII69L5N#oPe&y3H_ zU!uSCjb?tueJ%cG_O1SV(D$cfX=7tzj)XHY3ZM{$6B7Y=cLV@(EC3W_h9J(LoT(=S zn*0u!An>1fa_*iGCjcrDGd>!z8UQ$nsBsPeN?rhrk3|t)0JgW6>(u-^F{_V@4)U!q zUkV6<505uJ&H?|B$vOUqzW-^#cvc7SPx~+Ybqtk~Mf;Ti003-dSV?A0O#mtY000O8 z1OXZV1poj50RR91J^>p51poj50RR91Mgb@Q0{{R30RRF30EcM+1ONa40RR91P5=M^ z00000kN^Mx0RR91kN^Mx0RRG^0ssU600031001JO0{{d700031001Ze0000Y;861b z000SaNLh0L02UXN$RxC0-#u*^Ajdw7j34J}_Oevd#hdSQlD49ssp2v-b2%WxyUA7oNHZ47Wew!w=D~c+J8cbw z=T-mQv`b%Ri=`Tub7vX@GJXDb0~C)2Jq7oWYx1+UGcdLRAU-j{dS5>IzVEEffRM)S z=?T5=^yz@aYA4smj!yyth~e!-$JJgz5X=bjZYSQYCTn6>LgSEW5=r732>^{1Sp<6x z-<)U7O=-TF@j@0FJ?$dTrpA!9mwtZ91wK0JKMkQ zNmBfFKrm^FBIbkRw@3S~ *Detection rules (SIEM)* -> *Create new rule*. The *Create new rule* page displays. -. To create a rule that uses ES|QL, select **ES|QL**, -then write a query. There are two types of ES|QL queries: +Use {ref}/esql.html[{esql}] to query your source events and aggregate event data. Query results are returned in a table with rows and columns. Each row becomes an alert. + +To create an {esql} rule: + +. Go to *Rules* -> *Detection rules (SIEM)* -> *Create new rule*. The *Create new rule* page appears. +. Select **{esql}**, then write a <>. ++ +TIP: Click the help icon (image:images/esql-ref-button.png[Click the ES|QL help icon,20,20]) to open the in-product reference documentation for all {esql} commands and functions. ++ +[NOTE] +==== +To avoid conflicts between the {ref}/esql-commands.html#esql-limit[`LIMIT`] value (which defines how many rows are returned in the results table) and the <> value (which determines the maximum number of alerts a rule can generate per rule execution), make sure the `max_signals` value does not exceed the `LIMIT` value. -.. **Aggregating query**: This is a query that uses the `STATS...BY` grouping commands. Query results cannot be matched with a particular document in Elasticsearch. For example: +If you do not define a `LIMIT` value, the rule automatically uses the `max_signals` value. +==== + -[esql] +. Click *Continue* to <>. + +[float] +[[esql-rule-query-types]] +==== {esql} query types + +{esql} rule queries are loosely categorized into two types: aggregating and non-aggregating. + +[float] +[[esql-agg-query]] +===== Aggregating query + +Aggregating queries use {ref}/esql-functions-operators.html#esql-agg-functions[`STATS...BY`] functions to aggregate source event data. Alerts generated by an {esql} rule with an aggregating query only contain the fields returned by the query. + +Here is an example aggregating query: + +[source,esql] +---- +FROM logs-* +| STATS host_count = COUNT(host.name) BY host.name +| SORT host_count DESC +| WHERE host_count > 20 +---- + +- This query starts by searching logs from indices that match the pattern `logs-*`. +- The query then aggregates the count of events by `host.name`. +- Next, it sorts the result by `host_count` in descending order. +- Then, it filters for events where the `host_count` field appears more than 20 times during the specified rule interval. + +NOTE: Rules that use aggregating queries might create duplicate alerts. This can happen when events that occur in the additional look-back time are aggregated both in the current rule execution and in a previous rule execution. + +[float] +[[esql-non-agg-query]] +===== Non-aggregating query +Non-aggregating queries doesn't use `STATS...BY` functions and doesn't aggregate source event data. Alerts generated by an {esql} rule with a non-aggregating query only contain the fields returned by the query. + +Here is an example non-aggregating query: +[source,esql] ----- -FROM logs* -| STATS count = COUNT(host.name) BY host.name -| SORT host.name +FROM logs-* [metadata _id, _index, _version] +| WHERE event.category == "process" AND event.id == "8a4f500d" +| LIMIT 10 ----- +- This query starts by querying logs from indices that match the pattern `logs-*`. The `[metadata _id, _index, _version]` operator allows <>. +- Next, the query filters events where the `event.category` is a process and the `event.id` is `8a4f500d`. +- Then, it limits the output to the top 10 results. -.. **Non-aggregating query**: This is a query that _does not_ use the `STATS...BY` grouping commands. Each row in the query results can be tracked to a source document in Elasticsearch. -+ -For this type of query, use the operator `[metadata _id, _index, _version]` after defining the index source. This will allow for alerts to be deduplicated and linked to the source documents. For example: -+ -[esql] +[float] +[[esql-non-agg-query-dedupe]] +===== Turn on alert deduplication for rules using non-aggregating queries + +To deduplicate alerts, a query needs access to the `_id`, `_index`, and `_version` metadata fields of the queried source event documents. You can allow this by adding the `[metadata _id, _index, _version]` operator after the `FROM` source command, for example: + +[source,esql] ----- -FROM logs* [metadata _id, _index, _version] -| WHERE event.id == "test" +FROM logs-* [metadata _id, _index, _version] +| WHERE event.category == "process" AND event.id == "8a4f500d" | LIMIT 10 ----- -+ -Ensure, metadata properties `id`, `_index`, `_version` are carried over through pipe operators. -. Click *Continue* to <>. +When those metadata fields are provided, unique alert IDs are created for each alert generated by the query. + +When developing the query, make sure you don't {ref}/esql-commands.html#esql-drop[`DROP`] or filter out the `_id`, `_index`, or `_version` metadata fields. + +Here is an example of a query that fails to deduplicate alerts. It uses the `DROP` command to omit the `_id` property from the results table: + +[source,esql] +----- +FROM logs-* [metadata _id, _index, _version] +| WHERE event.category == "process" AND event.id == "8a4f500d" +| DROP _id +| LIMIT 10 +----- + +Here is another example of an invalid query that uses the `KEEP` command to only return `event.*` fields in the results table: + +[source,esql] +----- +FROM logs-* [metadata _id, _index, _version] +| WHERE event.category == "process" AND event.id == "8a4f500d" +| KEEP event.* +| LIMIT 10 +----- [float] [[rule-ui-basic-params]]