Merge branch '8.x' into redos-54-6190

docs/AI-for-security/llm-performance-matrix.asciidoc

@@ -13,4 +13,5 @@ This table describes the performance of various large language models (LLMs) for
 | *Assistant - Knowledge retrieval* | Good        |  Excellent              | Excellent            | Excellent          | Excellent         | Excellent | Great         | Excellent           | Excellent
 | *Attack Discovery*            | Great           |  Great                  | Excellent            | Poor               | Poor              | Great     | Poor          | Excellent           | Poor 
 |===
-
+
+NOTE: `Excellent` is the best rating, followed by `Great`, then by `Good`, and finally by `Poor`.

docs/cloud-native-security/cloud-native-security-index.asciidoc

@@ -41,6 +41,7 @@ include::cspm.asciidoc[leveloffset=+1]
 include::cspm-get-started-aws.asciidoc[leveloffset=+2]
 include::cspm-get-started-gcp.asciidoc[leveloffset=+2]
 include::cspm-get-started-azure.asciidoc[leveloffset=+2]
+include::cspm-permissions.asciidoc[leveloffset=+2]
 include::cspm-findings.asciidoc[leveloffset=+2]
 include::cspm-benchmark-rules.asciidoc[leveloffset=+2]
 include::cspm-cloud-posture-dashboard.asciidoc[leveloffset=+2]

docs/cloud-native-security/cspm-get-started-aws.asciidoc

@@ -10,17 +10,10 @@ This page explains how to get started monitoring the security posture of your cl
 .Requirements
 [sidebar]
 --
+* Minimum privileges vary depending on whether you need to read, write, or manage CSPM data and integrations. Refer to <<cspm-required-permissions>>.
 * The CSPM integration is available to all {ecloud} users. On-premise deployments require an https://www.elastic.co/pricing[Enterprise subscription].
 * CSPM only works in the `Default` {kib} space. Installing the CSPM integration on a different {kib} space will not work. 
 * CSPM is supported only on AWS, GCP, and Azure commercial cloud platforms, and AWS GovCloud. Other government cloud platforms are not supported. https://github.com/elastic/kibana/issues/new/choose[Click here to request support].
-* `Read` privileges for the following {es} indices:
-** `logs-cloud_security_posture.findings_latest-*`
-** `logs-cloud_security_posture.scores-*`
-* The following {kib} privileges:
-** Security: `Read`
-** Integrations: `Read`
-** Saved Objects Management: `Read` 
-** Fleet: `All`
 * The user who gives the CSPM integration AWS permissions must be an AWS account `admin`.
 --
 

docs/cloud-native-security/cspm-get-started-azure.asciidoc

@@ -10,17 +10,10 @@ This page explains how to get started monitoring the security posture of your cl
 .Requirements
 [sidebar]
 --
+* Minimum privileges vary depending on whether you need to read, write, or manage CSPM data and integrations. Refer to <<cspm-required-permissions>>.
 * The CSPM integration is available to all {ecloud} users. On-premise deployments require an https://www.elastic.co/pricing[Enterprise subscription].
 * CSPM only works in the `Default` {kib} space. Installing the CSPM integration on a different {kib} space will not work. 
 * CSPM is supported only on AWS, GCP, and Azure commercial cloud platforms, and AWS GovCloud. Other government cloud platforms are not supported. https://github.com/elastic/kibana/issues/new/choose[Click here to request support].
-* `Read` privileges for the following {es} indices:
-** `logs-cloud_security_posture.findings_latest-*`
-** `logs-cloud_security_posture.scores-*`
-* The following {kib} privileges:
-** Security: `Read`
-** Integrations: `Read`
-** Saved Objects Management: `Read` 
-** Fleet: `All`
 * The user who gives the CSPM integration permissions in Azure must be an Azure subscription `admin`.
 --
 

docs/cloud-native-security/cspm-get-started-gcp.asciidoc

@@ -10,17 +10,10 @@ This page explains how to get started monitoring the security posture of your GC
 .Requirements
 [sidebar]
 --
+* Minimum privileges vary depending on whether you need to read, write, or manage CSPM data and integrations. Refer to <<cspm-required-permissions>>.
 * The CSPM integration is available to all {ecloud} users. On-premise deployments require an https://www.elastic.co/pricing[Enterprise subscription].
 * CSPM only works in the `Default` {kib} space. Installing the CSPM integration on a different {kib} space will not work. 
 * CSPM is supported only on AWS, GCP, and Azure commercial cloud platforms, and AWS GovCloud. Other government cloud platforms are not supported. https://github.com/elastic/kibana/issues/new/choose[Click here to request support].
-* `Read` privileges for the following {es} indices:
-** `logs-cloud_security_posture.findings_latest-*`
-** `logs-cloud_security_posture.scores-*`
-* The following {kib} privileges:
-** Security: `Read`
-** Integrations: `Read`
-** Saved Objects Management: `Read` 
-** Fleet: `All`
 * The user who gives the CSPM integration GCP permissions must be a GCP project `admin`.
 --
 

docs/cloud-native-security/cspm-permissions.asciidoc

@@ -0,0 +1,61 @@
+[[cspm-required-permissions]]
+= CSPM privilege requirements
+
+This page lists required privileges for {elastic-sec}'s CSPM features. There are three access levels: read, write, and manage. Each access level and its requirements are described below.
+
+[discrete]
+== Read
+
+Users with these minimum permissions can view data on the **Findings** page and the Cloud Posture dashboard.
+
+[discrete]
+=== {es} index privileges
+`Read` privileges for the following {es} indices:
+
+* `logs-cloud_security_posture.findings_latest-*`
+* `logs-cloud_security_posture.scores-*`
+
+[discrete]
+=== {kib} privileges
+
+* `Security: Read`
+
+
+[discrete]
+== Write
+
+Users with these minimum permissions can view data on the **Findings** page and the Cloud Posture dashboard, create detection rules from the findings details flyout, and enable or disable benchmark rules.
+
+[discrete]
+=== {es} index privileges
+`Read` privileges for the following {es} indices:
+
+* `logs-cloud_security_posture.findings_latest-*`
+* `logs-cloud_security_posture.scores-*`
+
+[discrete]
+=== {kib} privileges
+
+* `Security: All`
+
+
+[discrete]
+== Manage
+
+Users with these minimum permissions can view data on the **Findings** page and the Cloud Posture dashboard, create detection rules from the findings details flyout, enable or disable benchmark rules, and install, update, or uninstall CSPM integrations and assets.
+
+[discrete]
+=== {es} index privileges
+`Read` privileges for the following {es} indices:
+
+* `logs-cloud_security_posture.findings_latest-*`
+* `logs-cloud_security_posture.scores-*`
+
+[discrete]
+=== {kib} privileges
+
+* `Security: All`
+* `Spaces: All`
+* `Fleet: All`
+* `Integrations: All`
+

docs/cloud-native-security/environment-variable-capture.asciidoc

@@ -28,9 +28,6 @@ To set up environment variable capture for an {agent} policy:
 . Enter the names of env vars you want to capture, separated by commas. For example: `PATH,USER`
 . Click *Save*.
 
-[role="screenshot"]
-image::images/env-var-capture.png[The "linux.advanced.capture_env_vars" advanced agent policy setting]
-
 [[find-cap-env-vars]]
 [discrete]
 == Find captured environment variables

docs/detections/detection-engine-intro.asciidoc

@@ -167,3 +167,9 @@ and you should contact your {kib} administrator.
 NOTE: For *self-managed* {stack} deployments only, this message may be
 displayed when the <<detections-permissions, `xpack.security.enabled`>>
 setting is not enabled in the `elasticsearch.yml` file. For more information, refer to <<detections-on-prem-requirements>>.
+
+[discrete]
+[[detections-logsdb-index-mode]]
+== Using logsdb index mode 
+
+To learn how your rules and alerts are affected by using the {ref}/logs-data-stream.html[logsdb index mode], refer to <<detections-logsdb-index-mode-impact>>. 

docs/detections/detections-index.asciidoc

@@ -2,6 +2,8 @@ include::detection-engine-intro.asciidoc[]
 
 include::detections-req.asciidoc[leveloffset=+1]
 
+include::detections-logsdb-impact.asciidoc[leveloffset=+1]
+
 include::about-rules.asciidoc[]
 
 

docs/detections/detections-logsdb-impact.asciidoc

@@ -0,0 +1,65 @@
+[[detections-logsdb-index-mode-impact]]
+= Using logsdb index mode with {elastic-sec}
+
+NOTE: To use the {ref}/mapping-source-field.html#synthetic-source[synthetic `_source`] feature, you must have the appropriate subscription. Refer to the subscription page for https://www.elastic.co/subscriptions/cloud[Elastic Cloud] and {subscriptions}[Elastic Stack/self-managed] for the breakdown of available features and their associated subscription tiers.
+
+This topic explains the impact of using logsdb index mode with {elastic-sec}.
+
+With logsdb index mode, the original `_source` field is not stored in the index but can be reconstructed using {ref}/mapping-source-field.html#synthetic-source[synthetic `_source`].
+
+When the `_source` is reconstructed, {ref}/mapping-source-field.html#synthetic-source-modifications[modifications] are possible. Therefore, there could be a mismatch between users' expectations and how fields are formatted.
+
+Continue reading to find out how this affects specific {elastic-sec} components. 
+
+[discrete]
+[[logsdb-alerts]]
+== Alerts
+
+When alerts are generated, the `_source` event is copied into the alert to retain the original data. When the logsdb index mode is applied, the `_source` event stored in the alert is reconstructed using synthetic `_source`.
+
+If you're switching to use logsdb index mode, the `_source` field stored in the alert might look different in certain situations:
+
+* {ref}/mapping-source-field.html#synthetic-source-modifications-leaf-arrays[Arrays can be reconstructed differently or deduplicated]
+* {ref}/mapping-source-field.html#synthetic-source-modifications-field-names[Field names] 
+* `geo_point` data fields (refer to {ref}/mapping-source-field.html#synthetic-source-modifications-ranges[Representation of ranges] and {ref}/mapping-source-field.html#synthetic-source-precision-loss-for-point-types[Reduced precision of `geo_point` values] for more information)
+
+Alerts generated by the following rule types could be affected:
+
+* Custom query
+* Event correlation (non-sequence only)
+* Non-aggregate rule types (for example, {esql} rules that use non-aggregating queries)
+
+Alerts that are generated by threshold, {ml}, and event correlation sequence rules are not affected since they do not contain copies of the original source.
+
+[discrete]
+[[logsdb-rule-actions]]
+== Rule actions
+
+While we do not recommend using `_source` for actions, in cases where the action relies on the `_source`, the same limitations and changes apply.
+
+If you send alert notifications by enabling {kibana-ref}/alerting-getting-started.html#alerting-concepts-actions[actions] to the external systems that have workflows or automations based on fields formatted from the original source, they may be affected. In particular, this can happen when the fields used are arrays of objects.
+
+We recommend checking and adjusting the rule actions using `_source` before switching to logsdb index mode.
+
+[discrete]
+[[logsdb-runtime-fields]]
+== Runtime fields
+
+Runtime fields that reference `_source` may be affected. Some runtime fields might not work and need to be adjusted. For example, if an event was indexed with the value of `agent.name` in the dot-notation form, it will be returned in the nested form and might not work. 
+
+The following is an example of accessing `_source` that works with the logsdb index mode enabled:
+
+[source,console]
+----
+"source": """  emit(params._source.agent.name + "_____" + doc['agent.name'].value ); """ 
+"source": """  emit(params._source['agent']['name'] + "_____" + doc['agent.name'].value );  """
+"source": """  emit(field('agent.name').get(null) + "_____" + doc['agent.name'].value ); """
+"source": """  emit($('agent.name', null) + "_____" + doc['agent.name'].value ); """
+----
+
+The following will not work with synthetic source (logsdb index mode enabled):
+
+[source,console]
+----
+"source": """  emit(params._source['agent.name'] + "_____" + doc['agent.name'].value );  """
+----

docs/release-notes.asciidoc

@@ -3,6 +3,7 @@
 
 This section summarizes the changes in each release.
 
+* <<release-notes-8.17.0, {elastic-sec} version 8.17.0>>
 * <<release-notes-8.16.1, {elastic-sec} version 8.16.1>>
 * <<release-notes-8.16.0, {elastic-sec} version 8.16.0>>
 * <<release-notes-8.15.5, {elastic-sec} version 8.15.5>>
@@ -69,6 +70,7 @@ This section summarizes the changes in each release.
 * <<release-notes-8.0.0, {elastic-sec} version 8.0.0>>
 * <<release-notes-8.0.0-rc2, {elastic-sec} version 8.0.0-rc2>>
 
+include::release-notes/8.17.asciidoc[]
 include::release-notes/8.16.asciidoc[]
 include::release-notes/8.15.asciidoc[]
 include::release-notes/8.14.asciidoc[]

docs/release-notes/8.16.asciidoc

@@ -9,6 +9,63 @@
 [[known-issue-8.16.1]]
 ==== Known issues
 
+// tag::known-issue[201820]
+[discrete]
+.The **Exceptions** tab won't properly load if exceptions contain comments with newline characters (`\n`)  
+[%collapsible]
+====
+*Details* +
+On December 5, 2024, it was discovered that the **Exceptions** tab won't load properly if any exceptions contain comments with newline characters (`\n`). This issue occurs when you upgrade to 8.16.0 or later ({kibana-issue}201820[#201820]).
+
+*Workaround* + 
+
+For custom rules:
+
+. From the **Rules** page, <<import-export-rules-ui,export>> the rule or rules with the affected exception lists. 
+. Modify the `.ndjson` file so `comments` no longer contain newline characters.
+. Return to the **Rules** page and <<import-export-rules-ui,re-import>> the rules. Make sure to select the **Overwrite existing exception lists with conflicting "list_id"** option.
+
+For prebuilt rules: 
+
+NOTE: If you only need to fix exceptions for the Elastic Endpoint rule, you can export and re-import its exception list from the <<shared-exception-lists,**Shared Exception Lists**>> page.
+
+. Follow these steps to fetch the affected exception list ID or IDs that are associated with the rule: 
+.. Find the affected rule's ID (`id`). From the **Rules** page, open the details of a rule, go to the page URL, and copy the string at the end. For example, in the URL http://host.name/app/security/rules/id/167a5f6f-2148-4792-8226-b5e7a58ef46e, the string at the end (`167a5f6f-2148-4792-8226-b5e7a58ef46e`) is the `id`.
+.. Specify the `id` when fetching the rule's details using the {api-kibana}/operation/operation-readrule[Retrieve a detection rule API]. Here is an example request that includes the `id`:
++
+[source,console]
+----
+curl -H 'Authorization: ApiKey API_KEY_HERE' -H 'kbn-xsrf: true' -H 'elastic-api-version: 2023-10-31' KIBANA_URL/api/detection_engine/rules?id=167a5f6f-2148-4792-8226-b5e7a58ef46e
+----
++
+.. The JSON response contains the `id`, `list_id`, and `namespace_type` values within the `exceptions_list` key (as shown below). You need these values when using the Exception list API to retrieve the affected exception list. 
++
+[source,console]
+----
+{
+  "id": "167a5f6f-2148-4792-8226-b5e7a58ef46e",
+  "exceptions_list": [
+    {
+      "id": "490525a2-eb66-4320-95b5-88bdd1302dc4",
+      "list_id": "f75aae6f-0229-413f-881d-81cb3abfbe2d",
+      "namespace_type": "single"
+    }
+  ]
+}
+----
++
+. Use the export exceptions API to retrieve the affected exception list. Insert the values for the `id`, `list_id`, and `namespace_type` parameters into the following API call:
++
+[source,console]
+----
+curl -XPOST -H 'Authorization: ApiKey API_KEY_HERE' -H 'kbn-xsrf: true' -H 'elastic-api-version: 2023-10-31' 'KIBANA_URL/api/exception_lists/_export?list_id=f75aae6f-0229-413f-881d-81cb3abfbe2d&id=490525a2-eb66-4320-95b5-88bdd1302dc4&namespace_type=single' -o list.ndjson
+----
++
+. Modify the exception list's `.ndjson` file to ensure `comments[].comment` values don't contain newline characters (`\n`).
+. Re-import the modified exception list using **Import exception lists** option on the <<shared-exception-lists,**Shared Exception Lists**>> page. The import will initially fail because the exception list already exists, and an option to overwrite the existing list will appear. Select the option, then resubmit the request to import the corrected exception list.
+====
+// end::known-issue[201820]
+
 // tag::known-issue[]
 [discrete]
 .Duplicate alerts can be produced from manually running threshold rules 
@@ -50,6 +107,63 @@ On November 12, 2024, it was discovered that manually running a custom query rul
 [[known-issue-8.16.0]]
 ==== Known issues
 
+// tag::known-issue[201820]
+[discrete]
+.The **Exceptions** tab won't properly load if exceptions contain comments with newline characters (`\n`)  
+[%collapsible]
+====
+*Details* +
+On December 5, 2024, it was discovered that the **Exceptions** tab won't load properly if any exceptions contain comments with newline characters (`\n`). This issue occurs when you upgrade to 8.16.0 or later ({kibana-issue}201820[#201820]). 
+
+*Workaround* + 
+
+For custom rules:
+
+. From the **Rules** page, <<import-export-rules-ui,export>> the rule or rules with the affected exception lists. 
+. Modify the `.ndjson` file so `comments` no longer contain newline characters.
+. Return to the **Rules** page and <<import-export-rules-ui,re-import>> the rules. Make sure to select the **Overwrite existing exception lists with conflicting "list_id"** option.
+
+For prebuilt rules: 
+
+NOTE: If you only need to fix exceptions for the Elastic Endpoint rule, you can export and re-import its exception list from the <<shared-exception-lists,**Shared Exception Lists**>> page.
+
+. Follow these steps to fetch the affected exception list ID or IDs that are associated with the rule: 
+.. Find the affected rule's ID (`id`). From the **Rules** page, open the details of a rule, go to the page URL, and copy the string at the end. For example, in the URL http://host.name/app/security/rules/id/167a5f6f-2148-4792-8226-b5e7a58ef46e, the string at the end (`167a5f6f-2148-4792-8226-b5e7a58ef46e`) is the `id`.
+.. Specify the `id` when fetching the rule's details using the {api-kibana}/operation/operation-readrule[Retrieve a detection rule API]. Here is an example request that includes the `id`:
++
+[source,console]
+----
+curl -H 'Authorization: ApiKey API_KEY_HERE' -H 'kbn-xsrf: true' -H 'elastic-api-version: 2023-10-31' KIBANA_URL/api/detection_engine/rules?id=167a5f6f-2148-4792-8226-b5e7a58ef46e
+----
++
+.. The JSON response contains the `id`, `list_id`, and `namespace_type` values within the `exceptions_list` key (as shown below). You need these values when using the Exception list API to retrieve the affected exception list. 
++
+[source,console]
+----
+{
+  "id": "167a5f6f-2148-4792-8226-b5e7a58ef46e",
+  "exceptions_list": [
+    {
+      "id": "490525a2-eb66-4320-95b5-88bdd1302dc4",
+      "list_id": "f75aae6f-0229-413f-881d-81cb3abfbe2d",
+      "namespace_type": "single"
+    }
+  ]
+}
+----
++
+. Use the export exceptions API to retrieve the affected exception list. Insert the values for the `id`, `list_id`, and `namespace_type` parameters into the following API call:
++
+[source,console]
+----
+curl -XPOST -H 'Authorization: ApiKey API_KEY_HERE' -H 'kbn-xsrf: true' -H 'elastic-api-version: 2023-10-31' 'KIBANA_URL/api/exception_lists/_export?list_id=f75aae6f-0229-413f-881d-81cb3abfbe2d&id=490525a2-eb66-4320-95b5-88bdd1302dc4&namespace_type=single' -o list.ndjson
+----
++
+. Modify the exception list's `.ndjson` file to ensure `comments[].comment` values don't contain newline characters (`\n`).
+. Re-import the modified exception list using **Import exception lists** option on the <<shared-exception-lists,**Shared Exception Lists**>> page. The import will initially fail because the exception list already exists, and an option to overwrite the existing list will appear. Select the option, then resubmit the request to import the corrected exception list.
+====
+// end::known-issue[201820]
+
 // tag::known-issue[]
 [discrete]
 .Attempting to edit an Elastic AI Assistant Knowledge Base index results in an error