diff --git a/docs/advanced-entity-analytics/entity-risk-scoring.asciidoc b/docs/advanced-entity-analytics/entity-risk-scoring.asciidoc index 6eb5042269..932e6b07fb 100644 --- a/docs/advanced-entity-analytics/entity-risk-scoring.asciidoc +++ b/docs/advanced-entity-analytics/entity-risk-scoring.asciidoc @@ -1,6 +1,11 @@ [[entity-risk-scoring]] = Entity risk scoring +[sidebar] +-- +If you’ve installed the original user and host risk score modules, refer to {security-guide-all}/8.11/host-risk-score.html[Host risk score] and {security-guide-all}/8.11/user-risk-score.html[User risk score]. +-- + beta::[] Entity risk scoring is an advanced {elastic-sec} analytics feature that helps security analysts detect changes in an entity's risk posture, hunt for new threats, and prioritize incident response. diff --git a/docs/advanced-entity-analytics/turn-on-risk-engine.asciidoc b/docs/advanced-entity-analytics/turn-on-risk-engine.asciidoc index 7cbd3c1e40..5391c71fad 100644 --- a/docs/advanced-entity-analytics/turn-on-risk-engine.asciidoc +++ b/docs/advanced-entity-analytics/turn-on-risk-engine.asciidoc @@ -23,7 +23,7 @@ image::images/preview-risky-entities.png[Preview of risky entities] [NOTE] ====== * To view risk score data, you must have alerts generated in your environment. -* If you previously installed the original <> and <> modules, and you're upgrading to {stack} version 8.11 or newer, refer to <>. +* If you previously installed the original user and host risk score modules, and you're upgrading to {stack} version 8.11 or newer, refer to <>. ====== If you're installing the risk scoring engine for the first time: diff --git a/docs/detections/alerts-view-details.asciidoc b/docs/detections/alerts-view-details.asciidoc index 5910de7e6f..f66fb14184 100644 --- a/docs/detections/alerts-view-details.asciidoc +++ b/docs/detections/alerts-view-details.asciidoc @@ -144,8 +144,6 @@ image::images/insights-section-rp.png[Insights section of the Overview tab, 65%] The Entities overview provides high-level details about the user and host that are related to the alert. Host and user risk classifications are also available with a https://www.elastic.co/pricing[Platinum subscription] or higher. -NOTE: <> and <> risk scores are technical preview features. - [role="screenshot"] image::images/entities-overview.png[Overview of the entity details section in the right panel, 60%] diff --git a/docs/experimental-features/beaconing-detection.asciidoc b/docs/experimental-features/beaconing-detection.asciidoc deleted file mode 100644 index 6b2301e5e5..0000000000 --- a/docs/experimental-features/beaconing-detection.asciidoc +++ /dev/null @@ -1,54 +0,0 @@ -[[network-beaconing-framework]] -== Network Beaconing - -This feature provides an early warning system for command and control beaconing activity. It monitors network traffic for indicators of compromise and provides analytics to add context to alerts and aid your threat hunting. - -[discrete] -=== Deploy the package - -To deploy the network beaconing framework in your environment, follow {integrations-docs}/beaconing#installation[these steps]. - -The installation package includes dashboards for monitoring beaconing activity in your environment. You can review signals using a Lens dashboard called Network beaconing. - -NOTE: If you want to modify any of the package components, you can install the package manually by following https://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/beaconing.md[these steps]. - -[role="screenshot"] -image::images/beaconing-detection-1.png[] - -[discrete] -=== Feature details - -This feature uses a {ref}/transforms.html[transform] to categorize network data by host and process name, then runs scripted metric aggregations on the host-process name pairs. For a given time window, the scripted metric aggregation checks each pair for the following: - -* Signals repeating at regular intervals, accounting for minor variations in those intervals. -* Low variation of bytes sent from source to destination. -* Low variation of bytes sent from destination to source. - -The transform, which runs every hour, also filters out common, known applications and IPs to reduce false positives. The transform outputs information about the detection, process, and host indicators, for example: - -[role="screenshot"] -image::images/beaconing-detection-2.png[] -The values highlighted above are typical of beaconing behavior and can help with your investigation. - -[discrete] -=== Further customizations - -Advanced users can also tune the scripted metric aggregation's parameters, such as jitter percentage or time window. To overwrite the default parameters: delete the transform, change the parameters, and restart the transform. The configurable parameters are: - -* `number_buckets_in_range`: The number of time buckets into which the time window is split. Using more buckets improves estimates for various statistics, but also increases resource usage. -* `time_bucket_length`: The length of each time bucket. A higher value indicates a longer time window. Set this to a higher value to check for very low-frequency beacons. -* `number_destination_ips`: The number of destination IPs to collect in results. Setting this to a higher value increases resource usage. -* `max_beaconing_bytes_cov`: The maximum coefficient of variation in the payload bytes for the low source and destination bytes variance test. Higher values increase the chance of flagging traffic as beaconing, increasing https://en.wikipedia.org/wiki/Precision_and_recall[recall] while reducing https://en.wikipedia.org/wiki/Precision_and_recall[precision]. -* `max_beaconing_count_rv`: The maximum relative variance in the bucket counts for the high-frequency beacon test. As with `max_beaconing_bytes_cov`, tuning this parameter involves a tradeoff between recall and precision. -* `truncate_at`: The lower and upper fraction of bucket values discarded when computing `max_beaconing_bytes_cov` and `max_beaconing_count_rv`. This allows you to ignore occasional changes in traffic patterns. However, if you retain too small a fraction of the data, these tests will be unreliable. -* `min_beaconing_count_autocovariance`: The minimum autocorrelation of the signal for the low-frequency beacon test. Lowering this value generally increases recall for malicious command and control beacons, while reducing precision. -* `max_jitter`: The maximum amount of https://en.wikipedia.org/wiki/Jitter[jitter] assumed to be possible for a periodic beacon, as a fraction of its period. - -You can also make changes to the transform query. The default query looks for beaconing activity over a 6-hour time range, but you can change it. - -Beaconing is not used exclusively by malware. Many legitimate, benign processes also exhibit beacon-like activity. To reduce false positives, default filters in the transform query exclude known beaconing processes and IPs that fall into two groups: - -* The source IP is local and the destination is remote. -* The destination IP is in a block of known Microsoft IP addresses. - -You can create additional filters to meet the needs of your environment. diff --git a/docs/experimental-features/experimental-features-intro.asciidoc b/docs/experimental-features/experimental-features-intro.asciidoc deleted file mode 100644 index c4612ae097..0000000000 --- a/docs/experimental-features/experimental-features-intro.asciidoc +++ /dev/null @@ -1,9 +0,0 @@ -[[sec-experimental-intro]] -= Technical preview - -The features in this section are experimental and may be changed or removed completely in future releases. Elastic will make a best effort to fix any issues, but experimental features are not supported to the same level as generally available (GA) features. - - -include::host-risk-score.asciidoc[] -include::user-risk-score.asciidoc[] - diff --git a/docs/experimental-features/host-risk-score.asciidoc b/docs/experimental-features/host-risk-score.asciidoc deleted file mode 100644 index 73f1f30c2b..0000000000 --- a/docs/experimental-features/host-risk-score.asciidoc +++ /dev/null @@ -1,327 +0,0 @@ -[[host-risk-score]] -== Host risk score - -NOTE: This page refers to the original user and host risk score modules. If you have the original modules installed, and you're running {stack} version 8.11 or newer, you can <>. -For information about the latest risk engine, refer to <>. - -NOTE: This feature is available for {stack} versions 7.16.0 and newer and requires a https://www.elastic.co/pricing[Platinum subscription] or higher. - -The host risk score feature highlights risky hosts from within your environment. It utilizes a transform with a scripted metric aggregation to calculate host risk scores based on alerts that were generated within the past five days. The transform runs hourly to update the score as new alerts are generated. - -Each rule's contribution to the host risk score is based on the rule's risk score (`signal.rule.risk_score`) and a time decay factor to reduce the impact of stale alerts. The risk score is calculated using a weighted sum where rules with higher time-corrected risk scores also have higher weights. Each host risk score is normalized on a scale of 0 to 100. - -Specific host attributes can boost the final risk score. For example, alert activity on a server poses a greater risk than that on a laptop. Therefore, the host risk score is 1.5 times higher if the host is a server. This boosted score is finalized after calculating the weighted sum of the time-corrected risks. - -The following table shows how risk levels are applied to a host, based on the normalized risk score: - -[width="100%",options="header"] -|============================================== -|Risk level |Host risk score - -|Unknown |< 20 -|Low |20-40 -|Moderate |40-70 -|High | 70-90 -|Critical | > 90 - - -|============================================== - -[[enable-host-risk-score]] -[discrete] -=== Enable host risk score - -NOTE: To enable the host risk score feature, you must have alerts in your environment. If you previously enabled host risk score and are upgrading the {stack} to 8.5–8.10, refer to <>. - -You can enable host risk score from the following places in the {security-app}: - -* The Entity Analytics dashboard -* The *Host risk* tab on the Hosts page -* The *Host risk* tab on a host's details page - -Or, in {kib}, you can enable host risk score in Console. - -To enable host risk score from the Entity Analytics dashboard: - -. In the {security-app}, go to *Dashboards* -> *Entity Analytics*. -. In the Host Risk Scores section, click *Enable* to install the module. - -To enable host risk score from the Hosts page: - -. Go to *Explore* -> *Hosts*. -. Select the *Host risk* tab, then click *Enable* to install the module. - -[role="screenshot"] -image::images/enable-hrs.png[Enable Host Risk Score button] - -To enable host risk score from a host's details page: - -. Go to *Explore* -> *Hosts*. -. Select the *All hosts* tab, then click a host name. -. On the details page, scroll down to the data tables, then select the *Host risk* tab. -. Click *Enable* to install the module. - -To enable host risk score from Console in {kib}, open a browser window and enter the following URL: - -[source,console] ----------------------------------- -{KibanaURL}/s/{spaceID}/app/dev_tools#/console?load_from={KibanaURL}/s/{spaceID}/internal/risk_score/prebuilt_content/dev_tool/enable_host_risk_score ----------------------------------- - -NOTE: If there's existing content in Console, scroll to the bottom to find the output loaded. - -TIP: If you receive an error message during the installation process, delete the host risk score module manually, then re-enable it. Refer to <> for more information. - -[[upgrade-host-risk-score]] -[discrete] -=== Upgrade host risk score - -If you previously enabled host risk score and you're upgrading to {stack} version 8.11 or newer, you can <>. - -Before upgrading, note the following: - -* Since older data is not preserved, previous host risk scores will be deleted, and new scores will be created. However, if you want to retain old host risk scores, you can reindex them _before_ upgrading. To learn how, refer to {ref}/docs-reindex.html[Reindex API]. New data will be stored in the `ml_host_risk_score_` and `ml_host_risk_score_latest_` indices. - -* You must edit your {cloud}/ec-manage-kibana-settings.html#ec-manage-kibana-settings[{kib} user settings] and remove the `xpack.securitySolution.enableExperimental:['riskyHostsEnabled']` feature flag. - -After this is done, you can proceed with upgrading the host risk score feature from any of the following places in the {security-app}: - -* The Entity Analytics dashboard -* The *Host risk* tab on the Hosts page -* The *Host risk* tab on a host's details page - -NOTE: After you enable or upgrade host risk score, you might get a message that says, "No host risk score data available to display." To verify that the transform that installs the host risk score module is picking up data, refer to <>. - -TIP: If you receive an error message during the upgrade process, delete the host risk score module manually, and then re-enable it. Refer to <> for more information. - -[[analyze-host-risk-score]] -[discrete] -=== Analyze host risk score data - -It is recommended you analyze hosts with the highest risk scores first -- those in the `Critical` and `Moderate` categories. Host risk score data appears in the following places in the {security-app}: - -The `host.risk.calculated_level` column in the Alerts table: - -[role="screenshot"] -image::images/hrs-alerts-table.png[Host risk score in the Alerts table] - -The *Insights* -> *Entities* section on the *Overview* tab within the alert details flyout: - -[role="screenshot"] -image::images/score-in-flyout.png[Host risk score in alert details flyout,65%] - -The *Host risk classification* column in the All hosts table on the Hosts page: - -[role="screenshot"] -image::images/hrs-all-hosts.png[Host risk score on the Hosts page] - -The *Host risk* tab on the Hosts page: - -[role="screenshot"] -image::advanced-entity-analytics/images/hosts-hr-data.png[Host risk score on the Hosts page] - -The Overview section on the host details page: - -[role="screenshot"] -image::images/hrs-overview-section.png[Host risk score in Overview section] - -The *Host risk* tab on the host details page: - -[role="screenshot"] -image::images/hosts-by-risk-details-page.png[Host risk score on the Hosts risk tab] - -You can also visualize host risk score data using prebuilt dashboards that are automatically imported when the feature is enabled. - -To access the dashboards: - -. In {kib}, go to *Analytics* -> *Dashboard*, then search for `risk score`. -. Select *Drilldown of Host Risk Score* to analyze the risk components of a host, or *Current Risk Score for Hosts* to display a list of current risky hosts in your environment. - -[role="screenshot"] -image::images/select-hrs-dashboard.png[Select host risk score dashboard] - -In this example, we'll explore the *Drilldown of Host Risk Score* dashboard. - -[role="screenshot"] -image::images/full-dashboard.png[Shows dashboard] - -Use the histogram to track how the risk score for a particular host has changed over time. To specify a date range, use the date and time picker, or drag and select a time range within the histogram. - -[role="screenshot"] -image::images/histogram.png[] - -To go to the host's details page, click any host's corresponding bar in the histogram, then select *Go to Host View*. - -[role="screenshot"] -image::images/go-to-host-view.png[] - -The histogram shows historical changes in a particular host's risk score(s). To specify a date range, use the date and time picker, or drag and select a time range within the histogram. - -[role="screenshot"] -image::images/data-tables.png[] - -[[troubleshoot-host-risk-score]] -[discrete] -=== Troubleshooting - -During the installation or upgrade process, you may receive the following error messages: - -* `Saved object already exists` -* `Transform already exists` -* `Ingest pipeline already exists` - -In this case, we recommend that you manually delete the host risk score module, then re-enable it. To manually delete the module: - -. Delete the host risk score saved objects: -.. From the {kib} main menu, go to **Stack Management** -> **Kibana** -> **Saved Objects**. -.. Delete the saved objects that have the `Host Risk Score - ` tag. -+ -[role="screenshot"] -image::images/delete-hrs-saved-objects.png[Delete host risk score saved objects] -.. Delete the `Host Risk Score - ` tag. -+ -[role="screenshot"] -image::images/delete-hrs-tag.png[Delete host risk score tag] -. Stop and delete the host risk score transforms. You can do this using the {kib} UI or the {ref}/stop-transform.html[Stop transform API] and {ref}/delete-transform.html[Delete transform API]. -** To delete the host risk score transforms using the {kib} UI: -.. From the {kib} main menu, go to **Stack Management** -> **Data** -> **Transforms**. -.. Stop the following transforms, then delete them: -*** `ml_hostriskscore_latest_transform_` -*** `ml_hostriskscore_pivot_transform_` -** To delete the host risk score transforms using the API, run the following commands in Console: -.. Stop and delete the latest transform: -+ -[source,console] ----------------------------------- -POST _transform/ml_hostriskscore_latest_transform_/_stop -DELETE _transform/ml_hostriskscore_latest_transform_ ----------------------------------- -.. Stop and delete the pivot transform: -+ -[source,console] ----------------------------------- -POST _transform/ml_hostriskscore_pivot_transform_/_stop -DELETE _transform/ml_hostriskscore_pivot_transform_ ----------------------------------- -. Delete the host risk score ingest pipeline. You can do this using the {kib} UI or the {ref}/delete-pipeline-api.html[Delete pipeline API]. -** To delete the host risk score ingest pipeline using the {kib} UI: -.. From the {kib} main menu, go to **Stack Management** -> **Ingest** -> **Ingest Pipelines**. -.. Delete the `ml_hostriskscore_ingest_pipeline_` ingest pipeline. -** To delete the host risk score ingest pipeline using the Delete pipeline API, run the following command in Console: -+ -[source,console] ----------------------------------- -DELETE /_ingest/pipeline/ml_hostriskscore_ingest_pipeline_ ----------------------------------- -. Delete the stored host risk score scripts using the {ref}/delete-stored-script-api.html[Delete stored script API]. In Console, run the following commands: -+ -[source,console] ----------------------------------- -DELETE _scripts/ml_hostriskscore_levels_script_ -DELETE _scripts/ml_hostriskscore_init_script_ -DELETE _scripts/ml_hostriskscore_map_script_ -DELETE _scripts/ml_hostriskscore_reduce_script_ ----------------------------------- - -After manually deleting the host risk score saved objects, transforms, ingest pipeline, and stored scripts, follow the steps to <>. - -[[verify-host-risk-score]] -=== Verify that host risk score data installed successfully (Optional) - -After you enable or upgrade host risk score, the following message may appear: - -[role="screenshot"] -image::images/restart-hrs.png[Restart host risk score] - -If so, click *Restart* and allow at least an hour for the data to be generated. If data still doesn't appear, verify that host risk score data has been generated: - -In {kib}, run the following commands in Console to query the `ml_host_risk_score_` index: - -[source,console] ----------------------------------- -GET ml_host_risk_score_/_search ----------------------------------- - -If no data returns, you'll need to check if the alerts index (.`alerts-security.alerts-`) had alert data when `ml_hostriskscore_pivot_transform_` was started. - -Example: - -[source,console] ----------------------------------- -GET transform/ml_hostriskscore_pivot_transform_/_stats?human=true ----------------------------------- - -Here's an example response: - -[source,console] ----------------------------------- -{ - "count": 1, - "transforms": [ - { - "id": "ml_hostriskscore_pivot_transform_", - "state": "started", - "node": { - "id": "H1tlwfTyRkWls-C0sarmHw", - "name": "instance-0000000000", - "ephemeral_id": "SBqlp5ywRuuop2gtcdCljA", - "transport_address": "10.43.255.164:19635", - "attributes": {} - }, - "stats": { - "pages_processed": 29, - "documents_processed": 11805, - "documents_indexed": 8, - "documents_deleted": 0, - "trigger_count": 9, - "index_time_in_ms": 52, - "index_total": 7, - "index_failures": 0, - "search_time_in_ms": 201, - "search_total": 29, - "search_failures": 0, - "processing_time_in_ms": 14, - "processing_total": 29, - "delete_time_in_ms": 0, - "exponential_avg_checkpoint_duration_ms": 59.02353261024906, - "exponential_avg_documents_indexed": 0.8762710605864747, - "exponential_avg_documents_processed": 1664.7724779548555 - }, - "checkpointing": { - "last": { - "checkpoint": 8, - "timestamp": "2022-10-17T14:49:50.315Z", - "timestamp_millis": 1666018190315, - "time_upper_bound": "2022-10-17T14:47:50.315Z", - "time_upper_bound_millis": 1666018070315 - }, - "operations_behind": 380, - "changes_last_detected_at_string": "2022-10-17T14:49:50.113Z", - "changes_last_detected_at": 1666018190113, - "last_search_time_string": "2022-10-17T14:49:50.113Z", - "last_search_time": 1666018190113 - } - } - ] -} ----------------------------------- - -Take note of the value from `time_upper_bound_millis` and enter it as a range query for the alerts index. - -Example: - -[source,console] ----------------------------------- -GET .alerts-security.alerts-/_search -{ - "query": { - "range": { - "@timestamp": { - "lt": 1666018070315 - } - } - } -} ----------------------------------- - -If there's no response, verify that relevant <> are running and that alert data is being generated. If there is a response, click *Restart* and allow an hour for the host risk data to appear. \ No newline at end of file diff --git a/docs/experimental-features/images/beaconing-detection-1.png b/docs/experimental-features/images/beaconing-detection-1.png deleted file mode 100644 index a7cc7663f1..0000000000 Binary files a/docs/experimental-features/images/beaconing-detection-1.png and /dev/null differ diff --git a/docs/experimental-features/images/beaconing-detection-2.png b/docs/experimental-features/images/beaconing-detection-2.png deleted file mode 100644 index 1c92671c94..0000000000 Binary files a/docs/experimental-features/images/beaconing-detection-2.png and /dev/null differ diff --git a/docs/experimental-features/images/dashboard.gif b/docs/experimental-features/images/dashboard.gif deleted file mode 100644 index b0f2ca830b..0000000000 Binary files a/docs/experimental-features/images/dashboard.gif and /dev/null differ diff --git a/docs/experimental-features/images/data-tables.png b/docs/experimental-features/images/data-tables.png deleted file mode 100644 index 89acfcc040..0000000000 Binary files a/docs/experimental-features/images/data-tables.png and /dev/null differ diff --git a/docs/experimental-features/images/delete-hrs-saved-objects.png b/docs/experimental-features/images/delete-hrs-saved-objects.png deleted file mode 100644 index c4c05024ad..0000000000 Binary files a/docs/experimental-features/images/delete-hrs-saved-objects.png and /dev/null differ diff --git a/docs/experimental-features/images/delete-hrs-tag.png b/docs/experimental-features/images/delete-hrs-tag.png deleted file mode 100644 index f35ad916d7..0000000000 Binary files a/docs/experimental-features/images/delete-hrs-tag.png and /dev/null differ diff --git a/docs/experimental-features/images/delete-urs-saved-objects.png b/docs/experimental-features/images/delete-urs-saved-objects.png deleted file mode 100644 index 4e41bb8590..0000000000 Binary files a/docs/experimental-features/images/delete-urs-saved-objects.png and /dev/null differ diff --git a/docs/experimental-features/images/delete-urs-tag.png b/docs/experimental-features/images/delete-urs-tag.png deleted file mode 100644 index 030e1e357b..0000000000 Binary files a/docs/experimental-features/images/delete-urs-tag.png and /dev/null differ diff --git a/docs/experimental-features/images/enable-hrs-details-pg.gif b/docs/experimental-features/images/enable-hrs-details-pg.gif deleted file mode 100644 index 14d7898159..0000000000 Binary files a/docs/experimental-features/images/enable-hrs-details-pg.gif and /dev/null differ diff --git a/docs/experimental-features/images/enable-hrs.png b/docs/experimental-features/images/enable-hrs.png deleted file mode 100644 index c77dfb7ce3..0000000000 Binary files a/docs/experimental-features/images/enable-hrs.png and /dev/null differ diff --git a/docs/experimental-features/images/enable-urs.png b/docs/experimental-features/images/enable-urs.png deleted file mode 100644 index e7ffde47ca..0000000000 Binary files a/docs/experimental-features/images/enable-urs.png and /dev/null differ diff --git a/docs/experimental-features/images/feature-flag.png b/docs/experimental-features/images/feature-flag.png deleted file mode 100644 index 55abffa37c..0000000000 Binary files a/docs/experimental-features/images/feature-flag.png and /dev/null differ diff --git a/docs/experimental-features/images/full-dashboard.png b/docs/experimental-features/images/full-dashboard.png deleted file mode 100644 index 073ffec098..0000000000 Binary files a/docs/experimental-features/images/full-dashboard.png and /dev/null differ diff --git a/docs/experimental-features/images/go-to-host-view.png b/docs/experimental-features/images/go-to-host-view.png deleted file mode 100644 index ce81f59580..0000000000 Binary files a/docs/experimental-features/images/go-to-host-view.png and /dev/null differ diff --git a/docs/experimental-features/images/go-to-host.png b/docs/experimental-features/images/go-to-host.png deleted file mode 100644 index e9fead98ee..0000000000 Binary files a/docs/experimental-features/images/go-to-host.png and /dev/null differ diff --git a/docs/experimental-features/images/histogram.png b/docs/experimental-features/images/histogram.png deleted file mode 100644 index 77b9fe6c13..0000000000 Binary files a/docs/experimental-features/images/histogram.png and /dev/null differ diff --git a/docs/experimental-features/images/host-risk-score-dev-tools-console.png b/docs/experimental-features/images/host-risk-score-dev-tools-console.png deleted file mode 100644 index 2e787d6d99..0000000000 Binary files a/docs/experimental-features/images/host-risk-score-dev-tools-console.png and /dev/null differ diff --git a/docs/experimental-features/images/host-risk-score-enable-dev-tools.png b/docs/experimental-features/images/host-risk-score-enable-dev-tools.png deleted file mode 100644 index 7fcea43afa..0000000000 Binary files a/docs/experimental-features/images/host-risk-score-enable-dev-tools.png and /dev/null differ diff --git a/docs/experimental-features/images/host-risk-score-import-dashboard.png b/docs/experimental-features/images/host-risk-score-import-dashboard.png deleted file mode 100644 index e460d4c951..0000000000 Binary files a/docs/experimental-features/images/host-risk-score-import-dashboard.png and /dev/null differ diff --git a/docs/experimental-features/images/host-score-overview.png b/docs/experimental-features/images/host-score-overview.png deleted file mode 100644 index 897603f7f8..0000000000 Binary files a/docs/experimental-features/images/host-score-overview.png and /dev/null differ diff --git a/docs/experimental-features/images/hosts-by-risk-details-page.png b/docs/experimental-features/images/hosts-by-risk-details-page.png deleted file mode 100644 index 7aca2d9346..0000000000 Binary files a/docs/experimental-features/images/hosts-by-risk-details-page.png and /dev/null differ diff --git a/docs/experimental-features/images/hrs-alerts-table.png b/docs/experimental-features/images/hrs-alerts-table.png deleted file mode 100644 index 7691cd8df9..0000000000 Binary files a/docs/experimental-features/images/hrs-alerts-table.png and /dev/null differ diff --git a/docs/experimental-features/images/hrs-all-hosts.png b/docs/experimental-features/images/hrs-all-hosts.png deleted file mode 100644 index dac02c56bb..0000000000 Binary files a/docs/experimental-features/images/hrs-all-hosts.png and /dev/null differ diff --git a/docs/experimental-features/images/hrs-overview-section.png b/docs/experimental-features/images/hrs-overview-section.png deleted file mode 100644 index 7b6bfc34f9..0000000000 Binary files a/docs/experimental-features/images/hrs-overview-section.png and /dev/null differ diff --git a/docs/experimental-features/images/kspm-1.png b/docs/experimental-features/images/kspm-1.png deleted file mode 100644 index c00479230b..0000000000 Binary files a/docs/experimental-features/images/kspm-1.png and /dev/null differ diff --git a/docs/experimental-features/images/kspm-2.png b/docs/experimental-features/images/kspm-2.png deleted file mode 100644 index 161bca094d..0000000000 Binary files a/docs/experimental-features/images/kspm-2.png and /dev/null differ diff --git a/docs/experimental-features/images/restart-hrs.png b/docs/experimental-features/images/restart-hrs.png deleted file mode 100644 index b323cc38f0..0000000000 Binary files a/docs/experimental-features/images/restart-hrs.png and /dev/null differ diff --git a/docs/experimental-features/images/restart-urs.png b/docs/experimental-features/images/restart-urs.png deleted file mode 100644 index 9ff8f7c480..0000000000 Binary files a/docs/experimental-features/images/restart-urs.png and /dev/null differ diff --git a/docs/experimental-features/images/score-in-flyout.png b/docs/experimental-features/images/score-in-flyout.png deleted file mode 100644 index 5aef84bab3..0000000000 Binary files a/docs/experimental-features/images/score-in-flyout.png and /dev/null differ diff --git a/docs/experimental-features/images/select-hrs-dashboard.png b/docs/experimental-features/images/select-hrs-dashboard.png deleted file mode 100644 index 834cfe25a1..0000000000 Binary files a/docs/experimental-features/images/select-hrs-dashboard.png and /dev/null differ diff --git a/docs/experimental-features/images/select-urs-dashboard.png b/docs/experimental-features/images/select-urs-dashboard.png deleted file mode 100644 index 0f64faa16c..0000000000 Binary files a/docs/experimental-features/images/select-urs-dashboard.png and /dev/null differ diff --git a/docs/experimental-features/images/urs-alerts-table.png b/docs/experimental-features/images/urs-alerts-table.png deleted file mode 100644 index 98bf66145e..0000000000 Binary files a/docs/experimental-features/images/urs-alerts-table.png and /dev/null differ diff --git a/docs/experimental-features/images/urs-details-page.png b/docs/experimental-features/images/urs-details-page.png deleted file mode 100644 index d24cb5a8b5..0000000000 Binary files a/docs/experimental-features/images/urs-details-page.png and /dev/null differ diff --git a/docs/experimental-features/images/urs-histogram.png b/docs/experimental-features/images/urs-histogram.png deleted file mode 100644 index fca00bc4e6..0000000000 Binary files a/docs/experimental-features/images/urs-histogram.png and /dev/null differ diff --git a/docs/experimental-features/images/urs-overview-section.png b/docs/experimental-features/images/urs-overview-section.png deleted file mode 100644 index dccc4bf61e..0000000000 Binary files a/docs/experimental-features/images/urs-overview-section.png and /dev/null differ diff --git a/docs/experimental-features/images/urs-score-flyout.png b/docs/experimental-features/images/urs-score-flyout.png deleted file mode 100644 index 9db5cb676a..0000000000 Binary files a/docs/experimental-features/images/urs-score-flyout.png and /dev/null differ diff --git a/docs/experimental-features/images/urs-table.png b/docs/experimental-features/images/urs-table.png deleted file mode 100644 index 8f8eabbe92..0000000000 Binary files a/docs/experimental-features/images/urs-table.png and /dev/null differ diff --git a/docs/experimental-features/images/users-by-risk-details-page.png b/docs/experimental-features/images/users-by-risk-details-page.png deleted file mode 100644 index 28c940deff..0000000000 Binary files a/docs/experimental-features/images/users-by-risk-details-page.png and /dev/null differ diff --git a/docs/experimental-features/images/usr-details-usr-risk-tab.png b/docs/experimental-features/images/usr-details-usr-risk-tab.png deleted file mode 100644 index 57a6d112b0..0000000000 Binary files a/docs/experimental-features/images/usr-details-usr-risk-tab.png and /dev/null differ diff --git a/docs/experimental-features/user-risk-score.asciidoc b/docs/experimental-features/user-risk-score.asciidoc deleted file mode 100644 index 3161f265dd..0000000000 --- a/docs/experimental-features/user-risk-score.asciidoc +++ /dev/null @@ -1,309 +0,0 @@ -[[user-risk-score]] -== User risk score - -NOTE: This page refers to the original user and host risk score modules. If you have the original modules installed, and you're running {stack} version 8.11 or newer, you can <>. -For information about the latest risk engine, refer to <>. - -NOTE: This feature is available for {stack} versions 8.3.0 and newer and requires a https://www.elastic.co/pricing[Platinum subscription] or higher. - -The user risk score feature highlights risky usernames in your environment. It utilizes a transform with a scripted metric aggregation to calculate user risk scores based on alerts generated within the past 90 days. The transform runs hourly to update scores as new alerts are generated. - -Each alert's contribution to the user risk score is based on the alert's risk score (`signal.rule.risk_score`). The risk score is calculated using a weighted sum where rules with higher time-corrected risk scores also have higher weights. Each risk score is normalized on a scale of 0 to 100. - -The following table shows how risk levels are applied to a username, based on the normalized risk score: - -[width="100%",options="header"] -|============================================== -|Risk level |User risk score - -|Unknown |< 20 -|Low |20-40 -|Moderate |40-70 -|High | 70-90 -|Critical | > 90 - -|============================================== - -[discrete] -[[deploy-user-risk-score]] -=== Enable user risk score - -You can enable user risk score from the following places in the {security-app}: - -* The Entity Analytics dashboard -* The *User risk* tab on the Users page -* The *User risk* tab on a user's details page - -Or, in {kib}, you can enable user risk score in Console. - -To enable user risk score from the Entity Analytics dashboard: - -. In the {security-app}, go to *Dashboards* -> *Entity Analytics*. -. In the User Risk Scores section, click *Enable* to install the module. - - -To enable user risk score from the Users page: - -. Go to *Explore* -> *Users*. -. Select the *User risk* tab, then click *Enable* to install the module. - -[role="screenshot"] -image::images/enable-urs.png[Enable User Risk score button] - -To enable user risk score from a user's details page: - -. Go to *Explore* -> *Users*. -. Select the *All users* tab, then click a user name. -. On the details page, scroll down to the data tables, then select the *User risk* tab. -. Click *Enable* to install the module. - -To enable user risk score from Console in {kib}, open a browser window and enter the following URL: - -[source,console] ----------------------------------- -{KibanaURL}/s/{spaceID}/app/dev_tools#/console?load_from={KibanaURL}/s/{spaceID}/internal/risk_score/prebuilt_content/dev_tool/enable_user_risk_score ----------------------------------- - -NOTE: If there's existing content in Console, scroll to the bottom to find the output loaded. - -TIP: If you receive an error message during the installation process, delete the user risk score module manually, and then re-enable it. Refer to <> for more information. - -[[upgrade-user-risk-score]] -[discrete] -=== Upgrade user risk score - -If you previously enabled user risk score and you're upgrading to {stack} version 8.11 or newer, you can <>. - -Before upgrading, note the following: - -* Since older data is not preserved, previous user risk scores will be deleted, and new scores will be created. However, if you want to retain old user risk scores, you can reindex them _before_ upgrading. To learn how, refer to {ref}/docs-reindex.html[Reindex API]. New data will be stored in the `ml_user_risk_score_` and `ml_user_risk_score_latest_` indices. - -* You must edit your {cloud}/ec-manage-kibana-settings.html#ec-manage-kibana-settings[{kib} user settings] and remove the `xpack.securitySolution.enableExperimental:['riskyUsersEnabled']` feature flag. - -After this is done, you can proceed with upgrading the user risk score feature from any of the following places in the {security-app}: - -* The Entity Analytics dashboard -* The *User risk* tab on the User page -* The *User risk* tab on a user's details page - -NOTE: After you enable or upgrade user risk score, you might get a message that says, "No user risk score data available to display." To verify that the transform that installs the user risk score module is picking up data, refer to <>. - -TIP: If you receive an error message during the installation process, delete the user risk score module manually, and then re-enable it. Refer to <> for more information. - -[[view-user-risk-score]] -[discrete] -=== Analyze user risk score data - -It is recommended you analyze users with the highest risk scores first -- those in the `Critical` and `Moderate` categories. User risk score data appears in the following places in the {security-app}: - -The `user.risk.calculated_level` column in the Alerts table: - -[role="screenshot"] -image::images/urs-alerts-table.png[User risk score in Alerts table] - -The *Insights* -> *Entities* section on the *Overview* tab within the alert details flyout - -[role="screenshot"] -image::images/urs-score-flyout.png[User risk score in alert details flyout,65%] - -The *User risk* tab on the Users page: - -[role="screenshot"] -image::images/users-by-risk-details-page.png[User risk score on Users risk tab] - -The Overview section on the user details page: - -[role="screenshot"] -image::images/urs-overview-section.png[User risk score in Overview section] - -The *User risk* tab on the user details page: - -[role="screenshot"] -image::images/usr-details-usr-risk-tab.png[User risk score on the user details page] - -You can also visualize user risk score data using prebuilt dashboards that are automatically imported when the feature is enabled. - -To access the dashboards: - -. In {kib}, go to *Analytics -> Dashboard*, then search for `risk score`. -. Select *Drilldown of User Risk Score* to analyze the risk components of a user, or *Current Risk Score for Users* to display a list of current risky users in your environment. - -In this example, we'll explore the *Drilldown of User Risk Score* dashboard. - -[role="screenshot"] -image::images/select-urs-dashboard.png[Select dashboard] - -The histogram shows historical changes in a particular user's risk score(s). To specify a date range, use the date and time picker, or drag and select a time range within the histogram. Click *View source dashboard* to view the top values of `user.name` and `risk.keyword`. - -[role="screenshot"] -image::images/urs-histogram.png[User risk score histogram] - -The data tables beneath the histogram display associated rules, users, and MITRE ATT&CK tactics seen for risky users. By default, the tables are sorted by risk, with the highest total risk scores at the top. Use this information to triage your highest risk users. - -[role="screenshot"] -image::images/dashboard.gif[User risk score dashboard] - -[[troubleshoot-user-risk-score]] -[discrete] -=== Troubleshooting - -During the installation or upgrade process, you may receive the following error messages: - -* `Saved object already exists` -* `Transform already exists` -* `Ingest pipeline already exists` - -In this case, we recommend that you manually delete the user risk score module, and then re-enable it. To manually delete the module: - -. Delete the user risk score saved objects: -.. From the {kib} main menu, go to **Stack Management** -> **Kibana** -> **Saved Objects**. -.. Delete the saved objects that have the `User Risk Score - ` tag. -+ -[role="screenshot"] -image::images/delete-urs-saved-objects.png[Delete user risk score saved objects] -.. Delete the `User Risk Score - ` tag. -+ -[role="screenshot"] -image::images/delete-urs-tag.png[Delete user risk score tag] -. Stop and delete the user risk score transforms. You can do this using the {kib} UI or the {ref}/stop-transform.html[Stop transform API] and {ref}/delete-transform.html[Delete transform API]. -** To delete the user risk score transforms using the {kib} UI: -.. From the {kib} main menu, go to **Stack Management** -> **Data** -> **Transforms**. -.. Stop the following transforms, then delete them: -*** `ml_userriskscore_latest_transform_` -*** `ml_userriskscore_pivot_transform_` -** To delete the user risk score transforms using the API, run the following commands in Console: -.. Stop and delete the latest transform: -+ -[source,console] ----------------------------------- -POST _transform/ml_userriskscore_latest_transform_/_stop -DELETE _transform/ml_userriskscore_latest_transform_ ----------------------------------- -.. Stop and delete the pivot transform: -+ -[source,console] ----------------------------------- -POST _transform/ml_userriskscore_pivot_transform_/_stop -DELETE _transform/ml_userriskscore_pivot_transform_ ----------------------------------- -. Delete the user risk score ingest pipeline. You can do this using the {kib} UI or the {ref}/delete-pipeline-api.html[Delete pipeline API]. -** To delete the user risk score ingest pipeline using the {kib} UI: -.. From the {kib} main menu, go to **Stack Management** -> **Ingest** -> **Ingest Pipelines**. -.. Delete the `ml_userriskscore_ingest_pipeline_` ingest pipeline. -** To delete the user risk score ingest pipeline using the Delete pipeline API, run the following command in Console: -+ -[source,console] ----------------------------------- -DELETE /_ingest/pipeline/ml_userriskscore_ingest_pipeline_ ----------------------------------- -. Delete the stored user risk score scripts using the {ref}/delete-stored-script-api.html[Delete stored script API]. In Console, run the following commands: -+ -[source,console] ----------------------------------- -DELETE _scripts/ml_userriskscore_levels_script_ -DELETE _scripts/ml_userriskscore_map_script_ -DELETE _scripts/ml_userriskscore_reduce_script_ ----------------------------------- - -After manually deleting the user risk score saved objects, transforms, ingest pipeline, and stored scripts, follow the steps to <>. - -[[verify-user-risk-score]] -=== Verify that user risk score data installed successfully (Optional) - -After you enable or upgrade user risk score, the following message may appear: - -[role="screenshot"] -image::images/restart-urs.png[Restart user risk score] - -If so, click *Restart* and allow at least an hour for the data to be generated. If data still doesn't appear, verify that user risk score data has been generated: - -In {kib}, run the following commands in Console to query the `ml_user_risk_score_` index: - -[source,console] ----------------------------------- -GET ml_user_risk_score_/_search ----------------------------------- - -If no data returns, you'll need to check if the alerts index (`.alerts-security.alerts-`) had alert data when `ml_userriskscore_pivot_transform_` was started. - -Example: - -[source,console] ----------------------------------- -GET transform/ml_userriskscore_pivot_transform_/_stats?human=true ----------------------------------- - -Here's an example response: - -[source,console] ----------------------------------- -{ - "count": 1, - "transforms": [ - { - "id": "ml_userriskscore_pivot_transform_", - "state": "started", - "node": { - "id": "H1tlwfTyRkWls-C0sarmHw", - "name": "instance-0000000000", - "ephemeral_id": "SBqlp5ywRuuop2gtcdCljA", - "transport_address": "10.43.255.164:19635", - "attributes": {} - }, - "stats": { - "pages_processed": 29, - "documents_processed": 11805, - "documents_indexed": 8, - "documents_deleted": 0, - "trigger_count": 9, - "index_time_in_ms": 52, - "index_total": 7, - "index_failures": 0, - "search_time_in_ms": 201, - "search_total": 29, - "search_failures": 0, - "processing_time_in_ms": 14, - "processing_total": 29, - "delete_time_in_ms": 0, - "exponential_avg_checkpoint_duration_ms": 59.02353261024906, - "exponential_avg_documents_indexed": 0.8762710605864747, - "exponential_avg_documents_processed": 1664.7724779548555 - }, - "checkpointing": { - "last": { - "checkpoint": 8, - "timestamp": "2022-10-17T14:49:50.315Z", - "timestamp_millis": 1666018190315, - "time_upper_bound": "2022-10-17T14:47:50.315Z", - "time_upper_bound_millis": 1666018070315 - }, - "operations_behind": 380, - "changes_last_detected_at_string": "2022-10-17T14:49:50.113Z", - "changes_last_detected_at": 1666018190113, - "last_search_time_string": "2022-10-17T14:49:50.113Z", - "last_search_time": 1666018190113 - } - } - ] -} ----------------------------------- - -Take note of the value from `time_upper_bound_millis` and enter it as a range query for the alerts index. - -Example: - -[source,console] ----------------------------------- -GET .alerts-security.alerts-/_search -{ - "query": { - "range": { - "@timestamp": { - "lt": 1666018070315 - } - } - } -} ----------------------------------- - -If there's no response, verify that relevant <> are running and that alert data is being generated. If there is a response, click *Restart* and allow an hour for the user risk data to appear. \ No newline at end of file diff --git a/docs/getting-started/users-page.asciidoc b/docs/getting-started/users-page.asciidoc index 35ba5fb8a7..f7df997574 100644 --- a/docs/getting-started/users-page.asciidoc +++ b/docs/getting-started/users-page.asciidoc @@ -24,7 +24,7 @@ Beneath the KPI charts are data tables, which are useful for viewing and investi * *All users*: A chronological list of unique user names, when they were last active, and the associated domains. * *Authentications*: A chronological list of user authentication events and associated details, such as the number of successes and failures, and the host name of the last successful destination. * *Anomalies*: Unusual activity discovered by machine learning jobs that contain user data. -* *User risk*: The latest recorded user risk score for each user, and its user risk classification. This feature requires a https://www.elastic.co/pricing[Platinum subscription] or higher and must be enabled to display the data. Click *Enable* on the *User risk* tab to get started. To learn more, refer to our <>. +* *User risk*: The latest recorded user risk score for each user, and its user risk classification. This feature requires a https://www.elastic.co/pricing[Platinum subscription] or higher and must be enabled to display the data. Click *Enable* on the *User risk* tab to get started. To learn more, refer to our <>. The Events table includes inline actions and several customization options. To learn more about what you can do with the data in these tables, refer to <>. diff --git a/docs/index.asciidoc b/docs/index.asciidoc index 6791f36c90..0194e513d9 100644 --- a/docs/index.asciidoc +++ b/docs/index.asciidoc @@ -46,8 +46,6 @@ include::reference/ref-index.asciidoc[] include::troubleshooting/troubleshooting-intro.asciidoc[] -include::experimental-features/experimental-features-intro.asciidoc[] - include::release-notes.asciidoc[] include::detections/prebuilt-rules/downloadable-packages/0-13-1/prebuilt-rules-0-13-1-appendix.asciidoc[] diff --git a/docs/management/hosts/hosts-overview.asciidoc b/docs/management/hosts/hosts-overview.asciidoc index d50ecfc63d..b8d12049cb 100644 --- a/docs/management/hosts/hosts-overview.asciidoc +++ b/docs/management/hosts/hosts-overview.asciidoc @@ -26,7 +26,7 @@ Beneath the KPI charts are data tables, categorized by individual tabs, which ar * *All hosts*: High-level host details. * *Uncommon processes*: Uncommon processes running on hosts. * *Anomalies*: Anomalies discovered by machine learning jobs. -* *Host risk*: The latest recorded host risk score for each host, and its host risk classification. This feature requires a https://www.elastic.co/pricing[Platinum subscription] or higher and must be enabled to display the data. Click *Enable* on the *Host risk* tab to get started. To learn more, refer to our <>. +* *Host risk*: The latest recorded host risk score for each host, and its host risk classification. This feature requires a https://www.elastic.co/pricing[Platinum subscription] or higher and must be enabled to display the data. Click *Enable* on the *Host risk* tab to get started. To learn more, refer to our <>. * *Sessions*: Linux process events that you can open in <>, an investigation tool that allows you to examine Linux process data at a hierarchal level. The tables within the *Events* and *Sessions* tabs include inline actions and several customization options. To learn more about what you can do with the data in these tables, refer to <>.