From c4fa1476acda692f737368e9bd7c23deee9b22ec Mon Sep 17 00:00:00 2001 From: Davis Plumlee Date: Thu, 8 Feb 2024 01:21:57 -0500 Subject: [PATCH 1/4] adds initial tests --- .../installation_and_upgrade.md | 85 +++++++++++++++++++ 1 file changed, 85 insertions(+) diff --git a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md index 19a15b64b5046..447d9d1d57a58 100644 --- a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md +++ b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md @@ -63,6 +63,11 @@ Status: `in progress`. The current test plan matches `Milestone 2` of the [Rule - [**Scenario: Properties with semantically equal values should not be shown as modified**](#scenario-properties-with-semantically-equal-values-should-not-be-shown-as-modified) - [**Scenario: Unchanged sections of a rule should be hidden by default**](#scenario-unchanged-sections-of-a-rule-should-be-hidden-by-default) - [**Scenario: Properties should be sorted alphabetically**](#scenario-properties-should-be-sorted-alphabetically) + - [Rule upgrade workflow: viewing rule changes in per-field diff view](#rule-upgrade-workflow-viewing-rule-changes-in-per-field-diff-view) + - [**Scenario: User can see changes in a side-by-side per-field diff view**](#scenario-user-can-see-changes-in-a-side-by-side-per-field-diff-view) + - [**Scenario: Field groupings should be rendered together in the same accordion panel**](#scenario-field-groupings-should-be-rendered-together-in-the-same-accordion-panel) + - [**Scenario: Undefined values are displayed with empty diffs**](#scenario-undefined-values-are-displayed-with-empty-diffs) + - [**Scenario: Field diff components have the same grouping and order as in rule details overview**](#scenario-field-diff-components-have-the-same-grouping-and-order-as-in-rule-details-overview) - [Rule upgrade workflow: misc cases](#rule-upgrade-workflow-misc-cases) - [**Scenario: User doesn't see the Rule Updates tab until the package installation is completed**](#scenario-user-doesnt-see-the-rule-updates-tab-until-the-package-installation-is-completed) - [Error handling](#error-handling) @@ -949,6 +954,86 @@ When a user expands all hidden sections Then all properties of the rule should be sorted alphabetically ``` +### Rule upgrade workflow: viewing rule changes in per-field diff view + +#### **Scenario: User can see changes in a side-by-side per-field diff view** + +**Automation**: 1 e2e test + +```Gherkin +Given X prebuilt rules are installed in Kibana +And for Y of these rules new versions are available +When user opens the Rule Updates table and selects a rule +Then the per-field upgrade preview should open +And rule changes should be displayed in a two-column diff view with each field in its own accordion component +And all field diff accordions should be open by default +And correct rule version numbers should be displayed in their respective columns +When the user selects another rule without closing the preview +Then the preview should display the changes for the newly selected rule +``` + +#### **Scenario: Field groupings should be rendered together in the same accordion panel** + +**Automation**: 1 UI integration test + +```Gherkin +Given a prebuilt rule is installed in Kibana +And this rule contains one or more values +When user opens the upgrade preview +The diff accordion panel should display its grouped rule properties +And each property should have its name displayed inside the panel + +Examples: +| field | +| data_source | +| kql_query | +| eql_query | +| esql_query | +| threat_query | +| rule_schedule | +| rule_name_override | +| timestamp_override | +| timeline_template | +| building_block | +| threshold | +``` + +#### **Scenario: Undefined values are displayed with empty diffs** + +**Automation**: 1 UI integration test + +```Gherkin +Given a prebuilt rule is installed in Kibana +And this rule has field in the version that didn't exist in the version +When a user opens the upgrade preview +Then the preview should open +And the old/new field should render an empty panel + +Examples: +| version_one | version_two | +| target | current | +| current | target | +``` + +#### **Scenario: Field diff components have the same grouping and order as in rule details overview** + +**Automation**: 1 UI integration test + +```Gherkin +Given a prebuilt rule is installed in Kibana +And this rule has multiple fields that are different between the current and target version +When a user opens the upgrade preview +Then the multiple field diff accordions should be sorted in the same order as on the rule details overview tab +And the field diff accordions should be grouped inside its corresponding
accordion + +Examples: +| section | +| About | +| Definition | +| Schedule | +| Setup Guide | +``` + ### Rule upgrade workflow: misc cases #### **Scenario: User doesn't see the Rule Updates tab until the package installation is completed** From 9d944f2fdc88e16ece7808dcd19e8635c8b4b724 Mon Sep 17 00:00:00 2001 From: Davis Plumlee Date: Thu, 8 Feb 2024 15:13:45 -0500 Subject: [PATCH 2/4] adds some more detail --- .../prebuilt_rules/installation_and_upgrade.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md index 447d9d1d57a58..b4fa408dd8ae5 100644 --- a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md +++ b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md @@ -958,7 +958,7 @@ Then all properties of the rule should be sorted alphabetically #### **Scenario: User can see changes in a side-by-side per-field diff view** -**Automation**: 1 e2e test +**Automation**: 1 e2e test + 1 UI integration test ```Gherkin Given X prebuilt rules are installed in Kibana @@ -970,6 +970,8 @@ And all field diff accordions should be open by default And correct rule version numbers should be displayed in their respective columns When the user selects another rule without closing the preview Then the preview should display the changes for the newly selected rule + +CASE: Test case should work when target rule version contains a different rule type than current ``` #### **Scenario: Field groupings should be rendered together in the same accordion panel** @@ -981,7 +983,7 @@ Given a prebuilt rule is installed in Kibana And this rule contains one or more values When user opens the upgrade preview The diff accordion panel should display its grouped rule properties -And each property should have its name displayed inside the panel +And each property should have its name displayed inside the panel above its value Examples: | field | @@ -1025,6 +1027,7 @@ And this rule has multiple fields that are different between the current and tar When a user opens the upgrade preview Then the multiple field diff accordions should be sorted in the same order as on the rule details overview tab And the field diff accordions should be grouped inside its corresponding
accordion +And any
accordion that doesn't have fields inside it shouldn't be displayed Examples: | section | From 246a9393bc29bd64c041aee8458a89d5c3bc0933 Mon Sep 17 00:00:00 2001 From: Davis Plumlee Date: Fri, 9 Feb 2024 13:01:34 -0500 Subject: [PATCH 3/4] addresses comments --- .../prebuilt_rules/installation_and_upgrade.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md index b4fa408dd8ae5..456ba27a7a64e 100644 --- a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md +++ b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md @@ -958,7 +958,7 @@ Then all properties of the rule should be sorted alphabetically #### **Scenario: User can see changes in a side-by-side per-field diff view** -**Automation**: 1 e2e test + 1 UI integration test +**Automation**: 1 e2e test ```Gherkin Given X prebuilt rules are installed in Kibana @@ -970,8 +970,18 @@ And all field diff accordions should be open by default And correct rule version numbers should be displayed in their respective columns When the user selects another rule without closing the preview Then the preview should display the changes for the newly selected rule +``` + +#### **Scenario: User can see changes when updated rule is a different rule type** + +**Automation**: 1 UI integration test -CASE: Test case should work when target rule version contains a different rule type than current +```Gherkin +Given a prebuilt rule is installed in Kibana +And this rule has an update available that changes the rule type +When user opens the upgrade preview +Then the rule type changes should be displayed in grouped field diffs with corresponding query fields +And a tooltip is displayed with information about changing rule types ``` #### **Scenario: Field groupings should be rendered together in the same accordion panel** @@ -985,7 +995,7 @@ When user opens the upgrade preview The diff accordion panel should display its grouped rule properties And each property should have its name displayed inside the panel above its value -Examples: +CASES: | field | | data_source | | kql_query | From b7ba1b9f1e49dcf51fbd1ee545f33615810986f6 Mon Sep 17 00:00:00 2001 From: Davis Plumlee Date: Mon, 12 Feb 2024 00:44:07 -0500 Subject: [PATCH 4/4] addresses comments --- .../prebuilt_rules/installation_and_upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md index 456ba27a7a64e..3a21a58ae79d7 100644 --- a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md +++ b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md @@ -995,7 +995,7 @@ When user opens the upgrade preview The diff accordion panel should display its grouped rule properties And each property should have its name displayed inside the panel above its value -CASES: +Examples: | field | | data_source | | kql_query |