Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Test plan for prebuilt rule flyout #167727

Merged
merged 5 commits into from
Oct 17, 2023
Merged

[Security Solution] Test plan for prebuilt rule flyout #167727

Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
238a950
Update the test plan
nikitaindik Oct 2, 2023
08b0c71
Refactor test plan
nikitaindik Oct 3, 2023
9aaae8a
Fix a typo
nikitaindik Oct 3, 2023
fd427cd
Code review fixes
banderror Oct 5, 2023
1168939
Add scenarios to check for empty sections and tabs
nikitaindik Oct 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
142 changes: 140 additions & 2 deletions ...esting/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,76 @@ Status: `in progress`. The current test plan matches `Milestone 2` of the [Rule
- Rule installation and upgrade workflows should work with packages containing up to 15000 historical rule versions. This is the max number of versions of all rules in the package. This limit is enforced by Fleet.
- Kibana should not crash with Out Of Memory exception during package installation.
- For test purposes, it should be possible to use detection rules package versions lower than the latest.
- In the prebuilt rule preview a tab that doesn't have any sections should not be displayed and a section that doesn't have any properties also should not be displayed
nikitaindik marked this conversation as resolved.
Show resolved Hide resolved


```Gherkin
Prebuilt rule properties examples:
| rule_type | property | section | tab |
│ All rule types │ Author │ About │ Overview │
│ All rule types │ Building block │ About │ Overview │
│ All rule types │ Severity │ About │ Overview │
│ All rule types │ Severity override │ About │ Overview │
│ All rule types │ Risk score │ About │ Overview │
│ All rule types │ Risk score override │ About │ Overview │
│ All rule types │ Reference URLs │ About │ Overview │
│ All rule types │ False positive examples │ About │ Overview │
│ All rule types │ Custom highlighted fields │ About │ Overview │
│ All rule types │ License │ About │ Overview │
│ All rule types │ Rule name override │ About │ Overview │
│ All rule types │ MITRE ATT&CK™ │ About │ Overview │
│ All rule types │ Timestamp override │ About │ Overview │
│ All rule types │ Tags │ About │ Overview │
│ All rule types │ Type │ Definition │ Overview │
│ All rule types │ Related integrations │ Definition │ Overview │
│ All rule types │ Required fields │ Definition │ Overview │
│ All rule types │ Timeline template │ Definition │ Overview │
│ All rule types │ Runs every │ Schedule │ Overview │
│ All rule types │ Additional look-back time │ Schedule │ Overview │
│ All rule types │ Setup guide │ Setup guide │ Overview │
│ All rule types │ Investigation guide │ Investigation guide │ Investigation guide │
│ Custom Query │ Index patterns │ Definition │ Overview │
│ Custom Query │ Data view ID │ Definition │ Overview │
│ Custom Query │ Data view index pattern │ Definition │ Overview │
│ Custom Query │ Custom query │ Definition │ Overview │
│ Custom Query │ Filters │ Definition │ Overview │
│ Custom Query │ Saved query name │ Definition │ Overview │
│ Custom Query │ Saved query filters │ Definition │ Overview │
│ Custom Query │ Saved query │ Definition │ Overview │
│ Custom Query │ Suppress alerts by │ Definition │ Overview │
│ Custom Query │ Suppress alerts for │ Definition │ Overview │
│ Custom Query │ If a suppression field is missing │ Definition │ Overview │
│ Machine Learning │ Anomaly score threshold │ Definition │ Overview │
│ Machine Learning │ Machine Learning job │ Definition │ Overview │
│ Threshold │ Threshold │ Definition │ Overview │
│ Threshold │ Index patterns │ Definition │ Overview │
│ Threshold │ Data view ID │ Definition │ Overview │
│ Threshold │ Data view index pattern │ Definition │ Overview │
│ Threshold │ Custom query │ Definition │ Overview │
│ Threshold │ Filters │ Definition │ Overview │
│ Event Correlation │ EQL query │ Definition │ Overview │
│ Event Correlation │ Filters │ Definition │ Overview │
│ Event Correlation │ Index patterns │ Definition │ Overview │
│ Event Correlation │ Data view ID │ Definition │ Overview │
│ Event Correlation │ Data view index pattern │ Definition │ Overview │
│ Indicator Match │ Indicator index patterns │ Definition │ Overview │
│ Indicator Match │ Indicator mapping │ Definition │ Overview │
│ Indicator Match │ Indicator filters │ Definition │ Overview │
│ Indicator Match │ Indicator index query │ Definition │ Overview │
│ Indicator Match │ Index patterns │ Definition │ Overview │
│ Indicator Match │ Data view ID │ Definition │ Overview │
│ Indicator Match │ Data view index pattern │ Definition │ Overview │
│ Indicator Match │ Custom query │ Definition │ Overview │
│ Indicator Match │ Filters │ Definition │ Overview │
│ New Terms │ Fields │ Definition │ Overview │
│ New Terms │ History Window Size │ Definition │ Overview │
│ New Terms │ Index patterns │ Definition │ Overview │
│ New Terms │ Data view ID │ Definition │ Overview │
│ New Terms │ Data view index pattern │ Definition │ Overview │
│ New Terms │ Custom query │ Definition │ Overview │
│ New Terms │ Filters │ Definition │ Overview │
│ ES|QL │ ES|QL query │ Definition │ Overview │
```

## Scenarios

Expand Down Expand Up @@ -366,7 +436,7 @@ Given no prebuilt rules are installed in Kibana
And there are X prebuilt rules available to install
When user opens the Add Rules page
Then prebuilt rules available for installation should be displayed in the table
When user installs one individual rule
When user installs one individual rule without previewing it
Then success message should be displayed after installation
And the installed rule should be removed from the table
When user navigates back to the Rule Management page
Expand Down Expand Up @@ -429,6 +499,39 @@ Then user should see a message indicating that all available rules have been ins
And user should see a CTA that leads to the Rule Management page
```

#### **Scenario: User can preview a rule before installing**

**Automation**: 1 e2e test

```Gherkin
Given no prebuilt rules are installed in Kibana
And there are at least 2 rules available to install
When user opens the Add Rules page
Then the user should be able to open a preview for a rule
When the preview is open the user should be able to close it
Then the user should be able to open a preview for another rule
When the preview is open the user should be able to install a rule using a CTA in the rule preview
Then success message should be displayed after installation
And the installed rule should be removed from the table
When user navigates back to the Rule Management page
Then user should see a CTA to install prebuilt rules
And user should see the number of rules available to install as 1
```

#### **Scenario: User can see correct rule information in preview before installing**

**Automation**: 1 e2e test

```Gherkin
Given no prebuilt rules are installed in Kibana
And there are X prebuilt rules available to install
When user opens the Add Rules page
Then the user should be able to open a preview for a rule
When the rule preview is open
Then for every property defined in a rule the user should see property information in the correct tab and section of the preview (see Prebuilt rule properties examples)
And selecting another rule in the table should update the preview content
```

### Rule installation workflow: filtering, sorting, pagination

TODO: add scenarios
Expand Down Expand Up @@ -467,7 +570,7 @@ And for Y of the installed rules there are new versions available
And user is on the Rule Management page
When user opens the Rule Updates table
Then Y rules available for upgrade should be displayed in the table
When user upgrades one individual rule
When user upgrades one individual rule without previewing it
Then success message should be displayed after upgrade
And the upgraded rule should be removed from the table
And user should see the number of rules available to upgrade decreased by 1
Expand Down Expand Up @@ -517,6 +620,41 @@ And user should NOT see the Rule Updates table

TODO: add scenarios

#### **Scenario: User can preview a rule before upgrading**

**Automation**: 1 e2e test

```Gherkin
Given at least 2 prebuilt rules are installed in Kibana
And for at least 2 of the installed rules there are new versions available
And user is on the Rule Management page
When user opens the Rule Updates table
Then all rules available for upgrade should be displayed in the table
And user should be able to open a preview for a rule
When the preview is open the user should be able to close it
Then the user should be able to open a preview for another rule
When the preview is open the user should be able to upgrade a rule using an action in the rule preview
Then success message should be displayed after upgrade
And the upgraded rule should be removed from the table
And user should see the number of rules available to upgrade as 1
```
nikitaindik marked this conversation as resolved.
Show resolved Hide resolved

#### **Scenario: User can see correct rule information in preview before upgrading**

**Automation**: 1 e2e test

```Gherkin
Given X prebuilt rules are installed in Kibana
And for Y of the installed rules there are new versions available
And user is on the Rule Management page
When user opens the Rule Updates table
Then Y rules available for upgrade should be displayed in the table
And the user should be able to open a preview for a rule
When the rule preview is open
Then for every property defined in a rule the user should see property information in the correct tab and section of the preview (see Prebuilt rule properties examples)
And selecting another rule in the table should update the preview content
```

### Rule upgrade workflow: misc cases

#### **Scenario: User doesn't see the Rule Updates tab until the package installation is completed**
Expand Down