[Security Solution] Test plan for prebuilt rule customization #204888
Conversation
dplumlee
added
test-plan
v9.0.0
Team:Detections and Resp
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
|
Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management) |
dplumlee
added
the
release_note:skip
|
@pborgonovi Please review this one |
There was a problem hiding this comment.
@dplumlee Thanks for adding Prebuilt Rules Customization test plan 🙏
I reviewed the test plan and left suggestions to make the test plan more robust and improve readability.
It looks like this test plan should include the following scenarios as well
- A test scenario (with examples) for each field affecting isCustomized
- Custom rules shouldn't have "Modified" badge on rule details page
- Custom rules shouldn't have "Modified" badge on rule management table
- Scenarios covering
ruleSource - Users can navigate to rule edit page from prebuilt rule details
Feel free to reach me to discuss it over Zoom in case of questions.
|
|
||
| ```Gherkin | ||
| Given a space with at least one prebuilt rule installed | ||
| And the rule is unmodified |
There was a problem hiding this comment.
| And the rule is unmodified | |
| And the rule is non-customized |
| ```Gherkin | ||
| Given a space with at least one prebuilt rule installed | ||
| And the rule is unmodified | ||
| When a user edits the rule from the rule edit page to something different than the original version |
There was a problem hiding this comment.
| When a user edits the rule from the rule edit page to something different than the original version | |
| When user changes any rule field value (so it differs from the base version) in rule edit form | |
| And saves the form |
Editing a rule is possible only via rule editing form but the same terminology isn't applicable to CRUD API. Change rule field sounds more generic. WDYT?
| And the ruleSource should be "external" | ||
| And isCustomized should be true |
There was a problem hiding this comment.
Should ruleSource and isCustomized be explained in terminology?
| **Automation**: 1 cypress test. | ||
|
|
||
| ```Gherkin | ||
| Given a space with at least one customized prebuilt rule installed |
There was a problem hiding this comment.
| Given a space with at least one customized prebuilt rule installed | |
| Given a space with at least one prebuilt rule installed | |
| And it is customized |
|
|
||
| ```Gherkin | ||
| Given a space with at least one customized prebuilt rule installed | ||
| When a user edits the rule from the rule edit page to something different than the original version |
There was a problem hiding this comment.
| When a user edits the rule from the rule edit page to something different than the original version | |
| When user changes any rule field value (so it differs from the base version) in rule edit form | |
| And saves the form |
|
|
||
| ### Assumptions | ||
|
|
||
| - Rule package used will have all rule versions present (no missing base versions) |
There was a problem hiding this comment.
| - Rule package used will have all rule versions present (no missing base versions) | |
| - Rule package used will have all previous rule versions present (no missing base versions) |
|
|
||
| ### Terminology | ||
|
|
||
| - **Base version**: The version of the rule we ship with the rule package, can be thought of as the "original" version of the rule. |
There was a problem hiding this comment.
| - **Base version**: The version of the rule we ship with the rule package, can be thought of as the "original" version of the rule. | |
| - **Base version**: Prebuilt rule asset we ship in the rule package corresponding to the currently installed prebuilt rules. It represents "original" version of the rule. During prebuilt rules installation prebuilt rule assets data is copied over and becomes an installed prebuilt rule. |
There was a problem hiding this comment.
Can we say that it's a prebuilt rule asset whose "version" field matches the "version" of an installed rule? WDYT?
|
|
||
| - **Base version**: The version of the rule we ship with the rule package, can be thought of as the "original" version of the rule. | ||
|
|
||
| - **Customized prebuilt rule**: A prebuilt rule that has been changed by the user from the base version of the rule. Also referred to as "Modified" in the UI. |
There was a problem hiding this comment.
| - **Customized prebuilt rule**: A prebuilt rule that has been changed by the user from the base version of the rule. Also referred to as "Modified" in the UI. | |
| - **Customized prebuilt rule**: An installed prebuilt rule that has been changed by the user in the way rule fields semantically differ from the base version. Also referred to as "Modified" in the UI. |
|
|
||
| - **Customized prebuilt rule**: A prebuilt rule that has been changed by the user from the base version of the rule. Also referred to as "Modified" in the UI. | ||
|
|
||
| - **Non-customized prebuilt rule**: A prebuilt rule that has no change from the base version of the rule. |
There was a problem hiding this comment.
| - **Non-customized prebuilt rule**: A prebuilt rule that has no change from the base version of the rule. | |
| - **Non-customized prebuilt rule**: An installed prebuilt rule that has rule fields values matching the base version. |
|
|
||
| - **Non-customized prebuilt rule**: A prebuilt rule that has no change from the base version of the rule. | ||
|
|
||
| - **Custom rules**: A rule created by the user themselves |
There was a problem hiding this comment.
| - **Custom rules**: A rule created by the user themselves | |
| - **Custom rule**: A rule created by the user themselves |
| ```Gherkin | ||
| Given a space with at least one customized prebuilt rule installed | ||
| When a user navigates to that rule's detail page | ||
| Then the rule's isCustomized value should be true |
There was a problem hiding this comment.
I think it should be snake case, since it'll come from our API
| Then the rule's isCustomized value should be true | |
| Then the rule's `is_customized` value should be true |
If you are going to incorporate this suggestion, please check other places as well
There was a problem hiding this comment.
Thanks for the PR, @dplumlee! Overall, the test coverage looks solid. I’ve left a couple of comments for you to consider.
One potential improvement: should we add a scenario to ensure that the “Edit” buttons in the rule details page and rule management table are clickable when the feature flag is enabled?
Additionally, do we want to include a test to verify that non-modifiable fields, such as version and id are uneditable?”
Looking forward to your thoughts!
Summary
Addresses #202068
Adds test plan for rule customization features related to the milestone 3 prebuilt rule customization epic