Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Test plans for prebuilt rule import and export #191116

Merged
merged 9 commits into from
Nov 11, 2024
+192 −0
Merged

[Security Solution] Test plans for prebuilt rule import and export #191116

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
133712c
WIP: Adding test plans for prebuilt rule import
rylnd Aug 22, 2024
b01c616
WIP: more scenarios for rule importing
rylnd Aug 22, 2024
53163b4
Update tests following logical update
rylnd Oct 21, 2024
e629c20
Add/clarify/update Prebuilt Rule Import test scenarios
rylnd Oct 21, 2024
b245c35
Add tests for exporting of prebuilt rules
rylnd Oct 21, 2024
341de39
Fix whitespace errors
rylnd Oct 23, 2024
00b2008
Merge branch 'main' into ryland/prebuilt_rule_import_test_plan
elasticmachine Nov 6, 2024
b255895
Align import/export test plans to established template
rylnd Nov 8, 2024
100a639
Merge branch 'main' into ryland/prebuilt_rule_import_test_plan
rylnd Nov 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions ...solution/docs/testing/test_plans/detection_response/prebuilt_rules/exporting.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
Feature: Exporting Prebuilt Rules

Scenario: Exporting prebuilt rule individually
Given a space with prebuilt rules installed
When the user selects "Export rule" from the "All actions" dropdown on the rule's page
Then the rule should be exported as an NDJSON file
And it should include an "immutable" field with a value of true
And its "ruleSource" "type" should be "external"
And its "ruleSource" "isCustomized" value should depend on whether the rule was customized

Scenario: Exporting prebuilt rules in bulk
Given a space with prebuilt rules installed
When the user selects prebuilt rules in the alerts table
And chooses "Export" from bulk actions
Then the selected rules should be exported as an NDJSON file
And they should include an "immutable" field with a value of true
And their "ruleSource" "type" should be "external"
And their "ruleSource" "isCustomized" should depend on whether the rule was customized

Scenario: Exporting both prebuilt and custom rules in bulk
Given a space with prebuilt and custom rules installed
When the user selects prebuilt rules in the alerts table
And chooses "Export" from bulk actions
Then the selected rules should be exported as an NDJSON file
And the prebuilt rules should include an "immutable" field with a value of true
And the custom rules should include an "immutable" field with a value of false
And the prebuilt rules' "ruleSource" "type" should be "external"
And the custom rules' "ruleSource" "type" should be "internal"

Scenario: Exporting beyond the export limit
Given a space with prebuilt and custom rules installed
And the number of rules is greater than the export limit (defaults to 10_000)
Then the request should be rejected as a bad request
72 changes: 72 additions & 0 deletions ...solution/docs/testing/test_plans/detection_response/prebuilt_rules/importing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
Feature: Importing Prebuilt Rules

Scenario: Importing an unmodified prebuilt rule with a matching rule_id and version
Given the import payload contains a prebuilt rule with a matching rule_id and version, identical to the published rule
When the user imports the rule
Then the rule should be created or updated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth having the specificity between these two here? What causes the rule to be created vs updated?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dplumlee @pborgonovi brought up similar questions, which were discussed here, but the TL;DR:

When the test states Then the rule should be created or updated, that's really a shorthand for:

If the rule is new, create it
If the rule exists, update or reject based on the value of overwrite

But since that's all orthogonal to the prebuilt customization stuff, I didn't think it was an important thing to split out in these tests.

And the ruleSource type should be "external"
And isCustomized should be false

Scenario: Importing a customized prebuilt rule with a matching rule_id and version
Given the import payload contains a prebuilt rule with a matching rule_id and version, modified from the published version
When the user imports the rule
Then the rule should be created or updated
And the ruleSource type should be "external"
And isCustomized should be true

Scenario: Importing a prebuilt rule with a matching rule_id but no matching version
Given the import payload contains a prebuilt rule with a matching rule_id but no matching version
When the user imports the rule
Then the rule should be created or updated
And the ruleSource type should be "external"
And isCustomized should be true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rylnd the 2 scenarios above are independent of the overwrite flag?
For both cases, the import should be successful even with flag overwrite=false?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pborgonovi good question! Most of these tests deal with whether the incoming rule(s) match prebuilt assets based on rule_id and version, since that's where most of the nuance in this feature lies.

When the test states Then the rule should be created or updated, that's really a shorthand for:

  1. If the rule is new, create it
  2. If the rule exists, update or reject based on the value of overwrite

The overwriting functionality is orthogonal to this feature and so isn't really touched on in these tests; "known rule_id" and "matching rule_id" refer to the prebuilt assets, as whether/which rules are actually installed is not relevant.

If they were to be added here, the scenarios dealing with overwrite would look like:

Scenario: Importing an existing prebuilt rule is rejected by default
Scenario: Importing an existing prebuilt rule is allowed if overwrite is true


Scenario: Importing a prebuilt rule with a non-existent rule_id
Given the import payload contains a prebuilt rule with a non-existent rule_id
When the user imports the rule
Then the rule should be created
And the ruleSource type should be "internal"

Scenario: Importing a prebuilt rule without a rule_id field
Given the import payload contains a prebuilt rule without a rule_id field
When the user imports the rule
Then the import should be rejected with a message "rule_id field is required"

Scenario: Importing a prebuilt rule with a matching rule_id but missing a version field
Given the import payload contains a prebuilt rule without a version field
When the user imports the rule
Then the import should be rejected with a message "version field is required"

Scenario: Importing an existing custom rule missing a version field
Given the import payload contains an existing custom rule without a version field
When the user imports the rule
Then the rule should be updated
And the ruleSource type should be "internal"
And the "version" field should be set to the existing rule's "version"

Scenario: Importing a new custom rule missing a version field
Given the import payload contains a new custom rule without a version field
When the user imports the rule
Then the rule should be created
And the ruleSource type should be "internal"
And the "version" field should be set to 1

Scenario: Importing a rule with overwrite flag set to true
Given the import payload contains a rule with an existing rule_id
And the overwrite flag is set to true
When the user imports the rule
Then the rule should be overwritten
And the ruleSource type should be calculated based on the rule_id and version

Scenario: Importing a rule with overwrite flag set to false
Given the import payload contains a rule with an existing rule_id
And the overwrite flag is set to false
When the user imports the rule
Then the import should be rejected with a message "rule_id already exists"

Scenario: Importing both custom and prebuilt rules
Given the import payload contains modified and unmodified, custom and prebuilt rules
When the user imports the rule
Then custom rules should be created or updated, with versions defaulted to 1
And prebuilt rules should be created or updated,
And prebuilt rules missing versions should be rejected