[Security Solution] Users can Customize Prebuilt Detection Rules: Milestone 4 (DRAFT)

[banderror]

Epic: https://github.com/elastic/security-team/issues/1974 (internal)
Milestones: << • >>

Status: Draft.

Summary

Milestone 4: Improve prebuilt rule customization, upgrade, and installation UX.

This meta ticket is created to simplify tracking of various tickets related to the epic, and to make this public information so our users can track the progress.

Useful info:

• RFC: Prebuilt Rules Customization - a new technical design document describing in detail how prebuilt rule customization should work and affect other existing features.
• Prebuilt Rules Customization Technical Design - an older document from Milestone 1.

Product and UX improvements

Rule customization UX

Preview Give feedback

1. Show customized prebuilt rules in the Rules table. A customized rule should be marked with some indicator.
2. Rule Details page: show what fields are customized and what are these customizations exactly (e.g. show a diff between the base and the current versions)
3. Ability to reset customizations and revert back to the base version
4. [Security Solution] Include namespace of source indices in index patterns of prebuilt rules #183616
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement needs product +6
5. Add BE-side validation for ESQL query field
6. Add BE-side validation for EQL query field
7. Add BE-side validation for KQL query field

Rule installation and upgrade UX

Preview Give feedback

1. [Security Solution] Show callouts when new prebuilt rules or rule updates are available #160270
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +4
2. [Security Solution] Implement filtering of Rule Installation and Rule Upgrade tables by Elastic Rules and Customized Elastic rules (DRAFT) #180396
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp triage_needed +5
3. [Security Solution] Inform users about missing base versions during prebuilt rule upgrade #186880
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement +5
4. Implement a preview of the applied changes in the Rule Upgrade Flyout and show it before calling the upgrade/_perform API endpoint
5. [Security Solution] Indicate conflicting rules in the Rule Upgrade table (DRAFT) #190500
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp needs product +5
   
6. [Security Solution] Implement UI for updating prebuilt rule to a new rule type (proper solution, DRAFT) #203145
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement needs product +6
   

Rule upgrade, diff algorithms

Preview Give feedback

1. [Security Solution] Implement array of objects diff algorithm #180161
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement +5
   
2. [Security Solution] Implement MITRE ATT&CK® field diff algorithm #187660
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement +5
   
3. [Security Solution] Implement query filters diff algorithm #190241
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement +5
   
4. [Security Solution] Assign proper diff algorithms to all rule fields #148191
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +4
   

"Last Updated" field in the UI

Preview Give feedback

1. [FR] Add source_updated_at to Rule Schema as a Build Time Field detection-rules#2826
   backlog enhancement v8.10 +3
2. [Security Solution] Add source_updated_at field to PrebuiltRuleAsset #176286
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +4
3. [Security Solution] Add source_updated_at field to RuleResponse via ResponseFields #174740
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +4
4. [Security Solution] [Prebuilt Rules Customization] Add "Last Updated" column in the Add Elastic Rules and Rule Updates tables #174767
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management +3

Bugs

Bugs: rule installation and upgrade

Preview Give feedback

1. [Security Solution] Installing prebuilt rules by 2 users at the same time causes rules to be duplicated (DRAFT) #180196
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug triage_needed +6
2. [Security Solution] Prebuilt rules being duplicated on upgrade #174847
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug impact:high +6
3. [Security Solution] Rule Update failure on 8.13 from 7.17.18 #177852
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug impact:medium +6
   
4. [Security Solution] Error when upgrading a rule that has an action referencing a deleted connector #198771
   Feature:Prebuilt Detection Rules Feature:Rule Actions Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug impact:medium needs product sdh-linked +9
5. [Security Solution] Rule is not updated and is followed by 'Rule failed to update' message when user attempts to upgrade a rule linked to a deleted shared exception list #198845
   Feature:Prebuilt Detection Rules Feature:Rule Exceptions Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug impact:medium +7
6. [Security Solution] Placeholder “Semver” Appears in the Related Integrations “Version” Text Field in Rule Upgrade Flyout #200079
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug impact:low +6
7. [Security Solution] Changes in Final Column Persist After Dismissal but Are Lost Upon Navigating Tabs #202799
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug impact:low +6
8. [Security Solution] Inconsistent naming for prebuilt rules across rule installation and flows #202783
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug impact:low ui-copy +7
   

Bugs: rule import and export

Preview Give feedback

1. [Security Solution] Rule import creates extra rules while importing a large number of rules #176207
   Feature:Rule Import/Export Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug impact:high +7
   
2. [Security Solution] Prevent creation of rules with the same rule_id when importing rules concurrently #177283
   Feature:Rule Import/Export Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug impact:high +6
3. [Security Solution] Error when exporting a rule that has an action referencing a deleted connector #178221
   Feature:Rule Actions Feature:Rule Import/Export Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug impact:medium needs product +8
4. https://github.com/elastic/security-team/issues/8644

Bugs: misc

Preview Give feedback

1. [Security Solution] Large number of related integrations can impact performance on rule creation/edit form #183607
   Feature:Rule Creation Feature:Rule Edit Team: SecuritySolution Team:Detection Engine Team:Detection Rule Management Team:Detections and Resp bug impact:medium performance +9
   
2. [Security Solution] See what's new in Prebuilt Security Detection Rules link on Add Elastic Rules page works with clicking anywhere on the blank space available on the right side of the page. #194275
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug impact:low +6

Technical improvements and debt

Schema migration from immutable to rule_source

Preview Give feedback

1. [Security Solution] Detection rule migration mechanism #187651
   Feature:Alerting/RulesFramework Feature:Rule Management Team: SecuritySolution Team:Detection Engine Team:Detection Rule Management Team:Detections and Resp Team:ResponseOps +7
2. [Security Solution] Migrate security rules encrypted saved objects to new schema with ruleSource field (BLOCKED) #184113
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp blocked +5
3. [Security Solution] Update rule searching to search by rule_source (BLOCKED) #180126
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp blocked +5
4. [Security Solution] Mark immutable as optional in the internal rule schema and stop writing it to rules (BLOCKED) #182573
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp blocked +5
5. [Security Solution] Deprecate the immutable field (DRAFT)(BLOCKED) #180269
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp triage_needed +5

Fleet package with prebuilt rules

Preview Give feedback

1. [Security Solution] Implement reliable tests to catch OOMs during rules package installation #188090
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp technical debt +5
2. [Security Solution] Alternative mechanism for distributing prebuilt rules #187649
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp needs product +5
3. [Security Solution] Splitting the package with prebuilt rules #187648
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp needs product +5
4. Support "content-only" integrations package-spec#351
   Team:Ecosystem discuss +2

Refactoring

Preview Give feedback

1. [Security Solution] Create types or tests to link Rule Schema and Diffable Rule schema #194484
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +4
   
2. [Security Solution] Apply same defaults values in extractDiffableCommonFields as in convertCreateAPIToInternalSchema #180165
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp refactoring +5
   
3. [Security Solution] DetectionRulesClient refactoring. Part 3 #187656
   Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp refactoring +5
4. [Security Solution] Implement a Zod transformation from snake_case to camelCase and vice versa #180121
   Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp technical debt +4
5. [Security Solution] Improve rule diff algorithm performance by splitting diff calculation into async chunks #180164
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp performance refactoring +6
   

Tests

Preview Give feedback

1. [Security Solution] Move existing test plans of Rules Management team from Google docs to Kibana repo #180451
   Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp technical debt test-plan +6
   
2. [Security Solution] Write tests for filtering, sorting, pagination in the Rule Installation and Upgrade tables #166215
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp technical debt test-coverage +6

Performance

Preview Give feedback

1. [Security Solution] /upgrade/_perform performance improvements #199101
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp performance +5
2. [Security Solution] Create asynchronous rules import API #195633
   Feature:Prebuilt Detection Rules Feature:Rule Import/Export Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +5
3. Import API: revisit [Security Solution] Move calculation of rule source outside of applyRuleUpdate #199720 and reimplement properly from the software design standpoint
4. Import API: implement bulk rule creation / updates in the RulesClient and use it in the import endpoint

Misc

Preview Give feedback

No tasks being tracked yet.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)