[Security Solution] Allow importing prebuilt rules at the API level

[jpdjere]

Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174168

Summary

• Allow importing prebuilt rules at the API level
• API users should be able to import both custom and prebuilt rules, including any customizations that might have been done over their modifiable fields.
• An imported prebuilt rule should continue to track the rule asset from security_detection_engine package if the rule_id matches. This means that a user will be able to upgrade a previously imported prebuilt rule, visualize any diffs on any customized fields, and continue to receive updates on the rule.
• To allow for importing of Elastic prebuilt rules, we will not rely on the rule_source or the immutable fields (which are not part of the import endpoint parameters), but we will rather calculate them dynamically based on the rule_id and version request parameters.

Acceptance criteria

• Changes need to be hidden behind prebuiltRulesCustomizationEnabled feature flag
• Modify the Rule Import schema as described in the RFC to allow the importing of prebuilt rules. (TBD: do we hide this change behind a FF?)
• Modify the import handler logic as follows:

From the RFC:

The logic to importing a rule is as follows:

• First, read the import endpoint request parameters rule_id and version. These two are two required parameters in the endpoint.

• Secondly, check that the security_detection_engine Fleet package is installed and install it if it isn't. We will need the package to be installed to check if a rule that is being imported is an Elastic prebuilt rule.

• Then, using the rule_id and version, attempt to fetch the corresponding security-rule asset from ES.

• If a matching rule_id and version is found, that means that the rule is an Elastic prebuilt rule, and we should therefore dynamically calculate the rule's rule_source field and its subfields:

  • type: should be always external since a matching external security-rule was found.
  • is_customized: should be calculated based on the differences between the security-rule asset's fields and the rule fields from the import request. If any of them are different, i.e. have diverged from the base version, is_customized should be true.

{ 
  immutable: true,
  ruleSource: {
    type: 'external',
    isCustomized: <either true or false, depending on payload>
  }
}

• Finally, using the import payload, plus the rule's security-rule asset fields and the calculated rule_source fields, create the rule, or update it if already exists in Kibana.

If a matching rule_id is found, but the version is not found, it means there are some versions of this prebuilt rule known to Kibana, which means we should identify the rule being imported as prebuilt. The prebuilt rules package has a limit on the number of historical rule versions, and we can't assume that for a given rule_id we will always have ALL historical versions available as security-rule assets.

In this case, we will set the rule's params to be:

{ 
  immutable: true,
  ruleSource: {
    type: 'external',
    isCustomized: false
  }
}

If a matching rule_id is NOT found, that means that the rule is a custom rule. And rule_source will be simply:

{ 
  immutable: false,
  ruleSource: {
    type: 'internal',
  }
}

And we can finally create a rule or update an existing rule using the request payload and the calculated rule_source field.

Background

1. See table for Importing rules, with expected import payload input and expected rule outputs
2. See how to handle different scenarios for the version parameter

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[banderror]

Hey @jpdjere, tomorrow we'll have Milestone 3 onboarding meetings with Ryland and Zhenia, and I think it would be great if we could finalize descriptions for all the tickets we're going to ask them to help us with. Could you please prioritize writing a description at least for this one before the meetings?

[banderror]

Implemented in:

• [Security Solution] Allow importing of prebuilt rules via the API #190198
• [Security Solution] Test plans for prebuilt rule import and export #191116

Follow-up work:

• [Security Solution] Benchmark performance of importing a large number of prebuilt rules #195632
• [Security Solution] Move calculation of rule source outside of applyRuleUpdate #199720