Skip to content

Commit

Permalink
[8.x] [Security Solution] Allow exporting of prebuilt rules via the A…
Browse files Browse the repository at this point in the history
…PI (elastic#194498) (elastic#196447)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Security Solution] Allow exporting of prebuilt rules via the API
(elastic#194498)](elastic#194498)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Ryland
Herrick","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-15T21:26:25Z","message":"[Security
Solution] Allow exporting of prebuilt rules via the API (elastic#194498)\n\n##
Summary\r\n\r\nThis PR introduces the backend functionality necessary to
export\r\nprebuilt rules via our existing export APIs:\r\n\r\n1. Export
Rules - POST /rules/_export \r\n2. Bulk Actions - POST
/rules/_bulk_action \r\n\r\nThe [Prebuilt Rule
Customization\r\nRFC](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/docs/rfcs/detection_response/prebuilt_rules_customization.md)\r\ngoes
into detail, and the export-specific issue is
described\r\n[here](https://github.com/elastic/kibana/issues/180167#issue-2227974379).\r\n\r\n\r\n##
Steps to Review\r\n1. Enable the Feature Flag:
`prebuiltRulesCustomizationEnabled`\r\n1. Install the prebuilt rules
package via fleet \r\n1. Install some prebuilt rules, and obtain a
prebuilt rule's `rule_id`,\r\ne.g.
`ac8805f6-1e08-406c-962e-3937057fa86f`\r\n1. Export the rule via the
export route, e.g. (in Dev Tools):\r\n\r\n POST
kbn:api/detection_engine/rules/_export\r\n \r\nNote that you may need to
use the CURL equivalent for these requests, as\r\nthe dev console does
not seem to handle file responses:\r\n\r\ncurl --location --request
POST\r\n'http://localhost:5601/api/detection_engine/rules/_export?exclude_export_details=true&file_name=exported_rules.ndjson'\r\n\\\r\n
--header 'kbn-xsrf: true' \\\r\n --header 'elastic-api-version:
2023-10-31' \\\r\n --header 'Authorization: Basic
waefoijawoefiajweo=='\r\n\r\n1. Export the rule via bulk actions, e.g.
(in Dev Tools):\r\n\r\n POST
kbn:api/detection_engine/rules/_bulk_action\r\n {\r\n \"action\":
\"export\"\r\n }\r\n \r\n1. Observe that the exported rules' fields are
correct, especially\r\n`rule_source` and `immutable` (see tests added
here for examples).\r\n\r\n### Checklist\r\n\r\n- [
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [ ] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n### For maintainers\r\n\r\n- [ ]
This was checked for breaking API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"b67bd83ea93909d809206b1004c306a11fd8ee3f","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","v8.16.0","backport:version"],"title":"[Security Solution] Allow
exporting of prebuilt rules via the
API","number":194498,"url":"https://github.com/elastic/kibana/pull/194498","mergeCommit":{"message":"[Security
Solution] Allow exporting of prebuilt rules via the API (elastic#194498)\n\n##
Summary\r\n\r\nThis PR introduces the backend functionality necessary to
export\r\nprebuilt rules via our existing export APIs:\r\n\r\n1. Export
Rules - POST /rules/_export \r\n2. Bulk Actions - POST
/rules/_bulk_action \r\n\r\nThe [Prebuilt Rule
Customization\r\nRFC](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/docs/rfcs/detection_response/prebuilt_rules_customization.md)\r\ngoes
into detail, and the export-specific issue is
described\r\n[here](https://github.com/elastic/kibana/issues/180167#issue-2227974379).\r\n\r\n\r\n##
Steps to Review\r\n1. Enable the Feature Flag:
`prebuiltRulesCustomizationEnabled`\r\n1. Install the prebuilt rules
package via fleet \r\n1. Install some prebuilt rules, and obtain a
prebuilt rule's `rule_id`,\r\ne.g.
`ac8805f6-1e08-406c-962e-3937057fa86f`\r\n1. Export the rule via the
export route, e.g. (in Dev Tools):\r\n\r\n POST
kbn:api/detection_engine/rules/_export\r\n \r\nNote that you may need to
use the CURL equivalent for these requests, as\r\nthe dev console does
not seem to handle file responses:\r\n\r\ncurl --location --request
POST\r\n'http://localhost:5601/api/detection_engine/rules/_export?exclude_export_details=true&file_name=exported_rules.ndjson'\r\n\\\r\n
--header 'kbn-xsrf: true' \\\r\n --header 'elastic-api-version:
2023-10-31' \\\r\n --header 'Authorization: Basic
waefoijawoefiajweo=='\r\n\r\n1. Export the rule via bulk actions, e.g.
(in Dev Tools):\r\n\r\n POST
kbn:api/detection_engine/rules/_bulk_action\r\n {\r\n \"action\":
\"export\"\r\n }\r\n \r\n1. Observe that the exported rules' fields are
correct, especially\r\n`rule_source` and `immutable` (see tests added
here for examples).\r\n\r\n### Checklist\r\n\r\n- [
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [ ] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n### For maintainers\r\n\r\n- [ ]
This was checked for breaking API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"b67bd83ea93909d809206b1004c306a11fd8ee3f"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/194498","number":194498,"mergeCommit":{"message":"[Security
Solution] Allow exporting of prebuilt rules via the API (elastic#194498)\n\n##
Summary\r\n\r\nThis PR introduces the backend functionality necessary to
export\r\nprebuilt rules via our existing export APIs:\r\n\r\n1. Export
Rules - POST /rules/_export \r\n2. Bulk Actions - POST
/rules/_bulk_action \r\n\r\nThe [Prebuilt Rule
Customization\r\nRFC](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/docs/rfcs/detection_response/prebuilt_rules_customization.md)\r\ngoes
into detail, and the export-specific issue is
described\r\n[here](https://github.com/elastic/kibana/issues/180167#issue-2227974379).\r\n\r\n\r\n##
Steps to Review\r\n1. Enable the Feature Flag:
`prebuiltRulesCustomizationEnabled`\r\n1. Install the prebuilt rules
package via fleet \r\n1. Install some prebuilt rules, and obtain a
prebuilt rule's `rule_id`,\r\ne.g.
`ac8805f6-1e08-406c-962e-3937057fa86f`\r\n1. Export the rule via the
export route, e.g. (in Dev Tools):\r\n\r\n POST
kbn:api/detection_engine/rules/_export\r\n \r\nNote that you may need to
use the CURL equivalent for these requests, as\r\nthe dev console does
not seem to handle file responses:\r\n\r\ncurl --location --request
POST\r\n'http://localhost:5601/api/detection_engine/rules/_export?exclude_export_details=true&file_name=exported_rules.ndjson'\r\n\\\r\n
--header 'kbn-xsrf: true' \\\r\n --header 'elastic-api-version:
2023-10-31' \\\r\n --header 'Authorization: Basic
waefoijawoefiajweo=='\r\n\r\n1. Export the rule via bulk actions, e.g.
(in Dev Tools):\r\n\r\n POST
kbn:api/detection_engine/rules/_bulk_action\r\n {\r\n \"action\":
\"export\"\r\n }\r\n \r\n1. Observe that the exported rules' fields are
correct, especially\r\n`rule_source` and `immutable` (see tests added
here for examples).\r\n\r\n### Checklist\r\n\r\n- [
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [ ] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n### For maintainers\r\n\r\n- [ ]
This was checked for breaking API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"b67bd83ea93909d809206b1004c306a11fd8ee3f"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Ryland Herrick <[email protected]>
  • Loading branch information
kibanamachine and rylnd authored Oct 15, 2024
1 parent c54e6a9 commit 75f078b
Show file tree
Hide file tree
Showing 6 changed files with 379 additions and 15 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,8 @@ export const performBulkActionRoute = (
rules.map(({ params }) => params.ruleId),
exporter,
request,
actionsClient
actionsClient,
config.experimentalFeatures.prebuiltRulesCustomizationEnabled
);

const responseBody = `${exported.rulesNdjson}${exported.exceptionLists}${exported.actionConnectors}${exported.exportDetails}`;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,10 @@ import {
} from '../../../../../../../common/api/detection_engine/rule_management';
import type { SecuritySolutionPluginRouter } from '../../../../../../types';
import type { ConfigType } from '../../../../../../config';
import { getNonPackagedRulesCount } from '../../../logic/search/get_existing_prepackaged_rules';
import {
getNonPackagedRulesCount,
getRulesCount,
} from '../../../logic/search/get_existing_prepackaged_rules';
import { getExportByObjectIds } from '../../../logic/export/get_export_by_object_ids';
import { getExportAll } from '../../../logic/export/get_export_all';
import { buildSiemResponse } from '../../../../routes/utils';
Expand Down Expand Up @@ -57,6 +60,8 @@ export const exportRulesRoute = (

const client = getClient({ includedHiddenTypes: ['action'] });
const actionsExporter = getExporter(client);
const { prebuiltRulesCustomizationEnabled } = config.experimentalFeatures;

try {
const exportSizeLimit = config.maxRuleImportExportSize;
if (request.body?.objects != null && request.body.objects.length > exportSizeLimit) {
Expand All @@ -65,10 +70,19 @@ export const exportRulesRoute = (
body: `Can't export more than ${exportSizeLimit} rules`,
});
} else {
const nonPackagedRulesCount = await getNonPackagedRulesCount({
rulesClient,
});
if (nonPackagedRulesCount > exportSizeLimit) {
let rulesCount = 0;

if (prebuiltRulesCustomizationEnabled) {
rulesCount = await getRulesCount({
rulesClient,
filter: '',
});
} else {
rulesCount = await getNonPackagedRulesCount({
rulesClient,
});
}
if (rulesCount > exportSizeLimit) {
return siemResponse.error({
statusCode: 400,
body: `Can't export more than ${exportSizeLimit} rules`,
Expand All @@ -84,14 +98,16 @@ export const exportRulesRoute = (
request.body.objects.map((obj) => obj.rule_id),
actionsExporter,
request,
actionsClient
actionsClient,
prebuiltRulesCustomizationEnabled
)
: await getExportAll(
rulesClient,
exceptionsClient,
actionsExporter,
request,
actionsClient
actionsClient,
prebuiltRulesCustomizationEnabled
);

const responseBody = request.query.exclude_export_details
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import type { ISavedObjectsExporter, KibanaRequest } from '@kbn/core/server';
import type { ExceptionListClient } from '@kbn/lists-plugin/server';
import type { RulesClient } from '@kbn/alerting-plugin/server';
import type { ActionsClient } from '@kbn/actions-plugin/server';
import { getNonPackagedRules } from '../search/get_existing_prepackaged_rules';
import { getNonPackagedRules, getRules } from '../search/get_existing_prepackaged_rules';
import { getExportDetailsNdjson } from './get_export_details_ndjson';
import { transformAlertsToRules } from '../../utils/utils';
import { getRuleExceptionsForExport } from './get_export_rule_exceptions';
Expand All @@ -23,14 +23,18 @@ export const getExportAll = async (
exceptionsClient: ExceptionListClient | undefined,
actionsExporter: ISavedObjectsExporter,
request: KibanaRequest,
actionsClient: ActionsClient
actionsClient: ActionsClient,
prebuiltRulesCustomizationEnabled?: boolean
): Promise<{
rulesNdjson: string;
exportDetails: string;
exceptionLists: string | null;
actionConnectors: string;
prebuiltRulesCustomizationEnabled?: boolean;
}> => {
const ruleAlertTypes = await getNonPackagedRules({ rulesClient });
const ruleAlertTypes = prebuiltRulesCustomizationEnabled
? await getRules({ rulesClient, filter: '' })
: await getNonPackagedRules({ rulesClient });
const rules = transformAlertsToRules(ruleAlertTypes);

const exportRules = rules.map((r) => transformRuleToExportableFormat(r));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,15 +29,21 @@ export const getExportByObjectIds = async (
ruleIds: string[],
actionsExporter: ISavedObjectsExporter,
request: KibanaRequest,
actionsClient: ActionsClient
actionsClient: ActionsClient,
prebuiltRulesCustomizationEnabled?: boolean
): Promise<{
rulesNdjson: string;
exportDetails: string;
exceptionLists: string | null;
actionConnectors: string;
prebuiltRulesCustomizationEnabled?: boolean;
}> =>
withSecuritySpan('getExportByObjectIds', async () => {
const rulesAndErrors = await fetchRulesByIds(rulesClient, ruleIds);
const rulesAndErrors = await fetchRulesByIds(
rulesClient,
ruleIds,
prebuiltRulesCustomizationEnabled
);
const { rules, missingRuleIds } = rulesAndErrors;

// Retrieve exceptions
Expand Down Expand Up @@ -76,7 +82,8 @@ interface FetchRulesResult {

const fetchRulesByIds = async (
rulesClient: RulesClient,
ruleIds: string[]
ruleIds: string[],
prebuiltRulesCustomizationEnabled?: boolean
): Promise<FetchRulesResult> => {
// It's important to avoid too many clauses in the request otherwise ES will fail to process the request
// with `too_many_clauses` error (see https://github.com/elastic/kibana/issues/170015). The clauses limit
Expand Down Expand Up @@ -110,7 +117,7 @@ const fetchRulesByIds = async (

return matchingRule != null &&
hasValidRuleType(matchingRule) &&
matchingRule.params.immutable !== true
(prebuiltRulesCustomizationEnabled || matchingRule.params.immutable !== true)
? {
rule: transformRuleToExportableFormat(internalRuleToAPIResponse(matchingRule)),
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,6 @@ import { FtrProviderContext } from '../../../../../../ftr_provider_context';
export default ({ loadTestFile }: FtrProviderContext): void => {
describe('Rules Management - Prebuilt Rules - Update Prebuilt Rules Package', function () {
loadTestFile(require.resolve('./is_customized_calculation'));
loadTestFile(require.resolve('./rules_export'));
});
};
Loading

0 comments on commit 75f078b

Please sign in to comment.