[Security Solution] Error when importing a duplicated Endpoint Security rule with default exceptions #198461

banderror · 2024-10-31T03:16:43Z

Related to: #143864

Summary

If you try to import a duplicated Endpoint Security rule (so, it means it's a custom one) that has an exception item added to its default exception list (not the endpoint list), then overall the import fails, and:

the rule gets imported
the exception list doesn't

Steps to reproduce:

Install the prebuilt Endpoint Security rule.
Duplicate it. Delete the original Endpoint Security rule. Rename the duplicate from Endpoint Security [Duplicate] to Endpoint Security (not sure if the renaming matters).
At this point, you have a custom rule called Endpoint Security.
Add an exception item to its default exception list (Rule Details page -> Rule exceptions tab).
Export this rule. This will generate an ndjson file containing the rule + its default exception list + it's exception item.
Delete the Endpoint Security rule.
Try to import the ndjson file.
See the error.

Details

Use this minimal ndjson to reproduce the bug:

{"id":"ed9732ef-5a40-4f9c-8f91-524598b59b59","updated_at":"2024-10-31T02:56:54.261Z","updated_by":"elastic","created_at":"2024-10-31T02:55:36.680Z","created_by":"elastic","name":"Endpoint Security","tags":["Data Source: Elastic Defend"],"interval":"5m","enabled":false,"revision":3,"description":"Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.","risk_score":47,"severity":"medium","license":"Elastic License v2","output_index":"","meta":{"from":"5m","kibana_siem_app_url":""},"rule_name_override":"message","timestamp_override":"event.ingested","timestamp_override_fallback_disabled":false,"author":["Elastic"],"false_positives":[],"from":"now-600s","rule_id":"e456b162-a64f-4b62-a344-ca3cd589cf6d","max_signals":10000,"risk_score_mapping":[{"field":"event.risk_score","operator":"equals","value":""}],"severity_mapping":[{"field":"event.severity","operator":"equals","severity":"low","value":"21"},{"field":"event.severity","operator":"equals","severity":"medium","value":"47"},{"field":"event.severity","operator":"equals","severity":"high","value":"73"},{"field":"event.severity","operator":"equals","severity":"critical","value":"99"}],"threat":[],"to":"now","references":[],"version":103,"exceptions_list":[{"id":"endpoint_list","list_id":"endpoint_list","type":"endpoint","namespace_type":"agnostic"},{"id":"2618dcb5-7bb2-4436-9c68-fd75208feaa4","list_id":"3fa90098-d556-4c38-863f-644c99766126","type":"rule_default","namespace_type":"single"}],"immutable":false,"rule_source":{"type":"internal"},"related_integrations":[{"package":"endpoint","version":"^8.2.0"}],"required_fields":[],"setup":"## Setup\n\nThis rule is configured to generate more **Max alerts per run** than the default 1000 alerts per run set for all rules. This is to ensure that it captures as many alerts as possible.\n\n**IMPORTANT:** The rule's **Max alerts per run** setting can be superseded by the `xpack.alerting.rules.run.alerts.max` Kibana config setting, which determines the maximum alerts generated by _any_ rule in the Kibana alerting framework. For example, if `xpack.alerting.rules.run.alerts.max` is set to 1000, this rule will still generate no more than 1000 alerts even if its own **Max alerts per run** is set higher.\n\nTo make sure this rule can generate as many alerts as it's configured in its own **Max alerts per run** setting, increase the `xpack.alerting.rules.run.alerts.max` system setting accordingly.\n\n**NOTE:** Changing `xpack.alerting.rules.run.alerts.max` is not possible in Serverless projects.","type":"query","language":"kuery","index":["logs-endpoint.alerts-*"],"query":"event.kind:alert and event.module:(endpoint and not endgame)\n","filters":[],"actions":[]}
{"_version":"WzE1MjQsMV0=","created_at":"2024-10-31T02:56:12.355Z","created_by":"elastic","description":"Exception list containing exceptions for rule with id: ed9732ef-5a40-4f9c-8f91-524598b59b59","id":"2618dcb5-7bb2-4436-9c68-fd75208feaa4","immutable":false,"list_id":"3fa90098-d556-4c38-863f-644c99766126","name":"Exceptions for rule - Endpoint Security [Duplicate]","namespace_type":"single","os_types":[],"tags":["default_rule_exception_list"],"tie_breaker_id":"5bc67024-9c58-4912-b78d-c1962593f7dd","type":"rule_default","updated_at":"2024-10-31T02:56:12.355Z","updated_by":"elastic","version":1}
{"_version":"WzE1MjUsMV0=","comments":[],"created_at":"2024-10-31T02:56:12.878Z","created_by":"elastic","description":"Exception list item","entries":[{"type":"match","field":"test","value":"test","operator":"included"}],"id":"3f478fb9-39b2-421d-b64d-689ed3c77503","item_id":"ee0b2657-e4bf-4d02-9528-d92e5eb8aa26","list_id":"3fa90098-d556-4c38-863f-644c99766126","name":"test","namespace_type":"single","os_types":[],"tags":[],"tie_breaker_id":"f1391be9-a2a9-4134-bbad-035a91cdc9f9","type":"simple","updated_at":"2024-10-31T02:56:12.878Z","updated_by":"elastic"}
{"exported_count":3,"exported_rules_count":1,"missing_rules":[],"missing_rules_count":0,"exported_exception_list_count":1,"exported_exception_list_item_count":1,"missing_exception_list_item_count":0,"missing_exception_list_items":[],"missing_exception_lists":[],"missing_exception_lists_count":0,"exported_action_connector_count":0,"missing_action_connection_count":0,"missing_action_connections":[],"excluded_action_connection_count":0,"excluded_action_connections":[]}

When you try to import it, it will call the following import API endpoint which will respond with the following error:

POST /api/detection_engine/rules/_import?overwrite=false&overwrite_exceptions=false&overwrite_action_connectors=false

{
    "exceptions_success": false,
    "exceptions_success_count": 0,
    "exceptions_errors": [
        {
            "list_id": "3fa90098-d556-4c38-863f-644c99766126",
            "error": {
                "status_code": 409,
                "message": "Found that list_id: \"3fa90098-d556-4c38-863f-644c99766126\" already exists. Import of list_id: \"3fa90098-d556-4c38-863f-644c99766126\" skipped."
            }
        },
        {
            "list_id": "3fa90098-d556-4c38-863f-644c99766126",
            "item_id": "ee0b2657-e4bf-4d02-9528-d92e5eb8aa26",
            "error": {
                "status_code": 409,
                "message": "Found that item_id: \"ee0b2657-e4bf-4d02-9528-d92e5eb8aa26\" already exists. Import of item_id: \"ee0b2657-e4bf-4d02-9528-d92e5eb8aa26\" skipped."
            }
        }
    ],
    "rules_count": 1,
    "success": false,
    "success_count": 1,
    "errors": [
        {
            "rule_id": "e456b162-a64f-4b62-a344-ca3cd589cf6d",
            "error": {
                "status_code": 400,
                "message": "Rule with rule_id: \"e456b162-a64f-4b62-a344-ca3cd589cf6d\" references a non existent exception list of list_id: \"endpoint_list\". Reference has been removed."
            }
        },
        {
            "rule_id": "e456b162-a64f-4b62-a344-ca3cd589cf6d",
            "error": {
                "status_code": 400,
                "message": "Rule with rule_id: \"e456b162-a64f-4b62-a344-ca3cd589cf6d\" references a non existent exception list of list_id: \"3fa90098-d556-4c38-863f-644c99766126\". Reference has been removed."
            }
        }
    ],
    "action_connectors_errors": [],
    "action_connectors_warnings": [],
    "action_connectors_success": true,
    "action_connectors_success_count": 0
}

The text was updated successfully, but these errors were encountered:

elasticmachine · 2024-10-31T03:16:46Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2024-10-31T03:16:46Z

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine · 2024-10-31T03:16:46Z

Pinging @elastic/security-detection-engine (Team:Detection Engine)

pborgonovi · 2024-10-31T15:21:40Z

I've reproduced this scenario by updating/not updating rule name to validate observation in step 2:

not sure if the renaming matters

Updating rule name:

Not updating rule name:

Same error is observed in all the scenarios: existing list_id exception error is thrown, rule is imported and reference is removed:

{
    "exceptions_success": false,
    "exceptions_success_count": 0,
    "exceptions_errors": [
        {
            "list_id": "a3b80051-f69e-4988-88df-6a5d1575affc",
            "error": {
                "status_code": 409,
                "message": "Found that list_id: \"a3b80051-f69e-4988-88df-6a5d1575affc\" already exists. Import of list_id: \"a3b80051-f69e-4988-88df-6a5d1575affc\" skipped."
            }
        },
        {
            "list_id": "a3b80051-f69e-4988-88df-6a5d1575affc",
            "item_id": "f78f75eb-ed1e-4d6f-96ac-0990fd71bba8",
            "error": {
                "status_code": 409,
                "message": "Found that item_id: \"f78f75eb-ed1e-4d6f-96ac-0990fd71bba8\" already exists. Import of item_id: \"f78f75eb-ed1e-4d6f-96ac-0990fd71bba8\" skipped."
            }
        }
    ],
    "rules_count": 1,
    "success": false,
    "success_count": 1,
    "errors": [
        {
            "rule_id": "c72725e0-0d35-4059-ba2b-2fa240c9d3a3",
            "error": {
                "status_code": 400,
                "message": "Rule with rule_id: \"c72725e0-0d35-4059-ba2b-2fa240c9d3a3\" references a non existent exception list of list_id: \"a3b80051-f69e-4988-88df-6a5d1575affc\". Reference has been removed."
            }
        }
    ],
    "action_connectors_errors": [],
    "action_connectors_warnings": [],
    "action_connectors_success": true,
    "action_connectors_success_count": 0
}

…le types of exception lists (elastic#198868) ## Summary Fixes elastic#198461 When a rule import file has both single-namespace and namespace-agnostic exception lists, there was a bug in the logic that fetched the existing exception lists after importing them. A missing set of parentheses caused a KQL query that should have read `(A OR B) AND (C OR D)` to be `(A OR B) AND C OR D`, meaning that the logic was satisfied by `D` alone instead of requiring `A` or `B` to be true along with `D`. In this case `A` and `B` are filters on `exception-list` and `exception-list-agnostic` SO attributes so that we (should) only be looking at the list container objects, i.e. `exception-list.attributes.list_type: list`. `C` and `D` are filters by `list_id`, e.g. `exception-list.attributes.list_id: (test_list_id)`. Without the extra parentheses around `C OR D`, the query finds both `list` and `item` documents for the list IDs specified in `D`. When the `findExceptionList` logic encounters a list item unexpectedly, it still tries to convert the SO into our internal representation of an exception list with `transformSavedObjectToExceptionList`. Most fields are shared between lists and items, which makes it confusing to debug. However, the `type` of items can only be `simple`, whereas lists have a variety of types. During the conversion, the `type` field of the resulting object is defaulted to `detection` if the `type` field of the SO doesn't match the allowed list type values. Since the related SDH involved importing a `rule_default` exception list instead, the list types didn't match up when the import route compared the exception list on the rule to import vs the "existing list" (which was actually a list item coerced into a list container schema with `type: detection`) and import fails. (cherry picked from commit 0cc2e56)

…le types of exception lists (elastic#198868) ## Summary Fixes elastic#198461 When a rule import file has both single-namespace and namespace-agnostic exception lists, there was a bug in the logic that fetched the existing exception lists after importing them. A missing set of parentheses caused a KQL query that should have read `(A OR B) AND (C OR D)` to be `(A OR B) AND C OR D`, meaning that the logic was satisfied by `D` alone instead of requiring `A` or `B` to be true along with `D`. In this case `A` and `B` are filters on `exception-list` and `exception-list-agnostic` SO attributes so that we (should) only be looking at the list container objects, i.e. `exception-list.attributes.list_type: list`. `C` and `D` are filters by `list_id`, e.g. `exception-list.attributes.list_id: (test_list_id)`. Without the extra parentheses around `C OR D`, the query finds both `list` and `item` documents for the list IDs specified in `D`. When the `findExceptionList` logic encounters a list item unexpectedly, it still tries to convert the SO into our internal representation of an exception list with `transformSavedObjectToExceptionList`. Most fields are shared between lists and items, which makes it confusing to debug. However, the `type` of items can only be `simple`, whereas lists have a variety of types. During the conversion, the `type` field of the resulting object is defaulted to `detection` if the `type` field of the SO doesn't match the allowed list type values. Since the related SDH involved importing a `rule_default` exception list instead, the list types didn't match up when the import route compared the exception list on the rule to import vs the "existing list" (which was actually a list item coerced into a list container schema with `type: detection`) and import fails.

banderror assigned yctercero Oct 31, 2024

banderror added the sdh-linked label Oct 31, 2024

yctercero assigned marshallmain and unassigned yctercero Oct 31, 2024

marshallmain removed the triage_needed label Nov 4, 2024

marshallmain mentioned this issue Nov 4, 2024

[Security Solution][Detection Engine] Fix importing rules with multiple types of exception lists #198868

Merged

marshallmain closed this as completed in #198868 Nov 13, 2024

marshallmain closed this as completed in 0cc2e56 Nov 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] Error when importing a duplicated Endpoint Security rule with default exceptions #198461

[Security Solution] Error when importing a duplicated Endpoint Security rule with default exceptions #198461

banderror commented Oct 31, 2024 •

edited

Loading

elasticmachine commented Oct 31, 2024

elasticmachine commented Oct 31, 2024

elasticmachine commented Oct 31, 2024

pborgonovi commented Oct 31, 2024 •

edited

Loading

[Security Solution] Error when importing a duplicated Endpoint Security rule with default exceptions #198461

[Security Solution] Error when importing a duplicated Endpoint Security rule with default exceptions #198461

Comments

banderror commented Oct 31, 2024 • edited Loading

Summary

Details

elasticmachine commented Oct 31, 2024

elasticmachine commented Oct 31, 2024

elasticmachine commented Oct 31, 2024

pborgonovi commented Oct 31, 2024 • edited Loading

banderror commented Oct 31, 2024 •

edited

Loading

pborgonovi commented Oct 31, 2024 •

edited

Loading