[Security Solution][Detection Engine] fixes IM rule failure when frozen tier node is not available #200621

vitaliidm · 2024-11-18T17:04:06Z

Summary

addresses https://github.com/elastic/security-team/issues/11117

How to test

Create a deployment with cold and frozen data tiers and use following commands to create index and ILM

Data tiers commands

PUT /_cluster/settings
{
  "persistent": {
    "indices.lifecycle.poll_interval": "10s"
  }
}


PUT /_ilm/policy/filtering_data_tiers
{
  "policy": {
    "phases": {
        "frozen": {
          "min_age": "10s",
          "actions": {
            "searchable_snapshot": {
              "snapshot_repository": "found-snapshots",
              "force_merge_index": true
            }
          }
        },
        "hot": {
          "min_age": "0ms",
          "actions": {
            "set_priority": {
              "priority": 100
            }
          }
        }
    }
  }
}


PUT /_index_template/filtering_data_tiers_template
{
  "index_patterns": [
    "filtering_data_tiers*"
  ],
  "template": {
    "settings": {
      "index.lifecycle.name": "filtering_data_tiers",
      "index.lifecycle.rollover_alias": "test-filtering_data_tiers"
    },
    "mappings": {
      "_meta": {
        "version": "1.6.0"
      },
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "host": {
          "properties": {
            "name": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        }
      }
    }
  }
}

PUT /filtering_data_tiers-000001
{
  "aliases": {
    "filtering_data_tiers": {
      "is_write_index": true
    }
  }
}


POST filtering_data_tiers/_doc
{
  "@timestamp": "2024-07-08T17:00:01.000Z",
  "host.name": "test-0"
}

Wait until document moves to frozen tier
Run another set of commands to persist document in hot tier

Data tiers commands

PUT /_ilm/policy/filtering_data_tiers
{
  "policy": {
    "phases": {
        "frozen": {
          "min_age": "100h",
          "actions": {
            "searchable_snapshot": {
              "snapshot_repository": "found-snapshots",
              "force_merge_index": true
            }
          }
        },
        "hot": {
          "min_age": "0ms",
          "actions": {
            "set_priority": {
              "priority": 100
            }
          }
        }
    }
  }
}


PUT /filtering_data_tiers-000002
{
  "aliases": {
    "filtering_data_tiers": {
      "is_write_index": true
    }
  }
}


POST filtering_data_tiers/_doc
{
  "@timestamp": "2024-11-08T17:00:01.000Z",
  "host.name": "test-1"
}

Pause frozen tier node (admin permissions needed for this) or increase memory of it, forcing node to become unavailable for short period of time.
Run IM rule with advanced setting filtering out frozen data tier
Rule should not fail and generate an alert from document in a hot tier

…en tier node is not available

elasticmachine · 2024-11-19T13:16:24Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2024-11-19T13:16:26Z

Pinging @elastic/security-detection-engine (Team:Detection Engine)

elasticmachine · 2024-11-19T15:08:04Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: b3fbf9d

Failed CI Steps

Jest Tests #17

Test Failures

[job] [logs] Jest Tests #17 / get_column_renderer should render a suricata row data when it is a suricata row

Metrics [docs]

✅ unchanged

History

💔 Build #252308 failed 1d035b2
💔 Build #252294 failed c9a0bb9

cc @vitaliidm

kibanamachine · 2024-11-20T10:00:32Z

Starting backport for target branches: 8.16, 8.18, 8.x

https://github.com/elastic/kibana/actions/runs/11931008234

@timestamp

…en tier node is not available (elastic#200621) ## Summary - addresses elastic/security-team#11117 ### How to test 1. Create a deployment with cold and frozen data tiers and use following commands to create index and ILM <details> <summary>Data tiers commands</summary> ```JSON PUT /_cluster/settings { "persistent": { "indices.lifecycle.poll_interval": "10s" } } PUT /_ilm/policy/filtering_data_tiers { "policy": { "phases": { "frozen": { "min_age": "10s", "actions": { "searchable_snapshot": { "snapshot_repository": "found-snapshots", "force_merge_index": true } } }, "hot": { "min_age": "0ms", "actions": { "set_priority": { "priority": 100 } } } } } } PUT /_index_template/filtering_data_tiers_template { "index_patterns": [ "filtering_data_tiers*" ], "template": { "settings": { "index.lifecycle.name": "filtering_data_tiers", "index.lifecycle.rollover_alias": "test-filtering_data_tiers" }, "mappings": { "_meta": { "version": "1.6.0" }, "properties": { "@timestamp": { "type": "date" }, "host": { "properties": { "name": { "type": "keyword", "ignore_above": 1024 } } } } } } } PUT /filtering_data_tiers-000001 { "aliases": { "filtering_data_tiers": { "is_write_index": true } } } POST filtering_data_tiers/_doc { "@timestamp": "2024-07-08T17:00:01.000Z", "host.name": "test-0" } ``` </details> 2. Wait until document moves to frozen tier 3. Run another set of commands to persist document in hot tier <details> <summary>Data tiers commands</summary> ```JSON PUT /_ilm/policy/filtering_data_tiers { "policy": { "phases": { "frozen": { "min_age": "100h", "actions": { "searchable_snapshot": { "snapshot_repository": "found-snapshots", "force_merge_index": true } } }, "hot": { "min_age": "0ms", "actions": { "set_priority": { "priority": 100 } } } } } } PUT /filtering_data_tiers-000002 { "aliases": { "filtering_data_tiers": { "is_write_index": true } } } POST filtering_data_tiers/_doc { "@timestamp": "2024-11-08T17:00:01.000Z", "host.name": "test-1" } ``` </details> 4. Pause frozen tier node (admin permissions needed for this) or increase memory of it, forcing node to become unavailable for short period of time. 5. Run IM rule with [advanced setting](https://www.elastic.co/guide/en/security/current/advanced-settings.html#exclude-cold-frozen-data-rule-executions) filtering out frozen data tier 6. Rule should not fail and generate an alert from document in a hot tier --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit ee397d6)

@timestamp

…en tier node is not available (elastic#200621) ## Summary - addresses elastic/security-team#11117 ### How to test 1. Create a deployment with cold and frozen data tiers and use following commands to create index and ILM <details> <summary>Data tiers commands</summary> ```JSON PUT /_cluster/settings { "persistent": { "indices.lifecycle.poll_interval": "10s" } } PUT /_ilm/policy/filtering_data_tiers { "policy": { "phases": { "frozen": { "min_age": "10s", "actions": { "searchable_snapshot": { "snapshot_repository": "found-snapshots", "force_merge_index": true } } }, "hot": { "min_age": "0ms", "actions": { "set_priority": { "priority": 100 } } } } } } PUT /_index_template/filtering_data_tiers_template { "index_patterns": [ "filtering_data_tiers*" ], "template": { "settings": { "index.lifecycle.name": "filtering_data_tiers", "index.lifecycle.rollover_alias": "test-filtering_data_tiers" }, "mappings": { "_meta": { "version": "1.6.0" }, "properties": { "@timestamp": { "type": "date" }, "host": { "properties": { "name": { "type": "keyword", "ignore_above": 1024 } } } } } } } PUT /filtering_data_tiers-000001 { "aliases": { "filtering_data_tiers": { "is_write_index": true } } } POST filtering_data_tiers/_doc { "@timestamp": "2024-07-08T17:00:01.000Z", "host.name": "test-0" } ``` </details> 2. Wait until document moves to frozen tier 3. Run another set of commands to persist document in hot tier <details> <summary>Data tiers commands</summary> ```JSON PUT /_ilm/policy/filtering_data_tiers { "policy": { "phases": { "frozen": { "min_age": "100h", "actions": { "searchable_snapshot": { "snapshot_repository": "found-snapshots", "force_merge_index": true } } }, "hot": { "min_age": "0ms", "actions": { "set_priority": { "priority": 100 } } } } } } PUT /filtering_data_tiers-000002 { "aliases": { "filtering_data_tiers": { "is_write_index": true } } } POST filtering_data_tiers/_doc { "@timestamp": "2024-11-08T17:00:01.000Z", "host.name": "test-1" } ``` </details> 4. Pause frozen tier node (admin permissions needed for this) or increase memory of it, forcing node to become unavailable for short period of time. 5. Run IM rule with [advanced setting](https://www.elastic.co/guide/en/security/current/advanced-settings.html#exclude-cold-frozen-data-rule-executions) filtering out frozen data tier 6. Rule should not fail and generate an alert from document in a hot tier --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit ee397d6)

kibanamachine · 2024-11-20T10:05:23Z

💔 Some backports could not be created

Status	Branch	Result
✅	8.16
❌	8.18	The branch "8.18" does not exist
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 200621

Questions ?

Please refer to the Backport tool documentation

@timestamp

…n frozen tier node is not available (#200621) (#200877) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution][Detection Engine] fixes IM rule failure when frozen tier node is not available (#200621)](#200621)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Vitalii Dmyterko <[email protected]>

kibanamachine · 2024-11-22T10:48:51Z

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

@timestamp

…en frozen tier node is not available (#200621) (#200876) # Backport This will backport the following commits from `main` to `8.16`: - [[Security Solution][Detection Engine] fixes IM rule failure when frozen tier node is not available (#200621)](#200621)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  --------- Co-authored-by: Vitalii Dmyterko <[email protected]> Co-authored-by: Paul Tavares <[email protected]>

@timestamp

…en tier node is not available (elastic#200621) ## Summary - addresses elastic/security-team#11117 ### How to test 1. Create a deployment with cold and frozen data tiers and use following commands to create index and ILM <details> <summary>Data tiers commands</summary> ```JSON PUT /_cluster/settings { "persistent": { "indices.lifecycle.poll_interval": "10s" } } PUT /_ilm/policy/filtering_data_tiers { "policy": { "phases": { "frozen": { "min_age": "10s", "actions": { "searchable_snapshot": { "snapshot_repository": "found-snapshots", "force_merge_index": true } } }, "hot": { "min_age": "0ms", "actions": { "set_priority": { "priority": 100 } } } } } } PUT /_index_template/filtering_data_tiers_template { "index_patterns": [ "filtering_data_tiers*" ], "template": { "settings": { "index.lifecycle.name": "filtering_data_tiers", "index.lifecycle.rollover_alias": "test-filtering_data_tiers" }, "mappings": { "_meta": { "version": "1.6.0" }, "properties": { "@timestamp": { "type": "date" }, "host": { "properties": { "name": { "type": "keyword", "ignore_above": 1024 } } } } } } } PUT /filtering_data_tiers-000001 { "aliases": { "filtering_data_tiers": { "is_write_index": true } } } POST filtering_data_tiers/_doc { "@timestamp": "2024-07-08T17:00:01.000Z", "host.name": "test-0" } ``` </details> 2. Wait until document moves to frozen tier 3. Run another set of commands to persist document in hot tier <details> <summary>Data tiers commands</summary> ```JSON PUT /_ilm/policy/filtering_data_tiers { "policy": { "phases": { "frozen": { "min_age": "100h", "actions": { "searchable_snapshot": { "snapshot_repository": "found-snapshots", "force_merge_index": true } } }, "hot": { "min_age": "0ms", "actions": { "set_priority": { "priority": 100 } } } } } } PUT /filtering_data_tiers-000002 { "aliases": { "filtering_data_tiers": { "is_write_index": true } } } POST filtering_data_tiers/_doc { "@timestamp": "2024-11-08T17:00:01.000Z", "host.name": "test-1" } ``` </details> 4. Pause frozen tier node (admin permissions needed for this) or increase memory of it, forcing node to become unavailable for short period of time. 5. Run IM rule with [advanced setting](https://www.elastic.co/guide/en/security/current/advanced-settings.html#exclude-cold-frozen-data-rule-executions) filtering out frozen data tier 6. Rule should not fail and generate an alert from document in a hot tier --------- Co-authored-by: kibanamachine <[email protected]>

@timestamp

…en tier node is not available (elastic#200621) ## Summary - addresses elastic/security-team#11117 ### How to test 1. Create a deployment with cold and frozen data tiers and use following commands to create index and ILM <details> <summary>Data tiers commands</summary> ```JSON PUT /_cluster/settings { "persistent": { "indices.lifecycle.poll_interval": "10s" } } PUT /_ilm/policy/filtering_data_tiers { "policy": { "phases": { "frozen": { "min_age": "10s", "actions": { "searchable_snapshot": { "snapshot_repository": "found-snapshots", "force_merge_index": true } } }, "hot": { "min_age": "0ms", "actions": { "set_priority": { "priority": 100 } } } } } } PUT /_index_template/filtering_data_tiers_template { "index_patterns": [ "filtering_data_tiers*" ], "template": { "settings": { "index.lifecycle.name": "filtering_data_tiers", "index.lifecycle.rollover_alias": "test-filtering_data_tiers" }, "mappings": { "_meta": { "version": "1.6.0" }, "properties": { "@timestamp": { "type": "date" }, "host": { "properties": { "name": { "type": "keyword", "ignore_above": 1024 } } } } } } } PUT /filtering_data_tiers-000001 { "aliases": { "filtering_data_tiers": { "is_write_index": true } } } POST filtering_data_tiers/_doc { "@timestamp": "2024-07-08T17:00:01.000Z", "host.name": "test-0" } ``` </details> 2. Wait until document moves to frozen tier 3. Run another set of commands to persist document in hot tier <details> <summary>Data tiers commands</summary> ```JSON PUT /_ilm/policy/filtering_data_tiers { "policy": { "phases": { "frozen": { "min_age": "100h", "actions": { "searchable_snapshot": { "snapshot_repository": "found-snapshots", "force_merge_index": true } } }, "hot": { "min_age": "0ms", "actions": { "set_priority": { "priority": 100 } } } } } } PUT /filtering_data_tiers-000002 { "aliases": { "filtering_data_tiers": { "is_write_index": true } } } POST filtering_data_tiers/_doc { "@timestamp": "2024-11-08T17:00:01.000Z", "host.name": "test-1" } ``` </details> 4. Pause frozen tier node (admin permissions needed for this) or increase memory of it, forcing node to become unavailable for short period of time. 5. Run IM rule with [advanced setting](https://www.elastic.co/guide/en/security/current/advanced-settings.html#exclude-cold-frozen-data-rule-executions) filtering out frozen data tier 6. Rule should not fail and generate an alert from document in a hot tier --------- Co-authored-by: kibanamachine <[email protected]>

[Security Solution][Detection Engine] fixes IM rule failure when froz…

d3af315

…en tier node is not available

vitaliidm self-assigned this Nov 18, 2024

[Security Solution][Detection Engine] add issue reference

c9a0bb9

kibanamachine and others added 3 commits November 19, 2024 10:36

[CI] Auto-commit changed files from 'make api-docs'

06f4b55

Merge branch 'main' into de_9_0/im-pit-new-flag

1d035b2

Merge branch 'main' into de_9_0/im-pit-new-flag

b3fbf9d

vitaliidm requested a review from marshallmain November 19, 2024 13:16

vitaliidm marked this pull request as ready for review November 19, 2024 13:16

vitaliidm requested a review from a team as a code owner November 19, 2024 13:16

marshallmain approved these changes Nov 19, 2024

View reviewed changes

vitaliidm merged commit ee397d6 into elastic:main Nov 20, 2024
44 checks passed

kibanamachine mentioned this pull request Nov 20, 2024

[8.16] [Security Solution][Detection Engine] fixes IM rule failure when frozen tier node is not available (#200621) #200876

Merged

kibanamachine mentioned this pull request Nov 20, 2024

[8.x] [Security Solution][Detection Engine] fixes IM rule failure when frozen tier node is not available (#200621) #200877

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Nov 22, 2024

kibanamachine added v8.16.2 and removed backport missing Added to PRs automatically when the are determined to be missing a backport. labels Nov 22, 2024

vitaliidm removed the v8.16.0 label Nov 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Detection Engine] fixes IM rule failure when frozen tier node is not available #200621

[Security Solution][Detection Engine] fixes IM rule failure when frozen tier node is not available #200621

vitaliidm commented Nov 18, 2024 •

edited by kibanamachine

Loading

elasticmachine commented Nov 19, 2024

elasticmachine commented Nov 19, 2024

elasticmachine commented Nov 19, 2024

kibanamachine commented Nov 20, 2024

kibanamachine commented Nov 20, 2024

kibanamachine commented Nov 22, 2024

[Security Solution][Detection Engine] fixes IM rule failure when frozen tier node is not available #200621

[Security Solution][Detection Engine] fixes IM rule failure when frozen tier node is not available #200621

Conversation

vitaliidm commented Nov 18, 2024 • edited by kibanamachine Loading

Summary

How to test

elasticmachine commented Nov 19, 2024

elasticmachine commented Nov 19, 2024

elasticmachine commented Nov 19, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

History

kibanamachine commented Nov 20, 2024

kibanamachine commented Nov 20, 2024

💔 Some backports could not be created

Manual backport

Questions ?

kibanamachine commented Nov 22, 2024

vitaliidm commented Nov 18, 2024 •

edited by kibanamachine

Loading