[Security Solution] Prebuilt rules being duplicated on upgrade

[zkink]

Epics: https://github.com/elastic/security-team/issues/1974 (internal), #179907
Kibana version: 8.11.3
Elasticsearch version: 8.11.3
Server OS version: Cloud
Browser version: Edge
Browser OS version: 120.0.2210.133
Original install method (e.g. download page, yum, from source, etc.): Cloud

Summary

Describe the bug:
When updating rules, instances appear to be duplicating some rules as part of the update. Within our cluster, we can see the same rule exist multiple times. Both rules appear to be prebuilt rules that have been created by the author "Elastic". This is a few example rules where this can be seen to have occurred.

Steps to reproduce:
Go to Detection Rules (SIEM)
Click on Rule Updates
Click on Update All
Once updates have been performed, sometimes (this appears to be an interment issue), rules can be found in Custom rather than prebuilt.

Expected behavior:
What would be expected is for rules to update rather than creating a new record/ID.

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:
This appears to be a long-term issue that has been intermittent. It appears only sometimes when rules are updated, and looking in a different instance, a different rule could be duplicated. When looking, it appears to be randomly occurring rather than occurring for the same rule each time.

Reference: elastic/detection-rules#3383

Rule with same Rule ID existing in instance with 2 different alerts IDS:
"hits": [
{
"_index": ".kibana_alerting_cases_8.8.0_001",
"_id": "alert:1b6df8c0-2599-11ee-8571-99b83d216ac8",
"_score": 26.424747,
"_source": {
"alert": {
"params": {
"ruleId": "cac91072-d165-11ec-a764-f661ea17fbce"
}
}
}
},
{
"_index": ".kibana_alerting_cases_8.8.0_001",
"_id": "alert:d76ede10-402c-11ee-bc86-b1f97eb72d5d",
"_score": 26.424747,
"_source": {
"alert": {
"params": {
"ruleId": "cac91072-d165-11ec-a764-f661ea17fbce"
}
}

[zkink]

@jpdjere

Requested ticket be moved to this repo

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[jpdjere]

Reference from elastic/detection-rules#3383

Duplicated rules:
Enumeration of Kernel Modules
Rule ID 1: aeffacb0-199f-11ee-be89-1b8cf30adee4
Rule ID 2: 2d8aaff0-4027-11ee-bc86-b1f97eb72d5d

Abnormal Process ID or Lock File Created
Rule ID 1: 1b6df8c0-2599-11ee-8571-99b83d216ac8
Rule ID 2: d76ede10-402c-11ee-bc86-b1f97eb72d5d

[jpdjere]

@zkink

Thanks for reopening the ticket in this repo, and for the provided data on the duplicated rules. One more ask: could you tell me which version of the Prebuilt Rules package are/were you on? Right before and after the rule updates appeared. As your deployment is on cloud, the Prebuilt Rules package is independent from the Kibana version, as it is distributed out-of-band via Fleet. You can navigate to the Integrations page and search for Prebuilt Security Detection Rules. Click on it to see the current installed version.

[zkink]

We are currently on 8.10.9. This has not been updated for a few of the more recent versions due to finding issues like this in the past few updates that we have performed.

As far of versions, I am unsure where some of these occurred from as we have seen it intermittently for different rules in different environments over the last year.

[jpdjere]

Hi @zkink

I'm investigating a possible bug of rules being duplicated while updating in cases in which the updates change a rule's type (for example, from EQL to New Terms). Would you mind running one more request for me in an instance where you have duplicated rules and paste the result here?

GET .kibana_alerting_cases/_search
{
  "_source": [
    "alert.params.ruleId",
    "alert.params.type",
    "alert.params.version",
    "alert.name"
  ],
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "type": "alert"
          }
        },
        {
          "match_phrase": {
            "alert.name": "Abnormal Process ID or Lock File Created" // <--- name of the duplicated rule here
          }
        }
      ]
    }
  }
}

[jpdjere]

Also @zkink , just wanted to note that what you are describing was a known issue that was fixed in 8.9. See the first bullet in the Release Notes.

I'm moving forward with the investigation of these duplications-while-upgrading that you reported with the assumption that they happened in a version that was 8.9 or higher, but please double check and let me know if it wasn't. We might be seeing a bug that was fixed already.

[zkink]

Do you have a date for when 8.9 would of been released? I can go through and confirm rules we are seeing this with after that time window.

[jpdjere]

@zkink July 25th, 2023. https://github.com/elastic/kibana/releases/tag/v8.9.0

[zkink]

From what I can tell, we are seeing this activity with rule updates we performed on aug 21st. This is when it states the new rule was created at.

[jpdjere]

@zkink Is there a way you could check and be sure whether you had already updated Kibana to 8.9 by the time that you upgraded the rule and the duplication issue happened? It's a pretty close date, from the release on July 25th to August 21st, maybe the update to 8.9 had not yet taken place on your side.

If we can make sure that you were in a version previous to 8.9 on August 21st, we know that what you are reporting is a fixed bug; and you could have certainty that the issue won't be happening again.

[zkink]

IT appears that our updates for 8.9 occurred around the date of 8/3/2023

[tdurden82]

This behavior was observed when upgrading from 8.10.2 to 8.12.2. Deleting the older index made alerting functional again. Rules were duplicated in lists, but we were unable to view, edit, create, or delete rules.

{"statusCode":400,"error":"Bad Request","message":"alias [.kibana_alerting_cases_8.12.2] has more than one index associated with it [.kibana_alerting_cases_8.10.2_001, .kibana_alerting_cases_8.12.2_001], can't execute a single index op: illegal_argument_exception

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)