Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Include all historical rule versions in the prebuilt rules package #4311

Closed
xcrzx opened this issue Dec 17, 2024 · 5 comments
Closed

Include all historical rule versions in the prebuilt rules package #4311

xcrzx opened this issue Dec 17, 2024 · 5 comments
Assignees
@shashank-elastic
Labels
bug Something isn't working Team: TRADE

Comments

@xcrzx
Copy link

xcrzx commented Dec 17, 2024

Since the rule package version 8.17.1 does not include all historical rule versions, I’ve downloaded all published rule packages compatible with Kibana 8.x and consolidated all previously published rule versions into a single package: security_detection_engine-8.17.2.zip

  • The package includes all rules from versions 1.0.1 to 8.17.1.
  • Total rule versions: 11,554.
  • It might contain deprecated rule versions; I have not checked for that.

I tested the package locally, and it resolves the issue with missing base rule versions observed earlier. We should use this package as the basis for releasing future packages that include the full rule history.
@xcrzx xcrzx added bug Something isn't working Team: TRADE labels Dec 17, 2024
@xcrzx xcrzx mentioned this issue Jan 6, 2025
Closed
@shashank-elastic
Copy link
Contributor

shashank-elastic commented Jan 7, 2025 •
edited
Loading

We have filtered the assets to exclude deprecated rules to use for 8.17.2 package

Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json
skipped deprecated: a5f0d057-d540-44f5-924d-c6a2ae92f045
skipped deprecated: 0c9a14d9-d65d-486f-9b5b-91e4e6b22bd0
skipped deprecated: ccc55af4-9882-4c67-87b4-449a7ae8079c
skipped deprecated: df959768-b0c9-4d45-988c-5606a2be8e5a
skipped deprecated: 699e9fdb-b77c-4c01-995c-1c15019b9c43
skipped deprecated: 231876e7-4d1f-4d63-a47c-47dd1acdc1cb
skipped deprecated: 1859ce38-6a50-422b-a5e8-636e231ea0cd
skipped deprecated: 2f0bae2d-bf20-4465-be86-1311addebaa3
skipped deprecated: 9cf7a0ae-2404-11ed-ae7d-f661ea17fbce
skipped deprecated: 2f0bae2d-bf20-4465-be86-1311addebaa3
skipped deprecated: 4b1a807a-4e7b-414e-8cea-24bf580f6fc5
skipped deprecated: 4b1a807a-4e7b-414e-8cea-24bf580f6fc5
skipped deprecated: 6f683345-bb10-47a7-86a7-71e9c24fb358
skipped deprecated: 2f0bae2d-bf20-4465-be86-1311addebaa3
skipped deprecated: da986d2c-ffbf-4fd6-af96-a88dbf68f386
skipped deprecated: 0c9a14d9-d65d-486f-9b5b-91e4e6b22bd0
skipped deprecated: 6506c9fd-229e-4722-8f0f-69be759afd2a
skipped deprecated: 231876e7-4d1f-4d63-a47c-47dd1acdc1cb
skipped deprecated: 7b08314d-47a0-4b71-ae4e-16544176924f
skipped deprecated: eb6a3790-d52d-11ec-8ce9-f661ea17fbce
skipped deprecated: eb6a3790-d52d-11ec-8ce9-f661ea17fbce
skipped deprecated: 28738f9f-7427-4d23-bc69-756708b5f624
skipped deprecated: 97da359b-2b61-4a40-b2e4-8fc48cf7a294
skipped deprecated: dd7f1524-643e-11ed-9e35-f661ea17fbcd
skipped deprecated: 2377946d-0f01-4957-8812-6878985f515d
skipped deprecated: ccc55af4-9882-4c67-87b4-449a7ae8079c
skipped deprecated: 43303fd4-4839-4e48-b2b2-803ab060758d
skipped deprecated: a5f0d057-d540-44f5-924d-c6a2ae92f045
skipped deprecated: d6450d4e-81c6-46a3-bd94-079886318ed5
skipped deprecated: 43303fd4-4839-4e48-b2b2-803ab060758d
skipped deprecated: 231876e7-4d1f-4d63-a47c-47dd1acdc1cb
skipped deprecated: 699e9fdb-b77c-4c01-995c-1c15019b9c43
skipped deprecated: 7b08314d-47a0-4b71-ae4e-16544176924f
skipped deprecated: 9cf7a0ae-2404-11ed-ae7d-f661ea17fbce
skipped deprecated: f52362cd-baf1-4b6d-84be-064efc826461
skipped deprecated: 10754992-28c7-4472-be5b-f3770fd04f2d
skipped deprecated: 4b1a807a-4e7b-414e-8cea-24bf580f6fc5
skipped deprecated: 041d4d41-9589-43e2-ba13-5680af75ebc2
skipped deprecated: a5f0d057-d540-44f5-924d-c6a2ae92f045
skipped deprecated: 4973e46b-a663-41b8-a875-ced16dda2bb0
skipped deprecated: e9b4a3c7-24fc-49fd-a00f-9c938031eef1
skipped deprecated: 72d33577-f155-457d-aad3-379f9b750c97
skipped deprecated: 0c9a14d9-d65d-486f-9b5b-91e4e6b22bd0
skipped deprecated: 699e9fdb-b77c-4c01-995c-1c15019b9c43
skipped deprecated: ccc55af4-9882-4c67-87b4-449a7ae8079c
skipped deprecated: fd3fc25e-7c7c-4613-8209-97942ac609f6
skipped deprecated: dd7f1524-643e-11ed-9e35-f661ea17fbcd
skipped deprecated: 699e9fdb-b77c-4c01-995c-1c15019b9c43
skipped deprecated: 28738f9f-7427-4d23-bc69-756708b5f624
skipped deprecated: eb6a3790-d52d-11ec-8ce9-f661ea17fbce
skipped deprecated: eb6a3790-d52d-11ec-8ce9-f661ea17fbce
skipped deprecated: 28738f9f-7427-4d23-bc69-756708b5f624
skipped deprecated: 231876e7-4d1f-4d63-a47c-47dd1acdc1cb
skipped deprecated: ccc55af4-9882-4c67-87b4-449a7ae8079c
skipped deprecated: 699e9fdb-b77c-4c01-995c-1c15019b9c43
skipped deprecated: 0c9a14d9-d65d-486f-9b5b-91e4e6b22bd0
skipped deprecated: 125417b8-d3df-479f-8418-12d7e034fee3
skipped deprecated: a5f0d057-d540-44f5-924d-c6a2ae92f045
skipped deprecated: 4973e46b-a663-41b8-a875-ced16dda2bb0
skipped deprecated: cab4f01c-793f-4a54-a03e-e5d85b96d7af
skipped deprecated: 041d4d41-9589-43e2-ba13-5680af75ebc2
skipped deprecated: 4b1a807a-4e7b-414e-8cea-24bf580f6fc5
skipped deprecated: 0c9a14d9-d65d-486f-9b5b-91e4e6b22bd0
skipped deprecated: 231876e7-4d1f-4d63-a47c-47dd1acdc1cb
skipped deprecated: 7b08314d-47a0-4b71-ae4e-16544176924f
skipped deprecated: 28896382-7d4f-4d50-9b72-67091901fd26
skipped deprecated: 90e28af7-1d96-4582-bf11-9a1eff21d0e5
skipped deprecated: 28738f9f-7427-4d23-bc69-756708b5f624
skipped deprecated: 2f0bae2d-bf20-4465-be86-1311addebaa3
skipped deprecated: 301571f3-b316-4969-8dd0-7917410030d3
skipped deprecated: 28896382-7d4f-4d50-9b72-67091901fd26
skipped deprecated: eb6a3790-d52d-11ec-8ce9-f661ea17fbce
skipped deprecated: eb6a3790-d52d-11ec-8ce9-f661ea17fbce
skipped deprecated: ccc55af4-9882-4c67-87b4-449a7ae8079c
skipped deprecated: 28738f9f-7427-4d23-bc69-756708b5f624
skipped deprecated: dd7f1524-643e-11ed-9e35-f661ea17fbcd
skipped deprecated: 8fed8450-847e-43bd-874c-3bbf0cd425f3
skipped deprecated: 28896382-7d4f-4d50-9b72-67091901fd26
skipped deprecated: 9cf7a0ae-2404-11ed-ae7d-f661ea17fbce
skipped deprecated: 699e9fdb-b77c-4c01-995c-1c15019b9c43
skipped deprecated: 43303fd4-4839-4e48-b2b2-803ab060758d
skipped deprecated: 0c9a14d9-d65d-486f-9b5b-91e4e6b22bd0
skipped deprecated: 041d4d41-9589-43e2-ba13-5680af75ebc2
skipped deprecated: 041d4d41-9589-43e2-ba13-5680af75ebc2
skipped deprecated: 4973e46b-a663-41b8-a875-ced16dda2bb0
skipped deprecated: a5f0d057-d540-44f5-924d-c6a2ae92f045
skipped deprecated: fb9937ce-7e21-46bf-831d-1ad96eac674d
skipped deprecated: 0c9a14d9-d65d-486f-9b5b-91e4e6b22bd0
skipped deprecated: d6450d4e-81c6-46a3-bd94-079886318ed5
skipped deprecated: 699e9fdb-b77c-4c01-995c-1c15019b9c43
skipped deprecated: dd7f1524-643e-11ed-9e35-f661ea17fbcd
skipped deprecated: 28738f9f-7427-4d23-bc69-756708b5f624
skipped deprecated: 0c9a14d9-d65d-486f-9b5b-91e4e6b22bd0
skipped deprecated: 041d4d41-9589-43e2-ba13-5680af75ebc2
skipped deprecated: eb6a3790-d52d-11ec-8ce9-f661ea17fbce
skipped deprecated: 28896382-7d4f-4d50-9b72-67091901fd26
skipped deprecated: 0c9a14d9-d65d-486f-9b5b-91e4e6b22bd0
skipped deprecated: 20dc4620-3b68-4269-8124-ca5091e00ea8
skipped deprecated: 4b1a807a-4e7b-414e-8cea-24bf580f6fc5
skipped deprecated: ccc55af4-9882-4c67-87b4-449a7ae8079c
skipped deprecated: 28738f9f-7427-4d23-bc69-756708b5f624
skipped deprecated: dd7f1524-643e-11ed-9e35-f661ea17fbcd
skipped deprecated: 2377946d-0f01-4957-8812-6878985f515d
skipped deprecated: 0968cfbd-40f0-4b1c-b7b1-a60736c7b241
skipped deprecated: ccc55af4-9882-4c67-87b4-449a7ae8079c
skipped deprecated: 83b2c6e5-e0b2-42d7-8542-8f3af86a1acb
skipped deprecated: 041d4d41-9589-43e2-ba13-5680af75ebc2
skipped deprecated: 4b1a807a-4e7b-414e-8cea-24bf580f6fc5
skipped deprecated: 86c3157c-a951-4a4f-989b-2f0d0f1f9518
skipped deprecated: e0dacebe-4311-4d50-9387-b17e89c2e7fd
skipped deprecated: 6ea71ff0-9e95-475b-9506-2580d1ce6154
skipped deprecated: 231876e7-4d1f-4d63-a47c-47dd1acdc1cb
skipped deprecated: 699e9fdb-b77c-4c01-995c-1c15019b9c43
skipped deprecated: 231876e7-4d1f-4d63-a47c-47dd1acdc1cb
skipped deprecated: 2f0bae2d-bf20-4465-be86-1311addebaa3
skipped deprecated: e0dacebe-4311-4d50-9387-b17e89c2e7fd
skipped deprecated: df959768-b0c9-4d45-988c-5606a2be8e5a
skipped deprecated: ee619805-54d7-4c56-ba6f-7717282ddd73
skipped deprecated: 041d4d41-9589-43e2-ba13-5680af75ebc2
skipped deprecated: 041d4d41-9589-43e2-ba13-5680af75ebc2
skipped deprecated: 89583d1b-3c2e-4606-8b74-0a9fd2248e88
skipped deprecated: 231876e7-4d1f-4d63-a47c-47dd1acdc1cb
skipped deprecated: 699e9fdb-b77c-4c01-995c-1c15019b9c43
skipped deprecated: 5e87f165-45c2-4b80-bfa5-52822552c997
skipped deprecated: ccc55af4-9882-4c67-87b4-449a7ae8079c
skipped deprecated: dd7f1524-643e-11ed-9e35-f661ea17fbcd
skipped deprecated: 28738f9f-7427-4d23-bc69-756708b5f624
skipped deprecated: e0dacebe-4311-4d50-9387-b17e89c2e7fd
skipped deprecated: 3605a013-6f0c-4f7d-88a5-326f5be262ec
Filtered files have been moved to filtered_all_assets.
(.venv) 
detection-rules on  main [$?] is 📦 v0.3.12 via 🐍 v3.12.5 (.venv) on ☁️  [email protected] took 5s 
❯ ls -ltr filtered_all_assets/ | wc -l 
   11425
(.venv) 
detection-rules on  main [$?] is 📦 v0.3.12 via 🐍 v3.12.5 (.venv) on ☁️  [email protected] 
❯

cc @Mikaayenson

@shashank-elastic shashank-elastic mentioned this issue Jan 7, 2025
Open
@shashank-elastic
Copy link
Contributor

When creating a beta package for 8.17.2 - elastic/integrations#12261

We observed many files having mismatched ID from file name that causes error like below


122. kibana object file [/opt/buildkite-agent/builds/bk-agent-prod-gcp-1736318837319164324/elastic/integrations/packages/security_detection_engine/kibana/security_rule/e88d1fe9-b2f4-48d4-bace-a026dc745d4b_7.json] defines non-matching ID [e88d1fe9-b2f4-48d4-bace-a026dc745d4b]
--
  | 123. kibana object file [/opt/buildkite-agent/builds/bk-agent-prod-gcp-1736318837319164324/elastic/integrations/packages/security_detection_engine/kibana/security_rule/ea09ff26-3902-4c53-bb8e-24b7a5d029dd_5.json] defines non-matching ID [ea09ff26-3902-4c53-bb8e-24b7a5d029dd]
  | 124. kibana object file [/opt/buildkite-agent/builds/bk-agent-prod-gcp-1736318837319164324/elastic/integrations/packages/security_detection_engine/kibana/security_rule/ebb200e8-adf0-43f8-a0bb-4ee5b5d852c6_311.json] defines non-matching ID [ebb200e8-adf0-43f8-a0bb-4ee5b5d852c6]
  | 125. kibana object file [/opt/buildkite-agent/builds/bk-agent-prod-gcp-1736318837319164324/elastic/integrations/packages/security_detection_engine/kibana/security_rule/ebfe1448-7fac-4d59-acea-181bd89b1f7f_111.json] defines non-matching ID [ebfe1448-7fac-4d59-acea-181bd89b1f7f]
  | 126. kibana object file [/opt/buildkite-agent/builds/bk-agent-prod-gcp-1736318837319164324/elastic/integrations/packages/security_detection_engine/kibana/security_rule/ee39a9f7-5a79-4b0a-9815-d36b3cf28d3e_103.json]

We used the below script to correct them

import json

def check_and_correct_json_files(directory):
    mismatched_files = []
    
    for filename in os.listdir(directory):
        if filename.endswith('.json'):
            filepath = os.path.join(directory, filename)
            with open(filepath, 'r') as file:
                try:
                    data = json.load(file)
                    file_id = filename.replace('.json', '')
                    if data.get('id') != file_id:
                        mismatched_files.append(filename)
                        data['id'] = file_id
                        with open(filepath, 'w') as corrected_file:
                            json.dump(data, corrected_file, indent=4)
                except json.JSONDecodeError:
                    print(f"Error decoding JSON in file: {filename}")
    
    return mismatched_files

# Example usage
directory_path = '/Users/shashankks/elastic_workspace/integrations/packages/security_detection_engine/kibana/security_rule'
mismatched_files = check_and_correct_json_files(directory_path)
if mismatched_files:
    print("Corrected mismatched files:")
    for file in mismatched_files:
        print(file)
else:
    print("All files had matching 'id' fields.")

@shashank-elastic
Copy link
Contributor

shashank-elastic commented Jan 8, 2025 •
edited
Loading

Once these corrections are done and adding all historical rules which are not depracated and the latest rules from current release we have about 11455 rules

❯ ls -ltr kibana/security_rule | wc -l
   11455

Spot checked some of the missing rules from issue -#4312

Image Image

These rules were present.

@shashank-elastic
Copy link
Contributor

BetPackage Live - https://epr.elastic.co/package/security_detection_engine/8.17.2-beta.2/ with 11455 Rules

@shashank-elastic
Copy link
Contributor

GA Package is available

Image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shashank-elastic shashank-elastic

Labels
bug Something isn't working Team: TRADE
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@xcrzx @shashank-elastic