[Security Solution] OOMs during prebuilt rules package installation

[xcrzx]

Epic: #174168

Summary

We've seen a number of incidents with OOMs when installing the security_detection_engine Fleet package that contains a significant number of prebuilt rules (> 5000 in Serverless or > 15000 locally).

We need to find and mitigate all memory bottlenecks during package installation to unblock the release of the prebuilt rule customization epic.

Known memory issues

Beta Give feedback

1. [Fleet] All package SOs are being read into memory during package upgrade or force installation #187975
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet technical debt +7
   
2. [Fleet] Split bulk delete into chunks to avoid high memory consumption #188208
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet technical debt +7
   
3. [Fleet] Split asset installation into chunks to avoid high memory pressure #189043
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet +6
   

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[kpollich]

@xcrzx - I wonder if we could introduce another option aside from force (which causes Fleet to uninstall all previous assets before installing new ones) that would allow Fleet to skip the initial deletion of old assets and just run the bulk import operation as an optimization here.

[xcrzx]

@xcrzx - I wonder if we could introduce another option aside from force (which causes Fleet to uninstall all previous assets before installing new ones) that would allow Fleet to skip the initial deletion of old assets and just run the bulk import operation as an optimization here.

Yes, incremental installation might help for our use case. I'm not sure how SO import handles conflicts when writing new assets, but if we can instruct it to ignore existing saved objects or filter them before passing them to the import function, that should work.

[banderror]

@xcrzx I think we can consider this done at this point. The issue with OOMs has been mitigated, and follow-up improvements will be done as part of #192350 and #187645.