Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fleet] All package SOs are being read into memory during package upgrade or force installation #187975

Closed
Tracked by #187969
, #174168
...
xcrzx opened this issue Jul 10, 2024 · 4 comments · Fixed by #188004
Closed
Tracked by #187969
, #174168
...

[Fleet] All package SOs are being read into memory during package upgrade or force installation #187975

xcrzx opened this issue Jul 10, 2024 · 4 comments · Fixed by #188004
Assignees
@xcrzx
Labels
8.16 candidate Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team:Fleet Team label for Observability Data Collection Fleet team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. technical debt Improvement of the software architecture and operational architecture

Comments

@xcrzx
Copy link
Contributor

xcrzx commented Jul 10, 2024

Related to: #187969

Summary

When upgrading or re-installing a package, all saved objects from a previous package are being loaded into memory using the bulkResolve method of the SO client:

kibana/x-pack/plugins/fleet/server/services/epm/packages/remove.ts

Lines 123 to 126 in 045aafc

const { resolved_objects: resolvedObjects } = await savedObjectsClient.bulkResolve(
installedObjects,
{ namespace }
);

This creates unnecessary memory pressure for packages containing thousands of saved objects, like the security_detection_engine package.

We can mitigate that by skipping the resolution altogether for all packages installed in Kibana 8.x. The resolution is only needed for package upgrades from 7.x to 8.x.
@xcrzx xcrzx added technical debt Improvement of the software architecture and operational architecture Team:Fleet Team label for Observability Data Collection Fleet team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules 8.16 candidate labels Jul 10, 2024
@xcrzx xcrzx self-assigned this Jul 10, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

This was referenced Jul 10, 2024
Closed
Open
Merged
xcrzx added a commit that referenced this issue Jul 12, 2024
@xcrzx
[Fleet] Avoid loading package saved objects into memory before deleti…
415ed2e 
…ng them (#188004)

**Resolves: #187975

## Summary

When upgrading or re-installing a package, all saved objects from a
previous package are loaded into memory using `bulkResolve`. This
creates unnecessary memory pressure for packages containing thousands of
saved objects, like the `security_detection_engine` package.

To mitigate that, we are now skipping saved object resolution for
packages known to be installed in `8.x`.

While testing locally on a package containing ~5000 detection rules, I
observed a significant drop in memory usage, from 1.17GB to 1.05GB at
peak.

**Before:**
![Screenshot 2024-07-11 at 11 11
06](https://github.com/elastic/kibana/assets/1938181/e535569e-03f0-404e-9937-4867839846ae)

**After:**
![Screenshot 2024-07-11 at 11 02
32](https://github.com/elastic/kibana/assets/1938181/889e6ade-9cd7-4cbc-a220-3e8bc507fba5)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xcrzx xcrzx

Labels
8.16 candidate Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team:Fleet Team label for Observability Data Collection Fleet team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. technical debt Improvement of the software architecture and operational architecture
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Fleet] Avoid loading package saved objects into memory before deleting them xcrzx/kibana
2 participants
@xcrzx @elasticmachine