[Security Solution] Benchmark performance of importing a large number of prebuilt rules #195632
Labels
8.18 candidate
Feature:Prebuilt Detection Rules
Security Solution Prebuilt Detection Rules area
Feature:Rule Import/Export
Security Solution Detection Rule Import & Export workflow
Team:Detection Rule Management
Security Detection Rule Management Team
Team:Detections and Resp
Security Detection Response Team
Team: SecuritySolution
Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
v8.18.0
Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174168
Summary
With #180168, we're going to introduce additional logic to the import endpoint for calculating the rule source object. Some of this logic will be run once for a given import call, some of it will be run multiple times for each rule being imported. Some of it can be IO-heavy (installing the package, fetching historical rule versions and ids), some of it can be CPU-heavy (calculating a diff for each rule).
Based on our prior observations, the rules import endpoint times out when importing a large number of rules. I think the number can be around 2-3k rules. Now, with the additional logic, the endpoint is going to be even heavier and can start timing out with a lower number of rules in the ndjson payload.
We would like to:
If we find some easy performance optimizations to do, we might reconsider working on #195633.
Testing performance
Profiling
It could be done by sending APM data from your local Kibana to your personal remote Elastic APM and then using APM as a profiler. This remote Elastic APM can be spinned up in production cloud as a normal deployment with APM. Staging cloud is known to be problematic for this use case.
The text was updated successfully, but these errors were encountered: