Skip to content

Commit

Permalink
[Security Solution] [Timeline] Consolidate reduces, remove unneeded a…
Browse files Browse the repository at this point in the history
…sync/awaits, other small fixes (#197168)

## Summary

For most of 8.x, both anecdotally from users and in development,
timeline search strategy based apis would often seem slower than the
equivalent search in discover or elsewhere in kibana, and I have long
suspected that this came from how the timeline sever code formatted the
elasticsearch responses for use in the UI, and while working on
something else, noticed even higher than normal occurrences in logs of
"][http.server.Kibana] Event loop utilization for
/internal/search/timelineSearchStrategy exceeded threshold of..." and so
I tried to refactor all of the functions in place as much as possible,
keeping the apis similar, most of the unit tests, etc, but removing as
many as possible of the Promise.alls, reduce within reduce, etc. This
has lead to a substantial improvement in performance, as you can see
below, and with larger result sets, I think the difference would only be
more noticeable.

After fix:
~40 ms for formatTimelineData with ~1000 docs
<img width="1470" alt="image"
src="https://github.com/user-attachments/assets/c664f940-aa37-4335-9204-2a9300fbafa0">
Before fix:
~18000 ms for formatTimelineData with ~1000 docs
<img width="1464" alt="image"
src="https://github.com/user-attachments/assets/124fa327-13b9-41ef-9489-8d27f853590c">


[chrome_profile_timeline_slow.cpuprofile](https://github.com/user-attachments/files/17825602/chrome_profile_timeline_slow.cpuprofile)

[chrome_profile_timeline_fast.cpuprofile](https://github.com/user-attachments/files/17825606/chrome_profile_timeline_fast.cpuprofile)
I've attached the chrome devtools profiles for each, the time was
measured with the function:

```
async function measureAwait<T>(promise: Promise<T>, label: string): Promise<T> {
  const start = performance.now();
  try {
    const result = await promise;
    const duration = performance.now() - start;
    console.log(`${label} took ${duration}ms`);
    return result;
  } catch (error) {
    const duration = performance.now() - start;
    console.log(`${label} failed after ${duration}ms`);
    throw error;
  }
}
```

Wrapped around the call to formatTimelineData in
x-pack/plugins/timelines/server/search_strategy/timeline/factory/events/all/index.ts


### Checklist

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
  • Loading branch information
kqualters-elastic authored Nov 22, 2024
1 parent ad61b9d commit 30fb8dd
Show file tree
Hide file tree
Showing 20 changed files with 740 additions and 1,060 deletions.
46 changes: 23 additions & 23 deletions packages/kbn-securitysolution-t-grid/src/mock/mock_event_details.ts
Original file line number Diff line number Diff line change
Expand Up @@ -291,6 +291,29 @@ export const eventDetailsFormattedFields = [
originalValue: [`{"lon":118.7778,"lat":32.0617}`],
values: [`{"lon":118.7778,"lat":32.0617}`],
},
{
category: 'threat',
field: 'threat.enrichments',
isObjectArray: true,
originalValue: [
'{"matched.field":["matched_field","other_matched_field"],"indicator.first_seen":["2021-02-22T17:29:25.195Z"],"indicator.provider":["yourself"],"indicator.type":["custom"],"matched.atomic":["matched_atomic"],"lazer":[{"great.field":["grrrrr"]},{"great.field":["grrrrr_2"]}]}',
'{"matched.field":["matched_field_2"],"indicator.first_seen":["2021-02-22T17:29:25.195Z"],"indicator.provider":["other_you"],"indicator.type":["custom"],"matched.atomic":["matched_atomic_2"],"lazer":[{"great.field":[{"wowoe":[{"fooooo":["grrrrr"]}],"astring":"cool","aNumber":1,"neat":true}]}]}',
'{"matched.field":["host.name"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["FFEtSYIBZ61VHL7LvV2j"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
'{"matched.field":["host.hostname"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
'{"matched.field":["host.architecture"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["x86_64"]}',
'{"matched.field":["host.name"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
'{"matched.field":["host.hostname"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["CFErSYIBZ61VHL7LIV1N"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
],
values: [
'{"matched.field":["matched_field","other_matched_field"],"indicator.first_seen":["2021-02-22T17:29:25.195Z"],"indicator.provider":["yourself"],"indicator.type":["custom"],"matched.atomic":["matched_atomic"],"lazer":[{"great.field":["grrrrr"]},{"great.field":["grrrrr_2"]}]}',
'{"matched.field":["matched_field_2"],"indicator.first_seen":["2021-02-22T17:29:25.195Z"],"indicator.provider":["other_you"],"indicator.type":["custom"],"matched.atomic":["matched_atomic_2"],"lazer":[{"great.field":[{"wowoe":[{"fooooo":["grrrrr"]}],"astring":"cool","aNumber":1,"neat":true}]}]}',
'{"matched.field":["host.name"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["FFEtSYIBZ61VHL7LvV2j"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
'{"matched.field":["host.hostname"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
'{"matched.field":["host.architecture"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["x86_64"]}',
'{"matched.field":["host.name"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
'{"matched.field":["host.hostname"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["CFErSYIBZ61VHL7LIV1N"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
],
},
{
category: 'threat',
field: 'threat.enrichments.matched.field',
Expand Down Expand Up @@ -376,27 +399,4 @@ export const eventDetailsFormattedFields = [
originalValue: ['FFEtSYIBZ61VHL7LvV2j', 'E1EtSYIBZ61VHL7Ltl3m', 'CFErSYIBZ61VHL7LIV1N'],
values: ['FFEtSYIBZ61VHL7LvV2j', 'E1EtSYIBZ61VHL7Ltl3m', 'CFErSYIBZ61VHL7LIV1N'],
},
{
category: 'threat',
field: 'threat.enrichments',
isObjectArray: true,
originalValue: [
'{"matched.field":["matched_field","other_matched_field"],"indicator.first_seen":["2021-02-22T17:29:25.195Z"],"indicator.provider":["yourself"],"indicator.type":["custom"],"matched.atomic":["matched_atomic"],"lazer":[{"great.field":["grrrrr"]},{"great.field":["grrrrr_2"]}]}',
'{"matched.field":["matched_field_2"],"indicator.first_seen":["2021-02-22T17:29:25.195Z"],"indicator.provider":["other_you"],"indicator.type":["custom"],"matched.atomic":["matched_atomic_2"],"lazer":[{"great.field":[{"wowoe":[{"fooooo":["grrrrr"]}],"astring":"cool","aNumber":1,"neat":true}]}]}',
'{"matched.field":["host.name"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["FFEtSYIBZ61VHL7LvV2j"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
'{"matched.field":["host.hostname"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
'{"matched.field":["host.architecture"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["x86_64"]}',
'{"matched.field":["host.name"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
'{"matched.field":["host.hostname"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["CFErSYIBZ61VHL7LIV1N"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
],
values: [
'{"matched.field":["matched_field","other_matched_field"],"indicator.first_seen":["2021-02-22T17:29:25.195Z"],"indicator.provider":["yourself"],"indicator.type":["custom"],"matched.atomic":["matched_atomic"],"lazer":[{"great.field":["grrrrr"]},{"great.field":["grrrrr_2"]}]}',
'{"matched.field":["matched_field_2"],"indicator.first_seen":["2021-02-22T17:29:25.195Z"],"indicator.provider":["other_you"],"indicator.type":["custom"],"matched.atomic":["matched_atomic_2"],"lazer":[{"great.field":[{"wowoe":[{"fooooo":["grrrrr"]}],"astring":"cool","aNumber":1,"neat":true}]}]}',
'{"matched.field":["host.name"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["FFEtSYIBZ61VHL7LvV2j"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
'{"matched.field":["host.hostname"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
'{"matched.field":["host.architecture"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["x86_64"]}',
'{"matched.field":["host.name"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
'{"matched.field":["host.hostname"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["CFErSYIBZ61VHL7LIV1N"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
],
},
];

This file was deleted.

148 changes: 0 additions & 148 deletions x-pack/plugins/security_solution/common/utils/field_formatters.ts

This file was deleted.

89 changes: 0 additions & 89 deletions x-pack/plugins/security_solution/common/utils/to_array.ts

This file was deleted.

Loading

0 comments on commit 30fb8dd

Please sign in to comment.