[8.16] [Security Solution] [Timeline] Consolidate reduces, remove unn…

…eeded async/awaits, other small fixes (elastic#197168) (elastic#201456)

# Backport

This will backport the following commits from `main` to `8.16`:
- [[Security Solution] [Timeline] Consolidate reduces, remove unneeded
async/awaits, other small fixes
(elastic#197168)](elastic#197168)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Kevin
Qualters","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-22T18:24:54Z","message":"[Security
Solution] [Timeline] Consolidate reduces, remove unneeded async/awaits,
other small fixes (elastic#197168)\n\n## Summary\r\n\r\nFor most of 8.x, both
anecdotally from users and in development,\r\ntimeline search strategy
based apis would often seem slower than the\r\nequivalent search in
discover or elsewhere in kibana, and I have long\r\nsuspected that this
came from how the timeline sever code formatted the\r\nelasticsearch
responses for use in the UI, and while working on\r\nsomething else,
noticed even higher than normal occurrences in logs
of\r\n\"][http.server.Kibana] Event loop utilization
for\r\n/internal/search/timelineSearchStrategy exceeded threshold
of...\" and so\r\nI tried to refactor all of the functions in place as
much as possible,\r\nkeeping the apis similar, most of the unit tests,
etc, but removing as\r\nmany as possible of the Promise.alls, reduce
within reduce, etc. This\r\nhas lead to a substantial improvement in
performance, as you can see\r\nbelow, and with larger result sets, I
think the difference would only be\r\nmore noticeable.\r\n\r\nAfter
fix:\r\n~40 ms for formatTimelineData with ~1000 docs\r\n<img
width=\"1470\"
alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/c664f940-aa37-4335-9204-2a9300fbafa0\">\r\nBefore
fix:\r\n~18000 ms for formatTimelineData with ~1000 docs\r\n<img
width=\"1464\"
alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/124fa327-13b9-41ef-9489-8d27f853590c\">\r\n\r\n\r\n[chrome_profile_timeline_slow.cpuprofile](https://github.com/user-attachments/files/17825602/chrome_profile_timeline_slow.cpuprofile)\r\n\r\n[chrome_profile_timeline_fast.cpuprofile](https://github.com/user-attachments/files/17825606/chrome_profile_timeline_fast.cpuprofile)\r\nI've
attached the chrome devtools profiles for each, the time was\r\nmeasured
with the function:\r\n\r\n```\r\nasync function measureAwait<T>(promise:
Promise<T>, label: string): Promise<T> {\r\n const start =
performance.now();\r\n try {\r\n const result = await promise;\r\n const
duration = performance.now() - start;\r\n console.log(`${label} took
${duration}ms`);\r\n return result;\r\n } catch (error) {\r\n const
duration = performance.now() - start;\r\n console.log(`${label} failed
after ${duration}ms`);\r\n throw error;\r\n }\r\n}\r\n```\r\n\r\nWrapped
around the call to formatTimelineData
in\r\nx-pack/plugins/timelines/server/search_strategy/timeline/factory/events/all/index.ts\r\n\r\n\r\n###
Checklist\r\n\r\n- [ ] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"30fb8dd5bb97b5001030ed9eed355ab4fffc9070","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Threat
Hunting:Investigations","backport:prev-major"],"title":"[Security
Solution] [Timeline] Consolidate reduces, remove unneeded async/awaits,
other small
fixes","number":197168,"url":"https://github.com/elastic/kibana/pull/197168","mergeCommit":{"message":"[Security
Solution] [Timeline] Consolidate reduces, remove unneeded async/awaits,
other small fixes (elastic#197168)\n\n## Summary\r\n\r\nFor most of 8.x, both
anecdotally from users and in development,\r\ntimeline search strategy
based apis would often seem slower than the\r\nequivalent search in
discover or elsewhere in kibana, and I have long\r\nsuspected that this
came from how the timeline sever code formatted the\r\nelasticsearch
responses for use in the UI, and while working on\r\nsomething else,
noticed even higher than normal occurrences in logs
of\r\n\"][http.server.Kibana] Event loop utilization
for\r\n/internal/search/timelineSearchStrategy exceeded threshold
of...\" and so\r\nI tried to refactor all of the functions in place as
much as possible,\r\nkeeping the apis similar, most of the unit tests,
etc, but removing as\r\nmany as possible of the Promise.alls, reduce
within reduce, etc. This\r\nhas lead to a substantial improvement in
performance, as you can see\r\nbelow, and with larger result sets, I
think the difference would only be\r\nmore noticeable.\r\n\r\nAfter
fix:\r\n~40 ms for formatTimelineData with ~1000 docs\r\n<img
width=\"1470\"
alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/c664f940-aa37-4335-9204-2a9300fbafa0\">\r\nBefore
fix:\r\n~18000 ms for formatTimelineData with ~1000 docs\r\n<img
width=\"1464\"
alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/124fa327-13b9-41ef-9489-8d27f853590c\">\r\n\r\n\r\n[chrome_profile_timeline_slow.cpuprofile](https://github.com/user-attachments/files/17825602/chrome_profile_timeline_slow.cpuprofile)\r\n\r\n[chrome_profile_timeline_fast.cpuprofile](https://github.com/user-attachments/files/17825606/chrome_profile_timeline_fast.cpuprofile)\r\nI've
attached the chrome devtools profiles for each, the time was\r\nmeasured
with the function:\r\n\r\n```\r\nasync function measureAwait<T>(promise:
Promise<T>, label: string): Promise<T> {\r\n const start =
performance.now();\r\n try {\r\n const result = await promise;\r\n const
duration = performance.now() - start;\r\n console.log(`${label} took
${duration}ms`);\r\n return result;\r\n } catch (error) {\r\n const
duration = performance.now() - start;\r\n console.log(`${label} failed
after ${duration}ms`);\r\n throw error;\r\n }\r\n}\r\n```\r\n\r\nWrapped
around the call to formatTimelineData
in\r\nx-pack/plugins/timelines/server/search_strategy/timeline/factory/events/all/index.ts\r\n\r\n\r\n###
Checklist\r\n\r\n- [ ] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"30fb8dd5bb97b5001030ed9eed355ab4fffc9070"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/197168","number":197168,"mergeCommit":{"message":"[Security
Solution] [Timeline] Consolidate reduces, remove unneeded async/awaits,
other small fixes (elastic#197168)\n\n## Summary\r\n\r\nFor most of 8.x, both
anecdotally from users and in development,\r\ntimeline search strategy
based apis would often seem slower than the\r\nequivalent search in
discover or elsewhere in kibana, and I have long\r\nsuspected that this
came from how the timeline sever code formatted the\r\nelasticsearch
responses for use in the UI, and while working on\r\nsomething else,
noticed even higher than normal occurrences in logs
of\r\n\"][http.server.Kibana] Event loop utilization
for\r\n/internal/search/timelineSearchStrategy exceeded threshold
of...\" and so\r\nI tried to refactor all of the functions in place as
much as possible,\r\nkeeping the apis similar, most of the unit tests,
etc, but removing as\r\nmany as possible of the Promise.alls, reduce
within reduce, etc. This\r\nhas lead to a substantial improvement in
performance, as you can see\r\nbelow, and with larger result sets, I
think the difference would only be\r\nmore noticeable.\r\n\r\nAfter
fix:\r\n~40 ms for formatTimelineData with ~1000 docs\r\n<img
width=\"1470\"
alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/c664f940-aa37-4335-9204-2a9300fbafa0\">\r\nBefore
fix:\r\n~18000 ms for formatTimelineData with ~1000 docs\r\n<img
width=\"1464\"
alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/124fa327-13b9-41ef-9489-8d27f853590c\">\r\n\r\n\r\n[chrome_profile_timeline_slow.cpuprofile](https://github.com/user-attachments/files/17825602/chrome_profile_timeline_slow.cpuprofile)\r\n\r\n[chrome_profile_timeline_fast.cpuprofile](https://github.com/user-attachments/files/17825606/chrome_profile_timeline_fast.cpuprofile)\r\nI've
attached the chrome devtools profiles for each, the time was\r\nmeasured
with the function:\r\n\r\n```\r\nasync function measureAwait<T>(promise:
Promise<T>, label: string): Promise<T> {\r\n const start =
performance.now();\r\n try {\r\n const result = await promise;\r\n const
duration = performance.now() - start;\r\n console.log(`${label} took
${duration}ms`);\r\n return result;\r\n } catch (error) {\r\n const
duration = performance.now() - start;\r\n console.log(`${label} failed
after ${duration}ms`);\r\n throw error;\r\n }\r\n}\r\n```\r\n\r\nWrapped
around the call to formatTimelineData
in\r\nx-pack/plugins/timelines/server/search_strategy/timeline/factory/events/all/index.ts\r\n\r\n\r\n###
Checklist\r\n\r\n- [ ] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"30fb8dd5bb97b5001030ed9eed355ab4fffc9070"}}]}]
BACKPORT-->

Co-authored-by: Kevin Qualters <[email protected]>

packages/kbn-securitysolution-t-grid/src/mock/mock_event_details.ts

@@ -291,6 +291,29 @@ export const eventDetailsFormattedFields = [
     originalValue: [`{"lon":118.7778,"lat":32.0617}`],
     values: [`{"lon":118.7778,"lat":32.0617}`],
   },
+  {
+    category: 'threat',
+    field: 'threat.enrichments',
+    isObjectArray: true,
+    originalValue: [
+      '{"matched.field":["matched_field","other_matched_field"],"indicator.first_seen":["2021-02-22T17:29:25.195Z"],"indicator.provider":["yourself"],"indicator.type":["custom"],"matched.atomic":["matched_atomic"],"lazer":[{"great.field":["grrrrr"]},{"great.field":["grrrrr_2"]}]}',
+      '{"matched.field":["matched_field_2"],"indicator.first_seen":["2021-02-22T17:29:25.195Z"],"indicator.provider":["other_you"],"indicator.type":["custom"],"matched.atomic":["matched_atomic_2"],"lazer":[{"great.field":[{"wowoe":[{"fooooo":["grrrrr"]}],"astring":"cool","aNumber":1,"neat":true}]}]}',
+      '{"matched.field":["host.name"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["FFEtSYIBZ61VHL7LvV2j"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
+      '{"matched.field":["host.hostname"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
+      '{"matched.field":["host.architecture"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["x86_64"]}',
+      '{"matched.field":["host.name"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
+      '{"matched.field":["host.hostname"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["CFErSYIBZ61VHL7LIV1N"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
+    ],
+    values: [
+      '{"matched.field":["matched_field","other_matched_field"],"indicator.first_seen":["2021-02-22T17:29:25.195Z"],"indicator.provider":["yourself"],"indicator.type":["custom"],"matched.atomic":["matched_atomic"],"lazer":[{"great.field":["grrrrr"]},{"great.field":["grrrrr_2"]}]}',
+      '{"matched.field":["matched_field_2"],"indicator.first_seen":["2021-02-22T17:29:25.195Z"],"indicator.provider":["other_you"],"indicator.type":["custom"],"matched.atomic":["matched_atomic_2"],"lazer":[{"great.field":[{"wowoe":[{"fooooo":["grrrrr"]}],"astring":"cool","aNumber":1,"neat":true}]}]}',
+      '{"matched.field":["host.name"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["FFEtSYIBZ61VHL7LvV2j"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
+      '{"matched.field":["host.hostname"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
+      '{"matched.field":["host.architecture"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["x86_64"]}',
+      '{"matched.field":["host.name"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
+      '{"matched.field":["host.hostname"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["CFErSYIBZ61VHL7LIV1N"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
+    ],
+  },
   {
     category: 'threat',
     field: 'threat.enrichments.matched.field',
@@ -376,27 +399,4 @@ export const eventDetailsFormattedFields = [
     originalValue: ['FFEtSYIBZ61VHL7LvV2j', 'E1EtSYIBZ61VHL7Ltl3m', 'CFErSYIBZ61VHL7LIV1N'],
     values: ['FFEtSYIBZ61VHL7LvV2j', 'E1EtSYIBZ61VHL7Ltl3m', 'CFErSYIBZ61VHL7LIV1N'],
   },
-  {
-    category: 'threat',
-    field: 'threat.enrichments',
-    isObjectArray: true,
-    originalValue: [
-      '{"matched.field":["matched_field","other_matched_field"],"indicator.first_seen":["2021-02-22T17:29:25.195Z"],"indicator.provider":["yourself"],"indicator.type":["custom"],"matched.atomic":["matched_atomic"],"lazer":[{"great.field":["grrrrr"]},{"great.field":["grrrrr_2"]}]}',
-      '{"matched.field":["matched_field_2"],"indicator.first_seen":["2021-02-22T17:29:25.195Z"],"indicator.provider":["other_you"],"indicator.type":["custom"],"matched.atomic":["matched_atomic_2"],"lazer":[{"great.field":[{"wowoe":[{"fooooo":["grrrrr"]}],"astring":"cool","aNumber":1,"neat":true}]}]}',
-      '{"matched.field":["host.name"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["FFEtSYIBZ61VHL7LvV2j"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
-      '{"matched.field":["host.hostname"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
-      '{"matched.field":["host.architecture"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["x86_64"]}',
-      '{"matched.field":["host.name"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
-      '{"matched.field":["host.hostname"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["CFErSYIBZ61VHL7LIV1N"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
-    ],
-    values: [
-      '{"matched.field":["matched_field","other_matched_field"],"indicator.first_seen":["2021-02-22T17:29:25.195Z"],"indicator.provider":["yourself"],"indicator.type":["custom"],"matched.atomic":["matched_atomic"],"lazer":[{"great.field":["grrrrr"]},{"great.field":["grrrrr_2"]}]}',
-      '{"matched.field":["matched_field_2"],"indicator.first_seen":["2021-02-22T17:29:25.195Z"],"indicator.provider":["other_you"],"indicator.type":["custom"],"matched.atomic":["matched_atomic_2"],"lazer":[{"great.field":[{"wowoe":[{"fooooo":["grrrrr"]}],"astring":"cool","aNumber":1,"neat":true}]}]}',
-      '{"matched.field":["host.name"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["FFEtSYIBZ61VHL7LvV2j"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
-      '{"matched.field":["host.hostname"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
-      '{"matched.field":["host.architecture"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["x86_64"]}',
-      '{"matched.field":["host.name"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["E1EtSYIBZ61VHL7Ltl3m"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
-      '{"matched.field":["host.hostname"],"matched.index":["im"],"matched.type":["indicator_match_rule"],"matched.id":["CFErSYIBZ61VHL7LIV1N"],"matched.atomic":["MacBook-Pro-de-Gloria.local"]}',
-    ],
-  },
 ];