Adds support for event loop utilization to the core metrics service #153717

lukeelmers · 2023-03-24T22:48:57Z

This updates the metrics service with a collector for event loop utilization, based on some prior art by @Bamieh ❤️

To learn more about ELU, I recommend this article from nodesource.

Things included:

Introduces ELU metrics collector
Updates ops metrics logger to include utilization
Updates Kibana status page with a tile for utilization
- This wasn't strictly necessary, happy to drop it if folks think it isn't useful
/api/status, /api/stats are automatically updated as a result of this change

Things not included:

I didn't add a utilization threshold tracker as @Bamieh had put in his original POC.
- Main reasoning for this is that it's easy enough to add later, and I think we need to see this metric in production for awhile before we get a good idea of what a helpful threshold would be for logging purposes. I'm still happy to go back and add this if someone feels strongly about it.

Note to reviewers: I restructured one of the test files a bit, so reviewing will be easier if you view the diff with Hide whitespace enabled.

lukeelmers

self review

lukeelmers · 2023-03-24T22:50:20Z

packages/core/apps/core-apps-browser-internal/src/status/lib/load_status.ts

+        defaultMessage: 'Heap used out of {heapTotal}',
+        values: { heapTotal: numeral(metrics.process.memory.heap.size_limit).format('0.00 b') },


To avoid restructuring the whole status page, I combined the two separate heap tiles into one that includes both heapUsedInBytes and sizeLimit.

lukeelmers · 2023-03-24T22:52:49Z

packages/core/apps/core-apps-browser-internal/src/status/lib/load_status.ts

+        },
+      }),
+      value: metrics.process.event_loop_utilization.utilization,
+      type: 'float',


We're autoformatting floats to two decimal places... it's nice to see the utilization values with a bit more precision, but I don't know if it's worth a one-off implementation just for this metric. I tried updating all floats to five decimal places, but it blew up the layout, so I decided to leave it alone since this is a nice-to-have item anyway 🤷

Yeah, it's fine ihmo. That page has been a pain every time we want to change/add/remove anything...

lukeelmers · 2023-03-24T22:58:13Z

packages/core/metrics/core-metrics-server-internal/src/logging/get_ops_metrics_log.ts

@@ -55,6 +55,11 @@ export function getEcsOpsMetricsLog(metrics: OpsMetrics) {
      ).format('0.000')} }`
    : '';

+  const eventLoopUtilizationVal = process?.event_loop_utilization;
+  const eventLoopUtilizationMsg = eventLoopUtilizationVal
+    ? ` utilization: ${numeral(process?.event_loop_utilization.utilization).format('0.00000')}`


Including active/idle here didn't seem useful enough to justify the increased message length. But it is easy to add if anyone feels otherwise.

Not sure either to see how they could be useful. The utilization ratio seems the only valuable info

elasticmachine · 2023-03-27T21:50:36Z

Pinging @elastic/kibana-core (Team:Core)

lukeelmers · 2023-03-28T04:46:05Z

@elasticmachine merge upstream

pgayvallet

Looking good to me (once the tests are fixed!)

pgayvallet · 2023-03-28T07:42:35Z

packages/core/apps/core-apps-browser-internal/src/status/lib/load_status.ts

+        },
+      }),
+      value: metrics.process.event_loop_utilization.utilization,
+      type: 'float',


Yeah, it's fine ihmo. That page has been a pain every time we want to change/add/remove anything...

packages/core/metrics/core-metrics-collectors-server-internal/src/process.test.ts

pgayvallet · 2023-03-28T07:57:49Z

packages/core/metrics/core-metrics-server-internal/src/logging/get_ops_metrics_log.ts

@@ -55,6 +55,11 @@ export function getEcsOpsMetricsLog(metrics: OpsMetrics) {
      ).format('0.000')} }`
    : '';

+  const eventLoopUtilizationVal = process?.event_loop_utilization;
+  const eventLoopUtilizationMsg = eventLoopUtilizationVal
+    ? ` utilization: ${numeral(process?.event_loop_utilization.utilization).format('0.00000')}`


Not sure either to see how they could be useful. The utilization ratio seems the only valuable info

Bamieh

LGTM! It is good that you skipped using the ELU class I have from the POC because it is very specific to the collector (to avoid double logging and double incrementing the counters). Using the nodeJS eventLoopUtilization method directly as you did looks correct to me 👍

lukeelmers · 2023-03-29T04:13:20Z

@elasticmachine merge upstream

lukeelmers · 2023-03-29T20:16:19Z

@elasticmachine merge upstream

kibana-ci · 2023-03-29T21:35:56Z

💚 Build Succeeded

Buildkite Build
Commit: f89431a

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`core`	352.2KB	352.4KB	+285.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/core-metrics-server`	54	55	+1

ESLint disabled in files

id	before	after	diff
`@kbn/core-metrics-collectors-server-internal`	2	1	-1

ESLint disabled line counts

id	before	after	diff
`securitySolution`	432	435	+3

Total ESLint disabled count

id	before	after	diff
`@kbn/core-metrics-collectors-server-internal`	3	2	-1
`securitySolution`	512	515	+3
total			+2

History

💚 Build #116281 succeeded f361aee
💔 Build #115891 failed 4755065
💔 Build #115854 failed 09b4167
💛 Build #115826 was flaky 934f88bd735098a714d1d5787a1bb72bb8f9456e
💔 Build #115804 failed ac136c0b79748f3f93b19b9f2eb77d91eac28bc2
💔 Build #115479 failed 3eb01768e9a89d16c55a3750fd9e3c4e49f5fae1

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @lukeelmers

…lastic#153717)

lukeelmers commented Mar 24, 2023

View reviewed changes

lukeelmers force-pushed the feat/elu branch 3 times, most recently from efe9de7 to 934f88b Compare March 27, 2023 20:41

lukeelmers added v8.8.0 release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting labels Mar 27, 2023

lukeelmers added 3 commits March 27, 2023 15:49

Add ELU to core metrics service.

1461f3a

Add ELU to ops metrics logs.

23617c1

Add utilization to Kibana status page.

09b4167

lukeelmers force-pushed the feat/elu branch from 934f88b to 09b4167 Compare March 27, 2023 21:49

lukeelmers marked this pull request as ready for review March 27, 2023 21:50

lukeelmers requested a review from a team as a code owner March 27, 2023 21:50

lukeelmers added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Epic:KBNA-8605 labels Mar 27, 2023

lukeelmers self-assigned this Mar 27, 2023

Merge branch 'main' into feat/elu

4755065

pgayvallet approved these changes Mar 28, 2023

View reviewed changes

Bamieh approved these changes Mar 28, 2023

View reviewed changes

Bamieh mentioned this pull request Mar 28, 2023

[Telemetry] track event loop utilization #103477

Closed

Merge branch 'main' into feat/elu

f361aee

rayafratkina removed the Epic:KBNA-8605 label Mar 29, 2023

Clean up metrics collectors test mocks.

f778544

Merge branch 'main' into feat/elu

f89431a

lukeelmers enabled auto-merge (squash) March 29, 2023 20:17

lukeelmers merged commit 9517d06 into elastic:main Mar 29, 2023

lukeelmers deleted the feat/elu branch March 29, 2023 21:35

jgowdyelastic pushed a commit to jgowdyelastic/kibana that referenced this pull request Mar 30, 2023

Adds support for event loop utilization to the core metrics service (e…

f53bd78

…lastic#153717)

lukeelmers mentioned this pull request Apr 6, 2023

Add ELU fields to metricbeat Kibana module. elastic/beats#35020

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds support for event loop utilization to the core metrics service #153717

Adds support for event loop utilization to the core metrics service #153717

lukeelmers commented Mar 24, 2023 •

edited

Loading

lukeelmers left a comment

lukeelmers Mar 24, 2023

lukeelmers Mar 24, 2023

pgayvallet Mar 28, 2023

lukeelmers Mar 24, 2023

pgayvallet Mar 28, 2023

elasticmachine commented Mar 27, 2023

lukeelmers commented Mar 28, 2023

pgayvallet left a comment

pgayvallet Mar 28, 2023

pgayvallet Mar 28, 2023

Bamieh left a comment •

edited

Loading

lukeelmers commented Mar 29, 2023

lukeelmers commented Mar 29, 2023

kibana-ci commented Mar 29, 2023

API count

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

		defaultMessage: 'Heap used out of {heapTotal}',
		values: { heapTotal: numeral(metrics.process.memory.heap.size_limit).format('0.00 b') },

Adds support for event loop utilization to the core metrics service #153717

Adds support for event loop utilization to the core metrics service #153717

Conversation

lukeelmers commented Mar 24, 2023 • edited Loading

lukeelmers left a comment

Choose a reason for hiding this comment

lukeelmers Mar 24, 2023

Choose a reason for hiding this comment

lukeelmers Mar 24, 2023

Choose a reason for hiding this comment

pgayvallet Mar 28, 2023

Choose a reason for hiding this comment

lukeelmers Mar 24, 2023

Choose a reason for hiding this comment

pgayvallet Mar 28, 2023

Choose a reason for hiding this comment

elasticmachine commented Mar 27, 2023

lukeelmers commented Mar 28, 2023

pgayvallet left a comment

Choose a reason for hiding this comment

pgayvallet Mar 28, 2023

Choose a reason for hiding this comment

pgayvallet Mar 28, 2023

Choose a reason for hiding this comment

Bamieh left a comment • edited Loading

Choose a reason for hiding this comment

lukeelmers commented Mar 29, 2023

lukeelmers commented Mar 29, 2023

kibana-ci commented Mar 29, 2023

💚 Build Succeeded

Metrics [docs]

Page load bundle

API count

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

History

lukeelmers commented Mar 24, 2023 •

edited

Loading

Bamieh left a comment •

edited

Loading