[Data Usage] updates to metrics api #195640

neptunian · 2024-10-09T16:05:56Z

validates autoOps response data using mock data and new type
processes autoOps data to return an object of {x,y} values from our API instead of array of [timestamp, value]. updates UI accordingly
requires data_streams from the /metrics api
changes the /metrics route from GET to POST, better suited for more complex and larger payloads

neptunian · 2024-10-09T18:37:19Z

x-pack/plugins/data_usage/server/routes/internal/usage_metrics_handler.ts

          expand_wildcards: 'all',
        });

-      const hasDataStreams = dataStreamsResponse.length > 0;


esClient.indices.getDataStream will throw an error if a requested data stream is not found. since data streams are required, the logic below would not be met.

x-pack/plugins/data_usage/server/routes/internal/usage_metrics_handler.ts

ashokaditya · 2024-10-10T12:22:13Z

x-pack/plugins/data_usage/public/app/data_usage.tsx

+    dataStreams: [
+      '.alerts-ml.anomaly-detection-health.alerts-default',
+      '.alerts-stack.alerts-default',
+    ],


Q: With respect to UX, should we use all the datastreams from the API here and have them all selected when we land on the page even if the list is very long, say 100 or more?

Probably a question for a PM. Personally, I would say the "top" 10 would suffice. Otherwise we are potentially making the initial experience very slow unnecessarily.

Yeah, that is my concern. Also since we're pulling the datastreams list from another API (/_metering/stats) that doesn't allow for paging, I'm not sure how we're going to handle cases when users have a huge list of data streams.

At the moment I think the plan is to not handle it and just display them all in the dropdown but only select the largest by storage size of 10. The dropdown with all the streams will ideally scroll and not go off the page.

ashokaditya · 2024-10-10T12:32:30Z

x-pack/plugins/data_usage/common/rest_types/usage_metrics.ts

-        }),
-      ])
-    ),
+    dataStreams: schema.arrayOf(schema.string(), {


This schema has to be a oneOf(string, array). It should work with api/path?dataStreams=xyz and with api/path?dataStreams=xyz&dataStreams=pqr. The first one is a string and the second one is an array. So you can use the previous schema def without the maybe wrapper.

Right. I think in my head I was thinking this should be a POST request and we could simplify it and metricTypes to be arrays only. Given they can pass in as many data streams as they want I think we should probably switch to a POST as I can see this URL getting very large. wdyt?

x-pack/plugins/data_usage/server/routes/internal/usage_metrics_handler.ts

ashokaditya · 2024-10-10T14:31:11Z

x-pack/plugins/data_usage/common/rest_types/usage_metrics.ts

+      if (trimmedValues.some((v) => !v.length)) {
+        return '[metricTypes] list cannot contain empty values';
+      } else if (trimmedValues.some((v) => !isValidMetricType(v))) {
+        return `[metricTypes] must be one of ${METRIC_TYPE_VALUES.join(', ')}`;
+      }


Should also validate a string input instead of an array then the error should say that it expects and array and not a string. could not parse array value from json input is not valuable info to fix the request input, IMO.

ashokaditya · 2024-10-10T14:31:42Z

x-pack/plugins/data_usage/common/rest_types/usage_metrics.ts

+  dataStreams: schema.arrayOf(schema.string(), {
+    minSize: 1,
+    validate: (values) => {
+      if (values.map((v) => v.trim()).some((v) => !v.length)) {


same validation here for a string intput instead of array

ashokaditya · 2024-10-10T17:28:58Z

x-pack/plugins/data_usage/public/app/data_usage.tsx

@@ -42,37 +41,37 @@ export const DataUsage = () => {
    setUrlDateRangeFilter,
  } = useDataUsageMetricsUrlParams();

-  const [queryParams, setQueryParams] = useState<UsageMetricsRequestSchemaQueryParams>({
+  const [metricsFilters, setMetricsFilters] = useState<UsageMetricsRequestSchemaQueryParams>({


note: This file is renamed in my PR. you can leave it here as it is and I'll resolve conflicts in my PR.

kibanamachine · 2024-10-10T20:00:08Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11281103237

elasticmachine · 2024-10-10T20:00:19Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 5ffe666

Failed CI Steps

Jest Tests #1

Test Failures

[job] [logs] Jest Tests #1 / Description it renders

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`dataUsage`	229.9KB	229.7KB	-210.0B

History

💔 Build #241277 failed ba52d99
💔 Build #241202 failed a574727
💔 Build #240868 failed 1163d1f
💚 Build #240794 succeeded 08b6a92

- validates autoOps response data using mock data and new type - processes autoOps data to return an object of {x,y} values from our API instead of array of [timestamp, value]. updates UI accordingly (cherry picked from commit 3ec1908)

kibanamachine · 2024-10-10T20:05:03Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

# Backport This will backport the following commits from `main` to `8.x`: - [[Data Usage] process autoops mock data (#195640)](#195640)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Sandra G <[email protected]>

process autoops mock data

08b6a92

neptunian added release_note:skip Skip the PR/issue when compiling release notes backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) labels Oct 9, 2024

neptunian requested review from a team as code owners October 9, 2024 16:05

neptunian enabled auto-merge (squash) October 9, 2024 17:23

make data_streams required for metrics api

1163d1f

neptunian changed the title ~~[Data Usage] process autoops mock data~~ [Data Usage] updates to metrics api Oct 9, 2024

neptunian commented Oct 9, 2024

View reviewed changes

ashokaditya reviewed Oct 9, 2024

View reviewed changes

x-pack/plugins/data_usage/server/routes/internal/usage_metrics_handler.ts Show resolved Hide resolved

ashokaditya reviewed Oct 9, 2024

View reviewed changes

x-pack/plugins/data_usage/server/routes/internal/usage_metrics_handler.ts Show resolved Hide resolved

ashokaditya reviewed Oct 10, 2024

View reviewed changes

neptunian added 3 commits October 10, 2024 09:12

fix tests

2efd672

remove object

5746492

change metrics endpoint from GET to POST and update tests

a574727

ashokaditya reviewed Oct 10, 2024

View reviewed changes

ashokaditya approved these changes Oct 10, 2024

View reviewed changes

Merge branch 'main' into data-usage-process-data-autoops

ba52d99

neptunian mentioned this pull request Oct 10, 2024

[DataUsage][Serverless] Data usage metrics page enhancements #195556

Merged

6 tasks

ashokaditya reviewed Oct 10, 2024

View reviewed changes

Merge branch 'main' into data-usage-process-data-autoops

5ffe666

neptunian merged commit 3ec1908 into elastic:main Oct 10, 2024
21 checks passed

kibanamachine added the v9.0.0 label Oct 10, 2024

kibanamachine mentioned this pull request Oct 10, 2024

[8.x] [Data Usage] process autoops mock data (#195640) #195819

Merged

kibanamachine added the v8.16.0 label Oct 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Data Usage] updates to metrics api #195640

[Data Usage] updates to metrics api #195640

neptunian commented Oct 9, 2024 •

edited by kibanamachine

Loading

neptunian Oct 9, 2024

ashokaditya Oct 10, 2024

neptunian Oct 10, 2024 •

edited

Loading

ashokaditya Oct 10, 2024

neptunian Oct 10, 2024 •

edited

Loading

ashokaditya Oct 10, 2024

neptunian Oct 10, 2024

ashokaditya Oct 10, 2024

ashokaditya Oct 10, 2024

ashokaditya Oct 10, 2024

kibanamachine commented Oct 10, 2024

elasticmachine commented Oct 10, 2024

kibanamachine commented Oct 10, 2024

[Data Usage] updates to metrics api #195640

[Data Usage] updates to metrics api #195640

Conversation

neptunian commented Oct 9, 2024 • edited by kibanamachine Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neptunian Oct 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neptunian Oct 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibanamachine commented Oct 10, 2024

elasticmachine commented Oct 10, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Async chunks

History

kibanamachine commented Oct 10, 2024

💚 All backports created successfully

Questions ?

neptunian commented Oct 9, 2024 •

edited by kibanamachine

Loading

neptunian Oct 10, 2024 •

edited

Loading

neptunian Oct 10, 2024 •

edited

Loading