Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fleet] Use metering API in serverless #200063

Merged
merged 4 commits into from
Nov 18, 2024

Conversation

nchaulet
Copy link
Member

@nchaulet nchaulet commented Nov 13, 2024

Summary

Resolve #198737

Fix the GET /api/fleet/datastreams used to power the datastreams tab in fleet to not use _datastreams/_stats in serverless but the metering API instead.

Tests

I added specific test in serverless for that endpoint

And you can test locally by running kibana in serverless and check the datastreams tab is working

Screenshot 2024-11-13 at 12 51 06 PM

@nchaulet nchaulet added bug Fixes for quality problems that affect the customer experience release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) labels Nov 13, 2024
@nchaulet nchaulet self-assigned this Nov 13, 2024
@nchaulet nchaulet requested review from a team as code owners November 13, 2024 18:24
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@nchaulet
Copy link
Member Author

@elasticmachine merge upstream

@@ -51,10 +56,22 @@ export const getListHandler: RequestHandler = async (context, request, response)
};

try {
const useMeteringApi = appContextService.getConfig()?.internal?.useMeteringApi;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if we could use the metering API in stateful too, or is it only for serverless?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately it's only available in serverless

dataStreamService.getAllFleetDataStreams(esClient),
dataStreamService.getAllFleetDataStreamsStats(esClient),
useMeteringApi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we could simplify by moving these out of Promise.all and await the dataStreamService call below in the conditional

? undefined
: dataStreamService.getAllFleetDataStreamsStats(elasticsearch.client.asSecondaryAuthUser),
useMeteringApi
? dataStreamService.getAllFleetMeteringStats(elasticsearch.client.asSecondaryAuthUser)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why use asSecondaryAuthUser here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The API is only available as internal, and calling it as current user fail for that reason.

Copy link
Contributor

@juliaElastic juliaElastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, added a few questions

Copy link
Contributor

@jloleysens jloleysens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Config changes LGTM

Copy link
Member

@azasypkin azasypkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes in config/serverless.yml LGTM

@nchaulet
Copy link
Member Author

@elasticmachine merge upstream

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

cc @nchaulet

@jlind23
Copy link
Contributor

jlind23 commented Nov 18, 2024

@elastic/security-solution can we get some eyes on this please?

Copy link
Contributor

@paul-tavares paul-tavares left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

cc/ @joeypoon

@nchaulet nchaulet merged commit 1fd3f41 into elastic:main Nov 18, 2024
27 checks passed
@nchaulet nchaulet deleted the feature-fix-data-streams-serverless branch November 18, 2024 21:36
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11901548609

@kibanamachine
Copy link
Contributor

💔 All backports failed

Status Branch Result
8.x Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 200063

Questions ?

Please refer to the Backport tool documentation

@nchaulet
Copy link
Member Author

💚 All backports created successfully

Status Branch Result
8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

nchaulet added a commit to nchaulet/kibana that referenced this pull request Nov 19, 2024
(cherry picked from commit 1fd3f41)

# Conflicts:
#	x-pack/plugins/fleet/server/config.ts
nchaulet added a commit that referenced this pull request Nov 19, 2024
# Backport

This will backport the following commits from `main` to `8.x`:
- [[Fleet] Use metering API in serverless
(#200063)](#200063)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Nicolas
Chaulet","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-18T21:36:27Z","message":"[Fleet]
Use metering API in serverless
(#200063)","sha":"1fd3f412e13ed234524915cc87d058f05b840528","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:skip","Team:Fleet","v9.0.0","backport:prev-minor"],"number":200063,"url":"https://github.com/elastic/kibana/pull/200063","mergeCommit":{"message":"[Fleet]
Use metering API in serverless
(#200063)","sha":"1fd3f412e13ed234524915cc87d058f05b840528"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/200063","number":200063,"mergeCommit":{"message":"[Fleet]
Use metering API in serverless
(#200063)","sha":"1fd3f412e13ed234524915cc87d058f05b840528"}}]}]
BACKPORT-->
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Dec 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) bug Fixes for quality problems that affect the customer experience release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v8.17.0 v9.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Fleet] Datastream API is broken in serverless
8 participants