Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Remote clusters] Per cluster status call #194420

Merged
merged 9 commits into from
Oct 7, 2024
Merged

[Remote clusters] Per cluster status call #194420

merged 9 commits into from
Oct 7, 2024

Conversation

sabarasaba
Copy link
Member

@sabarasaba sabarasaba commented Sep 30, 2024 •
edited by kibanamachine
Loading

Summary

We agregate all the cluster names in order to do a single resolveCluster API call to get the cluster statuses. If we have many and with long names, this could trigger a error such as 400: Bad Request. An HTTP line is larger than 4096 bytes.. This PR changes this code so that we do a resolveCluster call per cluster instead, in order to avoid this sort of problems.

How to test
  • Start up kibana and es
  • Setup a remote cluster locally using the guidelines
  • Navigate to Stack management -> Remote cluster
  • Verify that:
    • The test remote cluster is connected
    • Disconnect the remote cluster, and verify that the remote cluster is now disconnected in kibana

@sabarasaba sabarasaba self-assigned this Sep 30, 2024
@sabarasaba sabarasaba added Feature:CCR and Remote Clusters Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more v9.0.0 backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) v8.15.0 v8.16.0 labels Sep 30, 2024
@sabarasaba sabarasaba marked this pull request as ready for review September 30, 2024 13:06
@sabarasaba sabarasaba requested a review from a team as a code owner September 30, 2024 13:06
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-management (Team:Kibana Management)

@sabarasaba sabarasaba added the release_note:skip Skip the PR/issue when compiling release notes label Sep 30, 2024
afharo
afharo reviewed Sep 30, 2024
View reviewed changes
Copy link
Member

@afharo afharo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We moved from 1 request to N requests (and triggering all requests in parallel).

I wish we could batch cluster resolution (like 10 clusters per call?) to minimize the impact or, at least, use pMap (from p-map) to control the number of concurrent requests.

x-pack/plugins/remote_clusters/server/routes/api/get_route.ts Outdated Show resolved Hide resolved
x-pack/plugins/remote_clusters/server/routes/api/get_route.ts Outdated Show resolved Hide resolved
@sabarasaba
Copy link
Member Author

@elasticmachine merge upstream

@sabarasaba
Copy link
Member Author

Thanks for reviewing @afharo, I've separated the requests in batches of 10 and also requestTimeout at least until we can resolve #194834

afharo
afharo approved these changes Oct 4, 2024
View reviewed changes
Copy link
Member

@afharo afharo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

mattkime
mattkime requested changes Oct 4, 2024
View reviewed changes
Copy link
Contributor

@mattkime mattkime left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works well but I think the concurrency can be improved.

x-pack/plugins/remote_clusters/server/routes/api/get_route.ts Show resolved Hide resolved
@mattkime mattkime removed the v8.15.0 label Oct 7, 2024
@sabarasaba
Copy link
Member Author

@elasticmachine merge upstream

@sabarasaba sabarasaba enabled auto-merge (squash) October 7, 2024 14:36
@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group1/config.ts / alerting api integration security and spaces enabled Alerts - Group 1 alerts backfill rule runs ad hoc backfill task should run all execution sets of a scheduled backfill and correctly generate alerts

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
remoteClusters 78.8KB 78.8KB +1.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @sabarasaba

@sabarasaba sabarasaba merged commit 77c89d2 into elastic:main Oct 7, 2024
24 of 25 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11219542147

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Oct 7, 2024
@sabarasaba
[Remote clusters] Per cluster status call (elastic#194420)
3a9e3a3 
(cherry picked from commit 77c89d2)
@kibanamachine kibanamachine mentioned this pull request Oct 7, 2024
Merged
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
✅ 8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Oct 7, 2024
@kibanamachine @sabarasaba
[8.x] [Remote clusters] Per cluster status call (#194420) (#195290)
04c5679 
# Backport

This will backport the following commits from `main` to `8.x`:
- [[Remote clusters] Per cluster status call
(#194420)](#194420)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Ignacio
Rivas","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-07T16:07:41Z","message":"[Remote
clusters] Per cluster status call
(#194420)","sha":"77c89d22f7863e8e55214c3aa79dc16026cc08e3","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Feature:CCR
and Remote Clusters","Team:Kibana
Management","release_note:skip","v9.0.0","backport:prev-minor","v8.16.0"],"title":"[Remote
clusters] Per cluster status
call","number":194420,"url":"https://github.com/elastic/kibana/pull/194420","mergeCommit":{"message":"[Remote
clusters] Per cluster status call
(#194420)","sha":"77c89d22f7863e8e55214c3aa79dc16026cc08e3"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/194420","number":194420,"mergeCommit":{"message":"[Remote
clusters] Per cluster status call
(#194420)","sha":"77c89d22f7863e8e55214c3aa79dc16026cc08e3"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Ignacio Rivas <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@afharo afharo afharo approved these changes

@mattkime mattkime mattkime approved these changes

Assignees

@sabarasaba sabarasaba

Labels
backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) Feature:CCR and Remote Clusters release_note:skip Skip the PR/issue when compiling release notes Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more v8.16.0 v9.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@sabarasaba @elasticmachine @kibana-ci @kibanamachine @mattkime @afharo