Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Observability] Create observability-specific setting for excluding data tiers from queries #192570

Conversation

crespocarlos
Copy link
Contributor

@crespocarlos crespocarlos commented Sep 11, 2024

part of #190559

Summary

This PR introduces a new Advanced Settings under Observabilty to provide a way of configuring the exclusion of indices in the data_cold and/or data_frozen tiers from queries.

The change will help to address issues encountered in O11y, most specifically in APM, and could also affect Infra and other features, with unbounded queries targeting the frozen tier.

For reviewers

This PR replaces #192276

@obltmachine
Copy link

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@crespocarlos crespocarlos changed the title [Observability] 190559 create observability excluded data tiers setting [Observability] Create observability-specific setting for excluding data tiers from queries Sep 11, 2024
@crespocarlos crespocarlos force-pushed the 190559-create-observability-excluded-data-tiers-setting branch 2 times, most recently from 0e65086 to 2b4cd56 Compare September 11, 2024 13:05
@crespocarlos crespocarlos force-pushed the 190559-create-observability-excluded-data-tiers-setting branch from 2b4cd56 to cc7e886 Compare September 11, 2024 13:07
@crespocarlos
Copy link
Contributor Author

/ci

'xpack.observability.advancedSettings.searchExcludedDataTiersDesc',
{
defaultMessage: `Specify the data tiers to exclude from search, such as data_cold and/or data_frozen.
When configured, indices allocated in the selected tiers will be ignored from search requests.`,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll add here which parts of o11y will be impacted

@crespocarlos crespocarlos marked this pull request as ready for review September 12, 2024 14:22
@crespocarlos crespocarlos requested review from a team as code owners September 12, 2024 14:22
@crespocarlos crespocarlos added release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team labels Sep 12, 2024
@botelastic botelastic bot added ci:project-deploy-observability Create an Observability project Team:obs-ux-management Observability Management User Experience Team labels Sep 12, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

Copy link
Member

@maryam-saeidi maryam-saeidi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Obs-ux-management code changes LGTM

@crespocarlos
Copy link
Contributor Author

@elasticmachine merge upstream

Copy link
Contributor

@jloleysens jloleysens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did not test locally, but core changes LGTM.

@crespocarlos
Copy link
Contributor Author

@elasticmachine merge upstream

@crespocarlos
Copy link
Contributor Author

@elasticmachine merge upstream

@kibana-ci
Copy link
Collaborator

kibana-ci commented Sep 16, 2024

💚 Build Succeeded

  • Buildkite Build
  • Commit: 9536464
  • Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-192570-953646412c26

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/management-settings-ids 140 141 +1
observability 698 705 +7
total +8
Unknown metric groups

API count

id before after diff
@kbn/management-settings-ids 142 143 +1
observability 705 712 +7
total +8

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Copy link
Member

@Bamieh Bamieh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

telemetry changes LGTM. added a new telemetry field observability:searchExcludedDataTiers

@crespocarlos crespocarlos merged commit f029f80 into elastic:main Sep 16, 2024
22 checks passed
@crespocarlos crespocarlos deleted the 190559-create-observability-excluded-data-tiers-setting branch September 16, 2024 14:58
@crespocarlos
Copy link
Contributor Author

💔 Some backports could not be created

Status Branch Result
8.x
8.15 Conflict resolution was aborted by the user

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 192570

Questions ?

Please refer to the Backport tool documentation

crespocarlos added a commit to crespocarlos/kibana that referenced this pull request Sep 23, 2024
…ata tiers from queries (elastic#192570)

part of [elastic#190559](elastic#190559)

## Summary

This PR introduces a new `Advanced Settings` under `Observabilty` to
provide a way of configuring the exclusion of indices in the `data_cold`
and/or `data_frozen` tiers from queries.

The change will help to address issues encountered in O11y, most
specifically in APM, and could also affect Infra and other features,
with unbounded queries targeting the frozen tier.

### For reviewers

This PR replaces elastic#192276

---------

Co-authored-by: Elastic Machine <[email protected]>
(cherry picked from commit f029f80)
@crespocarlos
Copy link
Contributor Author

💚 All backports created successfully

Status Branch Result
8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

crespocarlos added a commit that referenced this pull request Sep 24, 2024
…ding data tiers from queries (#192570) (#193784)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Observability] Create observability-specific setting for excluding
data tiers from queries
(#192570)](#192570)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Carlos
Crespo","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-09-16T14:58:13Z","message":"[Observability]
Create observability-specific setting for excluding data tiers from
queries (#192570)\n\npart of
[#190559](https://github.com/elastic/kibana/issues/190559)\r\n\r\n##
Summary\r\n\r\nThis PR introduces a new `Advanced Settings` under
`Observabilty` to\r\nprovide a way of configuring the exclusion of
indices in the `data_cold`\r\nand/or `data_frozen` tiers from
queries.\r\n\r\nThe change will help to address issues encountered in
O11y, most\r\nspecifically in APM, and could also affect Infra and other
features,\r\nwith unbounded queries targeting the frozen
tier.\r\n\r\n### For reviewers\r\n\r\nThis PR replaces
https://github.com/elastic/kibana/pull/192276\r\n\r\n---------\r\n\r\nCo-authored-by:
Elastic Machine
<[email protected]>","sha":"f029f8086a6731b5f435775c915d46e110a34658","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:skip","v9.0.0","ci:project-deploy-observability","Team:obs-ux-infra_services","Team:obs-ux-management"],"number":192570,"url":"https://github.com/elastic/kibana/pull/192570","mergeCommit":{"message":"[Observability]
Create observability-specific setting for excluding data tiers from
queries (#192570)\n\npart of
[#190559](https://github.com/elastic/kibana/issues/190559)\r\n\r\n##
Summary\r\n\r\nThis PR introduces a new `Advanced Settings` under
`Observabilty` to\r\nprovide a way of configuring the exclusion of
indices in the `data_cold`\r\nand/or `data_frozen` tiers from
queries.\r\n\r\nThe change will help to address issues encountered in
O11y, most\r\nspecifically in APM, and could also affect Infra and other
features,\r\nwith unbounded queries targeting the frozen
tier.\r\n\r\n### For reviewers\r\n\r\nThis PR replaces
https://github.com/elastic/kibana/pull/192276\r\n\r\n---------\r\n\r\nCo-authored-by:
Elastic Machine
<[email protected]>","sha":"f029f8086a6731b5f435775c915d46e110a34658"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192570","number":192570,"mergeCommit":{"message":"[Observability]
Create observability-specific setting for excluding data tiers from
queries (#192570)\n\npart of
[#190559](https://github.com/elastic/kibana/issues/190559)\r\n\r\n##
Summary\r\n\r\nThis PR introduces a new `Advanced Settings` under
`Observabilty` to\r\nprovide a way of configuring the exclusion of
indices in the `data_cold`\r\nand/or `data_frozen` tiers from
queries.\r\n\r\nThe change will help to address issues encountered in
O11y, most\r\nspecifically in APM, and could also affect Infra and other
features,\r\nwith unbounded queries targeting the frozen
tier.\r\n\r\n### For reviewers\r\n\r\nThis PR replaces
https://github.com/elastic/kibana/pull/192276\r\n\r\n---------\r\n\r\nCo-authored-by:
Elastic Machine
<[email protected]>","sha":"f029f8086a6731b5f435775c915d46e110a34658"}}]}]
BACKPORT-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting ci:project-deploy-observability Create an Observability project release_note:skip Skip the PR/issue when compiling release notes Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team Team:obs-ux-management Observability Management User Experience Team v8.16.0 v9.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.