Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create search:search_excluded_data_tiers setting #192276

Closed
wants to merge 4 commits into from
Closed

Create search:search_excluded_data_tiers setting #192276

wants to merge 4 commits into from
+34 −0

Conversation

crespocarlos
Copy link
Contributor

@crespocarlos crespocarlos commented Sep 6, 2024 •
edited
Loading

part of #190559

Summary

This PR introduces a new Kibana-wide Advanced Settings under Search, enabling solutions to exclude indices in the data_cold and/or data_frozen tiers from their queries.

The change will help to address issues encountered by O11y, Stack Monitoring, and Security Solutions with unbounded queries targeting the frozen tier. By centralizing this setting, we prevent each solution from implementing its own version.

For example, the securitySolution:excludedDataTiersForRuleExecution setting in Security could be replaced by the one introduced in this PR.

Additionally, this setting could potentially replace the soon to be deprecated includeFrozen setting.

Follow-up

It might be a good idea to change the get_search_params.ts and other places in the data plugin to look at the configuration.

@obltmachine
Copy link

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@crespocarlos crespocarlos force-pushed the 190559-create-kibana-wide-excluded-tiers-setting branch from d525cca to c555f1b Compare September 6, 2024 15:32
@crespocarlos
Copy link
Contributor Author

/ci

@crespocarlos crespocarlos force-pushed the 190559-create-kibana-wide-excluded-tiers-setting branch 2 times, most recently from ff94759 to 94d1444 Compare September 9, 2024 09:10
@crespocarlos crespocarlos force-pushed the 190559-create-kibana-wide-excluded-tiers-setting branch from 94d1444 to 1e4487c Compare September 9, 2024 09:11
@crespocarlos
Copy link
Contributor Author

@elasticmachine merge upstream

@crespocarlos
Copy link
Contributor Author

/ci

@crespocarlos
Copy link
Contributor Author

/ci

@crespocarlos crespocarlos added Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL. Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting Team:Monitoring Stack Monitoring team labels Sep 9, 2024
@crespocarlos crespocarlos marked this pull request as ready for review September 9, 2024 13:29
@crespocarlos crespocarlos requested review from a team as code owners September 9, 2024 13:29
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

@crespocarlos @smith
Adjust title
40c65ca 
Co-authored-by: Nathan L Smith <[email protected]>
@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
aiops 547.4KB 547.6KB +189.0B
datasetQuality 208.7KB 208.7KB +63.0B
discover 797.2KB 797.2KB +63.0B
eventAnnotationListing 227.1KB 227.2KB +63.0B
lens 1.5MB 1.5MB +63.0B
securitySolution 19.7MB 19.7MB +63.0B
slo 852.1KB 852.2KB +63.0B
total +567.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
data 418.9KB 418.9KB +63.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@lucabelluccini
Copy link
Contributor

Would this also be applied to the Synthetics (being part of O11y I would say so)?

@crespocarlos
Copy link
Contributor Author

Would this also be applied to the Synthetics (being part of O11y I would say so)?

The filtering out of data tiers won't be automatic. Teams will have to implement that usage of the parameter created here to decide when the filter needs to be applied.

kertal
kertal requested changes Sep 11, 2024
View reviewed changes
Copy link
Member

@kertal kertal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thx for taking care of this, however need to block this for now. we need internal alignment about introducing a new setting for this, will follow up in slack

jloleysens
jloleysens approved these changes Sep 11, 2024
View reviewed changes
Copy link
Contributor

@jloleysens jloleysens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did not test locally, but Core changes LGTM

This was referenced Sep 11, 2024
Merged
Closed
@crespocarlos
Copy link
Contributor Author

I had a chat with @elastic/kibana-data-discovery and we've agreed that it will be better to move forward with a solution-specific setting. I'm closing this in favor of #192570

crespocarlos added a commit that referenced this pull request Sep 16, 2024
@crespocarlos @elasticmachine
[Observability] Create observability-specific setting for excluding d…
f029f80 
…ata tiers from queries (#192570)

part of [#190559](#190559)

## Summary

This PR introduces a new `Advanced Settings` under `Observabilty` to
provide a way of configuring the exclusion of indices in the `data_cold`
and/or `data_frozen` tiers from queries.

The change will help to address issues encountered in O11y, most
specifically in APM, and could also affect Infra and other features,
with unbounded queries targeting the frozen tier.

### For reviewers

This PR replaces #192276

---------

Co-authored-by: Elastic Machine <[email protected]>
crespocarlos added a commit to crespocarlos/kibana that referenced this pull request Sep 23, 2024
@crespocarlos
[Observability] Create observability-specific setting for excluding d…
e9f7d68 
…ata tiers from queries (elastic#192570)

part of [elastic#190559](elastic#190559)

## Summary

This PR introduces a new `Advanced Settings` under `Observabilty` to
provide a way of configuring the exclusion of indices in the `data_cold`
and/or `data_frozen` tiers from queries.

The change will help to address issues encountered in O11y, most
specifically in APM, and could also affect Infra and other features,
with unbounded queries targeting the frozen tier.

### For reviewers

This PR replaces elastic#192276

---------

Co-authored-by: Elastic Machine <[email protected]>
(cherry picked from commit f029f80)
crespocarlos referenced this pull request Sep 24, 2024
@crespocarlos
[8.x] [Observability] Create observability-specific setting for exclu…
e20ccc4 
…ding data tiers from queries (#192570) (#193784)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Observability] Create observability-specific setting for excluding
data tiers from queries
(#192570)](#192570)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Carlos
Crespo","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-09-16T14:58:13Z","message":"[Observability]
Create observability-specific setting for excluding data tiers from
queries (#192570)\n\npart of
[#190559](https://github.com/elastic/kibana/issues/190559)\r\n\r\n##
Summary\r\n\r\nThis PR introduces a new `Advanced Settings` under
`Observabilty` to\r\nprovide a way of configuring the exclusion of
indices in the `data_cold`\r\nand/or `data_frozen` tiers from
queries.\r\n\r\nThe change will help to address issues encountered in
O11y, most\r\nspecifically in APM, and could also affect Infra and other
features,\r\nwith unbounded queries targeting the frozen
tier.\r\n\r\n### For reviewers\r\n\r\nThis PR replaces
https://github.com/elastic/kibana/pull/192276\r\n\r\n---------\r\n\r\nCo-authored-by:
Elastic Machine
<[email protected]>","sha":"f029f8086a6731b5f435775c915d46e110a34658","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:skip","v9.0.0","ci:project-deploy-observability","Team:obs-ux-infra_services","Team:obs-ux-management"],"number":192570,"url":"https://github.com/elastic/kibana/pull/192570","mergeCommit":{"message":"[Observability]
Create observability-specific setting for excluding data tiers from
queries (#192570)\n\npart of
[#190559](https://github.com/elastic/kibana/issues/190559)\r\n\r\n##
Summary\r\n\r\nThis PR introduces a new `Advanced Settings` under
`Observabilty` to\r\nprovide a way of configuring the exclusion of
indices in the `data_cold`\r\nand/or `data_frozen` tiers from
queries.\r\n\r\nThe change will help to address issues encountered in
O11y, most\r\nspecifically in APM, and could also affect Infra and other
features,\r\nwith unbounded queries targeting the frozen
tier.\r\n\r\n### For reviewers\r\n\r\nThis PR replaces
https://github.com/elastic/kibana/pull/192276\r\n\r\n---------\r\n\r\nCo-authored-by:
Elastic Machine
<[email protected]>","sha":"f029f8086a6731b5f435775c915d46e110a34658"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192570","number":192570,"mergeCommit":{"message":"[Observability]
Create observability-specific setting for excluding data tiers from
queries (#192570)\n\npart of
[#190559](https://github.com/elastic/kibana/issues/190559)\r\n\r\n##
Summary\r\n\r\nThis PR introduces a new `Advanced Settings` under
`Observabilty` to\r\nprovide a way of configuring the exclusion of
indices in the `data_cold`\r\nand/or `data_frozen` tiers from
queries.\r\n\r\nThe change will help to address issues encountered in
O11y, most\r\nspecifically in APM, and could also affect Infra and other
features,\r\nwith unbounded queries targeting the frozen
tier.\r\n\r\n### For reviewers\r\n\r\nThis PR replaces
https://github.com/elastic/kibana/pull/192276\r\n\r\n---------\r\n\r\nCo-authored-by:
Elastic Machine
<[email protected]>","sha":"f029f8086a6731b5f435775c915d46e110a34658"}}]}]
BACKPORT-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@smith smith smith left review comments

@kertal kertal kertal requested changes

@jloleysens jloleysens jloleysens approved these changes

@markov00 markov00 markov00 approved these changes

Assignees
No one assigned
Labels
backport:skip This commit does not require backporting release_note:skip Skip the PR/issue when compiling release notes Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL. Team:Monitoring Stack Monitoring team Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
9 participants
@crespocarlos @obltmachine @elasticmachine @kibana-ci @lucabelluccini @smith @kertal @markov00 @jloleysens