[Security Solution][DataQuality Dashboard][Serverless] Fix stats api filtering logic #196528
Assignees
Labels
bug
Fixes for quality problems that affect the customer experience
Feature:Data Health Quality
Data health quality dashboard and related features
fixed
Team:Threat Hunting:Explore
Team:Threat Hunting
Security Solution Threat Hunting Team
v8.16.1
v8.17.0
v9.0.0
Comments
kapral18
added
bug
|
Pinging @elastic/security-threat-hunting (Team:Threat Hunting) |
|
Pinging @elastic/security-threat-hunting-explore (Team:Threat Hunting:Explore) |
kapral18
added a commit
to kapral18/kibana
that referenced
this issue
Oct 26, 2024
…lableIndices in get_index_stats addresses elastic#196528 - Remove unused get_available_indices.ts params helper file. - Change fetchAvailableIndices to use creation_date from _cat api instead of targeting @timestamp field of indices
kapral18
added a commit
to kapral18/kibana
that referenced
this issue
Oct 30, 2024
…lableIndices in get_index_stats addresses elastic#196528 - Remove unused get_available_indices.ts params helper file. - Change fetchAvailableIndices to use creation_date from _cat api instead of targeting @timestamp field of indices
kapral18
added a commit
that referenced
this issue
Oct 31, 2024
…lableIndices in get_index_stats (#197065) addresses #196528 - Remove unused get_available_indices.ts params helper file. - Change fetchAvailableIndices to use creation_date from _cat api instead of targeting @timestamp field of indices ## UI Changes: Before:  After:  --------- Co-authored-by: kibanamachine <[email protected]>
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Oct 31, 2024
…lableIndices in get_index_stats (elastic#197065) addresses elastic#196528 - Remove unused get_available_indices.ts params helper file. - Change fetchAvailableIndices to use creation_date from _cat api instead of targeting @timestamp field of indices ## UI Changes: Before:  After:  --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit ac013b4)
kibanamachine
added a commit
that referenced
this issue
Oct 31, 2024
…chAvailableIndices in get_index_stats (#197065) (#198525) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution][Data Quality Dashboard][Serverless] Fix fetchAvailableIndices in get_index_stats (#197065)](#197065) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Karen Grigoryan","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-31T13:07:36Z","message":"[Security Solution][Data Quality Dashboard][Serverless] Fix fetchAvailableIndices in get_index_stats (#197065)\n\naddresses #196528\r\n\r\n- Remove unused get_available_indices.ts params helper file.\r\n- Change fetchAvailableIndices to use creation_date from _cat api\r\ninstead of targeting @timestamp field of indices\r\n\r\n## UI Changes:\r\n\r\nBefore:\r\n\r\n\r\nAfter:\r\n\r\n\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"ac013b4a99d68ac1596a19d94a7094b4284a200a","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Threat Hunting","Team:Threat Hunting:Explore","backport:prev-minor","ci:project-deploy-security"],"title":"[Security Solution][Data Quality Dashboard][Serverless] Fix fetchAvailableIndices in get_index_stats","number":197065,"url":"https://github.com/elastic/kibana/pull/197065","mergeCommit":{"message":"[Security Solution][Data Quality Dashboard][Serverless] Fix fetchAvailableIndices in get_index_stats (#197065)\n\naddresses #196528\r\n\r\n- Remove unused get_available_indices.ts params helper file.\r\n- Change fetchAvailableIndices to use creation_date from _cat api\r\ninstead of targeting @timestamp field of indices\r\n\r\n## UI Changes:\r\n\r\nBefore:\r\n\r\n\r\nAfter:\r\n\r\n\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"ac013b4a99d68ac1596a19d94a7094b4284a200a"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/197065","number":197065,"mergeCommit":{"message":"[Security Solution][Data Quality Dashboard][Serverless] Fix fetchAvailableIndices in get_index_stats (#197065)\n\naddresses #196528\r\n\r\n- Remove unused get_available_indices.ts params helper file.\r\n- Change fetchAvailableIndices to use creation_date from _cat api\r\ninstead of targeting @timestamp field of indices\r\n\r\n## UI Changes:\r\n\r\nBefore:\r\n\r\n\r\nAfter:\r\n\r\n\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"ac013b4a99d68ac1596a19d94a7094b4284a200a"}}]}] BACKPORT--> Co-authored-by: Karen Grigoryan <[email protected]>
kapral18
added a commit
to kapral18/kibana
that referenced
this issue
Nov 18, 2024
…lableIndices in get_index_stats (elastic#197065) addresses elastic#196528 - Remove unused get_available_indices.ts params helper file. - Change fetchAvailableIndices to use creation_date from _cat api instead of targeting @timestamp field of indices ## UI Changes: Before:  After:  --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit ac013b4)
kapral18
added a commit
that referenced
this issue
Nov 18, 2024
…tchAvailableIndices in get_index_stats (#197065) (#200551) # Backport This will backport the following commits from `main` to `8.16`: - [[Security Solution][Data Quality Dashboard][Serverless] Fix fetchAvailableIndices in get_index_stats (#197065)](#197065) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Karen Grigoryan","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-31T13:07:36Z","message":"[Security Solution][Data Quality Dashboard][Serverless] Fix fetchAvailableIndices in get_index_stats (#197065)\n\naddresses #196528\r\n\r\n- Remove unused get_available_indices.ts params helper file.\r\n- Change fetchAvailableIndices to use creation_date from _cat api\r\ninstead of targeting @timestamp field of indices\r\n\r\n## UI Changes:\r\n\r\nBefore:\r\n\r\n\r\nAfter:\r\n\r\n\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"ac013b4a99d68ac1596a19d94a7094b4284a200a","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Threat Hunting","Team:Threat Hunting:Explore","backport:prev-minor","ci:project-deploy-security","v8.17.0"],"number":197065,"url":"https://github.com/elastic/kibana/pull/197065","mergeCommit":{"message":"[Security Solution][Data Quality Dashboard][Serverless] Fix fetchAvailableIndices in get_index_stats (#197065)\n\naddresses #196528\r\n\r\n- Remove unused get_available_indices.ts params helper file.\r\n- Change fetchAvailableIndices to use creation_date from _cat api\r\ninstead of targeting @timestamp field of indices\r\n\r\n## UI Changes:\r\n\r\nBefore:\r\n\r\n\r\nAfter:\r\n\r\n\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"ac013b4a99d68ac1596a19d94a7094b4284a200a"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/197065","number":197065,"mergeCommit":{"message":"[Security Solution][Data Quality Dashboard][Serverless] Fix fetchAvailableIndices in get_index_stats (#197065)\n\naddresses #196528\r\n\r\n- Remove unused get_available_indices.ts params helper file.\r\n- Change fetchAvailableIndices to use creation_date from _cat api\r\ninstead of targeting @timestamp field of indices\r\n\r\n## UI Changes:\r\n\r\nBefore:\r\n\r\n\r\nAfter:\r\n\r\n\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"ac013b4a99d68ac1596a19d94a7094b4284a200a"}},{"branch":"8.x","label":"v8.17.0","labelRegex":"^v8.17.0$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/198525","number":198525,"state":"MERGED","mergeCommit":{"sha":"0a561873792de54c5266884a47a0e0819dc7aea3","message":"[8.x] [Security Solution][Data Quality Dashboard][Serverless] Fix fetchAvailableIndices in get_index_stats (#197065) (#198525)\n\n# Backport\n\nThis will backport the following commits from `main` to `8.x`:\n- [[Security Solution][Data Quality Dashboard][Serverless] Fix\nfetchAvailableIndices in get_index_stats\n(#197065)](https://github.com/elastic/kibana/pull/197065)\n\n<!--- Backport version: 9.4.3 -->\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT [{\"author\":{\"name\":\"Karen\nGrigoryan\",\"email\":\"[email protected]\"},\"sourceCommit\":{\"committedDate\":\"2024-10-31T13:07:36Z\",\"message\":\"[Security\nSolution][Data Quality Dashboard][Serverless] Fix fetchAvailableIndices\nin get_index_stats (#197065)\\n\\naddresses #196528\\r\\n\\r\\n- Remove unused\nget_available_indices.ts params helper file.\\r\\n- Change\nfetchAvailableIndices to use creation_date from _cat api\\r\\ninstead of\ntargeting @timestamp field of indices\\r\\n\\r\\n## UI\nChanges:\\r\\n\\r\\nBefore:\\r\\n\\r\\n\\r\\nAfter:\\r\\n\\r\\n\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by:\nkibanamachine\n<[email protected]>\",\"sha\":\"ac013b4a99d68ac1596a19d94a7094b4284a200a\",\"branchLabelMapping\":{\"^v9.0.0$\":\"main\",\"^v8.17.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:skip\",\"v9.0.0\",\"Team:Threat\nHunting\",\"Team:Threat\nHunting:Explore\",\"backport:prev-minor\",\"ci:project-deploy-security\"],\"title\":\"[Security\nSolution][Data Quality Dashboard][Serverless] Fix fetchAvailableIndices\nin\nget_index_stats\",\"number\":197065,\"url\":\"https://github.com/elastic/kibana/pull/197065\",\"mergeCommit\":{\"message\":\"[Security\nSolution][Data Quality Dashboard][Serverless] Fix fetchAvailableIndices\nin get_index_stats (#197065)\\n\\naddresses #196528\\r\\n\\r\\n- Remove unused\nget_available_indices.ts params helper file.\\r\\n- Change\nfetchAvailableIndices to use creation_date from _cat api\\r\\ninstead of\ntargeting @timestamp field of indices\\r\\n\\r\\n## UI\nChanges:\\r\\n\\r\\nBefore:\\r\\n\\r\\n\\r\\nAfter:\\r\\n\\r\\n\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by:\nkibanamachine\n<[email protected]>\",\"sha\":\"ac013b4a99d68ac1596a19d94a7094b4284a200a\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[],\"targetPullRequestStates\":[{\"branch\":\"main\",\"label\":\"v9.0.0\",\"branchLabelMappingKey\":\"^v9.0.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/197065\",\"number\":197065,\"mergeCommit\":{\"message\":\"[Security\nSolution][Data Quality Dashboard][Serverless] Fix fetchAvailableIndices\nin get_index_stats (#197065)\\n\\naddresses #196528\\r\\n\\r\\n- Remove unused\nget_available_indices.ts params helper file.\\r\\n- Change\nfetchAvailableIndices to use creation_date from _cat api\\r\\ninstead of\ntargeting @timestamp field of indices\\r\\n\\r\\n## UI\nChanges:\\r\\n\\r\\nBefore:\\r\\n\\r\\n\\r\\nAfter:\\r\\n\\r\\n\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by:\nkibanamachine\n<[email protected]>\",\"sha\":\"ac013b4a99d68ac1596a19d94a7094b4284a200a\"}}]}]\nBACKPORT-->\n\nCo-authored-by: Karen Grigoryan <[email protected]>"}}]}] BACKPORT-->
|
Is this the new expected behavior?
If so, please feel free to close the ticket. Thanks!! :) |
|
@MadameSheema yes our currently fixed implementation shows incompatible field mappings even if no data is present for mappings that are either created within a week or are created earlier but have received data within a week |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug:
Currently serverless kibana DQD
/internal/ecs_data_quality_dashboard/stats/{pattern}apilogic fetches only indices that have data in a specific range (1 week)
kibana/x-pack/plugins/ecs_data_quality_dashboard/server/helpers/get_available_indices.ts
Line 34 in 47a497c
This effectively filters out any indices that have docs without
@timestampfields or don't have any docs yet.This is a problem because ess implementation doesn't do that and shows missing
@timestampfield in the latest check report both in incompatible fields and as part of missing ecs compliant fields ui.To align these implementations we need to use index creation data api from es (example
GET /_cat/indices/.alerts-security.alerts-default?h=creation.date) instead of relying on filtering docs by@timestampsin serverless.This results in empty pattern tables in serverless for empty indices or indices without @timestamp, which is confusing, and when of empty indices are stacked it pushes out the non-empty ones out of the view.
This can be circumvented with a temporary fix to close them by default or sort them by content, but it's arguably easier to just fix the core issue altogether.
Kibana/Elasticsearch Stack version: all supported versions
Steps to reproduce:
Current behavior:
Pattern is showing but empty
Expected behavior:
Pattern is showing latest index to check
Screenshots (if relevant):
The text was updated successfully, but these errors were encountered: