Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution][Data Quality Dashboard][Serverless] results rollup incompatible counts not aligned #191053

Closed
angorayc opened this issue Aug 22, 2024 · 2 comments
Closed

[Security Solution][Data Quality Dashboard][Serverless] results rollup incompatible counts not aligned #191053

angorayc opened this issue Aug 22, 2024 · 2 comments
Assignees
@kapral18
Labels
bug Fixes for quality problems that affect the customer experience Feature:Data Health Quality Data health quality dashboard and related features fixed Team:Threat Hunting:Explore Team:Threat Hunting Security Solution Threat Hunting Team v8.16.1 v8.17.0 v9.0.0

Comments

@angorayc
Copy link
Contributor

angorayc commented Aug 22, 2024 •
edited by kapral18
Loading

Describe the bug:

In serverless, stats api returns time restricted selection of index names that we use to render individual index results in the table, but results rollup data is not time restricted, it's being fetched unfiltered and is presented as is hence the discrepancy. (6 v.s. 2 in the screenshot)

Kibana/Elasticsearch Stack version: all versions

Steps to reproduce:

Precondition: Have failed index checks with incompatible fields from indices containing data older than 1 week

  1. Visit data quality dashboard in serverless
  2. Pick data view with pattern that includes said indices
  3. See data quality pattern show incompatible fields count bigger then the sum of individual index check incompatible field counts

Current behavior:

Incompatible fields from old indices are being counted into the sum of pattern incompatible fields

Expected behavior:

Incompatible fields from old indices shouldn't be fetched in the first place by specifying startDate, endDate for results_latest call that should filter indices on the backend and return already filtered indices, thus resolving the problem altogether

Screenshots (if relevant):
image (3)
@angorayc angorayc added bug Fixes for quality problems that affect the customer experience Team:Threat Hunting:Explore Feature:Data Health Quality Data health quality dashboard and related features labels Aug 22, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-threat-hunting-explore (Team:Threat Hunting:Explore)

@kapral18 kapral18 self-assigned this Aug 22, 2024
@kapral18 kapral18 changed the title Data quality dashboard incompatible counts not aligned [Security Solution][Data quality dashboard][Serverless] results rollup incompatible counts not aligned Oct 22, 2024
kapral18 added a commit to kapral18/kibana that referenced this issue Nov 7, 2024
@kapral18
[Security Solution][Data Quality Dashboard] add default start and end…
456a48e 
… time support

addresses elastic#191053

- Introduce `defaultStartTime` and `defaultEndTime` props across data quality context and panels.
- Update hooks to handle new time parameters and include them in storage results queries.
- Modify server-side helpers and routes to process and filter indices based on the provided time range.
- Update related tests to accommodate the new time parameters.
@kapral18 kapral18 mentioned this issue Nov 7, 2024
Merged
kapral18 added a commit to kapral18/kibana that referenced this issue Nov 7, 2024
@kapral18
[Security Solution][Data Quality Dashboard][Serverless] add start/end…
1a5a72f 
… time support for latest_results

addresses elastic#191053

- Introduce `defaultStartTime` and `defaultEndTime` props across data
quality context and panels for fetching latest_results and align them
with serverless default time range of last week
- Update hooks to handle new time parameters and include them in storage
results queries.
- Modify server-side helpers and routes to process and filter indices
based on the provided time range.
- Update related tests to accommodate the new time parameters.
kapral18 added a commit to kapral18/kibana that referenced this issue Nov 7, 2024
@kapral18
[Security Solution][Data Quality Dashboard][Serverless] add start/end…
c5f4423 
… time support for latest_results

addresses elastic#191053

- Introduce `defaultStartTime` and `defaultEndTime` props across data
quality context and panels for fetching latest_results and align them
with serverless default time range of last week
- Update hooks to handle new time parameters and include them in storage
results queries.
- Modify server-side helpers and routes to process and filter indices
based on the provided time range.
- Update related tests to accommodate the new time parameters.
kapral18 added a commit to kapral18/kibana that referenced this issue Nov 7, 2024
@kapral18
[Security Solution][Data Quality Dashboard][Serverless] add start/end…
611e05f 
… time support for latest_results

addresses elastic#191053

- Introduce `defaultStartTime` and `defaultEndTime` props across data
quality context and panels for fetching latest_results and align them
with serverless default time range of last week
- Update hooks to handle new time parameters and include them in storage
results queries.
- Modify server-side helpers and routes to process and filter indices
based on the provided time range.
- Update related tests to accommodate the new time parameters.
kapral18 added a commit to kapral18/kibana that referenced this issue Nov 7, 2024
@kapral18
[Security Solution][Data Quality Dashboard][Serverless] add start/end…
21273b0 
… time support for latest_results

addresses elastic#191053

- Introduce `defaultStartTime` and `defaultEndTime` props across data
quality context and panels for fetching latest_results and align them
with serverless default time range of last week
- Update hooks to handle new time parameters and include them in storage
results queries.
- Modify server-side helpers and routes to process and filter indices
based on the provided time range.
- Update related tests to accommodate the new time parameters.
kapral18 added a commit to kapral18/kibana that referenced this issue Nov 7, 2024
@kapral18
[Security Solution][Data Quality Dashboard][Serverless] add start/end…
f732192 
… time support for latest_results

addresses elastic#191053

- Introduce `defaultStartTime` and `defaultEndTime` props across data
quality context and panels for fetching latest_results and align them
with serverless default time range of last week
- Update hooks to handle new time parameters and include them in storage
results queries.
- Modify server-side helpers and routes to process and filter indices
based on the provided time range.
- Update related tests to accommodate the new time parameters.
@kapral18 kapral18 added Team:Threat Hunting Security Solution Threat Hunting Team v8.16.1 and removed Team:Threat Hunting Security Solution Threat Hunting Team labels Nov 7, 2024
kapral18 added a commit that referenced this issue Nov 7, 2024
@kapral18
[Security Solution][Data Quality Dashboard][Serverless] add start/end…
1df04ae 
… time support for latest_results (#199248)

addresses #191053

- Introduce `defaultStartTime` and `defaultEndTime` props across data
quality context and panels for fetching latest_results and align them
with serverless default time range of last week
- Update hooks to handle new time parameters and include them in storage
results queries.
- Modify server-side helpers and routes to process and filter indices
based on the provided time range.
- Update related tests to accommodate the new time parameters.
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Nov 7, 2024
@kapral18
[Security Solution][Data Quality Dashboard][Serverless] add start/end…
0ebec5d 
… time support for latest_results (elastic#199248)

addresses elastic#191053

- Introduce `defaultStartTime` and `defaultEndTime` props across data
quality context and panels for fetching latest_results and align them
with serverless default time range of last week
- Update hooks to handle new time parameters and include them in storage
results queries.
- Modify server-side helpers and routes to process and filter indices
based on the provided time range.
- Update related tests to accommodate the new time parameters.

(cherry picked from commit 1df04ae)
@kapral18 kapral18 added the fixed label Nov 7, 2024
kibanamachine added a commit that referenced this issue Nov 7, 2024
@kibanamachine @kapral18
[8.x] [Security Solution][Data Quality Dashboard][Serverless] add sta…
de09e3a 
…rt/end time support for latest_results (#199248) (#199385)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Security Solution][Data Quality Dashboard][Serverless] add start/end
time support for latest_results
(#199248)](#199248)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Karen
Grigoryan","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-07T17:42:31Z","message":"[Security
Solution][Data Quality Dashboard][Serverless] add start/end time support
for latest_results (#199248)\n\naddresses #191053\r\n\r\n- Introduce
`defaultStartTime` and `defaultEndTime` props across data\r\nquality
context and panels for fetching latest_results and align them\r\nwith
serverless default time range of last week\r\n- Update hooks to handle
new time parameters and include them in storage\r\nresults queries.\r\n-
Modify server-side helpers and routes to process and filter
indices\r\nbased on the provided time range.\r\n- Update related tests
to accommodate the new time
parameters.","sha":"1df04aef8d5859507c85a2ad37594075e9054b70","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Threat
Hunting","Team:Threat
Hunting:Explore","backport:prev-minor","ci:cloud-deploy","ci:project-deploy-security"],"title":"[Security
Solution][Data Quality Dashboard][Serverless] add start/end time support
for
latest_results","number":199248,"url":"https://github.com/elastic/kibana/pull/199248","mergeCommit":{"message":"[Security
Solution][Data Quality Dashboard][Serverless] add start/end time support
for latest_results (#199248)\n\naddresses #191053\r\n\r\n- Introduce
`defaultStartTime` and `defaultEndTime` props across data\r\nquality
context and panels for fetching latest_results and align them\r\nwith
serverless default time range of last week\r\n- Update hooks to handle
new time parameters and include them in storage\r\nresults queries.\r\n-
Modify server-side helpers and routes to process and filter
indices\r\nbased on the provided time range.\r\n- Update related tests
to accommodate the new time
parameters.","sha":"1df04aef8d5859507c85a2ad37594075e9054b70"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/199248","number":199248,"mergeCommit":{"message":"[Security
Solution][Data Quality Dashboard][Serverless] add start/end time support
for latest_results (#199248)\n\naddresses #191053\r\n\r\n- Introduce
`defaultStartTime` and `defaultEndTime` props across data\r\nquality
context and panels for fetching latest_results and align them\r\nwith
serverless default time range of last week\r\n- Update hooks to handle
new time parameters and include them in storage\r\nresults queries.\r\n-
Modify server-side helpers and routes to process and filter
indices\r\nbased on the provided time range.\r\n- Update related tests
to accommodate the new time
parameters.","sha":"1df04aef8d5859507c85a2ad37594075e9054b70"}}]}]
BACKPORT-->

Co-authored-by: Karen Grigoryan <[email protected]>
mbondyra pushed a commit to mbondyra/kibana that referenced this issue Nov 8, 2024
@kapral18 @mbondyra
[Security Solution][Data Quality Dashboard][Serverless] add start/end…
f9f0959 
… time support for latest_results (elastic#199248)

addresses elastic#191053

- Introduce `defaultStartTime` and `defaultEndTime` props across data
quality context and panels for fetching latest_results and align them
with serverless default time range of last week
- Update hooks to handle new time parameters and include them in storage
results queries.
- Modify server-side helpers and routes to process and filter indices
based on the provided time range.
- Update related tests to accommodate the new time parameters.
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this issue Nov 8, 2024
@kapral18 @CAWilson94
[Security Solution][Data Quality Dashboard][Serverless] add start/end…
59e9308 
… time support for latest_results (elastic#199248)

addresses elastic#191053

- Introduce `defaultStartTime` and `defaultEndTime` props across data
quality context and panels for fetching latest_results and align them
with serverless default time range of last week
- Update hooks to handle new time parameters and include them in storage
results queries.
- Modify server-side helpers and routes to process and filter indices
based on the provided time range.
- Update related tests to accommodate the new time parameters.
kapral18 added a commit to kapral18/kibana that referenced this issue Nov 18, 2024
@kapral18
[Security Solution][Data Quality Dashboard][Serverless] add start/end…
7db9349 
… time support for latest_results (elastic#199248)

addresses elastic#191053

- Introduce `defaultStartTime` and `defaultEndTime` props across data
quality context and panels for fetching latest_results and align them
with serverless default time range of last week
- Update hooks to handle new time parameters and include them in storage
results queries.
- Modify server-side helpers and routes to process and filter indices
based on the provided time range.
- Update related tests to accommodate the new time parameters.

(cherry picked from commit 1df04ae)
kapral18 added a commit that referenced this issue Nov 18, 2024
@kapral18
[8.16] [Security Solution][Data Quality Dashboard][Serverless] add st…
b900fa0 
…art/end time support for latest_results (#199248) (#200635)

# Backport

This will backport the following commits from `main` to `8.16`:
- [[Security Solution][Data Quality Dashboard][Serverless] add start/end
time support for latest_results
(#199248)](#199248)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Karen
Grigoryan","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-07T17:42:31Z","message":"[Security
Solution][Data Quality Dashboard][Serverless] add start/end time support
for latest_results (#199248)\n\naddresses #191053\r\n\r\n- Introduce
`defaultStartTime` and `defaultEndTime` props across data\r\nquality
context and panels for fetching latest_results and align them\r\nwith
serverless default time range of last week\r\n- Update hooks to handle
new time parameters and include them in storage\r\nresults queries.\r\n-
Modify server-side helpers and routes to process and filter
indices\r\nbased on the provided time range.\r\n- Update related tests
to accommodate the new time
parameters.","sha":"1df04aef8d5859507c85a2ad37594075e9054b70","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Threat
Hunting","Team:Threat
Hunting:Explore","backport:prev-minor","ci:cloud-deploy","ci:project-deploy-security","v8.17.0"],"number":199248,"url":"https://github.com/elastic/kibana/pull/199248","mergeCommit":{"message":"[Security
Solution][Data Quality Dashboard][Serverless] add start/end time support
for latest_results (#199248)\n\naddresses #191053\r\n\r\n- Introduce
`defaultStartTime` and `defaultEndTime` props across data\r\nquality
context and panels for fetching latest_results and align them\r\nwith
serverless default time range of last week\r\n- Update hooks to handle
new time parameters and include them in storage\r\nresults queries.\r\n-
Modify server-side helpers and routes to process and filter
indices\r\nbased on the provided time range.\r\n- Update related tests
to accommodate the new time
parameters.","sha":"1df04aef8d5859507c85a2ad37594075e9054b70"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/199248","number":199248,"mergeCommit":{"message":"[Security
Solution][Data Quality Dashboard][Serverless] add start/end time support
for latest_results (#199248)\n\naddresses #191053\r\n\r\n- Introduce
`defaultStartTime` and `defaultEndTime` props across data\r\nquality
context and panels for fetching latest_results and align them\r\nwith
serverless default time range of last week\r\n- Update hooks to handle
new time parameters and include them in storage\r\nresults queries.\r\n-
Modify server-side helpers and routes to process and filter
indices\r\nbased on the provided time range.\r\n- Update related tests
to accommodate the new time
parameters.","sha":"1df04aef8d5859507c85a2ad37594075e9054b70"}},{"branch":"8.x","label":"v8.17.0","labelRegex":"^v8.17.0$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/199385","number":199385,"state":"MERGED","mergeCommit":{"sha":"de09e3af76d4bd1ce029830bd866e20b46e3aa9e","message":"[8.x]
[Security Solution][Data Quality Dashboard][Serverless] add start/end
time support for latest_results (#199248) (#199385)\n\n#
Backport\n\nThis will backport the following commits from `main` to
`8.x`:\n- [[Security Solution][Data Quality Dashboard][Serverless] add
start/end\ntime support for
latest_results\n(#199248)](https://github.com/elastic/kibana/pull/199248)\n\n<!---
Backport version: 9.4.3 -->\n\n### Questions ?\nPlease refer to the
[Backport
tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT
[{\"author\":{\"name\":\"Karen\nGrigoryan\",\"email\":\"[email protected]\"},\"sourceCommit\":{\"committedDate\":\"2024-11-07T17:42:31Z\",\"message\":\"[Security\nSolution][Data
Quality Dashboard][Serverless] add start/end time support\nfor
latest_results (#199248)\\n\\naddresses #191053\\r\\n\\r\\n-
Introduce\n`defaultStartTime` and `defaultEndTime` props across
data\\r\\nquality\ncontext and panels for fetching latest_results and
align them\\r\\nwith\nserverless default time range of last week\\r\\n-
Update hooks to handle\nnew time parameters and include them in
storage\\r\\nresults queries.\\r\\n-\nModify server-side helpers and
routes to process and filter\nindices\\r\\nbased on the provided time
range.\\r\\n- Update related tests\nto accommodate the new
time\nparameters.\",\"sha\":\"1df04aef8d5859507c85a2ad37594075e9054b70\",\"branchLabelMapping\":{\"^v9.0.0$\":\"main\",\"^v8.17.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:skip\",\"v9.0.0\",\"Team:Threat\nHunting\",\"Team:Threat\nHunting:Explore\",\"backport:prev-minor\",\"ci:cloud-deploy\",\"ci:project-deploy-security\"],\"title\":\"[Security\nSolution][Data
Quality Dashboard][Serverless] add start/end time
support\nfor\nlatest_results\",\"number\":199248,\"url\":\"https://github.com/elastic/kibana/pull/199248\",\"mergeCommit\":{\"message\":\"[Security\nSolution][Data
Quality Dashboard][Serverless] add start/end time support\nfor
latest_results (#199248)\\n\\naddresses #191053\\r\\n\\r\\n-
Introduce\n`defaultStartTime` and `defaultEndTime` props across
data\\r\\nquality\ncontext and panels for fetching latest_results and
align them\\r\\nwith\nserverless default time range of last week\\r\\n-
Update hooks to handle\nnew time parameters and include them in
storage\\r\\nresults queries.\\r\\n-\nModify server-side helpers and
routes to process and filter\nindices\\r\\nbased on the provided time
range.\\r\\n- Update related tests\nto accommodate the new
time\nparameters.\",\"sha\":\"1df04aef8d5859507c85a2ad37594075e9054b70\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[],\"targetPullRequestStates\":[{\"branch\":\"main\",\"label\":\"v9.0.0\",\"branchLabelMappingKey\":\"^v9.0.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/199248\",\"number\":199248,\"mergeCommit\":{\"message\":\"[Security\nSolution][Data
Quality Dashboard][Serverless] add start/end time support\nfor
latest_results (#199248)\\n\\naddresses #191053\\r\\n\\r\\n-
Introduce\n`defaultStartTime` and `defaultEndTime` props across
data\\r\\nquality\ncontext and panels for fetching latest_results and
align them\\r\\nwith\nserverless default time range of last week\\r\\n-
Update hooks to handle\nnew time parameters and include them in
storage\\r\\nresults queries.\\r\\n-\nModify server-side helpers and
routes to process and filter\nindices\\r\\nbased on the provided time
range.\\r\\n- Update related tests\nto accommodate the new
time\nparameters.\",\"sha\":\"1df04aef8d5859507c85a2ad37594075e9054b70\"}}]}]\nBACKPORT-->\n\nCo-authored-by:
Karen Grigoryan <[email protected]>"}}]}] BACKPORT-->
@kapral18 kapral18 changed the title [Security Solution][Data quality dashboard][Serverless] results rollup incompatible counts not aligned [Security Solution][Data Quality Dashboard][Serverless] results rollup incompatible counts not aligned Nov 19, 2024
@MadameSheema
Copy link
Member

Issue seems to be fixed.

Image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kapral18 kapral18

Labels
bug Fixes for quality problems that affect the customer experience Feature:Data Health Quality Data health quality dashboard and related features fixed Team:Threat Hunting:Explore Team:Threat Hunting Security Solution Threat Hunting Team v8.16.1 v8.17.0 v9.0.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kapral18 @angorayc @elasticmachine @MadameSheema