Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[8.12] [RAM] Fix rule log aggregations when filtering >1k logs (#1…
…72228) (#174557) # Backport This will backport the following commits from `main` to `8.12`: - [[RAM] Fix rule log aggregations when filtering >1k logs (#172228)](#172228) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Zacqary Adam Xeper","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-01-09T23:17:13Z","message":"[RAM] Fix rule log aggregations when filtering >1k logs (#172228)\n\n## Summary\r\n\r\nFixes #171409 \r\n\r\nThis fixes weird behavior when attempting to filter more than 1000 rule\r\nexecution logs.\r\n\r\nThe execution log aggregator had two problems:\r\n\r\n- It had a max bucket limit of 1000, while the KPI agg had a max bucket\r\nlimit of 10000, leading to inconsistent results\r\n- For pagination, it reported the total number of logs by using a\r\ncardinality aggregator, which wasn't subject to any bucket limit at all\r\n\r\nThe endpoint responds with a `total` (the cardinality) and an array of\r\n`data` (a paginated slice of the aggregated logs). The way the data\r\ntable UI works is that it takes the `total` value, caps it at 1000, and\r\nthen generates that many blank rows to fill with whatever's in the\r\n`data` array.\r\n\r\nSo as seen in the issue, we could easily run into a situation where:\r\n\r\n1. User enters a filter query that last appeared more than 1000 logs ago\r\n2. The cardinality agg accurately reports the number of logs that should\r\nbe fetched, and generates enough blank rows for these logs\r\n3. The actual execution log agg hits the 1000 bucket limit, and returns\r\nan empty `data` array\r\n\r\nThis PR fixes this by using a bucket sum aggregation to determine the\r\ndata table's `total` instead of a cardinality aggregation.\r\n\r\nIt also ups the bucket limit from 1000 to 10000, to match the bucket\r\nlimit from the summary KPI endpoint. This prevents a situation where:\r\n\r\n1. User enters a filter query that last appeared in <1000 logs, but more\r\nthan 1000 logs ago\r\n2. Summary KPI widget, with a bucket limit of 10000, accurately displays\r\na count of the <1000 logs this query matches\r\n3. The data table is empty because the execution log bucket limit tops\r\nout at 1000\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>\r\nCo-authored-by: Xavier Mouligneau <[email protected]>","sha":"9d32f889683cb6bc099d561d94e02af2feaf0d10","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","Team:ResponseOps","Feature:Alerting/RulesManagement","v8.12.0","v8.13.0"],"title":"[RAM] Fix rule log aggregations when filtering >1k logs","number":172228,"url":"https://github.com/elastic/kibana/pull/172228","mergeCommit":{"message":"[RAM] Fix rule log aggregations when filtering >1k logs (#172228)\n\n## Summary\r\n\r\nFixes #171409 \r\n\r\nThis fixes weird behavior when attempting to filter more than 1000 rule\r\nexecution logs.\r\n\r\nThe execution log aggregator had two problems:\r\n\r\n- It had a max bucket limit of 1000, while the KPI agg had a max bucket\r\nlimit of 10000, leading to inconsistent results\r\n- For pagination, it reported the total number of logs by using a\r\ncardinality aggregator, which wasn't subject to any bucket limit at all\r\n\r\nThe endpoint responds with a `total` (the cardinality) and an array of\r\n`data` (a paginated slice of the aggregated logs). The way the data\r\ntable UI works is that it takes the `total` value, caps it at 1000, and\r\nthen generates that many blank rows to fill with whatever's in the\r\n`data` array.\r\n\r\nSo as seen in the issue, we could easily run into a situation where:\r\n\r\n1. User enters a filter query that last appeared more than 1000 logs ago\r\n2. The cardinality agg accurately reports the number of logs that should\r\nbe fetched, and generates enough blank rows for these logs\r\n3. The actual execution log agg hits the 1000 bucket limit, and returns\r\nan empty `data` array\r\n\r\nThis PR fixes this by using a bucket sum aggregation to determine the\r\ndata table's `total` instead of a cardinality aggregation.\r\n\r\nIt also ups the bucket limit from 1000 to 10000, to match the bucket\r\nlimit from the summary KPI endpoint. This prevents a situation where:\r\n\r\n1. User enters a filter query that last appeared in <1000 logs, but more\r\nthan 1000 logs ago\r\n2. Summary KPI widget, with a bucket limit of 10000, accurately displays\r\na count of the <1000 logs this query matches\r\n3. The data table is empty because the execution log bucket limit tops\r\nout at 1000\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>\r\nCo-authored-by: Xavier Mouligneau <[email protected]>","sha":"9d32f889683cb6bc099d561d94e02af2feaf0d10"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","branchLabelMappingKey":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/172228","number":172228,"mergeCommit":{"message":"[RAM] Fix rule log aggregations when filtering >1k logs (#172228)\n\n## Summary\r\n\r\nFixes #171409 \r\n\r\nThis fixes weird behavior when attempting to filter more than 1000 rule\r\nexecution logs.\r\n\r\nThe execution log aggregator had two problems:\r\n\r\n- It had a max bucket limit of 1000, while the KPI agg had a max bucket\r\nlimit of 10000, leading to inconsistent results\r\n- For pagination, it reported the total number of logs by using a\r\ncardinality aggregator, which wasn't subject to any bucket limit at all\r\n\r\nThe endpoint responds with a `total` (the cardinality) and an array of\r\n`data` (a paginated slice of the aggregated logs). The way the data\r\ntable UI works is that it takes the `total` value, caps it at 1000, and\r\nthen generates that many blank rows to fill with whatever's in the\r\n`data` array.\r\n\r\nSo as seen in the issue, we could easily run into a situation where:\r\n\r\n1. User enters a filter query that last appeared more than 1000 logs ago\r\n2. The cardinality agg accurately reports the number of logs that should\r\nbe fetched, and generates enough blank rows for these logs\r\n3. The actual execution log agg hits the 1000 bucket limit, and returns\r\nan empty `data` array\r\n\r\nThis PR fixes this by using a bucket sum aggregation to determine the\r\ndata table's `total` instead of a cardinality aggregation.\r\n\r\nIt also ups the bucket limit from 1000 to 10000, to match the bucket\r\nlimit from the summary KPI endpoint. This prevents a situation where:\r\n\r\n1. User enters a filter query that last appeared in <1000 logs, but more\r\nthan 1000 logs ago\r\n2. Summary KPI widget, with a bucket limit of 10000, accurately displays\r\na count of the <1000 logs this query matches\r\n3. The data table is empty because the execution log bucket limit tops\r\nout at 1000\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>\r\nCo-authored-by: Xavier Mouligneau <[email protected]>","sha":"9d32f889683cb6bc099d561d94e02af2feaf0d10"}}]}] BACKPORT--> Co-authored-by: Zacqary Adam Xeper <[email protected]>
- Loading branch information
1 parent
47b78f0
commit cec6529