Failing Test: x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/rule_actions/bulk_actions/bulk_edit_rules_actions.cy·ts #171101
Assignees
Labels
8.13 candidate
failed-test
A test failure on a tracked branch, potentially flaky-test
Feature:Rule Management
Security Solution Detection Rule Management area
legit-flake
Test was triaged and marked as an actual flake.
Team:Detection Rule Management
Security Detection Rule Management Team
Team:Detections and Resp
Security Detection Response Team
Team: SecuritySolution
Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
Comments
jpdjere
added
test
triage_needed
Team:Detections and Resp
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
banderror
added
failed-test
nikitaindik
added a commit
that referenced
this issue
Jan 10, 2024
## Summary **Resolves: #171101 200 runs of bulk_edit_rules_actions.cy.ts in ESS env: [*Buildkite 4776*](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4776) 200 runs of bulk_edit_rules_actions.cy.ts in Serverless env: [*Buildkite 4777*](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4777) All tests were green except a couple tests that stopped abruptly mid-way because of CI runner timeouts. So there weren't failed tests. Two issues were causing fails: 1. `"hunter_no_actions"` role that was used in "User with no privileges can't add rule actions" test doesn't exist in Serverless env. Changed it to `"t1_analyst"` – it exists in both ESS and Serverless and doesn't give permission to edit rules. 2. Race condition caused by disabled auto-refresh - In the `beforeAll` hook the auto-refresh is disabled for the Rule Management page. - Then `excessivelyInstallAllPrebuiltRules` is called, which installs all 1000+ prebuilt rules (only in Serverless, because ESS has 0 prebuilt rules available to install). - While the installation is in progress the Rule Management page loads and displays 9 rules. - Then the test selects all rules (9) and executes a bulk update on them. - Once the bulk action succeeds, the user sees a toast with "1000+ actions have been updated" while the test expects "9 actions have been updated" because of disabled auto-refresh. - I decided to skip installing all the Elastic prebuilt rules because the operation is very heavy and we check that bulk actions work for prebuilt rules anyways since two test prebuilt rules are created and installed in beforeAll.
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Jan 10, 2024
## Summary **Resolves: elastic#171101 200 runs of bulk_edit_rules_actions.cy.ts in ESS env: [*Buildkite 4776*](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4776) 200 runs of bulk_edit_rules_actions.cy.ts in Serverless env: [*Buildkite 4777*](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4777) All tests were green except a couple tests that stopped abruptly mid-way because of CI runner timeouts. So there weren't failed tests. Two issues were causing fails: 1. `"hunter_no_actions"` role that was used in "User with no privileges can't add rule actions" test doesn't exist in Serverless env. Changed it to `"t1_analyst"` – it exists in both ESS and Serverless and doesn't give permission to edit rules. 2. Race condition caused by disabled auto-refresh - In the `beforeAll` hook the auto-refresh is disabled for the Rule Management page. - Then `excessivelyInstallAllPrebuiltRules` is called, which installs all 1000+ prebuilt rules (only in Serverless, because ESS has 0 prebuilt rules available to install). - While the installation is in progress the Rule Management page loads and displays 9 rules. - Then the test selects all rules (9) and executes a bulk update on them. - Once the bulk action succeeds, the user sees a toast with "1000+ actions have been updated" while the test expects "9 actions have been updated" because of disabled auto-refresh. - I decided to skip installing all the Elastic prebuilt rules because the operation is very heavy and we check that bulk actions work for prebuilt rules anyways since two test prebuilt rules are created and installed in beforeAll. (cherry picked from commit 53ffb14)
kibanamachine
referenced
this issue
Jan 10, 2024
#174599) # Backport This will backport the following commits from `main` to `8.12`: - [[Security Solution] Unskip bulk actions Cypress tests (#174365)](#174365) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Nikita Indik","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-01-10T13:23:31Z","message":"[Security Solution] Unskip bulk actions Cypress tests (#174365)\n\n## Summary\r\n\r\n**Resolves: https://github.com/elastic/kibana/issues/171101**\r\n\r\n200 runs of bulk_edit_rules_actions.cy.ts in ESS env: [*Buildkite\r\n4776*](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4776)\r\n200 runs of bulk_edit_rules_actions.cy.ts in Serverless env: [*Buildkite\r\n4777*](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4777)\r\n\r\nAll tests were green except a couple tests that stopped abruptly mid-way\r\nbecause of CI runner timeouts. So there weren't failed tests.\r\n\r\nTwo issues were causing fails:\r\n1. `\"hunter_no_actions\"` role that was used in \"User with no privileges\r\ncan't add rule actions\" test doesn't exist in Serverless env. Changed it\r\nto `\"t1_analyst\"` – it exists in both ESS and Serverless and doesn't\r\ngive permission to edit rules.\r\n2. Race condition caused by disabled auto-refresh\r\n- In the `beforeAll` hook the auto-refresh is disabled for the Rule\r\nManagement page.\r\n- Then `excessivelyInstallAllPrebuiltRules` is called, which installs\r\nall 1000+ prebuilt rules (only in Serverless, because ESS has 0 prebuilt\r\nrules available to install).\r\n- While the installation is in progress the Rule Management page loads\r\nand displays 9 rules.\r\n- Then the test selects all rules (9) and executes a bulk update on\r\nthem.\r\n- Once the bulk action succeeds, the user sees a toast with \"1000+\r\nactions have been updated\" while the test expects \"9 actions have been\r\nupdated\" because of disabled auto-refresh.\r\n- I decided to skip installing all the Elastic prebuilt rules because\r\nthe operation is very heavy and we check that bulk actions work for\r\nprebuilt rules anyways since two test prebuilt rules are created and\r\ninstalled in beforeAll.","sha":"53ffb143ed1e105f448632c27c1d7b858e13386e","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","test-coverage","Team:Detections and Resp","Team: SecuritySolution","Feature:Rule Management","Team:Detection Rule Management","v8.12.1","v8.13.0"],"title":"[Security Solution] Unskip bulk actions Cypress tests","number":174365,"url":"https://github.com/elastic/kibana/pull/174365","mergeCommit":{"message":"[Security Solution] Unskip bulk actions Cypress tests (#174365)\n\n## Summary\r\n\r\n**Resolves: https://github.com/elastic/kibana/issues/171101**\r\n\r\n200 runs of bulk_edit_rules_actions.cy.ts in ESS env: [*Buildkite\r\n4776*](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4776)\r\n200 runs of bulk_edit_rules_actions.cy.ts in Serverless env: [*Buildkite\r\n4777*](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4777)\r\n\r\nAll tests were green except a couple tests that stopped abruptly mid-way\r\nbecause of CI runner timeouts. So there weren't failed tests.\r\n\r\nTwo issues were causing fails:\r\n1. `\"hunter_no_actions\"` role that was used in \"User with no privileges\r\ncan't add rule actions\" test doesn't exist in Serverless env. Changed it\r\nto `\"t1_analyst\"` – it exists in both ESS and Serverless and doesn't\r\ngive permission to edit rules.\r\n2. Race condition caused by disabled auto-refresh\r\n- In the `beforeAll` hook the auto-refresh is disabled for the Rule\r\nManagement page.\r\n- Then `excessivelyInstallAllPrebuiltRules` is called, which installs\r\nall 1000+ prebuilt rules (only in Serverless, because ESS has 0 prebuilt\r\nrules available to install).\r\n- While the installation is in progress the Rule Management page loads\r\nand displays 9 rules.\r\n- Then the test selects all rules (9) and executes a bulk update on\r\nthem.\r\n- Once the bulk action succeeds, the user sees a toast with \"1000+\r\nactions have been updated\" while the test expects \"9 actions have been\r\nupdated\" because of disabled auto-refresh.\r\n- I decided to skip installing all the Elastic prebuilt rules because\r\nthe operation is very heavy and we check that bulk actions work for\r\nprebuilt rules anyways since two test prebuilt rules are created and\r\ninstalled in beforeAll.","sha":"53ffb143ed1e105f448632c27c1d7b858e13386e"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","branchLabelMappingKey":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/174365","number":174365,"mergeCommit":{"message":"[Security Solution] Unskip bulk actions Cypress tests (#174365)\n\n## Summary\r\n\r\n**Resolves: https://github.com/elastic/kibana/issues/171101**\r\n\r\n200 runs of bulk_edit_rules_actions.cy.ts in ESS env: [*Buildkite\r\n4776*](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4776)\r\n200 runs of bulk_edit_rules_actions.cy.ts in Serverless env: [*Buildkite\r\n4777*](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4777)\r\n\r\nAll tests were green except a couple tests that stopped abruptly mid-way\r\nbecause of CI runner timeouts. So there weren't failed tests.\r\n\r\nTwo issues were causing fails:\r\n1. `\"hunter_no_actions\"` role that was used in \"User with no privileges\r\ncan't add rule actions\" test doesn't exist in Serverless env. Changed it\r\nto `\"t1_analyst\"` – it exists in both ESS and Serverless and doesn't\r\ngive permission to edit rules.\r\n2. Race condition caused by disabled auto-refresh\r\n- In the `beforeAll` hook the auto-refresh is disabled for the Rule\r\nManagement page.\r\n- Then `excessivelyInstallAllPrebuiltRules` is called, which installs\r\nall 1000+ prebuilt rules (only in Serverless, because ESS has 0 prebuilt\r\nrules available to install).\r\n- While the installation is in progress the Rule Management page loads\r\nand displays 9 rules.\r\n- Then the test selects all rules (9) and executes a bulk update on\r\nthem.\r\n- Once the bulk action succeeds, the user sees a toast with \"1000+\r\nactions have been updated\" while the test expects \"9 actions have been\r\nupdated\" because of disabled auto-refresh.\r\n- I decided to skip installing all the Elastic prebuilt rules because\r\nthe operation is very heavy and we check that bulk actions work for\r\nprebuilt rules anyways since two test prebuilt rules are created and\r\ninstalled in beforeAll.","sha":"53ffb143ed1e105f448632c27c1d7b858e13386e"}}]}] BACKPORT--> Co-authored-by: Nikita Indik <[email protected]>
delanni
pushed a commit
to delanni/kibana
that referenced
this issue
Jan 11, 2024
## Summary **Resolves: elastic#171101 200 runs of bulk_edit_rules_actions.cy.ts in ESS env: [*Buildkite 4776*](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4776) 200 runs of bulk_edit_rules_actions.cy.ts in Serverless env: [*Buildkite 4777*](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4777) All tests were green except a couple tests that stopped abruptly mid-way because of CI runner timeouts. So there weren't failed tests. Two issues were causing fails: 1. `"hunter_no_actions"` role that was used in "User with no privileges can't add rule actions" test doesn't exist in Serverless env. Changed it to `"t1_analyst"` – it exists in both ESS and Serverless and doesn't give permission to edit rules. 2. Race condition caused by disabled auto-refresh - In the `beforeAll` hook the auto-refresh is disabled for the Rule Management page. - Then `excessivelyInstallAllPrebuiltRules` is called, which installs all 1000+ prebuilt rules (only in Serverless, because ESS has 0 prebuilt rules available to install). - While the installation is in progress the Rule Management page loads and displays 9 rules. - Then the test selects all rules (9) and executes a bulk update on them. - Once the bulk action succeeds, the user sees a toast with "1000+ actions have been updated" while the test expects "9 actions have been updated" because of disabled auto-refresh. - I decided to skip installing all the Elastic prebuilt rules because the operation is very heavy and we check that bulk actions work for prebuilt rules anyways since two test prebuilt rules are created and installed in beforeAll.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Summary
The whole test suite in:
was skipped as flake was reported during the CI ran of #170663
NOTE: This test suite was only reported as flakey in 8.12, so it was skipped in
mainonly (i.e. was not backported to 8.11)The text was updated successfully, but these errors were encountered: