Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: Rules Management - Rule Bulk Actions Integration Tests - ESS Env - Trial License.x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_bulk_actions/trial_license_complete_tier/perform_bulk_action_ess·ts - Rules Management - Rule Bulk Action API @ess perform_bulk_action - ESS specific logic should disable rules and migrate actions #196462

Open
2 of 3 tasks
kibanamachine opened this issue Oct 16, 2024 · 17 comments · Fixed by #197660
Open
2 of 3 tasks

Failing test: Rules Management - Rule Bulk Actions Integration Tests - ESS Env - Trial License.x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_bulk_actions/trial_license_complete_tier/perform_bulk_action_ess·ts - Rules Management - Rule Bulk Action API @ess perform_bulk_action - ESS specific logic should disable rules and migrate actions #196462

kibanamachine opened this issue Oct 16, 2024 · 17 comments · Fixed by #197660
Assignees
@maximpn
Labels
8.18 candidate failed-test A test failure on a tracked branch, potentially flaky-test Feature:Rule Management Security Solution Detection Rule Management area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Oct 16, 2024 •
edited by banderror
Loading

A test failed on a tracked branch

TypeError: Cannot read properties of undefined (reading 'uuid')
    at Context.<anonymous> (perform_bulk_action_ess.ts:209:37)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Object.apply (wrap_function.js:74:16)

First failure: kibana-on-merge - main

Todo
Preview Give feedback
  1. [Security Solution] Unskip perform bulk action ESS FTR tests #197660
    Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp backport:prev-major failed-test release_note:skip test v8.16.0 v8.17.0 v9.0.0
    maximpn
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Oct 16, 2024
@botelastic botelastic bot added the needs-team Issues missing a team label label Oct 16, 2024
@kibanamachine kibanamachine added the Team:Detection Rule Management Security Detection Rule Management Team label Oct 16, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Oct 16, 2024
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.x

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.x

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@jbudz
Copy link
Member

jbudz commented Oct 16, 2024

/skip

kibanamachine added a commit that referenced this issue Oct 16, 2024
@kibanamachine
skip failing test suite (#196462)
9bca8b7
@kibanamachine
Copy link
Contributor Author

Skipped

main: 9bca8b7
8.x: 5f86c95

kibanamachine added a commit that referenced this issue Oct 16, 2024
@kibanamachine
skip failing test suite (#196462)
5f86c95
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.x

@banderror banderror added Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Feature:Rule Management Security Solution Detection Rule Management area 8.16 candidate labels Oct 22, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@banderror
Copy link
Contributor

@maximpn @xcrzx 👋

I assigned you to this one and #196470. By the end of this week we need to figure out who can work on these. The goal is to understand if these test failures are indications of a bug - in which case we will have less than one week for fixing it for v8.16.0 according to the release schedule.

@maximpn maximpn mentioned this issue Oct 28, 2024
Merged
@maximpn
Copy link
Contributor

maximpn commented Oct 28, 2024

@banderror I created #197660 to address flakiness reported in this ticket and in #196470. It seems we might have race condition when performing bulk operations on enabled rules picked up by the task manager at the same time. I left only two functional tests with enabled rules since these tests check rule disabling. After applying my changes don't see any flakiness.

@maximpn maximpn closed this as completed in bf37a01 Nov 5, 2024
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Nov 5, 2024
@maximpn
[Security Solution] Unskip perform bulk action ESS FTR tests (elastic…
34cd80b 
…#197660)

**Closes: elastic#196470
**Closes: elastic#196462

## Summary

This PR unskips `perform_bulk_action_ess.ts` functional test.

## Details

`perform_bulk_action_ess.ts` includes a number of functional tests where some of them were flaky. Investigation revealed that creating enabled rules and performing bulk actions may lead to race conditions. Under that conditions rule's SO isn't updated as expected. For example legacy rule actions aren't persisted in rule's SO when it's expected by the test.

This PR includes `perform_bulk_action_ess.ts` refactoring to create disabled rules in the majority of tests. Enabled rules are created only in tests checking behavior upon rules disabling. These tests were checked multiple times and didn't demonstrate flakiness.

Additionally `@kbn/expect` was replaced with `expect` to make asserting more transparent and avoid unclear error messages.

## Flaky test results

[100 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7274) 🟢

(cherry picked from commit bf37a01)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Nov 5, 2024
@maximpn
[Security Solution] Unskip perform bulk action ESS FTR tests (elastic…
3792382 
…#197660)

**Closes: elastic#196470
**Closes: elastic#196462

## Summary

This PR unskips `perform_bulk_action_ess.ts` functional test.

## Details

`perform_bulk_action_ess.ts` includes a number of functional tests where some of them were flaky. Investigation revealed that creating enabled rules and performing bulk actions may lead to race conditions. Under that conditions rule's SO isn't updated as expected. For example legacy rule actions aren't persisted in rule's SO when it's expected by the test.

This PR includes `perform_bulk_action_ess.ts` refactoring to create disabled rules in the majority of tests. Enabled rules are created only in tests checking behavior upon rules disabling. These tests were checked multiple times and didn't demonstrate flakiness.

Additionally `@kbn/expect` was replaced with `expect` to make asserting more transparent and avoid unclear error messages.

## Flaky test results

[100 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7274) 🟢

(cherry picked from commit bf37a01)
@kibanamachine kibanamachine reopened this Nov 5, 2024
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.16

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.16

@maximpn maximpn mentioned this issue Nov 6, 2024
Merged
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.16

maximpn added a commit that referenced this issue Nov 7, 2024
@maximpn
[8.16] [Security Solution] Skip a flaky legacy actions migration upon…
e29cc5f 
… bulk action FTR test (#199200)

**Addresses:** #196462

## Summary

`perform_bulk_action_ess.ts`'s `ESS specific logic` -> `should disable rules and migrate actions` FTR test is flaky in `8.16`. This skips only this particular test.

## More details on flakiness

Flakiness isn't reproducible in `main` which is `v9.0` currently.  It's suspected that modification of rule's SO interferences with rules bulk action performing a legacy action migration. Though a quick fix like waiting for rule execution completion before performing bulk action didn't help. It requires further investigation.

Another idea `8.16` could require some PR from to be backported which fixed this issue in `main`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@maximpn maximpn

Labels
8.18 candidate failed-test A test failure on a tracked branch, potentially flaky-test Feature:Rule Management Security Solution Detection Rule Management area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Security Solution] Unskip perform bulk action ESS FTR tests maximpn/kibana
6 participants
@xcrzx @jbudz @maximpn @banderror @elasticmachine @kibanamachine