Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: Rules Management - Rule Bulk Actions Integration Tests - ESS Env - Trial License.x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_bulk_actions/trial_license_complete_tier/perform_bulk_action_ess·ts - Rules Management - Rule Bulk Action API @ess perform_bulk_action - ESS specific logic edit action should migrate legacy actions on edit #196470

Closed
kibanamachine opened this issue Oct 16, 2024 · 21 comments · Fixed by #197660
Closed

Failing test: Rules Management - Rule Bulk Actions Integration Tests - ESS Env - Trial License.x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_bulk_actions/trial_license_complete_tier/perform_bulk_action_ess·ts - Rules Management - Rule Bulk Action API @ess perform_bulk_action - ESS specific logic edit action should migrate legacy actions on edit #196470

kibanamachine opened this issue Oct 16, 2024 · 21 comments · Fixed by #197660
Assignees
@maximpn
Labels
8.16 candidate failed-test A test failure on a tracked branch, potentially flaky-test Feature:Rule Management Security Solution Detection Rule Management area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Oct 16, 2024 •
edited
Loading

A test failed on a tracked branch

TypeError: Cannot read properties of undefined (reading 'uuid')
    at Context.<anonymous> (perform_bulk_action_ess.ts:400:42)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Object.apply (wrap_function.js:74:16)

First failure: kibana-on-merge - main
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Oct 16, 2024
@botelastic botelastic bot added the needs-team Issues missing a team label label Oct 16, 2024
@kibanamachine kibanamachine added the Team:Detection Rule Management Security Detection Rule Management Team label Oct 16, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Oct 16, 2024
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.x

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.x

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@jbudz
Copy link
Member

jbudz commented Oct 16, 2024

/skip

kibanamachine added a commit that referenced this issue Oct 16, 2024
@kibanamachine
skip failing test suite (#196470)
4edb91b
@kibanamachine
Copy link
Contributor Author

Skipped

main: 4edb91b
8.x: 4213ff6

kibanamachine added a commit that referenced this issue Oct 16, 2024
@kibanamachine
skip failing test suite (#196470)
4213ff6
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.x

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.x

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@banderror banderror added the Team:Detections and Resp Security Detection Response Team label Oct 22, 2024
@banderror banderror added Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Feature:Rule Management Security Solution Detection Rule Management area 8.16 candidate labels Oct 22, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@banderror banderror mentioned this issue Oct 22, 2024
Open
@maximpn maximpn mentioned this issue Oct 25, 2024
Merged
maximpn added a commit that referenced this issue Nov 5, 2024
@maximpn
[Security Solution] Unskip perform bulk action ESS FTR tests (#197660)
bf37a01 
**Closes: #196470
**Closes: #196462

## Summary

This PR unskips `perform_bulk_action_ess.ts` functional test.

## Details

`perform_bulk_action_ess.ts` includes a number of functional tests where some of them were flaky. Investigation revealed that creating enabled rules and performing bulk actions may lead to race conditions. Under that conditions rule's SO isn't updated as expected. For example legacy rule actions aren't persisted in rule's SO when it's expected by the test.

This PR includes `perform_bulk_action_ess.ts` refactoring to create disabled rules in the majority of tests. Enabled rules are created only in tests checking behavior upon rules disabling. These tests were checked multiple times and didn't demonstrate flakiness.

Additionally `@kbn/expect` was replaced with `expect` to make asserting more transparent and avoid unclear error messages. 

## Flaky test results

[100 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7274) 🟢
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Nov 5, 2024
@maximpn
[Security Solution] Unskip perform bulk action ESS FTR tests (elastic…
34cd80b 
…#197660)

**Closes: elastic#196470
**Closes: elastic#196462

## Summary

This PR unskips `perform_bulk_action_ess.ts` functional test.

## Details

`perform_bulk_action_ess.ts` includes a number of functional tests where some of them were flaky. Investigation revealed that creating enabled rules and performing bulk actions may lead to race conditions. Under that conditions rule's SO isn't updated as expected. For example legacy rule actions aren't persisted in rule's SO when it's expected by the test.

This PR includes `perform_bulk_action_ess.ts` refactoring to create disabled rules in the majority of tests. Enabled rules are created only in tests checking behavior upon rules disabling. These tests were checked multiple times and didn't demonstrate flakiness.

Additionally `@kbn/expect` was replaced with `expect` to make asserting more transparent and avoid unclear error messages.

## Flaky test results

[100 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7274) 🟢

(cherry picked from commit bf37a01)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Nov 5, 2024
@maximpn
[Security Solution] Unskip perform bulk action ESS FTR tests (elastic…
3792382 
…#197660)

**Closes: elastic#196470
**Closes: elastic#196462

## Summary

This PR unskips `perform_bulk_action_ess.ts` functional test.

## Details

`perform_bulk_action_ess.ts` includes a number of functional tests where some of them were flaky. Investigation revealed that creating enabled rules and performing bulk actions may lead to race conditions. Under that conditions rule's SO isn't updated as expected. For example legacy rule actions aren't persisted in rule's SO when it's expected by the test.

This PR includes `perform_bulk_action_ess.ts` refactoring to create disabled rules in the majority of tests. Enabled rules are created only in tests checking behavior upon rules disabling. These tests were checked multiple times and didn't demonstrate flakiness.

Additionally `@kbn/expect` was replaced with `expect` to make asserting more transparent and avoid unclear error messages.

## Flaky test results

[100 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7274) 🟢

(cherry picked from commit bf37a01)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@maximpn maximpn

Labels
8.16 candidate failed-test A test failure on a tracked branch, potentially flaky-test Feature:Rule Management Security Solution Detection Rule Management area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Security Solution] Unskip perform bulk action ESS FTR tests maximpn/kibana
6 participants
@xcrzx @jbudz @maximpn @banderror @elasticmachine @kibanamachine