Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: Security Solution Cypress.x-pack/plugins/security_solution/public/management/cypress/e2e/automated_response_actions/automated_response_actions·cy·ts - Automated Response Actions From alerts "before all" hook for "should have generated endpoint and rule" "before all" hook for "should have generated endpoint and rule" #168340

Open
kibanamachine opened this issue Oct 9, 2023 · 284 comments · Fixed by #179743, #181655, #186073 or #188013
Open

Failing test: Security Solution Cypress.x-pack/plugins/security_solution/public/management/cypress/e2e/automated_response_actions/automated_response_actions·cy·ts - Automated Response Actions From alerts "before all" hook for "should have generated endpoint and rule" "before all" hook for "should have generated endpoint and rule" #168340

kibanamachine opened this issue Oct 9, 2023 · 284 comments · Fixed by #179743, #181655, #186073 or #188013
Assignees
@gergoabraham
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Defend Workflows “EDR Workflows” sub-team of Security Solution

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Oct 9, 2023 •
edited
Loading

A test failed on a tracked branch

CypressError: `cy.task('createEndpointHost')` timed out after waiting `600000ms`.

https://on.cypress.io/api/task

Because this error occurred during a `before all` hook we are skipping the remaining tests in the current suite: `Automated Response Actions`

Although you have test retries enabled, we do not retry tests when `before all` or `after all` hooks fail
    at <unknown> (http://localhost:5652/__cypress/runner/cypress_runner.js:135256:70)
    at tryCatcher (http://localhost:5652/__cypress/runner/cypress_runner.js:1807:23)
    at <unknown> (http://localhost:5652/__cypress/runner/cypress_runner.js:4186:41)
    at tryCatcher (http://localhost:5652/__cypress/runner/cypress_runner.js:1807:23)
    at Promise._settlePromiseFromHandler (http://localhost:5652/__cypress/runner/cypress_runner.js:1519:31)
    at Promise._settlePromise (http://localhost:5652/__cypress/runner/cypress_runner.js:1576:18)
    at Promise._settlePromise0 (http://localhost:5652/__cypress/runner/cypress_runner.js:1621:10)
    at Promise._settlePromises (http://localhost:5652/__cypress/runner/cypress_runner.js:1697:18)
    at _drainQueueStep (http://localhost:5652/__cypress/runner/cypress_runner.js:2407:12)
    at _drainQueue (http://localhost:5652/__cypress/runner/cypress_runner.js:2400:9)
    at Async._drainQueues (http://localhost:5652/__cypress/runner/cypress_runner.js:2416:5)
    at Async.drainQueues (http://localhost:5652/__cypress/runner/cypress_runner.js:2286:14)
From Your Spec Code:
    at createEndpointHost (webpack:///./tasks/create_endpoint_host.ts:15:12)
    at Context.eval (webpack:///./e2e/automated_response_actions/automated_response_actions.cy.ts:39:38)

First failure: CI Build - main
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Oct 9, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label Oct 9, 2023
@kibanamachine kibanamachine added the Team:Defend Workflows “EDR Workflows” sub-team of Security Solution label Oct 9, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Oct 9, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

mistic added a commit that referenced this issue Oct 13, 2023
@mistic
skip flaky suite (#168340)
50c59af
@mistic
Copy link
Member

mistic commented Oct 13, 2023

Skipped.

main: 50c59af

dej611 pushed a commit to dej611/kibana that referenced this issue Oct 17, 2023
@mistic @dej611
skip flaky suite (elastic#168340)
618c21c
@szwarckonrad szwarckonrad mentioned this issue Oct 25, 2023
Merged
szwarckonrad added a commit that referenced this issue Oct 25, 2023
@szwarckonrad @patrykkopycinski
[EDR Workflows][E2E] Recreate agent on createEndpointHost task fail (#…
1fdcb41 
…169092)

Restart vagrant vm on error during `beforeAll` task `createEndpointHost`

Defend Workflows Cypress suite ran 300 times through flaky test runner:
1. 100x
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3699
2. 50x
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3707
3. 50x
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3708
4. 50x
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3709
5. 50x
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3710


Flaky test runner runs with `createEndpointHost` task failure with
successful recovery:
1.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3710#018b62fd-9ae9-4988-b1e0-ab0f04d8efdc
2.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3710#018b62fd-9ae6-4340-992b-1474ee0f114b
3.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3708#018b62fd-578e-4817-ae1c-8c58e8774eec
4.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3708#018b62fd-5787-4245-85a6-cb446e42bc73
5.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3707#018b62fc-fc17-407e-88de-d0b43b6b1d44
(failed due to unrelated issue)
6.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3699#018b61d9-d2c3-430c-b3e3-72b9fbb22d24
7.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3699#018b61d9-d2c6-4315-b828-b3218a70f209
8.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3699#018b61d9-d2c7-4ff7-9a70-7354f90179e0
9.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3699#018b61d9-d2d7-418f-b043-049e5effb26f
10.
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3699#018b61d9-d2da-47cc-b4ea-a4d4de3ba0a0

New errors not spotted before that got to do with env set up:

1. `vagrant up` failed:
1.1
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3708#018b62fd-5787-4245-85a6-cb446e42bc73
1.2
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3699#018b61d9-d2d0-4a52-87d9-34caa8927465

2. `CypressError: `cy.task('indexFleetEndpointPolicy')` timed out after
waiting `60000ms`.:
2.1
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3707#018b62fc-fc04-40d4-b155-46f094681edb
2.2
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3699#018b61d9-d2c9-4ebb-9174-eb9d79d04d02
2.3
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3699#018b61d9-d2dc-438f-94b0-9f94ae95701c
    

Closes:
#168284
#169343
#169468
#169469
#169467
#169465
#169466
#169157
#168719
#168427
#168359
#168340
#169689

---------

Co-authored-by: Patryk Kopyciński <[email protected]>
@kibanamachine kibanamachine reopened this Nov 7, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kevinlog
Copy link
Contributor

kevinlog commented Nov 7, 2023

New failure, looks like a rate limit, 429, on the VM url. Is it possible to just keep a local copy of the VM for these tests instead of downloading it every time?

cc @dasansol92

==> default: VM not created. Moving on...
--
  | Bringing machine 'default' up with 'virtualbox' provider...
  | ==> default: Box 'ubuntu/jammy64' could not be found. Attempting to find and install...
  | default: Box Provider: virtualbox
  | default: Box Version: >= 0
  | ERROR Error: Command failed with exit code 1: vagrant up
  | The box 'ubuntu/jammy64' could not be found or
  | could not be accessed in the remote catalog. If this is a private
  | box on HashiCorp's Vagrant Cloud, please verify you're logged in via
  | `vagrant login`. Also, please double-check the name. The expanded
  | URL and error message are shown below:
  |  
  | URL: ["https://vagrantcloud.com/ubuntu/jammy64"]
  | Error: The requested URL returned error: 429
  | at makeError (/opt/local-ssd/buildkite/builds/kb-n2-4-virt-935156c9a35dc919/elastic/kibana-on-merge/kibana/node_modules/execa/lib/error.js:60:11)
  | at handlePromise (/opt/local-ssd/buildkite/builds/kb-n2-4-virt-935156c9a35dc919/elastic/kibana-on-merge/kibana/node_modules/execa/index.js:118:26)
  | at processTicksAndRejections (node:internal/process/task_queues:95:5)
  | at async createVagrantVm (/opt/local-ssd/buildkite/builds/kb-n2-4-virt-935156c9a35dc919/elastic/kibana-on-merge/kibana/x-pack/plugins/security_solution/scripts/endpoint/common/endpoint_host_services.ts:188:5)
  | at async Promise.all (index 0)
  | at async createAndEnrollEndpointHost (/opt/local-ssd/buildkite/builds/kb-n2-4-virt-935156c9a35dc919/elastic/kibana-on-merge/kibana/x-pack/plugins/security_solution/scripts/endpoint/common/endpoint_host_services.ts:85:49)
  | at async attemptCreateEndpointHost (/opt/local-ssd/buildkite/builds/kb-n2-4-virt-935156c9a35dc919/elastic/kibana-on-merge/kibana/x-pack/plugins/security_solution/public/management/cypress/support/data_loaders.ts:340:27)
  | info Caught error when setting up the agent: Error: Command failed with exit code 1: vagrant up
  | The box 'ubuntu/jammy64' could not be found or
  | could not be accessed in the remote catalog. If this is a private
  | box on HashiCorp's Vagrant Cloud, please verify you're logged in via
  | `vagrant login`. Also, please double-check the name. The expanded
  | URL and error message are shown below:
  |  
  | URL: ["https://vagrantcloud.com/ubuntu/jammy64"]
  | Error: The requested URL returned error: 429

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

mistic added a commit that referenced this issue Nov 8, 2023
@mistic
skip flaky suite (#168340)
70fb99b
@mistic mistic added the blocker label Nov 8, 2023
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

1 similar comment
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine kibanamachine reopened this Oct 15, 2024
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gergoabraham gergoabraham

Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Defend Workflows “EDR Workflows” sub-team of Security Solution
Projects
None yet
Milestone
No milestone
9 participants
@ashokaditya @jbudz @mistic @dasansol92 @elasticmachine @szwarckonrad @gergoabraham @kibanamachine @kevinlog