From 80c575f06718bad8a86246241bedc2dac91eeb26 Mon Sep 17 00:00:00 2001 From: Robert Austin Date: Fri, 6 Oct 2023 11:58:31 -0400 Subject: [PATCH] Enable Security's Cypress tests on all PRs (#167516) ## Summary Security Solution writes e2e and other tests using Cypress. In the past, these tests, if they failed on a tracked branch, couldn't be easily skipped. They also weren't run in parallelized jobs. For primarily these reasons, they didn't run on most Kibana PRs. This PR moves these Cypress tests back to the main PR pipeline. Tests that fail on tracked branches create (or update) Github issues which can be used with the skip-test github workflow script to easily skip the failing tests. The pipeline steps are parallelized and run in under 40 minutes. ### Open Questions - [ ] Should this PR enable Serverless Security Defend Workflows Cypress Tests @patrykkopycinski ### Some buildkite pipelines that used to run only on Security PRs now run on all PRs: These steps run on all PRs with these changes - Security Solution Cypress Tests (general tests that haven't been organized into an area team) - Explore tests - Investigations Tests - Defend Workflows Tests - Defend Workflows Serverless - Threat Intel Tests - OS Query Tests - Security Solution Burning Changed Specs (these run only recently changed specs a few extra times) - Security Solution OpenAPI codegen - OSQuery burning - OSQuery Serverless
And these already run on all PRs - Serverless Security Cypress Tests - Serverless Explore tests - Serverless Investigations Tests
### Security Cypress tests run in the main `on merge` pipeline instead of the `on merge unsupported ftrs` pipeline: These steps run in the `on merge` pipeline with these changes: - Security Solution Cypress Tests - Explore Cypress Tests - Investigations Cypress Tests - Defend Workflows Cypress Tests - Defend Workflows Serverless Cypress Tests - Threat Intelligence Cypress Tests - Osquery Cypress Tests
and these already run on the `on merge` pipeline - Serverless Security Cypress Tests - Serverless Explore - Security Solution Cypress Tests - Serverless Investigations - Security Solution Cypress Tests
### Additional work to be done: We need to consolidate build steps, enhance test skipping to support Cypress-grep flags, avoid out-of-memory errors in cypress, enhance parallelization, improve Cypress reporting, and probably other things. These are tracked separately. Reach out to me if you need details. ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --- .buildkite/pipelines/on_merge.yml | 84 +++++++++++ .../pipelines/on_merge_unsupported_ftrs.yml | 60 -------- .buildkite/pipelines/pull_request/base.yml | 130 ++++++++++++++++++ .../pull_request/defend_workflows.yml | 26 ---- .../pull_request/osquery_cypress.yml | 40 ------ .../pull_request/security_solution.yml | 54 -------- .../pull_request/threat_intelligence.yml | 12 -- .../pipelines/pull_request/pipeline.ts | 42 ------ packages/kbn-openapi-generator/README.md | 21 +-- x-pack/plugins/threat_intelligence/README.md | 9 +- 10 files changed, 227 insertions(+), 251 deletions(-) delete mode 100644 .buildkite/pipelines/pull_request/defend_workflows.yml delete mode 100644 .buildkite/pipelines/pull_request/osquery_cypress.yml delete mode 100644 .buildkite/pipelines/pull_request/security_solution.yml delete mode 100644 .buildkite/pipelines/pull_request/threat_intelligence.yml diff --git a/.buildkite/pipelines/on_merge.yml b/.buildkite/pipelines/on_merge.yml index d567ac16af588..5427a3081730d 100644 --- a/.buildkite/pipelines/on_merge.yml +++ b/.buildkite/pipelines/on_merge.yml @@ -115,6 +115,90 @@ steps: - exit_status: '*' limit: 1 + - command: .buildkite/scripts/steps/functional/security_solution.sh + label: 'Security Solution Cypress Tests' + agents: + queue: n2-4-spot + depends_on: build + timeout_in_minutes: 60 + parallelism: 16 + retry: + automatic: + - exit_status: '*' + limit: 1 + + - command: .buildkite/scripts/steps/functional/security_solution_explore.sh + label: 'Explore - Security Solution Cypress Tests' + agents: + queue: n2-4-spot + depends_on: build + timeout_in_minutes: 60 + parallelism: 4 + retry: + automatic: + - exit_status: '*' + limit: 1 + + - command: .buildkite/scripts/steps/functional/security_solution_investigations.sh + label: 'Investigations - Security Solution Cypress Tests' + agents: + queue: n2-4-spot + depends_on: build + timeout_in_minutes: 60 + parallelism: 8 + retry: + automatic: + - exit_status: '*' + limit: 1 + + - command: .buildkite/scripts/steps/functional/defend_workflows.sh + label: 'Defend Workflows Cypress Tests' + agents: + queue: n2-4-virt + depends_on: build + timeout_in_minutes: 60 + parallelism: 6 + retry: + automatic: + - exit_status: '*' + limit: 1 + + - command: .buildkite/scripts/steps/functional/defend_workflows_serverless.sh + label: 'Defend Workflows Cypress Tests on Serverless' + agents: + queue: n2-4-virt + depends_on: build + timeout_in_minutes: 120 + parallelism: 2 + retry: + automatic: + - exit_status: '*' + limit: 1 + + - command: .buildkite/scripts/steps/functional/threat_intelligence.sh + label: 'Threat Intelligence Cypress Tests' + agents: + queue: n2-4-spot + depends_on: build + timeout_in_minutes: 120 + parallelism: 2 + retry: + automatic: + - exit_status: '*' + limit: 1 + + - command: .buildkite/scripts/steps/functional/osquery_cypress.sh + label: 'Osquery Cypress Tests' + agents: + queue: n2-4-spot + depends_on: build + timeout_in_minutes: 50 + parallelism: 6 + retry: + automatic: + - exit_status: '*' + limit: 1 + - command: '.buildkite/scripts/steps/functional/on_merge_unsupported_ftrs.sh' label: Trigger unsupported ftr tests timeout_in_minutes: 10 diff --git a/.buildkite/pipelines/on_merge_unsupported_ftrs.yml b/.buildkite/pipelines/on_merge_unsupported_ftrs.yml index ceef6e06ef96f..6dee27db71659 100644 --- a/.buildkite/pipelines/on_merge_unsupported_ftrs.yml +++ b/.buildkite/pipelines/on_merge_unsupported_ftrs.yml @@ -49,66 +49,6 @@ steps: - exit_status: '*' limit: 1 - - command: .buildkite/scripts/steps/functional/security_solution.sh - label: 'Security Solution Tests' - agents: - queue: n2-4-spot - depends_on: build - timeout_in_minutes: 120 - parallelism: 14 - retry: - automatic: - - exit_status: '*' - limit: 1 - - - command: .buildkite/scripts/steps/functional/security_solution_investigations.sh - label: 'Investigations - Security Solution Cypress Tests' - agents: - queue: n2-4-spot - depends_on: build - timeout_in_minutes: 120 - parallelism: 6 - retry: - automatic: - - exit_status: '*' - limit: 1 - - - command: .buildkite/scripts/steps/functional/security_solution_explore.sh - label: 'Explore - Security Solution Tests' - agents: - queue: n2-4-spot - depends_on: build - timeout_in_minutes: 60 - parallelism: 2 - retry: - automatic: - - exit_status: '*' - limit: 1 - - - command: .buildkite/scripts/steps/functional/defend_workflows.sh - label: 'Defend Workflows Cypress Tests' - agents: - queue: n2-4-virt - depends_on: build - timeout_in_minutes: 120 - parallelism: 6 - retry: - automatic: - - exit_status: '*' - limit: 1 - - - command: .buildkite/scripts/steps/functional/osquery_cypress.sh - label: 'Osquery Cypress Tests' - agents: - queue: n2-4-spot - depends_on: build - timeout_in_minutes: 50 - parallelism: 6 - retry: - automatic: - - exit_status: '*' - limit: 1 - - command: .buildkite/scripts/steps/functional/synthetics_plugin.sh label: 'Synthetics @elastic/synthetics Tests' agents: diff --git a/.buildkite/pipelines/pull_request/base.yml b/.buildkite/pipelines/pull_request/base.yml index 04861de3dfac9..0a03db978fa21 100644 --- a/.buildkite/pipelines/pull_request/base.yml +++ b/.buildkite/pipelines/pull_request/base.yml @@ -93,6 +93,136 @@ steps: - exit_status: '*' limit: 1 + - command: .buildkite/scripts/steps/functional/security_solution.sh + label: 'Security Solution Cypress Tests' + agents: + queue: n2-4-spot + depends_on: build + timeout_in_minutes: 60 + parallelism: 16 + retry: + automatic: + - exit_status: '*' + limit: 1 + + - command: .buildkite/scripts/steps/functional/security_solution_explore.sh + label: 'Explore - Security Solution Cypress Tests' + agents: + queue: n2-4-spot + depends_on: build + timeout_in_minutes: 60 + parallelism: 4 + retry: + automatic: + - exit_status: '*' + limit: 1 + + - command: .buildkite/scripts/steps/functional/security_solution_investigations.sh + label: 'Investigations - Security Solution Cypress Tests' + agents: + queue: n2-4-spot + depends_on: build + timeout_in_minutes: 60 + parallelism: 8 + retry: + automatic: + - exit_status: '*' + limit: 1 + + - command: .buildkite/scripts/steps/functional/defend_workflows.sh + label: 'Defend Workflows Cypress Tests' + agents: + queue: n2-4-virt + depends_on: build + timeout_in_minutes: 60 + parallelism: 6 + retry: + automatic: + - exit_status: '*' + limit: 1 + + - command: .buildkite/scripts/steps/functional/defend_workflows_serverless.sh + label: 'Defend Workflows Cypress Tests on Serverless' + agents: + queue: n2-4-virt + depends_on: build + timeout_in_minutes: 120 + parallelism: 2 + retry: + automatic: + - exit_status: '*' + limit: 1 + + - command: .buildkite/scripts/steps/functional/threat_intelligence.sh + label: 'Threat Intelligence Cypress Tests' + agents: + queue: n2-4-spot + depends_on: build + timeout_in_minutes: 120 + parallelism: 2 + retry: + automatic: + - exit_status: '*' + limit: 1 + + - command: .buildkite/scripts/steps/functional/osquery_cypress.sh + label: 'Osquery Cypress Tests' + agents: + queue: n2-4-spot + depends_on: build + timeout_in_minutes: 50 + parallelism: 6 + retry: + automatic: + - exit_status: '*' + limit: 1 + artifact_paths: + - 'target/kibana-osquery/**/*' + + - command: .buildkite/scripts/steps/functional/security_solution_burn.sh + label: 'Security Solution Cypress tests, burning changed specs' + agents: + queue: n2-4-spot + depends_on: build + timeout_in_minutes: 60 + parallelism: 1 + retry: + automatic: false + soft_fail: true + + - command: .buildkite/scripts/steps/code_generation/security_solution_codegen.sh + label: 'Security Solution OpenAPI codegen' + agents: + queue: n2-2-spot + timeout_in_minutes: 60 + parallelism: 1 + + - command: .buildkite/scripts/steps/functional/osquery_cypress_burn.sh + label: 'Osquery Cypress Tests, burning changed specs' + agents: + queue: n2-4-spot + depends_on: build + timeout_in_minutes: 50 + soft_fail: true + retry: + automatic: false + artifact_paths: + - 'target/kibana-osquery/**/*' + + - command: .buildkite/scripts/steps/functional/security_serverless_osquery.sh + label: 'Serverless Osquery Cypress Tests' + agents: + queue: n2-4-spot + depends_on: build + timeout_in_minutes: 50 + parallelism: 6 + retry: + automatic: + - exit_status: '*' + limit: 1 + artifact_paths: + - 'target/kibana-osquery/**/*' + # status_exception: Native role management is not enabled in this Elasticsearch instance # - command: .buildkite/scripts/steps/functional/security_serverless_defend_workflows.sh # label: 'Serverless Security Defend Workflows Cypress Tests' diff --git a/.buildkite/pipelines/pull_request/defend_workflows.yml b/.buildkite/pipelines/pull_request/defend_workflows.yml deleted file mode 100644 index da79114cbebfb..0000000000000 --- a/.buildkite/pipelines/pull_request/defend_workflows.yml +++ /dev/null @@ -1,26 +0,0 @@ -steps: - - command: .buildkite/scripts/steps/functional/defend_workflows.sh - label: 'Defend Workflows Cypress Tests' - agents: - queue: n2-4-virt - depends_on: build - timeout_in_minutes: 60 - parallelism: 6 - retry: - automatic: - - exit_status: '*' - limit: 1 - - - command: .buildkite/scripts/steps/functional/defend_workflows_serverless.sh - label: 'Defend Workflows Cypress Tests on Serverless' - agents: - queue: n2-4-virt - depends_on: build - timeout_in_minutes: 120 - parallelism: 2 - retry: - automatic: - - exit_status: '*' - limit: 1 - - diff --git a/.buildkite/pipelines/pull_request/osquery_cypress.yml b/.buildkite/pipelines/pull_request/osquery_cypress.yml deleted file mode 100644 index 07e26e8f1ff6b..0000000000000 --- a/.buildkite/pipelines/pull_request/osquery_cypress.yml +++ /dev/null @@ -1,40 +0,0 @@ -steps: - - command: .buildkite/scripts/steps/functional/osquery_cypress.sh - label: 'Osquery Cypress Tests' - agents: - queue: n2-4-spot - depends_on: build - timeout_in_minutes: 50 - parallelism: 6 - retry: - automatic: - - exit_status: '*' - limit: 1 - artifact_paths: - - "target/kibana-osquery/**/*" - - - command: .buildkite/scripts/steps/functional/osquery_cypress_burn.sh - label: 'Osquery Cypress Tests, burning changed specs' - agents: - queue: n2-4-spot - depends_on: build - timeout_in_minutes: 50 - soft_fail: true - retry: - automatic: false - artifact_paths: - - "target/kibana-osquery/**/*" - - - command: .buildkite/scripts/steps/functional/security_serverless_osquery.sh - label: 'Serverless Osquery Cypress Tests' - agents: - queue: n2-4-spot - depends_on: build - timeout_in_minutes: 50 - parallelism: 6 - retry: - automatic: - - exit_status: '*' - limit: 1 - artifact_paths: - - "target/kibana-osquery/**/*" diff --git a/.buildkite/pipelines/pull_request/security_solution.yml b/.buildkite/pipelines/pull_request/security_solution.yml deleted file mode 100644 index c679d9397f7ad..0000000000000 --- a/.buildkite/pipelines/pull_request/security_solution.yml +++ /dev/null @@ -1,54 +0,0 @@ -steps: - - command: .buildkite/scripts/steps/functional/security_solution.sh - label: 'Security Solution Cypress Tests' - agents: - queue: n2-4-spot - depends_on: build - timeout_in_minutes: 60 - parallelism: 16 - retry: - automatic: - - exit_status: '*' - limit: 1 - - - command: .buildkite/scripts/steps/functional/security_solution_explore.sh - label: 'Explore - Security Solution Cypress Tests' - agents: - queue: n2-4-spot - depends_on: build - timeout_in_minutes: 60 - parallelism: 4 - retry: - automatic: - - exit_status: '*' - limit: 1 - - - command: .buildkite/scripts/steps/functional/security_solution_investigations.sh - label: 'Investigations - Security Solution Cypress Tests' - agents: - queue: n2-4-spot - depends_on: build - timeout_in_minutes: 60 - parallelism: 8 - retry: - automatic: - - exit_status: '*' - limit: 1 - - - command: .buildkite/scripts/steps/functional/security_solution_burn.sh - label: 'Security Solution Cypress tests, burning changed specs' - agents: - queue: n2-4-spot - depends_on: build - timeout_in_minutes: 60 - parallelism: 1 - retry: - automatic: false - soft_fail: true - - - command: .buildkite/scripts/steps/code_generation/security_solution_codegen.sh - label: 'Security Solution OpenAPI codegen' - agents: - queue: n2-2-spot - timeout_in_minutes: 60 - parallelism: 1 diff --git a/.buildkite/pipelines/pull_request/threat_intelligence.yml b/.buildkite/pipelines/pull_request/threat_intelligence.yml deleted file mode 100644 index b91be5faffdec..0000000000000 --- a/.buildkite/pipelines/pull_request/threat_intelligence.yml +++ /dev/null @@ -1,12 +0,0 @@ -steps: - - command: .buildkite/scripts/steps/functional/threat_intelligence.sh - label: 'Threat Intelligence Cypress Tests' - agents: - queue: n2-4-spot - depends_on: build - timeout_in_minutes: 120 - parallelism: 2 - retry: - automatic: - - exit_status: '*' - limit: 1 diff --git a/.buildkite/scripts/pipelines/pull_request/pipeline.ts b/.buildkite/scripts/pipelines/pull_request/pipeline.ts index 80d1312af6e64..7a7fa0f59b9c7 100644 --- a/.buildkite/scripts/pipelines/pull_request/pipeline.ts +++ b/.buildkite/scripts/pipelines/pull_request/pipeline.ts @@ -59,38 +59,6 @@ const uploadPipeline = (pipelineContent: string | object) => { pipeline.push(getPipeline('.buildkite/pipelines/pull_request/kbn_handlebars.yml')); } - if ( - (await doAnyChangesMatch([ - /^src\/plugins\/controls/, - /^packages\/kbn-securitysolution-.*/, - /^x-pack\/plugins\/lists/, - /^x-pack\/plugins\/security_solution/, - /^x-pack\/plugins\/timelines/, - /^x-pack\/plugins\/triggers_actions_ui\/public\/application\/sections\/action_connector_form/, - /^x-pack\/plugins\/triggers_actions_ui\/public\/application\/sections\/alerts_table/, - /^x-pack\/plugins\/triggers_actions_ui\/public\/application\/context\/actions_connectors_context\.tsx/, - /^x-pack\/test\/defend_workflows_cypress/, - /^x-pack\/test\/security_solution_cypress/, - /^fleet_packages\.json/, // It contains reference to prebuilt detection rules, we want to run security solution tests if it changes - ])) || - GITHUB_PR_LABELS.includes('ci:all-cypress-suites') - ) { - pipeline.push(getPipeline('.buildkite/pipelines/pull_request/security_solution.yml')); - pipeline.push(getPipeline('.buildkite/pipelines/pull_request/defend_workflows.yml')); - pipeline.push(getPipeline('.buildkite/pipelines/pull_request/osquery_cypress.yml')); - } - - if ( - (await doAnyChangesMatch([ - /^x-pack\/plugins\/threat_intelligence/, - /^x-pack\/test\/threat_intelligence_cypress/, - /^x-pack\/plugins\/security_solution\/public\/threat_intelligence/, - ])) || - GITHUB_PR_LABELS.includes('ci:all-cypress-suites') - ) { - pipeline.push(getPipeline('.buildkite/pipelines/pull_request/threat_intelligence.yml')); - } - if ( (await doAnyChangesMatch([ /^src\/plugins\/data/, @@ -140,16 +108,6 @@ const uploadPipeline = (pipelineContent: string | object) => { GITHUB_PR_LABELS.includes('ci:all-cypress-suites') ) { pipeline.push(getPipeline('.buildkite/pipelines/pull_request/fleet_cypress.yml')); - pipeline.push(getPipeline('.buildkite/pipelines/pull_request/defend_workflows.yml')); - pipeline.push(getPipeline('.buildkite/pipelines/pull_request/osquery_cypress.yml')); - } - - if ( - ((await doAnyChangesMatch([/^x-pack\/plugins\/osquery/, /^x-pack\/test\/osquery_cypress/])) || - GITHUB_PR_LABELS.includes('ci:all-cypress-suites')) && - !GITHUB_PR_LABELS.includes('ci:skip-cypress-osquery') - ) { - pipeline.push(getPipeline('.buildkite/pipelines/pull_request/osquery_cypress.yml')); } if (await doAnyChangesMatch([/^x-pack\/plugins\/exploratory_view/])) { diff --git a/packages/kbn-openapi-generator/README.md b/packages/kbn-openapi-generator/README.md index fc75a76827934..4a3ed910bca29 100644 --- a/packages/kbn-openapi-generator/README.md +++ b/packages/kbn-openapi-generator/README.md @@ -94,6 +94,7 @@ export const InstallPrebuiltRulesResponse = z.object({ timelines_updated: z.number().int().min(0), }); ``` + ## Programmatic API Alternatively, you can use the code generator programmatically. You can create a script file and run it with `node` command. This could be useful if you want to set up code generation in your CI pipeline. Here's an example of such script: @@ -135,26 +136,26 @@ check_for_changed_files "yarn openapi:generate" true This scripts sets up the minimal environment required fro code generation and runs the code generation script. Then it checks if there are any changes and commits them if there are any using the `check_for_changed_files` function. -Then add the code generation script to your plugin build pipeline. Open your plugin build pipeline, for example `.buildkite/pipelines/pull_request/security_solution.yml`, and add the following command to the steps list adjusting the path to your code generation script: +Then add the code generation script to your plugin build pipeline. Open your plugin build pipeline, for example `.buildkite/pipelines/pull_request/base.yml`, and add the following command to the steps list adjusting the path to your code generation script: ```yaml - - command: .buildkite/scripts/steps/code_generation/security_solution_codegen.sh - label: 'Security Solution OpenAPI codegen' - agents: - queue: n2-2-spot - timeout_in_minutes: 60 - parallelism: 1 +- command: .buildkite/scripts/steps/code_generation/security_solution_codegen.sh + label: 'Security Solution OpenAPI codegen' + agents: + queue: n2-2-spot + timeout_in_minutes: 60 + parallelism: 1 ``` Now on every pull request the code generation script will run and commit the changes if there are any. ## OpenAPI Schema -The code generator supports the OpenAPI definitions described in the request, response, and component sections of the document. +The code generator supports the OpenAPI definitions described in the request, response, and component sections of the document. For every API operation (GET, POST, etc) it is required to specify the `operationId` field. This field is used to generate the name of the generated types. For example, if the `operationId` is `InstallPrebuiltRules` then the generated types will be named `InstallPrebuiltRulesResponse` and `InstallPrebuiltRulesRequest`. If the `operationId` is not specified then the code generation will throw an error. -The `x-codegen-enabled` field is used to enable or disable code generation for the operation. If it is not specified then code generation is disabled by default. This field could be also used to disable code generation of common components described in the `components` section of the OpenAPI specification. +The `x-codegen-enabled` field is used to enable or disable code generation for the operation. If it is not specified then code generation is disabled by default. This field could be also used to disable code generation of common components described in the `components` section of the OpenAPI specification. Keep in mind that disabling code generation for common components that are referenced by external OpenAPI specifications could lead to errors during code generation. @@ -198,4 +199,4 @@ paths: application/json: schema: $ref: '../../../model/rule_schema/rule_schemas.schema.yaml#/components/schemas/RuleResponse' -``` +``` \ No newline at end of file diff --git a/x-pack/plugins/threat_intelligence/README.md b/x-pack/plugins/threat_intelligence/README.md index da2cf37fe749a..b708a4fecaa3d 100755 --- a/x-pack/plugins/threat_intelligence/README.md +++ b/x-pack/plugins/threat_intelligence/README.md @@ -84,14 +84,9 @@ You can use this approach to load separate data dumps for every test case, to co ### CI Execution -The entry point for PR testing is `.buildkite/pipelines/pull_request/threat_intelligence.yml` file, see that for details on +The entry point for PR testing is `.buildkite/pipelines/pull_request/base.yml` file, see that for details on how the test suite is executed & extra options regarding parallelism, retrying etc. -E2E tests for this plugin will only be executed if any of the files changed within the PR matches dependency list here: -`.buildkite/scripts/pipelines/pull_request/pipeline.ts` - -It is also possible to run all tests by attaching a PR flag: `ci:all-cypress-suites`. - ## FAQ ### How is the Threat Intelligence code loaded in Kibana? @@ -110,4 +105,4 @@ See [CONTRIBUTING.md](https://github.com/elastic/kibana/blob/main/x-pack/plugins ## Issues -Please report any issues in [this GitHub project](https://github.com/orgs/elastic/projects/758/). +Please report any issues in [this GitHub project](https://github.com/orgs/elastic/projects/758/). \ No newline at end of file