[scout] add ci pipeline #202707

dmlemeshko · 2024-12-03T12:43:34Z

Summary

Adding basic pipeline to run scout UI tests for stateful and 3 serverless projects. Few tests were skipped for Security project because esArchives used in tests ingest index that are restricted for default project roles.

It is an opt-in pipeline to verify kbn/scout changes does not break the existing tests. It is only for start and we plan to re-work into a mature pipeline later.

New opt-in pipeline was added

dmlemeshko · 2024-12-03T13:11:30Z

packages/kbn-scout/src/playwright/fixtures/test/validate_tags.ts

+export const validateTagsFixture = base.extend<{ validateTags: void }>({
+  validateTags: [
+    async ({}, use, testInfo) => {
+      if (testInfo.tags.length === 0) {
+        throw new Error(`At least one tag is required: ${supportedTags.join(', ')}`);
+      }
+
+      const invalidTags = testInfo.tags.filter((tag: string) => !supportedTags.includes(tag));
+      if (invalidTags.length > 0) {
+        throw new Error(
+          `Unsupported tag(s) found in test suite "${testInfo.title}": ${invalidTags.join(
+            ', '
+          )}. ` + `Supported tags are: ${supportedTags.join(', ')}.`
+        );
+      }
+
+      await use();
+    },
+    { auto: true },
+  ],
+});


The goal: notify test author about requirement of tags definition and validating its correctness. I also introduces aliases through tags variable to minimize spelling error.

Note: it only work when you run tests in IDE; when you run tests via npx playwright test --grep <tag> playwright is doing filtering in advance and no tags or incorrect tags will cause test to be filtered out.

That said, for CI we will need to come up with some script to validate tags being defined. But I still believe there is a benefit of raising error during test development process

Example: adding non-supported tag @xyz like

test.describe( 'Discover app - saved searches', { tag: ['@ess', '@svlSearch', '@svlOblt', '@xyz'] }, () => {

will lead to runtime error

dmlemeshko · 2024-12-03T13:12:16Z

x-pack/plugins/discover_enhanced/ui_tests/tests/error_handling.spec.ts

 import { test, testData } from '../fixtures';

-test.describe('Discover app - errors', { tag: ['@ess'] }, () => {
+test.describe('Discover app - errors', { tag: tags.ESS_ONLY }, () => {


alias to simplify the process and avoid spelling/incorrect tagging issues

…into scout/add-ci-pipeline

dmlemeshko · 2024-12-03T16:28:02Z

x-pack/plugins/discover_enhanced/ui_tests/tests/saved_searches.spec.ts

+  };
+
+  test.beforeAll(async ({ esArchiver, kbnClient, uiSettings }) => {
+    await esArchiver.loadIfNeeded(testData.ES_ARCHIVES.ECOMMERCE);


This test will work for Security project only with admin role, that has access to all indexes. It probably means generating new saved objects, leaving for later

dmlemeshko · 2024-12-03T16:29:29Z

x-pack/plugins/discover_enhanced/ui_tests/tests/value_suggestions_non_time_based.spec.ts

-  { tag: ['@ess', '@svlSecurity', '@svlOblt', '@svlSearch'] },
+  { tag: ['@ess', '@svlSearch', '@svlOblt'] },
+  // TODO: Update to use an ES archive with an index accessible to 'viewer'
+  // for running this test against the Security serverless project.
  () => {
    test.beforeAll(async ({ esArchiver, kbnClient, uiSettings }) => {
      await esArchiver.loadIfNeeded(testData.ES_ARCHIVES.NO_TIME_FIELD);


Similar issue: viewer has access to limited indexes ( '.siem-signals*', '.lists-', '.items-') and test requires re-work.

dmlemeshko · 2024-12-03T16:35:07Z

packages/kbn-scout/src/playwright/runner/run_tests.ts

+        args: [
+          'test',
+          `--config=${playwrightConfigPath}`,
+          `--grep=${playwrightGrepTag}`,


Passing one of @ess , svlSearch, ... tags to playwright runner

dmlemeshko · 2024-12-09T16:21:23Z

packages/kbn-scout/src/config/serverless/serverless.base.config.ts

+      '--xpack.cloud.profile_url=/user/settings/',
+      '--xpack.cloud.projects_url=/projects/',
+      '--xpack.cloud.serverless.project_id=fakeprojectid',
+      '--xpack.cloud.users_and_roles_url=/account/members/',


I copied missing arguments from FTR svl config

dmlemeshko · 2024-12-09T16:35:59Z

.buildkite/scripts/pipelines/pull_request/pipeline.ts

+        /^x-pack\/plugins\/discover_enhanced\/ui_tests/,
+        /^packages\/kbn-scout/,
+      ])) ||
+      GITHUB_PR_LABELS.includes('ci:scout-ui-tests')


ci:scout-ui-tests, later we can add ci:scout-api-tests, wdyt?

dolaru

Perused over the changes and nothing looks out of place to me, but someone more experienced in this code base should approve.

…into scout/add-ci-pipeline

pheyos

LGTM
I have a few things in mind that we should discuss and potentially follow up on. But in the interest of moving forward, we should merge this PR and handle the items in separate PRs.

elasticmachine · 2024-12-11T11:15:52Z

💚 Build Succeeded

Buildkite Build
Commit: ca92b74

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/scout`	86	90	+4

Unknown metric groups

API count

id	before	after	diff
`@kbn/scout`	117	121	+4

History

💚 Build #258589 succeeded 868095c
💔 Build #258372 failed ffae7d3
💔 Build #256563 failed 2774f7b
💚 Build #256431 succeeded 3bf06d8

cc @dmlemeshko

## Summary Adding basic pipeline to run scout UI tests for stateful and 3 serverless projects. Few tests were skipped for Security project because esArchives used in tests ingest index that are restricted for default project roles. It is an opt-in pipeline to verify kbn/scout changes does not break the existing tests. It is only for start and we plan to re-work into a mature pipeline later. New opt-in [pipeline](https://buildkite.com/elastic/kibana-pull-request/builds/258589#0193afc4-bbd6-4200-8c5f-a7e4a8073e1d) was added <img width="1659" alt="Screenshot 2024-12-10 at 11 31 35" src="https://github.com/user-attachments/assets/1c19fa46-4e66-4796-ac6d-c2c96c74fa8e">

## Summary Adding basic pipeline to run scout UI tests for stateful and 3 serverless projects. Few tests were skipped for Security project because esArchives used in tests ingest index that are restricted for default project roles. It is an opt-in pipeline to verify kbn/scout changes does not break the existing tests. It is only for start and we plan to re-work into a mature pipeline later. New opt-in [pipeline](https://buildkite.com/elastic/kibana-pull-request/builds/258589#0193afc4-bbd6-4200-8c5f-a7e4a8073e1d) was added <img width="1659" alt="Screenshot 2024-12-10 at 11 31 35" src="https://github.com/user-attachments/assets/1c19fa46-4e66-4796-ac6d-c2c96c74fa8e"> (cherry picked from commit 87b2a12) # Conflicts: # x-pack/plugins/discover_enhanced/ui_tests/tests/saved_searches.spec.ts

dmlemeshko · 2024-12-18T16:17:04Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

# Backport This will backport the following commits from `main` to `8.x`: - [[scout] add ci pipeline (#202707)](#202707)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)

dmlemeshko added 3 commits December 3, 2024 13:01

[kbn/scout] update script to use tagging

b06b1e3

[ci] add pipeline to run stateful tests

2196b50

Merge remote-tracking branch 'upstream/main' into scout/add-ci-pipeline

3bf06d8

dmlemeshko changed the title ~~Scout/add ci pipeline~~ [scout] add ci pipeline Dec 3, 2024

dmlemeshko commented Dec 3, 2024

View reviewed changes

refactor and add serverless tests

15db0aa

dmlemeshko self-assigned this Dec 3, 2024

dmlemeshko added 4 commits December 3, 2024 17:23

exclude security project where viewer has no permission to index

11648e6

Merge branch 'main' into scout/add-ci-pipeline

21828c6

bump timeout to 30 min

d82f4cd

Merge branch 'scout/add-ci-pipeline' of github.com:dmlemeshko/kibana …

ffa7a7b

…into scout/add-ci-pipeline

dmlemeshko commented Dec 3, 2024

View reviewed changes

update script

2774f7b

dmlemeshko added release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting v9.0.0 test:scout labels Dec 3, 2024

dmlemeshko commented Dec 3, 2024

View reviewed changes

dmlemeshko added 10 commits December 4, 2024 10:05

Merge remote-tracking branch 'upstream/main' into scout/add-ci-pipeline

9fe0a76

Merge remote-tracking branch 'upstream/main' into scout/add-ci-pipeline

b1feb44

update config

3a966c4

Merge remote-tracking branch 'upstream/main' into scout/add-ci-pipeline

7277a38

[kbn-scout] fix svl configs to run Kibana build

ea67b2d

update config type

a250a59

Merge remote-tracking branch 'upstream/main' into scout/add-ci-pipeline

040f224

update ci script

7fb9639

update pipeline script

2b49bff

Merge branch 'main' into scout/add-ci-pipeline

ffae7d3

dmlemeshko commented Dec 9, 2024

View reviewed changes

update pipeline

a514f73

dmlemeshko marked this pull request as ready for review December 9, 2024 16:59

dmlemeshko requested review from a team as code owners December 9, 2024 16:59

dmlemeshko requested review from dolaru, pheyos and mistic December 9, 2024 16:59

dolaru reviewed Dec 9, 2024

View reviewed changes

dmlemeshko added 3 commits December 10, 2024 09:49

improve script

302ad67

Merge branch 'scout/add-ci-pipeline' of github.com:dmlemeshko/kibana …

1a06832

…into scout/add-ci-pipeline

Merge branch 'main' into scout/add-ci-pipeline

868095c

mistic approved these changes Dec 10, 2024

View reviewed changes

Merge branch 'main' into scout/add-ci-pipeline

ca92b74

pheyos approved these changes Dec 11, 2024

View reviewed changes

dmlemeshko merged commit 87b2a12 into elastic:main Dec 11, 2024
9 checks passed

dmlemeshko mentioned this pull request Dec 18, 2024

[8.x] [scout] add ci pipeline (#202707) #204796

Merged

dmlemeshko added the v8.18.0 label Dec 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[scout] add ci pipeline #202707

[scout] add ci pipeline #202707

dmlemeshko commented Dec 3, 2024 •

edited

Loading

dmlemeshko Dec 3, 2024 •

edited

Loading

dmlemeshko Dec 9, 2024 •

edited

Loading

dmlemeshko Dec 3, 2024

dmlemeshko Dec 3, 2024

dmlemeshko Dec 3, 2024

dmlemeshko Dec 3, 2024

dmlemeshko Dec 9, 2024

dmlemeshko Dec 9, 2024 •

edited

Loading

dolaru left a comment

pheyos left a comment

elasticmachine commented Dec 11, 2024

API count

dmlemeshko commented Dec 18, 2024

[scout] add ci pipeline #202707

[scout] add ci pipeline #202707

Conversation

dmlemeshko commented Dec 3, 2024 • edited Loading

Summary

dmlemeshko Dec 3, 2024 • edited Loading

Choose a reason for hiding this comment

dmlemeshko Dec 9, 2024 • edited Loading

Choose a reason for hiding this comment

dmlemeshko Dec 3, 2024

Choose a reason for hiding this comment

dmlemeshko Dec 3, 2024

Choose a reason for hiding this comment

dmlemeshko Dec 3, 2024

Choose a reason for hiding this comment

dmlemeshko Dec 3, 2024

Choose a reason for hiding this comment

dmlemeshko Dec 9, 2024

Choose a reason for hiding this comment

dmlemeshko Dec 9, 2024 • edited Loading

Choose a reason for hiding this comment

dolaru left a comment

Choose a reason for hiding this comment

pheyos left a comment

Choose a reason for hiding this comment

elasticmachine commented Dec 11, 2024

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

API count

History

dmlemeshko commented Dec 18, 2024

💚 All backports created successfully

Questions ?

dmlemeshko commented Dec 3, 2024 •

edited

Loading

dmlemeshko Dec 3, 2024 •

edited

Loading

dmlemeshko Dec 9, 2024 •

edited

Loading

dmlemeshko Dec 9, 2024 •

edited

Loading