[SLO] [Alerting] deployment agnostic slo burn rate rule tests #187924

mgiota · 2024-07-09T23:12:49Z

Addresses #179549
Relates to #183113

Update

Since the Appex QA team has taken on deployment agnostic tests, a lot of the original implementation of this PR has changed. Now that the Appex QA team has provided a current directly to write deployment agnostic tests, the burn rate rule tests have been moved here.

To finish onboarding the burn rate rule test to this new framework, the following was done.

Add an oblt.stateful.config.ts file to complement the existing oblt.serverless.config.ts file to ensure the tests are run in CI
Ensure our test config is added to the buildkite pipepline
Add the alerting service to the new deployment_agnostic/services directory.
Port the tests over to the new deployment_agnostic directory

To run serverless

node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts
node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts --grep="Burn rate rule"

To run stateful

node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts
node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts --grep="Burn rate rule"

For context, I've kept the history from the original PR description below.

🍒 History

A new type of config file will be allowed for API integration and functional tests within the x-pack/test folder, using a pattern of *.serverless.config.ts — these config files will specify configuration needed to run a set of tests in a serverless deployment context.

FTR tests already make use of the it.tags(['my-tag-1', 'my-tag-2']) pattern, and so we would like to stay with that pattern rather than introducing a new type of mocha tag in the it block's "description" as it was introduced in this POC. The difference with the previous PR in terms of tagging is that we use suiteTags instead of mochaOps

Adding following in the config files:

serverless config

suiteTags: {
        include: ['serverless'],
      },

ess config

suiteTags: {
        include: ['ess'],
      },

and then adding this.tags(['serverless', 'ess']) in the test suite instructs the test runner to run the same test suite in both environments.

In order to keep things simple, we stay with the current skip approach, which means that flaky tests will be skipped for all environments by appending .skip() to the suite or to specific test cases.

Description

This PR uses suiteTags for tagging the tests appropriately. We decide through following labels in which environment the tests are going to be executed:
- @ess: Runs in an ESS environment (on-prem installation) as part of the CI validation on PRs.
- @serverless: Runs in a serverless environment.
It introduces a new folder x-pack/test/observability_solution_api_integration which will serve as a centralized location for all tests by obs-ux-management team that must be run in Serverless and ESS environments. A list of all tests can be found in the R&D issue
Within this folder, there is a "config" subdirectory that stores base configurations specific to both the Serverless and ESS environments. These configurations build upon the base configuration provided by test_serverless and api_integration, incorporating additional settings such as environment variables and tagging options.
The file x-pack/test/observability_solution_api_integration/test_suites/alerting/burn_rate/burn_rate_rule.ts is functional in both Serverless and ESS
It removes the existing burn rate rule from x-pack/test_serverless/api_integration/test_suites/observability/alerting/burn_rate/burn_rate_rule.ts
The alertingApi and sloApi services are moved to test/api_integration servers

In the screenshot below you can see the test_suites folder structure, after having migrated the current slo burn rate rule. We recommend having an alerting and slo subfolders. Rest observability apps could be added as another subfolder under test_suites. As part of this PR, the alerting > burn_rate subfolders are created.

How to run locally

You can navigate into the new observability_solution_api_integration folder and use following commands to run the tests in serverless and ess environments accordingly. You can find more information in the README file of the observability_solution_api_integration folder.

cd x-pack/test/observability_solution_api_integration

// SERVERLESS
npm run alerting_burn_rate:server:serverless
npm run alerting_burn_rate:runner:serverless

// ESS
npm run alerting_burn_rate:server:ess
npm run alerting_burn_rate:runner:ess

CI

It includes a new entry in the ftr_configs.yml to execute the newly added tests in the pipeline.
It involves the addition of suiteTags in both serverless/config.base.ts and ess/config.base.ts. In the case of serverless, it includes @serverless while excluding @skipInServerless. Similarly, for ess, it includes @ess and excludes @skipInEss.

Quality Gates and MKI pipeline

The Platform team will support config files within x-pack/test folder with a pattern of *.serverless.config.ts, so these tests will be included in Kibana's Quality gates and will be run against a real MKI environment.

obltmachine · 2024-07-09T23:13:01Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

mgiota · 2024-07-10T08:05:38Z

x-pack/test/api_integration/services/slo_api.ts

+  FetchHistoricalSummaryResponse,
+} from '@kbn/slo-schema';
+import * as t from 'io-ts';
+import { FtrProviderContext } from '../../functional/ftr_provider_context';


@pheyos I was trying to recall why I load the ftr provider context from this location, and I think I got inspired by this change of yours. Does this feel right? Before it was like that import { FtrProviderContext } from '../ftr_provider_context';

FYI: I already moved this service to https://github.com/elastic/kibana/blob/main/x-pack/test/api_integration/deployment_agnostic/services/slo_api.ts

mgiota · 2024-07-10T08:24:20Z

x-pack/test/api_integration/services/slo_api.ts

+} from '@kbn/slo-schema';
+import * as t from 'io-ts';
+import { FtrProviderContext } from '../../functional/ftr_provider_context';
+import { RoleCredentials } from '../../../test_serverless/shared/services';


Role based testing was introduced in this PR. As part of that PR an svlUserManager service was used within the test itselft to create an api role for admin for serverless.

With current PR, where the goal is to have only one test used by both ess and serverless, current approach doesn't work. We would need to check the environment within the test. What I tried instead was to move this role-based logic within the slo_api service itself. The slo api will check the service and will create the api key on serverless accordingly.

This worked out well. As you can see below in case of ess it creates all the predefined roles and users, where as on serverless it only creates an api key.

ess

serverless

Let's explore if this approach feels right, and then I will proceed applying the changes to the alerting_api service as well. My concern with this approach is that slo_api service is added within the deployment agnostic services, but internally it uses the config service to check if it is serverless or stateful. What are your thoughts on this? Any other alternatives?

mgiota · 2024-07-10T08:27:24Z

x-pack/test/observability_solution_api_integration/README.md

+- Test suites are prefixed with specific tags to determine their execution in particular environments or to exclude them from specific environments.
+
+- We are using the following tags:
+    * `@ess`: Runs in an ESS environment (on-prem installation) as part of the CI validation on PRs.


@pheyos Let's agree on the tags we want to use.

no need to apply tags, at least at the moment. We have 2 config files:

x-pack/test/api_integration/deployment_agnostic/oblt.stateful.config.ts

x-pack/test/api_integration/deployment_agnostic/oblt.serverless.config.ts

and index file to load all Oblt tests:

x-pack/test/api_integration/deployment_agnostic/oblt.index.ts

If the test is added to deployment_agnostic/ folder, it is assumed by default to be run both in serverless and stateful without extra filtering

x-pack/test_serverless/api_integration/test_suites/observability/index.ts

x-pack/test/api_integration/services/index.ts

x-pack/test/api_integration/deployment_agnostic/oblt.stateful.config.ts

….config.ts

.eslintrc.js

…mgiota/kibana into slo_burn_rate_deployment_agnostic_tests

dominiqueclarke

LGTM

mgiota · 2024-08-20T13:47:43Z

Thanks @dominiqueclarke and @dmlemeshko for taking care of this! I am testing now locally the PR and doing a code review

mgiota · 2024-08-20T15:32:27Z

@dominiqueclarke @dmlemeshko I noticed something weird. When I ran the tests for stateful using above mentioned commands:

node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/oblt.stateful.config.ts
node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/oblt.stateful.config.ts --grep="Burn rate rule"

Serverless Observability - Deployment-agnostic api integration tests is being printed.

Screen.Recording.2024-08-20.at.18.28.25.mov

dominiqueclarke · 2024-08-20T16:21:20Z

@dominiqueclarke @dmlemeshko I noticed something weird. When I ran the tests for stateful using above mentioned commands:
node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/oblt.stateful.config.ts
node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/oblt.stateful.config.ts --grep="Burn rate rule"
Serverless Observability - Deployment-agnostic api integration tests is being printed.

Screen.Recording.2024-08-20.at.18.28.25.mov

That is weird, I can see the override for the junit name here https://github.com/elastic/kibana/pull/187924/files#diff-3e77dd5588b23a46609fd3451860d5c3cc53b682f5d33b60430cf0d1aab76bd2R13

mgiota · 2024-08-20T20:54:00Z

@dmlemeshko Here's the commit, where I re-organized things in order to have seperate index files for loading tests in serverless and stateful. Similar to what you try to achieve in this PR.

## Summary While reviewing #187924 I realised that Solution Teams will have their own stateful FTR configs and the overall number of configs in `api_integration/deployment_agnostic` folder might grow up soon and it is better to re-organise it in advance. With new structure we have `configs` folder with 2 sub-folders: `stateful` and `serverless` <img width="494" alt="image" src="https://github.com/user-attachments/assets/c844ce2c-51b6-498c-b51f-0cd9203d84ea"> When _Platform_ teams add deployment-agnostic tests, it is expected to be loaded in `configs/stateful/platform.index.ts` and _at least one_ of `<project>.serverless.config` under `configs/serverless` When _Solution_ team adds deployment-agnostic tests (e.g. one of Oblt teams), it is expected to be loaded _both_ in `configs/stateful/oblt.index.ts` and `configs/serverless/oblt.index.ts`

pheyos

Looks good overall!
Also gave it a test run against an MKI project and it passed ✔️
Left two small nits.

x-pack/test/api_integration/deployment_agnostic/services/alerting_api.ts

…ing_api.ts Co-authored-by: Robert Oskamp <[email protected]>

mgiota · 2024-08-21T12:59:05Z

@elasticmachine merge upstream

kibana-ci · 2024-08-21T14:38:01Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 85273a2

Failed CI Steps

FTR Configs #90

Test Failures

[job] [logs] FTR Configs #90 / Rules Management - Prebuilt Rules - Prebuilt Rules Management @ess @serverless @skipInServerlessMKI Prebuilt Rules status get_prebuilt_rules_status should return empty structure when no prebuilt rule assets

Metrics [docs]

✅ unchanged

History

💛 Build #228781 was flaky 91a4511
💚 Build #228710 succeeded 0a2b008
💛 Build #228603 was flaky ea47dbf
💛 Build #228555 was flaky dd8ee61
💚 Build #227865 succeeded 52ecd8d
💔 Build #227824 failed 5f9ae2c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @mgiota

mgiota mentioned this pull request Jul 9, 2024

[Observability solution] [SLO] Run burn rate api tests in serverless & ess using mocha tagging #183113

Closed

4 tasks

mgiota self-assigned this Jul 10, 2024

deployment agnostic slo burn rate rule tests

f495936

mgiota force-pushed the slo_burn_rate_deployment_agnostic_tests branch from dcb77d1 to f495936 Compare July 10, 2024 07:47

mgiota commented Jul 10, 2024

View reviewed changes

x-pack/test_serverless/api_integration/test_suites/observability/index.ts Show resolved Hide resolved

neptunian mentioned this pull request Jul 24, 2024

[Observability AI Assistant] Add tests for serverless projects #180925

Open

dominiqueclarke self-requested a review August 12, 2024 15:49

dominiqueclarke added 6 commits August 12, 2024 12:51

merge main

95feb99

add burn rate tests to agnostic api test directory

647e38b

add builtkite config

4876a31

remove unused directory

2ee040b

update codeowners

f28913a

adjust alerting api

fb4d553

dominiqueclarke reviewed Aug 13, 2024

View reviewed changes

x-pack/test/api_integration/services/index.ts Outdated Show resolved Hide resolved

Update x-pack/test/api_integration/services/index.ts

f1d7143

dominiqueclarke reviewed Aug 13, 2024

View reviewed changes

x-pack/test/api_integration/deployment_agnostic/oblt.stateful.config.ts Outdated Show resolved Hide resolved

sUpdate x-pack/test/api_integration/deployment_agnostic/oblt.stateful…

7d70b1a

….config.ts

dominiqueclarke reviewed Aug 13, 2024

View reviewed changes

.eslintrc.js Outdated Show resolved Hide resolved

Update .eslintrc.js

390ebb6

dominiqueclarke changed the title ~~deployment agnostic slo burn rate rule tests~~ [SLO] [Alerting] deployment agnostic slo burn rate rule tests Aug 13, 2024

dominiqueclarke added chore release_note:skip Skip the PR/issue when compiling release notes v8.15.0 v8.16.0 and removed v8.15.0 labels Aug 13, 2024

dominiqueclarke marked this pull request as ready for review August 13, 2024 12:38

Merge branch 'slo_burn_rate_deployment_agnostic_tests' of github.com:…

52ecd8d

…mgiota/kibana into slo_burn_rate_deployment_agnostic_tests

dmlemeshko mentioned this pull request Aug 19, 2024

[ftr] re-group configs for deployment agnostic tests #190672

Merged

dominiqueclarke approved these changes Aug 20, 2024

View reviewed changes

Merge branch 'main' into slo_burn_rate_deployment_agnostic_tests

dd8ee61

remove unused path

ea47dbf

configs for deployment agnostic tests

0a2b008

mgiota requested a review from dmlemeshko August 21, 2024 08:28

Merge branch 'main' into slo_burn_rate_deployment_agnostic_tests

91a4511

dmlemeshko approved these changes Aug 21, 2024

View reviewed changes

pheyos reviewed Aug 21, 2024

View reviewed changes

x-pack/test/api_integration/deployment_agnostic/services/alerting_api.ts Outdated Show resolved Hide resolved

x-pack/test/api_integration/deployment_agnostic/services/alerting_api.ts Outdated Show resolved Hide resolved

Update x-pack/test/api_integration/deployment_agnostic/services/alert…

364eea3

…ing_api.ts Co-authored-by: Robert Oskamp <[email protected]>

elasticmachine and others added 2 commits August 21, 2024 22:59

Merge branch 'main' into slo_burn_rate_deployment_agnostic_tests

c3f0329

better error message

85273a2

mgiota enabled auto-merge (squash) August 21, 2024 14:31

mgiota merged commit e3d6cf6 into elastic:main Aug 21, 2024
27 checks passed

kibanamachine added the backport:skip This commit does not require backporting label Aug 21, 2024

yngrdyn mentioned this pull request Sep 3, 2024

[Dataset quality] Meta - Use deployment-agnostic testing in api tests #191980

Open

This was referenced Oct 2, 2024

Migrate Custom threshold rule tests to deployment agnostic solution #183378

Closed

Migrate Metric threshold rule tests to deployment agnostic solution #183401

Closed

Migrate SLO API tests to deployment agnostic solution #183397

Closed

mgiota mentioned this pull request Oct 10, 2024

Migrate ES Query tests to deployment agnostic solution #183395

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SLO] [Alerting] deployment agnostic slo burn rate rule tests #187924

[SLO] [Alerting] deployment agnostic slo burn rate rule tests #187924

mgiota commented Jul 9, 2024 •

edited

Loading

obltmachine commented Jul 9, 2024

mgiota Jul 10, 2024

dmlemeshko Aug 14, 2024

mgiota Jul 10, 2024 •

edited

Loading

mgiota Jul 10, 2024

dmlemeshko Aug 14, 2024

dominiqueclarke left a comment

mgiota commented Aug 20, 2024

mgiota commented Aug 20, 2024

dominiqueclarke commented Aug 20, 2024

mgiota commented Aug 20, 2024

pheyos left a comment

mgiota commented Aug 21, 2024

kibana-ci commented Aug 21, 2024

[SLO] [Alerting] deployment agnostic slo burn rate rule tests #187924

[SLO] [Alerting] deployment agnostic slo burn rate rule tests #187924

Conversation

mgiota commented Jul 9, 2024 • edited Loading

Update

🍒 History

Description

How to run locally

CI

Quality Gates and MKI pipeline

obltmachine commented Jul 9, 2024

🤖 GitHub comments

mgiota Jul 10, 2024

Choose a reason for hiding this comment

dmlemeshko Aug 14, 2024

Choose a reason for hiding this comment

mgiota Jul 10, 2024 • edited Loading

Choose a reason for hiding this comment

mgiota Jul 10, 2024

Choose a reason for hiding this comment

dmlemeshko Aug 14, 2024

Choose a reason for hiding this comment

dominiqueclarke left a comment

Choose a reason for hiding this comment

mgiota commented Aug 20, 2024

mgiota commented Aug 20, 2024

dominiqueclarke commented Aug 20, 2024

mgiota commented Aug 20, 2024

pheyos left a comment

Choose a reason for hiding this comment

mgiota commented Aug 21, 2024

kibana-ci commented Aug 21, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

History

mgiota commented Jul 9, 2024 •

edited

Loading

mgiota Jul 10, 2024 •

edited

Loading