[APM] Migrate settings API tests to be deployment-agnostic #200762

iblancof · 2024-11-19T15:33:58Z

Closes #198989
Part of #193245

This PR contains the changes to migrate settings test folder to deployment-agnostic testing strategy.

Not Migrated

agent_configuration: Not available in Serverless.
anomaly_detection/no_access: Involves the noAccess user role; we are only migrating tests for viewer, editor, or admin roles.
anomaly_detection/update_to_v3: Involves the deletion of ML jobs; we will wait until an "ml" service is available to properly migrate these tests.
anomaly_detection/write_user: Involves the deletion of ML jobs; we will wait until an "ml" service is available to properly migrate these tests.

Partially Migrated

anomaly_detection/read_user: Involves the apmAllPrivilegesWithoutWriteSettingsUser role; only tests for the read role have been migrated.
anomaly_detection/write_user: Involves the apmReadPrivilegesWithWriteSettingsUser role; only tests for the write role have been migrated.
apm_indices: Tests based on license have not been migrated.
custom_link: Involves the apmReadPrivilegesWithWriteSettingsUser role; only tests for the trial write role have been migrated.
agent_keys: Involves the manageOwnAgentKeysUser and createAndAllAgentKeysUser roles; only tests for the write role have been migrated.

How to test

Serverless

node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.apm.serverless.config.ts
node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.apm.serverless.config.ts

It's recommended to be run against MKI

Stateful

node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.apm.stateful.config.ts 
node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.apm.stateful.config.ts

Checks

(OPTIONAL, only if a test has been unskipped) Run flaky test suite
local run for serverless
local run for stateful
MKI run for serverless

…ests

…tests-settings-to-be-deployment-agnostic-api-tests

elasticmachine · 2024-11-19T15:34:01Z

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

github-actions · 2024-11-19T15:34:11Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

dmlemeshko · 2024-11-19T15:51:06Z

...ntegration/deployment_agnostic/apis/observability/apm/settings/agent_keys/agent_keys.spec.ts

+    before(async () => {
+      roleAuthc = await samlAuth.createM2mApiKeyWithRoleScope('editor');
+    });


API key should be invalidated in after hook:

after(async () => { await samlAuth.invalidateM2mApiKeyWithRoleScope(roleAuthc); });

I missed that, thank you!

e183e46

dmlemeshko

LGTM, few nits

dmlemeshko · 2024-11-19T15:53:40Z

...ntegration/deployment_agnostic/apis/observability/apm/settings/agent_keys/agent_keys.spec.ts

+    describe('When the user does not have the required cluster privileges', () => {
+      it('should return an error when creating an agent key', async () => {
+        const error = await expectToReject<ApmApiError>(() =>
+          createAgentKey(apmApiClient.writeUser, roleAuthc)


Could you explain why do we use different sources of authentication here: afaik apmApiClient.writeUser is already authorised with Editor privileges and a another "fresh" API key is passed?

The POST /api/apm/agent_keys 2023-10-31 endpoint, when used with the apmApiClient, specifically requires the roleAuthc parameter.

From what I observed in the code, this is necessary because the endpoint is not considered internal, so additional credential information must be included in the request headers.

It seems the credentials require an API key, but the apmApiClient does not have one by default. To address this, we need to create a new role with the appropriate API key attached, ensuring the credentials are correctly passed in the request headers.

I’m not sure about the reasoning behind handling this outside of the apmApiClient instead of within it. Maybe it’s related to the need to invalidate the API key after the tests are run, avoiding the creation and invalidation of keys for every request that requires one.

Perhaps @crespocarlos could share more details about why it’s done this way in case you want to know more about it.

Now that I'm looking at it, it is weird. We need to improve it.

roleAuthc needs to be passed to apmApi because, as you've said, we need to destroy the key after the test runs, but calling apmApi service with writeUser and then having to pass the apiKey looks very strange.

For now, lets not use the apmApi to call public APIs. It's a momentary compromise on the strongly typed response, but it'd be less confusing. Let's use the SupertestWithRoleScopeType in this case.

we could type the response manually as:

let response: APIReturnType<'POST /api/apm/agent_keys 2023-10-31'>

Suggestion: iblancof#1

I’ll be using the new apmApiClient.publicApi introduced in iblancof#1.

d50e6bf

crespocarlos · 2024-11-20T09:30:08Z

x-pack/test/api_integration/deployment_agnostic/apis/observability/apm/index.ts

@@ -12,31 +12,32 @@ export default function apmApiIntegrationTests({
 }: DeploymentAgnosticFtrProviderContext) {
  describe('APM', function () {


Thank you for reorg the list. This was bothering me so much.

...ntegration/deployment_agnostic/apis/observability/apm/settings/agent_keys/agent_keys.spec.ts

crespocarlos · 2024-11-20T09:47:29Z

...ntegration/deployment_agnostic/apis/observability/apm/settings/agent_keys/agent_keys.spec.ts

+    describe('When the user does not have the required cluster privileges', () => {
+      it('should return an error when creating an agent key', async () => {
+        const error = await expectToReject<ApmApiError>(() =>
+          createAgentKey(apmApiClient.writeUser, roleAuthc)


Now that I'm looking at it, it is weird. We need to improve it.

roleAuthc needs to be passed to apmApi because, as you've said, we need to destroy the key after the test runs, but calling apmApi service with writeUser and then having to pass the apiKey looks very strange.

...egration/deployment_agnostic/apis/observability/apm/settings/anomaly_detection/basic.spec.ts

...egration/deployment_agnostic/apis/observability/apm/settings/apm_indices/apm_indices.spec.ts

...egration/deployment_agnostic/apis/observability/apm/settings/custom_link/custom_link.spec.ts

…lity/apm/settings/custom_link/custom_link.spec.ts Co-authored-by: Carlos Crespo <[email protected]>

…lity/apm/settings/apm_indices/apm_indices.spec.ts Co-authored-by: Carlos Crespo <[email protected]>

…tests-settings-to-be-deployment-agnostic-api-tests

…ngs-to-be-deployment-agnostic-api-tests' of https://github.com/iblancof/kibana into 198989-apm-migrate-test-apm_api_integration-tests-settings-to-be-deployment-agnostic-api-tests

…tests-settings-to-be-deployment-agnostic-api-tests

dmlemeshko

Thanks for refactoring apmApi service, code LGTM

…tests-settings-to-be-deployment-agnostic-api-tests

elasticmachine · 2024-11-21T21:18:39Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: f5c0e14
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-200762-f5c0e147e77c

Failed CI Steps

Jest Tests #18

Test Failures

[job] [logs] Jest Tests #18 / RootPrivilegesCallout should render callout requiring root privileges

Metrics [docs]

✅ unchanged

History

cc @iblancof

kibanamachine · 2024-11-21T22:09:13Z

Starting backport for target branches: 8.17, 8.x

https://github.com/elastic/kibana/actions/runs/11962493177

kibanamachine · 2024-11-21T22:14:02Z

💔 All backports failed

Status	Branch	Result
❌	8.17	Backport failed because of merge conflicts
❌	8.x	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 200762

Questions ?

Please refer to the Backport tool documentation

iblancof · 2024-11-22T09:24:40Z

💚 All backports created successfully

Status	Branch	Result
✅	8.17

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

iblancof · 2024-11-22T09:25:53Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…00762) Closes elastic#198989 Part of elastic#193245 This PR contains the changes to migrate `settings` test folder to deployment-agnostic testing strategy. **Not Migrated** - `agent_configuration`: Not available in Serverless. - `anomaly_detection/no_access`: Involves the `noAccess` user role; we are only migrating tests for `viewer`, `editor`, or `admin` roles. - `anomaly_detection/update_to_v3`: Involves the deletion of ML jobs; we will wait until an "ml" service is available to properly migrate these tests. - `anomaly_detection/write_user`: Involves the deletion of ML jobs; we will wait until an "ml" service is available to properly migrate these tests. **Partially Migrated** - `anomaly_detection/read_user`: Involves the `apmAllPrivilegesWithoutWriteSettingsUser` role; only tests for the `read` role have been migrated. - `anomaly_detection/write_user`: Involves the `apmReadPrivilegesWithWriteSettingsUser` role; only tests for the `write` role have been migrated. - `apm_indices`: Tests based on license have not been migrated. custom_link: Involves the `apmReadPrivilegesWithWriteSettingsUser` role; only tests for the trial `write` role have been migrated. - `agent_keys`: Involves the `manageOwnAgentKeysUser` and `createAndAllAgentKeysUser` roles; only tests for the `write` role have been migrated. ### How to test - Serverless ``` node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.apm.serverless.config.ts node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.apm.serverless.config.ts ``` It's recommended to be run against [MKI](https://github.com/crespocarlos/kibana/blob/main/x-pack/test_serverless/README.md#run-tests-on-mki) - Stateful ``` node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.apm.stateful.config.ts node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.apm.stateful.config.ts ``` ## Checks - [ ] (OPTIONAL, only if a test has been unskipped) Run flaky test suite - [x] local run for serverless - [x] local run for stateful - [x] MKI run for serverless  --------- Co-authored-by: Carlos Crespo <[email protected]> Co-authored-by: Carlos Crespo <[email protected]> (cherry picked from commit 05bf56f) # Conflicts: # x-pack/test/api_integration/deployment_agnostic/apis/observability/apm/index.ts

…00762) (#201327) # Backport This will backport the following commits from `main` to `8.17`: - [[APM] Migrate settings API tests to be deployment-agnostic (#200762)](#200762)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) \r\n\r\n---------\r\n\r\nCo-authored-by: Carlos Crespo <[email protected]>\r\nCo-authored-by: Carlos Crespo <[email protected]>","sha":"05bf56f3365e9f273bad6d29cc5855d9c3607fc7","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","apm","backport:prev-minor","ci:project-deploy-observability","Team:obs-ux-infra_services","v8.17.0"],"number":200762,"url":"https://github.com/elastic/kibana/pull/200762","mergeCommit":{"message":"[APM] Migrate settings API tests to be deployment-agnostic (#200762)\n\nCloses https://github.com/elastic/kibana/issues/198989\r\nPart of https://github.com/elastic/kibana/issues/193245\r\n\r\nThis PR contains the changes to migrate `settings` test folder to\r\ndeployment-agnostic testing strategy.\r\n\r\n\r\n**Not Migrated**\r\n- `agent_configuration`: Not available in Serverless.\r\n- `anomaly_detection/no_access`: Involves the `noAccess` user role; we\r\nare only migrating tests for `viewer`, `editor`, or `admin` roles.\r\n- `anomaly_detection/update_to_v3`: Involves the deletion of ML jobs; we\r\nwill wait until an \"ml\" service is available to properly migrate these\r\ntests.\r\n- `anomaly_detection/write_user`: Involves the deletion of ML jobs; we\r\nwill wait until an \"ml\" service is available to properly migrate these\r\ntests.\r\n\r\n**Partially Migrated**\r\n- `anomaly_detection/read_user`: Involves the\r\n`apmAllPrivilegesWithoutWriteSettingsUser` role; only tests for the\r\n`read` role have been migrated.\r\n- `anomaly_detection/write_user`: Involves the\r\n`apmReadPrivilegesWithWriteSettingsUser` role; only tests for the\r\n`write` role have been migrated.\r\n- `apm_indices`: Tests based on license have not been migrated.\r\ncustom_link: Involves the `apmReadPrivilegesWithWriteSettingsUser` role;\r\nonly tests for the trial `write` role have been migrated.\r\n- `agent_keys`: Involves the `manageOwnAgentKeysUser` and\r\n`createAndAllAgentKeysUser` roles; only tests for the `write` role have\r\nbeen migrated.\r\n\r\n### How to test\r\n\r\n\r\n- Serverless\r\n\r\n```\r\nnode scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.apm.serverless.config.ts\r\nnode scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.apm.serverless.config.ts \r\n```\r\n\r\nIt's recommended to be run against\r\n[MKI](https://github.com/crespocarlos/kibana/blob/main/x-pack/test_serverless/README.md#run-tests-on-mki)\r\n\r\n- Stateful\r\n```\r\nnode scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.apm.stateful.config.ts \r\nnode scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.apm.stateful.config.ts \r\n```\r\n\r\n## Checks\r\n\r\n- [ ] (OPTIONAL, only if a test has been unskipped) Run flaky test suite\r\n- [x] local run for serverless\r\n- [x] local run for stateful\r\n- [x] MKI run for serverless \r\n\r\n\r\n\r\n---------\r\n\r\nCo-authored-by: Carlos Crespo <[email protected]>\r\nCo-authored-by: Carlos Crespo <[email protected]>","sha":"05bf56f3365e9f273bad6d29cc5855d9c3607fc7"}},"sourceBranch":"main","suggestedTargetBranches":["8.17"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/200762","number":200762,"mergeCommit":{"message":"[APM] Migrate settings API tests to be deployment-agnostic (#200762)\n\nCloses https://github.com/elastic/kibana/issues/198989\r\nPart of https://github.com/elastic/kibana/issues/193245\r\n\r\nThis PR contains the changes to migrate `settings` test folder to\r\ndeployment-agnostic testing strategy.\r\n\r\n\r\n**Not Migrated**\r\n- `agent_configuration`: Not available in Serverless.\r\n- `anomaly_detection/no_access`: Involves the `noAccess` user role; we\r\nare only migrating tests for `viewer`, `editor`, or `admin` roles.\r\n- `anomaly_detection/update_to_v3`: Involves the deletion of ML jobs; we\r\nwill wait until an \"ml\" service is available to properly migrate these\r\ntests.\r\n- `anomaly_detection/write_user`: Involves the deletion of ML jobs; we\r\nwill wait until an \"ml\" service is available to properly migrate these\r\ntests.\r\n\r\n**Partially Migrated**\r\n- `anomaly_detection/read_user`: Involves the\r\n`apmAllPrivilegesWithoutWriteSettingsUser` role; only tests for the\r\n`read` role have been migrated.\r\n- `anomaly_detection/write_user`: Involves the\r\n`apmReadPrivilegesWithWriteSettingsUser` role; only tests for the\r\n`write` role have been migrated.\r\n- `apm_indices`: Tests based on license have not been migrated.\r\ncustom_link: Involves the `apmReadPrivilegesWithWriteSettingsUser` role;\r\nonly tests for the trial `write` role have been migrated.\r\n- `agent_keys`: Involves the `manageOwnAgentKeysUser` and\r\n`createAndAllAgentKeysUser` roles; only tests for the `write` role have\r\nbeen migrated.\r\n\r\n### How to test\r\n\r\n\r\n- Serverless\r\n\r\n```\r\nnode scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.apm.serverless.config.ts\r\nnode scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.apm.serverless.config.ts \r\n```\r\n\r\nIt's recommended to be run against\r\n[MKI](https://github.com/crespocarlos/kibana/blob/main/x-pack/test_serverless/README.md#run-tests-on-mki)\r\n\r\n- Stateful\r\n```\r\nnode scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.apm.stateful.config.ts \r\nnode scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.apm.stateful.config.ts \r\n```\r\n\r\n## Checks\r\n\r\n- [ ] (OPTIONAL, only if a test has been unskipped) Run flaky test suite\r\n- [x] local run for serverless\r\n- [x] local run for stateful\r\n- [x] MKI run for serverless \r\n\r\n\r\n\r\n---------\r\n\r\nCo-authored-by: Carlos Crespo <[email protected]>\r\nCo-authored-by: Carlos Crespo <[email protected]>","sha":"05bf56f3365e9f273bad6d29cc5855d9c3607fc7"}},{"branch":"8.17","label":"v8.17.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->

…0762) (#201328) # Backport This will backport the following commits from `main` to `8.x`: - [[APM] Migrate settings API tests to be deployment-agnostic (#200762)](#200762)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) \r\n\r\n---------\r\n\r\nCo-authored-by: Carlos Crespo <[email protected]>\r\nCo-authored-by: Carlos Crespo <[email protected]>","sha":"05bf56f3365e9f273bad6d29cc5855d9c3607fc7","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","apm","backport:prev-minor","ci:project-deploy-observability","Team:obs-ux-infra_services","v8.17.0"],"number":200762,"url":"https://github.com/elastic/kibana/pull/200762","mergeCommit":{"message":"[APM] Migrate settings API tests to be deployment-agnostic (#200762)\n\nCloses https://github.com/elastic/kibana/issues/198989\r\nPart of https://github.com/elastic/kibana/issues/193245\r\n\r\nThis PR contains the changes to migrate `settings` test folder to\r\ndeployment-agnostic testing strategy.\r\n\r\n\r\n**Not Migrated**\r\n- `agent_configuration`: Not available in Serverless.\r\n- `anomaly_detection/no_access`: Involves the `noAccess` user role; we\r\nare only migrating tests for `viewer`, `editor`, or `admin` roles.\r\n- `anomaly_detection/update_to_v3`: Involves the deletion of ML jobs; we\r\nwill wait until an \"ml\" service is available to properly migrate these\r\ntests.\r\n- `anomaly_detection/write_user`: Involves the deletion of ML jobs; we\r\nwill wait until an \"ml\" service is available to properly migrate these\r\ntests.\r\n\r\n**Partially Migrated**\r\n- `anomaly_detection/read_user`: Involves the\r\n`apmAllPrivilegesWithoutWriteSettingsUser` role; only tests for the\r\n`read` role have been migrated.\r\n- `anomaly_detection/write_user`: Involves the\r\n`apmReadPrivilegesWithWriteSettingsUser` role; only tests for the\r\n`write` role have been migrated.\r\n- `apm_indices`: Tests based on license have not been migrated.\r\ncustom_link: Involves the `apmReadPrivilegesWithWriteSettingsUser` role;\r\nonly tests for the trial `write` role have been migrated.\r\n- `agent_keys`: Involves the `manageOwnAgentKeysUser` and\r\n`createAndAllAgentKeysUser` roles; only tests for the `write` role have\r\nbeen migrated.\r\n\r\n### How to test\r\n\r\n\r\n- Serverless\r\n\r\n```\r\nnode scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.apm.serverless.config.ts\r\nnode scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.apm.serverless.config.ts \r\n```\r\n\r\nIt's recommended to be run against\r\n[MKI](https://github.com/crespocarlos/kibana/blob/main/x-pack/test_serverless/README.md#run-tests-on-mki)\r\n\r\n- Stateful\r\n```\r\nnode scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.apm.stateful.config.ts \r\nnode scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.apm.stateful.config.ts \r\n```\r\n\r\n## Checks\r\n\r\n- [ ] (OPTIONAL, only if a test has been unskipped) Run flaky test suite\r\n- [x] local run for serverless\r\n- [x] local run for stateful\r\n- [x] MKI run for serverless \r\n\r\n\r\n\r\n---------\r\n\r\nCo-authored-by: Carlos Crespo <[email protected]>\r\nCo-authored-by: Carlos Crespo <[email protected]>","sha":"05bf56f3365e9f273bad6d29cc5855d9c3607fc7"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/200762","number":200762,"mergeCommit":{"message":"[APM] Migrate settings API tests to be deployment-agnostic (#200762)\n\nCloses https://github.com/elastic/kibana/issues/198989\r\nPart of https://github.com/elastic/kibana/issues/193245\r\n\r\nThis PR contains the changes to migrate `settings` test folder to\r\ndeployment-agnostic testing strategy.\r\n\r\n\r\n**Not Migrated**\r\n- `agent_configuration`: Not available in Serverless.\r\n- `anomaly_detection/no_access`: Involves the `noAccess` user role; we\r\nare only migrating tests for `viewer`, `editor`, or `admin` roles.\r\n- `anomaly_detection/update_to_v3`: Involves the deletion of ML jobs; we\r\nwill wait until an \"ml\" service is available to properly migrate these\r\ntests.\r\n- `anomaly_detection/write_user`: Involves the deletion of ML jobs; we\r\nwill wait until an \"ml\" service is available to properly migrate these\r\ntests.\r\n\r\n**Partially Migrated**\r\n- `anomaly_detection/read_user`: Involves the\r\n`apmAllPrivilegesWithoutWriteSettingsUser` role; only tests for the\r\n`read` role have been migrated.\r\n- `anomaly_detection/write_user`: Involves the\r\n`apmReadPrivilegesWithWriteSettingsUser` role; only tests for the\r\n`write` role have been migrated.\r\n- `apm_indices`: Tests based on license have not been migrated.\r\ncustom_link: Involves the `apmReadPrivilegesWithWriteSettingsUser` role;\r\nonly tests for the trial `write` role have been migrated.\r\n- `agent_keys`: Involves the `manageOwnAgentKeysUser` and\r\n`createAndAllAgentKeysUser` roles; only tests for the `write` role have\r\nbeen migrated.\r\n\r\n### How to test\r\n\r\n\r\n- Serverless\r\n\r\n```\r\nnode scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.apm.serverless.config.ts\r\nnode scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.apm.serverless.config.ts \r\n```\r\n\r\nIt's recommended to be run against\r\n[MKI](https://github.com/crespocarlos/kibana/blob/main/x-pack/test_serverless/README.md#run-tests-on-mki)\r\n\r\n- Stateful\r\n```\r\nnode scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.apm.stateful.config.ts \r\nnode scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.apm.stateful.config.ts \r\n```\r\n\r\n## Checks\r\n\r\n- [ ] (OPTIONAL, only if a test has been unskipped) Run flaky test suite\r\n- [x] local run for serverless\r\n- [x] local run for stateful\r\n- [x] MKI run for serverless \r\n\r\n\r\n\r\n---------\r\n\r\nCo-authored-by: Carlos Crespo <[email protected]>\r\nCo-authored-by: Carlos Crespo <[email protected]>","sha":"05bf56f3365e9f273bad6d29cc5855d9c3607fc7"}},{"branch":"8.17","label":"v8.17.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/201327","number":201327,"state":"OPEN"}]}] BACKPORT-->

…00762) Closes elastic#198989 Part of elastic#193245 This PR contains the changes to migrate `settings` test folder to deployment-agnostic testing strategy. **Not Migrated** - `agent_configuration`: Not available in Serverless. - `anomaly_detection/no_access`: Involves the `noAccess` user role; we are only migrating tests for `viewer`, `editor`, or `admin` roles. - `anomaly_detection/update_to_v3`: Involves the deletion of ML jobs; we will wait until an "ml" service is available to properly migrate these tests. - `anomaly_detection/write_user`: Involves the deletion of ML jobs; we will wait until an "ml" service is available to properly migrate these tests. **Partially Migrated** - `anomaly_detection/read_user`: Involves the `apmAllPrivilegesWithoutWriteSettingsUser` role; only tests for the `read` role have been migrated. - `anomaly_detection/write_user`: Involves the `apmReadPrivilegesWithWriteSettingsUser` role; only tests for the `write` role have been migrated. - `apm_indices`: Tests based on license have not been migrated. custom_link: Involves the `apmReadPrivilegesWithWriteSettingsUser` role; only tests for the trial `write` role have been migrated. - `agent_keys`: Involves the `manageOwnAgentKeysUser` and `createAndAllAgentKeysUser` roles; only tests for the `write` role have been migrated. ### How to test - Serverless ``` node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.apm.serverless.config.ts node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.apm.serverless.config.ts ``` It's recommended to be run against [MKI](https://github.com/crespocarlos/kibana/blob/main/x-pack/test_serverless/README.md#run-tests-on-mki) - Stateful ``` node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.apm.stateful.config.ts node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.apm.stateful.config.ts ``` ## Checks - [ ] (OPTIONAL, only if a test has been unskipped) Run flaky test suite - [x] local run for serverless - [x] local run for stateful - [x] MKI run for serverless  --------- Co-authored-by: Carlos Crespo <[email protected]> Co-authored-by: Carlos Crespo <[email protected]>

iblancof added 8 commits November 18, 2024 11:58

Create settings/anomaly_detection basic deployment agnostic api tests

1c3c51a

Create settings/anomaly_detection read user deployment agnostic api t…

744db47

…ests

Create settings/custom_link deployment agnostic api tests

eeb1971

Create settings/apm_indices deployment agnostic api tests

a825c95

Create settings/agent_keys deployment agnostic api tests

86c262f

Create tests loader for settings deployment agnostic tests

8c4f48a

Merge branch 'main' into 198989-apm-migrate-test-apm_api_integration-…

4a4c838

…tests-settings-to-be-deployment-agnostic-api-tests

Clean stateful tests

ed0bc6b

iblancof added release_note:skip Skip the PR/issue when compiling release notes apm backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team v8.17.0 labels Nov 19, 2024

iblancof self-assigned this Nov 19, 2024

iblancof requested review from a team as code owners November 19, 2024 15:33

botelastic bot added the ci:project-deploy-observability Create an Observability project label Nov 19, 2024

iblancof requested a review from crespocarlos November 19, 2024 15:37

iblancof mentioned this pull request Nov 19, 2024

[APM] ML API deployment-agnostic API tests migration #200764

Open

dmlemeshko reviewed Nov 19, 2024

View reviewed changes

iblancof added 3 commits November 19, 2024 17:07

Refactor imports to use type-only imports in APM settings test files

e74f551

Invalidate samlAuth

e183e46

Remove unused constant

48a98c9

dmlemeshko reviewed Nov 19, 2024

View reviewed changes

crespocarlos reviewed Nov 20, 2024

View reviewed changes

iblancof and others added 3 commits November 20, 2024 11:34

Update x-pack/test/api_integration/deployment_agnostic/apis/observabi…

8c61a72

…lity/apm/settings/custom_link/custom_link.spec.ts Co-authored-by: Carlos Crespo <[email protected]>

Update x-pack/test/api_integration/deployment_agnostic/apis/observabi…

41e3a23

…lity/apm/settings/apm_indices/apm_indices.spec.ts Co-authored-by: Carlos Crespo <[email protected]>

Remove migrated test

2cb8be0

iblancof added 5 commits November 20, 2024 17:28

Remove duplicated test file import

e2ed6f5

Merge branch 'main' into 198989-apm-migrate-test-apm_api_integration-…

61a9a0c

…tests-settings-to-be-deployment-agnostic-api-tests

Remove test case that was not deployment agnostic

4f8ad0c

Remove duplicated writeUser test

f1e40f1

Merge branch '198989-apm-migrate-test-apm_api_integration-tests-setti…

681d40a

…ngs-to-be-deployment-agnostic-api-tests' of https://github.com/iblancof/kibana into 198989-apm-migrate-test-apm_api_integration-tests-settings-to-be-deployment-agnostic-api-tests

elastic deleted a comment from kibanamachine Nov 21, 2024

iblancof requested a review from dmlemeshko November 21, 2024 09:44

iblancof added 2 commits November 21, 2024 12:34

Merge branch 'main' into 198989-apm-migrate-test-apm_api_integration-…

96db562

…tests-settings-to-be-deployment-agnostic-api-tests

Merge branch 'main' into 198989-apm-migrate-test-apm_api_integration-…

e8371fb

…tests-settings-to-be-deployment-agnostic-api-tests

dmlemeshko approved these changes Nov 21, 2024

View reviewed changes

iblancof added 2 commits November 21, 2024 18:09

Add deleteSavedObject again afterEach

269f33f

Merge branch 'main' into 198989-apm-migrate-test-apm_api_integration-…

ea617dd

…tests-settings-to-be-deployment-agnostic-api-tests

iblancof enabled auto-merge (squash) November 21, 2024 17:13

Merge branch 'main' into 198989-apm-migrate-test-apm_api_integration-…

f5c0e14

…tests-settings-to-be-deployment-agnostic-api-tests

iblancof merged commit 05bf56f into elastic:main Nov 21, 2024
24 checks passed

kibanamachine added the v9.0.0 label Nov 21, 2024

iblancof mentioned this pull request Nov 22, 2024

[8.17] [APM] Migrate settings API tests to be deployment-agnostic (#200762) #201327

Merged

iblancof mentioned this pull request Nov 22, 2024

[8.x] [APM] Migrate settings API tests to be deployment-agnostic (#200762) #201328

Merged

kibanamachine added the v8.18.0 label Nov 22, 2024

iblancof deleted the 198989-apm-migrate-test-apm_api_integration-tests-settings-to-be-deployment-agnostic-api-tests branch December 2, 2024 13:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[APM] Migrate settings API tests to be deployment-agnostic #200762

[APM] Migrate settings API tests to be deployment-agnostic #200762

iblancof commented Nov 19, 2024 •

edited by kibanamachine

Loading

elasticmachine commented Nov 19, 2024

github-actions bot commented Nov 19, 2024

dmlemeshko Nov 19, 2024

iblancof Nov 19, 2024

dmlemeshko left a comment

dmlemeshko Nov 19, 2024

iblancof Nov 20, 2024

crespocarlos Nov 20, 2024

crespocarlos Nov 20, 2024 •

edited

Loading

crespocarlos Nov 20, 2024

iblancof Nov 20, 2024

crespocarlos Nov 20, 2024

crespocarlos Nov 20, 2024

dmlemeshko left a comment

elasticmachine commented Nov 21, 2024 •

edited

Loading

kibanamachine commented Nov 21, 2024

kibanamachine commented Nov 21, 2024

iblancof commented Nov 22, 2024

iblancof commented Nov 22, 2024

		@@ -12,31 +12,32 @@ export default function apmApiIntegrationTests({
		}: DeploymentAgnosticFtrProviderContext) {
		describe('APM', function () {

[APM] Migrate settings API tests to be deployment-agnostic #200762

[APM] Migrate settings API tests to be deployment-agnostic #200762

Conversation

iblancof commented Nov 19, 2024 • edited by kibanamachine Loading

How to test

Checks

elasticmachine commented Nov 19, 2024

github-actions bot commented Nov 19, 2024

🤖 GitHub comments

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmlemeshko left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crespocarlos Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmlemeshko left a comment

Choose a reason for hiding this comment

elasticmachine commented Nov 21, 2024 • edited Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

History

kibanamachine commented Nov 21, 2024

kibanamachine commented Nov 21, 2024

💔 All backports failed

Manual backport

Questions ?

iblancof commented Nov 22, 2024

💚 All backports created successfully

Questions ?

iblancof commented Nov 22, 2024

💚 All backports created successfully

Questions ?

iblancof commented Nov 19, 2024 •

edited by kibanamachine

Loading

crespocarlos Nov 20, 2024 •

edited

Loading

elasticmachine commented Nov 21, 2024 •

edited

Loading