Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[APM] Migrate latency API tests to be deployment-agnostic #199802

Conversation

iblancof
Copy link
Contributor

@iblancof iblancof commented Nov 12, 2024

How to test

Closes #198978
Part of #193245

This PR contains the changes to migrate latency test folder to deployment-agnostic testing strategy.

How to test

  • Serverless
node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts
node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts --grep="APM"

It's recommended to be run against MKI

  • Stateful
node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts
node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts --grep="APM"

Checks

  • (OPTIONAL, only if a test has been unskipped) Run flaky test suite
  • local run for serverless
  • local run for stateful
  • MKI run for serverless

@iblancof iblancof added release_note:skip Skip the PR/issue when compiling release notes apm backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team v8.17.0 labels Nov 12, 2024
@iblancof iblancof self-assigned this Nov 12, 2024
@iblancof iblancof changed the title 198978 apm migrate test apm api integration tests latency to be deployment agnostic api tests [APM] Migrate latency API tests to be deployment-agnostic Nov 12, 2024
@iblancof
Copy link
Contributor Author

Flaky test runner going on here to check if we can remove flaky tests comments.

@iblancof iblancof marked this pull request as ready for review November 12, 2024 14:37
@iblancof iblancof requested review from a team as code owners November 12, 2024 14:37
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

Copy link
Contributor

@rmyz rmyz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

Copy link
Member

@wayneseymour wayneseymour left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 questions

before(async () => {
apmSynthtraceEsClient = await synthtrace.createApmSynthtraceEsClient();
});

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need an after call to clean up, like this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, and it's there


before(async () => {
apmSynthtraceEsClient = await synthtrace.createApmSynthtraceEsClient();
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need an after call to clean up, like this?

Copy link
Contributor Author

@iblancof iblancof Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, and it's there

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7385

[✅] x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts: 25/25 tests passed.
[✅] x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts: 25/25 tests passed.

see run history

Copy link
Contributor

@crespocarlos crespocarlos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can now remove this: x-pack/test_serverless/api_integration/test_suites/observability/apm_api_integration/service_maps/service_maps.ts

or incorporate into the service_maps.speec.ts from this PR.

@iblancof
Copy link
Contributor Author

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7385

[✅] x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts: 25/25 tests passed. [✅] x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts: 25/25 tests passed.

see run history

All runs with the runner went well, but it did fail once on my local setup, so let's keep the "flaky" test comment just in case.

@iblancof
Copy link
Contributor Author

We can now remove this: x-pack/test_serverless/api_integration/test_suites/observability/apm_api_integration/service_maps/service_maps.ts

or incorporate into the service_maps.spec.ts from this PR.

Hey @crespocarlos, could you clarify why you’re relating the files?

I noticed we have three different service_maps.spec.ts files, and I would expect each one to test different aspects of service_map. So, it seems logical to keep them separated.

Screenshot 2024-11-12 at 17 33 03

Please let me know if I misunderstood your point, because from my perspective, we shouldn’t mix latency, throughput, and general tests.

@crespocarlos
Copy link
Contributor

We can now remove this: x-pack/test_serverless/api_integration/test_suites/observability/apm_api_integration/service_maps/service_maps.ts
or incorporate into the service_maps.spec.ts from this PR.

Hey @crespocarlos, could you clarify why you’re relating the files?

I noticed we have three different service_maps.spec.ts files, and I would expect each one to test different aspects of service_map. So, it seems logical to keep them separated.

Screenshot 2024-11-12 at 17 33 03 Please let me know if I misunderstood your point, because from my perspective, we shouldn’t mix latency, throughput, and general tests.

Sorry, my bad. I thought this was the service_maps folder. nvm

…tests-latency-to-be-deployment-agnostic-api-tests
@elasticmachine
Copy link
Contributor

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #10 / upgrade assistant Kibana API Deprecations returns deprecated APIs when the api is called

Metrics [docs]

✅ unchanged

cc @iblancof

@iblancof iblancof merged commit 411573b into elastic:main Nov 13, 2024
22 checks passed
@iblancof iblancof deleted the 198978-apm-migrate-test-apm_api_integration-tests-latency-to-be-deployment-agnostic-api-tests branch November 13, 2024 10:40
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11815572208

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Nov 13, 2024
…9802)

### How to test
Closes elastic#198978
Part of elastic#193245

This PR contains the changes to migrate `latency` test folder to
deployment-agnostic testing strategy.

### How to test

- Serverless

```
node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts
node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts --grep="APM"
```

It's recommended to be run against
[MKI](https://github.com/crespocarlos/kibana/blob/main/x-pack/test_serverless/README.md#run-tests-on-mki)

- Stateful
```
node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts
node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts --grep="APM"
```

## Checks

- [ ] (OPTIONAL, only if a test has been unskipped) Run flaky test suite
- [x] local run for serverless
- [x] local run for stateful
- [x] MKI run for serverless

(cherry picked from commit 411573b)
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Nov 13, 2024
) (#199972)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[APM] Migrate latency API tests to be deployment-agnostic
(#199802)](#199802)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Irene
Blanco","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-13T10:40:06Z","message":"[APM]
Migrate latency API tests to be deployment-agnostic (#199802)\n\n### How
to test\r\nCloses
https://github.com/elastic/kibana/issues/198978\r\nPart of
https://github.com/elastic/kibana/issues/193245\r\n\r\nThis PR contains
the changes to migrate `latency` test folder to\r\ndeployment-agnostic
testing strategy.\r\n\r\n### How to test\r\n\r\n\r\n-
Serverless\r\n\r\n```\r\nnode scripts/functional_tests_server --config
x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts\r\nnode
scripts/functional_test_runner --config
x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts
--grep=\"APM\"\r\n```\r\n\r\nIt's recommended to be run
against\r\n[MKI](https://github.com/crespocarlos/kibana/blob/main/x-pack/test_serverless/README.md#run-tests-on-mki)\r\n\r\n-
Stateful\r\n```\r\nnode scripts/functional_tests_server --config
x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts\r\nnode
scripts/functional_test_runner --config
x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts
--grep=\"APM\"\r\n```\r\n\r\n## Checks\r\n\r\n- [ ] (OPTIONAL, only if a
test has been unskipped) Run flaky test suite\r\n- [x] local run for
serverless\r\n- [x] local run for stateful\r\n- [x] MKI run for
serverless","sha":"411573bded37b3e338d6709ddfadf2c95987b0f5","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","apm","backport:prev-minor","Team:obs-ux-infra_services","v8.17.0"],"title":"[APM]
Migrate latency API tests to be deployment-agnostic
","number":199802,"url":"https://github.com/elastic/kibana/pull/199802","mergeCommit":{"message":"[APM]
Migrate latency API tests to be deployment-agnostic (#199802)\n\n### How
to test\r\nCloses
https://github.com/elastic/kibana/issues/198978\r\nPart of
https://github.com/elastic/kibana/issues/193245\r\n\r\nThis PR contains
the changes to migrate `latency` test folder to\r\ndeployment-agnostic
testing strategy.\r\n\r\n### How to test\r\n\r\n\r\n-
Serverless\r\n\r\n```\r\nnode scripts/functional_tests_server --config
x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts\r\nnode
scripts/functional_test_runner --config
x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts
--grep=\"APM\"\r\n```\r\n\r\nIt's recommended to be run
against\r\n[MKI](https://github.com/crespocarlos/kibana/blob/main/x-pack/test_serverless/README.md#run-tests-on-mki)\r\n\r\n-
Stateful\r\n```\r\nnode scripts/functional_tests_server --config
x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts\r\nnode
scripts/functional_test_runner --config
x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts
--grep=\"APM\"\r\n```\r\n\r\n## Checks\r\n\r\n- [ ] (OPTIONAL, only if a
test has been unskipped) Run flaky test suite\r\n- [x] local run for
serverless\r\n- [x] local run for stateful\r\n- [x] MKI run for
serverless","sha":"411573bded37b3e338d6709ddfadf2c95987b0f5"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/199802","number":199802,"mergeCommit":{"message":"[APM]
Migrate latency API tests to be deployment-agnostic (#199802)\n\n### How
to test\r\nCloses
https://github.com/elastic/kibana/issues/198978\r\nPart of
https://github.com/elastic/kibana/issues/193245\r\n\r\nThis PR contains
the changes to migrate `latency` test folder to\r\ndeployment-agnostic
testing strategy.\r\n\r\n### How to test\r\n\r\n\r\n-
Serverless\r\n\r\n```\r\nnode scripts/functional_tests_server --config
x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts\r\nnode
scripts/functional_test_runner --config
x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts
--grep=\"APM\"\r\n```\r\n\r\nIt's recommended to be run
against\r\n[MKI](https://github.com/crespocarlos/kibana/blob/main/x-pack/test_serverless/README.md#run-tests-on-mki)\r\n\r\n-
Stateful\r\n```\r\nnode scripts/functional_tests_server --config
x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts\r\nnode
scripts/functional_test_runner --config
x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts
--grep=\"APM\"\r\n```\r\n\r\n## Checks\r\n\r\n- [ ] (OPTIONAL, only if a
test has been unskipped) Run flaky test suite\r\n- [x] local run for
serverless\r\n- [x] local run for stateful\r\n- [x] MKI run for
serverless","sha":"411573bded37b3e338d6709ddfadf2c95987b0f5"}},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Irene Blanco <[email protected]>
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Nov 18, 2024
…9802)

### How to test
Closes elastic#198978
Part of elastic#193245

This PR contains the changes to migrate `latency` test folder to
deployment-agnostic testing strategy.

### How to test


- Serverless

```
node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts
node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts --grep="APM"
```

It's recommended to be run against
[MKI](https://github.com/crespocarlos/kibana/blob/main/x-pack/test_serverless/README.md#run-tests-on-mki)

- Stateful
```
node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts
node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts --grep="APM"
```

## Checks

- [ ] (OPTIONAL, only if a test has been unskipped) Run flaky test suite
- [x] local run for serverless
- [x] local run for stateful
- [x] MKI run for serverless
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Nov 18, 2024
…9802)

### How to test
Closes elastic#198978
Part of elastic#193245

This PR contains the changes to migrate `latency` test folder to
deployment-agnostic testing strategy.

### How to test


- Serverless

```
node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts
node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts --grep="APM"
```

It's recommended to be run against
[MKI](https://github.com/crespocarlos/kibana/blob/main/x-pack/test_serverless/README.md#run-tests-on-mki)

- Stateful
```
node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts
node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts --grep="APM"
```

## Checks

- [ ] (OPTIONAL, only if a test has been unskipped) Run flaky test suite
- [x] local run for serverless
- [x] local run for stateful
- [x] MKI run for serverless
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
apm backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) release_note:skip Skip the PR/issue when compiling release notes Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team v8.17.0 v9.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[APM] Migrate /test/apm_api_integration/tests/latency to be deployment-agnostic API tests
6 participants