Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[APM][Otel] Use fields instead of _source on APM queries #195242

Merged
merged 34 commits into from
Oct 15, 2024
Merged

[APM][Otel] Use fields instead of _source on APM queries #195242

merged 34 commits into from
Oct 15, 2024

Conversation

crespocarlos
Copy link
Contributor

@crespocarlos crespocarlos commented Oct 7, 2024 •
edited by kibanamachine
Loading

closes #192606

Summary

v2 based on the work done in this PR #192608 and the suggestion from Dario #194424

This PR replaces the _source usage in APM queries with fields to support Otel data. The idea is to get rid of existing UI errors we have and make sure that otel data is shown correctly in the UI.

One way to check it is using the e2e PoC.

@reakaleek
Copy link
Member

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@crespocarlos crespocarlos added apm apm:opentelemetry APM UI - OTEL Work Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team labels Oct 7, 2024
rmyz
rmyz reviewed Oct 7, 2024
View reviewed changes
Copy link
Contributor

@rmyz rmyz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments about using _source instead of fields regarding errors, other than that LGTM, will be checking it locally today 🚀

...bility_solution/apm/server/routes/errors/get_error_groups/get_error_group_main_statistics.ts Show resolved Hide resolved
...bility_solution/apm/server/routes/errors/get_error_groups/get_error_group_main_statistics.ts Show resolved Hide resolved
...servability_solution/apm/server/routes/errors/get_error_groups/get_error_group_sample_ids.ts Show resolved Hide resolved
x-pack/plugins/observability_solution/apm/server/routes/services/get_service_metadata_icons.ts Outdated Show resolved Hide resolved
x-pack/plugins/observability_solution/apm/server/routes/traces/get_trace_items.ts Outdated Show resolved Hide resolved
x-pack/plugins/observability_solution/apm_data_access/server/utils/unflatten_known_fields.ts Show resolved Hide resolved
@jennypavlova jennypavlova mentioned this pull request Oct 7, 2024
Closed
@crespocarlos
Copy link
Contributor Author

/ci

This was referenced Oct 7, 2024
Closed
Closed
@kibana-ci
Copy link
Collaborator

kibana-ci commented Oct 7, 2024 •
edited
Loading

💔 Build Failed

Failed CI Steps

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

miloszmarcinkowski
miloszmarcinkowski approved these changes Oct 7, 2024
View reviewed changes
Copy link
Contributor

@miloszmarcinkowski miloszmarcinkowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! It was a great idea to take advantage of APMEventClient returning response type.

@AlexanderWert AlexanderWert linked an issue Oct 11, 2024 that may be closed by this pull request
[APM][Otel] PoC Otel data with APM UI: Replacing _source with fields #192606
Closed
@crespocarlos
Copy link
Contributor Author

@elasticmachine merge upstream

@crespocarlos
Copy link
Contributor Author

@elasticmachine merge upstream

elasticmachine and others added 3 commits October 14, 2024 11:28
@elasticmachine
Merge branch 'main' into 192606-move-source-to-fields
3bfc69d
@jennypavlova
Fix missing exception error in error details
e8dce71
@crespocarlos
Merge pull request #5 from jennypavlova/196113-apmotelerrors-without-…
cba684f 
…an-error-type-result-in-error-details-page-crashing

[APM][Otel] Errors without an error type result in error details page crashing
@jennypavlova jennypavlova mentioned this pull request Oct 14, 2024
Closed
@jennypavlova jennypavlova mentioned this pull request Oct 14, 2024
Closed
@elasticmachine
Copy link
Contributor

elasticmachine commented Oct 15, 2024 •
edited
Loading

💚 Build Succeeded

  • Buildkite Build
  • Commit: db19165
  • Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-195242-db191655d6c9

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
apm 1844 1880 +36

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/apm-types 316 333 +17

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
apm 3.4MB 3.4MB +224.0B

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id before after diff
apm 118 119 +1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
apm 36.4KB 38.0KB +1.5KB
Unknown metric groups

API count

id before after diff
@kbn/apm-types 317 334 +17

ESLint disabled line counts

id before after diff
@kbn/observability-utils 1 2 +1

Total ESLint disabled count

id before after diff
@kbn/observability-utils 1 2 +1

History

@crespocarlos crespocarlos changed the title [DO NOT MERGE][APM][Otel] Use fields instead of _source on APM queries [APM][Otel] Use fields instead of _source on APM queries Oct 15, 2024
@crespocarlos crespocarlos merged commit 7235ed0 into elastic:main Oct 15, 2024
51 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11343408262

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Oct 15, 2024
@crespocarlos
[APM][Otel] Use fields instead of _source on APM queries (elastic…
09f2c7c 
…#195242)

closes elastic#192606

## Summary

v2 based on the work done in this PR
elastic#192608 and the suggestion from
Dario elastic#194424

This PR replaces the _source usage in APM queries with fields to support
Otel data. The idea is to get rid of existing UI errors we have and make
sure that otel data is shown correctly in the UI.

One way to check it is using the [e2e
PoC](https://github.com/elastic/otel-apm-e2e-poc/blob/main/README.md).

---------

Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>
Co-authored-by: Jenny <[email protected]>
(cherry picked from commit 7235ed0)
@kibanamachine kibanamachine mentioned this pull request Oct 15, 2024
Merged
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
✅ 8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

@crespocarlos crespocarlos deleted the 192606-move-source-to-fields branch October 15, 2024 10:16
kibanamachine added a commit that referenced this pull request Oct 15, 2024
@kibanamachine @crespocarlos
[8.x] [APM][Otel] Use &#x60;fields&#x60; instead of &#x60;_source&#x6…
ba5a8fc 
…0; on APM queries (#195242) (#196265)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[APM][Otel] Use &#x60;fields&#x60; instead of &#x60;_source&#x60; on
APM queries (#195242)](#195242)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Carlos
Crespo","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-15T09:38:44Z","message":"[APM][Otel]
Use `fields` instead of `_source` on APM queries (#195242)\n\ncloses
https://github.com/elastic/kibana/issues/192606\r\n\r\n##
Summary\r\n\r\nv2 based on the work done in this
PR\r\nhttps://github.com//pull/192608 and the suggestion
from\r\nDario https://github.com/elastic/kibana/pull/194424\r\n\r\nThis
PR replaces the _source usage in APM queries with fields to
support\r\nOtel data. The idea is to get rid of existing UI errors we
have and make\r\nsure that otel data is shown correctly in the
UI.\r\n\r\nOne way to check it is using the
[e2e\r\nPoC](https://github.com/elastic/otel-apm-e2e-poc/blob/main/README.md).\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<[email protected]>\r\nCo-authored-by:
Elastic Machine
<[email protected]>\r\nCo-authored-by: Jenny
<[email protected]>","sha":"7235ed0425100bbf04ff157d0af7980875473c99","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","v9.0.0","apm","apm:opentelemetry","backport:prev-minor","ci:project-deploy-observability","Team:obs-ux-infra_services"],"title":"[APM][Otel]
Use `fields` instead of `_source` on APM
queries","number":195242,"url":"https://github.com/elastic/kibana/pull/195242","mergeCommit":{"message":"[APM][Otel]
Use `fields` instead of `_source` on APM queries (#195242)\n\ncloses
https://github.com/elastic/kibana/issues/192606\r\n\r\n##
Summary\r\n\r\nv2 based on the work done in this
PR\r\nhttps://github.com//pull/192608 and the suggestion
from\r\nDario https://github.com/elastic/kibana/pull/194424\r\n\r\nThis
PR replaces the _source usage in APM queries with fields to
support\r\nOtel data. The idea is to get rid of existing UI errors we
have and make\r\nsure that otel data is shown correctly in the
UI.\r\n\r\nOne way to check it is using the
[e2e\r\nPoC](https://github.com/elastic/otel-apm-e2e-poc/blob/main/README.md).\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<[email protected]>\r\nCo-authored-by:
Elastic Machine
<[email protected]>\r\nCo-authored-by: Jenny
<[email protected]>","sha":"7235ed0425100bbf04ff157d0af7980875473c99"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/195242","number":195242,"mergeCommit":{"message":"[APM][Otel]
Use `fields` instead of `_source` on APM queries (#195242)\n\ncloses
https://github.com/elastic/kibana/issues/192606\r\n\r\n##
Summary\r\n\r\nv2 based on the work done in this
PR\r\nhttps://github.com//pull/192608 and the suggestion
from\r\nDario https://github.com/elastic/kibana/pull/194424\r\n\r\nThis
PR replaces the _source usage in APM queries with fields to
support\r\nOtel data. The idea is to get rid of existing UI errors we
have and make\r\nsure that otel data is shown correctly in the
UI.\r\n\r\nOne way to check it is using the
[e2e\r\nPoC](https://github.com/elastic/otel-apm-e2e-poc/blob/main/README.md).\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<[email protected]>\r\nCo-authored-by:
Elastic Machine
<[email protected]>\r\nCo-authored-by: Jenny
<[email protected]>","sha":"7235ed0425100bbf04ff157d0af7980875473c99"}}]}]
BACKPORT-->

Co-authored-by: Carlos Crespo <[email protected]>
@jennypavlova jennypavlova mentioned this pull request Oct 30, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AlexanderWert AlexanderWert AlexanderWert left review comments

@dgieselaar dgieselaar dgieselaar left review comments

@miloszmarcinkowski miloszmarcinkowski miloszmarcinkowski approved these changes

@rmyz rmyz rmyz approved these changes

@jennypavlova jennypavlova jennypavlova approved these changes

@felixbarny felixbarny Awaiting requested review from felixbarny

@MiriamAparicio MiriamAparicio Awaiting requested review from MiriamAparicio

@bryce-b bryce-b Awaiting requested review from bryce-b

Assignees
No one assigned
Labels
apm:opentelemetry APM UI - OTEL Work apm backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) ci:project-deploy-observability Create an Observability project release_note:fix Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team v8.16.0 v9.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[APM][Otel] PoC Otel data with APM UI: Replacing _source with fields