[Fleet] Create intermediate objects when using dynamic mappings #169981
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🤖 GitHub commentsExpand to view the GitHub comments
Just comment with:
|
jsoriano
commented
Oct 26, 2023
| labels: { | ||
| type: "object", | ||
| dynamic: true, | ||
| }, |
There was a problem hiding this comment.
Example of intermediary object that is created as static property for dynamic templates.
jsoriano
commented
Oct 26, 2023
| dynamic: true, | ||
| }, | ||
| }, | ||
| }, |
There was a problem hiding this comment.
Example of intermediary objects created using dynamic mappings, as they contain dynamic parts.
jsoriano
commented
Oct 26, 2023
...plugins/fleet/server/services/epm/elasticsearch/template/__snapshots__/template.test.ts.snap
Show resolved
Hide resolved
…-ref HEAD~1..HEAD --fix'
…-ref HEAD~1..HEAD --fix'
…ppings' into fleet-create-object-dynamic-mappings
jsoriano
changed the title
|
@elasticmachine merge upstream |
botelastic
bot
added
the
Team:Fleet
|
Pinging @elastic/fleet (Team:Fleet) |
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]
History
To update your PR or re-run it, just comment with: cc @jsoriano |
kyungeunni
approved these changes
Nov 6, 2023
There was a problem hiding this comment.
I can confirm that intermediate objects are generated in the mappings so thus apmpackage can be installed with the correct index templates thanks to this change.
Thank you so much for fixing this, @jsoriano !
8 tasks
kibanamachine
added
v8.12.0
backport:skip
1 task
jsoriano
added a commit
that referenced
this pull request
Jan 31, 2024
…jects (#175970) When there are multiple dynamic templates with the same name, we only use the first appearance. Add an exception for intermediate objects, so we use the last appearance. This way we avoid that less specific dynamic templates created for intermediate objects match before more specific dynamic templates defined in packages. This was a problem for example in the Prometheus package, where there are definitions for `prometheus.*.value` and `prometheus.*.histogram`. For both cases Fleet tries to create a dynamic template for the intermediate objects `prometheus.*`. If this dynamic template is inserted before the dynamic template for the histograms, it matches for them, causing mapping errors on ingestion, like the following one: ``` mapper [prometheus.queue_duration.histogram.values] cannot be changed from type [float] to [long] ``` With this change, when multiple definitions generate the same dynamic template for intermediate objects, only the last one ends up in the template. These intermediate objects were introduced in 8.12, after #169981.
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jan 31, 2024
…jects (elastic#175970) When there are multiple dynamic templates with the same name, we only use the first appearance. Add an exception for intermediate objects, so we use the last appearance. This way we avoid that less specific dynamic templates created for intermediate objects match before more specific dynamic templates defined in packages. This was a problem for example in the Prometheus package, where there are definitions for `prometheus.*.value` and `prometheus.*.histogram`. For both cases Fleet tries to create a dynamic template for the intermediate objects `prometheus.*`. If this dynamic template is inserted before the dynamic template for the histograms, it matches for them, causing mapping errors on ingestion, like the following one: ``` mapper [prometheus.queue_duration.histogram.values] cannot be changed from type [float] to [long] ``` With this change, when multiple definitions generate the same dynamic template for intermediate objects, only the last one ends up in the template. These intermediate objects were introduced in 8.12, after elastic#169981. (cherry picked from commit 7274f6e)
kibanamachine
referenced
this pull request
Jan 31, 2024
…iate objects (#175970) (#175991) # Backport This will backport the following commits from `main` to `8.12`: - [[Fleet] Fix conflicting dynamic template mappings for intermediate objects (#175970)](#175970) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Jaime Soriano Pastor","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-01-31T14:51:39Z","message":"[Fleet] Fix conflicting dynamic template mappings for intermediate objects (#175970)\n\nWhen there are multiple dynamic templates with the same name, we only\r\nuse the first appearance.\r\nAdd an exception for intermediate objects, so we use the last\r\nappearance. This way we avoid that less specific dynamic templates\r\ncreated for intermediate objects match before more specific dynamic\r\ntemplates defined in packages.\r\n\r\nThis was a problem for example in the Prometheus package, where there\r\nare definitions for `prometheus.*.value` and `prometheus.*.histogram`.\r\nFor both cases Fleet tries to create a dynamic template for the\r\nintermediate objects `prometheus.*`. If this dynamic template is\r\ninserted before the dynamic template for the histograms, it matches for\r\nthem, causing mapping errors on ingestion, like the following one:\r\n```\r\nmapper [prometheus.queue_duration.histogram.values] cannot be changed from type [float] to [long]\r\n```\r\nWith this change, when multiple definitions generate the same dynamic\r\ntemplate for intermediate objects, only the last one ends up in the\r\ntemplate.\r\n\r\nThese intermediate objects were introduced in 8.12, after https://github.com/elastic/kibana/pull/169981.","sha":"7274f6e2e152928418a677fcb6b9c3550b8a6227","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Fleet","backport:prev-minor","v8.12.1","v8.13.0"],"title":"[Fleet] Fix conflicting dynamic template mappings for intermediate objects","number":175970,"url":"https://github.com/elastic/kibana/pull/175970","mergeCommit":{"message":"[Fleet] Fix conflicting dynamic template mappings for intermediate objects (#175970)\n\nWhen there are multiple dynamic templates with the same name, we only\r\nuse the first appearance.\r\nAdd an exception for intermediate objects, so we use the last\r\nappearance. This way we avoid that less specific dynamic templates\r\ncreated for intermediate objects match before more specific dynamic\r\ntemplates defined in packages.\r\n\r\nThis was a problem for example in the Prometheus package, where there\r\nare definitions for `prometheus.*.value` and `prometheus.*.histogram`.\r\nFor both cases Fleet tries to create a dynamic template for the\r\nintermediate objects `prometheus.*`. If this dynamic template is\r\ninserted before the dynamic template for the histograms, it matches for\r\nthem, causing mapping errors on ingestion, like the following one:\r\n```\r\nmapper [prometheus.queue_duration.histogram.values] cannot be changed from type [float] to [long]\r\n```\r\nWith this change, when multiple definitions generate the same dynamic\r\ntemplate for intermediate objects, only the last one ends up in the\r\ntemplate.\r\n\r\nThese intermediate objects were introduced in 8.12, after https://github.com/elastic/kibana/pull/169981.","sha":"7274f6e2e152928418a677fcb6b9c3550b8a6227"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","branchLabelMappingKey":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/175970","number":175970,"mergeCommit":{"message":"[Fleet] Fix conflicting dynamic template mappings for intermediate objects (#175970)\n\nWhen there are multiple dynamic templates with the same name, we only\r\nuse the first appearance.\r\nAdd an exception for intermediate objects, so we use the last\r\nappearance. This way we avoid that less specific dynamic templates\r\ncreated for intermediate objects match before more specific dynamic\r\ntemplates defined in packages.\r\n\r\nThis was a problem for example in the Prometheus package, where there\r\nare definitions for `prometheus.*.value` and `prometheus.*.histogram`.\r\nFor both cases Fleet tries to create a dynamic template for the\r\nintermediate objects `prometheus.*`. If this dynamic template is\r\ninserted before the dynamic template for the histograms, it matches for\r\nthem, causing mapping errors on ingestion, like the following one:\r\n```\r\nmapper [prometheus.queue_duration.histogram.values] cannot be changed from type [float] to [long]\r\n```\r\nWith this change, when multiple definitions generate the same dynamic\r\ntemplate for intermediate objects, only the last one ends up in the\r\ntemplate.\r\n\r\nThese intermediate objects were introduced in 8.12, after https://github.com/elastic/kibana/pull/169981.","sha":"7274f6e2e152928418a677fcb6b9c3550b8a6227"}}]}] BACKPORT--> Co-authored-by: Jaime Soriano Pastor <[email protected]>
CoenWarmer
pushed a commit
to CoenWarmer/kibana
that referenced
this pull request
Feb 15, 2024
…jects (elastic#175970) When there are multiple dynamic templates with the same name, we only use the first appearance. Add an exception for intermediate objects, so we use the last appearance. This way we avoid that less specific dynamic templates created for intermediate objects match before more specific dynamic templates defined in packages. This was a problem for example in the Prometheus package, where there are definitions for `prometheus.*.value` and `prometheus.*.histogram`. For both cases Fleet tries to create a dynamic template for the intermediate objects `prometheus.*`. If this dynamic template is inserted before the dynamic template for the histograms, it matches for them, causing mapping errors on ingestion, like the following one: ``` mapper [prometheus.queue_duration.histogram.values] cannot be changed from type [float] to [long] ``` With this change, when multiple definitions generate the same dynamic template for intermediate objects, only the last one ends up in the template. These intermediate objects were introduced in 8.12, after elastic#169981.
fkanout
pushed a commit
to fkanout/kibana
that referenced
this pull request
Mar 4, 2024
…jects (elastic#175970) When there are multiple dynamic templates with the same name, we only use the first appearance. Add an exception for intermediate objects, so we use the last appearance. This way we avoid that less specific dynamic templates created for intermediate objects match before more specific dynamic templates defined in packages. This was a problem for example in the Prometheus package, where there are definitions for `prometheus.*.value` and `prometheus.*.histogram`. For both cases Fleet tries to create a dynamic template for the intermediate objects `prometheus.*`. If this dynamic template is inserted before the dynamic template for the histograms, it matches for them, causing mapping errors on ingestion, like the following one: ``` mapper [prometheus.queue_duration.histogram.values] cannot be changed from type [float] to [long] ``` With this change, when multiple definitions generate the same dynamic template for intermediate objects, only the last one ends up in the template. These intermediate objects were introduced in 8.12, after elastic#169981.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
For
dynamic: "runtime", and possibly other configurations different todynamic: true, intermediate objects need to exist for dynamic mappings, otherwise ingestion can fail withMissing intermediate object ....This pull request includes these intermediate objects, and avoids the creation of static properties with wildcards, that is probably not what is expected for these mappings.
Intermediate objects are included as static properties for parts of the name before the wildcard, and as dynamic templates when the full path has wildcards.
In the modified test files are examples of any of these.
This is more an issue since the following recent fixes, that create the actual dynamic mappings for some cases where only empty objects were created before.
This attempts to unblock elastic/apm-server#11822.
Checklist
Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.
When forming the risk matrix, consider some of the following examples and how they may potentially impact the change: