Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[8.12] [Fleet] Update Fleet's custom ingest pipeline names to av…
…oid collisions + add descriptions to each pipeline (#175448) (#175547) # Backport This will backport the following commits from `main` to `8.12`: - [[Fleet] Update Fleet's custom ingest pipeline names to avoid collisions + add descriptions to each pipeline (#175448)](#175448) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Kyle Pollich","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-01-25T14:07:43Z","message":"[Fleet] Update Fleet's custom ingest pipeline names to avoid collisions + add descriptions to each pipeline (#175448)\n\n## Summary\r\n\r\nCloses https://github.com/elastic/kibana/issues/175254\r\nRef https://github.com/elastic/kibana/issues/168019\r\nRef https://github.com/elastic/kibana/pull/170270\r\n\r\nIn 8.12.0, Fleet unintentionally shipped a breaking change in\r\nhttps://github.com//pull/170270 for APM users who make use\r\nof a custom `traces-apm` data stream. If a user had previously defined\r\nthis ingest pipeline to customize documents ingested for the\r\n`traces-apm` data stream (defined\r\n[here](https://github.com/elastic/integrations/blob/9a36183f0bd12e39a957d2f7bd65f3de4ee685b1/packages/apm/data_stream/traces/manifest.yml#L2-L3),\r\nthen they would unexpectedly see that pipeline called when documents\r\nwere ingested to the `traces-apm.rum` and `traces-apm.sampled`\r\ndatastreams as well.\r\n\r\nThis PR addresses this collision by adding a `.package` suffix to the\r\n\"package level\" ingest pipeline introduced in 8.12.0.\r\n\r\nSo, in 8.12.0 a processor would be defined as such on the\r\n`traces-apm.rum` or `traces-apm.sampled` ingest pipeline\r\n\r\n```\r\n{\r\n \"pipeline\": {\r\n \"name\": \"traces-apm@custom\",\r\n \"ignore_missing_pipeline\": true,\r\n }\r\n},\r\n```\r\n\r\nThis PR replaces the pipeline with one that looks as follows:\r\n\r\n```\r\n{\r\n \"pipeline\": {\r\n \"name\": \"traces-apm.package@custom\",\r\n \"ignore_missing_pipeline\": true,\r\n \"description\": \"[Fleet] Pipeline for all data streams of type `traces` defined by the `apm` integration\"\r\n }\r\n},\r\n```\r\n\r\n**To be clear: this is a breaking change if you have defined the\r\n`traces-apm@custom` integration on 8.12. In 8.12.1, it will no longer be\r\ncalled for documents ingested to the `traces-apm`, `traces-apm.rum`, or\r\n`traces-apm.sampled` data streams. You will need to rename your pipeline\r\nto `traces-apm.package@custom` to preserve this behavior.**\r\n\r\nThis change also applies to `logs-elastic_agent.*` ingest pipelines. See\r\n[this\r\ncomment](https://github.com/elastic/kibana/issues/175254#issuecomment-1906202137)\r\nfor more information.\r\n\r\nThere is still technically room for a collision, though it's unlikely,\r\nif the data stream name is `package`. This will be handled by a package\r\nspec validation proposed in\r\nhttps://github.com/elastic/package-spec/issues/699.\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>","sha":"9fe5a66faf4e06fc444c6078edafc29e91126f8d","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:breaking","Team:Fleet","backport:prev-minor","v8.12.1","v8.13.0"],"title":"[Fleet] Update Fleet's custom ingest pipeline names to avoid collisions + add descriptions to each pipeline","number":175448,"url":"https://github.com/elastic/kibana/pull/175448","mergeCommit":{"message":"[Fleet] Update Fleet's custom ingest pipeline names to avoid collisions + add descriptions to each pipeline (#175448)\n\n## Summary\r\n\r\nCloses https://github.com/elastic/kibana/issues/175254\r\nRef https://github.com/elastic/kibana/issues/168019\r\nRef https://github.com/elastic/kibana/pull/170270\r\n\r\nIn 8.12.0, Fleet unintentionally shipped a breaking change in\r\nhttps://github.com//pull/170270 for APM users who make use\r\nof a custom `traces-apm` data stream. If a user had previously defined\r\nthis ingest pipeline to customize documents ingested for the\r\n`traces-apm` data stream (defined\r\n[here](https://github.com/elastic/integrations/blob/9a36183f0bd12e39a957d2f7bd65f3de4ee685b1/packages/apm/data_stream/traces/manifest.yml#L2-L3),\r\nthen they would unexpectedly see that pipeline called when documents\r\nwere ingested to the `traces-apm.rum` and `traces-apm.sampled`\r\ndatastreams as well.\r\n\r\nThis PR addresses this collision by adding a `.package` suffix to the\r\n\"package level\" ingest pipeline introduced in 8.12.0.\r\n\r\nSo, in 8.12.0 a processor would be defined as such on the\r\n`traces-apm.rum` or `traces-apm.sampled` ingest pipeline\r\n\r\n```\r\n{\r\n \"pipeline\": {\r\n \"name\": \"traces-apm@custom\",\r\n \"ignore_missing_pipeline\": true,\r\n }\r\n},\r\n```\r\n\r\nThis PR replaces the pipeline with one that looks as follows:\r\n\r\n```\r\n{\r\n \"pipeline\": {\r\n \"name\": \"traces-apm.package@custom\",\r\n \"ignore_missing_pipeline\": true,\r\n \"description\": \"[Fleet] Pipeline for all data streams of type `traces` defined by the `apm` integration\"\r\n }\r\n},\r\n```\r\n\r\n**To be clear: this is a breaking change if you have defined the\r\n`traces-apm@custom` integration on 8.12. In 8.12.1, it will no longer be\r\ncalled for documents ingested to the `traces-apm`, `traces-apm.rum`, or\r\n`traces-apm.sampled` data streams. You will need to rename your pipeline\r\nto `traces-apm.package@custom` to preserve this behavior.**\r\n\r\nThis change also applies to `logs-elastic_agent.*` ingest pipelines. See\r\n[this\r\ncomment](https://github.com/elastic/kibana/issues/175254#issuecomment-1906202137)\r\nfor more information.\r\n\r\nThere is still technically room for a collision, though it's unlikely,\r\nif the data stream name is `package`. This will be handled by a package\r\nspec validation proposed in\r\nhttps://github.com/elastic/package-spec/issues/699.\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>","sha":"9fe5a66faf4e06fc444c6078edafc29e91126f8d"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","branchLabelMappingKey":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/175448","number":175448,"mergeCommit":{"message":"[Fleet] Update Fleet's custom ingest pipeline names to avoid collisions + add descriptions to each pipeline (#175448)\n\n## Summary\r\n\r\nCloses https://github.com/elastic/kibana/issues/175254\r\nRef https://github.com/elastic/kibana/issues/168019\r\nRef https://github.com/elastic/kibana/pull/170270\r\n\r\nIn 8.12.0, Fleet unintentionally shipped a breaking change in\r\nhttps://github.com//pull/170270 for APM users who make use\r\nof a custom `traces-apm` data stream. If a user had previously defined\r\nthis ingest pipeline to customize documents ingested for the\r\n`traces-apm` data stream (defined\r\n[here](https://github.com/elastic/integrations/blob/9a36183f0bd12e39a957d2f7bd65f3de4ee685b1/packages/apm/data_stream/traces/manifest.yml#L2-L3),\r\nthen they would unexpectedly see that pipeline called when documents\r\nwere ingested to the `traces-apm.rum` and `traces-apm.sampled`\r\ndatastreams as well.\r\n\r\nThis PR addresses this collision by adding a `.package` suffix to the\r\n\"package level\" ingest pipeline introduced in 8.12.0.\r\n\r\nSo, in 8.12.0 a processor would be defined as such on the\r\n`traces-apm.rum` or `traces-apm.sampled` ingest pipeline\r\n\r\n```\r\n{\r\n \"pipeline\": {\r\n \"name\": \"traces-apm@custom\",\r\n \"ignore_missing_pipeline\": true,\r\n }\r\n},\r\n```\r\n\r\nThis PR replaces the pipeline with one that looks as follows:\r\n\r\n```\r\n{\r\n \"pipeline\": {\r\n \"name\": \"traces-apm.package@custom\",\r\n \"ignore_missing_pipeline\": true,\r\n \"description\": \"[Fleet] Pipeline for all data streams of type `traces` defined by the `apm` integration\"\r\n }\r\n},\r\n```\r\n\r\n**To be clear: this is a breaking change if you have defined the\r\n`traces-apm@custom` integration on 8.12. In 8.12.1, it will no longer be\r\ncalled for documents ingested to the `traces-apm`, `traces-apm.rum`, or\r\n`traces-apm.sampled` data streams. You will need to rename your pipeline\r\nto `traces-apm.package@custom` to preserve this behavior.**\r\n\r\nThis change also applies to `logs-elastic_agent.*` ingest pipelines. See\r\n[this\r\ncomment](https://github.com/elastic/kibana/issues/175254#issuecomment-1906202137)\r\nfor more information.\r\n\r\nThere is still technically room for a collision, though it's unlikely,\r\nif the data stream name is `package`. This will be handled by a package\r\nspec validation proposed in\r\nhttps://github.com/elastic/package-spec/issues/699.\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>","sha":"9fe5a66faf4e06fc444c6078edafc29e91126f8d"}}]}] BACKPORT--> Co-authored-by: Kyle Pollich <[email protected]>
- Loading branch information
1 parent
b281d4c
commit 5d4d356