From 1acd2e3cb33c0151ac9412d3bf3fcff297b584c6 Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Mon, 22 Jan 2024 13:12:55 -0500 Subject: [PATCH 1/4] Document possible collisions with '@custom ingest pipelines' --- docs/en/ingest-management/data-streams.asciidoc | 14 +++++++++++++- .../release-notes/release-notes-8.12.asciidoc | 1 + 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/docs/en/ingest-management/data-streams.asciidoc b/docs/en/ingest-management/data-streams.asciidoc index 5c9746d71..a05352705 100644 --- a/docs/en/ingest-management/data-streams.asciidoc +++ b/docs/en/ingest-management/data-streams.asciidoc @@ -179,7 +179,7 @@ Starting in version 8.4, all default ingest pipelines call a non-existent and no If left uncreated, this pipeline has no effect on your data. However, if added to a data stream and customized, this pipeline can be used for custom data processing, adding fields, sanitizing data, and more. -Staring in version 8.12, ingest pipelines can be configured to process events at various levels of customization. +Starting in version 8.12, ingest pipelines can be configured to process events at various levels of customization. `global@custom`:: Apply processing to all events @@ -268,6 +268,18 @@ PUT _ingest/pipeline/metrics-system.cpu@custom Custom pipelines can directly contain processors or you can use the pipeline processor to call other pipelines that can be shared across multiple data streams or integrations. These pipelines will persist across all version upgrades. +[[data-streams-pipelines-warning]] +[WARNING] +==== +If you have a custom pipeline defined that matches the naming scheme used for any {fleet} custom ingest pipelines, this can produce unintended results. For example, if you have a pipeline named like one of the following: + +* `global@custom` +* `traces@custom` +* `traces-apm@custom` + +The pipeline may be unexpectedly called for other data streams in other integrations. To avoid this problem, avoid the naming schemes defined above when naming your custom pipelines. +==== + See <> to get started. [[data-streams-ilm-tutorial]] diff --git a/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc b/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc index 797426efa..b129da662 100644 --- a/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc +++ b/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc @@ -96,6 +96,7 @@ The 8.12.0 release Added the following new and notable features. {fleet}:: * Add {agent} upgrade states and display each agent's progress through the upgrade process. See <> for details. ({kibana-pull}167539[#167539]) * Add support for preconfigured output secrets. ({kibana-pull}172041[#172041]) +* Add support for pipelines to process events at various levels of customization. ({kibana-pull}170270[#170270]) * Add UI components to create and edit output secrets. ({kibana-pull}169429[#169429]) * Add support for remote ES output. ({kibana-pull}169252[#169252]) * Add the ability to specify secrets in outputs. ({kibana-pull}169221[#169221]) From e791e2315759f1d71f990cc97baf3f279686600a Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Mon, 22 Jan 2024 13:29:49 -0500 Subject: [PATCH 2/4] Add breaking change --- .../release-notes/release-notes-8.12.asciidoc | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc b/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc index b129da662..fe1105200 100644 --- a/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc +++ b/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc @@ -35,6 +35,23 @@ Review important information about {fleet-server} and {agent} for the 8.12.0 rel {agent}:: * Update Go version to 1.20.12. {agent-pull}3885[#3885] +[discrete] +[[breaking-changes-8.12.0]] +=== Breaking changes + +Breaking changes can prevent your application from optimal operation and +performance. Before you upgrade, review the breaking changes, then mitigate the +impact to your application. + +[discrete] +[[breaking-170270]] +.Possible naming collisions with {fleet} custom ingest pipelines +[%collapsible] +==== +*Details* + +Starting in this release, {fleet} <> can be configured to process events at various levels of customization. If you have a custom pipeline already defined that matches the name of a {fleet} custom ingest pipeline, it may be unexpectedly called for other data streams in other integrations. For more information, refer to {kibana-pull}PR[#170270] and the {fleet} ingest pipelines documentation. +==== + [discrete] [[known-issues-8.12.0]] === Known issues From 9a9077f027b6b3865d96d0aab467d9581c311b62 Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Mon, 22 Jan 2024 14:26:46 -0500 Subject: [PATCH 3/4] Update breaking change with link to investigation issue 175254 --- .../ingest-management/release-notes/release-notes-8.12.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc b/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc index fe1105200..e97713a56 100644 --- a/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc +++ b/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc @@ -49,7 +49,7 @@ impact to your application. [%collapsible] ==== *Details* + -Starting in this release, {fleet} <> can be configured to process events at various levels of customization. If you have a custom pipeline already defined that matches the name of a {fleet} custom ingest pipeline, it may be unexpectedly called for other data streams in other integrations. For more information, refer to {kibana-pull}PR[#170270] and the {fleet} ingest pipelines documentation. +Starting in this release, {fleet} <> can be configured to process events at various levels of customization. If you have a custom pipeline already defined that matches the name of a {fleet} custom ingest pipeline, it may be unexpectedly called for other data streams in other integrations. For more information, refer to {kibana-pull}PR[#175254] in which the problem is being investigated, and also the {fleet} ingest pipelines documentation. ==== [discrete] From 9af293db2443df657bf751478d4cef58b18142e5 Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Tue, 23 Jan 2024 09:34:17 -0500 Subject: [PATCH 4/4] Change PR link to issue --- .../ingest-management/release-notes/release-notes-8.12.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc b/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc index e97713a56..8997d5b7f 100644 --- a/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc +++ b/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc @@ -49,7 +49,7 @@ impact to your application. [%collapsible] ==== *Details* + -Starting in this release, {fleet} <> can be configured to process events at various levels of customization. If you have a custom pipeline already defined that matches the name of a {fleet} custom ingest pipeline, it may be unexpectedly called for other data streams in other integrations. For more information, refer to {kibana-pull}PR[#175254] in which the problem is being investigated, and also the {fleet} ingest pipelines documentation. +Starting in this release, {fleet} <> can be configured to process events at various levels of customization. If you have a custom pipeline already defined that matches the name of a {fleet} custom ingest pipeline, it may be unexpectedly called for other data streams in other integrations. For more information, refer to {kibana-issue}175254[#175254] in which the problem is being investigated, and also the {fleet} ingest pipelines documentation. ==== [discrete]