From ad67da80ed1a517828940c69e74a78fa6b559e4a Mon Sep 17 00:00:00 2001 From: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> Date: Tue, 23 Jan 2024 11:29:05 -0500 Subject: [PATCH] Document possible collisions with '@custom ingest pipelines' (#840) * Document possible collisions with '@custom ingest pipelines' * Add breaking change * Update breaking change with link to investigation issue 175254 * Change PR link to issue (cherry picked from commit 851757041e3a6c69bc0ef454c88c5b6d2dac19e4) --- .../en/ingest-management/data-streams.asciidoc | 14 +++++++++++++- .../release-notes/release-notes-8.12.asciidoc | 18 ++++++++++++++++++ 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/docs/en/ingest-management/data-streams.asciidoc b/docs/en/ingest-management/data-streams.asciidoc index aa9630310..28e7cf913 100644 --- a/docs/en/ingest-management/data-streams.asciidoc +++ b/docs/en/ingest-management/data-streams.asciidoc @@ -172,7 +172,7 @@ Starting in version 8.4, all default ingest pipelines call a non-existent and no If left uncreated, this pipeline has no effect on your data. However, if added to a data stream and customized, this pipeline can be used for custom data processing, adding fields, sanitizing data, and more. -Staring in version 8.12, ingest pipelines can be configured to process events at various levels of customization. +Starting in version 8.12, ingest pipelines can be configured to process events at various levels of customization. `global@custom`:: Apply processing to all events @@ -261,6 +261,18 @@ PUT _ingest/pipeline/metrics-system.cpu@custom Custom pipelines can directly contain processors or you can use the pipeline processor to call other pipelines that can be shared across multiple data streams or integrations. These pipelines will persist across all version upgrades. +[[data-streams-pipelines-warning]] +[WARNING] +==== +If you have a custom pipeline defined that matches the naming scheme used for any {fleet} custom ingest pipelines, this can produce unintended results. For example, if you have a pipeline named like one of the following: + +* `global@custom` +* `traces@custom` +* `traces-apm@custom` + +The pipeline may be unexpectedly called for other data streams in other integrations. To avoid this problem, avoid the naming schemes defined above when naming your custom pipelines. +==== + See <> to get started. [[data-streams-ilm-tutorial]] diff --git a/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc b/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc index 797426efa..8997d5b7f 100644 --- a/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc +++ b/docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc @@ -35,6 +35,23 @@ Review important information about {fleet-server} and {agent} for the 8.12.0 rel {agent}:: * Update Go version to 1.20.12. {agent-pull}3885[#3885] +[discrete] +[[breaking-changes-8.12.0]] +=== Breaking changes + +Breaking changes can prevent your application from optimal operation and +performance. Before you upgrade, review the breaking changes, then mitigate the +impact to your application. + +[discrete] +[[breaking-170270]] +.Possible naming collisions with {fleet} custom ingest pipelines +[%collapsible] +==== +*Details* + +Starting in this release, {fleet} <> can be configured to process events at various levels of customization. If you have a custom pipeline already defined that matches the name of a {fleet} custom ingest pipeline, it may be unexpectedly called for other data streams in other integrations. For more information, refer to {kibana-issue}175254[#175254] in which the problem is being investigated, and also the {fleet} ingest pipelines documentation. +==== + [discrete] [[known-issues-8.12.0]] === Known issues @@ -96,6 +113,7 @@ The 8.12.0 release Added the following new and notable features. {fleet}:: * Add {agent} upgrade states and display each agent's progress through the upgrade process. See <> for details. ({kibana-pull}167539[#167539]) * Add support for preconfigured output secrets. ({kibana-pull}172041[#172041]) +* Add support for pipelines to process events at various levels of customization. ({kibana-pull}170270[#170270]) * Add UI components to create and edit output secrets. ({kibana-pull}169429[#169429]) * Add support for remote ES output. ({kibana-pull}169252[#169252]) * Add the ability to specify secrets in outputs. ({kibana-pull}169221[#169221])