Skip to content

Commit

Permalink
Document possible collisions with '@Custom ingest pipelines' (#840) (#…
Browse files Browse the repository at this point in the history
…849)

* Document possible collisions with '@Custom ingest pipelines'

* Add breaking change

* Update breaking change with link to investigation issue 175254

* Change PR link to issue

(cherry picked from commit 8517570)

Co-authored-by: David Kilfoyle <[email protected]>
  • Loading branch information
mergify[bot] and kilfoyle authored Jan 23, 2024
1 parent 9cdca79 commit e3fb3fe
Show file tree
Hide file tree
Showing 2 changed files with 31 additions and 1 deletion.
14 changes: 13 additions & 1 deletion docs/en/ingest-management/data-streams.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ Starting in version 8.4, all default ingest pipelines call a non-existent and no
If left uncreated, this pipeline has no effect on your data. However, if added to a data stream and customized,
this pipeline can be used for custom data processing, adding fields, sanitizing data, and more.

Staring in version 8.12, ingest pipelines can be configured to process events at various levels of customization.
Starting in version 8.12, ingest pipelines can be configured to process events at various levels of customization.

`global@custom`::
Apply processing to all events
Expand Down Expand Up @@ -261,6 +261,18 @@ PUT _ingest/pipeline/metrics-system.cpu@custom
Custom pipelines can directly contain processors or you can use the pipeline processor to call other pipelines that can be shared across multiple data streams or integrations.
These pipelines will persist across all version upgrades.

[[data-streams-pipelines-warning]]
[WARNING]
====
If you have a custom pipeline defined that matches the naming scheme used for any {fleet} custom ingest pipelines, this can produce unintended results. For example, if you have a pipeline named like one of the following:
* `global@custom`
* `traces@custom`
* `traces-apm@custom`
The pipeline may be unexpectedly called for other data streams in other integrations. To avoid this problem, avoid the naming schemes defined above when naming your custom pipelines.
====

See <<data-streams-pipeline-tutorial>> to get started.

[[data-streams-ilm-tutorial]]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,23 @@ Review important information about {fleet-server} and {agent} for the 8.12.0 rel
{agent}::
* Update Go version to 1.20.12. {agent-pull}3885[#3885]

[discrete]
[[breaking-changes-8.12.0]]
=== Breaking changes

Breaking changes can prevent your application from optimal operation and
performance. Before you upgrade, review the breaking changes, then mitigate the
impact to your application.

[discrete]
[[breaking-170270]]
.Possible naming collisions with {fleet} custom ingest pipelines
[%collapsible]
====
*Details* +
Starting in this release, {fleet} <<data-streams-pipelines,ingest pipelines>> can be configured to process events at various levels of customization. If you have a custom pipeline already defined that matches the name of a {fleet} custom ingest pipeline, it may be unexpectedly called for other data streams in other integrations. For more information, refer to {kibana-issue}175254[#175254] in which the problem is being investigated, and also the {fleet} ingest pipelines documentation.
====

[discrete]
[[known-issues-8.12.0]]
=== Known issues
Expand Down Expand Up @@ -96,6 +113,7 @@ The 8.12.0 release Added the following new and notable features.
{fleet}::
* Add {agent} upgrade states and display each agent's progress through the upgrade process. See <<view-upgrade-status>> for details. ({kibana-pull}167539[#167539])
* Add support for preconfigured output secrets. ({kibana-pull}172041[#172041])
* Add support for pipelines to process events at various levels of customization. ({kibana-pull}170270[#170270])
* Add UI components to create and edit output secrets. ({kibana-pull}169429[#169429])
* Add support for remote ES output. ({kibana-pull}169252[#169252])
* Add the ability to specify secrets in outputs. ({kibana-pull}169221[#169221])
Expand Down

0 comments on commit e3fb3fe

Please sign in to comment.