Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document possible collisions with '@custom ingest pipelines' #840

Merged
merged 4 commits into from
Jan 23, 2024
Merged

Document possible collisions with '@custom ingest pipelines' #840

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1acd2e3
Document possible collisions with '@custom ingest pipelines'
kilfoyle Jan 22, 2024
e791e23
Add breaking change
kilfoyle Jan 22, 2024
9a9077f
Update breaking change with link to investigation issue 175254
kilfoyle Jan 22, 2024
9af293d
Change PR link to issue
kilfoyle Jan 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion docs/en/ingest-management/data-streams.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,7 @@ Starting in version 8.4, all default ingest pipelines call a non-existent and no
If left uncreated, this pipeline has no effect on your data. However, if added to a data stream and customized,
this pipeline can be used for custom data processing, adding fields, sanitizing data, and more.

Staring in version 8.12, ingest pipelines can be configured to process events at various levels of customization.
Starting in version 8.12, ingest pipelines can be configured to process events at various levels of customization.

`global@custom`::
Apply processing to all events
Expand Down Expand Up @@ -268,6 +268,18 @@ PUT _ingest/pipeline/metrics-system.cpu@custom
Custom pipelines can directly contain processors or you can use the pipeline processor to call other pipelines that can be shared across multiple data streams or integrations.
These pipelines will persist across all version upgrades.

[[data-streams-pipelines-warning]]
[WARNING]
====
If you have a custom pipeline defined that matches the naming scheme used for any {fleet} custom ingest pipelines, this can produce unintended results. For example, if you have a pipeline named like one of the following:

* `global@custom`
* `traces@custom`
* `traces-apm@custom`
Comment on lines +276 to +278
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for including some examples. I'll work on getting a full set of collision candidates (if it even exceeds this) on elastic/kibana#175254


The pipeline may be unexpectedly called for other data streams in other integrations. To avoid this problem, avoid the naming schemes defined above when naming your custom pipelines.
====

See <<data-streams-pipeline-tutorial>> to get started.

[[data-streams-ilm-tutorial]]
Expand Down
18 changes: 18 additions & 0 deletions docs/en/ingest-management/release-notes/release-notes-8.12.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,23 @@ Review important information about {fleet-server} and {agent} for the 8.12.0 rel
{agent}::
* Update Go version to 1.20.12. {agent-pull}3885[#3885]

[discrete]
[[breaking-changes-8.12.0]]
=== Breaking changes

Breaking changes can prevent your application from optimal operation and
performance. Before you upgrade, review the breaking changes, then mitigate the
impact to your application.

[discrete]
[[breaking-170270]]
.Possible naming collisions with {fleet} custom ingest pipelines
[%collapsible]
====
*Details* +
Starting in this release, {fleet} <<data-streams-pipelines,ingest pipelines>> can be configured to process events at various levels of customization. If you have a custom pipeline already defined that matches the name of a {fleet} custom ingest pipeline, it may be unexpectedly called for other data streams in other integrations. For more information, refer to {kibana-issue}175254[#175254] in which the problem is being investigated, and also the {fleet} ingest pipelines documentation.
====

[discrete]
[[known-issues-8.12.0]]
=== Known issues
Expand Down Expand Up @@ -96,6 +113,7 @@ The 8.12.0 release Added the following new and notable features.
{fleet}::
* Add {agent} upgrade states and display each agent's progress through the upgrade process. See <<view-upgrade-status>> for details. ({kibana-pull}167539[#167539])
* Add support for preconfigured output secrets. ({kibana-pull}172041[#172041])
* Add support for pipelines to process events at various levels of customization. ({kibana-pull}170270[#170270])
* Add UI components to create and edit output secrets. ({kibana-pull}169429[#169429])
* Add support for remote ES output. ({kibana-pull}169252[#169252])
* Add the ability to specify secrets in outputs. ({kibana-pull}169221[#169221])
Expand Down