[REQUEST]: [Fleet] [APM] Document 8.12.0 breaking change in APM ingest pipelines in 8.12.0 release notes

[kpollich]

Description

In 8.12.0, there's a breaking change with APM data streams where the package name apm overlaps with the dataset name, e.g. the traces-apm data stream defined here.

As of 8.12.0, Fleet adds pipeline processors to all its managed ingest pipelines that will incur a collision in this case. This results in the traces-apm@custom pipeline being called by a preexisting traces-apm.sampled@custom (predates 8.12.0, this was the only extension point that existed previously) ingest pipeline which will be unexpected and potentially detrimental based on a given user's ingest workflow.

We need to retroactively update the 8.12.0 release notes to mention this.

Relates to #838

Resources

elastic/kibana#175254

Collaboration

(I forgot to select an option)

Docs team to provide initial content with support from stakeholder team.

Point of contact.

Main contact: @elastic/fleet @kpollich @nchaulet @simitt

Stakeholders: @elastic/fleet

[kpollich]

@kilfoyle - Should I close this in favor of just doing the release notes update + other docs work in #838, or would you prefer to have separate issues?

[kilfoyle]

@kpollich This issue is so much more clear than the one I opened, so let's keep this one and I'll close 838. Thanks a lot for providing the details.

[simitt]

nit: this also impacts the traces-apm.rum* data stream, as it will also get the traces-apm pipeline applied with the changed logic.

[kpollich]

@simitt - Thank you. I'd like to spend some time today trying to grep the integrations repo for other cases where we'll see this overlap in any packages.

[kpollich]

@kilfoyle - we've finished identifying the full scope of this change and I've documented a manual workaround below. The workaround is not a permanent fix, but should allow users to disable the unwanted pipeline behavior until a fix is shipped in 8.12.1:

Affected ingest pipelines

APM

• traces-apm
• traces-apm.rum
• traces-apm.sampled

For APM, if you had previously defined an ingest pipeline of the form traces-apm@custom to customize the ingestion of documents ingested to the traces-apm data stream (see docs), then by nature of the new @custom hooks introduced in elastic/kibana#168019, the traces-apm@custom pipeline will be called as a pipeline processor in both the traces-apm.rum and traces-apm.sampled ingest pipelines. See below for a comparison of the relevant processors blocks for each of these pipeline before and after upgrading to 8.12.0:

// traces-apm-8.x.x
{
	"pipeline": {
		"name": "traces-apm@custom",
		"ignore_missing_pipeline": true
	}
}

// traces-apm-8.12.0
{
	"pipeline": {
		"name": "global@custom",
		"ignore_missing_pipeline": true
	}
},
{
	"pipeline": {
		"name": "traces@custom",
		"ignore_missing_pipeline": true
	}
},
{
	"pipeline": {
		"name": "traces-apm@custom",
		"ignore_missing_pipeline": true
	}
},
{
	"pipeline": {
		"name": "traces-apm@custom", <--- Duplicate pipeline entry
		"ignore_missing_pipeline": true
	}
}

// traces-apm.rum-8.x.x
{
	"pipeline": {
		"name": "traces-apm.rum@custom",
		"ignore_missing_pipeline": true
	}
}

// traces-apm.rum-8.12.0
{
	"pipeline": {
		"name": "global@custom",
		"ignore_missing_pipeline": true
	}
},
{
	"pipeline": {
		"name": "traces@custom",
		"ignore_missing_pipeline": true
	}
},
{
	"pipeline": {
		"name": "traces-apm@custom", <--- Collides with `traces-apm@custom` that may be preexisting
		"ignore_missing_pipeline": true
	}
},
{
	"pipeline": {
		"name": "traces-apm.rum@custom",
		"ignore_missing_pipeline": true
	}
}

// traces-apm.sampled-8.x.x
{
	"pipeline": {
		"name": "traces-apm.rum@custom",
		"ignore_missing_pipeline": true
	}
}

// traces-apm.sampled-8.12.0
{
	"pipeline": {
		"name": "global@custom",
		"ignore_missing_pipeline": true
	}
},
{
	"pipeline": {
		"name": "traces@custom",
		"ignore_missing_pipeline": true
	}
},
{
	"pipeline": {
		"name": "traces-apm@custom", <--- Collides with `traces-apm@custom` that may be preexisting
		"ignore_missing_pipeline": true
	}
},
{
	"pipeline": {
		"name": "traces-apm.sampled@custom",
		"ignore_missing_pipeline": true
	}
}

The immediate workaround to avoid this unwanted behavior is to edit both the traces-apm.rum and traces-apm.sampled ingest pipelines to no longer include the traces-apm@custom pipeline processor. Please note this is a temporary workaround, and this change will be undone if the APM integration is upgraded or reinstalled.

Elastic Agent

The elastic_agent integration is subject to the same type of breaking change as described in APM above. The following ingest pipelines are impacted:

• logs-elastic_agent
• logs-elastic_agent.apm_server
• logs-elastic_agent.auditbeat
• logs-elastic_agent.cloud_defend
• logs-elastic_agent.cloudbeat
• logs-elastic_agent.endpoint_security
• logs-elastic_agent.filebeat
• logs-elastic_agent.filebeat_input
• logs-elastic_agent.fleet_server
• logs-elastic_agent.heartbeat
• logs-elastic_agent.metricbeat
• logs-elastic_agent.osquerybeat
• logs-elastic_agent.packetbeat
• logs-elastic_agent.pf_elastic_collector
• logs-elastic_agent.pf_elastic_symbolizer
• logs-elastic_agent.pf_host_agent

The behavior is similar to what's described for APM above: pipelines like logs-elastic_agent.filebeat will include a pipeline processor that calls logs-elastic_agent@custom. If you have custom processing logic defined in a logs-elastic_agent@custom ingest pipeline, it will be called by all of the pipelines listed above.

The workaround is the same: remove the logs-elastic_agent@custom pipeline processor from all of the ingest pipelines listed above.

---

If we want, we can add this in its entirety to the docs or associated KB article (cc @lucabelluccini), or we can update elastic/kibana#175254 to include the workaround steps directly as it's linked in the docs now.

[kilfoyle]

Thanks @kpollich! I can add this exactly as written into the breaking change, just to ensure people see it. I'll get a PR opened up shortly.

[lucabelluccini]

Thank you @kpollich & @kilfoyle for the thorough investigation.