Ingesting new jvm metrics in 8.9 and 8.10 breaks upgrade to 8.11 and stops ingestion

[carsonip]

Java agent v1.39.0+ sending new jvm metrics to apm-server 8.9 and 8.10 will create runtime fields, causing upgrade to 8.11.0 or 8.11.1 to silently fail and stop ingesting APM metrics.

APM server error logs after upgrade:

failed to index document in 'metrics-apm.internal-default' (fail_processor_exception): Document produced by APM Server v8.11.1, which is newer than the installed APM integration (v8.10.3-preview-1695284222). The APM integration must be upgraded.

Fleet error on integration package upgrade:

Failed installing package [apm] due to error: [ResponseError: mapper_parsing_exception
	Root causes:
		mapper_parsing_exception: Field [jvm.memory.non_heap.pool.committed] attempted to shadow a time_series_metric]

Related PRs:
#11303
#10968

This should be fixed by elastic/kibana#171712

[6fears7]

We experienced this same issue upgrading from 8.10.3 to 8.11.1.

In the logs, we noticed that there were a lot of error logs that stated we were in 8.10.3 and the integration needed to be 8.11+

{"log.level":"error","@timestamp":"2023-11-27T07:51:38.915-0500","log.origin":{"file.name":"[email protected]/appender.go","file.line":312},"message":"failed to index document in 'metrics-apm.internal-default' (fail_processor_exception): Document produced by APM Server v8.11.1, which is newer than the installed APM integration (v8.10.3-preview-1695284222). The APM integration must be upgraded.","service.name":"apm-server","ecs.version":"1.6.0"}

When attempting to upgrade the integration via Fleet UI, I received,
Field [jvm.memory.non_heap.pool.committed] attempted to shadow a time_series_metric].

I then attempted a reinstall, same issue. Then an uninstall, same issue. We have managed to stand up our APM servers as standalone without an APM integration via Fleet UI and can receive some data, but this is obviously a concern to us long-term.

[simitt]

Hi @6fears7 , sorry to hear you are running into this bug. A fix was merged already in elastic/kibana#171712 and is planned to be part of the next patch release. We can update here once the fix is released.

[sunilemanjee]

Our customer is experiencing the same issue. what is the work around?

[simitt]

@sunilemanjee 8.11.2 was released last week, upgrading to this version should resolve the problems.

[kruskall]

A PR was merged to mention the issue and the related fix in the known-issues section of the docs. I think we can close this.

[simitt]

Docs listing the known issues for 8.11