[8.16](backport #4459) APM: add known issue of lazy rollover bug (#4467)

* APM: add known issue of lazy rollover bug (#4459)

Signed-off-by: inge4pres <[email protected]>
Co-authored-by: simitt <[email protected]>
(cherry picked from commit 66a73ef)

* Clean up new APM known issue (#4463) (#4466)

* clean up

* further clarifications

---------

Co-authored-by: Francesco Gualazzi <[email protected]>
(cherry picked from commit c7cb6c1)

# Conflicts:
#	docs/en/observability/apm/known-issues.asciidoc

Co-authored-by: Colleen McGinnis <[email protected]>

---------

Co-authored-by: Francesco Gualazzi <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Colleen McGinnis <[email protected]>

docs/en/observability/apm/known-issues.asciidoc

@@ -21,6 +21,50 @@ _Versions: XX.XX.XX, YY.YY.YY, ZZ.ZZ.ZZ_
 // If applicable, link to fix
 ////
 
+[discrete]
+== Upgrading to v8.15.x may cause ingestion to fail
+
+_Elastic Stack versions: 8.15.0+_
+
+// The conditions in which this issue occurs
+The issue only occurs when _upgrading_ the {stack} from 8.12.2 or lower directly to any 8.15.x version.
+The issue does _not_ occur when creating a _new_ cluster using any 8.15.x version, or when upgrading
+from 8.12.2 to 8.13.x or 8.14.x and then to 8.15.x.
+
+// Describe why it happens
+In APM Servers versions prior to 8.13.0, an ingestion pipeline exists to perform a check on the version.
+The version check would fail any APM document produced with a different version of APM server compared to the version of the installed APM’s ingest pipeline.
+In 8.13.0 the version check in the ingest pipeline was removed.
+Due to the combination of an internal change in how apm data management assets are set up from 8.15 onwards and a bug in Elasticsearch, 
+related to https://github.com/elastic/elasticsearch/issues/112781[lazy rollover of data streams], the ingestion pipeline conducting the version check is not removed on upgrade and prevents the ingestion of data.
+
+// How to fix it
+If the deployment is running 8.15.0, upgrade the deployment to 8.15.1 or above.
+A manual rollover of all APM data streams is required to pick up the new index templates and remove the faulty ingest pipeline version check.
+Perform the following requests to Elasticsearch (they are assuming the `default` namespace is used, adjust if necessary):
+
+[source,txt]
+----
+POST /traces-apm-default/_rollover
+POST /traces-apm.rum-default/_rollover
+POST /logs-apm.error-default/_rollover
+POST /logs-apm.app-default/_rollover
+POST /metrics-apm.app-default/_rollover
+POST /metrics-apm.internal-default/_rollover
+POST /metrics-apm.service_destination.1m-default/_rollover
+POST /metrics-apm.service_destination.10m-default/_rollover
+POST /metrics-apm.service_destination.60m-default/_rollover
+POST /metrics-apm.service_summary.1m-default/_rollover
+POST /metrics-apm.service_summary.10m-default/_rollover
+POST /metrics-apm.service_summary.60m-default/_rollover
+POST /metrics-apm.service_transaction.1m-default/_rollover
+POST /metrics-apm.service_transaction.10m-default/_rollover
+POST /metrics-apm.service_transaction.60m-default/_rollover
+POST /metrics-apm.transaction.1m-default/_rollover
+POST /metrics-apm.transaction.10m-default/_rollover
+POST /metrics-apm.transaction.60m-default/_rollover
+----
+
 [discrete]
 == Upgrading to v8.15.0 may cause APM indices to lose their lifecycle policy