From 6f46b2f486be528ed9ac75627c2c4cf210a3cbe7 Mon Sep 17 00:00:00 2001 From: "mergify[bot]" <37929162+mergify[bot]@users.noreply.github.com> Date: Fri, 1 Nov 2024 10:45:44 -0500 Subject: [PATCH] [8.16](backport #4459) APM: add known issue of lazy rollover bug (#4467) * APM: add known issue of lazy rollover bug (#4459) Signed-off-by: inge4pres Co-authored-by: simitt (cherry picked from commit 66a73ef17b50143e6f6f12bc430bc197e8905b46) * Clean up new APM known issue (#4463) (#4466) * clean up * further clarifications --------- Co-authored-by: Francesco Gualazzi (cherry picked from commit c7cb6c152c0705a83734c11563a9601eba5a29da) # Conflicts: # docs/en/observability/apm/known-issues.asciidoc Co-authored-by: Colleen McGinnis --------- Co-authored-by: Francesco Gualazzi Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Colleen McGinnis --- .../observability/apm/known-issues.asciidoc | 44 +++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index 3ed733cd2c..72f8990b39 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -21,6 +21,50 @@ _Versions: XX.XX.XX, YY.YY.YY, ZZ.ZZ.ZZ_ // If applicable, link to fix //// +[discrete] +== Upgrading to v8.15.x may cause ingestion to fail + +_Elastic Stack versions: 8.15.0+_ + +// The conditions in which this issue occurs +The issue only occurs when _upgrading_ the {stack} from 8.12.2 or lower directly to any 8.15.x version. +The issue does _not_ occur when creating a _new_ cluster using any 8.15.x version, or when upgrading +from 8.12.2 to 8.13.x or 8.14.x and then to 8.15.x. + +// Describe why it happens +In APM Servers versions prior to 8.13.0, an ingestion pipeline exists to perform a check on the version. +The version check would fail any APM document produced with a different version of APM server compared to the version of the installed APM’s ingest pipeline. +In 8.13.0 the version check in the ingest pipeline was removed. +Due to the combination of an internal change in how apm data management assets are set up from 8.15 onwards and a bug in Elasticsearch, +related to https://github.com/elastic/elasticsearch/issues/112781[lazy rollover of data streams], the ingestion pipeline conducting the version check is not removed on upgrade and prevents the ingestion of data. + +// How to fix it +If the deployment is running 8.15.0, upgrade the deployment to 8.15.1 or above. +A manual rollover of all APM data streams is required to pick up the new index templates and remove the faulty ingest pipeline version check. +Perform the following requests to Elasticsearch (they are assuming the `default` namespace is used, adjust if necessary): + +[source,txt] +---- +POST /traces-apm-default/_rollover +POST /traces-apm.rum-default/_rollover +POST /logs-apm.error-default/_rollover +POST /logs-apm.app-default/_rollover +POST /metrics-apm.app-default/_rollover +POST /metrics-apm.internal-default/_rollover +POST /metrics-apm.service_destination.1m-default/_rollover +POST /metrics-apm.service_destination.10m-default/_rollover +POST /metrics-apm.service_destination.60m-default/_rollover +POST /metrics-apm.service_summary.1m-default/_rollover +POST /metrics-apm.service_summary.10m-default/_rollover +POST /metrics-apm.service_summary.60m-default/_rollover +POST /metrics-apm.service_transaction.1m-default/_rollover +POST /metrics-apm.service_transaction.10m-default/_rollover +POST /metrics-apm.service_transaction.60m-default/_rollover +POST /metrics-apm.transaction.1m-default/_rollover +POST /metrics-apm.transaction.10m-default/_rollover +POST /metrics-apm.transaction.60m-default/_rollover +---- + [discrete] == Upgrading to v8.15.0 may cause APM indices to lose their lifecycle policy