From 4d980b108baef7ace0c8571144015b87502367a8 Mon Sep 17 00:00:00 2001 From: mdbirnstiehl Date: Tue, 3 Dec 2024 11:26:54 -0600 Subject: [PATCH 1/4] add mapping issues --- .../logs-troubleshooting.asciidoc | 52 ++++++++++++++----- 1 file changed, 39 insertions(+), 13 deletions(-) diff --git a/docs/en/observability/logs-troubleshooting.asciidoc b/docs/en/observability/logs-troubleshooting.asciidoc index 531be77c97..7a8d3fe215 100644 --- a/docs/en/observability/logs-troubleshooting.asciidoc +++ b/docs/en/observability/logs-troubleshooting.asciidoc @@ -1,11 +1,19 @@ [[logs-troubleshooting]] = Troubleshoot logs +Use this page to find possible solutions for errors your encountering with your logs. This troubleshooting page is broken into two sections: + +- <> +- <> + +[[logs-onboarding-troubleshooting]] +== Common onboarding issues + This section provides possible solutions for errors you might encounter while onboarding your logs. [discrete] [[logs-troubleshooting-insufficient-priv]] -== User does not have permissions to create API key +=== User does not have permissions to create API key If you don't have the required privileges to create an API key, you'll see the following error message: @@ -21,7 +29,7 @@ to the role of the authenticated user. [discrete] [[logs-troubleshooting-insufficient-priv-solution]] -=== Solution +==== Solution You need to either: @@ -30,7 +38,7 @@ You need to either: [discrete] [[logs-troubleshooting-API-key-failed]] -== Failed to create API key +=== Failed to create API key If you don't have the privileges to create `savedObjects` in {kib}, you'll see the following error message: @@ -43,14 +51,14 @@ Something went wrong: Unable to create observability-onboarding-state [discrete] [[logs-troubleshooting-API-key-failed-solution]] -=== Solution +==== Solution You need an administrator to give you the `Saved Objects Management` {kib} privilege to generate the required `observability-onboarding-state` flow state. Once you have the necessary privileges, restart the onboarding flow. [discrete] [[logs-troubleshooting-kib-not-accessible]] -== {kib} not accessible from host +=== {kib} not accessible from host If {kib} is not accessible from the host, you'll see the following error message after pasting the *Install the {agent}* instructions into the host: @@ -61,7 +69,7 @@ Failed to connect to {host} port {port} after 0 ms: Connection refused [discrete] [[logs-troubleshooting-kib-not-accessible-solution]] -=== Solution +==== Solution The host needs access to {kib}. Port `443` must be open and the deployment's {es} endpoint must be reachable. Locate your project's endpoint from *Help menu (image:images/help-icon.png[]) → Connection details*. @@ -73,7 +81,7 @@ curl https://your-endpoint.elastic.cloud [discrete] [[logs-troubleshooting-download-agent]] -== Download {agent} failed +=== Download {agent} failed If the host was able to download the installation script but cannot connect to the public artifact repository, you'll see the following error message: @@ -86,7 +94,7 @@ Failed to download Elastic Agent, see script for error. [discrete] [[logs-troubleshooting-download-agent-solution]] -=== Solutions +==== Solutions * If the combination of the {agent} version and operating system architecture is not available, you'll see the following error message: + @@ -108,7 +116,7 @@ To fix this, delete previous downloads and restart the onboarding. [discrete] [[logs-troubleshooting-install-agent]] -== Install {agent} failed +=== Install {agent} failed If an {agent} already exists on your host, you'll see the following error message: @@ -121,19 +129,37 @@ Failed to install Elastic Agent, see script for error. [discrete] [[logs-troubleshooting-install-agent-solution]] -=== Solution +==== Solution You can uninstall the current {agent} using the `elastic-agent uninstall` command, and run the script again. WARNING: Uninstalling the current {agent} removes the entire current setup, including the existing configuration. [discrete] [[logs-troubleshooting-wait-for-logs]] -== Waiting for Logs to be shipped... step never completes +=== Waiting for Logs to be shipped... step never completes If the *Waiting for Logs to be shipped...* step never completes, logs are not being shipped to {es}, and there is most likely an issue with your {agent} configuration. [discrete] [[logs-troubleshooting-wait-for-logs-solution]] -=== Solution +==== Solution + +Inspect the {agent} logs for errors. See the {fleet-guide}/debug-standalone-agents.html#inspect-standalone-agent-logs[Debug standalone {agent}s] documentation for more on finding errors in {agent} logs. + +[[logs-common-mapping-troubleshooting]] +== Mapping and pipeline issues + +This section provides possible solutions for errors you might encounter while onboarding your logs. + +[[logs-mapping-troubleshooting-keyword]] +=== Keyword fields are too long + + +[[logs-mapping-troubleshooting-format-mismatch]] +=== Date format mismatch + +[[logs-mapping-troubleshooting-grok-mismatch]] +=== Grok or dissect pattern mismatch -Inspect the {agent} logs for errors. See the {fleet-guide}/debug-standalone-agents.html#inspect-standalone-agent-logs[Debug standalone {agent}s] documentation for more on finding errors in {agent} logs. \ No newline at end of file +[[logs-mapping-troubleshooting-string-mismatch]] +=== String/number or Float/int mismatch \ No newline at end of file From 93479041dab1d41904d56bb32eb6aa94a8e429d8 Mon Sep 17 00:00:00 2001 From: mdbirnstiehl Date: Thu, 26 Dec 2024 13:50:54 -0600 Subject: [PATCH 2/4] add keyword mitigations --- .../logs-troubleshooting.asciidoc | 29 +++++++++++++++++-- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/docs/en/observability/logs-troubleshooting.asciidoc b/docs/en/observability/logs-troubleshooting.asciidoc index 7a8d3fe215..e61ae4bdd6 100644 --- a/docs/en/observability/logs-troubleshooting.asciidoc +++ b/docs/en/observability/logs-troubleshooting.asciidoc @@ -1,7 +1,7 @@ [[logs-troubleshooting]] = Troubleshoot logs -Use this page to find possible solutions for errors your encountering with your logs. This troubleshooting page is broken into two sections: +Use this page to find possible solutions for errors your encountering with your logs. This troubleshooting page is broken into the following sections: - <> - <> @@ -149,11 +149,34 @@ Inspect the {agent} logs for errors. See the {fleet-guide}/debug-standalone-agen [[logs-common-mapping-troubleshooting]] == Mapping and pipeline issues -This section provides possible solutions for errors you might encounter while onboarding your logs. +This section provides possible solutions for mapping and pipeline issues you might encounter with your logs. -[[logs-mapping-troubleshooting-keyword]] +[[logs-mapping-troubleshooting-keyword-limit]] === Keyword fields are too long +The `keyword` field limit is 32,766 bytes. When indexing a document, if your `keyword` field length exceeds this limit, you'll see an error similar to the following: + +``` +max_bytes_length_exceeded_exception: bytes can be at most 32766 in length +``` + +[discrete] +[[logs-mapping-troubleshooting-keyword-limit-solution]] +==== Solution +Avoid this error using one of the following options: + +*Stop indexing the field:* If you don't need the `keyword` field for aggregation or search, set `"index":false` in the index template to stop indexing the field. + +*Increase `ignore_above`:* To index any `keyword` fields under the field limit, but not those over the limit, set the `ignore_above` setting in the index template. + +NOTE: The value for `ignore_above` is the _character count_, but Lucene counts bytes. +If you use UTF-8 text with many non-ASCII characters, you may want to set the limit to `32766 / 4 = 8191` since UTF-8 characters may occupy at most 4 bytes. + +Refer to the {ref}/ignore-above.html[`ignore_above`] docs for more information. + +*Convert the `keyword` field to a `text` field:* To continue indexing the field while avoiding length limits, you can convert the `keyword` field to a `text` field. To do this: +. Create a new index with the `text` field data type. +. Reindex from the `_source` field of the source index using the {ref}/docs-reindex.html[`_reindex` API]. [[logs-mapping-troubleshooting-format-mismatch]] === Date format mismatch From 68d977f9ae1788deaab8b9f7ff7f4f10c9358b26 Mon Sep 17 00:00:00 2001 From: mdbirnstiehl Date: Mon, 30 Dec 2024 11:31:23 -0600 Subject: [PATCH 3/4] add date format mismatch --- .../logs-troubleshooting.asciidoc | 39 +++++++++++++++++-- 1 file changed, 36 insertions(+), 3 deletions(-) diff --git a/docs/en/observability/logs-troubleshooting.asciidoc b/docs/en/observability/logs-troubleshooting.asciidoc index e61ae4bdd6..836b90495d 100644 --- a/docs/en/observability/logs-troubleshooting.asciidoc +++ b/docs/en/observability/logs-troubleshooting.asciidoc @@ -155,9 +155,10 @@ This section provides possible solutions for mapping and pipeline issues you mig === Keyword fields are too long The `keyword` field limit is 32,766 bytes. When indexing a document, if your `keyword` field length exceeds this limit, you'll see an error similar to the following: -``` +[source, plaintext] +---- max_bytes_length_exceeded_exception: bytes can be at most 32766 in length -``` +---- [discrete] [[logs-mapping-troubleshooting-keyword-limit-solution]] @@ -178,8 +179,40 @@ Refer to the {ref}/ignore-above.html[`ignore_above`] docs for more information. . Create a new index with the `text` field data type. . Reindex from the `_source` field of the source index using the {ref}/docs-reindex.html[`_reindex` API]. -[[logs-mapping-troubleshooting-format-mismatch]] +[discrete] +[[logs-mapping-troubleshooting-date-mismatch]] === Date format mismatch +If the format of the `date` field in your document doesn't match the format set in your index template, you'll see an error similar to the following: + +[source, plaintext] +---- +failed to parse field [date] of type [date] in document with id 'KGcZb3cBqhj6kAxank_x'. +---- + +[discrete] +[[logs-mapping-troubleshooting-date-solution]] +==== Solution +Avoid this error by adding the format of the mismatched date to your index template. +Multiple formats can be specified by separating them with `||` as a separator. +Each format will be tried in turn until a matching format is found. +For example: + +[source,console,id=date-format-example] +---- +PUT my-index-000001 +{ + "mappings": { + "properties": { + "date": { + "type": "date", + "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis" + } + } + } +} +------ + +Refer to the {ref}/date.html[`date` field type] docs for more information. [[logs-mapping-troubleshooting-grok-mismatch]] === Grok or dissect pattern mismatch From dd830f2ff7dcb59030813d357d61fb51f6258e5f Mon Sep 17 00:00:00 2001 From: mdbirnstiehl Date: Thu, 2 Jan 2025 15:01:37 -0600 Subject: [PATCH 4/4] fix code block --- docs/en/observability/logs-troubleshooting.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/observability/logs-troubleshooting.asciidoc b/docs/en/observability/logs-troubleshooting.asciidoc index 836b90495d..29b5ae9ddc 100644 --- a/docs/en/observability/logs-troubleshooting.asciidoc +++ b/docs/en/observability/logs-troubleshooting.asciidoc @@ -210,7 +210,7 @@ PUT my-index-000001 } } } ------- +---- Refer to the {ref}/date.html[`date` field type] docs for more information.