From b3c2c88e2fed6f53a94672aee235b0c3200cddbd Mon Sep 17 00:00:00 2001 From: "mergify[bot]" <37929162+mergify[bot]@users.noreply.github.com> Date: Wed, 29 Jan 2025 10:07:15 -0600 Subject: [PATCH] [8.16](backport #4693) Add common pipeline and mapping issues and mitigations (#4798) * Add common pipeline and mapping issues and mitigations (#4693) (cherry picked from commit 056d34a3287fafaf12e58fbf8a64ddab23eed33e) --------- Co-authored-by: Mike Birnstiehl <114418652+mdbirnstiehl@users.noreply.github.com> Co-authored-by: github-actions[bot] --- .../logs-troubleshooting.asciidoc | 131 ++++++++++++++++-- 1 file changed, 118 insertions(+), 13 deletions(-) diff --git a/docs/en/observability/logs-troubleshooting.asciidoc b/docs/en/observability/logs-troubleshooting.asciidoc index 531be77c97..747a3994ed 100644 --- a/docs/en/observability/logs-troubleshooting.asciidoc +++ b/docs/en/observability/logs-troubleshooting.asciidoc @@ -1,11 +1,20 @@ [[logs-troubleshooting]] = Troubleshoot logs +Use this page to find possible solutions for errors your encountering with your logs. This troubleshooting page is divided into the following sections: + +- <> +- <> + +[discrete] +[[logs-onboarding-troubleshooting]] +== Common onboarding issues + This section provides possible solutions for errors you might encounter while onboarding your logs. [discrete] [[logs-troubleshooting-insufficient-priv]] -== User does not have permissions to create API key +=== User does not have permissions to create API key If you don't have the required privileges to create an API key, you'll see the following error message: @@ -21,7 +30,7 @@ to the role of the authenticated user. [discrete] [[logs-troubleshooting-insufficient-priv-solution]] -=== Solution +==== Solution You need to either: @@ -30,7 +39,7 @@ You need to either: [discrete] [[logs-troubleshooting-API-key-failed]] -== Failed to create API key +=== Failed to create API key If you don't have the privileges to create `savedObjects` in {kib}, you'll see the following error message: @@ -43,14 +52,14 @@ Something went wrong: Unable to create observability-onboarding-state [discrete] [[logs-troubleshooting-API-key-failed-solution]] -=== Solution +==== Solution You need an administrator to give you the `Saved Objects Management` {kib} privilege to generate the required `observability-onboarding-state` flow state. Once you have the necessary privileges, restart the onboarding flow. [discrete] [[logs-troubleshooting-kib-not-accessible]] -== {kib} not accessible from host +=== {kib} not accessible from host If {kib} is not accessible from the host, you'll see the following error message after pasting the *Install the {agent}* instructions into the host: @@ -61,7 +70,7 @@ Failed to connect to {host} port {port} after 0 ms: Connection refused [discrete] [[logs-troubleshooting-kib-not-accessible-solution]] -=== Solution +==== Solution The host needs access to {kib}. Port `443` must be open and the deployment's {es} endpoint must be reachable. Locate your project's endpoint from *Help menu (image:images/help-icon.png[]) → Connection details*. @@ -73,7 +82,7 @@ curl https://your-endpoint.elastic.cloud [discrete] [[logs-troubleshooting-download-agent]] -== Download {agent} failed +=== Download {agent} failed If the host was able to download the installation script but cannot connect to the public artifact repository, you'll see the following error message: @@ -86,7 +95,7 @@ Failed to download Elastic Agent, see script for error. [discrete] [[logs-troubleshooting-download-agent-solution]] -=== Solutions +==== Solutions * If the combination of the {agent} version and operating system architecture is not available, you'll see the following error message: + @@ -108,7 +117,7 @@ To fix this, delete previous downloads and restart the onboarding. [discrete] [[logs-troubleshooting-install-agent]] -== Install {agent} failed +=== Install {agent} failed If an {agent} already exists on your host, you'll see the following error message: @@ -121,19 +130,115 @@ Failed to install Elastic Agent, see script for error. [discrete] [[logs-troubleshooting-install-agent-solution]] -=== Solution +==== Solution + You can uninstall the current {agent} using the `elastic-agent uninstall` command, and run the script again. WARNING: Uninstalling the current {agent} removes the entire current setup, including the existing configuration. [discrete] [[logs-troubleshooting-wait-for-logs]] -== Waiting for Logs to be shipped... step never completes +=== Waiting for Logs to be shipped... step never completes If the *Waiting for Logs to be shipped...* step never completes, logs are not being shipped to {es}, and there is most likely an issue with your {agent} configuration. [discrete] [[logs-troubleshooting-wait-for-logs-solution]] -=== Solution +==== Solution + +Inspect the {agent} logs for errors. See the {fleet-guide}/debug-standalone-agents.html#inspect-standalone-agent-logs[Debug standalone {agent}s] documentation for more on finding errors in {agent} logs. + +[discrete] +[[logs-common-mapping-troubleshooting]] +== Mapping and pipeline issues + +This section provides possible solutions for mapping and pipeline issues you might encounter with your logs. + +[discrete] +[[logs-mapping-troubleshooting-keyword-limit]] +=== Keyword fields are too long + +The `keyword` field limit is 32,766 bytes. When indexing a document, if your `keyword` field length exceeds this limit, you'll see an error similar to the following: + +[source, plaintext] +---- +max_bytes_length_exceeded_exception: bytes can be at most 32766 in length +---- + +[discrete] +[[logs-mapping-troubleshooting-keyword-limit-solution]] +==== Solution + +Avoid this error using one of the following options: + +*Stop indexing the field:* If you don't need the `keyword` field for aggregation or search, set `"index":false` in the index template to stop indexing the field. + +*Convert the `keyword` field to a `text` field:* To continue indexing the field while avoiding length limits, you can convert the `keyword` field to a `text` field. + +NOTE: Aggregations on this field would no longer be supported, but the contents would be searchable. + +To convert the `keyword` field to a `text` field: + +. Create a new index with the `text` field data type. +. Reindex from the `_source` field of the source index using the {ref}/docs-reindex.html[`_reindex` API]. + +[discrete] +[[logs-mapping-troubleshooting-date-mismatch]] +=== Date format mismatch + +If the format of the `date` field in your document doesn't match the format set in your index template, you'll see an error similar to the following: + +[source, plaintext] +---- +failed to parse field [date] of type [date] in document with id 'KGcZb3cBqhj6kAxank_x'. +---- + +[discrete] +[[logs-mapping-troubleshooting-date-solution]] +==== Solution + +Add the format of the mismatched date to your index template. +Multiple formats can be specified by separating them with `||` as a separator. +Each format will be tried in turn until a matching format is found. +For example: + +[source,console,id=date-format-example] +---- +PUT my-index-000001 +{ + "mappings": { + "properties": { + "date": { + "type": "date", + "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis" + } + } + } +} +---- + +Refer to the {ref}/date.html[`date` field type] docs for more information. + +[discrete] +[[logs-mapping-troubleshooting-grok-mismatch]] +=== Grok or dissect pattern mismatch + +If the pattern in your grok or dissect processor doesn't match the format of your document, you'll see an error similar to the following: + +[source, plaintext] +---- +Provided Grok patterns do not match field value... +---- + +[discrete] +[[logs-mapping-troubleshooting-grok-solution]] +==== Solution + +Make sure your {ref}/grok-processor.html[grok] or {ref}/dissect-processor.html[dissect] processor pattern matches your log document format. + +You can build and debug grok patterns in {kib} using the {kibana-ref}/xpack-grokdebugger.html[Grok Debugger]. Find the *Grok Debugger* by navigating to the *Developer tools* page using the +navigation menu or the global search field. + +From here, you can enter sample data representative of the log document you're trying to ingest and the Grok pattern you want to apply to the data. -Inspect the {agent} logs for errors. See the {fleet-guide}/debug-standalone-agents.html#inspect-standalone-agent-logs[Debug standalone {agent}s] documentation for more on finding errors in {agent} logs. \ No newline at end of file +If you don't see any *Structured Data* when you simulate the grok pattern, iterate on the pattern until you find the error. \ No newline at end of file