-
Notifications
You must be signed in to change notification settings - Fork 228
OTRM Troubleshooting doc #4984
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
OTRM Troubleshooting doc #4984
Changes from 3 commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
e13d6ca
OYRM Troubleshooting doc
JV0812 b61b4e7
minor fix
JV0812 32a4627
Update troubleshooting.md
JV0812 a0efb35
Update docs/send-data/opentelemetry-collector/remote-management/troub…
JV0812 7452ca2
Update docs/send-data/opentelemetry-collector/remote-management/troub…
JV0812 a4eee0a
Update docs/send-data/opentelemetry-collector/remote-management/troub…
JV0812 24752c4
Update docs/send-data/opentelemetry-collector/remote-management/troub…
JV0812 6ceb242
Merge branch 'main' into OTRM-troubleshooting
JV0812 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
122 changes: 122 additions & 0 deletions
122
docs/send-data/opentelemetry-collector/remote-management/troubleshooting.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| --- | ||
| id: troubleshooting | ||
| title: Troubleshooting | ||
| description: Learn about common troubleshooting scenarios of Sumo Logic OpenTelemetry collector remote management. | ||
| --- | ||
|
|
||
| This document contains common troubleshooting scenarios about the Sumo Logic OpenTelemetry collector remote management. | ||
|
|
||
| ## Source template specific issues | ||
|
|
||
| ### Docker: Path Permission Denied | ||
|
|
||
| #### Error message | ||
|
|
||
| ` | ||
| Error: failed to setup configuration components: cannot start pipelines: permission denied while trying to connect to the Docker daemon socket at unix:///cav/run/docker.sock: Get "http://****> | ||
| collector server run finished with error: failed to setup configuration components: cannot start pipelines: permission denied while trying to connect to the Docker daemon soc> | ||
| ` | ||
|
|
||
| #### Solution | ||
|
|
||
| 1. Ensure that you have satisfied the [Docker source template pre-requisites](/docs/send-data/opentelemetry-collector/remote-management/source-templates/docker/#prerequisites). | ||
| 1. Restart the agent. | ||
|
|
||
| ### Windows: Failed to apply OpAMP agent remote config | ||
|
|
||
| #### Error message | ||
|
|
||
| ` | ||
| Failed to start service: invalid configuration: no receiver configuration specified in config | ||
| ` | ||
|
|
||
| #### Solution | ||
|
|
||
| 1. Make sure that you have selected the **Event Channels/Scrappers** sub-options for the source template config. | ||
| 1. Delete the invalid configuration from machine path `/etc/otelcol-sumo/opamp.d` for Linux/Mac operating system or `C:\ProgramData\Sumo Logic\OpenTelemetry Collector\config\opamp.d` for Windows operating system. | ||
| 1. Update the invalid inputs, delete invalid configuration source templates, or unlink the agent from UI. | ||
| 1. Restart the agent. | ||
|
|
||
| ### Apache: Missing hostname / Invalid hostname | ||
|
|
||
| #### Error message | ||
|
|
||
| ` | ||
| Failed to apply OpAMP agent remote config {"kind": "extension", "name": "opamp", "error": "cannot save the OpAMP effective config to /etc/otelcol-sumo/opamp.d: cannot validate config: receivers::apache/ce121f50-***nha-aidn: missing hostname: 'localhost'"} | ||
| ` | ||
|
|
||
| #### Solution | ||
|
|
||
| 1. Ensure that you have satisfied the [Apache source template pre-requisites](/docs/send-data/opentelemetry-collector/remote-management/source-templates/apache/#prerequisites). | ||
| 1. Delete the invalid configuration from machine path `/etc/otelcol-sumo/opamp.d` for Linux/Mac operating system or `C:\ProgramData\Sumo Logic\OpenTelemetry Collector\config\opamp.d` for Windows operating system. | ||
| 1. Update the invalid inputs, delete invalid configuration source templates, or unlink the agent from UI. | ||
| 1. Restart the agent. | ||
|
|
||
| ### Syslog: Failed to listen on interface | ||
|
|
||
| #### Error message | ||
|
|
||
| ` | ||
| Jan 17 11:34:14 ip-***-***-***-*** otelcol-sumo.sh[1234]: Error: failed to setup configuration components: cannot start pipelines: start stanza: failed to listen on interface: failed to configure tcp listener: listen tcp 0.0.0.0:***: bind: permission denied | ||
| Jan 17 11:34:14 ip-***-***-***-*** otelcol-sumo.sh[1234]: 2025/01/17 11:34:14 collector server run finished with error: failed to setup configuration components: cannot start pipelines: start stanza: failed to listen on interface: failed to configure tcp listener: listen tcp 0.0.0.0:***: bind: permission denied | ||
| ` | ||
|
|
||
| #### Solution | ||
|
|
||
| 1. Delete the invalid configuration from machine path `/etc/otelcol-sumo/opamp.d` for Linux/Mac operating system or `C:\ProgramData\Sumo Logic\OpenTelemetry Collector\config\opamp.d` for Windows operating system. | ||
| 1. Update the invalid inputs (Ensure that valid ports have the right permissions), delete invalid configuration source templates, or unlink the agent from UI. | ||
JV0812 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| 1. Restart the agent. | ||
|
|
||
| ## Agent disconnected from opamp server | ||
|
|
||
| #### Error message | ||
|
|
||
| ` | ||
| Agent Health Warning on UI : agent disconnected from opamp server. | ||
| ` | ||
|
|
||
| #### Solution | ||
|
|
||
| - Restart the agent. | ||
JV0812 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ## Failed to apply OpAMP agent remote config | ||
|
|
||
| #### Error messages | ||
|
|
||
| Error message 1: | ||
|
|
||
| ` | ||
| Failed to apply OpAMP agent remote config {"kind": "extension", "name": "opamp", "error": "cannot save the OpAMP effective config to /etc/otelcol-sumo/opamp.d: cannot validate config: processors::filter/logs/5c4f1d86-7946-4174-952f-bb52f016d2cd: unable to parse OTTL condition \"IsMatch(body, \\\"~!@#$%^&*()_+{}|:\\\\\\\"<>?/.,';\\\\\\\\][=-`\\\")\": couldn't create function: the pattern supplied to IsMatch is not a valid regexp pattern: error parsing regexp: missing closing ]: [=-``"} | ||
JV0812 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| ` | ||
|
|
||
| Error message 2: | ||
|
|
||
| ` | ||
| collector server run finished with error: failed to setup configuration components: failed to build pipelines: failed to create "filelog/8ee0d9e8-92cf-46c8-97be-764c67e59a83" receiver for data type "logs": compile line end regex: error parsing regexp: unexpected | ||
| ` | ||
|
|
||
| #### Solution | ||
|
|
||
| 1. Ensure the rules comply with [RE2 syntax](https://github.com/google/re2/wiki/Syntax). | ||
| 1. Delete the invalid configuration from machine path `/etc/otelcol-sumo/opamp.d` for Linux/Mac operating system or `C:\ProgramData\Sumo Logic\OpenTelemetry Collector\config\opamp.d` for Windows operating system. | ||
| 1. Update the invalid inputs, delete invalid configuration source templates, or unlink the agent from UI. For example, replace the invalid regex with valid regex. | ||
|
|
||
| ## Agent crashes completely | ||
|
|
||
| If there is a runtime error that leads the agent crash unexpectedly, follow the below steps to rectify the issue: | ||
|
|
||
| 1. Delete the invalid configuration from machine path `/etc/otelcol-sumo/opamp.d` for Linux/Mac operating system or `C:\ProgramData\Sumo Logic\OpenTelemetry Collector\config\opamp.d` for Windows operating system. | ||
| 1. Update the invalid inputs, delete invalid configuration source templates, or unlink the agent from UI. | ||
| 1. Restart the agent. | ||
|
|
||
| ## Agent stuck in the config processing loop | ||
|
|
||
| If the agent repeatedly fails to process the effective configuration, it will be unable to perform any other operations, including sending heartbeat signals to the server. The server may interpret the absence of these heartbeats as an indication that the agent is offline, which can complicate recovery efforts. To resolve this issue, follow the below-mentioned steps: | ||
|
|
||
| 1. Delete the invalid configuration from machine path `/etc/otelcol-sumo/opamp.d` for Linux/Mac operating system or `C:\ProgramData\Sumo Logic\OpenTelemetry Collector\config\opamp.d` for Windows operating system. | ||
| 1. Update the invalid inputs, delete invalid configuration source templates, or unlink the agent from the user interface. | ||
| 1. Restart the agent. | ||
|
|
||
| ## Agent sends failure response to the server | ||
|
|
||
| If the agent attempts to apply the effective configuration but encounters errors, such as incompatible or invalid configuration parameters, it will send a failure response to the Opamp server to indicate the problem. This issue, however, disrupts data ingestion and leads to service interruptions. To resolve this, you can update the invalid inputs, delete any invalid configuration source templates, or unlink the agent from the UI. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.