From 35cdf58b28ad25bf1d3e0dcb0ef9db95653f5860 Mon Sep 17 00:00:00 2001 From: Aleksandra Spilkowska Date: Thu, 18 Sep 2025 16:57:29 +0200 Subject: [PATCH 1/6] Add connectivity issues --- .../ingest/opentelemetry/connectivity.md | 121 ++++++++++++++++++ troubleshoot/ingest/opentelemetry/toc.yml | 1 + 2 files changed, 122 insertions(+) create mode 100644 troubleshoot/ingest/opentelemetry/connectivity.md diff --git a/troubleshoot/ingest/opentelemetry/connectivity.md b/troubleshoot/ingest/opentelemetry/connectivity.md new file mode 100644 index 0000000000..933d9d9e9b --- /dev/null +++ b/troubleshoot/ingest/opentelemetry/connectivity.md @@ -0,0 +1,121 @@ +--- +navigation_title: Connectivity issues +description: Troubleshoot connectivity issues between EDOT SDKs, the Collector, and Elastic. +applies_to: + stack: all + serverless: all +products: + - id: observability + - id: edot-collector + - id: edot-sdk +--- + +# Connectivity issues with EDOT + +Connectivity problems occur when the OpenTelemetry SDK or Collector cannot reach the Elastic endpoint. Even with correct proxy settings, network restrictions such as blocked ports or firewalls can prevent data from flowing. + + +## Symptoms + +You might see one or more of the following error messages: + +- `connection refused` +- `network unreachable` +- `i/o timeout` +- `tls: handshake failure` + +These errors may appear either in application logs (from the SDK) or in the Collector logs. + + +## Causes + +Connectivity errors usually trace back to one of the following issues: + +- **Firewall or port blocking** + + Outbound traffic on port `443` is blocked by corporate firewalls or network policies. + +- **Endpoint errors** + + The endpoint is unreachable or not listening on the specified port: + + - `connection refused`: endpoint not listening + - `network unreachable`: VPN, routing, or DNS failure + - `timeout`: traffic dropped by firewall, proxy, or load balancer + +- **Proxy misconfiguration** + + Proxy environment variables (`HTTP_PROXY`, `HTTPS_PROXY`) may be set correctly but the proxy itself lacks access to Elastic or restricts ports. Refer to [Proxy settings](opentelemetry://reference/edot-collector/config/proxy.md) for more information. + + +### Differentiating SDK vs Collector issues + +Errors can look similar whether they come from an SDK or the Collector, but identifying the source helps you isolate the problem: + +* **SDK** + +Application logs report failures when the SDK cannot send data to the Collector or directly to Elastic. These often appear as `connection refused` or `timeout` messages. If seen, verify that the Collector endpoint is reachable. + +* **The Collector** + +Collector logs show export failures when it cannot forward data to Elastic. Look for messages like `cannot send spans` or `failed to connect to `. If present, confirm the Collector’s exporters configuration and network access. + + +## Resolution + +Before you dig into SDK or Collector configuration, confirm that your environment can reach the Elastic endpoint by performing the following checks: + +:::::{stepper} + +::::{step} Verify DNS resolution + +Make sure the hostname for your Elastic endpoint resolves correctly: + +```bash +nslookup +``` + +:::: + +::::{step} Test network reachability + +```bash +ping +``` + +:::: + +::::{step} Check open ports + +Test whether the required OTLP ports are open (default `443` for HTTPS): + +```bash +nc -vz 443 +``` + +:::: + +::::{step} Verify TLS/SSL + +Check that TLS certificates can be validated: + +```bash +openssl s_client -connect :443 +``` + +:::: + +::::: + +If any of these steps fail, the issue is likely caused by network infrastructure rather than your SDK or Collector configuration. + + +### Next steps + +If basic checks and configuration look correct but issues persist, collect more details before escalating: + +* Review proxy settings. For more information, refer to [Proxy settings](opentelemetry://reference/edot-collector/config/proxy.md). + +* If ports are confirmed open but errors persist, [enable debug logging in the SDK](../opentelemetry/edot-sdks/enable-debug-logging.md) or [in the Collector](../opentelemetry/edot-collector/enable-debug-logging.md) for more detail. + +* Contact your network administrator with test results if you suspect firewall restrictions. \ No newline at end of file diff --git a/troubleshoot/ingest/opentelemetry/toc.yml b/troubleshoot/ingest/opentelemetry/toc.yml index 0d41459130..eeaf7d82c1 100644 --- a/troubleshoot/ingest/opentelemetry/toc.yml +++ b/troubleshoot/ingest/opentelemetry/toc.yml @@ -23,4 +23,5 @@ toc: - file: edot-sdks/enable-debug-logging.md - file: edot-sdks/proxy.md - file: no-data-in-kibana.md + - file: connectivity.md - file: contact-support.md From 4b3885ea1119703e23c2d70757615907faa0b2cb Mon Sep 17 00:00:00 2001 From: Aleksandra Spilkowska Date: Tue, 23 Sep 2025 13:44:40 +0200 Subject: [PATCH 2/6] Apply peer comments --- .../ingest/opentelemetry/connectivity.md | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/troubleshoot/ingest/opentelemetry/connectivity.md b/troubleshoot/ingest/opentelemetry/connectivity.md index 933d9d9e9b..7e0f4e5932 100644 --- a/troubleshoot/ingest/opentelemetry/connectivity.md +++ b/troubleshoot/ingest/opentelemetry/connectivity.md @@ -1,9 +1,10 @@ --- navigation_title: Connectivity issues -description: Troubleshoot connectivity issues between EDOT SDKs, the Collector, and Elastic. +description: Troubleshoot connectivity issues between EDOT SDKs, the EDOT Collector, and Elastic. applies_to: - stack: all - serverless: all + - serverless: all + - product: + edot_collector: ga products: - id: observability - id: edot-collector @@ -12,7 +13,7 @@ products: # Connectivity issues with EDOT -Connectivity problems occur when the OpenTelemetry SDK or Collector cannot reach the Elastic endpoint. Even with correct proxy settings, network restrictions such as blocked ports or firewalls can prevent data from flowing. +Connectivity problems occur when the EDOT SDKs or the EDOT Collector can't reach Elastic. Even with correct proxy settings, network restrictions such as blocked ports or firewalls can prevent data from flowing. ## Symptoms @@ -24,7 +25,7 @@ You might see one or more of the following error messages: - `i/o timeout` - `tls: handshake failure` -These errors may appear either in application logs (from the SDK) or in the Collector logs. +These errors might appear either in application logs (from the SDK) or in the Collector logs. ## Causes @@ -45,18 +46,18 @@ Connectivity errors usually trace back to one of the following issues: - **Proxy misconfiguration** - Proxy environment variables (`HTTP_PROXY`, `HTTPS_PROXY`) may be set correctly but the proxy itself lacks access to Elastic or restricts ports. Refer to [Proxy settings](opentelemetry://reference/edot-collector/config/proxy.md) for more information. + Proxy environment variables (`HTTP_PROXY`, `HTTPS_PROXY`) might be set correctly but the proxy itself lacks access to Elastic or restricts ports. Refer to [Proxy settings](opentelemetry://reference/edot-collector/config/proxy.md) for more information. -### Differentiating SDK vs Collector issues +### Differences between SDK and Collector issues -Errors can look similar whether they come from an SDK or the Collector, but identifying the source helps you isolate the problem: +Errors can look similar whether they come from an SDK or the Collector. Identifying the source helps you isolate the problem: -* **SDK** +#### SDK Application logs report failures when the SDK cannot send data to the Collector or directly to Elastic. These often appear as `connection refused` or `timeout` messages. If seen, verify that the Collector endpoint is reachable. -* **The Collector** +#### The Collector Collector logs show export failures when it cannot forward data to Elastic. Look for messages like `cannot send spans` or `failed to connect to `. If present, confirm the Collector’s exporters configuration and network access. From 558d74bc9d31891d07a9340087abb5813e022309 Mon Sep 17 00:00:00 2001 From: Aleksandra Spilkowska Date: Fri, 3 Oct 2025 14:18:49 +0200 Subject: [PATCH 3/6] update yaml --- troubleshoot/ingest/opentelemetry/connectivity.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/troubleshoot/ingest/opentelemetry/connectivity.md b/troubleshoot/ingest/opentelemetry/connectivity.md index 7e0f4e5932..36273446d1 100644 --- a/troubleshoot/ingest/opentelemetry/connectivity.md +++ b/troubleshoot/ingest/opentelemetry/connectivity.md @@ -2,8 +2,8 @@ navigation_title: Connectivity issues description: Troubleshoot connectivity issues between EDOT SDKs, the EDOT Collector, and Elastic. applies_to: - - serverless: all - - product: + serverless: all + product: edot_collector: ga products: - id: observability From 82d3ddb9be7edcd00da073885d4395e60d89a938 Mon Sep 17 00:00:00 2001 From: Aleksandra Spilkowska Date: Fri, 3 Oct 2025 14:56:17 +0200 Subject: [PATCH 4/6] Apply SME comments --- .../ingest/opentelemetry/connectivity.md | 45 +++++++++++++++++-- 1 file changed, 41 insertions(+), 4 deletions(-) diff --git a/troubleshoot/ingest/opentelemetry/connectivity.md b/troubleshoot/ingest/opentelemetry/connectivity.md index 36273446d1..2fc9b1554f 100644 --- a/troubleshoot/ingest/opentelemetry/connectivity.md +++ b/troubleshoot/ingest/opentelemetry/connectivity.md @@ -27,14 +27,35 @@ You might see one or more of the following error messages: These errors might appear either in application logs (from the SDK) or in the Collector logs. +Example (Collector): + +```text +2024-09-15T12:44:30Z error exporterhelper/queued_retry.go:149 Exporting failed. Rejecting data. Error: context deadline exceeded +``` + +Example (Python SDK): + +```text +opentelemetry.sdk ERROR OTLPSpanExporter - Failed to export spans: [Errno 111] Connection refused +``` ## Causes Connectivity errors usually trace back to one of the following issues: - **Firewall or port blocking** - - Outbound traffic on port `443` is blocked by corporate firewalls or network policies. + + Outbound traffic may be blocked by corporate firewalls or network policies. + + Check that the required protocol and port combination is allowed: + + - OTLP/HTTP: TCP 4318 + - OTLP/gRPC: TCP 4317 + - Elastic over HTTPS: TCP 443 + - Elastic over HTTP: TCP 9200 + + Also confirm whether your environment uses IPv4 or IPv6, as routing rules may differ. + - **Endpoint errors** @@ -51,12 +72,24 @@ Connectivity errors usually trace back to one of the following issues: ### Differences between SDK and Collector issues -Errors can look similar whether they come from an SDK or the Collector. Identifying the source helps you isolate the problem: +Errors can look similar whether they come from an SDK or the Collector. Identifying the source helps you isolate the problem. + +:::{note} +Note: Some SDKs support setting a proxy directly (for example, using `HTTPS_PROXY`). Refer to [Proxy settings for EDOT SDKs](../opentelemetry/edot-sdks/proxy.md) for details. +::: #### SDK Application logs report failures when the SDK cannot send data to the Collector or directly to Elastic. These often appear as `connection refused` or `timeout` messages. If seen, verify that the Collector endpoint is reachable. +For guidance on enabling logs in your SDK, see [Enable SDK debug logging](../opentelemetry/edot-sdks/enable-debug-logging.md). + +Example (Java SDK): + +```text +io.opentelemetry.exporter.otlp.internal.grpc.OkHttpGrpcExporter - Failed to export spans. Error: UNAVAILABLE: io exception +``` + #### The Collector Collector logs show export failures when it cannot forward data to Elastic. Look for messages like `cannot send spans` or `failed to connect to `. If present, confirm the Collector’s exporters configuration and network access. @@ -64,7 +97,11 @@ Collector logs show export failures when it cannot forward data to Elastic. Look ## Resolution -Before you dig into SDK or Collector configuration, confirm that your environment can reach the Elastic endpoint by performing the following checks: +Before you dig into SDK or Collector configuration, confirm that your environment can reach the Elastic endpoint. + +:::{note} +The examples below use Linux/Unix-style commands. On Windows or when testing IPv6, the equivalent tooling or syntax may differ (for example, `Test-NetConnection` in PowerShell). +::: :::::{stepper} From 2e6825a503511a041f6608d31d127174215509e6 Mon Sep 17 00:00:00 2001 From: Aleksandra Spilkowska Date: Fri, 3 Oct 2025 15:03:03 +0200 Subject: [PATCH 5/6] Add placeholders + minor changes --- troubleshoot/ingest/opentelemetry/connectivity.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/troubleshoot/ingest/opentelemetry/connectivity.md b/troubleshoot/ingest/opentelemetry/connectivity.md index 2fc9b1554f..adca014535 100644 --- a/troubleshoot/ingest/opentelemetry/connectivity.md +++ b/troubleshoot/ingest/opentelemetry/connectivity.md @@ -51,8 +51,8 @@ Connectivity errors usually trace back to one of the following issues: - OTLP/HTTP: TCP 4318 - OTLP/gRPC: TCP 4317 - - Elastic over HTTPS: TCP 443 - - Elastic over HTTP: TCP 9200 + - {{es}} (over HTTPS): 443 + - {{es}}: 9200 Also confirm whether your environment uses IPv4 or IPv6, as routing rules may differ. @@ -100,7 +100,7 @@ Collector logs show export failures when it cannot forward data to Elastic. Look Before you dig into SDK or Collector configuration, confirm that your environment can reach the Elastic endpoint. :::{note} -The examples below use Linux/Unix-style commands. On Windows or when testing IPv6, the equivalent tooling or syntax may differ (for example, `Test-NetConnection` in PowerShell). +The examples below use command syntax from Linux and macOS. On Windows or when testing IPv6, the equivalent tooling or syntax may differ (for example, `Test-NetConnection` in PowerShell). ::: :::::{stepper} From 13e92109b9c7729a21103d043a8b43daaf427659 Mon Sep 17 00:00:00 2001 From: Aleksandra Spilkowska Date: Mon, 6 Oct 2025 13:01:49 +0200 Subject: [PATCH 6/6] Add firewall to line 57 --- troubleshoot/ingest/opentelemetry/connectivity.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/troubleshoot/ingest/opentelemetry/connectivity.md b/troubleshoot/ingest/opentelemetry/connectivity.md index adca014535..6d2aa988ad 100644 --- a/troubleshoot/ingest/opentelemetry/connectivity.md +++ b/troubleshoot/ingest/opentelemetry/connectivity.md @@ -54,7 +54,7 @@ Connectivity errors usually trace back to one of the following issues: - {{es}} (over HTTPS): 443 - {{es}}: 9200 - Also confirm whether your environment uses IPv4 or IPv6, as routing rules may differ. + Also confirm whether your environment uses IPv4 or IPv6, as routing and firewall rules may differ. - **Endpoint errors**