Merge pull request #64772 from pavolloffay/otel-troubleshoot

TRACING-3258 | Document troubleshooting OTEL

Author: michaelryanpeter

_topic_maps/_topic_map.yml

@@ -3650,6 +3650,8 @@ Topics:
       File: distr-tracing-otel-configuring
     - Name: Use
       File: distr-tracing-otel-using
+    - Name: Troubleshooting
+      File: distr-tracing-otel-troubleshooting
     - Name: Migration
       File: distr-tracing-otel-migrating
     - Name: Removal

distr_tracing/distr_tracing_otel/distr-tracing-otel-troubleshooting.adoc

@@ -0,0 +1,13 @@
+:_content-type: ASSEMBLY
+[id="distr-tracing-otel-troubleshoot"]
+= Troubleshooting the {OTELShortName}
+include::_attributes/common-attributes.adoc[]
+:context: distr-tracing-otel-troubleshoot
+
+toc::[]
+
+The OpenTelemetry Collector offers multiple ways to measure its health as well as investigate data ingestion issues.
+
+include::modules/distr-tracing-otel-troubleshoot-logs.adoc[leveloffset=+1]
+include::modules/distr-tracing-otel-troubleshoot-metrics.adoc[leveloffset=+1]
+include::modules/distr-tracing-otel-troubleshoot-logging-exporter.adoc[leveloffset=+1]

modules/distr-tracing-otel-troubleshoot-logging-exporter.adoc

@@ -0,0 +1,30 @@
+// Module included in the following assemblies:
+//
+// * /distr_tracing/distr_tracing_otel/distr-tracing-otel-troubleshooting.adoc
+:_content-type: PROCEDURE
+[id="distr-tracing-otel-troubleshoot-logging-exporter_{context}"]
+= Logging exporter
+
+You can configure the logging exporter to export the collected data to the standard output.
+
+.Procedure
+
+. Configure the OpenTelemetry Collector custom resource as follows:
++
+[source,yaml]
+----
+  config: |
+    exporters:
+      logging:
+        verbosity: detailed
+    service:
+      pipelines:
+        traces:
+          exporters: [logging]
+        metrics:
+          exporters: [logging]
+        logs:
+          exporters: [logging]
+----
+
+. Use the `oc logs` command or the OpenShift console to export the logs to the standard output.

modules/distr-tracing-otel-troubleshoot-logs.adoc

@@ -0,0 +1,24 @@
+// Module included in the following assemblies:
+//
+// * /distr_tracing/distr_tracing_otel/distr-tracing-otel-troubleshooting.adoc
+:_content-type: PROCEDURE
+[id="distr-tracing-otel-troubleshoot-logs_{context}"]
+= Getting the OpenTelemetry Collector logs
+
+You can get the logs for the OpenTelemetry Collector as follows.
+
+.Procedure
+
+. Set the relevant log level in the OpenTelemetry Collector custom resource (CR):
++
+[source,yaml]
+----
+  config: |
+    service:
+      telemetry:
+        logs:
+          level: debug <1>
+----
+<1> Collector's log level. Select one of the following values: `info`, `warn`, `error`, or `debug`. Defaults to `info`.
+
+. Use the `oc logs` command or the OpenShift console to retrieve the logs.

modules/distr-tracing-otel-troubleshoot-metrics.adoc

@@ -0,0 +1,40 @@
+// Module included in the following assemblies:
+//
+// * /distr_tracing/distr_tracing_otel/distr-tracing-otel-troubleshooting.adoc
+:_content-type: PROCEDURE
+[id="distr-tracing-otel-troubleshoot-metrics_{context}"]
+= Exposing the metrics
+
+The OpenTelemetry Collector exposes the metrics about the data volumes it has processed. The following metrics are for spans, although similar metrics are exposed for metrics and logs signals:
+
+`otelcol_receiver_accepted_spans`:: The number of spans successfully pushed into the pipeline.
+`otelcol_receiver_refused_spans`:: The number of spans that could not be pushed into the pipeline.
+`otelcol_exporter_sent_spans`:: The number of spans successfully sent to destination.
+`otelcol_exporter_enqueue_failed_spans`:: The number of spans failed to be added to the sending queue.
+
+The operator creates a `<cr-name>-collector-monitoring` telemetry service that you can use to scrape the metrics endpoint.
+
+.Procedure
+
+. Enable the telemetry service by adding the following lines in the OpenTelemetry Collector custom resource:
+
++
+[source,yaml]
+----
+  config: |
+    service:
+      telemetry:
+        metrics:
+          address: ":8888" <1>
+----
+<1> The address on which internal collector metrics are exposed. Defaults to `:8888`.
+// TODO Operator 0.82.0 has spec.observability.metrics.enableMetrics config that creates ServiceMonitors for users
+
+. Retrieve the metrics by running the following command, which uses the port forwarding collector pod:
++
+[source,terminal]
+----
+$ oc port-forward <collector-pod>
+----
+
+. Access the metrics endpoint at `+http://localhost:8888/metrics+`.