|
1 | 1 | # Basic OpenTelemetry metrics example with custom error handler: |
2 | 2 |
|
3 | | -This example shows how to setup the custom error handler for self-diagnostics. |
4 | | - |
5 | | -## Custom Error Handling: |
6 | | - |
7 | | -A custom error handler is set up to capture and record errors using the `tracing` crate's `error!` macro. These errors are then exported to a collector using the `opentelemetry-appender-tracing` crate, which utilizes the OTLP log exporter over `HTTP/protobuf`. As a result, any errors generated by the configured OTLP metrics pipeline are funneled through this custom error handler for proper recording and export. |
| 3 | +This example shows how to self-diagnose OpenTelemetry by enabling its internal |
| 4 | +logs. OpenTelemetry crates publish internal logs when "internal-logs" feature is |
| 5 | +enabled. This feature is enabled by default. Internal logs are published using |
| 6 | +`tracing` events, and hence, a `tracing` subscriber must be configured without |
| 7 | +which the logs are simply discarded. |
8 | 8 |
|
9 | 9 | ## Filtering logs from external dependencies of OTLP Exporter: |
10 | 10 |
|
11 | | -The example configures a tracing `filter` to restrict logs from external crates (`hyper`, `tonic`, and `reqwest`) used by the OTLP Exporter to the `error` level. This helps prevent an infinite loop of log generation when these crates emit logs that are picked up by the tracing subscriber. |
12 | | - |
13 | | -## Ensure that the internally generated errors are logged only once: |
14 | | - |
15 | | -By using a hashset to track seen errors, the custom error handler ensures that the same error is not logged multiple times. This is particularly useful for handling scenarios where continuous error logging might occur, such as when the OpenTelemetry collector is not running. |
16 | | - |
17 | | - |
18 | | -## Usage |
19 | | - |
20 | | -### `docker-compose` |
21 | | - |
22 | | -By default runs against the `otel/opentelemetry-collector:latest` image, and uses `reqwest-client` |
23 | | -as the http client, using http as the transport. |
24 | | - |
25 | | -```shell |
26 | | -docker-compose up |
27 | | -``` |
28 | | - |
29 | | -In another terminal run the application `cargo run` |
30 | | - |
31 | | -The docker-compose terminal will display logs, traces, metrics. |
32 | | - |
33 | | -Press Ctrl+C to stop the collector, and then tear it down: |
34 | | - |
35 | | -```shell |
36 | | -docker-compose down |
37 | | -``` |
38 | | - |
39 | | -### Manual |
40 | | - |
41 | | -If you don't want to use `docker-compose`, you can manually run the `otel/opentelemetry-collector` container |
42 | | -and inspect the logs to see traces being transferred. |
43 | | - |
44 | | -On Unix based systems use: |
45 | | - |
46 | | -```shell |
47 | | -# From the current directory, run `opentelemetry-collector` |
48 | | -docker run --rm -it -p 4318:4318 -v $(pwd):/cfg otel/opentelemetry-collector:latest --config=/cfg/otel-collector-config.yaml |
49 | | -``` |
50 | | - |
51 | | -On Windows use: |
52 | | - |
53 | | -```shell |
54 | | -# From the current directory, run `opentelemetry-collector` |
55 | | -docker run --rm -it -p 4318:4318 -v "%cd%":/cfg otel/opentelemetry-collector:latest --config=/cfg/otel-collector-config.yaml |
56 | | -``` |
57 | | - |
58 | | -Run the app which exports logs, metrics and traces via OTLP to the collector |
59 | | - |
60 | | -```shell |
61 | | -cargo run |
62 | | -``` |
63 | | - |
64 | | -### Output: |
65 | | - |
66 | | -- If the docker instance for collector is running, below error should be logged into the container. There won't be any logs from the `hyper`, `reqwest` and `tonic` crates. |
67 | | -``` |
68 | | -otel-collector-1 | 2024-06-05T17:09:46.926Z info LogExporter {"kind": "exporter", "data_type": "logs", "name": "logging", "resource logs": 1, "log records": 1} |
69 | | -otel-collector-1 | 2024-06-05T17:09:46.926Z info ResourceLog #0 |
70 | | -otel-collector-1 | Resource SchemaURL: |
71 | | -otel-collector-1 | Resource attributes: |
72 | | -otel-collector-1 | -> telemetry.sdk.name: Str(opentelemetry) |
73 | | -otel-collector-1 | -> telemetry.sdk.version: Str(0.23.0) |
74 | | -otel-collector-1 | -> telemetry.sdk.language: Str(rust) |
75 | | -otel-collector-1 | -> service.name: Str(unknown_service) |
76 | | -otel-collector-1 | ScopeLogs #0 |
77 | | -otel-collector-1 | ScopeLogs SchemaURL: |
78 | | -otel-collector-1 | InstrumentationScope opentelemetry-appender-tracing 0.4.0 |
79 | | -otel-collector-1 | LogRecord #0 |
80 | | -otel-collector-1 | ObservedTimestamp: 2024-06-05 17:09:45.931951161 +0000 UTC |
81 | | -otel-collector-1 | Timestamp: 1970-01-01 00:00:00 +0000 UTC |
82 | | -otel-collector-1 | SeverityText: ERROR |
83 | | -otel-collector-1 | SeverityNumber: Error(17) |
84 | | -otel-collector-1 | Body: Str(OpenTelemetry metrics error occurred: Metrics error: Warning: Maximum data points for metric stream exceeded. Entry added to overflow. Subsequent overflows to same metric until next collect will not be logged.) |
85 | | -otel-collector-1 | Attributes: |
86 | | -otel-collector-1 | -> name: Str(event examples/self-diagnostics/src/main.rs:42) |
87 | | -otel-collector-1 | Trace ID: |
88 | | -otel-collector-1 | Span ID: |
89 | | -otel-collector-1 | Flags: 0 |
90 | | -otel-collector-1 | {"kind": "exporter", "data_type": "logs", "name": "logging"} |
91 | | -``` |
92 | | - |
93 | | -- The SDK will keep trying to upload metrics at regular intervals if the collector's Docker instance is down. To avoid a logging loop, internal errors like 'Connection refused' will be attempted to be logged only once. |
| 11 | +The example configures a tracing `filter` to restrict logs from external crates |
| 12 | +(`hyper`, `tonic`, and `reqwest` etc.) used by the OTLP Exporter to the `error` |
| 13 | +level. This helps prevent an infinite loop of log generation when these crates |
| 14 | +emit logs that are picked up by the tracing subscriber. This is only a |
| 15 | +workaround until [the root |
| 16 | +issue](https://github.com/open-telemetry/opentelemetry-rust/issues/761) is |
| 17 | +resolved. |
| 18 | + |
| 19 | +## Filtering logs to be send to OpenTelemetry itself |
| 20 | + |
| 21 | +If you use [OpenTelemetry Tracing |
| 22 | +Appender](../../opentelemetry-appender-tracing/README.md) to send `tracing` logs |
| 23 | +to OpenTelemetry, then enabling OpenTelemetry internal logs can also cause |
| 24 | +infinite, recursive logging. You can filter out all OpenTelemetry internal logs |
| 25 | +from being sent to [OpenTelemetry Tracing |
| 26 | +Appender](../../opentelemetry-appender-tracing/README.md) using a filter, like |
| 27 | +"add_directive("opentelemetry=off".parse().unwrap())" being done for tracing's |
| 28 | +`FmtSubscriber`. |
0 commit comments