Metrics-Reporter - openTelemetry connection refused

[NChittimalla]

Hello,

Trying to get the metrics of airbyte using open-telemetry and metric-reporter then airbyte-cron throws an error, and airbyte gets into an unhealthy state.

Details:
Airbyte Version: 0.63
Environment: Docker-compose
Steps to reproduce: I have deployed OpenTelemetry and its UP in prometheus but unable to get metrics.

Relevant log output
airbyte-api-server                | 
airbyte-api-server                |     ___    _      __          __
airbyte-api-server                |    /   |  (_)____/ /_  __  __/ /____
airbyte-api-server                |   / /| | / / ___/ __ \/ / / / __/ _ \
airbyte-api-server                |  / ___ |/ / /  / /_/ / /_/ / /_/  __/
airbyte-api-server                | /_/  |_/_/_/  /_.___/\__, /\__/\___/
airbyte-api-server                |                     /____/
airbyte-api-server                |         : airbyte-api-server :
airbyte-api-server                | --------------------------------------------------
airbyte-api-server                |  Will soon be available at http://localhost:8006/
airbyte-api-server                | --------------------------------------------------
airbyte-api-server                | 
airbyte-cron                      | 
airbyte-cron                      |     ___    _      __          __
airbyte-cron                      |    /   |  (_)____/ /_  __  __/ /____
airbyte-cron                      |   / /| | / / ___/ __ \/ / / / __/ _ \
airbyte-cron                      |  / ___ |/ / /  / /_/ / /_/ / /_/  __/
airbyte-cron                      | /_/  |_/_/_/  /_.___/\__, /\__/\___/
airbyte-cron                      |                     /____/
airbyte-cron                      |         : airbyte-cron :
airbyte-cron                      | 
airbyte-worker                    | 
airbyte-worker                    |     ___    _      __          __
airbyte-worker                    |    /   |  (_)____/ /_  __  __/ /____
airbyte-worker                    |   / /| | / / ___/ __ \/ / / / __/ _ \
airbyte-worker                    |  / ___ |/ / /  / /_/ / /_/ / /_/  __/
airbyte-worker                    | /_/  |_/_/_/  /_.___/\__, /\__/\___/
airbyte-worker                    |                     /____/
airbyte-worker                    |         : airbyte-workers :
airbyte-worker                    | 
airbyte-server                    | 
airbyte-server                    |     ___    _      __          __
airbyte-server                    |    /   |  (_)____/ /_  __  __/ /____
airbyte-server                    |   / /| | / / ___/ __ \/ / / / __/ _ \
airbyte-server                    |  / ___ |/ / /  / /_/ / /_/ / /_/  __/
airbyte-server                    | /_/  |_/_/_/  /_.___/\__, /\__/\___/
airbyte-server                    |                     /____/
airbyte-server                    |         : airbyte-server :
airbyte-server                    | --------------------------------------------------
airbyte-server                    |  Will soon be available at http://localhost:8000/
airbyte-server                    | --------------------------------------------------
airbyte-server                    | 
airbyte-connector-builder-server  | 
airbyte-connector-builder-server  |     ___    _      __          __
airbyte-connector-builder-server  |    /   |  (_)____/ /_  __  __/ /____
airbyte-connector-builder-server  |   / /| | / / ___/ __ \/ / / / __/ _ \
airbyte-connector-builder-server  |  / ___ |/ / /  / /_/ / /_/ / /_/  __/
airbyte-connector-builder-server  | /_/  |_/_/_/  /_.___/\__, /\__/\___/
airbyte-connector-builder-server  |                     /____/
airbyte-connector-builder-server  |         : airbyte-connector-builder-server :
airbyte-connector-builder-server  | 
airbyte-temporal                  | {"level":"info","ts":"2024-03-14T04:24:50.227Z","msg":"Current reachable members","component":"service-resolver","service":"worker","addresses":["172.24.0.2:7239"],"logging-call-at":"service_resolver.go:279"}
airbyte-temporal                  | {"level":"info","ts":"2024-03-14T04:24:50.229Z","msg":"Current reachable members","component":"service-resolver","service":"frontend","addresses":["172.24.0.2:7233"],"logging-call-at":"service_resolver.go:279"}
airbyte-temporal                  | {"level":"info","ts":"2024-03-14T04:24:50.229Z","msg":"Current reachable members","component":"service-resolver","service":"worker","addresses":["172.24.0.2:7239"],"logging-call-at":"service_resolver.go:279"}
airbyte-temporal                  | {"level":"info","ts":"2024-03-14T04:24:50.229Z","msg":"Current reachable members","component":"service-resolver","service":"history","addresses":["172.24.0.2:7234"],"logging-call-at":"service_resolver.go:279"}
airbyte-temporal                  | {"level":"info","ts":"2024-03-14T04:24:50.229Z","msg":"Current reachable members","component":"service-resolver","service":"worker","addresses":["172.24.0.2:7239"],"logging-call-at":"service_resolver.go:279"}
airbyte-temporal                  | {"level":"info","ts":"2024-03-14T04:24:50.229Z","msg":"Current reachable members","component":"service-resolver","service":"history","addresses":["172.24.0.2:7234"],"logging-call-at":"service_resolver.go:279"}
airbyte-temporal                  | {"level":"info","ts":"2024-03-14T04:24:50.230Z","msg":"Current reachable members","component":"service-resolver","service":"matching","addresses":["172.24.0.2:7235"],"logging-call-at":"service_resolver.go:279"}
airbyte-cron                      | 2024-03-14 04:24:50 INFO i.m.c.e.DefaultEnvironment(<init>):168 - Established active environments: [control-plane]
airbyte-worker                    | 2024-03-14 04:24:51 INFO i.m.c.e.DefaultEnvironment(<init>):168 - Established active environments: [control-plane]
airbyte-server                    | 2024-03-14 04:24:51 INFO i.m.c.e.DefaultEnvironment(<init>):168 - Established active environments: [control-plane]
airbyte-cron                      | 2024-03-14 04:24:51 INFO i.a.m.l.MetricClientFactory(initializeOpenTelemetryMetricClient):131 - Initializing OpenTelemetryMetricClient
airbyte-cron                      | 2024-03-14 04:24:51 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):147 - Waiting for temporal server...
airbyte-cron                      | 2024-03-14 04:24:51 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):158 - Waiting for namespace default to be initialized in temporal...
airbyte-cron                      | 2024-03-14 04:24:51 ERROR i.m.s.TaskExceptionHandler(handleCreationFailure):44 - Error creating scheduled task for bean [WorkspaceCleaner] Error instantiating bean of type  [io.airbyte.cron.jobs.WorkspaceCleaner]
airbyte-cron                      | 
airbyte-cron                      | Message: Invalid endpoint, must start with http:// or https://: 
airbyte-cron                      | Path Taken: new WorkspaceCleaner(MetricClient metricClient) --> new WorkspaceCleaner([MetricClient metricClient])
airbyte-cron                      | io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type  [io.airbyte.cron.jobs.WorkspaceCleaner]
airbyte-cron                      | 
airbyte-cron                      | Message: Invalid endpoint, must start with http:// or https://: 

Metric-reporter error logs :

2024-10-08 14:08:34 WARN i.m.s.r.u.Loggers$Slf4JLogger(warn):299 - [b2635dd2, L:/127.0.0.1:53291 - R:localhost/127.0.0.1:8125] An exception has been observed post termination, use DEBUG level to see the full stack: java.net.PortUnreachableException: recvAddress(..) failed: Connection refused

[AurimasNav]

I'm in the same situation it would be nice if you could share your experience.

[NChittimalla]

Check whether airbyte-cron, airbyte-server and airbyte-connector-builder-server getting error with " Message: Invalid endpoint, must start with http:// or https://: " if yes,

Solution : you might need to add the OTEL_COLLECTOR_ENDPOINT=${OTEL_COLLECTOR_ENDPOINT} environment variable to the airbyte-server, airbyte-cron, and airbyte-connector-builder-server services in docker-compose.yaml file.

[AurimasNav]

I'm running it on kubernetes and what confuses me is line in metrics container:

2024-10-10 07:06:15 WARN i.m.s.r.u.Loggers$Slf4JLogger(warn):299 - [b4196ff6, L:/127.0.0.1:49340 - R:localhost/127.0.0.1:8125] An exception has been observed post termination, use DEBUG level to see the full stack: java.net.PortUnreachableException: recvAddress(..) failed: Connection refused

starting to think it is not related to otel, since I can see that at least some metrics are comming to my otel, however another point of confusion is a log in airbyte-server:

2024-10-10 07:05:38 WARN i.a.m.l.MetricClientFactory(initialize):72 - MetricClient was not recognized or not provided. Accepted values are datadog or otel.

it does not seem like it is getting the value from helm values.yaml, must be helm deployment specific issue.

[NChittimalla]

you need to add METRIC_CLIENT=otel or METRIC_CLIENT=datadog in .env file, where airbyte-server uses this variable.
https://docs.airbyte.com/operator-guides/collecting-metrics as they mentioned in the doc.

[AurimasNav]

for helm deployments it should be enough to specify it once under global https://docs.airbyte.com/operator-guides/collecting-metrics#helm-chart-setup-instructions
however it does not look like that value is used for pods, I'm able to configure it by adding env vars for pods as a workaround.
Created an issue for it - #46701

[tautvydas-v]

Hey @AurimasNav , how do you deploy your Airbyte helm charts? Do you have all Airbyte components (worker, server, etc.) listed out as dependencies or just airbyte as a dependency? I've noticed that if you try to deploy only airbyte as a whole, pretty much no values are picked up by values.yaml file (therefore your error), but if you specify each component as a condition - then it works and global vars are picked up.

I'm also trying to test out metrics with OpenTelemetry, but the documentation is pretty scarce