Enable OpenTelemetry instrumentation

[devpro]

Description

Would it be possible to enable OpenTelemetry Go instrumentation in ngrok Ingress Controller?

OpenTelemetry, the most trending project of the CNCF after Kubernetes for many years, is the new standard for observability. It is currently the best way to send valuable information (3 pillars of observability) to the OpenTelemetry collector that can then share this data to any kind of data platform (such as Grafana, Elastic Stack, Splunk, etc.) just by configuration, in a decoupled and resilient way, avoiding any complicated solution or vendor lock-in.

Use Case

Be able to gather all observable data (logs, metrics, traces) directly from the containers, following the same standard, and be able to easily monitor, investigate, take action from dashboards where all information of the same system will be regrouped.

Related issues

No response

[salilsub]

Investigating, would need spec before development.

[alex-bezek]

Hello @devpro , we technically do support Open Telemetry metrics in this controller. There are some very basic docs here https://ngrok.com/docs/k8s/deployment-guide/#metrics saying to scrape port 8080 and we expose just the basic metrics included in kubebuilder. We’d like to add more custom metrics here as we have use cases and would document them as we add them. Is there anything else you are looking for in particular?

[devpro]

Hello @alex-bezek, thank you for your reply. I really like ngrok :)
Ideally, it would be nice to also have logs and traces, to cover the 3 pillars of observability, and get distributed tracing from the first entry in ngrok (link it with the calls to the apps/services).
I was thinking about having a way to configure an exporter like OpenTelemetry Collector URL so ngrok sends all this data with OpenTelemetry standard (through OTLP).

[alex-bezek]

Gotcha, thanks for the information @devpro . Today our logs and metrics FYI are just about the operator itself. Logs about how its creating/updating resources, and metrics from kubebuilder about the number of reconcile loops that have been run and whatnot. You are right, we don't have any distributed tracing setup.

What kind of traces would you be looking for that would provide the most value for you? For example, similar to how we expose metrics about the internal controllers of the operator, we could also build traces of our controller reconcilers. Or are you looking for traces about requests going through online tunnels and what components in the ngrok could, the controller, and your backend apps/services that respond.

[devpro]

Thank you for the detailed feedback!

I would be indeed very interested to know about the requests going through the ngrok Controller and ideally corelate them with the Kubernetes applications that are processing the requests. For supervision, investigation, optimization, distributed tracing is awesome and OpenTelemetry makes it very easy (a standard with span_id, parent_id).

For instance, currently I can see from my web app, with a timed flow, the different calls made to microservices, databases (like MongoDB), messaging (like Kafka). Being able to get the request from the very start of Kubernetes (Ingress Controller) would be great to get the full picture.