Help with jaeger OTLP metrics

[roiar]

Hello Bruno!
Hope you are doing well!

I am trying to explore micrometer integration with Quarkus
I used as bases the example from https://quarkus.io/guides/telemetry-micrometer-tutorial
Included dependency
<dependency> <groupId>io.quarkiverse.micrometer.registry</groupId> <artifactId>quarkus-micrometer-registry-otlp</artifactId> <version>3.2.4</version> </dependency>

In application.properties
quarkus.micrometer.registry-enabled-default=true quarkus.micrometer.export.otlp.enabled=true quarkus.micrometer.export.otlp.get-enabled=true quarkus.micrometer.export.otlp.url=http://localhost:4318/v1/metrics

In order to have monitor tab enabled on a local jaeger deployment I used the docker-compose from here Made also a small addition in port binding to have 4318 also available

Unfortunately the prime number metric is not presented on the monitor dashboard. Only Operations metrics of the application, regarding the latency, request rate etc...

Also http://localhost:4318/v1/metrics responds 404 page not found

Could you please help me?

Thanks in advance
Best Regards,
Roi Arapoglou

[brunobat]

Sure!
Some questions:

1. Which micrometer registry are you using to export your metrics?
2. http://localhost:4318 is the port for the HTTP protocol in the OTel Collector. The usual is using GRPC and port 4317.
3. What is the "prime number metric"? Is it a custom metric that you are creating in the code? Can you please provide a code snipet?
4. If you see "Operations metrics of the application, regarding the latency, request rate etc..." looks like you are already receiving data.

[roiar]

Inside pom

I have taken the example from here for the counters
https://quarkus.io/guides/telemetry-micrometer-tutorial#add-a-counter

In Jaeger monitor tool I cannot see the code counters

Inside Prometheus

And when I am trying to search Prometheus counters

[brunobat]

There are multiple problems happening here.
Let's leave the counter problem for later and solve first the connection issues.

If you want to use an OTLP registry you must not use the quarkus-micromenter-registry-prometheus. I assume you want to send all your data to an OTLP Collector, right?
If that's the case, is the collector that should forward the data to prometheus... using scraping of the latest Prometheus that is able to receive push data from the collector.

[roiar]

Right I want to send data to OTLP Collector, I removed the Prometheus registry dependency. Now The http://localhost:8080/q/metrics responds 404 not found

Here are my docker instances

And here is my application properties setup

Also after a while there is a warning log

14:45:30 WARN traceId=, parentId=, spanId=, sampled= [io.mi.re.ot.OtlpMeterRegistry] (otlp-metrics-publisher) Failed to publish metrics to OTLP receiver: java.net.ConnectException: Connection refused: no further information
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)

[brunobat]

There are many things to fix...
Please check this project as a reference: https://github.com/brunobat/quarkus-observability-demo/tree/main/quarkus-observability-demo-full
Especially: https://github.com/brunobat/quarkus-observability-demo/blob/main/quarkus-observability-demo-full/quarkus-observability-demo-full-legume/src/main/resources/application.properties
Please ignore everything Openshift related.
Also, this docker compose also as a reference: https://github.com/brunobat/quarkus-observability-demo/tree/main/docker-compose/basic_with_prometheus

[roiar]

Based on this docker compose the docker instances created are

Unfortunatelly the monitor tab in Jaeger is disabled

and still there are error logs

[brunobat]

Mind that is you are using the configs I provided above, The metrics will show up on the logs. No idea why Jaeger UI is disabled.
You will need to share your docker compose and Jaeger config.
otel-collector is the name of the machine... if all ports are shared on the local network, try localhost or 0.0.0.0

[roiar]

I ran docker-compose up on folder https://github.com/brunobat/quarkus-observability-demo/tree/main/docker-compose/basic_with_prometheus

I suspect that it has to do with configs they refer here

Could you please provide me with print screens where are the metrics printed and how do we query the Prometheus?

[brunobat]

As you can see on this line:

quarkus-observability-demo/docker-compose/basic_with_prometheus/otel-collector-config.yaml

Line 40 in b858978

exporters: [logging, prometheus]

These will be sent to the colector log (see the container output log) and prometheus, if properly configured. Sorry, but you'll have to research the prometheus queries.