Expose collected metrics on /q/metrics endpoint

[ShubhamRwt]

Hi,
Based on the chats on the previous discussion, we decided move forward using the JMX to scrape the kafka related metrics.
I am currently using Prometheus JmXCollector class to collect metrics from JMX. I wanted to know how we can expose these collected metrics on the same /q/metrics endpoint where micrometer metrics are exposed?

CC: @ppatierno

[quarkus-bot[bot]]

/cc @ebullient (metrics), @jmartisk (metrics)

[ppatierno]

/cc @brunobat

[ebullient]

There is a separate Quarkus JMX Micrometer registry present in the Quarkiverse project:

https://github.com/quarkiverse/quarkus-micrometer-registry

https://quarkiverse.github.io/quarkiverse-docs/quarkus-micrometer-registry/dev/micrometer-registry-jmx.html

You can have both registries enabled at the same time, but no, the JMX registry does not use the metrics URL. JMX is a different animal.

[ebullient]

Hmm. the way that is implemented, no. It is two separate data aggregations... The micrometer prometheus registry (this is not a quarkus thing) is collecting metrics from a higher level abstraction.. and you've already gone to a specific prometheus collector in strimzi..

well... no. hang on.

you could create a new (not-default) prometheus MeterRegistry that injects the one from strimzi... it would have to do some kind of munging between the two prometheus collectors at scrape time

[ppatierno]

you could create a new (not-default) prometheus MeterRegistry that injects the one from strimzi... it would have to do some kind of munging between the two prometheus collectors at scrape time

@ebullient can you elaborate a little bit more this idea? I have played around with what we have now and how to deal with the PrometheusMeterRegistry instance (injected via Quarkus extension) together with the JmxCollector but didn't find a way to converge metrics from the two different sources in one place.

The hack that I started to have in mind was about using an HTTP filter, to intercept the call on the /q/metrics endpoint and doing the same scraping we do today this way:

public String scrape() {
        StringBuilder sb = new StringBuilder();
        if (jmxCollectorRegistry != null) {
            sb.append(jmxCollectorRegistry.scrape());
        }
        if (meterRegistry instanceof PrometheusMeterRegistry) {
            sb.append(((PrometheusMeterRegistry) meterRegistry).scrape());
        }
        return sb.toString();
    }

but it seems that Quarkus doesn't hit any @ServerRequestFilter when it's a non-application path (i.e. '/q/metrics`) :-(

[ppatierno]

@ebullient I decided to go the same way we go today to get metrics from the two worlds (the MeterRegistry and JMX collector).
I defined our own /metrics endpoint (which is the one we have in the current bridge version) this way:

https://github.com/ppatierno/strimzi-kafka-bridge/blob/add-metrics-quarkus/src/main/java/io/strimzi/kafka/bridge/quarkus/RestBridge.java#L353

then that MetricsReporter component is just "scraping" on both JMX collector and the Quarkus injected MeterRegistry and then builds the output text:

https://github.com/ppatierno/strimzi-kafka-bridge/blob/add-metrics-quarkus/src/main/java/io/strimzi/kafka/bridge/quarkus/MetricsReporter.java#L38

The only drawnback is that the /q/metrics endpoint is still active but getting just the Quarkus metrics. I would like to "override" that endpoint (so our code to be called), or "suppress" or just mapping it to our /metrics endpoint and our code still executed.
I haven't found a good way to do that, so I opened this discussion #32671

[ebullient]

So. What I was trying to suggest is that you replace the default Prometheus Meter Registry with your own..

The default producer will be replaced if you make your own:

quarkus/extensions/micrometer/runtime/src/main/java/io/quarkus/micrometer/runtime/export/PrometheusMeterRegistryProducer.java

Line 20 in b687fbb

public class PrometheusMeterRegistryProducer {

You can also disable construction of the default Prometheus Meter Registry with a property (assuming you add your own instead):

quarkus.micrometer.export.prometheus.default-registry=false

As shown here (with StatsD):

https://quarkus.io/guides/micrometer#creating-a-customized-meterregistry

My thought was that you could extend the PrometheusMeterRegistry (which would catch micrometer metrics) and add what you need to merge in the other metrics. Given you have the producer, you can inject sources and control construction, and then you'd need to override the scrape method that accepts the content type:

https://github.com/micrometer-metrics/micrometer/blob/23804d52a78cc1ce4dd248169c76cf59f0f9d7ad/implementations/micrometer-registry-prometheus/src/main/java/io/micrometer/prometheus/PrometheusMeterRegistry.java#L117

The default handler just calls that scrape method and gets a string back (it doesn't otherwise know what happens):

quarkus/extensions/micrometer/runtime/src/main/java/io/quarkus/micrometer/runtime/export/handlers/PrometheusHandler.java

Line 63 in b687fbb

.end(Buffer.buffer(registry.scrape(acceptHeader)));

[ShubhamRwt]

Hi @ebullient I was trying to implement what you explained. When you say extend the PrometheusMeterRegistry you want me to do something like this --> public CustomMeterRegistryProducer extends PrometheusMeterRegistry, right and also then overriding the scrape as per our requirement? (CustomMeterRegistryProducer is the class I am creating)

[ShubhamRwt]

Hi @ebullient. I am now able to get the required metrics now on /q/metrics endpoint but yeah they are always enable. Anyway I can disable them?

[ebullient]

so now we just need to verify that your registry bean is being used, there are several scrape methods.. so make sure you're overriding the correct one

[ShubhamRwt]

https://gist.github.com/ShubhamRwt/fc7ff38ef1599467f7168ba623919d2a Can you have a look and see if I am going wrong somewhere. Foe ease I am just trying to create PrometheusMeteRegistry and added the code for jmx scraping in scrape method

[ShubhamRwt]

@ebullient ^

[ShubhamRwt]

@ebullient We are using quarkus 2.15.3 and there the doHandler method is like this

    private void doHandle(HttpServerResponse response) {
        response.putHeader("Content-Type", TextFormat.CONTENT_TYPE_004)
                .end(Buffer.buffer(registry.scrape()));
    }

Also I have quarkus.micrometer.export.prometheus.default-registry=false added this to my properties file and still the metrics are there on /q/metrics

[ShubhamRwt]

Also I found out if I do this

    @Produces
    @Singleton
    @Alternative
    public PrometheusMeterRegistry createPrometheusMeterRegistry() {
        return new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
    }

I get the jmx scraped metrics but not the http ones

But if I do this

    @Produces
    @Singleton
    @Alternative
    @Priority(Interceptor.Priority.APPLICATION + 100)
    public PrometheusMeterRegistry createPrometheusMeterRegistry() {
        return new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
    }

I get the http ones but not the jmx ones

[ShubhamRwt]

@ebullient Thanks for the instructions. Iam now able to get both the metrics on /q/metrics endpoint.

[ebullient]

you can change the metrics endpoint from /q/metrics to /metrics if you want. That's a normal configuration property

[ShubhamRwt]

Thanks @ebullient for clarifying all my doubts. Is there any documentation/example which I can refer to for changing the endpoint?

[ShubhamRwt]

@ebullient got the property. Thanks for all the help.

[ShubhamRwt]

Sorry forgot that there was grep being applied

[ShubhamRwt]

Thanks a lot again @ebullient

[ShubhamRwt]

I will be closing this discussion