Micrometer: How to filter out CLIENT_ERROR outcomes?

[gastaldi]

I have several 404 and 403 requests (probably caused by failed DoS attacks) like the following:

http_server_requests_seconds_bucket{method="POST",outcome="CLIENT_ERROR",status="403",uri="/invoker/EJBInvokerServlet",le="0.005",} 1.0
http_server_requests_seconds_bucket{method="POST",outcome="CLIENT_ERROR",status="403",uri="/wp-content/plugins/amty-thumb-recent-post/amtyThumbPostsAdminPg.php",le="0.005",} 1.0
http_server_requests_seconds_bucket{method="POST",outcome="CLIENT_ERROR",status="403",uri="/MUP",le="0.005",} 1.0
http_server_requests_seconds_bucket{method="PUT",outcome="CLIENT_ERROR",status="403",uri="/wp-content/plugins/w3-total-cache/pub/sns.php",le="0.005",} 1.0

Is there any way to filter them out to avoid having a gigantic output in /q/metrics?

[quarkus-bot[bot]]

/cc @brunobat (opentelemetry), @radcortez (opentelemetry)

[geoand]

Have you looked at using MeterFilter as mentioned here?

[geoand]

Isn't that just a tag? I don't see why it would not be available to a MeterFilter...

[gastaldi]

I don't see the Response anywhere in the MeterFilter API, but I am surely missing something

[geoand]

Meter.Id allows you to access all the tags does it not?

[gastaldi]

Ah-ha, Thank you! I overlooked that API. I'll give it a shot and post a solution below

[geoand]

🤞🏼

[gastaldi]

Thanks to @geoand I made that work by checking the tags in the Meter.Id API:

	if (id.getTags().contains(Outcome.CLIENT_ERROR.asTag())) {
	    return MeterFilter.deny().configure(id, config);
	}

Here is my complete MeterFilter implementation:

import java.util.Set;
import java.util.concurrent.TimeUnit;

import io.micrometer.core.instrument.Meter;
import io.micrometer.core.instrument.Tag;
import io.micrometer.core.instrument.config.MeterFilter;
import io.micrometer.core.instrument.distribution.DistributionStatisticConfig;
import jakarta.inject.Singleton;

/**
 * Enable histogram buckets for http.server.requests metric
 */
@Singleton
public class HistogramMeterFilter implements MeterFilter {
    private static final Set<String> WRITE_METHODS = Set.of("POST", "PUT", "PATCH", "DELETE");
    private static final Tag OP_READ = Tag.of("op", "r");
    private static final Tag OP_WRITE = Tag.of("op", "w");

    @Override
    public Meter.Id map(Meter.Id id) {
        if (isHttpServerRequests(id)) {
            String uri = id.getTag("uri");
            String method = id.getTag("method");
            if (uri == null || method == null) {
                return id;
            }
            if (WRITE_METHODS.contains(method) && uri.startsWith("/admin")) {
                return id.withTag(OP_WRITE);
            }
            return id.withTag(OP_READ);
        }
        return id;
    }

    @Override
    public DistributionStatisticConfig configure(Meter.Id id, DistributionStatisticConfig config) {
        if (isHttpServerRequests(id)) {
            if (id.getTags().contains(Outcome.CLIENT_ERROR.asTag())) {
                return MeterFilter.deny().configure(id, config);
            }
            return DistributionStatisticConfig.builder()
                    .percentilesHistogram(false) // histogram buckets (e.g. prometheus histogram_quantile)
                    .serviceLevelObjectives(
                            TimeUnit.MILLISECONDS.toNanos(5),
                            TimeUnit.MILLISECONDS.toNanos(10),
                            TimeUnit.MILLISECONDS.toNanos(25),
                            TimeUnit.MILLISECONDS.toNanos(50),
                            TimeUnit.MILLISECONDS.toNanos(250),
                            TimeUnit.MILLISECONDS.toNanos(500),
                            TimeUnit.MILLISECONDS.toNanos(1_000),
                            TimeUnit.MILLISECONDS.toNanos(2_500),
                            TimeUnit.MILLISECONDS.toNanos(5_000),
                            TimeUnit.MILLISECONDS.toNanos(10_000)) //slo slots
                    .build()
                    .merge(config);
        }
        return config;
    }

    private boolean isHttpServerRequests(Meter.Id id) {
        return "http.server.requests".equals(id.getName());
    }
}