High memory usage when using multiple custom readiness probes

[brunobastosg]

Hi all. I've been struggling with this problem for quite some time now.

I have a simple Quarkus app (version 2.16.4.Final), written in Kotlin, with a single POST endpoint. This endpoint needs to call 5 other services (using Rest Client Reactive) to function properly. So I created custom readiness probes using UrlHealthCheck to check if they're up.

However, when I do that, the memory keeps increasing until the pod is restarted with an OOMKilled. If I comment the custom readiness probes, then memory usage stays constant.

I also found that if I keep only one custom readiness probe, memory usage also stays constant (apparently). But as soon as I add an extra readiness probe, then it starts increasing.

The graph below shows this behaviour:

Previously, the readiness probes where implemented as below (a base class and the others inherit from it):

// base class
abstract class ExternalServiceHealthCheck : HealthCheck {

    abstract fun getName(): String

    abstract fun getUrl(): String

    override fun call(): HealthCheckResponse {
        return UrlHealthCheck(getUrl())
            .name(getName())
            .requestMethod(HttpMethod.GET)
            .statusCode(Response.Status.OK.statusCode)
            .call()
    }
}

@Readiness
@ApplicationScoped
class FooServiceHealthCheck : ExternalServiceHealthCheck() {

    @ConfigProperty(name = "url.foo.service")
    lateinit var serviceUrl: String

    override fun getName(): String {
        return "Foo Service"
    }

    override fun getUrl(): String {
        return "${serviceUrl}/health"
    }
}

@Readiness
@ApplicationScoped
class BarServiceHealthCheck : ExternalServiceHealthCheck() {

    @ConfigProperty(name = "url.bar.service")
    lateinit var serviceUrl: String

    override fun getName(): String {
        return "Bar Service"
    }

    override fun getUrl(): String {
        return "${serviceUrl}/health"
    }
}

// and so on... there were 5 of these in total

Then I changed the base class to an AsyncHealthCheck to see if it improved things (it didn't).

abstract class ExternalServiceHealthCheck : AsyncHealthCheck {

    abstract fun getName(): String

    abstract fun getUrl(): String

    override fun call(): Uni<HealthCheckResponse> {
        return Uni.createFrom()
            .item(UrlHealthCheck(getUrl())
                .name(getName())
                .requestMethod(HttpMethod.GET)
                .statusCode(Response.Status.OK.statusCode)
                .call())
    }
}

And finally, I moved everything to a single class, and annotated the methods instead of the class:

@ApplicationScoped
class ExternalServicesHealthCheck {

    @ConfigProperty(name = "url.foo.service")
    lateinit var fooServiceUrl: String

    @ConfigProperty(name = "url.bar.service")
    lateinit var barServiceUrl: String

    @ConfigProperty(name = "url.xyz.service")
    lateinit var xyzServiceUrl: String

    @ConfigProperty(name = "url.abc.service")
    lateinit var abcServiceUrl: String

    @ConfigProperty(name = "url.pqr.service")
    lateinit var pqrServiceUrl: String

    @Readiness
    fun checkFoo(): HealthCheck {
        return UrlHealthCheck("$fooServiceUrl/health")
            .name("Foo")
            .requestMethod(HttpMethod.GET)
            .statusCode(200)
    }

    @Readiness
    fun checkBar(): HealthCheck {
        return UrlHealthCheck("$barServiceUrl/health")
            .name("Bar")
            .requestMethod(HttpMethod.GET)
            .statusCode(200)
    }

    @Readiness
    fun checkXyz(): HealthCheck {
        return UrlHealthCheck("$xyzServiceUrl/health")
            .name("Xyz")
            .requestMethod(HttpMethod.GET)
            .statusCode(200)
    }

    @Readiness
    fun checkAbc(): HealthCheck {
        return UrlHealthCheck("$abcServiceUrl/health")
            .name("Abc")
            .requestMethod(HttpMethod.GET)
            .statusCode(200)
    }

    @Readiness
    fun checkPqr(): HealthCheck {
        return UrlHealthCheck("$pqrServiceUrl/health")
            .name("Pqr")
            .requestMethod(HttpMethod.GET)
            .statusCode(200)
    }
}

But the problem still persists.

I thought it could be a bug in Quarkus Smallrye Health. But I added extra custom readiness probes to another existing Quarkus project (also using 2.16.4.Final), and the problem didn't occur.

Oh, and I forgot to mention that when memory increases, both heap and non-heap memory usage stays low. So the issue appears to be in native memory. I enabled native memory tracking but couldn't find anything.

So, what I'd like to know is:

1 - Is anything wrong with the custom readiness probes' code?
2 - Is there anything I could check on my app's code to find the root cause of the problem?

Thanks in advance!

[quarkus-bot[bot]]

/cc @evanchooly (kotlin), @geoand (kotlin)

[geoand]

Very interesting...

Could you add a small sample application that exhibits this behavior so we can check?
That's the easiest and surest way to figure out what's going on

[geoand]

I don't see anything obviously wrong with the code, but that doesn't really mean much :)

[brunobastosg]

Good news. I enabled quarkus.vertx.prefer-native-transport yesterday evening and the problem appears to be solved.

I don't have an explanation, though.

[geoand]

@franz1981 is there any chance this is a leak in Netty?

[mkouba]

Hm, it seems that io.smallrye.health.checks.UrlHealthCheck opens a new HttpURLConnection for every health check. I wonder if it should at least call HttpURLConnection#disconnect() afterwards?

CC @xstefank

[geoand]

Good catch @mkouba!