Elasticsearch high CPU usage in idle state

[serhiynovos]

Hi. I have a few services build as a native executable. One of this services is using RestHighLevelClient to write or update indexes on elastic search. I noticed that this service has higher CPU usage in idle state.

dependencies that this service has

    implementation("io.quarkus:quarkus-smallrye-reactive-messaging-kafka")
    implementation("io.quarkus:quarkus-kotlin")
    implementation("io.quarkiverse.quarkus-elasticsearch-reactive:quarkus-elasticsearch-reactive:0.2.0")
    implementation("io.quarkus:quarkus-jackson")
    implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8")
    implementation("io.quarkus:quarkus-arc")
    implementation("io.quarkus:quarkus-smallrye-health")
    implementation("io.quarkus:quarkus-micrometer-registry-prometheus")
    testImplementation("io.quarkus:quarkus-junit5")

Other services has similar dependencies except elasticsearch client. Do you know what can be a problem ?

[quarkus-bot[bot]]

/cc @evanchooly (kotlin), @geoand (kotlin), @gsmet (elasticsearch), @loicmathieu (elasticsearch), @yrodiere (elasticsearch)

[gsmet]

If the scale at the left of your graph is actually meaningful, I wouldn't call 0.3% high.

The Elasticsearch client actually does things in the background such as checking it can still contact the node and do some node discovery. I wouldn't be surprised if it was this machinery creating the load.

If you're really curious about it, start your application in JVM mode and do some profiling of the idling phase.

[serhiynovos]

Grafana is configured not right and it's not 0.3, it's 3%. Yes it's not a high usage but in compare with other services with are connected to amqp and kafka it's high. Just wonder if it's normal elasticsearch client behavior or not, have never had before experience with it

[loicmathieu]

Discovery is disabled by default https://quarkus.io/guides/elasticsearch#quarkus-elasticsearch-restclient-lowlevel_quarkus.elasticsearch.discovery.enabled

If you enable it, it can explain it.

But anyway, 2% more CPU is not a lot, did you try to restart the service ?
Also, did you check that heap memory usage is not too high ?

If the service running Elasticsearch client uses more memory than the one that don't it can also explains things.

A few percent of CPU usage can be caused by a lot of things so it's very difficult, you can try to make a profile of your application but you may be chasing down something that is not an issue.