Easiest way to communicate between multiple instances of a Quarkus app ?

[omasseau]

Hello,

Is there an easy way to communicate between all instances of a Quarkus app ?

In my case I need to send simple commands to all the nodes of the cluster (ex: a command to ask for a local cache invalidation on each node, a command to change the log level of a given category on each node, ...)

Before using Quarkus I was using Wildfly which allowed to easily communicate between each node by using the Wildfly clustering API.

I would like to implement something similar in Quarkus but I don't know what is the best solution.

I know I could use quarkus messaging and spin up a message broker, but for me this is not an ideal solution. I would like some sort of direct communication between the application nodes without having to rely on another dedicated server that must be started and that could be a single point of failure.

For cache invalidation I could use infinispan, but again, I don't want to rely on a distributed cache and have to start a dedicated Infinispan server (And I understood that Infinispan embedded extension is not supported anymore).
Local caches are enough for my use case. I just need a way to send a message on each node to ask them to invalidate their local cache.

So is there any alternatives that don't require having to start an intermediate server just for communicating with other instances ?

I've read about jgroups and it seems that this is wat is used by Keycloak for clustering (Keycloak relies on Quarkus). But I don't really know if there are easier alternatives.

I've also read about Vert.X event bus. It seems there is a way to make it clustered if I look at the possible configuration :

But I don't see any documentation on how to configure the Vert.X cluster.
In Vert.X documentation they talk about the need to have a cluster manager configured for Vert.X : https://vertx.io/docs/vertx-core/java/#_cluster_managers

Which cluster manager does Quarkus use ? Is it possible to choose the implementation of the cluster manager in Quarkus ? How can this cluster manager be associated to Vert.X instance used for the event bus ?

EDIT: In the meantime I've found this to configure Vert.x clustered : https://stackoverflow.com/a/76464489/1879731

Note sure though if Vert.X is the appropiate solution to use to communicate between nodes ?

[emmanuelbernard]

hey Olivier,
Yes indeed we discourage such solutions because state sync should be out of cloud native apps (in backend services). Your case is subtly different in that you want to send signals. But for info think about a case where app instances are going down and up reguarly (say Kubernetes with nodes being decomissioned etc).

In that situation, I think a proper messaging system with proper ack and replay would be useful so that new instances canc atch up on your commands (e.g. the log changes). But if you still want to stick with what you have in mind, yes jgroups (that's the key library for this kind of operation) or the vert.x clustering/bus would be what I would explored.

Unfortunately before of what I said above we have not explored nor documented much these two paths.

@tsegismont might know about Quarkus + vert.x bus for that use case

[tsegismont]

The Vert.x clustered eventbus is not supported on Quarkus, we have helped some users in the past to hack their way around the different issues, but we can't guarantee the setup will work on the long term.

@omasseau there are many possible solutions to your problem, depending on the deployment environment (K8S?) and constraints. Just a few examples:

• the application exposes an endpoint for cache administration
• the application polls an external endpoint
• the applications polls the filesystem (NFS or volume on k8s)

[adomguo]

no deploy in k8s

[mschorsch]

@omasseau The infinispan team is managing a separate quarkus extensions including an embedded infinispan extension.

see

• infinispan-quarkus-embedded
• https://central.sonatype.com/artifact/org.infinispan/infinispan-quarkus-embedded/15.0.6.Final

[adomguo]

hello。 is there has any solution to solve this?

[mschorsch]

There is now an official Quarkus extension for infinispan embedded (https://docs.quarkiverse.io/quarkus-infinispan-embedded/dev/index.html#)