Quarkus 3 with Java 19 Virtual Threads for Kafka Consumer

[sandeep540]

I wrote a small Quarkus 3 App with Java 19 for Kafka Consumer with 5 consumer threads

https://github.com/sandeep540/quarkus-kafka-sink

Here in the sink method, I have annotated with @ Run on Virtual Thread - https://github.com/sandeep540/quarkus-kafka-sink/blob/main/src/main/java/org/acme/sink/MyReactiveMessagingApplication.java#L17

is this the correct approach? When I analyse the same I see 5 threads, can this be run 1 thread (5 Virtual)?

I took the Virtual Thread inspiration from

https://piotrminkowski.com/2022/10/06/quarkus-with-java-virtual-threads/

Please share your views, as this is my first experience with Java 19 - Virtual Threads

[quarkus-bot[bot]]

/cc @alesj (kafka), @cescoffier (kafka), @ozangunalp (kafka)

[cescoffier]

Kafka does not support virtual threads yet. I will be added in a later version (3.x), but I can't say which one. The main issue is related to the kafka expected behavior which requires tracking messages and handling acknowledgment/ commit properly.

Typically, the polling thread cannot be a virtual thread, and the consumer could with some work to make sure that we do not pin or monopolize the carrier thread.

On the producer side, the issue is a bit simpler. However, using a virtual thread to send is also problematic as it pins. Note that this issue does not happen on Quarkus, as our sender is asynchronous.

[sandeep540]

Thanks @cescoffier for detailed explanation

[sandeep540]

Also, In this example author is using Kotlin Coroutines to parallel to scale each consumer in a group - https://blog.rockthejvm.com/optimizing-kafka-clients-a-hands-on-guide/#73-partitions--consumers

I was trying to achieve the same with Quarkus 3+