Background job with hibernate-reactive without blocking the eventloop (java - ignore the area/kotlin label)!

[Froidoh]

Similar to #32665 I have a question

We are facing a similar problem.

We have a background job running in a k8s cluster and need to ensure it's only executed once at the same time.

Team decided to go with the proven DB approach instead of redis and use a shedlock in a db.

So whenever the job runs we need to get a lock-entry in the DB.

If there exists a lock that has not yet exceeded the maximum length a lock might live, we return.
If there is a lock that has exceeded it's maximum lifetime we delete it and attempt to create a new one. (race condition, others might try as well)
If there is no lock, we attempt to create a new one (again, race condition, others might try as well)

Once we have a lock, we execute our blocking logic - going to the filesystem, reading in some files and again (!) doing some db inserts via hibernate-reactive.

The processing of the files is what is blocking and depending on the file may take quite a few seconds. During that timeframe, when we block the eventloop, the pod gets killed because it's unresponsive.

To make matters worse we rely on quartz and schedule by some config item - so we can't hardcode something like:

@Scheduled(every = "1m")

We need it to be something like:

    @ConfigProperty(name = "cron.ourJob")
    String cronOurJob;

        var identity = "jobName";

        JobDetail job = JobBuilder.newJob(OurJob.class)
                .withIdentity(identity)
                .build();

        Trigger trigger = TriggerBuilder.newTrigger()
                .withIdentity(identity)
                .withSchedule(cronSchedule(cronOurJob))
                .startNow()
                .build();
        quartz.scheduleJob(job, trigger);

The problem is that we always faced the

HR000069: Detected use of the reactive Session from a different Thread than the one which was used to open the reactive Session - this suggests an invalid integration

et al exceptions... if we block the event loop or pods die like flies.

We haven't found a solution yet. I am currently migrating to quarkus 3 to see if we can make this work somehow.
It's interesting how alien reactive programming feels in Java. Never had any problems in any other language (including Scala and Kotlin) but with vert.x, mutiny and quarkus we keep on struggling.

I am open to switch to something else but hibernate-reactive, for the background-job stuff. Actually I am open for any hints how to resolve this.

Thanks in advance for any ideas!

[quarkus-bot[bot]]

/cc @DavideD (hibernate-reactive), @Sanne (hibernate-reactive), @evanchooly (kotlin), @gavinking (hibernate-reactive), @geoand (kotlin)

[Froidoh]

Thanks quarkus-bot, but I just mentioned kotlin in a sidenote... we use Java

[geoand]

I am pretty sure that Quartz allows you to specify the executor to be used. @machi1990 @mkouba do you guys have any examples where we use the EventLoop as the Quartz executor?

[machi1990]

Thanks @Froidoh we appreciate the details and the context.

There are suggestions from #33340 (reply in thread) and #33340 (reply in thread) can you try one of the two approaches?

[machi1990]

The main reason to use Quartz was that the developer initially doing this had the requirement that the schedule needs to be configurable. As the docs didn't mention this on the "quarkus schedule" page, they decided to use the extension and manually schedule with a cron-expression taken from a config.

Thank you. As of Quarkus 3, it is possible to schedule and configure jobs programmatic with the lightweight scheduler extension too: https://quarkus.io/guides/scheduler-reference#programmatic_scheduling hence my suggestion

[Froidoh]

Thanks @Froidoh we appreciate the details and the context.

There are suggestions from #33340 (reply in thread) and #33340 (reply in thread) can you try one of the two approaches?

I wanna say thanks @machi1990 and the others (don't want to ping all of you!) Thanks a bunch for helping out, for creating such a pleasant framework to work with, for being so responsive and for contributing so much to open source!

I am having a look at the suggested approaches. First step was to migrate to quarkus 3, to make life easier :)

Next step is to find out how to combine all the new knowledge. I still think the easiest approach would probably be to just create a new microservice for the background tasks so we can be blocking and use hibernate-orm without being reactive at all, but I will have a hard time justifying this wrt management because a java pod, even with quarkus, still needs more resources than a node.js one (this is not a diss, that's the metric that is ruling at the moment -.-)

[mkouba]

@Froidoh note that the code from the ProcessQueuedEmailJob#execute() will not work in Quarkus 3 because a HR Panache entity method execution is not offloaded on the current Vert.x context anymore. Details are described in the migration guide.

You need to use either Scheduler.JobDefinition.setAsyncTask() or VertxContextSupport.subscribeAndAwait() from the blocking thread.

[Froidoh]

@mkouba

Thank you

I am pretty sure I need to rewrite this in general - it is blocking by definition... at least the most important step in the middle is.
It's just the DB calls that could be async/non-blocking, but the cpu-bound task doesn't profit from being run on an eventloop... on the contrary, it is the main cause of headache.

Switching between eventloop and background threads - even if easily possible - seems like a waste here. The calls to the DB should be fast, they are not the performance bottleneck.

So to make life easier, to make the code easier, the approach I would take would be to run everything on the WorkerPool. As we established, that's not possible with hibernate-reactive and currently one cannot combine hibernate-reactive and hibernate-orm (blocking) in one quarkus application.

That is my dilemma right now.

I am sure we can come up with some smart workarounds, some code that switched the context in a safe manner... the question then is: if next week there is a new requirement, who can code it? Me? Maybe! Some one else? Not so much, as team-members are already complaining about the "non java"-syntax -.-

[Froidoh]

Currently investigating whether it's smarter (or even possible) to combine hibernate reactive and "hibernate blocking" in one and the same app.

Would mean that we have to double a lot of code to have a sync, blocking and an async, non blocking implementation (basically the coloring problem) but if it works... Right now because of the background jobs the pods die like flies.

The API endpoints that can be totally reactive and non-blocking work like a charm and we are happy with them (unless the background job blocks the eventloop again and all goes down)

[geoand]

Using blocking Hibernate for the background job would certainly make more sense. However at the time being I don't think we allow mixing Hibernate ORM and Hibernate Reactive in the same application (this restriction will likely be lifted in the future).

[Froidoh]

A bummer, maybe I need to extract this to another microservice for the time being then. Let's see. I'll await (pun intended) your judgement. We are obviously doing something very, very wrong here!