Automate Quarkus re-augmentation in Kubernetes environments

[andreaTP]

Hi all,
in the Keycloak project we are using Quarkus re-augmentation,
to give users a better experience(and the promised fast start-up time) while using the new operator on Kubernetes we want to "mask" this technical detail and perform the augmentation once.

Ref: keycloak/keycloak#9605

I implemented here: https://github.com/andreaTP/poc-operator-mutable-jar a POC to seamlessly swap the mutable jars with the "augmented" once in the final Keycloak deployment, the steps are the following:

• Create a Job with a Keycloak container sharing the same configuration as the main Deployment but running the build && sleep infinity command (to leave the pod live after the augmentation has been performed)
• Wait for the build to finish
• Collect the new mutable-jars from the Pod and store them back as Secrets (one file/one secret to avoid hitting the maximum size of a Secret of 1Mb)
• Delete the Job
• Create a Deployment for the main Keycloak container appropriately mounting the augmented files from the corresponding secrets

Here I'm asking if anyone else have attempted to do the same or something similar, if the approach is sound from the Quarkus perspective and if there is any additional guideline I should take into consideration.

Thanks a lot in advance!

[quarkus-bot[bot]]

/cc @geoand, @iocanel

[geoand]

I don't know of anyone who has done such a thing. @maxandersen @stuartwdouglas, perhaps you know?

[stuartwdouglas]

This sounds fairly hacky, in particular there is no guarantee that files will stay under 1mb.

[pedroigor]

Regardless of the approach, do you think this is a valid requirement for running mutable-jars in kubernetes?

I mean, is it worth investing time in finding a solution, or perhaps have something from Quarkus itself to allow re-augmenting/starting in a single JVM?

I know the latter is going completly against Quarkus's design though ... We are lucky of having mutable-jars :)

[andreaTP]

@stuartwdouglas you are right, I forgot to mention that this doesn't need to be perfect, we can always fallback to run the augmentation before starting, although the user will pay quite long start-up times.

The first attempt we did was using a PVC, it makes the approach less hackish (e.g. no size limit, and simple folder mounting) but the user will incur in additional costs (from the cloud provider).

The problem statement sounds as follows:

• the default Keycloak.X distribution ships an image that needs to perform the augmentation step
• we will suggest the users to build a custom image with the already augmented application
• we want to give to the users a nice "day-0" experience with operator + default image

The last point is compromised by the fact that we need to run the re-augmentation every time a pod starts.
I would be happy to hear about alternative ideas/approaches to solve this 🙂

[pedroigor]

@andreaTP Can't the operator store files using a persistent volume in Kubernetes? Or always mount the lib directory using a PV?

Store as secrets does not seem correct for me too. Mainly due to that limitation.

This sounds kinda similar to when keeping data when using databases so you mount some directories using PVs ...

[andreaTP]

The only reason why I moved away from PV/PVCs is that they are not "automatically" available everywhere and they usually cost money ...

One possibility is to default to Secrets if the file size fits and fallback to PVCs otherwise.

[pedroigor]

I see. It should not take much storage/iops though and a trade-off people need to consider.

I would like to have better options, but this approach is the best I can think of.

[andreaTP]

It should not take much storage/iops though and a trade-off people need to consider.

True, still, in case the Secret is too limited we can simply fail the seamless re-augmentation and point the user to use a custom "pre-augmented" docker image.

I would like to have better options

Me too 😞

[iocanel]

I can't comment on the re-augmentation part, but I will comment on the Secret. Why don't you use a real artifact repository to store artifacts?

[iocanel]

How about nexus ?

[andreaTP]

That's interesting! A nexus instance will automatically solve versioning and more problems, though it will require additional resources/maintenance/etc. for something that we are trying to "hide" from the user.

[iocanel]

An other alternative is to roll your own. I mean an ephemeral m2 repository is something as simple as a servlet backed by a local directory to implement. In the past we've done something similar in Fuse.

[andreaTP]

Sure, but it would need to be backed by a PersistentVolume, where this comment applies: #23056 (reply in thread)

[vmuzikar]

With the seamless re-augmentation, we're basically looking into a nice OOTB experience. And Nexus is yet another thing to configure, another dependency. Which IMO kinda contradicts what we expect from the re-augmentation.

The recommended prod solution to this is and – will remain – the custom image of Keycloak.