Remote Development mode with Quarkus v2.9.2 + Kubernetes v1.23.6 - what am I doing wrong?

[NMichas]

Hello, I'm trying to make a simple hello-world example working in remote development mode using Quarkus 2.9.2 and deploying on k8s 1.23.6 (k3s cluster).

I build and deploy my app in one step using:

./mvnw clean package \
   -Dquarkus.container-image.build=true \
   -Dquarkus.container-image.push=true \
   -Dquarkus.container-image.group=nassos \
   -Dquarkus.kubernetes.deployment-target=kubernetes \
   -Dquarkus.kubernetes.deploy=true \
   -Dquarkus.package.type=mutable-jar \
   -Dquarkus.live-reload.password=nassos \
   -Dquarkus.kubernetes.env.vars.QUARKUS_LAUNCH_DEVMODE=true

The above successfully deploys my app in k8s running in dev mode:

I then port forward the pod's 8080 port using kubectl to my local port 5000:

Finally, I try to connect with remote development mode from my machine:

./mvnw quarkus:remote-dev \
   -Dquarkus.package.type=mutable-jar \
   -Dquarkus.live-reload.password=esthesis \
   -Dquarkus.live-reload.url=http://localhost:5000

The client can't connect with the following error:

• I've tried switching logging to TRACE and running Maven on debug mode but no messages seem to be relevant to the above error (in fact, both client & server display no additional logging when trying to connect).
• I've also tried changing the quarkus-deployed service to LoadBalancer (getting an External IP with MetalLB running on my cluster), so that I don' t have to port forward, however I still get the same error.
• I've tried extending the connection timeout to 10 seconds.

Am I missing something obvious? Any ideas please?

[quarkus-bot[bot]]

/cc @geoand, @iocanel

[NMichas]

Deleted my image in Docker Hub, started all over again and now it works 🤷

[iocanel]

I am wondering if the issue is related with image pull policy being set to IfNotPresent or Never. This could cause your cluster ignoring the newer image and sticking to what is already available. A fresh start could possibly cause your cluster to pickup the latest image again.

Not sure if this is the case, but it definitely worths having a look. If my theory stands you will most likely hit the issue down the road.

[NMichas]

Thanks @iocanel. Actually, the default k8s manifests Quarkus creates have imagePullPolicy: Always, so that wasn't the problem.

I did some extra testing and realised where the problem was: When I build my image I'm passing -Dquarkus.live-reload.password=nassos and I was under the impression that this is somehow "baked" into the image; obviously, it's not, you still need to include it in application.yaml.

In my many attempts to make this work I made this change and forgot about it, so I thought there was a problem with k8s not pulling the updated image instead 😊