Create Infinispan caches/schemas again when Infinispan server killed

[jdussouillez]

Context

I have a Quarkus app (multiple pods) deployed on Kubernetes and a shared cache in a single Infinispan server (with the Infinispan operator) in the same namespace.

The cache schemas are handled by code using @AutoProtoSchemaBuilder. When I deploy a new app release, Quarkus starts and creates the caches/schemas (thanks to use-schema-registration option).

Kubernetes pods:

• myproject-api-1
• myproject-api-2
• myproject-api-3
• myproject-infinispan-1
• myproject-infinispan-config-listener-1

Problem

Sometimes, the Infinispan server pod might be killed for whatever reasons. When this happens, another Infinispan server pod is started, but Quarkus can't use it because it has no schemas/caches. My app handle this so it's not broken but my response times are slower as caching doesn't work anymore.

After the new instance is created, my caches/schemas are not recreated because Quarkus didn't start, it was already running.

Solutions

I would like to fix it automatically. What is the recommended way to do this? Do you know how people handle this?

Here's some solutions I thought of:

• Persistence
  • Is it possible to persist the schemas/cache configurations somewhere? From what I've read, it also caches the data but I don't want to. Caches can store thousands of keys/values and I don't care if I lost them, I just want to be able to get the cache configurations and schemas again to continue using the (now empty) caches like nothing happened.
• Rollout the quarkus app pods
  • This is my current solution. When I see that the cache reads/writes failed because the Infinispan instance changed, I rollout my pods to start Quarkus again
  • I guess it would be a solution to monitor the Infinispan and rollout the Quarkus pods when a new cache server instance is starting
• Externalize the cache configurations/schemas so it's created when the Infinispan server starts and not the Quarkus app.
  • Would work I guess but I would prefer to handle this in my Quarkus app and not in the deployment/Kubernetes configurations
• Tell the Quarkus app to create the caches and register the schemas again if a "cache not found" exception is thrown
  • This is what I tried to do but without success (see the reproducer). The code hangs and I need to kill the app. I tried multiple things but didn't find any production-usable solutions.
  • Is there a way to manually run what the use-schema-registration does in an already running Quarkus app?

Reproducer

https://github.com/jdussouillez/quarkus-infinispan-cache-reset

---

TLDR: What is the best way to recreate Infinispan caches/schemas after the server was killed, if possible from the Quarkus app itself without restarting it, and without persisting all cached data in a database/filesystem?

[quarkus-bot[bot]]

/cc @geoand (kubernetes), @gwenneg (cache), @iocanel (kubernetes), @karesti (infinispan), @wburns (infinispan)

[karesti]

Hi @jdussouillez

You can register the schema again in case you've lost it on a server reboot. The schema needs to be uploaded to the server, this is done with the remote cache manager. However, the API is not great now but we are providing good changes for 15.
https://github.com/infinispan/infinispan-simple-tutorials/blob/main/infinispan-remote/query/src/main/java/org/infinispan/tutorial/simple/remote/query/InfinispanRemoteQuery.java#L46

I'm going to create a new issue to try to cover this case, so you don't need to handle it yourself.

Another option is, if you are using the Infinispan Operator, pass the schema to the server so it's created on startup. The downside of this is that you need to sync the schemas for each release, disabling the "upload" from the app. This is a good option if your schema tends to be stable enough.

[jdussouillez]

Hi @karesti, thanks for the details!

The schema needs to be uploaded to the server, this is done with the remote cache manager.

Ok so I was on the right track but it felt like a quick workaround so I was wondering if there were better alternatives.

I need to try again to see if I can get a working implementation with Quarkus on top on Infinispan.

---

Another option is, if you are using the Infinispan Operator, pass the schema to the server so it's created on startup. The downside of this is that you need to sync the schemas for each release, disabling the "upload" from the app. This is a good option if your schema tends to be stable enough.

Yes this is what I thought of when I said "Externalize the cache configurations/schemas so it's created when the Infinispan server starts and not the Quarkus app". The procedure would be to disable the use-schema-registration and managing the schemas server side, outside the Quarkus app scope.

This will be my last solution as I would prefer the way it is: let the Infinispan client (= my Quarkus app) handle the schemas/caches instead of delegating this to the operator/devops team. We already do this with Liquibase and other connectors (Quarkus handle migrations at startup), it feels easier to maintain for the dev team when all this is done by Quarkus.

[jdussouillez]

I need to try again to see if I can get a working implementation with Quarkus on top on Infinispan.

FYI: I managed to get a working recovery from the app code itself. See branch code-recovery. I don't really know why but I had to duplicate the RemoteCacheManager instance (the one injected by Quarkus) and use this new cache manager to get it working (code).

[karesti]

@wburns or any other core team member might have additional ideas on this. I've just ping them in our chat

[ryanemerson]

@jdussouillez Schemas should survive server restarts. Can you share your Infinispan CR definition?

[ryanemerson]

@jdussouillez If you're deploying Infinispan via the Operator, then a StatefulSet is used for the deployment. So on pod restart the same PVC should be mounted to the pod and the cache definitions in the global state and schemas stored in ___protobuf_metadata cache should be reloaded from disk.

The only scenario where ^ shouldn't be the case is if you have explicitly configured Ephemeral Storage via spec.service.container.ephemeralStorage=true in your Infinispan CR

[jdussouillez]

@ryanemerson Oh OK I get it now. Thanks!

Option ephemeralStorage is enabled. I didn't know this, I will check with the devops team to see what solutions we have to store the cache definitions somewhere.

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: {{ ... }}
  annotations:
    {{- ... }}
    infinispan.org/podTargetLabels: {{ ...}}
    infinispan.org/podTargetAnnotations: {{ ... }}
  labels:
    {{- ... }}
spec:
  replicas: {{ ... }}
  service:
    type: DataGrid
    replicationFactor: 1
    container:
      storage: 1Gi
      ephemeralStorage: true
  autoscale:
    maxMemUsagePercent: 70
    maxReplicas: 1
    minMemUsagePercent: 30
    minReplicas: 1
  container:
    cpu: {{ ... }}
    memory: {{ ... }}
  security:
    endpointSecretName: {{ ... }}
    authorization:
      enabled: true

[karesti]

@ryanemerson for the issue I opened in Quarkus, maybe I just need to provide some insights on this ? In the docs.

[ryanemerson]

This is specifically related to the semantics of Operator deployments and is non-default behaviour that is documented in the Operator documentation:

Defines whether storage is ephemeral or permanent. Set the value to true to use ephemeral storage, which means all data in storage is deleted when clusters shut down or restart. The default value is false, which means storage is permanent.

The same issue would apply for non-quarkus clients as well, so IMO this shouldn't be part of the Quarkus docs.

[karesti]

Yeah, but I can point to that from the docs, in case this is reproduced again