Use /health/live and /health/ready probes in supported servers

[ryanemerson]

Requires: #2277

Closes #2384

[Crumby]

'TestCustomImage' seems to be failing

[ryanemerson]

infinispan/infinispan#14869 required to fix the TestGracefulShutdownWithTwoReplicas failures.

[ryanemerson]

PR is on hold as we have decided to postpone reconsider the infinispan/infinispan#14802 implementation.

[pminz]

Hi @ryanemerson
I see infinispan/infinispan#14802 issue is closed now, so are we good to move ahead with this PR ?
We are targeting this for our next release.

[ryanemerson]

This PR should be fine to merge into main, however I think it would be prudent for us to first resolve the TestGracefulShutdownWithTwoReplicas failure on main as I believe this is related to the recent changes to the server lifecycle introduced in 16.0. \cc @Crumby @jabolina

[jabolina]

Hey, @ryanemerson, can you point me to some logs or a run with the failure I can look to, please?

[ryanemerson]

@jabolina The failure occurs on every test run on the main branch. Here's the latest: https://github.com/infinispan/infinispan-operator/actions/runs/16445456480

You can see the logs by downloading the infinispan-test-logs artifact.

[jabolina]

I've created infinispan/infinispan#15404 to handle a problem I found while running the graceful shutdown locally. I'll still investigate more, as the logs in the test shows a different exception:

13:48:03,073 ERROR () [org.infinispan.manager.DefaultCacheManager] ISPN000710: Failed to initialize global registry
org.infinispan.commons.CacheConfigurationException: ISPN000438: Cache with name 'memcachedCache' is defined more than once!
	at org.infinispan.configuration.parsing.CacheParser.checkDuplicateCacheName(CacheParser.java:918)
	at org.infinispan.configuration.parsing.CacheParser.getConfigurationBuilder(CacheParser.java:902)
	at org.infinispan.configuration.parsing.CacheParser.parseReplicatedCache(CacheParser.java:782)
	at org.infinispan.configuration.parsing.Parser.parseCaches(Parser.java:792)
	at org.infinispan.configuration.parsing.Parser.parseContainer(Parser.java:696)
	at org.infinispan.configuration.parsing.Parser.readElement(Parser.java:93)
	at org.infinispan.configuration.parsing.ParserRegistry.parseElement(ParserRegistry.java:213)
	at org.infinispan.configuration.parsing.ParserRegistry.parse(ParserRegistry.java:196)
	at org.infinispan.configuration.parsing.ParserRegistry.parse(ParserRegistry.java:182)
	at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.load(OverlayLocalConfigurationStorage.java:145)
	at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.loadAllCaches(OverlayLocalConfigurationStorage.java:129)
	at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart(GlobalConfigurationManagerImpl.java:126)
	at org.infinispan.CoreModule.cacheManagerStarted(CoreModule.java:19)
	at org.infinispan.factories.GlobalComponentRegistry.modulesManagerStarted(GlobalComponentRegistry.java:301)
	at org.infinispan.factories.GlobalComponentRegistry.postStart(GlobalComponentRegistry.java:275)
	at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:703)
	at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:669)
	at org.infinispan.server.SecurityActions.lambda$startCacheManager$0(SecurityActions.java:42)
	at org.infinispan.security.Security.doPrivileged(Security.java:62)
	at org.infinispan.server.SecurityActions.startCacheManager(SecurityActions.java:44)
	at org.infinispan.server.Server.lambda$startCacheManager$7(Server.java:552)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
	at java.base/java.util.concurrent.ThreadPerTaskExecutor$TaskRunner.run(ThreadPerTaskExecutor.java:314)
	at java.base/java.lang.VirtualThread.run(VirtualThread.java:329)