[BUG] Docker HEALTHCHECK can increase load when server is busy

[adamretter]

The Docker image is configured with the following HEALTHCHECK to determine if the Elemental server is healthy:

HEALTHCHECK CMD [ "java", \
    "org.exist.start.Main", "client", \
    "--no-gui",  \
    "--user", "guest", "--password", "guest", \
    "--xpath", "system:get-version()" ]

The Docker HEALTHCHECK documentation says that this by default will be run every 30 seconds. If Docker does not receive a response within a 30 second timeout , it will retry. If there are 3 failures the container will be marked unhealthy.

The above HEALTHCHECK makes an XML-RPC call from the Java Admin Client to the Elemental server.

The issue is that there does not appear to be a timeout on the XML-RPC call, and so during times of high-load multiple XML-RPC calls can be opened by Docker HEALTHCHECK, and remain open to the server.

To prevent these XML-RPC calls staying open, we should add a configurable timeout to the XML-RPC call, and we should perhaps consider adding a dedicated 'health-check' command to the Java Admin Client that takes care of correctly setting the XML-RPC call timeout.