How to poll a Quarkus app in a container and use a custom health endpoint

[flowt-au]

This is more of a "how to" than a question. Also looking for suggestions for improvement.

I have a Quarkus app with health checking enabled running in a container on port 9000. I can make a health request to that app on q/health from another Quarkus app (call it app2). If the status is UP, it returns the health json and http status 200. If the status is DOWN it returns status 503 (Service not available) and no data. This is the usual thing.

However, my use-case is I am using docker-java to create the container of the target Quarkus app and I need to poll for it being ready so I can start sending requests to it. I want to keep checking the health for (say) 6 seconds and I wanted to receive the health data even if the health status was DOWN (so I can see what didn't work and get some extra data back via "withData").

Here is what I am doing in case others need to so something like this. Any suggestions for improvement are welcome.

Kudos to @xstefank for help via: this SO answer

First, in the target Quarkus app I created a "readiness" check as per the Quarkus guide. Nothing special here. It can be configured to return UP or DOWN for testing.

@Readiness
@ApplicationScoped
public class ReadinessCheck implements HealthCheck {

    // From https://quarkus.io/guides/smallrye-health

    // FYI: The data you supply here gets merged into the complete result that the health check returns.
   
    // Set default true or false for testing
    @ConfigProperty(name = "database.up", defaultValue = "true")
    boolean databaseUp;

    @Override
    public HealthCheckResponse call() {

        HealthCheckResponseBuilder responseBuilder = HealthCheckResponse.named("My container health check");

        try {
            verifyAllIsReady();
            responseBuilder.up()
                .withData("someprop", "somevalue");

        } catch (IllegalStateException e) {
            // eg cannot access the database
            responseBuilder.down()
                    .withData("error", e.getMessage()); // pass the exception message
        }

        return responseBuilder.build();
    }

    private void verifyAllIsReady() {
        if (!databaseUp) {
            throw new IllegalStateException("Database is down");
        }
    }
}

Next, in the target Quarkus app I created a standard REST endpoint called /myhealth to use instead of the q/health endpoint. My custom health endpoint returns the health data and status 200.

import io.smallrye.health.SmallRyeHealth;
import io.smallrye.health.SmallRyeHealthReporter;

// the other imports....

@ApplicationScoped
@Path("/myhealth")
public class CustomHttpHealthEndpoint {
  private final SmallRyeHealthReporter reporter;

  public CustomHttpHealthEndpoint(SmallRyeHealthReporter reporter) {
    this.reporter = reporter;
  }

  @GET
  @Produces(MediaType.APPLICATION_JSON)
  public JsonObject getHealth() {

    // Get the health
    SmallRyeHealth health = reporter.getHealth();

    // Get the data from all health checks, including, in my case, REDIS via the quarkus redis extension, and my ReadinessCheck.java
    JsonObject payload = health.getPayload();

    System.out.println("getHealth() payload =" + payload.toString());

    // We return the json payload, whether the status is UP or DOWN,
    // so your caller can know WHY it is down,
    // and optionally to get the extra data you send via "withData"
    // from your ReadinessCheck.java

    return payload;
  }
}

From a browser, http://localhost:9000/myhealth and http://localhost:9000/q/health both return the health data when the status is UP, but /myhealth returns a status 200 and the data even when the status is DOWN.

In the Quarkus app requesting the health data I created a proxy:

import org.eclipse.microprofile.rest.client.annotation.ClientHeaderParam;
// other imports...

// This class is an INTERFACE which you implement and optionally extend
// to provide the methods that execute the REST requests.

// @Path("/q/health")
@Path("/myhealth")
@ClientHeaderParam(name = "Content-Type", value = "application/json")
public interface OrgHealthProxy {
  // We dont need a pojo, just use JsonObject so we have more flexibility about what we might get back.

  @GET
  @Produces(MediaType.APPLICATION_JSON)
  Uni<JsonObject> GETAsUni();
}

The container is created and run via docker-java. ie in summary:

// Create the container
container = dockerClient.createContainerCmd(imageName)
    .withName(containerName)
    .withEnv(envArgs)
    .withHostConfig(hostConfig)
    .exec();

containerId = container.getId();

// Start (ie run) the container
dockerClient.startContainerCmd(containerId).exec();

Then I call the proxy via this function:

import org.eclipse.microprofile.rest.client.RestClientBuilder;
// other imports...

@ApplicationScoped
public class OrgContainerHealth {

  Boolean isDev = LaunchMode.current().toString().equals("DEVELOPMENT");

  /**
   * Get the health status of the container
   * @param containerName
   * @param portNumberStr
   * @return
   */
  public Uni<JsonObject> getHealthDataFromOrgContainer(String containerName, String portNumberStr) {

    String networkHost = "";
    String portStr = "";
 
    if (isDev) {
      // This works when this app is running in DEV mode.
      // Talk to the Org container via its port number on localhost.
      networkHost = "http://localhost";
      portStr = portNumberStr;
    } else {
      // We are running from inside a container so use the docker network and container name
      networkHost = "http://" + containerName;
      portStr = "8080";
    }

    URI uri = URI.create(networkHost + ":" + portStr);

    OrgHealthProxy orgHealthProxy = RestClientBuilder.newBuilder()
        .baseUri(uri)
        .build(OrgHealthProxy.class);

    // Start the timer
    Instant startReq = Instant.now();

    // The JsonObject of health data we will return
    JsonObject healthJsonObject = new JsonObject();

    // Our timeout in ms
    int healthCheckTimeout = 6000;

    return orgHealthProxy.GETAsUni()
        .onItem().transform(orgHealthJsonObject -> {
          Log.info("getHealthDataFromOrgContainer REST request received response =" + orgHealthJsonObject.toString());

          healthJsonObject.clear();
          healthJsonObject.mergeIn(orgHealthJsonObject, true);

          // Get the overall status string: UP | DOWN
          String status = orgHealthJsonObject.getString("status");

          if (status.equals("UP")) {
            Instant endReq = Instant.now(); // Stop the timer
            long durationMs = Duration.between(startReq, endReq).toMillis(); // Calculate the time taken
            
            // Add these properties for the caller
            healthJsonObject.put("timeoutMs", healthCheckTimeout);
            healthJsonObject.put("timeToStartMs", durationMs);

            // Return the health data as a JsonObject so the caller has the actual data.
            return healthJsonObject;
          } else {
            String errMsg = "ERROR: OrgContainerHealth.getHealthDataFromOrgContainer() the Organisation container is running but not ready. Status=" + status;
            Log.error(errMsg);

            // Throw the error so we retry onFailure()
            throw new Error(errMsg);
          }
        })

        // Log the failure
        .onFailure().invoke(throwable -> Log.info( "getHealthDataFromOrgContainer REST request " + containerName + " failed because: " + throwable))

        // Retry the request every 500ms
        .onFailure().retry().withBackOff(Duration.ofMillis(500), Duration.ofMillis(500))

        // Keep trying for eg 6 seconds
        .expireIn(healthCheckTimeout)

        // The FINAL failure, if / when we expire.
        .onFailure().recoverWithItem(throwable -> {
          Log.info("getHealthDataFromOrgContainer HEALTH request for " + containerName + " finally failed after "
              + healthCheckTimeout + "ms because: " + throwable);

          healthJsonObject.put("timeoutMs", healthCheckTimeout);
          healthJsonObject.put("timeToStartMs", healthCheckTimeout);

          // Add the failure reason in case it is useful.
          healthJsonObject.put("finalFailureReason", throwable.toString());

          // Return the health data we got from the Org container.
          // Your caller can decide what to do.
          return healthJsonObject;
        });

  } // getHealthDataFromOrgContainer

}

So, when it runs, I get a couple of getHealthDataFromOrgContainer REST request for container xxxx failed because: io.vertx.core.http.HttpClosedException: Connection was closed because the container isn't accepting requests yet.

Then, when it is UP, I get the health JSON and return it.

If it is not up within the timeout period, I get the health json and return that by recovering from the failure. My caller can then decide what to do about the failure after inspecting the overall status and the sub-status data.

Anyway, maybe helpful for someone.
Murray

[xstefank]

Hi @flowt-au,

This is again not specific to Health extension, but I'll try to help.

When you use the reactive REST client, all status codes other than 200 are, by default, converted to exceptions. This is why you don't see the DOWN Health JSON in your logs/retries of your Uni. The target application, however, always responds with the JSON, as this is spec required.

Here I tried to reproduce your application - https://github.com/xstefank/call-health.

Here you can see a custom exception mapper that forwards the response - https://github.com/xstefank/call-health/blob/main/src/main/java/org/acme/OrgHealthProxy.java#L30-L38. According to https://quarkus.io/guides/rest-client-reactive#using-clientexceptionmapper if you would like to learn more.

And then you can pick the response (because you know it's the WebApplicationException) in the failure handler - https://github.com/xstefank/call-health/blob/main/src/main/java/org/acme/OrgContainerHealth.java#L95-L98.

Hopefully, this helps. Feel free to ask if you have more questions.

Martin

[flowt-au]

Thanks again for your time and detailed response, Martin, much appreciated. Yes, much better to use the exception mapper which I hadn't thought about. :-)