Provide an optional timeout configuration setting to prevent hanging during health checks

[kgiusti]

I've "accidentally" created a health check that failed to terminate. It basically failed into an infinite loop. This prevented the system from recovering via rollback:

[   ***] Job greenboot-healthcheck.service/start running (59min 59s / no limit)
[    **] Job greenboot-healthcheck.service/start running (1h / no limit)
[     *] Job greenboot-healthcheck.service/start running (1h / no limit)
[    **] Job greenboot-healthcheck.service/start running (1h 1s / no limit)

etc.

Would it be possible to add a greenboot configuration option to place a maximum limit to how long the checks can run? If the limit is hit then greenboot could mark the check as failed an rollback.

[arortiz-rh]

This would definitely be possible to implement. I did a little research and found that there is a timeout command you can use to set a time limit on the command. We could also run the scripts in a child process and then automatically kill the child process if it doesn't end itself within a specified amount of time, then log that as an error. I think the latter would be easier to configure and reliably check. There might be other solutions as well, but these are what first came to mind

[kgiusti]

FWIW: here's my test healthcheck c program that sleeps for an hour. It does it's best to mask signals during the sleep which makes it slightly harder to stop.

#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <sys/select.h>
#include <signal.h>


int main(int argc, char *argv[])
{

    sigset_t mask;
    struct timespec timeout = {.tv_sec = 3600};  // default 1 hour

    if (argc > 1) {
        errno = 0;
        long to = strtol(argv[1], 0, 10);
        if (errno) {
            perror("invalid input");
            abort();
        }
        if (to < 1) {
            fprintf(stderr, "input must be > 0 (was %ld)", to);
            abort();
        }
        timeout.tv_sec = (time_t) to;
    }

    if (sigfillset(&mask) != 0) {
        perror("sigfillset failed");
        abort();
    }

    fprintf(stdout, "Sleeping for %ld seconds...", (long) timeout.tv_sec);
    fflush(stdout);
    pselect(0, 0, 0, 0, &timeout, &mask);
    return 0;
}