What is the exact, complete behaviour of health checks?

[rbarrois]

Vector supports various methods to define which health checks to run; but the documentation doesn't describe the actual behaviour when such health checks fail.

Configuring health checks

There are a several methods for configuring health checks execution:

• A per-sink healthcheck.enabled boolean flag;
• A global healthchecks section, with two options:
  • enabled to enable/disable them globally;
  • require_healthy (or --require-healthy) to refuse starting when any health check fails
• The --no-environment flag for vector validate disables health checks execution on validation

Health check impacts

What happens when a sink health check is enabled and fails?

• Unreachable at startup:
  1. If vector validate is part of the startup sequence (ExecStartPre = vector validate in a SystemD unit), the unit won't start (unless --no-environment is set);
  2. If healthchecks.require_healthy is configured, vector will fail to start
  3. If none of those checks are performed (either no vector validate or with --no-environment; no require_healthy), what happens? Will vector retry the health check until it succeeds? See Health checks should retry #16810
• Turns unreachable while running:
  • Are those health checks even run once vector has started?
  • Does Vector retry the health check until it succeeds? How often?
• Unreachable at reload:
  • If a health check fails when the process reloads (newly added sink or existing one), will the vector instance stop?

[StephenWakely]

Currently, the health checks are just run once on startup. There are no retries, we would like to change this as per the issue you have linked.

If require_healthy is configured, Vector doesn't start. Otherwise, Vector continues and attempts to run as normal. Sometimes this works, other times it results in all the events erroring because they cannot be sent to the destination. It very much depends on the circumstances.

On a restart, healthchecks are run for new sinks only. If require_healthy is true and the healthcheck fails, Vector will continue to run but with the old configuration.

Hope that answers your question. Any other questions, please ask.