Add readiness and liveness endpoints #366
Description
HTTP endpoints used to assess the readiness and liveness state of daemons running across the Z3 stack should be available and consistent. Readiness indicates "should this service receive traffic?" while liveness indicates answers "is this service broken? (and potentially needs its orchestrator to restart it?)"
Zallet should expose GET /ready and GET /healthy, ideally running on a dedicated port for security. ("readyz" and "healthz" are even more standard but I would make this match Zebra, and eventually Zaino when it has readiness and liveness)
This is a common ops pattern for container probes and load balancers, and it also keeps us aligned with what Zebra already does.
User stories:
- As an infrastructure engineer responsible for ensuring that zallet is online and reliable for my users, I need to point my orchestration stack at readiness and liveness endpoints in order to know if zallets is running in a healthy state, and to know when it is ready to accept end-user connections via a load balancer.
- As an infrastructure engineer I want to configure my observability tools to notify the team when our zallet deployment is unhealthy or not ready to receive user traffic.
- As a privacy-conscious person wanting to self-host zallet, I want a simple "docker compose up" way to launch an entire Z3 stack, and know that each service within the stack only communicates once it is ready, using readiness and liveness endpoints behind the scenes to define healthy behavior in its docker-compose.yml or similar.
- As an infrastructure engineer exposing zallet RPCs to other teams within my organization, I want to ensure that readiness and liveness endpoints are only accessible on a dedicated port so that I have full control over my zallet deployment's security.
Related: