Add Liveness and Readiness checks to provider services.

[TobyPinfold]

Is your feature request related to a problem? Please describe

Currently there are provider service deployments that are not using readiness and liveness checks. We should implement this checks to improve the service resilience for its clients and dependencies.

Describe the solution you would like

There already exists the required http endpoints for the readiness and liveness checks. These should be correctly implemented to check on their dependencies as required, and configured to be used by k8s via the provider service deployment.

Given the existing KFP provider is going to be removed in the near future, it can remain returning 200 OK responses for all health check calls.

The VAI provider can implement the dependency checks as normally expected.

Describe alternative solutions you have considered

N/A

Additional context

N/A

[chr12c]

Considering kfp provider v1 is going to be reworked entirely into v2, and that the exposure of the deployment's container spec liveness/readiness probe is going to be shared across vai provider and kfp provider, I'd say it's better to do this after v1 is removed rather than try to implement checks on it's various dependencies.

[TobyPinfold]

Considering kfp provider v1 is going to be reworked entirely into v2, and that the exposure of the deployment's container spec liveness/readiness probe is going to be shared across vai provider and kfp provider, I'd say it's better to do this after v1 is removed rather than try to implement checks on it's various dependencies.

Updated the issue to reflect specific changes per each provider, so there is less wasted effort implementing dependency checks on kfp provider

[TobyPinfold]

Just looking into this further, there appears to be no way of checking many of the google managed services connections health, ie GCS and VertexAI. You can create a form of health check on pubsub by asserting the permission are as expected per subscription but its not quite the same as the normal expectations