trainer get_job_logs should wait for pods to be running when follow=True #182
Description
What you would like to be added?
When running trainer.get_job_logs(job_id, follow=True), currently if the relevant pod does not exist, or is pending, the function exits without returning anything. Instead when follow=true, it would be good to wait for the pod to be running (potentially with some timeout)
sdk/kubeflow/trainer/backends/kubernetes/backend.py
Lines 330 to 344 in 53e744e
Why is this needed?
When follow=False, the user expects an empty response if no pod logs exist, but a common use case for follow=True is to watch a job immediately after submitting it, in this case the pods are unlikely to exist, meaning with the current behaviour the user must implement their own logic to wait for the pod.
Love this feature?
Give it a 👍 We prioritize the features with most 👍