ClusterTrainingRuntime Status

[szaher]

What you would like to be added?

add a more detailed status to the TrainingRuntime and ClusterTrainingRuntime CRDs to reflect image is Ok and list of installed packages.

Why is this needed?

As MLOps Engineer I want to make sure the Training Runtime created is good to be used and provides a list of pre-installed python packages for AI Practitioners.

As AI Practitioner I want to know what packages are installed on the runtime image before starting the training and to decide which additional packages are required to be installed on top of it without running a TrainingJob every time I want to retrieve these information.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[astefanutti]

/area api
/retitle ClusterTrainingRuntime Status

[astefanutti]

/remove-label lifecycle/needs-triage

[astefanutti]

To add on top of the description, listing the packages in a ClusterTrainingRuntime / TrainingRuntime is possible with the TrainerClient in the SDK by calling get_runtime_packages, but this creates a "spurious" TrainJob which is confusing and has some side-effects.

A possible approach would be to have the ClusterTrainingRuntime controller spawning a one-off Pod to exec the command when the ClusterTrainingRuntime is created and populating its status as well as setting a ready condition.

cc @kubeflow/kubeflow-trainer-team

[andreyvelich]

Yeah, using CRD status to expose this information might be tricky, since Python packages usually dynamic, and installed once the TrainJob is created.

[astefanutti]

Yeah, using CRD status to expose this information might be tricky, since Python packages usually dynamic, and installed once the TrainJob is created.

My understanding is these should be the packages already installed in the ClusterTrainingRuntime image(s), not those dynamically added to by the TrainJob.

[andreyvelich]

But for CTR, even if we pin the packages, we can't guarantee the version of transitive dependencies. For example, DeepSpeed: https://github.com/deepspeedai/DeepSpeed/blob/master/requirements/requirements.txt#L5

Actually, I am wrong, since the runtime image is pre-built.

[astefanutti]

Actually, I am wrong, since the runtime image is pre-built.

Absolutely.

[sksingh2005]

Is this issue still open sir ??