Add node selector configuration for Python SDK (katib.tune)

[hahahannes]

What you would like to be added?

It would be nice to be able to specify on which node Katib jobs would run. This corresponds basically to the node_selector field of a Pod.

For example:

katib_client.tune(
    name=name,
    objective=objective,
    parameters=parameters,
    objective_metric_name="loss",
    max_trial_count=5,
    resources_per_trial={"cpu": "2"},
    node_selector="nvidia.com/gpu.product": "NVIDIA-H100-NVL"}
)

https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/kubeflow/katib/api/katib_client.py#L179

Why is this needed?

If you have a heterogeneous cluster with node configs, e.g. different GPU vendors, models, ... a user might want to specify which node should be used.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[github-actions]

🎉 Welcome to the Kubeflow Katib repo! 🎉

Thanks for opening your first issue! We're happy to have you as part of our community 🚀

Next steps:

• Our team will review your issue soon! cc @kubeflow/wg-automl-leads
• Check out the Contributing Guide and the Kubeflow Contributor Guide
• Join the Kubeflow Slack channels: https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels
• Join the AutoML & Training WG meetings: https://bit.ly/2PWVCkV

Feel free to ask questions in the comments. Thanks again for contributing! 🙏

[haroon0x]

I'd like to work on this.

[harshavardhanreddy06]

/assign me

[google-oss-prow]

@harshavardhanreddy06: GitHub didn't allow me to assign the following users: me.

Note that only kubeflow members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

Details

In response to this:

/assign me

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.