"zero-trust" security / networking for training jobs

[astefanutti]

What you would like to be added?

Secure, ideally by default, the data plane of the jobs managed by the training operator.

This would include:

• The creation of NetworkPolicies that prevent ingress traffic to the training jobs, i.e., only intra-job Pod-to-Pod communication is allowed
• The configuration of (m)TLS for Pod-to-Pod communication wherever possible, or provide some documentation on how to achieve it, possibly using external solution like a service mesh for example.

Why is this needed?

In multi-tenant setups, it's important to guarantee tenants are isolated from each other.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[andreyvelich]

Thank you for creating this @astefanutti!
Creating a dedicated page for Kubeflow Trainer operators would be nice: https://www.kubeflow.org/docs/components/trainer/operator-guides/

/area docs
/remove-label lifecycle/needs-triage
/remove-lifecycle stale

[andreyvelich]

cc @kubeflow/wg-manifests-leads @juliusvonkohout from the security point of view.

[juliusvonkohout]

I think istio support with mtls for the trainer component would already cover your needs.

[astefanutti]

I think istio support with mtls for the trainer component would already cover your needs.

I wonder if Istio would support direct GPU interconnect and other high-performance network fabric.

[juliusvonkohout]

I think istio support with mtls for the trainer component would already cover your needs.

I wonder if Istio would support direct GPU interconnect and other high-performance network fabric.

Are sou sure that this even runs trough the Kubernetes network stack? Shouldn't this happen at lower levels?

[tenzen-y]

I think istio support with mtls for the trainer component would already cover your needs.

I wonder if Istio would support direct GPU interconnect and other high-performance network fabric.

In my experience, this is so challenging. HPC cluster often directly uses SRIO-V VirtualFunctions generated by physical interconnect devices.

[astefanutti]

I think istio support with mtls for the trainer component would already cover your needs.

I wonder if Istio would support direct GPU interconnect and other high-performance network fabric.

Are sou sure that this even runs trough the Kubernetes network stack? Shouldn't this happen at lower levels?

You're right that's not always the case. With RoCE, collective communication still flow over Ethernet, possibly on secondary network interfaces.

Having said that, I'd be curious to have some data on how relevant encryption is for collective communication, and if the performance hit isn't just too high vs. the benefits.

[tenzen-y]

I think istio support with mtls for the trainer component would already cover your needs.

I wonder if Istio would support direct GPU interconnect and other high-performance network fabric.

Are sou sure that this even runs trough the Kubernetes network stack? Shouldn't this happen at lower levels?

You're right that's not always the case. With RoCE, collective communication still flow over Ethernet, possibly on secondary network interfaces.

Having said that, I'd be curious to have some data on how relevant encryption is for collective communication, and if the performance hit isn't just too high vs. the benefits.

I know NVIDIA spectrum switch supports offloading collective communication. It might be able to make communications secure and improve collective communication performance, but I did not actually verify the behavior.