[Feature Request] Support Network Plugins Like MACVLAN, IPVLAN,SRIOV,LOOPBACK,DUMMY,VLAN,TAP etc

[infinitydon]

Description:

I’d like to request a new feature for Google DRANET Project to support CNI plugins like macvlan and ipvlan, similar to how Multus CNI enables multi-network interfaces in Kubernetes. Additionally, the feature should support pluggable IPAM solutions (e.g., host-local, static) without requiring code changes.

---

Proposed Features:

1. Support for CNI Plugins:

   • Add native support for CNI plugins like macvlan and ipvlan (also other similar CNI plugins, ipvlan and macvlan seems the most used).
   • Maybe introduce a single ResourceClaim that can be shared by multiple Pods, as the master interface remains the same. This would reduce resource overhead and simplify configuration.

2. Pluggable IPAM Support:

   • Similar IPAM solutions like host-local and static.
   • Investigate support for cluster-wide IPAM solutions like Whereabouts and Spiderpool. If not natively supported, ensure the design is extensible to accommodate such solutions in the future.

3. META CNIs:

   • Similar solutions for things like VRF, SBR etc.

---

Questions:

1. Will this project become an upstream k8s project because I think there is an ongoing similar discussion?
2. If the answer to the above question is yes, will it replace multiple k8snetworkplumbingwg projects like sriov-cni, sriov-device-plugin (DPDK and kernel interfaces), ovs-cni, whereabouts?

---

Looking forward to your feedback!

Thanks

[aojea]

Thanks for the detailed feature request and for your interest in DRANet!

You're right to point out the importance of CNI and its widespread use in Kubernetes. However, DRANet takes a fundamentally different approach to network management. Instead of relying on the CNI framework, which is stateless and limited in scalability and performance, DRANet aims to be more cloud-native and declarative.

Here's a breakdown of the key differences and why we've chosen this direction:

• Declarative API: DRANet provides a declarative API that simplifies network configuration based on Dynamic Resources Allocation. This allows cluster admins to define available resources and lets users request those resources without needing to delve into low-level CNI configurations. This separation of concerns promotes easier management and scalability.
• Stateful Operations: Unlike CNI, which is stateless, DRANet enables stateful operations for better control and reconciliation of network resources using NRI, the Network Runtime Interface (NRI) for tighter integration with container runtimes, offering advantages in performance, efficiency and reliability.
• Focus on Cloud-Native: DRANet is designed with cloud-native principles in mind, aiming for greater flexibility, scalability, and automation compared to traditional CNI-based approaches.

Regarding your specific points:

DRANet support for macvlan, ipvlan, IPAM and others advanced features will be different than CNI:

• macvlan or ipvlan are just different modes of sharing a master device, we have a KEP in kubernetes targeting this use case, see KEP-4815: DRA Partitionable Devices per example.
• IPAM Support and advanced networking features like VRF and SBR can be accomodated in DRANet architecture but as mentioned before, there will be designed in a different way, still need to gather more use cases and feedback about them.

You're correct that there are ongoing discussions around similar projects in the Kubernetes community. DRANet is not intended to replace projects like sriov-cni, sriov-device-plugin, ovs-cni, or whereabouts. These projects serve specific use cases and have their own strengths. DRANet offers a different approach with a focus on declarative configuration, cloud-native integration, and enhanced reliability.

The reason DRANet is not currently in the Kubernetes organization is twofold:

• Existing Project: There's already an existing project within the Kubernetes networking space, and we want to avoid duplication of effort.
• Google-Specific Features: DRANet currently includes some Google-specific features so the Google OSS github projects fits better.

[infinitydon]

Thanks for the response.

How do you see https://docs.google.com/document/d/1pe_0aOsI35BEsQJ-FhFH9Z_pWQcU2uqwAnOx2NIx6OY/edit?tab=t.0#heading=h.fo1yo94x96wg as per multiNet feature that is been worked upon in the upstream, don't you think the efforts are similar?

• IPAM Support and advanced networking features like VRF and SBR can be accomodated in DRANet architecture but as mentioned before, there will be designed in a different way, still need to gather more use cases and feedback about them.

As per use cases, not sure how you want to go about it but these advanced features will be relevant to Telco deployments where advanced routing capabilities will be required.

Also will dranet be able to inject user-space interfaces (DPDK) into PODs?

[aojea]

How do you see https://docs.google.com/document/d/1pe_0aOsI35BEsQJ-FhFH9Z_pWQcU2uqwAnOx2NIx6OY/edit?tab=t.0#heading=h.fo1yo94x96wg as per multiNet feature that is been worked upon in the upstream, don't you think the efforts are similar?

I'm SIG net TL in Kubernetes and I was the one that redirected that effort to use DRA https://docs.google.com/presentation/d/1Vdr7BhbYXeWjwmLjGmqnUkvJr_eOUdU0x-JxfXWxUT8/edit#slide=id.g30fe42b1ba8_0_80 :)

As per use cases, not sure how you want to go about it but these advanced features will be relevant to Telco deployments where advanced routing capabilities will be required.

When I talk with storage people the use cases are clear, they deploy appliances in datacenters that are connected through different vlanes on the kubernetes cluster nodes, and their Pods need to have a leg in the cluster and an additional interface in a Vlan on the storage network.

When I try to get the use cases for telco I only got, "I need to use dpdk or I need to use vrfs or I need to use X", but those are not user stories, those are connectivity solutions to some user stories.
Those solutions were created decades ago with a focus on an operator doing manual configuration and did not evolve since then, and those solutions does not scale or are prepared for a more dynamic world as is today ... if we want to really evolve we need to know these user stories and revisit those solutions and define new workflows, it has to be self service, there should be an administrator that prepare the infra and users that consume it, not an operator that has to do manual operations , instead of using a cli using yaml files, that does not scale

Also will dranet be able to inject user-space interfaces (DPDK) into PODs?

why not, it's been a long time since I had to use dpdk, you are now more familiar on how this model work, we need to expose the resources first so the users can consume them, how will this DPDK be exposed? are virtual devices that need to be created on demand or is there some interface already in the host we'll use?

[aojea]

I'm a bit busy with another high priority tasks, but I just got access to hardware so I can progress on this ... I will update the issue once I do more progress

[cyclinder]

We are also interested in this area, we (spiderpool) have recently been thinking about how to implement current CNI (including macvlan, ipvlan, sriov, etc.) capabilities through DRA. Our requirements include attaching multiple NICs to the Pod (similar to what is currently implemented in Multus), as well as dynamically plugging in NICs with PCI affinity to the GPU, and possibly RDMA device.

NRI + DRA is a nice framework, but we need to implement each CNI in the dra net plugin? This CHANGE is huge, so we plan to still work closely with Multus in early releases.

[aojea]

NRI + DRA is a nice framework, but we need to implement each CNI in the dra net plugin?

I don't feel is huge, lot of things are handled by the hostNetwork plugin that is already implemented,

as well as dynamically plugging in NICs with PCI affinity to the GPU, and possibly RDMA device.

This is the main use case I see these days and the one I will give priority

This CHANGE is huge, so we plan to still work closely with Multus in early releases.

MacVlan and ipvlan are very easy to implement but I didn't find good use cases for people, since most of them require external configuration of the vlans, multus is a good project and there is also https://github.com/kubernetes-sigs/cni-dra-driver , users should choose the one that better fit their needs,

[cyclinder]

MacVlan and ipvlan are very easy to implement but I didn't find good use cases for people

In this scenario, I prefer to read the CNI configuration in the dra-net plugin and directly call /opt/cni/bin/macvlan. For shared RDMA user requirements, we also need to consider how to mount RDMA devices into Pods. Additionally, we must consider the SR-IOV scenario.

This is the main use case I see these days and the one I will give priority

The key issue is that we cannot ensure whether a Pod has been allocated GPU resources when calling the dra-net plugin. We cannot determine the order of allocation between GPU and network card resources. Therefore, we cannot automatically allocate a PCI-affinity network card using the allocated GPU resources.

[aojea]

In this scenario, I prefer to read the CNI configuration in the dra-net plugin and directly call /opt/cni/bin/macvlan

the CNI config presumes you have a lot of previous knowledge (I want to create macvlan with these specific parameters) , that is a model that I don't see can work with autoscaler per example also this a source of problems when somehow the binary does not do what is expected, with hard to troubleshoot problems or depending on timeouts to finish the operation ... as I said in my first answer, I'm trying to understand the use cases this will be needed on more "cloud native" environments where the whole provisioning is more declarative, agree this is well solved with current model for static environments

The key issue is that we cannot ensure whether a Pod has been allocated GPU resources when calling the dra-net plugin

It does, although right now is a bit hacky, since it implies on sneaking onto the allocated claims to the Pod, there is work on this area in the wg-device-management group and this is an specific goal go DRA, to support exactly this (cc: @johnbelamaric )

[aojea]

This CHANGE is huge

adding multiple subinterface mode is not huge, it looks very simple, see in #57

In this scenario, I prefer to read the CNI configuration in the dra-net plugin and directly call /opt/cni/bin/macvlan

I have written in several places what that model limits functionality and is a pain to troubleshoot in production

The key issue is that we cannot ensure whether a Pod has been allocated GPU resources when calling the dra-net plugin.

that is for the end user to configure by matching by attributes

[aojea]

With the evolution of this project I've been evaluating this request and I reached a conclusion that having an uber network DRA driver is not the right direction. When I started this path of using DRA for networking one of the goals was to empower communities and people to create DRA network drivers to solve the existing problems and be able to create a "composable network", every admin should be able to pick different DRA network drivers to build their stack, I've also created a template project so simplify it https://github.com/aojea/dra-network-driver-template

As per today decision, we can always revisit since is hard to predict the future, I'm going to leave DRANET focused on the high performance network problem: RDMA , multinic with CPU alignment, ... that has a clear definition and a considerable scope specially with existing AI demands

For "MACVLAN, IPVLAN,SRIOV,LOOPBACK,DUMMY,VLAN,TAP " stay tuned, I still do not know where or when, but I want also to add SRv6, MPLS and more networking DRA Network drivers ...

Thanks for the feedback and the great discussion