Support Azure CNI Overlay

[jkroepke]

Is your feature request related to a problem? Please describe.
AKS offers a new Azure CNI Overlay as alternative to kubenet.

There is documented limitation that AGIC would not work with Azure CNI Overlay.

You can't use Application Gateway as an Ingress Controller (AGIC) for an overlay cluster.

Describe the solution you'd like

I would like to AGIC with Azure CNI Overlay.

[EppO]

AGIC adds the backend's pod IP addresses in the backend pool for a specific ingress as the Application Gateway can't use the Kubernetes service (which uses non-routable IP range outside of the cluster). Azure CNI overlay prevents the Application Gateway to reach the pods directly. I don't think it's a use case that is possible. Using a Kubernetes Service with Nodeport or LoadBalancer type maybe? that's an extra-hop but it might help the connectivity between the application gateway and the pods.

[jkroepke]

@EppO

AGIC works fine with kubenet which based on non routable IP range outside of the cluster. Exactly the same behavior. This can't blocker.

Pretty sure, AGIC gets deprecated soon.

[EppO]

what IPs are using in the App Gw's backend pool in case of kubelet? I'm curious how this works with NAT-ed pod IPs

[jkroepke]

The Pod IPs.

On AKS with kubenet, AKS is managing a Route Table. Each Node gets a /24 network form the POD CIDR and AKS will configure UDRs the subnet route table. Based on that, the. AppGW has the ability to route to the Pod IPs directly.

Ref:

• https://learn.microsoft.com/en-us/azure/aks/configure-kubenet#overview-of-kubenet-networking-with-your-own-subnet
• https://azure.github.io/application-gateway-kubernetes-ingress/how-tos/networking/#with-kubenet

[nicolasdesentryfy]

Is there no plan to support Application Gateway with Overlay ?
It's really odd because overlay is what is being recommended to use as network plugin method.

Here is a conflict:
We are using AKS Overlay with Azure Firewall. In the documentation, Azure Firewall recommends to use Application Gateway to keep the source IP in a X-Forwarded-For. However, we cannot, since Overlay doesn't support Application Gateway.
https://learn.microsoft.com/en-us/azure/architecture/example-scenario/gateway/firewall-application-gateway

Thus, we are blocked.
And we are back to Kubelet. The problem without Overlay is that the CNI plugin doesn't scale enough, ips are limited.

[jkroepke]

@nicolasdesentryfy I expect that once Application Gateway for Container is GA, AGIC may get deprecated anyways. I guess you have to think about an ingress replacements anyways in the future.

[nicolasdesentryfy]

@jkroepke The problem is Azure Firewall forces to use Application Gateway to keep the source IP in X-Forwarded-For , what do you use otherwise ?
Because otherwise, we were already using Nginx ingress.

Cloudflare for example adds a x-forwarded-for as a firewall reverse proxy but if we want to stay in Azure, what can we use to keep the source ip then. I don't see much option.

[jkroepke]

At least with an Azure Firewall, I would agree with you. Of couse, you can also configure the App Gateway manually and point it to an ingress-nginx deployment staticly via Loadbalancer.

[nicolasdesentryfy]

@jkroepke When you say configuring the App Gateway to point to ingress-nginx, what http rule would you configure then ?
Because I thought App Gateway is kinda replacing Nginx ingress and hosts the http redirection rules itself instead of nginx ingress ? Thanks

[ferhatguneri]

I've tried very simple thing to point application gateway to ingress nginx but guess what. It returns 502. Just a basic proxy configuration also doesn't work. Dear Azure Team, Do you think it should have been this much difficult to send a header from firewall? if not why you have it at all ? forget about firewall. Why you CNI does not support your very own Application gateway? Tell us what we need to do to send x-forwarded-for header to Kubernetes pods thru Firewall or Application Gateway while using Azure CNI in AKS?

[jkroepke]

@ferhatguneri If you create an nginx-ingress controller your AKS, then the ingress controller should have an IP from the AKS load balancer.

Create the Application Gateway and manually configure the IP of the loadbalencer as Backend in Application Gateway.

Thanks works fine. If you, check your network rules.

Please mention, that deploy ingress-controller on AKS requires special annotation.

AGIC will be deprecated anyways, once if Application Gaetway for Containers is GA.

[ferhatguneri]

@jkroepke My backend works perfectly even from firewall public IP. When I try to set my Ingress LB IP as backend of Application Gateway it just returns 502. About AGIC even if I want I can't use it. Because I use Azure CNI, Which is not supported. Long story short It shouldn't have been this much complicated to get a header...

[akshaysngupta]

Support for Overlay CNI is planned for next year.

[thuzzik]

@akshaysngupta Do you know if there is a roadmap or an estimated time? I think this missing feature is a big limitation. Thanks

[MaxAnderson95]

Any updates to share on this?