How to apply Pod Anti-Affinity rules?

[NicolasPires777]

Hello team,

I'm facing an issue with NGINX in my production environment. I am using NGINX Gateway Fabric with over 200,000 concurrent users.

Sometimes, all replicas of my NGINX worker pods are scheduled on a single machine. When this happens, the node hits its connection limit and can no longer respond to new connections, causing some of my clients to lose access.

How can I apply pod anti-affinity rules to the pods? I've already looked through the entire Helm chart but couldn't find an option for it.

P.S. We're using GKE Autopilot *

Thanks for your help!

[nginx-bot]

Hi @NicolasPires777! Welcome to the project! 🎉

Thanks for opening this issue!
Be sure to check out our Contributing Guidelines and the Issue Lifecycle while you wait for someone on the team to take a look at this.

[ciarams87]

Hi @NicolasPires777!

Yes, you can configure pod anti-affinity rules using the Helm chart!

You can set affinity rules in your Helm values under nginx.pod.affinity - see the commented examples in values.yaml. This populates the NginxProxy resource that's automatically created by the Helm chart and attaches to the GatewayClass, applying to all Gateways managed by this instance.

For per-Gateway affinity rules (if needed), you can create separate NginxProxy resources for individual Gateways - see the documentation on manually creating NginxProxies.

Additional options for your use case:

• topologySpreadConstraints is also supported under nginx.pod.topologySpreadConstraints as an alternative approach to distribute pods across nodes/zones

Example configuration:

nginx:
  replicas: 10
  pod:
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app.kubernetes.io/name: nginx-gateway-fabric
                app.kubernetes.io/component: nginx
            topologyKey: kubernetes.io/hostname

It's also worth noting that from the next release (v2.1.0) scheduled for later this month, WorkerConnections will also be a configurable option through the NginxProxy resource and Helm chart.

[NicolasPires777]

Hello @ciarams87 , thank you for the response!

I was looking at the wrong Helm values template, which is why I hadn't noticed the setting. I have now applied the configuration, and it's working. I will monitor it to confirm if it's sufficient for our production environment.

Regarding version 2.1, we are eagerly awaiting it, especially for the worker_connections configuration and the HPA support. Currently, I am using a custom image built from version 2.0.2 with worker_connections set to 65k. This solved the issue up to a point, but we eventually started hitting the GCP machine's resource ceiling. I have a meeting with a Google engineer next week to discuss this machine connection limit as well.

For now, I believe separating the pods onto different machines will resolve the problem. Thank you very much!

Best regards.

[ciarams87]

Great, delighted you're sorted for now @NicolasPires777!

The release is scheduled for the end of next week, so not too much longer to wait 😁