[Bug]: Upstream conf resolves to 127.0.0.1:8181 when an additional port is added to an upstream Deploy & Service

[MarkTopping]

Version

4.0.1

What Kubernetes platforms are you running on?

AKS Azure

Steps to reproduce

I've spotted that if you modify an existing deployment in a certain way (more on this below) then sometimes Nginx fails to resolve the correct IP address for the destination and so the conf file contains something like this at the top:

upstream example-minion-example.com-example.com-example-443 {zone example-minion-example.com-example.com-example-443 256k;
        random two least_conn;
        server 127.0.0.1:8181 max_fails=1 fail_timeout=10s max_conns=0;
}

# all content below removed from the example but also completely fine

The problem is this bit: server 127.0.0.1:8181

In my setup and in the environment where I'm testing this I have 2x replicas of the Nginx Ingress Controller pods.
On all occasions where I've replicated this issue, only 1 pod retains this invalid conf. During a config reload I can observe both pods initially with server 127.0.0.1:8181 but shortly after one of the Pods resolves to the correct address. i.e.

upstream example-minion-example.com-example.com-example-443 {zone example-minion-example.com-example.com-example-443 256k;
        random two least_conn;
        server 10.156.67.181:5001 max_fails=1 fail_timeout=10s max_conns=0;
}

In order to replicate this, I deploy a K8s Deployment, Service, and Ingress resource. The deployment spec configures 1 containerPort, and the Service also has 1 port (I start with 80), and the Ingress resource is configured to route all traffic to the Service on port 80.

Everything is fine at this point.

Then to trigger the issue, redeploy the aforementioned resources but the Deployment now configures 2x containerPorts, and the Service has 2 ports (80 + 443 - each mapping to one of the container ports), and the Ingress resource is updated to route all traffic to the Service on port 443.

One of the two Nginx Pods will handle this config just fine and will continue to route to the workload as expected; but the other will fail and start issuing 502 responses. Upon doing so the following logs will be presented (with debug loglevel set):

{"time":"2025-09-23T10:24:20.59820435Z","level":"DEBUG","source":{"file":"verify.go","line":90},"msg":"success, version 27 ensured. took: 128.491111ms"}
2025/09/23 10:24:20 [notice] 18#18: signal 17 (SIGCHLD) received from 204
2025/09/23 10:24:20 [notice] 18#18: worker process 204 exited with code 0
2025/09/23 10:24:20 [notice] 18#18: worker process 206 exited with code 0
2025/09/23 10:24:20 [notice] 18#18: signal 29 (SIGIO) received
2025/09/23 10:24:20 [notice] 18#18: signal 17 (SIGCHLD) received from 207
2025/09/23 10:24:20 [notice] 18#18: worker process 207 exited with code 0
2025/09/23 10:24:20 [notice] 18#18: signal 29 (SIGIO) received
2025/09/23 10:24:20 [notice] 18#18: signal 17 (SIGCHLD) received from 205
2025/09/23 10:24:20 [notice] 18#18: worker process 205 exited with code 0
2025/09/23 10:24:20 [notice] 18#18: signal 29 (SIGIO) received
{"time":"2025-09-23T10:24:20.846495916Z","level":"DEBUG","source":{"file":"endpoint_slice.go","line":33},"msg":"Removing EndpointSlice: example-pod-km6xm"}
{"time":"2025-09-23T10:24:20.846534116Z","level":"DEBUG","source":{"file":"task_queue.go","line":65},"msg":"Adding an element with a key: example-ns/example-pod-km6xm"}
{"time":"2025-09-23T10:24:20.846551817Z","level":"DEBUG","source":{"file":"task_queue.go","line":98},"msg":"Syncing example-ns/example-pod-km6xm"}
{"time":"2025-09-23T10:24:20.846561717Z","level":"DEBUG","source":{"file":"task_queue.go","line":77},"msg":"The queue has 0 element(s)"}
{"time":"2025-09-23T10:24:20.846568217Z","level":"DEBUG","source":{"file":"controller.go","line":1019},"msg":"Syncing example-ns/example-pod-km6xm"}
2025/09/23 10:25:58 [error] 211#211: *403 connect() failed (111: Connection refused) while connecting to upstream, client: 10.157.116.134, server: example.com, request: "GET /api/v1/example-path HTTP/1.1", upstream: "https://127.0.0.1:8181/api/v1/example-path", host: "example.com"
10.157.116.134 - - [23/Sep/2025:10:25:58 +0000] "GET /api/v1/example-path HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36" "xxx.xxx.xxx.xxx,yyy.yyy.yyy.yyy:37220,zzz.zzz.zzz.zzz"

_# Subsequent log lines then repeatedly show the 502 and connect refused entries as more requests come in_

In order to resolve the issue I simply need to trigger a config reload. I've demonstrated this works by either restarting the affected Nginx Pod; restarting the upstream workload; redeploying the upstream workload.

Note that I've so far been unable to replicate this issue when applying subsequent deployments. It seems to only affect the very first deployment which adds the additional port to the Service (or maybe it's the flip from port 80 to port 443 in the Ingress resource that might specifically cause it).

[github-actions]

Hi @MarkTopping thanks for reporting!

Be sure to check out the docs and the Contributing Guidelines while you wait for a human to take a look at this 🙂

Cheers!

[AlexFenlon]

Hi @MarkTopping,

Thank you for reporting the bug,

I will try replicate this bug and get back to you soon.

[AlexFenlon]

Hi @MarkTopping, I have been unable to replicate this at the moment using the instructions given.

You mention you are unable to replicate it on subsequent deployments but this happens everytime its the first deployment?

[MarkTopping]

HI @AlexFenlon
Thank you for your attention.

It doesn't happen on 'every' first deployment - but to be clear, first deployment here I mean the first deployment which adds the extra ports and changes the port which the existing ingress resource maps to - so more of an update than a first deploy.

Today for example we took a change through a couple of environments. I was on hand to restart any instances which might be affected as we were expecting the issue to manifest itself. In the first cluster we deployed the change into, both instances of the ingress controller correctly populated the correct upstream IP and with the revised container port. But on the subsequent cluster which we deployed into ~5 min later the issue repeated itself. One instance had a correct conf file, but the other had 127.0.0.1:8181 as the upstream IP.

I would say that I have about a 80% success rate in replicating the issue when bouncing between 2x deployments with the different port configurations.

I didn't mention it as I didn't think it would be relevant, but I'll add that this change also included some annotations on the ingress resource to enforce the use of TLS with the upstream.

It might help if I show you the ingress resource before and after. _Service names redacted _ :

Before / From:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.org/mergeable-ingress-type: minion
  name: minion-example.sub.example.com
  namespace: example
spec:
  ingressClassName: nginx
  rules:
  - host: example.sub.example.com
    http:
      paths:
      - backend:
          service:
            name: example-app
            port:
              number: 80
        path: /
        pathType: Prefix

After / To:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.org/location-snippets: |
      proxy_ssl_verify on;
      proxy_ssl_verify_depth 3;
      proxy_ssl_name example-app.example.svc.cluster.local;
      proxy_ssl_trusted_certificate /etc/ssl/certs/issuer.crt;
    nginx.org/mergeable-ingress-type: minion
    nginx.org/ssl-services: example-app
  name: minion-example.sub.example.com
  namespace: example
spec:
  ingressClassName: nginx
  rules:
  - host: example.sub.example.com
    http:
      paths:
      - backend:
          service:
            name: example-app
            port:
              number: 443
        path: /
        pathType: Prefix

I won't present the manifests, but to recap that in addition the Service went from 1 port (80) to 2 ports (80 + 443), and the pod spec on the deployment was also updated with an additional port.

Some thoughts from myself and a colleague:

1. Note we're using Helm for the deployment. Helm will deploy a Service before a Deployment. If Kubernetes creates the Service fractionally before the Deployment, then there is a small window during which the new Service will have a port (the new one) which maps to a non-existent container port since the Deployment with the new extra port is not yet in place. Maybe there is something in this which causes the issue.

2. A colleague who has previously contributed to this project wonders if it is an issue that only occurs when you have a HA control-plane. We didn't get chance to discuss much but I guess he was getting at 2x different API Server instances handling requests possible contributing to the issue. We're using Azure and they will deploy the control plane with HA. Could be worth you trying to replicate in that setup if you can and didn't already do so.

[vepatel]

Hey @MarkTopping, i've tried similar example but haven't been able to reproduce this likely race condition, can you please confirm if:

1. You're using helm chart for you application and Ingress deployments,
2. config of your cluster (platform and number of nodes etc)