http_x_forwarded_for and remote_addr headers does not contain the actual ip after injecting ingress with linkerd

[glen-uc]

In order to try the retries and timeout feature, i injected our ingress-nginx controller with side car as per mentioned in https://linkerd.io/2/tasks/using-ingress/#nginx

  nginx.ingress.kubernetes.io/configuration-snippet: |
  proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
  grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;

Everything seemed to work fine at first, but when i saw our application log http_x_forwarded_for and remote_addr headers did not contain the actual client ip but containerd 127.0.0.1, any idea how to avoid this?

[joewilliams]

We are also seeing this behavior. We are using gloo and linkerd and both with and without ingress mode our x-forwarded-for headers end up containing internal IP addresses rather than the IP of the client. Inspecting our logs the header is set properly coming out of envoy inside gloo but is overwritten it seems by linkerd when it egresses the gloo proxy pod, so our services only end up seeing the internal gloo ip addresses.