Traefik loses Client IP when injecting Linkerd

[BenB196]

Hi All,

I am trying to use linkerd with traefik and am running into a issue, where is I inject traefik with ingress, it causes all IPs in traefik logs to show up as 127.0.0.1. If I un-inject linkerd from traefik I see the client IPs in the traefik logs again.

I've also tried injecting linkerd via enabled as well and got the same issue.

I believe this is either an issue with how linkerd/traefik interacts, because if I replace traefik with nginx, everything works fine.

Any help on this would be greatly appreciated as I've been stuck on this issue for a while now.

See below for deployment specific stuff.

Linkerd - 2.10.0
Traefik - 2.4.8

Traefik deployed via Helm:

additionalArguments: []
additionalVolumeMounts: []
affinity: {}
autoscaling:
  enabled: false
deployment:
  additionalContainers: []
  additionalVolumes: []
  annotations: {}
  enabled: true
  imagePullSecrets: []
  initContainers: []
  kind: Deployment
  labels: {}
  podAnnotations:
    config.linkerd.io/proxy-cpu-limit: 300m
    config.linkerd.io/proxy-cpu-request: 100m
    config.linkerd.io/skip-inbound-ports: '80,443'
    linkerd.io/inject: ingress
  podLabels: {}
  replicas: 1
env: []
envFrom: []
experimental:
  kubernetesGateway:
    appLabelSelector: traefik
    certificates: []
    enabled: false
  plugins:
    enabled: false
globalArguments:
  - '--global.checknewversion'
  - '--global.sendanonymoususage'
hostNetwork: false
image:
  name: traefik
  pullPolicy: IfNotPresent
  tag: ''
ingressClass:
  enabled: false
  isDefaultClass: false
ingressRoute:
  dashboard:
    annotations: {}
    enabled: true
    labels: {}
logs:
  access:
    enabled: true
    fields:
      general:
        defaultmode: keep
        names: {}
      headers:
        defaultmode: drop
        names: {}
    filters: {}
  general:
    level: ERROR
nodeSelector: {}
persistence:
  accessMode: ReadWriteOnce
  annotations: {}
  enabled: false
  name: data
  path: /data
  size: 128Mi
pilot:
  enabled: false
  token: ''
podDisruptionBudget:
  enabled: false
podSecurityContext:
  fsGroup: 65532
podSecurityPolicy:
  enabled: false
ports:
  traefik:
    expose: true
    exposedPort: 9000
    port: 9000
    protocol: TCP
  web:
    expose: true
    exposedPort: 80
    port: 8000
    protocol: TCP
  websecure:
    expose: true
    exposedPort: 443
    port: 8443
    protocol: TCP
    tls:
      certResolver: ''
      domains: []
      enabled: false
      options: ''
priorityClassName: ''
providers:
  kubernetesCRD:
    enabled: true
    namespaces: []
  kubernetesIngress:
    enabled: true
    namespaces: []
    publishedService:
      enabled: false
rbac:
  enabled: true
  namespaced: false
resources: {}
rollingUpdate:
  maxSurge: 1
  maxUnavailable: 1
securityContext:
  capabilities:
    drop:
      - ALL
  readOnlyRootFilesystem: true
  runAsGroup: 65532
  runAsNonRoot: true
  runAsUser: 65532
service:
  annotations: {}
  enabled: true
  externalIPs: []
  labels: {}
  loadBalancerSourceRanges: []
  spec:
    externalTrafficPolicy: Local
  type: LoadBalancer
serviceAccount:
  name: ''
serviceAccountAnnotations: {}
tlsOptions: {}
tolerations: []
volumes: []
global:
  systemDefaultRegistry: ''

Linkerd deployed via cli:

linkerd install -L linkerd --ha

Using Metallb to provide the LoadBalancer IP

Testing using the linkerd-viz web.

---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: web-ingress-auth
  namespace: linkerd-viz
data:
  auth: LINKERD_DASHBOARD_BASIC_AUTH
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: web-l5d-header-middleware
  namespace: linkerd-viz
spec:
  headers:
    customRequestHeaders:
      l5d-dst-override: "web.linkerd-viz.svc.cluster.local:8084"
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: web-basic-auth
  namespace: linkerd-viz
spec:
  basicAuth:
    secret: web-ingress-auth
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: web-ingress
  namespace: linkerd-viz
spec:
  entryPoints:
    - web
  routes:
    - kind: Rule
      match: "Host(`LINKERD_DASHBOARD_URL`)"
      middlewares:
        - name: web-l5d-header-middleware
        - name: web-basic-auth
      services:
        - kind: Service
          name: web
          port: 8084

[cpretzer]

@BenB196 can you share the Traefik logs?

I'd start off with this by figuring out which value Traefik is using to log the IPs. Because the proxy is injected into the container and is handling requests for Traefik, it is technically coming from 127.0.0.1, so it would help to look at the headers that Traefik receives to find out if the IP address is in any of those headers like X-Forwarded-For or X-Real-IP

[BenB196]

@cpretzer thanks for the suggestion, but I don't think it helped. I enabled debug logging + header logging to see if I could see anything, but it looks like all of the IP headers contain only 127.0.0.1.

Traefik's docs regarding log format is a bit vague, but based off of https://doc.traefik.io/traefik/observability/access-logs/#format it looks like it uses remote_IP_address, which doesn't really help answer your question about what traefik uses for IP logging. But the logs do show that X-Forward-For and X-Real-IP are not populated/populating.

time="2021-04-06T12:07:39Z" level=debug msg="Authentication succeeded" middlewareType=BasicAuth middlewareName=linkerd-viz-web-basic-auth@kubernetescrd
time="2021-04-06T12:07:39Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":{},\"Host\":\"\",\"Path\":\"/api/tps-reports\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"resource_type=all\\u0026namespace=traefik\\u0026tcp_stats=true\\u0026window=1m\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"en-US,en;q=0.5\"],\"Authorization\":[\"Basic <snipped>\"],\"Cookie\":[\"_gcl_au=1.1.514921898.1614728158\"],\"Dnt\":[\"1\"],\"Referer\":[\"http://linkerd-dashboard-dev.example.com/namespaces/traefik\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0\"],\"X-Forwarded-Host\":[\"linkerd-dashboard-dev.example.com\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-69694756ff-clmnf\"],\"X-Real-Ip\":[\"127.0.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"linkerd-dashboard-dev.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"127.0.0.1:35022\",\"RequestURI\":\"/api/tps-reports?resource_type=all\\u0026namespace=traefik\\u0026tcp_stats=true\\u0026window=1m\",\"TLS\":null}"
time="2021-04-06T12:07:39Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":{},\"Host\":\"\",\"Path\":\"/api/tps-reports\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"resource_type=all\\u0026namespace=traefik\\u0026tcp_stats=true\\u0026window=1m\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"en-US,en;q=0.5\"],\"Authorization\":[\"Basic <snipped>\"],\"Cookie\":[\"_gcl_au=1.1.514921898.1614728158\"],\"Dnt\":[\"1\"],\"Referer\":[\"http://linkerd-dashboard-dev.example.com/namespaces/traefik\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0\"],\"X-Forwarded-Host\":[\"linkerd-dashboard-dev.example.com\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-69694756ff-clmnf\"],\"X-Real-Ip\":[\"127.0.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"linkerd-dashboard-dev.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"127.0.0.1:35022\",\"RequestURI\":\"/api/tps-reports?resource_type=all\\u0026namespace=traefik\\u0026tcp_stats=true\\u0026window=1m\",\"TLS\":null}" ForwardURL="http://10.42.3.140:8084"
time="2021-04-06T12:07:39Z" level=debug msg="Authentication succeeded" middlewareName=linkerd-viz-web-basic-auth@kubernetescrd middlewareType=BasicAuth

What is weird, is if I run a tap on the traefik namespace, I can see the real IP in the tap messages, but it looks like linkerd proxy isn't passing them to the traefik instance.

[BenB196]

@cpretzer after searching a bit more I came across #3403 which appears to be a similar issue with nginx. However, the solution of adding config.linkerd.io/skip-inbound-ports: 80,443 to the pod annotation does not seem to work with regards to traefik as I'm already using config.linkerd.io/skip-inbound-ports: 80,443 in my deployment config.

[olix0r]

Support for Forwarded headers is tracked by #4219