HTTP_X_FORWARDED_PROTO overwritten by k3d w. treafik ingress controller behind nginx reverse proxy w. ssl termination

[ghost]

Hi,
I've been breaking my head over the following problem for some time now.

We have an nginx-reverse-proxy exposed to the internet with an let's encrypt wildcard-certificate,
and trying to proxy_pass to the internal k3d cluster with an exposed http port.
Nginx is configured with:

location / {
	proxy_pass http://k3d_upstreamm;
	...
	proxy_set_header X-Real-Ip $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Proto $scheme;
	...
}

Everything is working fine, except that any ingress controller (i've tried treafik & nginx) rewrites the HTTP_X_FORWARDED_...-Headers.
HTTP_X_FORWARDED_PROTO=http instead of HTTP_X_FORWARDED_PROTO=https as passed by nginx-reverse-proxy.

For Treafik I tried to enable https://doc.traefik.io/traefik/v2.3/routing/entrypoints/#forwarded-headers using the provided HelmChartConfig and also by deploying Treafik-Ingress-Controller manually to be able to set "insecure" directly on the Pod. I would still think that should have worked - can anybody confirm or deny this ?

This is the HelmChartConfig I used:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    image:
      name: traefik
      tag: v2.6.1
    proxyProtocol:
      enabled: true
      trustedIPs:
        - 10.0.0.0/8 < k3d network
        - 172.0.0.0/8 < docker network
        - #.#.#.# < nginx-reverse-proxy
    forwardedHeaders:
      enabled: true
      trustedIPs:
        - 10.0.0.0/8 < k3d network
        - 172.0.0.0/8 < docker network
        - #.#.#.# < nginx-reverse-proxy

These are the HTTP-Headers as passed to the Application by ingress:

    [HTTP_X_REAL_IP] => 10.42.0.1
    [HTTP_X_FORWARDED_SERVER] => traefik-658d87c8b5-7qddl
    [HTTP_X_FORWARDED_PROTO] => http
    [HTTP_X_FORWARDED_PORT] => 80
    [HTTP_X_FORWARDED_HOST] => test.something-something.dark-side
    [HTTP_X_FORWARDED_FOR] => 10.42.0.1

Which makes sense because the ingress controller is working with http, but this breaks some things in the hosted applications, because they are relying on HTTP_X_FORWARDED_PROTO=https.

Has anybody done something similar and can point me in the right direction?

[iwilltry42]

Hi @fsmoak , thanks for starting this discussion!
It seems like this is not particularly related to k3d, right? Have you asking in the traefik forum? https://community.traefik.io/
The only thing in between your proxy and the ingress controller on k3d's end could be k3d-proxy, which only does plain tcp passthrough, so it's transparent and doesn't touch headers.
Additionally, if you want to keep this in the k3d community, you may want to ask over in our #k3d Slack channel at https://rancher-users.slack.com (sign up via https://slack.rancher.io/)