Need help installing teleport behind traefik reverse proxy

[BLucky-gh]

Hi,

I know this is a bit of an open question but I have been trying to set up teleport behind traefik the whole day and I'm going in circles. Does anyone have any tips on deploying teleport behind traefik or should I just give up and expose teleport on a different port?

It's just that the whole thing with ACME and certificates and stuff being done by both traefik and teleport, and teleport needing traefik to not terminate tls just makes everything way more complicated.

I even tried to just start teleport with the following teleport.yaml (with the key_file and cert_file being the tls certificates dumped from traefik) but I get a https: error: SSLError: HTTPSConnectionPool(host='tele.mydomain.tld', port=3025): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))) while doing a GET request to URL: https://tele.mydomain.tld:3025/ error (same error on all ports, same with 444) at best and an empty response at worst.

teleport.yaml that I got with teleport configure --public-addr tele.mydomain.tld --cert-file /etc/traefik/certs/certs/tele.mydomain.tldcrt --key-file /etc/traefik/certs/private/tele.mydomain.tld.key -o /etc/teleport/teleport.yaml --cluster-name telemydomain.tld

teleport:
  nodename: 151.mydomain.tld
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: INFO
    format:
      output: text
  ca_pin: []
  diag_addr: ""
auth_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3025
  cluster_name: tele.mydomain.tld
  proxy_listener_mode: multiplex
ssh_service:
  enabled: "yes"
  labels:
    env: example
  commands:
  - name: hostname
    command: [hostname]
    period: 1m0s
proxy_service:
  enabled: "yes"
  web_listen_addr: 0.0.0.0:444
  public_addr: tele.mydomain.tld:444
  https_keypairs:
  - key_file: /etc/traefik/certs/private/tele.mydomain.tld.key
    cert_file: /etc/traefik/certs/certs/tele.mydomain.tld.crt

teleport logs from journalctl:

INFO             Generating new host UUID: dc5d2c2a-ccb2-4514-b17b-fdb99f7a2ea5. service/service.go:691
INFO [PROC:1]    Admin has obtained credentials to connect to the cluster. service/connect.go:433
INFO [PROC:1]    The process successfully wrote the credentials and state of Admin to the disk. service/connect.go:474
INFO [PROC:1]    Service auth is creating new listener on 0.0.0.0:3025. service/signals.go:212
WARN [AUTH:1]    Configuration setting auth_service/advertise_ip is not set. guessing 172.23.0.1:3025. service/service.go:1455
WARN             No TLS Keys provided, using self-signed certificate. service/service.go:3965
WARN             Generating self-signed key and cert to /var/lib/teleport/webproxy_key.pem /var/lib/teleport/webproxy_cert.pem. service/service.go:3983
INFO [AUTH:1]    Auth service 9.2.4:v9.2.4-0-g86d887eff is starting on 172.23.0.1:3025. utils/cli.go:275
[AUTH]         Auth service 9.2.4:v9.2.4-0-g86d887eff is starting on 172.23.0.1:3025.
INFO [PROC:1]    Node has obtained credentials to connect to the cluster. service/connect.go:433
INFO [PROC:1]    Proxy has obtained credentials to connect to the cluster. service/connect.go:433
INFO [PROC:1]    The process successfully wrote the credentials and state of Node to the disk. service/connect.go:474
INFO [PROC:1]    The process successfully wrote the credentials and state of Proxy to the disk. service/connect.go:474
INFO [PROC:1]    Proxy: features loaded from auth server: Kubernetes:true App:true DB:true Desktop:true  service/connect.go:76
INFO [PROC:1]    Service proxy:web is creating new listener on 0.0.0.0:3080. service/signals.go:212
INFO [PROC:1]    Node: features loaded from auth server: Kubernetes:true App:true DB:true Desktop:true  service/connect.go:76
INFO             Loading TLS certificate /var/lib/teleport/webproxy_cert.pem and key /var/lib/teleport/webproxy_key.pem. service/service.go:3428
INFO [PROXY:SER] Reverse tunnel service 9.2.4:v9.2.4-0-g86d887eff is starting on . utils/cli.go:275
[PROXY]        Reverse tunnel service 9.2.4:v9.2.4-0-g86d887eff is starting on .
INFO [PROXY:SER] Starting 9.2.4:v9.2.4-0-g86d887eff on  using no cache service/service.go:2924
INFO [PROC:1]    Service node is creating new listener on 0.0.0.0:3022. service/signals.go:212
INFO [NODE:1]    Service 9.2.4:v9.2.4-0-g86d887eff is starting on 0.0.0.0:3022 no cache. service/service.go:1996
INFO [NODE:1]    Service 9.2.4:v9.2.4-0-g86d887eff is starting on 0.0.0.0:3022. utils/cli.go:275
[NODE]         Service 9.2.4:v9.2.4-0-g86d887eff is starting on 0.0.0.0:3022.
INFO [PROXY:SER] Web proxy service 9.2.4:v9.2.4-0-g86d887eff is starting on 0.0.0.0:3080. utils/cli.go:275
[PROXY]        Web proxy service 9.2.4:v9.2.4-0-g86d887eff is starting on 0.0.0.0:3080.
INFO [PROXY:SER] Web proxy service 9.2.4:v9.2.4-0-g86d887eff is starting on 0.0.0.0:3080. service/service.go:3013
INFO [PROXY:SER] SSH proxy service 9.2.4:v9.2.4-0-g86d887eff is starting on . utils/cli.go:275
[PROXY]        SSH proxy service 9.2.4:v9.2.4-0-g86d887eff is starting on .
INFO [PROXY:SER] SSH proxy service 9.2.4:v9.2.4-0-g86d887eff is starting on {  } service/service.go:3055
INFO [PROC:1]    The new service has started successfully. Starting syncing rotation status with period 10m0s. service/connect.go:486
INFO [DB:SERVIC] Starting Postgres proxy server on 0.0.0.0:3080. service/service.go:3217
INFO [DB:SERVIC] Starting Database TLS proxy server on 0.0.0.0:3080. service/service.go:3235
INFO [PROXY:SER] Starting proxy gRPC server on [::]:3080. service/service.go:3269
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log. service/service.go:2125
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload. service/service.go:2125
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/sessions. service/service.go:2125
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/sessions/default. service/service.go:2125
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log. service/service.go:2125
INFO [PROXY:SER] Starting TLS ALPN SNI proxy server on [::]:3080. service/service.go:3298
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload. service/service.go:2125
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/streaming. service/service.go:2125
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/streaming/default. service/service.go:2125
WARN [PROXY:1]   Restart watch on error: empty proxy list. resource-kind:proxy services/watcher.go:219

[webvictim]

If you want to use Traefik, first you'll need to remove proxy_listener_mode: multiplex from your configuration to disable TLS multiplexing (as this will break everything when you're not terminating TLS at Teleport itself).

Then, get Traefik to forward traffic to your Teleport container on port 444 (i.e. whatever your have configured for web_listen_addr)

You probably don't need to provide certs to Teleport - it will generate a self-signed cert for the backend connection if you don't provide one via https_keypairs. You'll need to disable TLS verification for Traefik's backend connection to Teleport in this case.

[BLucky-gh]

Can I have both a tcp and an http endpoint on the same port in traefik tho? putting it on another port defeats the purpose of reverse proxying it in the first place
I used port 444 in teleport config so that it doesn't conflict with port 443 already taken by traefik

[ar1em]

@hreidar thanks for your config.

I had issue with certificates:
ERROR: Get "https://teleport.cluster.local/v2/authorities/host?load_keys=false": x509: certificate is valid for 5ee1a3d9d4e7984e11d004510aedd2e6.a69d60489801155f78cdbc3ff905901e.traefik.default, not 6c6f63616c686f7374.teleport.cluster.local

It is important to use wildcard in HostSNI(`*`) because, if proxy_listener_mode: multiplex option is used, teleport uses SNI and ALPN extensions for TLS Routing and SNI not always is equal to domain name.

[ingowonner]

Thanks for the config @hreidar! I was able to get it to work, but stumbled upon the same error as @ar1em. His suggestion to use HostSNI(`*`) worked perfectly fine, but I found an alternative which does not require to use the wildcard here.
After I added {subdomain:[a-z0-9]+}.teleport.cluster.local to my HostSNIRegexp rule it started to work without throwing an error.

HostSNIRegexp(`teleport.example.com`, `{subdomain:[a-z]+}.teleport.example.com`, `{subdomain:[a-z0-9]+}.teleport.cluster.local`)

[strausmann]

I have now also extended my TCP service with @ingowonner to include the *.teleport.cluster.local.

I can't quite figure out why this works. But it does. I did not create a hosts entry for teleport.cluster.local, nor did I change the teleport server config itself.

I am using the AuthService in multiplex mode and Traefik only passes 443 through.

Thank you for the tip...

Thanks for the config @hreidar! I was able to get it to work, but stumbled upon the same error as @ar1em. His suggestion to use HostSNI(*) worked perfectly fine, but I found an alternative which does not require to use the wildcard here. After I added {subdomain:[a-z0-9]+}.teleport.cluster.local to my HostSNIRegexp rule it started to work without throwing an error.

HostSNIRegexp(`teleport.example.com`, `{subdomain:[a-z]+}.teleport.example.com`, `{subdomain:[a-z0-9]+}.teleport.cluster.local`)

[webvictim]

Here's some more information on teleport.cluster.local and what it does: https://goteleport.com/docs/management/admin/troubleshooting/#teleportclusterlocal

[r0fflerapt0r]

Hey Everyone, I know this is an old thread but I also have a question about deploying behind Traefik.
I have everything working as far as the container, I can access it on my local network on the usual port 3080. But when I try to access it from the domain I see this error: DEBU [MX:PROXY:] Closing HTTP connection: HTTP listener is disabled. multiplexer/multiplexer.go:241 every time I go to the domain from public. I've attached my docker-compose file, teleport.yaml and the logs from when I connect with local subnet on 3080 versus when I connect on public.

Ideally I'd like all items going through 443/80 because I don't like port forwarding, but I might have to for Teleport to work behind my reverse proxy :( I have 34 other containers running behind the proxy, nearly all with port 80/443 exposed to reverse proxy working great with SSL. Teleport is the only one so far I can't get working.
debugging-help.txt
_teleport_logs.txt

[webvictim]

Don't use --insecure-no-tls as it will disable TLS routing. You should instead configure Traefik to speak HTTPS to Teleport and ignore the self-signed certificate that Teleport generates for its web UI by default. That will likely get you working, and the steps detailed above with HostSNI should hopefully then enable you to use TLS routing successfully behind Traefik as a layer 4 LB.

[webvictim]

For anyone else interested, I also posted a writeup of how to do this with nginx here: #19093

[stefan-golinschi]

Here's a working example of teleport running behind traefik reverse proxy.

docker-compose.yaml:

teleport:
    image: public.ecr.aws/gravitational/teleport:11.2.3
    container_name: teleport
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=net.external"
      - "traefik.tcp.services.teleport-svc.loadBalancer.server.port=3080"
      - "traefik.tcp.routers.teleport.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.teleport.entrypoints=websecure"
      - "traefik.tcp.routers.teleport.tls.passthrough=true"
    volumes:
      - ./teleport/config:/etc/teleport
      - ./teleport/data:/var/lib/teleport
    ports:
      - 3025:3025
      - 3080:3080
      - 3023:3023
    networks:
      - net.external
    restart: unless-stopped

teleport.yaml:

version: v3
teleport:
  nodename: teleport.domain.com
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: INFO
    format:
      output: text
  ca_pin: ""
  diag_addr: ""
auth_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3025
  proxy_listener_mode: multiplex
ssh_service:
  enabled: "yes"
  commands:
  - name: hostname
    command: [hostname]
    period: 1m0s
proxy_service:
  web_listen_addr: 0.0.0.0:3080
  public_addr: teleport.domain.com:443
  enabled: "yes"
  https_keypairs: []
  https_keypairs_reload_interval: 0s
  acme: 
    enabled: yes
    email: blabla@blabla.bla

Using this setup, traefik will not terminate TLS, instead wil pass the TCP traffic to the 3080 port which is where teleport listens.
This way, teleport will handle the certificates & traffic decryption.

[birkanozer]

Here's a working example of teleport running behind traefik reverse proxy.

docker-compose.yaml:

teleport:
    image: public.ecr.aws/gravitational/teleport:11.2.3
    container_name: teleport
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=net.external"
      - "traefik.tcp.services.teleport-svc.loadBalancer.server.port=3080"
      - "traefik.tcp.routers.teleport.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.teleport.entrypoints=websecure"
      - "traefik.tcp.routers.teleport.tls.passthrough=true"
    volumes:
      - ./teleport/config:/etc/teleport
      - ./teleport/data:/var/lib/teleport
    ports:
      - 3025:3025
      - 3080:3080
      - 3023:3023
    networks:
      - net.external
    restart: unless-stopped

teleport.yaml:

version: v3
teleport:
  nodename: teleport.domain.com
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: INFO
    format:
      output: text
  ca_pin: ""
  diag_addr: ""
auth_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3025
  proxy_listener_mode: multiplex
ssh_service:
  enabled: "yes"
  commands:
  - name: hostname
    command: [hostname]
    period: 1m0s
proxy_service:
  web_listen_addr: 0.0.0.0:3080
  public_addr: teleport.domain.com:443
  enabled: "yes"
  https_keypairs: []
  https_keypairs_reload_interval: 0s
  acme: 
    enabled: yes
    email: blabla@blabla.bla

Using this setup, traefik will not terminate TLS, instead wil pass the TCP traffic to the 3080 port which is where teleport listens. This way, teleport will handle the certificates & traffic decryption.

Can you also share the Traefik's docker-compose file?

[codicodac]

This work perfectly but you don't need to expose as many port in multiplex mode,
in your compose you exposed 3 ports :

• 3025:3025
• 3080:3080
• 3023:3023

but you only need to expose 3080, this is the purpose of the multiplex mode.

And in fact, as the traffic is passthrough via TCP Traefik within the traefik network you don't even need to expose the 3080.

[urbaman]

Hi,

Trying to do Teleport behind Traefik here.

I always get the followig error when I try to access a webapp.

It's working flawlessly without Traefik (direct Teleport), and server access (ssh) works flawlessly with or without Traefik.

I am running Teleport in ClusterIP and Multiplex mode.

My Traefik CRD conf:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: traefik-teleport-cluster-https-redirect
  namespace: teleport-cluster
spec:
  redirectScheme:
    scheme: https
    permanent: true
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: traefik-teleport-cluster-security
  namespace: teleport-cluster
spec:
  headers:
    frameDeny: true
    sslRedirect: true
    browserXssFilter: true
    contentTypeNosniff: true
    stsIncludeSubdomains: true
    stsPreload: true
    stsSeconds: 31536000
---
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
  name: traefik-teleport-cluster-transport
  namespace: teleport-cluster
spec:
  insecureSkipVerify: true
---
apiVersion: traefik.containo.us/v1alpha1
kind: TLSOption
metadata:
  name: traefik-teleport-cluster-tlsoptions
  namespace: teleport-cluster
spec:
  minVersion: VersionTLS12
  cipherSuites:
    - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
    - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
    - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
    - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
    - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
    - TLS_AES_256_GCM_SHA384
    - TLS_AES_128_GCM_SHA256
    - TLS_CHACHA20_POLY1305_SHA256
    - TLS_FALLBACK_SCSV
  curvePreferences:
    - CurveP521
    - CurveP384
  sniStrict: false
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-teleport-cluster-websecure
  namespace: teleport-cluster
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: HostRegexp(`teleport.domain.com`, `{subdomain:[a-z]+}.teleport.domain.com`, `{subdomain:[a-z0-9]+}.teleport.cluster.local`)
      services:
        - name: teleport-cluster
          port: 443
          serversTransport: traefik-teleport-cluster-transport
      middlewares:
        - name: traefik-teleport-cluster-security
  tls:
    secretName: teleport-urbaman
    options:
      name: traefik-teleport-cluster-tlsoptions
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-teleport-cluster-web
  namespace: teleport-cluster
spec:
  entryPoints:
    - web
  routes:
    - kind: Rule
      match: HostRegexp(`teleport.domain.com`, `{subdomain:[a-z]+}.teleport.domain.com`, `{subdomain:[a-z0-9]+}.teleport.cluster.local`)
      services:
        - name: teleport-cluster
          port: 443
      middlewares:
        - name: traefik-teleport-cluster-https-redirect

[urbaman]

Ok, solved adding the

    customRequestHeaders:
      X-Forwarded-Proto: "https"

to the headers middleware, now everything runs flawlessly!

Thanks.

[urbaman]

Well, not really, only solved some apps, not all.

I do not understand why some seem to work, some not.

If I go the Traefik->Teleport->Traefk->App route, it works.
If I go the Traefik->Teleport->App, it still doesn't work.

Still trying to work on it, if you have any insight that would be awesome.

[urbaman]

Probably resolved completely: renaming the apps fixed it all, following this bug: #30160