Since upgrade from 15.x to 16.x cannot access nodes via teleport web

[StillLoading]

Hi,

I've gone through all changelogs but could not find anything that points to changes requiring manual intervention for teleport-web.
After upgrading to 16.x I can no longer access any node via the Web-GUI. SSH Session instantly disconnect and web applications open with "access denied" and a CSP error "because it violates the following Content Security Policy directive: "connect-src 'self' wss:".
Access via tsh still works though.

My server config looks like this:

version: v2
teleport:
  nodename: teleport01
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: DEBUG
    format:
      output: text
  ca_pin: []
  diag_addr: ""
  storage:
    type: sqlite
    sync: NORMAL
    journal: WAL
auth_service:
  enabled: "yes"
  cluster_name: tele.example.com
  authentication:
    second_factor: on
    webauthn:
      rp_id: tele.example.com
  proxy_listener_mode: multiplex
  session_recording: "proxy"
ssh_service:
  enabled: "yes"
  labels:
    hardware: virtual
  commands:
  - name: hostname
    command: [hostname]
    period: 1m0s
  - name: arch
    command: [uname, -p]
    period: 1h0m0s
proxy_service:
  enabled: "yes"
  web_listen_addr: 0.0.0.0:443
  public_addr: tele.example.com:443
  https_keypairs: []
  trust_x_forwarded_for: true
  acme:
    enabled: "yes"
    email: js@example.com

[webvictim]

I see you're setting trust_x_forwarded_for: true - is Teleport deployed behind a reverse proxy of some kind?

[StillLoading]

That option might just be from when I first setup the server. I was running in L7 mode at first but couldn't get apps to work, so I switched to TCP mode and let teleport handle the cert stuff.

I just swapped my NAT rule to point directly to the server, same issue.

Here's a log snippet of the app connection attempt

Jul 25 20:16:22 teleport01 teleport[1385]: 2024-07-25T20:16:22+02:00 DEBU [WEB]       Creating application web session for ipa01.tele.example.com in tele.example.com. web/apps.go:217
Jul 25 20:16:22 teleport01 teleport[1385]: 2024-07-25T20:16:22+02:00 DEBU [WEB]       Ensuring proxy can handle requests requests for application "ipa01". web/apps.go:222
Jul 25 20:16:22 teleport01 teleport[1385]: 2024-07-25T20:16:22+02:00 DEBU [PROXY:SER] Dialing from: "192.168.17.101:52354" to: "@local-node". trace.fields:map[cluster:tele.example.com] reversetunnel/localsite.go:307
Jul 25 20:16:22 teleport01 teleport[1385]: 2024-07-25T20:16:22+02:00 DEBU [PROXY:SER] Tunnel dialing to 48fd4ec9-a722-4a36-ad23-ea11ae40182b.tele.example.com, client source 192.168.17.101:52354 trace.fields:map[cluster:tele.example.com] reversetunnel/localsite.go:453
Jul 25 20:16:22 teleport01 teleport[1385]: 2024-07-25T20:16:22+02:00 DEBU [PROXY:SER] Connecting to 172.16.5.1:14716 through tunnel. trace.fields:map[cluster:tele.example.com] reversetunnel/localsite.go:865
Jul 25 20:16:22 teleport01 teleport[1385]: 2024-07-25T20:16:22+02:00 DEBU [PROXY:SER] Succeeded dialing from: "192.168.17.101:52354" to: "@local-node". trace.fields:map[cluster:tele.example.com] reversetunnel/localsite.go:313
Jul 25 20:16:22 teleport01 teleport[1385]: 2024-07-25T20:16:22+02:00 DEBU             generated user key for [js root steam -teleport-internal-join] with expiry on (1721973856) 2024-07-26 06:04:16.000752845 +0000 UTC keygen/keygen.go:153
Jul 25 20:16:22 teleport01 teleport[1385]: 2024-07-25T20:16:22+02:00 DEBU [CA]        Generating TLS certificate common_name:jzie dns_names:[] key_usage:5 not_after:2024-07-26 06:04:16.003070466 +0000 UTC tlsca/ca.go:1201
Jul 25 20:16:22 teleport01 teleport[1385]: 2024-07-25T20:16:22+02:00 INFO [AUDIT]     cert.create cert_type:user cluster_name:tele.example.com code:TC000I ei:0 event:cert.create client_ip:192.168.17.101 expires:2024-07-26T06:04:16.003070466Z logins:[js root steam -teleport-internal-join] prev_identity_expires:0001-01-01T00:00:00Z private_key_policy:none roles:[jzie editor auditor allow-user-steam] cluster_name:tele.example.com name: public_addr:ipa01.tele.example.com session_id:6726615684b69285eb33d8bf711d94e12d6dab65dffd3d03013ad694ce62bc5d route_to_cluster:tele.example.com teleport_cluster:tele.example.com aws_role_arns:<nil> db_names:<nil> db_users:<nil> kubernetes_groups:[] kubernetes_users:[] logins:<nil> windows_logins:<nil> usage:[usage:apps] user:jzie time:2024-07-25T18:16:22.444Z uid:0899fa6b-93b3-44c3-b70c-f0c65d9b0671 user_agent:grpc-go/1.64.1 events/emitter.go:288
Jul 25 20:16:22 teleport01 teleport[1385]: 2024-07-25T20:16:22+02:00 DEBU [AUTH]      Generated application web session for jzie with TTL 11h47m53.561864309s. auth/sessions.go:431
Jul 25 20:16:22 teleport01 teleport[1385]: 2024-07-25T20:16:22+02:00 INFO [AUDIT]     app.session.start addr.remote:192.168.17.101:52354 app_name:ipa01 app_public_addr:ipa01.tele.example.com app_uri:https://ipa01.int.example.com cluster_name:tele.example.com code:T2007I ei:0 event:app.session.start namespace:default private_key_policy:none public_addr:ipa01.tele.example.com server_id:92f4790b-c8cf-4641-8365-5c8e7fb7f3b0 server_version:16.1.0 sid:6726615684b69285eb33d8bf711d94e12d6dab65dffd3d03013ad694ce62bc5d time:2024-07-25T18:16:22.445Z uid:11155274-c8f4-475e-a1fd-f3e93b41db4a user:jzie user_kind:1 events/emitter.go:288
Jul 25 20:16:22 teleport01 teleport[1385]: 2024-07-25T20:16:22+02:00 INFO [AUDIT]     app.session.start addr.remote:192.168.17.101:52354 app_name:ipa01 app_public_addr:ipa01.tele.example.com app_uri:https://ipa01.int.example.com cluster_name:tele.example.com code:T2007I ei:0 event:app.session.start namespace:default private_key_policy:none public_addr:ipa01.tele.example.com server_id:92f4790b-c8cf-4641-8365-5c8e7fb7f3b0 server_version:16.1.0 sid:6726615684b69285eb33d8bf711d94e12d6dab65dffd3d03013ad694ce62bc5d time:2024-07-25T18:16:22.445Z uid:11155274-c8f4-475e-a1fd-f3e93b41db4a user:jzie user_kind:1 events/emitter.go:288

It's very weird, because if I rollback the server to v15 it just works again.

[webvictim]

So it works on 15.4.9 but not 16.0.0?

[StillLoading]

v15.4.7 is the version my last v15 backup is on. I could try upgrading that to 15.4.9 to see if it breaks at that point too.

[webvictim]

It'd be good to verify, but I'm sure there won't be a difference between 15.4.7 and 15.4.9. If 15.4.7 works and 16.0.0 doesn't, that's good to know.

[StillLoading]

Okay, I think I narrowed it down. The server is running OpenSUSE MicroOS and since that is a rolling release I had to pull the 15.4.9 bin directly from the downloads page. After swapping the binaries I noticed the Web GUI looked completely different. Access was still working too.
Rolled back and did the update via repository again, v16.1.0 and same issue. But the Web GUI looked different again. So I swapped the repo binary with the v16.1.0 from the download page again...Web GUI looks different again and everything works.

Looking at the package source (https://build.opensuse.org/package/show/openSUSE:Factory/teleport) I can see they are using a ~1 year old webasstes.tar.gz. Me thinks thats the culprit.