Rancher: Unknown error: websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header

[grig0701]

Description

Hi, we have deployed several clusters using this module, but with each deployment, there is a problem in rancher ui there is no option to view logs and execute shell in pod.

Logs rancher pods:

2023/09/11 12:54:46 [ERROR] Error during subscribe websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header
2023/09/11 12:54:46 [ERROR] Unknown error: websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header

rancher-webhook

E0911 07:34:18.349114       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request

Please help to fix this

Kube.tf file

module "kube-hetzner" {


  providers = {
    hcloud = hcloud
  }
  extra_firewall_rules = [
    {
      description     = "Access to MySQL cluster"
      direction       = "out"
      protocol        = "tcp"
      port            = "3306"
      source_ips      = [] # Won't be used for this rule
      destination_ips = ["0.0.0.0/0", "::/0"]
    },
    {
      description     = "Access to redis"
      direction       = "out"
      protocol        = "tcp"
      port            = "6379"
      source_ips      = [] # Won't be used for this rule
      destination_ips = ["0.0.0.0/0", "::/0"]
    },
  ]
  hcloud_token          = var.hcloud_token
  source                = "kube-hetzner/kube-hetzner/hcloud"
  ssh_public_key        = var.pub
  ssh_private_key       = var.priv
  network_region        = "eu-central"
  create_kubeconfig     = false
  create_kustomization  = false
  #
  use_control_plane_lb  = false
  lb_hostname           = "hostname"
  control_plane_lb_type = "lb11"
  enable_rancher        = true
  initial_k3s_channel   = "v1.25"
  rancher_hostname      = "hostname"

  cluster_name = "claster_name"

  automatically_upgrade_k3s = false
  automatically_upgrade_os  = false

  ingress_controller = "nginx"


  control_plane_nodepools = [
    {
      name        = "control-plane",
      server_type = "cx31",
      location    = "fsn1",
      labels      = ["role=master"],
      taints      = [],
      count       = 1

      # Enable automatic backups via Hetzner (default: false)
      # backups = true
    },

  ]

  agent_nodepools = [
    {
      name        = "agent-development",
      server_type = "cpx41",
      location    = "fsn1",
      labels      = ["environment=development", "role=worker"],
      taints      = [],
      count       = 2

      # Enable automatic backups via Hetzner (default: false)
      # backups = true
    },
    #
    {
      name        = "agent-redis-dev",
      server_type = "cx21",
      location    = "fsn1",
      labels      = ["environment=redis"],
      taints      = [],
      count       = 1

      # Enable automatic backups via Hetzner (default: false)
      # backups = true
    },
  ]

  load_balancer_type     = "lb11"
  load_balancer_location = "fsn1"

}

Screenshots

Platform

linux

[mysticaltech]

@grig0701 Please in your kube.tf, make sure to set a k3s version (via initial_k3s_channel) and rancher version that are compatible with reach other. The process should be explained in the comments near the rancher section.

[mysticaltech]

@grig0701 Thanks for trying. Please see this issue and the solution the user found. rancher/rancher#32751

All that comes to my mind honestly is an issue with the way the LB is configured. Maybe the rancher ingress needs to be tweaked to work with our platform and that can be done by setting custom rancher values in the kube.tf. Please try going that route and if you find the solution, don't hesitate to share.

[grig0701]

@mysticaltech I did a test deployment of the cluster and found that the new cluster has no problem with this. It is very strange. Because I couldn't get it to work on the clusters I deployed about two months ago. Rancher and ingress configurations are the same🤔🤔🤔

[mysticaltech]

@grig0701 The Latest version of the cluster has a lot of tweaks especially to the LB. Please try upgrading your old clusters. See the pinned discussion about the LB upgrade on what to expect, basically it will recreate it as now managed by terraform.

Apart from that it should be straightforward with just terraform init -upgrade while pointing to the latest version of the module.

[grig0701]

@mysticaltech Thank you! I updated the module, and the load balancer was also successfully updated, but unfortunately, the problem was not solved.

Also tried to update the kubernetes version from 1.25.12 to 1.25.14 but it didn't help.

[mysticaltech]

@grig0701 You might want to look at all the logs, see the debug section in the readme, also look at all the kube logs for the svc, rancher, node. Also, investigate selinux, see the pinned issue on the matter. You will find what's wrong! 🤞

[mysticaltech]

Also try to search for that error on the rancher repo, maybe you will find more info on it. On our end, we've never seen it!