Failed upgrade v1.33.7+k3s1 ➔ v1.34.3+k3s1.

[reefland]

Nodes are Ubuntu 24.04.3 LTS, bare metal.

Tried to upgrade: v1.33.7+k3s1 ➔ v1.34.3+k3s1. Reviewed the upgrade notes, did not see any breaking changes related to command-line switches, etc.

Approved PR to upgrade via K3s Upgrade Controller. The controlled selected k3s02 while control-plane, it is not the one with cluster init switch, that would be k3s01.

NAME      STATUS                        ROLES                       AGE      VERSION
dldsk01   Ready                         worker                      607d     v1.33.7+k3s1
k3s01     Ready                         control-plane,etcd,master   3y276d   v1.33.7+k3s1
k3s02     NotReady,SchedulingDisabled   control-plane,etcd,master   3y249d   v1.33.7+k3s1
k3s03     Ready                         control-plane,etcd,master   3y256d   v1.33.7+k3s1
k3s04     Ready                         worker                      3y22d    v1.33.7+k3s1
k3s05     Ready                         worker                      3y22d    v1.33.7+k3s1
k3s06     Ready                         worker                      2y315d   v1.33.7+k3s1

k3S service did not come back on-line. Manually stopped / started k3s service. A common error message I kept seeing was Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated]".

• Checked all nodes times are in-sync.
• Tried rebooting the node. Was still unable to get k3s service to start. After reboot this message kept repeating.

Dec 31 16:22:56 k3s02 k3s[1697]: E1231 16:22:56.002099    1697 remote_available_controller.go:462] "Unhandled Error" err="v1beta1.metrics.k8s.io failed with: apiservices.apiregistration.k8s.io \"v1beta1.metrics.k8s.io\" is forbidden: not yet ready to handle request" logger="UnhandledError"
Dec 31 16:22:56 k3s02 k3s[1697]: E1231 16:22:56.003296    1697 remote_available_controller.go:462] "Unhandled Error" err="v1beta1.external.metrics.k8s.io failed with: apiservices.apiregistration.k8s.io \"v1beta1.external.metrics.k8s.io\" is forbidden: not yet ready to handle request" logger="UnhandledError"
Dec 31 16:22:56 k3s02 k3s[1697]: E1231 16:22:56.245053    1697 autoregister_controller.go:198] "Unhandled Error" err="v1.resource.k8s.io failed with : apiservices.apiregistration.k8s.io \"v1.resource.k8s.io\" is forbidden: not yet ready to handle request" logger="UnhandledError"
Dec 31 16:22:56 k3s02 k3s[1697]: E1231 16:22:56.982143    1697 controller.go:113] "Unhandled Error" err="loading OpenAPI spec for \"v1beta1.external.metrics.k8s.io\" failed with: Error, could not get list of group versions for APIService" logger="UnhandledError"
Dec 31 16:22:56 k3s02 k3s[1697]: I1231 16:22:56.982156    1697 controller.go:126] OpenAPI AggregationController: action for item v1beta1.external.metrics.k8s.io: Rate Limited Requeue.
Dec 31 16:22:57 k3s02 k3s[1697]: E1231 16:22:57.032048    1697 storage_rbac.go:232] "Unhandled Error" err="unable to reconcile clusterrole.rbac.authorization.k8s.io/system:controller:resource-claim-controller: clusterroles.rbac.authorization.k8s.io \"system:controller:resource-claim-controller\" is forbidden: not yet ready to handle request" logger="UnhandledError"
Dec 31 16:23:05 k3s02 k3s[1697]: E1231 16:23:05.101479    1697 reflector.go:205] "Failed to watch" err="failed to list *v1beta1.MutatingAdmissionPolicy: the server could not find the requested resource" logger="UnhandledError" reflector="k8s.io/client-go@v1.34.3-k3s1/tools/cache/reflector.go:290" type="*v1beta1.MutatingAdmissionPolicy"
Dec 31 16:23:06 k3s02 k3s[1697]: E1231 16:23:06.333032    1697 controller.go:195] "Failed to update lease" err="leases.coordination.k8s.io \"apiserver-ahlivttnk5psgnd5rceobfurza\" is forbidden: not yet ready to handle request"
Dec 31 16:23:06 k3s02 k3s[1697]: E1231 16:23:06.405673    1697 autoregister_controller.go:198] "Unhandled Error" err="v1.resource.k8s.io failed with : apiservices.apiregistration.k8s.io \"v1.resource.k8s.io\" is forbidden: not yet ready to handle request" logger="UnhandledError"
Dec 31 16:23:07 k3s02 k3s[1697]: E1231 16:23:07.076573    1697 storage_rbac.go:232] "Unhandled Error" err="unable to reconcile clusterrole.rbac.authorization.k8s.io/system:controller:volumeattributesclass-protection-controller: clusterroles.rbac.authorization.k8s.io \"system:controller:volumeattributesclass-protection-controller\" is forbidden: not yet ready to handle request" logger="UnhandledError"
Dec 31 16:23:11 k3s02 k3s[1697]: E1231 16:23:11.003964    1697 remote_available_controller.go:462] "Unhandled Error" err="v1beta1.metrics.k8s.io failed with: apiservices.apiregistration.k8s.io \"v1beta1.metrics.k8s.io\" is forbidden: not yet ready to handle request" logger="UnhandledError"
Dec 31 16:23:11 k3s02 k3s[1697]: E1231 16:23:11.005012    1697 remote_available_controller.go:462] "Unhandled Error" err="v1beta1.external.metrics.k8s.io failed with: apiservices.apiregistration.k8s.io \"v1beta1.external.metrics.k8s.io\" is forbidden: not yet ready to handle request" logger="UnhandledError"
Dec 31 16:23:16 k3s02 k3s[1697]: E1231 16:23:16.335300    1697 controller.go:195] "Failed to update lease" err="leases.coordination.k8s.io \"apiserver-ahlivttnk5psgnd5rceobfurza\" is forbidden: not yet ready to handle request"
Dec 31 16:23:16 k3s02 k3s[1697]: E1231 16:23:16.726307    1697 autoregister_controller.go:198] "Unhandled Error" err="v1.resource.k8s.io failed with : apiservices.apiregistration.k8s.io \"v1.resource.k8s.io\" is forbidden: not yet ready to handle request" logger="UnhandledError"
Dec 31 16:23:17 k3s02 k3s[1697]: E1231 16:23:17.132559    1697 storage_rbac.go:264] "Unhandled Error" err="unable to reconcile clusterrolebinding.rbac.authorization.k8s.io/system:controller:resource-claim-controller: clusterrolebindings.rbac.authorization.k8s.io \"system:controller:resource-claim-controller\" is forbidden: not yet ready to handle request" logger="UnhandledError"
Dec 31 16:23:21 k3s02 k3s[1697]: E1231 16:23:21.336788    1697 reflector.go:205] "Failed to watch" err="failed to list *v1beta1.MutatingAdmissionPolicyBinding: the server could not find the requested resource" logger="UnhandledError" reflector="k8s.io/client-go@v1.34.3-k3s1/tools/cache/reflector.go:290" type="*v1beta1.MutatingAdmissionPolicyBinding"
Dec 31 16:23:26 k3s02 k3s[1697]: E1231 16:23:26.005546    1697 remote_available_controller.go:462] "Unhandled Error" err="v1beta1.metrics.k8s.io failed with: apiservices.apiregistration.k8s.io \"v1beta1.metrics.k8s.io\" is forbidden: not yet ready to handle request" logger="UnhandledError"
Dec 31 16:23:26 k3s02 k3s[1697]: E1231 16:23:26.006840    1697 remote_available_controller.go:462] "Unhandled Error" err="v1beta1.external.metrics.k8s.io failed with: apiservices.apiregistration.k8s.io \"v1beta1.external.metrics.k8s.io\" is forbidden: not yet ready to handle request" logger="UnhandledError"
Dec 31 16:23:26 k3s02 k3s[1697]: E1231 16:23:26.342351    1697 controller.go:195] "Failed to update lease" err="leases.coordination.k8s.io \"apiserver-ahlivttnk5psgnd5rceobfurza\" is forbidden: not yet ready to handle request"
Dec 31 16:23:27 k3s02 k3s[1697]: E1231 16:23:27.155750    1697 storage_rbac.go:264] "Unhandled Error" err="unable to reconcile clusterrolebinding.rbac.authorization.k8s.io/system:controller:volumeattributesclass-protection-controller: clusterrolebindings.rbac.authorization.k8s.io \"system:controller:volumeattributesclass-protection-controller\" is forbidden: not yet ready to handle request" logger="UnhandledError"

Rollback:

• Reverted the upgrade PR.
• Killed the upgrade controller plan job pods
• Stopped k3s service.
• Manually rolled back k3s02 binary back to v1.33.7+k3s1.
• Started K3s service; started without issue.
• Cluster is stable; no issues reported in AlertManager.

The important parts of k3s service look like:

ExecStart=/usr/local/bin/k3s \
    server \
        '--server' \
        'https://k3s01.[redacted]:6443' \
        '--token' \
        '[redacted]' \
        '--disable' \
        'traefik' \
        '--kubelet-arg=image-gc-low-threshold=50' \
        '--kubelet-arg=image-gc-high-threshold=65' \
        '--embedded-registry' \
        '--disable' \
        'servicelb' \
        '--tls-san=192.168.10.239' \

The Token above is the "short" type (32 chars); the file /var/lib/rancher/k3s/server/token has a "full" token and they all match.

Looking for suggestions on what needs to be changed before I attempt the upgrade again.

[brandond]

Please attach full logs from all involved nodes; just a couple lines is not enough to give any idea what's going on. The messages you included suggest that the metrics-server pod isn't running but that's not critical.