Tailscale Integration fails to Reconnect After Boot (sometimes)

[lukepolo]

Environmental Info:
K3s Version:

k3s version v1.34.3+k3s1 (48ffa7b)
go version go1.24.11

Node(s) CPU architecture, OS, and Version:

Linux luketestbox 5.15.0-168-generic #178-Ubuntu SMP Fri Jan 9 19:05:03 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

3 servers, 3 agents.

Describe the bug:

after reboot sometimes tailscale will be logged out.

 tailscale status
# Health check:
#     - You are logged out.
#     - Unable to connect to the Tailscale coordination server to synchronize the state of your tailnet. Peer reachability might degrade over time.

But if you restart the k3s daemon it will reconnect

root@teestbox:/home/qadmin# systemctl restart tailscaled
root@teestbox:/home/qadmin# tailscale status
100.x.201.x   teestbox           teestbox.tailcec4f.ts.net  linux  -
100.x.161.x     dev2      tagged-devices                linux  offline, last seen 22h ago
100.x.57.x   dev 1                  tagged-devices                linux  -
100.x.118.x  dev3             tagged-devices                linux  -
100.x.150.x    dev4                macOS  offline, last seen 93d ago

Steps To Reproduce:

• Install k3s with tailscale auth key integration.
• Reboot any node, check if tailscale reconnected

Expected behavior:

We expect tailscale to reconnect.

Actual behavior:

Tailscale sometimes does not connect (various errors, auth key is no longer valid (which is fine because tailscale has state directory that allows connection which a restart of the service makes it connect)

Additional context / logs:

this feels like a race condition as it does not happen all the time. This could be tailsscales problem or the way i set it up.

This happens on k3s and k3s-agent nodes.

[brandond]

Is k3s/tailscale coming up before the network on your node has settled?

[lukepolo]

Most likely the case, but im not sure how to verify.

[brandond]

Logs would probably be the best place to start?

[lukepolo]

sorry its been a while, here are some logs

journalctl -b -u k3s-agent -u tailscaled
Feb 20 19:29:34 luketestbox systemd[1]: Starting Lightweight Kubernetes...
Feb 20 19:29:35 luketestbox systemd[1]: Starting Tailscale node agent...
Feb 20 19:29:35 luketestbox tailscaled[897]: TPM: successfully read all properties
Feb 20 19:29:35 luketestbox tailscaled[897]: logtail started
Feb 20 19:29:35 luketestbox tailscaled[897]: Program starting: v1.94.1-t62c6f1cd7-g09fea6572, Go 1.25.5: []string{"/usr/sbin/tailscaled", "--state=/var/lib/tailscale/tailscaled.sta>
Feb 20 19:29:35 luketestbox tailscaled[897]: LogID: 30975a674c3a032baff4b6f6ef845671d50c3f43ec51de4e6b636cacde27e25f
Feb 20 19:29:35 luketestbox tailscaled[897]: logpolicy: using $STATE_DIRECTORY, "/var/lib/tailscale"
Feb 20 19:29:35 luketestbox tailscaled[897]: dns: [resolved-ping=yes rc=resolved resolved=file nm=no resolv-conf-mode=stub ret=systemd-resolved]
Feb 20 19:29:35 luketestbox tailscaled[897]: dns: using "systemd-resolved" mode
Feb 20 19:29:35 luketestbox tailscaled[897]: creating dns cleanup: route ip+net: no such network interface
Feb 20 19:29:36 luketestbox k3s[916]: time="2026-02-20T19:29:36Z" level=info msg="Starting k3s agent v1.34.3+k3s1 (48ffa7b6)"
Feb 20 19:29:36 luketestbox k3s[916]: time="2026-02-20T19:29:36Z" level=info msg="Starting VPN: tailscale"
Feb 20 19:29:37 luketestbox systemd[1]: Started Tailscale node agent.
Feb 20 19:29:37 luketestbox tailscaled[897]: wgengine.NewUserspaceEngine(tun "tailscale0") ...
Feb 20 19:29:37 luketestbox tailscaled[897]: dns: [resolved-ping=yes rc=resolved resolved=file nm=no resolv-conf-mode=stub ret=systemd-resolved]
Feb 20 19:29:37 luketestbox tailscaled[897]: dns: using "systemd-resolved" mode
Feb 20 19:29:37 luketestbox tailscaled[897]: dns: using *dns.resolvedManager
Feb 20 19:29:37 luketestbox tailscaled[897]: link state: interfaces.State{defaultRoute=eno1 ifs={eno1:[10.101.1.134/22 llu6]} v4=true v6=false}
Feb 20 19:29:37 luketestbox tailscaled[897]: magicsock: disco key = d:d8ef510e17d51969
Feb 20 19:29:37 luketestbox tailscaled[897]: Creating WireGuard device...
Feb 20 19:29:37 luketestbox tailscaled[897]: router: portUpdate(port=41641, network=udp6)
Feb 20 19:29:37 luketestbox tailscaled[897]: router: using firewall mode pref
Feb 20 19:29:37 luketestbox tailscaled[897]: Bringing WireGuard device up...
Feb 20 19:29:37 luketestbox tailscaled[897]: external route: up
Feb 20 19:29:37 luketestbox tailscaled[897]: Bringing router up...
Feb 20 19:29:37 luketestbox tailscaled[897]: router: default choosing iptables
Feb 20 19:29:37 luketestbox tailscaled[897]: router: netfilter running in iptables mode v6 = true, v6filter = true, v6nat = true
Feb 20 19:29:37 luketestbox tailscaled[897]: router: portUpdate(port=41641, network=udp4)
Feb 20 19:29:37 luketestbox tailscaled[897]: Clearing router settings...
Feb 20 19:29:37 luketestbox tailscaled[897]: Starting network monitor...
Feb 20 19:29:37 luketestbox tailscaled[897]: Engine created.
Feb 20 19:29:37 luketestbox tailscaled[897]: pm: using backend prefs for "profile-e936": Prefs{ra=false dns=true want=true routes=[] statefulFiltering=false nf=on update=check Pers>
Feb 20 19:29:37 luketestbox tailscaled[897]: envknob: PORT="41641"
Feb 20 19:29:37 luketestbox tailscaled[897]: monitor: gateway and self IP changed: gw=10.101.0.1 self=10.101.1.134
Feb 20 19:29:37 luketestbox tailscaled[897]: logpolicy: using $STATE_DIRECTORY, "/var/lib/tailscale"
Feb 20 19:29:37 luketestbox tailscaled[897]: linkChange: in state NoState; PAC or proxyConfig changed; updating routes
Feb 20 19:29:37 luketestbox tailscaled[897]: got LocalBackend in 15ms
Feb 20 19:29:37 luketestbox tailscaled[897]: Start
Feb 20 19:29:37 luketestbox tailscaled[897]: ipnext: active extensions: relayserver, taildrop, conn25, portlist, posture, clientupdate
Feb 20 19:29:37 luketestbox tailscaled[897]: Backend: logs: be:30975a674c3a032baff4b6f6ef845671d50c3f43ec51de4e6b636cacde27e25f fe:
Feb 20 19:29:37 luketestbox tailscaled[897]: control: client.Login(0)
Feb 20 19:29:37 luketestbox tailscaled[897]: control: doLogin(regen=false, hasUrl=false)
Feb 20 19:29:37 luketestbox tailscaled[897]: health(warnable=warming-up): error: Tailscale is starting. Please wait.
Feb 20 19:29:38 luketestbox tailscaled[897]: localapi: [POST] /localapi/v0/check-prefs
Feb 20 19:29:38 luketestbox tailscaled[897]: localapi: [POST] /localapi/v0/start
Feb 20 19:29:38 luketestbox tailscaled[897]: Start
Feb 20 19:29:38 luketestbox tailscaled[897]: Start: updated prefs: Prefs{ra=false dns=true want=true routes=[] statefulFiltering=false nf=on update=check Persist{o=, n=[xDZKL] u="q>
Feb 20 19:29:38 luketestbox tailscaled[897]: Backend: logs: be:30975a674c3a032baff4b6f6ef845671d50c3f43ec51de4e6b636cacde27e25f fe:
Feb 20 19:29:38 luketestbox tailscaled[897]: control: client.Login(0)
Feb 20 19:29:38 luketestbox tailscaled[897]: control: doLogin(regen=false, hasUrl=false)
Feb 20 19:29:38 luketestbox tailscaled[897]: control: client.Shutdown ...
Feb 20 19:29:38 luketestbox tailscaled[897]: control: updateRoutine: exiting
Feb 20 19:29:38 luketestbox tailscaled[897]: control: mapRoutine: exiting
Feb 20 19:29:38 luketestbox tailscaled[897]: control: authRoutine: exiting
Feb 20 19:29:38 luketestbox tailscaled[897]: control: Client.Shutdown done.
Feb 20 19:29:38 luketestbox tailscaled[897]: health(warnable=login-state): error: You are logged out. The last login error was: fetch control key: Get "https://controlplane.tailsca>
Feb 20 19:29:38 luketestbox tailscaled[897]: control: control server key from https://controlplane.tailscale.com: ts2021=[fSeS+], legacy=[nlFWp]
Feb 20 19:29:38 luketestbox tailscaled[897]: control: RegisterReq: onode= node=[xDZKL] fup=false nks=false
Feb 20 19:29:38 luketestbox tailscaled[897]: control: RegisterReq: got response; nodeKeyExpired=false, machineAuthorized=false; authURL=false
Feb 20 19:29:38 luketestbox tailscaled[897]: Received error: invalid key: API key does not exist
Feb 20 19:29:38 luketestbox tailscaled[897]: health(warnable=login-state): error: You are logged out. The last login error was: invalid key: API key does not exist
Feb 20 19:29:38 luketestbox k3s[916]: time="2026-02-20T19:29:38Z" level=fatal msg="Error: tailscale up failed: backend error: invalid key: API key does not exist\n: exit status 1"
Feb 20 19:29:38 luketestbox systemd[1]: k3s-agent.service: Main process exited, code=exited, status=1/FAILURE
Feb 20 19:29:38 luketestbox systemd[1]: k3s-agent.service: Failed with result 'exit-code'.
Feb 20 19:29:38 luketestbox systemd[1]: Failed to start Lightweight Kubernetes.
Feb 20 19:29:38 luketestbox tailscaled[897]: control: doLogin(regen=false, hasUrl=false)
Feb 20 19:29:38 luketestbox tailscaled[897]: control: RegisterReq: onode= node=[xDZKL] fup=false nks=false
Feb 20 19:29:42 luketestbox tailscaled[897]: health(warnable=warming-up): ok
Feb 20 19:29:44 luketestbox systemd[1]: k3s-agent.service: Scheduled restart job, restart counter is at 1.
Feb 20 19:29:44 luketestbox systemd[1]: Stopped Lightweight Kubernetes.
Feb 20 19:29:44 luketestbox systemd[1]: Starting Lightweight Kubernetes...
Feb 20 19:29:44 luketestbox k3s[1345]: time="2026-02-20T19:29:44Z" level=info msg="Starting k3s agent v1.34.3+k3s1 (48ffa7b6)"
Feb 20 19:29:44 luketestbox k3s[1345]: time="2026-02-20T19:29:44Z" level=info msg="Starting VPN: tailscale"
Feb 20 19:29:44 luketestbox tailscaled[897]: localapi: [POST] /localapi/v0/check-prefs
Feb 20 19:29:44 luketestbox tailscaled[897]: localapi: [POST] /localapi/v0/start
Feb 20 19:29:44 luketestbox tailscaled[897]: Start
Feb 20 19:29:44 luketestbox tailscaled[897]: Start: updated prefs: Prefs{ra=false dns=true want=true routes=[] statefulFiltering=false nf=on update=check Persist{o=, n=[xDZKL] u="q>
Feb 20 19:29:44 luketestbox tailscaled[897]: Backend: logs: be:30975a674c3a032baff4b6f6ef845671d50c3f43ec51de4e6b636cacde27e25f fe:
Feb 20 19:29:44 luketestbox tailscaled[897]: health(warnable=login-state): ok
Feb 20 19:29:44 luketestbox tailscaled[897]: control: client.Login(0)
Feb 20 19:29:44 luketestbox tailscaled[897]: control: client.Shutdown ...
Feb 20 19:29:44 luketestbox tailscaled[897]: control: updateRoutine: exiting
Feb 20 19:29:44 luketestbox tailscaled[897]: control: doLogin(regen=false, hasUrl=false)
Feb 20 19:29:44 luketestbox tailscaled[897]: control: mapRoutine: exiting
Feb 20 19:29:44 luketestbox tailscaled[897]: control: authRoutine: exiting
Feb 20 19:29:44 luketestbox tailscaled[897]: health(warnable=login-state): error: You are logged out.
Feb 20 19:29:44 luketestbox tailscaled[897]: control: Client.Shutdown done.
Feb 20 19:29:44 luketestbox tailscaled[897]: control: control server key from https://controlplane.tailscale.com: ts2021=[fSeS+], legacy=[nlFWp]

after a tailscaled restart

root@luketestbox:/home/qadmin# systemctl restart tailscaled
root@luketestbox:/home/qadmin# tailscale status
100.94.184.103   qx                    qx.tailcec4f.ts.net  linux  -
100.65.161.9     5stack-servers-2      tagged-devices       linux  offline, last seen 2d ago
100.113.57.101   dev                   tagged-devices       linux  -
100.116.118.126  fivestack             tagged-devices       linux  active; relay "ord", tx 19844 rx 26340
100.94.150.43    lukes-macbook-pro-qx  tagged-devices       macOS  offline, last seen 105d ago

[brandond]

says your keys are invalid?

Feb 20 19:29:38 luketestbox tailscaled[897]: health(warnable=login-state): error: You are logged out. The last login error was: fetch control key: Get "https://controlplane.tailsca>
Feb 20 19:29:38 luketestbox tailscaled[897]: control: control server key from https://controlplane.tailscale.com: ts2021=[fSeS+], legacy=[nlFWp]
Feb 20 19:29:38 luketestbox tailscaled[897]: control: RegisterReq: onode= node=[xDZKL] fup=false nks=false
Feb 20 19:29:38 luketestbox tailscaled[897]: control: RegisterReq: got response; nodeKeyExpired=false, machineAuthorized=false; authURL=false
Feb 20 19:29:38 luketestbox tailscaled[897]: Received error: invalid key: API key does not exist
Feb 20 19:29:38 luketestbox tailscaled[897]: health(warnable=login-state): error: You are logged out. The last login error was: invalid key: API key does not exist

That doesn't seem like something we can fix.

[lukepolo]

while the logs say they are incorrect, once i do a tailscaled restart it works. so im not sure if this is a tailscale issue or a k3s issue, as a restart of the tailscaled restart fixes it.

So the keys are valid, it just happens to fail the first time

[lukepolo]

ok i found this relevant issue , its probalby the same issue

tailscale/tailscale#17047

it semes that we have to remove the --auth-key parameter

[lukepolo]

For anyone that runs into this.

        sed -i '/vpn-auth/d' /etc/systemd/system/k3s-agent.service
        systemctl daemon-reload
        systemctl restart k3s-agent

this seems to work and doesn't break tailscale connection

i instal the k3s-agent like this

curl -sfL https://get.k3s.io | K3S_URL=https://${process.env.TAILSCALE_NODE_IP}:6443 K3S_TOKEN=${process.env.K3S_TOKEN} sh -s - --node-name ${gameServerNodeId} --vpn-auth="name=tailscale,joinKey=${game_server_nodes_by_pk.token}"

[brandond]

Seems like tailscaled does not handle having the join key set when it's restarted, if the key is expired. It should probably ignore that and use the existing creds, but instead it tries to reuse the api key to rejoin, which fails if the key no longer exists.