-
Notifications
You must be signed in to change notification settings - Fork 2.6k
Description
Environmental Info:
K3s Version:
k3s version v1.35.1+k3s1 (50fa2d7)
go version go1.25.6
Node(s) CPU architecture, OS, and Version:
Debian Bookworm - 6.1.0-43-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.162-1 (2026-02-08) x86_64 GNU/Linux
Cluster Configuration:
Single server with embedded SQLite (default configuration)
Describe the bug:
After upgrading from K3s v1.34.1+k3s1 to v1.35.1+k3s1, the cluster logs continuous errors every second related to parsing the compact revision:
E0215 09:43:39.094814 3513 compact.go:265] etcd: endpoint ([unix://kine.sock]): failed to parse compact revision: strconv.ParseInt: parsing "": invalid syntax
The root cause is a missing database migration in Kine. The format of the value column in compact_rev_key and compact_rev_key_apiserver rows changed between Kine v0.14.0 (bundled with K3s v1.34.1) and v0.14.11 (bundled with K3s v1.35.1):
| Entry | Old format (Kine v0.14.0) | New format (Kine v0.14.11) |
|---|---|---|
compact_rev_key |
empty (data in prev_revision column) |
0|<revision> |
compact_rev_key_apiserver |
<counter> |
<counter>|<revision> |
The new Kine version expects a pipe-delimited value but the existing database has the old format, causing strconv.ParseInt to fail when parsing an empty string.
Steps To Reproduce:
- Initial installation some time ago, not sure with which version I started.
- Upgraded to K3s v1.34.1+k3s1
- Cluster ran normally for some time with default SQLite backend
- Upgraded to v1.35.1+k3s1
- Restarted k3s with
systemctl restart k3s - Error appears continuously every second in logs
Expected behavior:
Kine should automatically migrate the compact_rev_key and compact_rev_key_apiserver value columns to the new format on startup, or the upgrade documentation should mention this migration step.
Actual behavior:
The error failed to parse compact revision: strconv.ParseInt: parsing "": invalid syntax is logged every second. The cluster appears to function but with continuous error spam and potential performance impact from retry loops.
Additional context / logs:
Verification that the database is intact but has old format:
# compact_rev_key has empty value column
sqlite3 /var/lib/rancher/k3s/server/db/state.db "SELECT id, name, prev_revision, length(value) FROM kine WHERE id = 1;"
# Returns: 1|compact_rev_key|12035781|0
# compact_rev_key_apiserver entries have old format (no pipe delimiter)
sqlite3 /var/lib/rancher/k3s/server/db/state.db "SELECT id, name, value FROM kine WHERE name = 'compact_rev_key_apiserver' ORDER BY id DESC LIMIT 1;"
# Returns: 14938860|compact_rev_key_apiserver|53525
# Expected format: 53525|14938860
Workaround:
systemctl stop k3s
# Fix compact_rev_key (id=1)
sqlite3 /var/lib/rancher/k3s/server/db/state.db \
"UPDATE kine SET value = '0|' || prev_revision WHERE id = 1 AND name = 'compact_rev_key';"
# Fix ALL compact_rev_key_apiserver entries with old format
sqlite3 /var/lib/rancher/k3s/server/db/state.db \
"UPDATE kine SET value = value || '|' || prev_revision WHERE name = 'compact_rev_key_apiserver' AND length(value) < 10;"
# Commit WAL changes
sqlite3 /var/lib/rancher/k3s/server/db/state.db "PRAGMA wal_checkpoint(TRUNCATE);"
systemctl start k3s
Or have I done something wrong when upgrading? Thanks!
Metadata
Metadata
Assignees
Labels
Type
Projects
Status