missing confstate information after upgrading etcd to v3.6

[DTLP]

Hi,

After upgrading etcd from v3.5.22 to v3.6.0 I'm observing the following in the logs:

{"level":"warn","ts":"2025-08-06T08:33:47.483064Z","caller":"etcdserver/server.go:2304","msg":"Failed to detect schema version","error":"missing confstate information"}
{"level":"error","ts":"2025-08-06T08:33:47.483160Z","caller":"version/monitor.go:120","msg":"failed to update storage version","cluster-version":"3.6.0","error":"cannot detect storage schema version: missing confstate information","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver/version.(*Monitor).UpdateStorageVersionIfNeeded\n\tgo.etcd.io/etcd/server/v3/etcdserver/version/monitor.go:120\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).monitorStorageVersion\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2345\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2526"}

The cluster reports healthy, but these logs are concerning.

$ etcdctl endpoint health --cluster
https://10.22.22.4:2379 is healthy: successfully committed proposal: took = 33.600101ms
https://10.22.22.132:2379 is healthy: successfully committed proposal: took = 36.27905ms
https://10.22.22.68:2379 is healthy: successfully committed proposal: took = 37.965187ms

What I know so far:

• Rolling back to v3.5.22 stops the error
• Upgrading to v3.6.4 brings the error back
• v2store was empty before the upgrade to v3.6
• Restoring etcd to a month old backup and rolling that to v3.6 leads to the same issue
• We have 2 similar clusters that didn't have this issue after the upgrade - one of them required cleaning upv2store

So at this point I'm not sure whether something is going wrong during the upgrade or if the issue was there prior the upgrade, but only surfaced in v3.6.

I would really appreciate some help getting to the bottom of this.

Thank you

[ahrtr]

{"level":"warn","ts":"2025-08-06T08:33:47.483064Z","caller":"etcdserver/server.go:2304","msg":"Failed to detect schema version","error":"missing confstate information"}

etcd can automatically populate the missing info. So warning message should disappear automatically.

If you want to debug this, try steps below,

Before upgrading to 3.6.4 (when you are still on 3.5.22), run command below to check whether the confState is present,

$ ./etcd-dump-db iterate-bucket ~/tmp/etcd/default.etcd/member/snap/db meta --decode
key="term", value=2
key="consistent_index", value=4
key="confState", value="{\"voters\":[10276657743932975437],\"auto_leave\":false}"

After upgrading to 3.6.x, if you see the warning message "missing confstate information", stop etcd and run the above command again to check the confState is present. But etcd should be automatically populate it even it's missing, everything should be still working well. Please let me know if you see any real issues.

[ahrtr]

BTW, the etcd-dump-db is https://github.com/etcd-io/etcd/tree/main/tools/etcd-dump-db

[DTLP]

Heya. Thank you for this.
Here's v3.5.20:

$ sudo ./etcd-dump-db iterate-bucket /var/lib/etcd/member/snap/db meta --decode
key="term", value="\x00\x00\x00\x00\x00\x00\x00\x04"
key="scheduledCompactRev", value="\x00\x00\x00\x00<c$0_\x00\x00\x00\x00\x00\x00\x00\x00"
key="finishedCompactRev", value="\x00\x00\x00\x00<c$0_\x00\x00\x00\x00\x00\x00\x00\x00"
key="consistent_index", value="\x00\x00\x00\x00\x00\x00\x14\xab"

and v3.6.4:

$ sudo ./etcd-dump-db-amd64 iterate-bucket /var/lib/etcd/member/snap/db meta --decode
key="term", value="\x00\x00\x00\x00\x00\x00\x00\x06"
key="scheduledCompactRev", value="\x00\x00\x00\x00<c-B_\x00\x00\x00\x00\x00\x00\x00\x00"
key="finishedCompactRev", value="\x00\x00\x00\x00<c-B_\x00\x00\x00\x00\x00\x00\x00\x00"
key="consistent_index", value="\x00\x00\x00\x00\x00\x00\x1dk"

But etcd should be automatically populate it even it's missing, everything should be still working well.

Unfortunately, in our case it just keeps printing those error messages over and over again. I left the cluster running over the weekend and it's still complaining about confstate being missing. Functionality-wise, however, everything seems to be fine.

[ahrtr]

I reproduced the issue.

The field confState was added since etcd v3.5. If an etcd cluster was created since v3.4, then it wouldn't have this field. The field will only be automatically populated during conf change (i.e. add/remove/update member). So when upgrading from 3.4 to 3.5, and from 3.5 to 3.6, the field will never be populated unless users add or remove or update members.

Note if you can't reproduce this issue, it should be because that there is no v2 snapshot, and etcdserver will replay all WAL records (including add member records) on bootstrap.

The simplest solution is to remove code snip below,

etcd/server/storage/schema/schema.go

Lines 103 to 106 in 05b8563

	confstate := UnsafeConfStateFromBackend(lg, tx)
	if confstate == nil {
	return v, fmt.Errorf("missing confstate information")
	}

[ahrtr]

Just raised an issue #20495

[DTLP]

Legend! Thank you so much for your help!

[ahrtr]

Resolved. The fix will be included in next etcd patch release v3.6.5.