How to handle irrelevant v2 custom content when upgrading to 3.6 of etcd

[Timothy-Dougherty]

Recently I've been trying to upgrade some etcd instances to 3.6 and I'm running into an error:

"msg":"illegal v2store content","error":"detected disallowed custom content in v2store"

Now we may have some old v2 custom content, but we migrated a long time ago to v3 and it might just been hanging around doing nothing.
We've also had the v2 endpoint disabled for a long time.

My question is how do we get etcd to ignore or remove this custom content when upgrading to 3.6?

I've tried using the flag:
- --v2-deprecation=write-only-drop-data
but etcd 3.5.21 says its invalid when it starts up.

Perhaps our original migration process was invalid, however all the data is so old that it's truely irrelevant now.

Any guidance would be greatly appreciated.

[lukasertl]

I have exactly the same issue. I even restarted the cluster with V2 enabled to identify any old V2 data, then deleted the old data.

etcdutl check v2store still complains that there is V2 data but I'm at a loss how to identify it.

[jmhbnz]

Hey @Timothy-Dougherty & @lukasertl - Thanks for your question.

If you have a testing environment available, try starting the 3.5.x cluster with v2 enabled you can then try running ETCDCTL_API=2 ./etcdctl ls to review what custom content is stored, followed by ETCDCTL_API=2 ./etcdctl rm <content> to remove keys then ETCDCTL_API=2 ./etcdctl rmdir <content> to remove any directories.

What is the output from ETCDCTL_API=2 ./etcdctl ls?

@ahrtr Do you have any further ideas on what steps to run here?

[ahrtr]

but we migrated a long time ago to v3 and it might just been hanging around doing nothing.

Did you remove the flag --enable-v2 or set it to false a long time ago when you are still on etcd 3.5? If yes, then the reason should be what you already pointed out that there is not much traffic/writes, so the the old snapshot (which contains custom data) isn't rotated yet. So a simple workaround is to set a small value (i.e. 5) to --snapshot-count, and write several key/values to create a new v2 snapshot.

[kvaikla]

etcd cluster is successfully migrated to 3.6.4

ty ahrtr

[DTLP]

$ etcd --version
etcd Version: 3.5.22
Git SHA: 7862c85
Go Version: go1.23.11
Go OS/Arch: linux/amd64

$ sudo ls -la /var/lib/etcd/member/snap/
total 36820
drwx------. 2 etcd etcd     4096 Jul 31 11:03 .
drwx------. 4 etcd etcd     4096 Jul 31 11:02 ..
-rw-r--r--. 1 etcd etcd   894493 Jul 30 22:15 000000000004a80e-00000000592a90b0.snap
-rw-r--r--. 1 etcd etcd   894494 Jul 31 01:23 000000000004a80e-00000000592c1751.snap
-rw-r--r--. 1 etcd etcd   894494 Jul 31 04:34 000000000004a80e-00000000592d9df2.snap
-rw-r--r--. 1 etcd etcd   894494 Jul 31 07:42 000000000004a80e-00000000592f2493.snap
-rw-r--r--. 1 etcd etcd   894486 Jul 31 11:02 000000000004a81b-000000005930acd2.snap
-rw-------. 1 etcd etcd 44797952 Jul 31 11:33 db


$ sudo etcdutl check v2store --data-dir /var/lib/etcd/
Error: detected custom content in v2store

$ ETCDCTL_API=2 etcdctl ls
$ ETCDCTL_API=2 etcdctl user list
$ ETCDCTL_API=2 etcdctl role list
root

$ curl http://127.0.0.1:2379/v2/keys/?recursive=true
{"action":"get","node":{"dir":true}}

In my case upgrading to v3.5.22 and v3.6.4 doesn't help, I'm still getting "error":"detected disallowed custom content in v2store for stage --v2-deprecation=write-only"

[ahrtr]

@DTLP can you share your latest *snap file?

[lukasertl]

@DLTP beat me to it - I just wanted to reply that 3.6.4 didn't fix the issue for me.

The situation looks to me exactly like @DLTP describes. The only V2 data remaining is the root role, and I can't delete it.

[DTLP]

@ahrtr, sorry, I'm afraid I can't share it. Perhaps I can open it up myself and send you the contents of the v2store?

[lukasertl]

Hi @ahrtr - I've shared my snap-file with you. Can you work with this?

[ahrtr]

You need to remove the users and roles (see below) using ETCDCTL_API=2 etcdctl commands. Afterwards, generate a new v2 snapshot by setting a very small value for --snapshot-count (i.e. 1).

users:

• patroni
• root

roles:

• guest
• postgres

Rough steps:

• Start the etcd instance with flag --enable-v2 and --snapshot-count=1
• Remove all users and roles using ETCDCTL_API=2 etcdctl commands, i.e. ETCDCTL_API=2 etcdctl role remove <role-name>;
• Add a key/value using command ETCDCTL_API=3 etcdctl put test test, and then ETCDCTL_API=3 etcdctl del test;
• Stop etcd, and remove --enable-v2 and rollback --snapshot-count change.
• Stop etcd and run etcdutl check v2store --data-dir /var/lib/etcd/. Pls let me know the result.
• ALso try to upgrade to 3.6

@DTLP @lukasertl please let me know if the steps above can resolve your issue.

[DTLP]

Thanks for this! I really appreciate the help.
This helped me realise that apparently I wasn't doing the initial ETCDCTL_API=2 etcdctl ls check correctly. We have an etcd systemd service with a custom config and it is set to store its data in a different directory and I mistakenly ran my commands against etcd with the default config. So we actually got plenty of stuff here to look at:

$ ETCDCTL_API=2 etcdctl ls
/coreos.com
/registry
$ ETCDCTL_API=2 etcdctl rm /coreos.com --recursive
$ ETCDCTL_API=2 etcdctl rm /registry --recursive

I have now successfully rolled to v3.6.4

[lukasertl]

Thanks @ahrtr - following your steps I'm now at a point where etcdutl check reports no v2 content.

However, after upgrading to 3.6.4 etcd panics on startup:

Aug 01 12:31:04 host bash[748264]: panic: detected disallowed v2 WAL for stage --v2-deprecation=write-only [recovered]
Aug 01 12:31:04 host bash[748264]:         panic: execute job failed
Aug 01 12:31:04 host bash[748264]: goroutine 231 [running]:
Aug 01 12:31:04 host bash[748264]: go.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x2?, 0x10001606074?, {0x0?, 0x0?, 0xc000314740?})
Aug 01 12:31:04 host bash[748264]:         go.uber.org/[email protected]/zapcore/entry.go:196 +0x54
Aug 01 12:31:04 host bash[748264]: go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc000303380, {0xc000133f00, 0x2, 0x2})
Aug 01 12:31:04 host bash[748264]:         go.uber.org/[email protected]/zapcore/entry.go:262 +0x24e
Aug 01 12:31:04 host bash[748264]: go.uber.org/zap.(*Logger).Panic(0x11e3d74?, {0x11f2865?, 0xfde840?}, {0xc000133f00, 0x2, 0x2})
Aug 01 12:31:04 host bash[748264]:         go.uber.org/[email protected]/logger.go:285 +0x51
Aug 01 12:31:04 host bash[748264]: go.etcd.io/etcd/pkg/v3/schedule.(*fifo).executeJob.func1()
Aug 01 12:31:04 host bash[748264]:         go.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:202 +0x2c8
Aug 01 12:31:04 host bash[748264]: panic({0xfde840?, 0xc0002fa210?})
Aug 01 12:31:04 host bash[748264]:         runtime/panic.go:791 +0x132
Aug 01 12:31:04 host bash[748264]: go.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x1?, 0x5667a4?, {0x0?, 0x0?, 0xc000418040?})
Aug 01 12:31:04 host bash[748264]:         go.uber.org/[email protected]/zapcore/entry.go:196 +0x54
Aug 01 12:31:04 host bash[748264]: go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0003036c0, {0xc000358d40, 0x1, 0x1})
Aug 01 12:31:04 host bash[748264]:         go.uber.org/[email protected]/zapcore/entry.go:262 +0x24e
Aug 01 12:31:04 host bash[748264]: go.uber.org/zap.(*Logger).Panic(0xc000359f80?, {0x122ef22?, 0x10?}, {0xc000358d40, 0x1, 0x1})
Aug 01 12:31:04 host bash[748264]:         go.uber.org/[email protected]/logger.go:285 +0x51
Aug 01 12:31:04 host bash[748264]: go.etcd.io/etcd/server/v3/etcdserver.v2ToV3Request(_, _)
Aug 01 12:31:04 host bash[748264]:         go.etcd.io/etcd/server/v3/etcdserver/apply_v2.go:31 +0x269
Aug 01 12:31:04 host bash[748264]: go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntryNormal(0xc000386a08, 0xc0003f7910, 0x0)
Aug 01 12:31:04 host bash[748264]:         go.etcd.io/etcd/server/v3/etcdserver/server.go:1953 +0x2a7
Aug 01 12:31:04 host bash[748264]: go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).apply(0xc000386a08, {0xc0005ce030, 0x2f8, 0x7fc2b5a4260d?}, 0xc0001ef0e0, 0xc0002e4770)
Aug 01 12:31:04 host bash[748264]:         go.etcd.io/etcd/server/v3/etcdserver/server.go:1894 +0x84c
Aug 01 12:31:04 host bash[748264]: go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntries(0xc000386a08, 0xc0001ef0e0, 0xc00024a480?)
Aug 01 12:31:04 host bash[748264]:         go.etcd.io/etcd/server/v3/etcdserver/server.go:1194 +0x274
Aug 01 12:31:04 host bash[748264]: go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyAll(0xc000386a08, 0xc0001ef0e0, 0xc00024a480)
Aug 01 12:31:04 host bash[748264]:         go.etcd.io/etcd/server/v3/etcdserver/server.go:979 +0x57
Aug 01 12:31:04 host bash[748264]: go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run.func6({0xc000238700?, 0xfe1e40?})
Aug 01 12:31:04 host bash[748264]:         go.etcd.io/etcd/server/v3/etcdserver/server.go:855 +0x1f
Aug 01 12:31:04 host bash[748264]: go.etcd.io/etcd/pkg/v3/schedule.job.Do(...)
Aug 01 12:31:04 host bash[748264]:         go.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:41
Aug 01 12:31:04 host bash[748264]: go.etcd.io/etcd/pkg/v3/schedule.(*fifo).executeJob(0xc000067f80?, {0x13a8d98?, 0xc0002e3128?}, 0x0?)
Aug 01 12:31:04 host bash[748264]:         go.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:206 +0x7d
Aug 01 12:31:04 host bash[748264]: go.etcd.io/etcd/pkg/v3/schedule.(*fifo).run(0xc000311880)
Aug 01 12:31:04 host bash[748264]:         go.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:187 +0x109
Aug 01 12:31:04 host bash[748264]: created by go.etcd.io/etcd/pkg/v3/schedule.NewFIFOScheduler in goroutine 181
Aug 01 12:31:04 host bash[748264]:         go.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:101 +0x16f

[ahrtr]

@lukasertl Are you still sending v2 requests to etcd cluster? Are you still running ETCDCTL_API=2 etcdctl ... commands after upgrading to 3.6.4?

[lukasertl]

Nevermind, I was able to find a workaround. On 3.5.22:

• set ETCD_MAX_WALS="1"
• set ETCD_SNAPSHOT_COUNT="1"
• restart the cluster
• upgrade to 3.6.4
• change the settings back

Apparently this cleaned out all the old v2 content. I upgraded successfully to 3.6.4.