Reducing DefaultSnapshotCatchUpEntries

[yipal]

Hi Folks,
I have an application what writes large values to etcd (can be larger than 1MB), and I believe I am running into #12548 where the etcd server consumes a lot of memory. For my application, I would like to limit the maximum memory consumption of etcd to something lower, like 3GB.

Even when I set

ETCD_AUTO_COMPACTION_MODE=revision
ETCD_AUTO_COMPACTION_RETENTION=200

the RSS memory consumption can exceed 6+GB.

However, when I reduce DefaultSnapshotCatchUpEntries to 100, the memory consumption is seems to stay below 3 GB.

In the code, there is a comment justifying the 5000 value for DefaultSnapshotCatchUpEntries, and I am curious when this value comes into play. What is the drawback if DefaultSnapshotCatchUpEntries is too small? Will a follower that takes a long time to do its local snapshot need to retrieve the entire snapshot from the leader instead of catching up on only the recent log entries?

// DefaultSnapshotCatchUpEntries is the number of entries for a slow follower
// to catch-up after compacting the raft storage entries.
// We expect the follower has a millisecond level latency with the leader.
// The max throughput is around 10K. Keep a 5K entries is enough for helping
// follower to catch up.
DefaultSnapshotCatchUpEntries uint64 = 5000

thank you

[serathius]

#17098