Increase in DB TOTAL size after upgrade from 3.4 to etcd v3.5.25 while ACTUAL size remains stable

[kavinnath]

We recently upgraded our etcd cluster from v3.4 to v3.5.25. The cluster is stable and functioning correctly.

However, we observed that the DB TOTAL size gradually increases while the ACTUAL space remains relatively stable.

• TOTAL size increased by ~10% within 24 hours after the upgrade.
• The growth rate has since slowed but still continues.
• Compaction runs successfully every 5 minutes.
• We have performed defragmentation, but the trend persists.

I suspect this behavior might be related to the freelist-type = map change introduced in v3.5 (Issue #11258), but I would appreciate confirmation from the maintainers.

Questions:

1. Is this expected behavior after upgrading to v3.5?
2. Could the freelist implementation change affect the reported TOTAL size?
3. Are there recommended settings or operational practices to control this growth?

[kavinnath]

As shown in the trend, the etcd total size began increasing immediately after the cluster upgrade. Interestingly, the actual database size has remained stable both before and after the upgrade. The etcd MVCC put and delete metrics are also steady, indicating that Kubernetes API traffic is not driving this growth.

Please note that the dip in the graph corresponds to defragmentation. Additionally, the rate of increase appears to be gradually accelerating. We are observing the same behavior in our test cluster, which has minimal traffic. This may be related to the architectural aspect mentioned earlier, though I could be mistaken.

Note: You might notice that one node does not show much space reclaimed after defragmentation. In reality, the space was reduced to the actual size post-defrag but increased again immediately after the etcd service started. For clarity, the etcd service was stopped before running the etcdutl defragmentation command. Because the Prometheus scrape interval is relatively high, the brief dip was not captured in the metrics.

But apart from the Total space issue the rest everything is stable. I could see improvement in Disk sync and memory usage post upgrade.

[ahrtr]

However, we observed that the DB TOTAL size gradually increases while the ACTUAL space remains relatively stable.

Can you clarify the TOTAL and ACTUAL sizes? Are you talking about the following metrics or something else?

• etcd_mvcc_db_total_size_in_bytes
• etcd_mvcc_db_total_size_in_use_in_bytes

Please also manually verify the total db size by running command something like below, and check the size of the file ${ETCD_DATA_DIR}/member/snap/db, they should be equal.

curl http://127.0.0.1:2379/metrics | grep etcd_mvcc_db_total_size_in_bytes

[kavinnath]

Can you clarify the TOTAL and ACTUAL sizes? Are you talking about the following metrics or something else?

Yeah i am referring to the metrics. Post upgrade to 3.5.25 the Actual size remain stable but the Total Size is gradually increasing.

etcd_mvcc_db_total_size_in_bytes. = Total Size
etcd_mvcc_db_total_size_in_use_in_bytes. = Actual Size

For More Understanding:

Please also manually verify the total db size by running command something like below, and check the size of the file ${ETCD_DATA_DIR}/member/snap/db, they should be equal.

yeah its matching.

Output Ref:
2.6G size = /data2/etcd/member/snap/db
etcd_mvcc_db_total_size_in_bytes = 2685493248

[ahrtr]

Output Ref:
2.6G size = /data2/etcd/member/snap/db
etcd_mvcc_db_total_size_in_bytes = 2685493248

OK, it means the metrics etcd_mvcc_db_total_size_in_bytes is accurate. Suggest to keep watching for a period of time (e.g. 4 hours, and 1 hour interval) to confirm that they are always be equal.

Not sure whether it's caused by the Kubernetes' behaviour changes. What's the Kubernetes versions before and after the upgrade?

we observed that the DB TOTAL size gradually increases

Does it come to a stable state eventually? If yes, how long? If not, you might want to perform the periodically compaction + defragmentation. Refer to https://github.com/ahrtr/etcd-defrag

[ahrtr]

also what's the OS version (get the systeminfo) and filesystem?

[kavinnath]

We are running Flatcar Linux with kernel version 6.6.60-flatcar.
The etcd data directory is hosted on an ext4 filesystem backed by SSD storage.

Q: Does it come to a stable state eventually? If yes, how long? If not, you might want to perform the periodically compaction + defragmentation. Refer to https://github.com/ahrtr/etcd-defrag

In my test cluster, the etcd storage trend stabilized after about 9–10 days. However, in the main cluster, it has not yet stabilized even after almost 7 days post-upgrade. The storage is still gradually increasing, although the growth rate has slowed compared to earlier. I expect it to settle down soon. Need to wait and watch.

Regarding the compaction we are running Kubernetes API server with default: --etcd-compaction-interval duration Default: 5m0s. Also verified the compaction is working fine.

Q: Not sure whether it's caused by the Kubernetes' behaviour changes. What's the Kubernetes versions before and after the upgrade?

I don’t believe this is due to a change in Kubernetes behavior, as we are not observing any abnormalities in traffic. etcd was upgraded to version 3.5.25, and Kubernetes was upgraded to version 1.29 the following day; however, the increase in total space was noticed immediately after the etcd upgrade, even before the Kubernetes upgrade took place.

The etcd PUT and DELETE request trends are also being shared, and the pattern remains consistent both pre- and post-upgrade.

[ahrtr]

I suspect this behavior might be related to the freelist-type = map change introduced in v3.5 (Issue #11258), but I would appreciate confirmation from the maintainers.

There is a behaviour change when allocating pages. Since a map is an unordered group of of elements, so there are some uncertainties which freelist block ( a span of continuous free pages) to allocate the pages from. To double check this, you can change the freelist type to array, and check whether you can still see the similar behaviour.

--backend-bbolt-freelist-type=array

In my test cluster, the etcd storage trend stabilized after about 9–10 days.

As long as it can stabelize eventually, then I don't think it's a big problem. You just need to deploy https://github.com/ahrtr/etcd-defrag in your environment.

[kavinnath]

Thanks for the input. To validate the behavior, the --backend-bbolt-freelist-type=array flag was enabled on one of the test etcd members. The procedure involved stopping the etcd service, performing a defragmentation using etcdutl, adding the flag, and then restarting the service.

Following this change, the growth trend appears more linear and stable. Monitoring will continue, as a longer observation period is required to gain clearer insight into the sustained behavior. Will keep you posted.

Ref:

[kavinnath]

Old defrag VS New degrag trend ( Along with freelist type change to Array )

[kavinnath]

Total space usage has remained stable for the past 9 hours after changing the flag --backend-bbolt-freelist-type=array, i.e., switching the freelist type from map to array on the specific node. This suggests that freelist-type=map may be the root cause of the abnormality we were observing. If I understand correctly, in etcd version 3.4.x, freelist-type=array is the default—am I right?

As long as it can stabelize eventually, then I don't think it's a big problem. You just need to deploy https://github.com/ahrtr/etcd-defrag in your environment.

It still hasn't stabilized. If it does not settle down soon, I am considering changing the flag --backend-bbolt-freelist-type=array on both my test and production clusters. I understand that this change may result in some performance trade-offs, as mentioned in this issue: #11258. Aside from that, are there any other drawbacks I should be aware of?

I will deploy https://github.com/ahrtr/etcd-defrag on my cluster. However, running defragmentation on a live member blocks the system from reading and writing data while it rebuilds its state. Additionally, some threads mention encountering 5xx errors during the defrag operation. Using etcdutl also requires periodic manual intervention.
Given these challenges, I am leaning toward enabling --backend-bbolt-freelist-type=array.

[ahrtr]

If I understand correctly, in etcd version 3.4.x, freelist-type=array is the default—am I right?

I think so.

I am considering changing the flag --backend-bbolt-freelist-type=array on both my test and production clusters.

It is NOT recommended. The map freelist type has much better performance than array.

I have also deployed https://github.com/ahrtr/etcd-defrag on my cluster. However, running defragmentation on a live member blocks the system from reading and writing data while it rebuilds its state. Additionally, some threads mention encountering 5xx errors during the defrag operation.

Consider using the defragmentation rules, and also deploy it as a cronjob.

[kavinnath]

Note: You might notice that one node does not show much space reclaimed after defragmentation. In reality, the space was reduced to the actual size post-defrag but increased again immediately after the etcd service started. For clarity, the etcd service was stopped before running the etcdutl defragmentation command. Because the Prometheus scrape interval is relatively high, the brief dip was not captured in the metrics.

As I mentioned earlier, after defragmentation, one of the nodes quickly returned to nearly the same space watermark as before, with minimal reduction in disk usage. This is another concern I have regarding the effectiveness of defragmentation. So far, I have observed this behavior only once on a specific node in the production cluster. I will monitor it again when the node approaches the quota limit. And really appreciate your continuous support and guidance.

I am unsure whether this overall increase in total space usage is a bug; however, it would be beneficial if future etcd releases further optimized free space management to help minimize the need for frequent defragmentation.