etcdDatabaseHighFragmentationRatio should change default rule

[zgfh]

https://github.com/etcd-io/etcd/blob/main/contrib/mixin/alerts/alerts.libsonnet#L222
I meet alert: etcdDatabaseHighFragmentationRatio

becuse the default quota space is 2G of etcd

I this is ok if there just 50M fragmentation
should we change the rule to 1024M? like this(1073741824=1024M):

(last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) < 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes > 1073741824

the default alert is

  (last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) < 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes > 104857600

link: prometheus-operator/kube-prometheus#2470

[jmhbnz]

Hi @zgfh - Thanks for raising this idea. We previously merged a change related to this in #15291. Did we miss something in that pr or is the change not working as expected?

[zgfh]

Thanks for your help
I think it would be better if the metrics are ignored unless total database size is 1024MB+

[LTP7534]

I also think that the etcd_mvcc_db_total_size_in_use_in_bytes setting to 100 MB is too sensitive.
It is easy to trigger alarms in some large clusters and we are not very willing to perform debris cleanup just because the debris space may have reached 100 MB

[ahrtr]

Note that there are two conditions, and there is an AND. The alert will only be triggered when both are true.

• (last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) < 0.5
• etcd_mvcc_db_total_size_in_use_in_bytes > 104857600

[jadolg]

@ahrtr Would it make sense to use etcd_server_quota_backend_bytes / 2 if we're not using the default quota given that this number is technically half that?