From be1d5c793c4cd17cb384a13142336a4b39ecb21c Mon Sep 17 00:00:00 2001 From: Louis PORTAY Date: Tue, 1 Apr 2025 12:39:28 +0200 Subject: [PATCH 1/2] docs(kapsule): add etcd clean-up tutorial --- .../kubernetes/how-to/recover-space-etcd.mdx | 54 +++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 pages/kubernetes/how-to/recover-space-etcd.mdx diff --git a/pages/kubernetes/how-to/recover-space-etcd.mdx b/pages/kubernetes/how-to/recover-space-etcd.mdx new file mode 100644 index 0000000000..91a04d0472 --- /dev/null +++ b/pages/kubernetes/how-to/recover-space-etcd.mdx @@ -0,0 +1,54 @@ +--- +meta: +title: Recovering ETCD Database space for a Kapsule/Kosmos cluster +description: Learn how to reclaim Database space for your Kubernetes Kapsule and Kosmos clusters to stay below your quota. +content: +h1: Recovering ETCD Database space for a Kapsule/Kosmos cluster +paragraph: Learn how to reclaim Database space for your Kubernetes Kapsule and Kosmos clusters to stay below your quota. +tags: kubernetes kapsule kosmos etcd +dates: +validation: 2025-04-01 +posted: 2025-04-01 +categories: + - containers +--- + +Kubernetes Kapsule clusters have quotas on the space they can occupy on an etcd database. See Kapsule [cluster-types](https://api.scaleway.com/k8s/v1/regions/fr-par/cluster-types) for details on each offer. +You can see your current cluster space consumption at any time in your cluster grafana dashboard (`Etcd disk usage` panel). +This guide helps you to free up space on your database to avoid reaching this limit. + + + +- [Created](/kubernetes/how-to/create-cluster/) a Kubernetes Kapsule cluster +- [Downloaded](/kubernetes/how-to/connect-cluster-kubectl/) the Kubeconfig + +* Looking for unused resources is a good approach, delete any Secrets, large ConfigMaps that are not used anymore in your cluster. + + ```sh + > kubectl -n $namespace delete $ConfigMapName + ``` + +* keep an eye on Helm Charts that are deploying a lot of custom resources (CRDs), they tend to fill up etcd space. You can find them by showing resource kinds + + ```sh + > kubectl api-resources + NAME SHORTNAMES APIVERSION NAMESPACED KIND + configmaps cm v1 true ConfigMap + endpoints ep v1 true Endpoints + events ev v1 true Event + cronjobs cj batch/v1 true CronJob + jobs batch/v1 true Job + [...] + ``` +look for resources with an external apiversion (not _v1_, _apps/v1_, _storage.k8s.io/v1_ or _batch/v1_ for example). + + + It is known that cluster with many nodes and with at least one GPU node may have a lot of _nodefeatures_ objects polluting their etcd space. We are working on a long-term fix but for now manually deleting these objects and downsizing the cluster (or upgrading to a dedicated offer with a bigger etcd quota) is the best solution. + + +* If you have a doubt on space taken by a resource, you can dump it to get its size + + ```sh + > kubectl get nodefeature -n kube-system $node-feature-name -o yaml | wc -c + 305545 // ~300KiB, big object + ``` \ No newline at end of file From d4abc683a26345794b4aa04db0c2e290e19001af Mon Sep 17 00:00:00 2001 From: Louis Portay <20128391+louisportay@users.noreply.github.com> Date: Tue, 1 Apr 2025 14:27:22 +0200 Subject: [PATCH 2/2] apply suggestions Co-authored-by: Benedikt Rollik --- pages/kubernetes/how-to/recover-space-etcd.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/pages/kubernetes/how-to/recover-space-etcd.mdx b/pages/kubernetes/how-to/recover-space-etcd.mdx index 91a04d0472..c5c1a2b2d9 100644 --- a/pages/kubernetes/how-to/recover-space-etcd.mdx +++ b/pages/kubernetes/how-to/recover-space-etcd.mdx @@ -1,10 +1,10 @@ --- meta: -title: Recovering ETCD Database space for a Kapsule/Kosmos cluster -description: Learn how to reclaim Database space for your Kubernetes Kapsule and Kosmos clusters to stay below your quota. +title: Recovering ETCD database space for a Kapsule/Kosmos cluster +description: Learn how to reclaim database space for your Kubernetes Kapsule and Kosmos clusters to stay below your quota. content: -h1: Recovering ETCD Database space for a Kapsule/Kosmos cluster -paragraph: Learn how to reclaim Database space for your Kubernetes Kapsule and Kosmos clusters to stay below your quota. +h1: Recovering ETCD database space for a Kapsule/Kosmos cluster +paragraph: Learn how to reclaim database space for your Kubernetes Kapsule and Kosmos clusters to stay below your quota. tags: kubernetes kapsule kosmos etcd dates: validation: 2025-04-01 @@ -40,7 +40,7 @@ This guide helps you to free up space on your database to avoid reaching this li jobs batch/v1 true Job [...] ``` -look for resources with an external apiversion (not _v1_, _apps/v1_, _storage.k8s.io/v1_ or _batch/v1_ for example). +Look for resources with an external apiversion (not _v1_, _apps/v1_, _storage.k8s.io/v1_ or _batch/v1_ for example). It is known that cluster with many nodes and with at least one GPU node may have a lot of _nodefeatures_ objects polluting their etcd space. We are working on a long-term fix but for now manually deleting these objects and downsizing the cluster (or upgrading to a dedicated offer with a bigger etcd quota) is the best solution.