.snapshots folder talking up all disk space on control planes

[mkreis]

Hi all,
my cluster failed due to disk pressure. It turns out the control planes disk space was used up to 99%, almost everything by the .snapshots folders.

Snapper shows the following snapshots:

k3s-control-plane-hel1-gtg:~ # snapper list
  # | Type   | Pre # | Date                     | User | Used Space | Cleanup | Description           | Userdata
----+--------+-------+--------------------------+------+------------+---------+-----------------------+--------------
 0  | single |       |                          | root |            |         | current               |
 1  | single |       | Tue Dec 12 02:17:53 2023 | root | 108.82 MiB | number  | first root filesystem | important=yes
 3- | single |       | Wed Dec 13 14:55:39 2023 | root | 161.13 MiB | number  | Snapshot Update of #2 |
27  | single |       | Fri Mar  8 00:57:44 2024 | root | 928.47 MiB | number  | Snapshot Update of #3 |
28  | single |       | Sat Mar  9 00:16:46 2024 | root | 928.47 MiB | number  | Snapshot Update of #3 |
29  | single |       | Sun Mar 10 00:24:03 2024 | root | 928.47 MiB | number  | Snapshot Update of #3 |
30  | single |       | Mon Mar 11 00:54:17 2024 | root | 928.47 MiB | number  | Snapshot Update of #3 |
31  | single |       | Tue Mar 12 01:42:25 2024 | root | 928.47 MiB | number  | Snapshot Update of #3 |
32  | single |       | Wed Mar 13 01:07:42 2024 | root | 928.47 MiB | number  | Snapshot Update of #3 |
33  | single |       | Thu Mar 14 00:28:37 2024 | root | 928.47 MiB | number  | Snapshot Update of #3 |
34  | single |       | Fri Mar 15 01:23:21 2024 | root | 928.47 MiB | number  | Snapshot Update of #3 |
35  | single |       | Sat Mar 16 01:28:36 2024 | root | 928.47 MiB | number  | Snapshot Update of #3 |
36  | single |       | Sun Mar 17 01:32:07 2024 | root | 928.47 MiB | number  | Snapshot Update of #3 |
....

Does anyone know what could be the reason for this and how I can prevent this happening again?

[mkreis]

Btw. I could resolve the situation by doing the following with each control plane:

• rebooting (it had issues to switch to the latest snapshot, which worked after reboot)
• execute snapper cleanup number

[derklaro]

Had the same issue, but deleted all snapshots using snapper delete 1-<max num> (after rebooting) - this skips all snapshots that are important.

[mkreis]

Hmm... ok. I hope this is something which can be fixed, I don't want to clean up manually every few weeks....