Node suffers from DiskPressure due to btrfs snapshots

[devamin]

Hello, and thank you for maintaining this wonderful project.

One of my K8s nodes is suffering from DiskPressure leading to evicted pods.

$ du -h / --max-depth=1 | sort -hr | head -n 10

du: cannot access '/proc/12134/task/12134/fd/4': No such file or directory
du: cannot access '/proc/12134/task/12134/fdinfo/4': No such file or directory
du: cannot access '/proc/12134/fd/3': No such file or directory
du: cannot access '/proc/12134/fdinfo/3': No such file or directory
40G     /
31G     /.snapshots
6.3G    /var
1.1G    /run
1020M   /usr
40M     /boot
28M     /etc
4.8M    /root
108K    /dev
0       /tmp

/.snapshots is taking the big part of the disk, how can I configure an auto cleaner, or how to clean this manual without affecting anything ?

[martinatwainobicom]

Hi,
I have exactly the same issue in my cluster on a control-plane node using kube-hetzner version: 2.11.7

Using snapper list I've got the following list of snapshots which matches the snapshots in /.snapshots directory.

| Type | Pre # | Date | User | Used Space | Cleanup | Description | Userdata

-----+--------+-------+--------------------------+------+------------+---------+------------------------+--------------
0 | single | | | root | | | current |
37 | single | | Tue Jan 16 01:16:13 2024 | root | 77.39 MiB | number | Snapshot Update of #36 | important=yes
38 | single | | Wed Jan 17 00:33:10 2024 | root | 38.06 MiB | number | Snapshot Update of #37 | important=yes
39 | single | | Thu Jan 18 01:43:05 2024 | root | 39.42 MiB | number | Snapshot Update of #38 | important=yes
40- | single | | Fri Jan 19 01:26:41 2024 | root | 75.70 MiB | number | Snapshot Update of #39 |
87 | single | | Thu Mar 7 01:09:02 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
88 | single | | Fri Mar 8 00:43:23 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
89 | single | | Sat Mar 9 00:58:52 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
90 | single | | Sun Mar 10 01:56:46 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
91 | single | | Mon Mar 11 01:18:26 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
92 | single | | Tue Mar 12 00:44:44 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
93 | single | | Wed Mar 13 00:42:18 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
94 | single | | Thu Mar 14 01:17:45 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
95 | single | | Fri Mar 15 00:35:10 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
96 | single | | Sat Mar 16 00:44:22 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
97 | single | | Sun Mar 17 00:39:56 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
98 | single | | Mon Mar 18 00:53:57 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
99 | single | | Tue Mar 19 01:13:20 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
100 | single | | Wed Mar 20 01:11:21 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
101 | single | | Thu Mar 21 00:18:58 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
102 | single | | Fri Mar 22 00:42:03 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
103 | single | | Sat Mar 23 01:26:08 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
104 | single | | Sun Mar 24 00:53:45 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
105 | single | | Mon Mar 25 01:31:40 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
106 | single | | Tue Mar 26 01:40:46 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
107 | single | | Wed Mar 27 01:52:48 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
108 | single | | Thu Mar 28 01:29:42 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
109 | single | | Fri Mar 29 00:44:38 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
110 | single | | Sat Mar 30 01:57:30 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
111 | single | | Sun Mar 31 00:41:59 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
112 | single | | Mon Apr 1 01:49:26 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
113 | single | | Tue Apr 2 00:45:10 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
114 | single | | Wed Apr 3 01:36:00 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
115 | single | | Thu Apr 4 01:20:39 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
116 | single | | Fri Apr 5 00:14:31 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
117 | single | | Sat Apr 6 00:04:50 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
118 | single | | Sun Apr 7 01:07:05 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |
119+ | single | | Mon Apr 8 00:15:49 2024 | root | 808.62 MiB | number | Snapshot Update of #40 |

Some interesting points here:

• Snapshots from 41-86 do not exist (they probably got deleted)
• Starting from Snapshot 87, each snapshot is quite large and they all have exactly the same size
• Starting from Snapshot 87, the description is always "Snapshot Update of 40"

In theory (I am absolutely no expert here!) it should be possible to delete a snapshot using: snapper delete [SNAPSHOTNUMBER]
https://documentation.suse.com/sles/12-SP5/html/SLES-all/cha-snapper.html

I'm looking forward to any help here, as I really do not want to run the control-plane node out of disk space.

[martinatwainobicom]

@devamin Maybe this helps - even though it is only a temporary solution.

I've tried to delete one snapshot but that did not work, because the default snapshot was depending on it. You can see a list of all snapshots related to the default one using: sudo btrfs subvolume show /

I also realized that the last reboot of the control plane node matched the 19th of January. So after this I:

• made a backup of the etcd store
• drained the control plane
• rebooted it

After reboot, the sizes of all the snapshots (snapper list) changed and where not the same anymore.
Furthermore with sudo btrfs subvolume show / the list of snapshots was empty and I could delete snapshots using snapper, resolving the disk pressure problem (for now).

[devamin]

Thank you @martinatwainobicom , I will try to apply this.

[Givemeurcookies]

I had the same issue 2-3 weeks ago, I did a cleanup on the control nodes and that fixed it. I've been running it this project for over a year now and this is the first time I've experienced this. Since more people experienced this around the same time, I suspect it's triggered by a bad build or something similar.