[Question] failed to find memory cgroup (v2) on Raspberry Pi

[egvimo]

What did you do

• How was the cluster created?
  • export K3D_FIX_CGROUPV2=1 ; k3d cluster create

What did you expect to happen

Cluster is up and running.

Screenshots or terminal output

INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-k3s-default'            
INFO[0000] Created volume 'k3d-k3s-default-images'      
INFO[0000] Starting new tools node...                   
INFO[0000] Starting Node 'k3d-k3s-default-tools'        
INFO[0001] Creating node 'k3d-k3s-default-server-0'     
INFO[0001] Creating LoadBalancer 'k3d-k3s-default-serverlb' 
INFO[0003] Pulling image 'docker.io/rancher/k3d-proxy:5.1.0' 
INFO[0013] Using the k3d-tools node to gather environment information 
INFO[0013] HostIP: using network gateway...             
INFO[0013] Starting cluster 'k3s-default'               
INFO[0013] Starting servers...                          
INFO[0014] Starting Node 'k3d-k3s-default-server-0'     
WARN[0022] warning: encountered fatal log from node k3d-k3s-default-server-0 (retrying 0/10): Ztime="2021-11-12T06:53:31.262056755Z" level=fatal msg="failed to find memory cgroup (v2)" 
WARN[0025] warning: encountered fatal log from node k3d-k3s-default-server-0 (retrying 1/10): Ztime="2021-11-12T06:53:34.601491108Z" level=fatal msg="failed to find memory cgroup (v2)" 
WARN[0029] warning: encountered fatal log from node k3d-k3s-default-server-0 (retrying 2/10): Ztime="2021-11-12T06:53:38.027542897Z" level=fatal msg="failed to find memory cgroup (v2)" 
WARN[0033] warning: encountered fatal log from node k3d-k3s-default-server-0 (retrying 3/10): Ztime="2021-11-12T06:53:42.382432089Z" level=fatal msg="failed to find memory cgroup (v2)" 
WARN[0034] warning: encountered fatal log from node k3d-k3s-default-server-0 (retrying 4/10): Ztime="2021-11-12T06:53:42.382432089Z" level=fatal msg="failed to find memory cgroup (v2)" 
WARN[0037] warning: encountered fatal log from node k3d-k3s-default-server-0 (retrying 5/10): Ztime="2021-11-12T06:53:46.172845418Z" level=fatal msg="failed to find memory cgroup (v2)" 
WARN[0038] warning: encountered fatal log from node k3d-k3s-default-server-0 (retrying 6/10): Ztime="2021-11-12T06:53:46.172845418Z" level=fatal msg="failed to find memory cgroup (v2)" 
WARN[0038] warning: encountered fatal log from node k3d-k3s-default-server-0 (retrying 7/10): Ztime="2021-11-12T06:53:46.172845418Z" level=fatal msg="failed to find memory cgroup (v2)" 
WARN[0039] warning: encountered fatal log from node k3d-k3s-default-server-0 (retrying 8/10): Ztime="2021-11-12T06:53:46.172845418Z" level=fatal msg="failed to find memory cgroup (v2)" 
WARN[0042] warning: encountered fatal log from node k3d-k3s-default-server-0 (retrying 9/10): Ztime="2021-11-12T06:53:50.724987867Z" level=fatal msg="failed to find memory cgroup (v2)" 
ERRO[0042] Failed Cluster Start: Failed to start server k3d-k3s-default-server-0: Node k3d-k3s-default-server-0 failed to get ready: error waiting for log line `k3s is up and running` from node 'k3d-k3s-default-server-0': stopped returning log lines 
ERRO[0042] Failed to create cluster >>> Rolling Back    
INFO[0042] Deleting cluster 'k3s-default'               
WARN[0042] Failed to get details for container /k3d-k3s-default-serverlb 
INFO[0042] Deleting cluster network 'k3d-k3s-default'   
INFO[0042] Deleting image volume 'k3d-k3s-default-images' 
FATA[0042] Cluster creation FAILED, all changes have been rolled back!

Which OS & Architecture

• Linux
• arm64
• Raspberry Pi OS Debian Bullseye (11)

Which version of k3d

k3d version v5.1.0
k3s version v1.21.5-k3s2 (default)

Which version of docker

Client:
 Version:           20.10.5+dfsg1
 API version:       1.41
 Go version:        go1.15.9
 Git commit:        55c4c88
 Built:             Wed Aug  4 19:55:15 2021
 OS/Arch:           linux/arm64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.5+dfsg1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.15.9
  Git commit:       363e9a8
  Built:            Wed Aug  4 19:55:15 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.4.5~ds1
  GitCommit:        1.4.5~ds1-2+deb11u1
 runc:
  Version:          1.0.0~rc93+ds1
  GitCommit:        1.0.0~rc93+ds1-5+b2
 docker-init:
  Version:          0.19.0
  GitCommit:

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 5
 Server Version: 20.10.5+dfsg1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1.4.5~ds1-2+deb11u1
 runc version: 1.0.0~rc93+ds1-5+b2
 init version: 
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.63-v8+
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.63GiB
 Name: raspberrypi
 ID: NDLB:O7ZA:KBT5:JDMM:XKAR:UANF:XKG5:7MH7:2ZL3:3BTP:GG3T:W7WD
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: Support for cgroup v2 is experimental

[iwilltry42]

Hi @egvimo , thanks for opening this issue (transferred it to a Discussion).
It's a matter of setting up your Raspberry Pi to enable the required CGroups (which are enabled on other systems by default).
Copied from here: k3s-io/k3s#2067 (comment)
Add cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 to the end of /boot/firmware/cmdline.txt or /boot/cmdline.txt and reboot.

[iwilltry42]

Maybe switching to cgroup v1 works in the meantime: #493 (comment)
(probably still needs the cmdline stuff)

[egvimo]

Okay, I've managed to downgrade to cgroup v1. Therefore I had to add the following to the /boot/cmdline.txt file:

systemd.unified_cgroup_hierarchy=0 cgroup_memory=1 cgroup_enable=memory

Unfortunately still no luck:

$ k3d cluster create
INFO[0000] Prep: Network                    
INFO[0000] Created network 'k3d-k3s-default'
INFO[0000] Created volume 'k3d-k3s-default-images'
INFO[0000] Starting new tools node...       
INFO[0000] Starting Node 'k3d-k3s-default-tools'
INFO[0001] Creating node 'k3d-k3s-default-server-0'
INFO[0001] Creating LoadBalancer 'k3d-k3s-default-serverlb'
INFO[0002] Using the k3d-tools node to gather environment information
INFO[0002] HostIP: using network gateway... 
INFO[0002] Starting cluster 'k3s-default'   
INFO[0002] Starting servers...              
INFO[0002] Starting Node 'k3d-k3s-default-server-0'
INFO[0018] Starting agents...               
INFO[0018] Starting helpers...              
INFO[0019] Starting Node 'k3d-k3s-default-serverlb'
ERRO[0021] Failed Cluster Start: Failed to add one or more helper nodes: runtime failed to start node 'k3d-k3s-default-serverlb': Failed to inspect container k3d-k3s-default-serverlb for node 06fd8b01fea391ed0b1cb3d037600e35bb26aae866b024b1a9af737e1746ed75: Error response from daemon: readlink /var/lib/docker/overlay2/l: invalid argument
ERRO[0021] Failed to create cluster >>> Rolling Back
INFO[0021] Deleting cluster 'k3s-default'   
WARN[0022] Failed to get details for container /k3d-k3s-default-serverlb
INFO[0024] Deleting cluster network 'k3d-k3s-default'
INFO[0024] Deleting image volume 'k3d-k3s-default-images'
FATA[0024] Cluster creation FAILED, all changes have been rolled back!

[egvimo]

I've figured out what causes the trouble with the system. It's the cgroup_enable=cpuset flag.
If I only add the other two flags (cgroup_memory=1 cgroup_enable=memory), like the error message suggests if using cgroup v1, the system stays stable.

With this knowledge I've successfully installed K3s (with cgroup v2). At the end only the two cgroup flags were necessary. Unfortunately still no luck with K3d. Same error as described above.

[egvimo]

I was a little too hasty. K3s works basically, but as I deployed some apps the CPU went up and stayed there (for all cores). So I think the cpuset flag is missing, but like I said it unfortunately doesn't work on the new OS.

[iwilltry42]

I'm not sure, but I think that if any extra action was needed, then it wouldn't work at all (i.e. complain that some cgroup is missing).
You can do e.g. cat /proc/cgroups to get some information about cgroups on your system.

[Isaaclot]

Add 'cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1' to the end line of file '/boot/cmdline.txt'
instand of new line.

I found my cmdline.text file ,every command is added on one line.
this issue help me a lot.
k3s-io/k3s#2067

[protogia]

For me it made also a difference if i wanted to start k3s.service on masternode and k3s-agent.service on a workernode.

All nodes were RaspberryPi-4 with raspbian-lite-64bit.

Masternode:

uname -r 
6.6.31+rpt-rpi-v8

k3s -v
k3s version v1.30.3+k3s1 (f6466040)
go version go1.22.5

Workernode:

uname -r 
6.6.31+rpt-rpi-v8

k3s -v
k3s version v1.30.3+k3s1 (f6466040)
go version go1.22.5

How i solved:

• To run k3s.service on masternode I needed to add cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 to /boot/cmdline.txt
• To run k3s-agent.service on workernode I needed to add cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 to /boot/firmware/cmdline.txt

Even though that cgroup-v2 was active (checked via mount | grep cgroup)