Kubernetes parameters

[Smolsada]

I am using dragonfly helm chart with resources restrictions.

resources:
  limits:
    cpu: "1000m"
    memory: "512Mi"
  requests:
    cpu: "500m"
    memory: "256Mi"

But I see that the dragonfly itself selects the resource parameters. This can lead to ooms, high CPU consumption (throttling).

I20230517 18:02:16.145570     1 init.cc:56] dragonfly running in opt mode.
I20230517 18:02:16.145646     1 dfly_main.cc:568] Starting dragonfly df-v1.2.1-0b13eaa9436097e6508b82c3ba593dc696f07c21
I20230517 18:02:16.145823     1 dfly_main.cc:599] maxmemory has not been specified. **Deciding myself**....
I20230517 18:02:16.145835     1 dfly_main.cc:603] **Found 13.29GiB** available memory. Setting maxmemory to 10.63GiB
W20230517 18:02:16.145880     1 dfly_main.cc:411] Kernel is older than 5.10, switching to epoll engine.
I20230517 18:02:16.149933     1 proactor_pool.cc:73] **Running 4 io threads**
I20230517 18:02:16.159684     1 server_family.cc:457] Data directory is "/data"
I20230517 18:02:16.162837    10 listener_interface.cc:96] sock[12] AcceptServer - listening on port 6379

How to correctly determine memory and number of threads for dragonfly to be good?

[talbii]

Dragonfly should automatically determine whether it is running inside a container (CGroup), this was introduced here.

Running using docker-compose seems to work fine, so this might be an issue with Helm/Kube

[Smolsada]

I don't know C++ well, but I'll try to figure it out. I would be grateful for any help :)

cat /proc/self/cgroup inside pod:

12:blkio:/kubepods/burstable/pod234cd89f-836a-455c-8389-fee693441908/65fd03ab27d90e42bb386fce84e52834032a83396d627ba3d664e8d4b5086076
11:pids:/kubepods/burstable/pod234cd89f-836a-455c-8389-fee693441908/65fd03ab27d90e42bb386fce84e52834032a83396d627ba3d664e8d4b5086076
10:rdma:/kubepods/burstable/pod234cd89f-836a-455c-8389-fee693441908/65fd03ab27d90e42bb386fce84e52834032a83396d627ba3d664e8d4b5086076
9:cpu,cpuacct:/kubepods/burstable/pod234cd89f-836a-455c-8389-fee693441908/65fd03ab27d90e42bb386fce84e52834032a83396d627ba3d664e8d4b5086076
8:perf_event:/kubepods/burstable/pod234cd89f-836a-455c-8389-fee693441908/65fd03ab27d90e42bb386fce84e52834032a83396d627ba3d664e8d4b5086076
7:freezer:/kubepods/burstable/pod234cd89f-836a-455c-8389-fee693441908/65fd03ab27d90e42bb386fce84e52834032a83396d627ba3d664e8d4b5086076
6:devices:/kubepods/burstable/pod234cd89f-836a-455c-8389-fee693441908/65fd03ab27d90e42bb386fce84e52834032a83396d627ba3d664e8d4b5086076
5:cpuset:/kubepods/burstable/pod234cd89f-836a-455c-8389-fee693441908/65fd03ab27d90e42bb386fce84e52834032a83396d627ba3d664e8d4b5086076
4:net_cls,net_prio:/kubepods/burstable/pod234cd89f-836a-455c-8389-fee693441908/65fd03ab27d90e42bb386fce84e52834032a83396d627ba3d664e8d4b5086076
3:memory:/kubepods/burstable/pod234cd89f-836a-455c-8389-fee693441908/65fd03ab27d90e42bb386fce84e52834032a83396d627ba3d664e8d4b5086076
2:hugetlb:/kubepods/burstable/pod234cd89f-836a-455c-8389-fee693441908/65fd03ab27d90e42bb386fce84e52834032a83396d627ba3d664e8d4b5086076
1:name=systemd:/kubepods/burstable/pod234cd89f-836a-455c-8389-fee693441908/65fd03ab27d90e42bb386fce84e52834032a83396d627ba3d664e8d4b5086076
0::/kubepods/burstable/pod234cd89f-836a-455c-8389-fee693441908/65fd03ab27d90e42bb386fce84e52834032a83396d627ba3d664e8d4b5086076

[Smolsada]

This version?

root@dragonfly-staging-849865c647-2mvrn:/sys/fs/cgroup/blkio# cat /etc/os-release;uname -r
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
5.4.0-124-generic

Also, here is the cgroup versions

root@dragonfly-staging-849865c647-2mvrn:/sys/fs/cgroup/blkio# grep cgroup /proc/filesystems
nodev   cgroup
nodev   cgroup2

root@dragonfly-staging-849865c647-2mvrn:/data# stat -fc %T /sys/fs/cgroup/
tmpfs

And /sys/fs/cgroup content

root@dragonfly-staging-849865c647-2mvrn:/sys/fs/cgroup# ls /sys/fs/cgroup
blkio  cpu,cpuacct  cpuset   freezer  memory   net_cls,net_prio  perf_event  rdma
cpu    cpuacct      devices  hugetlb  net_cls  net_prio          pids        systemd

[Smolsada]

You are right about CGroup v1.

[romange]

how do you see it?

also, I am surprised that k8s uses v1. is it intentional?

[talbii]

Yup, upgrading to Linux kernel 5.8 (or newer) would solve the issue for now.

Maybe on another day Dragonfly will support v1 detection as well, but for now upgrading will work.

Edit: Dragonfly now supports cgroup v1 as well. This was pushed in PR #1329, and should be available within v1.4. 😄

[Smolsada]

how do you see it?

I just did what it says here and saw tmpfs.

is it intentional?

Not sure. I'll try to ask my colleagues.

[Smolsada]

is it intentional?

We get such nodes from a cloud provider. We cannot influence it. 🤷‍♂️

[ashotland]

What provider are you using?
I think I encountered a similar issue with GCP Autopilot.
Upgrading the k8s version resulted with an a newer kernel.

[Smolsada]

I don't think the specific provider is important. We use multiple clouds. In my particular case, we don't want to upgrade k8s and version of Linux kernel does not depend on this.

But your answer can help to other people!

[romange]

@Letfdsada you can also manually override resource reservation by passing --proactor_threads= and --maxmemory= flags.
For example, --proactor_threads=1 and --maxmemory=400M

[Smolsada]

Yeah, so as not to be blocked, I will do so.
I also want to know how to set these parameters correctly, since the limits are not guaranteed.
So will it be right?

start parameters:
proactor_threads == limits.cpu ( == 1 in my case )
maxmemory == requests.memory == limits.memory (== 512M in my case, as i don't want to see ooms from non-guaranteed space)

[romange]

I am not familiar with k8s semantics of requests.memory and limits.memory.
It's worth setting maxmemory to a bit less than your machine capacity.