[question] Is possible to use Docker Server with TT enabled in kernel?

[ionutnechita]

Hi Hamad Al Marri,

Is it possible to have the TT in kernel and the docker server working properly?
Is there a problem with CGROUPs?

// level=warning msg="Unable to find cpu cgroup in mounts"

A good day.
Ionut.

[ptr1337]

@ionutnechita
Should work, but i dont think you wont profit much on it. Since the scheduler is mostly thought to provide a low latency. But you ca try it.

[ionutnechita]

I tried today, but the docker containers don't start.

[ionutnechita]

Maybe tomorrow I'll give you more details.

[ptr1337]

@ionutnechita

Do you have autogroup disabled ?

[ionutnechita]

I use this options:
CONFIG_TT_SCHED=y
CONFIG_TT_ACCOUNTING_STATS=y

And AUTOGROUP is disabled.

Patch set:
tt-5.15.patch - 0.3.7
high-hz.patch

[ptr1337]

@ionutnechita

Sorry, i was wrong. It does not work, since autogroup is disabled at default.

[hamadmarri]

hi @ionutnechita

could you please upload the .config

docker sometimes works without autogroup and sched_group

[ionutnechita]

I'm sending you 2 .config, one without TT and one with TT.

1. Without TT:
   config-5.15.9-lowlatency-non-tt.txt

2. With TT:
   config-5.15.9-lowlatency-with-tt.txt

My scenario:

1. Docker Server is installed on my host.
2. Use Kind tool for create new cluster. Docker Server is backend for new cluster.

./kind create cluster --config test.yml -v 1
Without TT, working correctly:
Creating cluster "kind" ...
DEBUG: docker/images.go:58] Image: kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6 present locally
✓ Ensuring node image (kindest/node:v1.21.1) 🖼
✓ Preparing nodes 📦 📦 📦 📦 📦 📦
✓ Configuring the external load balancer ⚖️
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
✓ Joining more control-plane nodes 🎮
✓ Joining worker nodes 🚜
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a nice day! 👋

./kind create cluster --config test.yml -v 1
With TT, not working correctly:
Creating cluster "kind" ...
DEBUG: docker/images.go:58] Image: kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6 present locally
✓ Ensuring node image (kindest/node:v1.21.1) 🖼
✓ Preparing nodes 📦 📦 📦 📦 📦 📦
✓ Configuring the external load balancer ⚖️
✗ Writing configuration 📜
ERROR: failed to create cluster: failed to generate kubeadm config content: failed to get kubernetes version from node: failed to get file: command "docker exec --privileged kind-worker3 cat /kind/version" failed with error: exit status 1
Command Output: Error response from daemon: Container b411eec646c160287d2c770eefa0fffc3498d30ecb9debdd838b4966c309094f is not running
Stack Trace:
sigs.k8s.io/kind/pkg/errors.WithStack
sigs.k8s.io/kind/pkg/errors/errors.go:59
sigs.k8s.io/kind/pkg/exec.(*LocalCmd).Run
sigs.k8s.io/kind/pkg/exec/local.go:124
sigs.k8s.io/kind/pkg/cluster/internal/providers/docker.(*nodeCmd).Run
sigs.k8s.io/kind/pkg/cluster/internal/providers/docker/node.go:146
sigs.k8s.io/kind/pkg/exec.OutputLines
sigs.k8s.io/kind/pkg/exec/helpers.go:81
sigs.k8s.io/kind/pkg/cluster/nodeutils.KubeVersion
sigs.k8s.io/kind/pkg/cluster/nodeutils/util.go:35
sigs.k8s.io/kind/pkg/cluster/internal/create/actions/config.getKubeadmConfig
sigs.k8s.io/kind/pkg/cluster/internal/create/actions/config/config.go:208
sigs.k8s.io/kind/pkg/cluster/internal/create/actions/config.(*Action).Execute.func1.1
sigs.k8s.io/kind/pkg/cluster/internal/create/actions/config/config.go:90
sigs.k8s.io/kind/pkg/errors.UntilErrorConcurrent.func1
sigs.k8s.io/kind/pkg/errors/concurrent.go:30
runtime.goexit
runtime/asm_amd64.s:1371

A good day.
Ionut.

[ptr1337]

@ionutnechita

There was also some time ago a cfs release, maybe that is a try worth.
Also you could use hamad's old scheduler, the cacule scheduler.

I also run that on my buildserver, the RDB version at it runs great.

Here you find the patches for 5.15 and 5.16.

https://github.com/CachyOS/cacule-cpu-scheduler

[hamadmarri]

Hi @ionutnechita

Could you please try with
UCLAMP_TASK enabled and also with HZ=300 or 1000

[ionutnechita]

Hi @hamadmarri

I try with UCLAMP_TASK enabled and also with HZ=300.
But not working.

config-5.15.9-lowlatency-with-tt-uclamp.txt

And i notice this in journalctl:
journalctl -u docker
...
level=warning msg="Unable to find cpu cgroup in mounts"

[ionutnechita]

I will test the next few days.
Thank you for help.

Happy New Year.

[hamadmarri]

@ionutnechita

You might also try the BORE scheduler https://github.com/firelzrd/bore-scheduler

[ionutnechita]

Hi @hamadmarri

I tried with BORE scheduler and working fine.
I think in the future I will use BORE or Cacule scheduler.

I don't think TT design supports that full docker support.
But this TT scheduler is very powerful.

Thanks for support.

A good day.