Failure to set a real time scheduler in CTDB when CephFS backend is used

[xhernandez]

When a SIT Environment is created using the CephFS backend, sometimes the CTDB service fails to start, logging this error:

Unable to set scheduler to SCHED_FIFO (Operation not permitted)

This seems related to BZ 1201952. The two workarounds proposed there work:

1. Increase the value of /sys/fs/cgroup/cpu/system.slice/cpu.rt_runtime_us.
2. Add the option realtime scheduling = false into the legacy section of ctdb.conf.

For now SIT Environment uses the second approach.

Upon further investigation, it seems that the system.slice subdirectory doesn't exist for other backends, but on CephFS, when cephadm is boostrapping the cluster, it appears (among others). Most probably this is related to container and/or systemd activity. This particular subdirectory seems to disappear automatically after some time, but sometimes it's already too late and CTDB has already attempted to start and failed.

Some additional related information provided by @phlogistonjohn (Thanks !!!):

• RFE: Allow cpu.rt_runtime_us to be set systemd/systemd#329
• https://bugzilla.redhat.com/show_bug.cgi?id=1229700
• https://bugs.freedesktop.org/show_bug.cgi?id=87570