/var/run/ceph-radosgw/ disappears on reboot

[gravitystorm]

0845115 introduced the /var/run/ceph-radosgw directory. Unfortunately that's not created in any of the init scripts, and disappears on reboot (Ubuntu 14.04 and likely others).

This means that on a reboot, the radosgw daemon won't work until the next chef run occurs, and even then not until radosgw is manually restarted.

[gravitystorm]

For what it's worth, here's my workaround. I have a wrapper cookbook, and within it I do the following:

template '/etc/init/radosgw-run-dir.conf' do
  variables(user: node['apache']['user'], group: node['apache']['group'])
end

templates/default/radosgw-run-dir.conf.erb:

# Managed by chef

start on starting radosgw

task

script
  if [ ! -d /var/run/ceph-radosgw ]; then
    mkdir -p /var/run/ceph-radosgw
    chown <%= @user %>:<%= @group %> /var/run/ceph-radosgw
    chmod 0755 /var/run/ceph-radosgw
  fi
end script

I'm no upstart expert, but it works for me. It basically jumps in just before any radosgw daemon starts, and checks if the run directory exists.

I've come to the conclusion that it's a bad idea to change the run directory away from what ceph expects. Within the packaged upstart files there are checks to ensure /var/run/ceph exists, but by changing dir in the chef recipe we miss out on these checks and it has the potential for trouble, like here. Do the other deployments like via ainsible or ceph-deploy also change the directory? Isn't it better to stick with the way that upstream expect it to work?

[guilhem]

It seems to be a problem with packaging (upstart scripts come from there).
I don't want to fix problem from another project (here ceph itself) in a cookbook (it add too much complexity).

But your wrapper seems to be perfect for a quick fix :)

[guilhem]

Oh no... I begin to understand...
I will try to find a solution

[guilhem]

Woo, problem is bigger than expected: http://tracker.ceph.com/issues/11453

[ktdreyer]

As background on that ticket "run RGW as root" ticket, see also http://tracker.ceph.com/issues/9001 . The long-term plan for RGW and everything else is to simply run as a "ceph" UID that will have rights the appropriate locations on disk.

By the way, for systemd we create /var/run/ceph with https://github.com/ceph/ceph/blob/master/systemd/ceph.tmpfiles.d . It looks like we need to handle this within the upstart configuration files, too?

Please note, if you guys are having to hack around packaging bugs, it would be great to have Redmine tickets filed upstream (and assign them to me :)