Apache not started after reboot

[e21347]

Cookbook version

0.4.5

Chef-client version

12.21.1

Platform Details

Ubuntu 16.04

Scenario:

Getting apache to start after reboot without having to run chef-client again.

Steps to Reproduce:

# metadata.rb
...
depends 'apt', '~> 6.1.0'
depends 'httpd', '~> 0.4.5'

# recipes/default.rb

include_recipe 'apt::default'

httpd_service 'default' do
  action [:create, :start]
end

httpd_config 'default' do
  source 'mysite.cnf.erb'
  notifies :restart, 'httpd_service[default]'
  action :create
end

# templates/default/mysite.cnf.erb

<VirtualHost *:80>
  DocumentRoot /var/www/html

  ErrorLog /var/log/apache2-default/error.log
  LogFormat "%v:%p %a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
  CustomLog /var/log/apache2-default/access.log vhost_combined
</VirtualHost>

kitchen converge
kitchen login

(all is good at this point)

vagrant@default-ubuntu-1604:~$ sudo reboot

Expected Result:

Apache started after reboot.

Actual Result:

Apache cannot be started after reboot. /var/lock/apache2-default is missing.

vagrant@default-ubuntu-1604:~$ systemctl status apache2-default.service
● apache2-default.service - The Apache HTTP Server
   Loaded: loaded (/etc/systemd/system/apache2-default.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2017-06-22 05:51:04 UTC; 10min ago
  Process: 1208 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
  Process: 1144 ExecStart=/usr/sbin/apache2 -f /etc/apache2-default/apache2.conf -DFOREGROUND (code=exited, status=1/FAILURE)
 Main PID: 1144 (code=exited, status=1/FAILURE)

Jun 22 05:51:04 default-ubuntu-1604 systemd[1]: Started The Apache HTTP Server.
Jun 22 05:51:04 default-ubuntu-1604 apache2[1144]: AH00526: Syntax error on line 6 of /etc/apache2-default/apache2.conf:
Jun 22 05:51:04 default-ubuntu-1604 apache2[1144]: Invalid Mutex directory in argument file:/var/lock/apache2-default
Jun 22 05:51:04 default-ubuntu-1604 systemd[1]: apache2-default.service: Main process exited, code=exited, status=1/FAILURE
Jun 22 05:51:04 default-ubuntu-1604 kill[1208]: kill: failed to parse argument: '': No such file or directory
Jun 22 05:51:04 default-ubuntu-1604 systemd[1]: apache2-default.service: Control process exited, code=exited status=1
Jun 22 05:51:04 default-ubuntu-1604 systemd[1]: apache2-default.service: Unit entered failed state.
Jun 22 05:51:04 default-ubuntu-1604 systemd[1]: apache2-default.service: Failed with result 'exit-code'.

# /etc/apache2-default/apache2.conf
ServerName default-ubuntu-1604
ServerRoot /etc/apache2-default
PidFile /var/run/apache2/apache2-default.pid
User www-data
Group www-data
Mutex file:/var/lock/apache2-default default
Timeout 400
ErrorLog /var/log/apache2-default/error_log
LogLevel warn
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5
DefaultType None
HostnameLookups off

Listen 0.0.0.0:80

IncludeOptional mods-enabled/*.load
IncludeOptional conf-enabled/*.conf
IncludeOptional mods-enabled/*.conf
IncludeOptional sites-enabled/*.conf

[iennae]

Thanks for reporting. We'll look into a fix for this.

[zit-hb]

As a side note, the same is true for the PidFile on Ubuntu 16. Thanks for the work!

Jun 22 05:51:04 default-ubuntu-1604 kill[1208]: kill: failed to parse argument: '': No such file or directory

[lieblinger]

It's just the same here for me on debian jessie. The /run/lock/apache2-default directory is not there and will not be created on a reboot.

A quick fix for me was to edit the helpers_debian.rb to:
def mutex
return nil if apache_version.to_f < 2.4
"file:/var/lock/apache2 default"
end`

httpd_service_debian_systemd.rb:
template "/etc/systemd/system/#{apache_name}.service" do
source 'systemd/httpd.service.erb'
owner 'root'
group 'root'
mode '0644'
cookbook 'httpd'
variables(
binary_name: 'apache2ctl',
apache_name: apache_name,
config_relative_path: config_file_relative_path
)
action :create
end

And it is working. But I am not sure if this will work with multiple instances.

Maybe we have to provide an init script which is being called from the systemd for each instance?

[JoshuaCWebDeveloper]

I'd just like to chime in and say I have observed both of the above issues on Ubuntu 16.04:

• The /run/lock/apache2-default directory is not created.
• The apache2.conf file looks for the pid file at /run/apache2/apache2-default.pid (The directory /run/apache2-default is created, the conf file is just looking for the wrong directory.)

I tried running the same recipe on Ubuntu 14.04 and did not encounter either of the above issues.

[nathwill]

confirming this is still an issue on Ubuntu 16.04; i was able to force a functional config by monkey patching the mutex method in the helper (though that's obviously not ideal):

wyrd:student-docker-images nathwill$ cat site-cookbooks/student-images/libraries/helpers_debian.rb 
module HttpdCookbook
  module Helpers
    module Debian
      def mutex
        return nil if apache_version.to_f < 2.4
        "file:/var/lock/apache2 default"
      end
    end
  end
end

strangely, only "apache2" worked as an argument, i tried also "apache2_default" & "apache2default", neither of which apache accepted... 🤷‍♂️

[JoshuaCWebDeveloper]

I didn't feel like patching the cookbook, so I decided to instead overwrite the default systemd .service script with a custom one that creates the directories the cookbook is currently looking for.

Template apace2.service.erb:

[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=simple
Environment=LANG=C

ExecStartPre=/bin/bash -c "mkdir -p /run/apache2  && mkdir -p /run/lock/apache2-default"
ExecStart=/usr/sbin/apache2 -f /etc/apache2-default/apache2.conf -DFOREGROUND
ExecReload=/usr/sbin/apache2 -f /etc/apache2-default/apache2.conf -k graceful
ExecStop=/bin/kill -WINCH ${MAINPID}

KillSignal=SIGCONT
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Then in my recipe, right after the httpd_service 'default' resource:

# Override default apache2-default.service file with custom one that creates directories
# (Workaround for: https://github.com/chef-cookbooks/httpd/issues/116)
template "/etc/systemd/system/apache2-default.service" do
  source "apache2.service.erb"
  mode '0775'
  owner "root"
  group "root"
end

execute "systemctl daemon-reload"

When you run the recipe, chef will output the following diff on the .service file::

@@ -6,6 +6,7 @@
 Type=simple
 Environment=LANG=C

+ExecStartPre=/bin/bash -c "mkdir -p /run/apache2 && mkdir -p /run/lock/apache2-default"
 ExecStart=/usr/sbin/apache2 -f /etc/apache2-default/apache2.conf -DFOREGROUND
 ExecReload=/usr/sbin/apache2 -f /etc/apache2-default/apache2.conf -k graceful
 ExecStop=/bin/kill -WINCH ${MAINPID}

This method should work for multiple instances, provided that you create a separate template resource for each httpd_service resource.

Edit: Updated recipe block to include daemon-reload command.