check-ncm-cdispd does not restart the services if it crashes

[aka7]

Since the move to systemd unit file, if ncm-dispd process dies check-ncm-cdispd script does not start it. If proccess dies, cron logs shows this.

Redirecting to /bin/systemctl status ncm-cdispd.service
Daemon ncm-cdispd has been intentionally stopped. Not restarted.

even though it was NOT stopped intentionally.

reason

# systemctl status ncm-cdispd
* ncm-cdispd.service - Configuration Dispatch Daemon waits for new incoming configuration profiles
   Loaded: loaded (/usr/lib/systemd/system/ncm-cdispd.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2025-06-22 02:10:53 EDT; 4 weeks 1 days ago
 Main PID: 940 (ncm-cdispd)
   CGroup: /system.slice/ncm-cdispd.service
           `-940 /usr/bin/perl -Tw /usr/sbin/ncm-cdispd
........

# echo $?
0
# kill -9 940
# systemctl status ncm-cdispd
* ncm-cdispd.service - Configuration Dispatch Daemon waits for new incoming configuration profiles
   Loaded: loaded (/usr/lib/systemd/system/ncm-cdispd.service; enabled; vendor preset: enabled)
   Active: failed (Result: signal) since Mon 2025-07-21 14:39:38 EDT; 13s ago
  Process: 940 ExecStart=/usr/sbin/ncm-cdispd (code=killed, signal=KILL)
 Main PID: 940 (code=killed, signal=KILL)
# echo $?
3

status code from systemclt is 3 regardless of if process crashes or is stopped. Therefore, it always hits this, https://github.com/quattor/ncm-cdispd/blob/main/src/main/scripts/check-ncm-cdispd#L23

since its now systemd unit file, I think fix here is to add restart=on-failure in unit file. https://github.com/quattor/ncm-cdispd/blob/main/src/main/resources/systemd/ncm-cdispd.service

look to perhaps add

[Service]
Restart=on-failure
RestartSec=5

and git rid of check-ncm-cdispd and the cron ? thoughts?

I added this through template dropin file, and seems to work during my tests.

[jrha]

Yeah, that looks sensible to me.
We should probably work out how to fix check-ncm-cdispd though, as it's supposed to be able to be used as a monitoring test.

[jrha]

Indeed systemctl doesn't have much richness in exit code meaning:

Value	Use in systemd
0	unit is active
1	unit not failed (used by is-failed)
2	unused
3	unit is not active
4	no such unit

https://www.freedesktop.org/software/systemd/man/latest/systemctl.html#Exit%20status

But treating 3 as "intentionally stopped" is definitely wrong.

[jrha]

On second thoughts, maybe we don't need a check wrapper script, any decent monitoring system should have native systemctl support anyway.

[aka7]

for time being we have added a drop-in file to restart=on-failure for time being, but do we know the reason why it sometimes dies? fact that there is cron to start suggests this has been a common issue of the daemon crashing?