Add UUID-based unit names to systemd-runs

[mdione-cloudian]

We run CI on a salt based system, which runs a few executions per day (runs are long and we're just a few developers). In some of these runs we see this error message: Failed to start transient scope unit. I dug around and found that (at least) service: reload:True is run with systemd-run. Frankly, I looked at minion logs with a level of DEBUG and I can't really find other runs using the same ID at the same time; others do, but they are run both way before and way later (in terms of tens of minutes) and I can see logs of them starting and successfully finishing. I mention this because

all units that ran and failed are kept in memory until the user explicitly resets their failure state with systemctl reset-failed or an equivalent command

Looking for this error in systemd's user mailing list I found this mail message https://lists.freedesktop.org/archives/systemd-devel/2015-October/034591.html that says:

The code for this is actually really naive... the number is just the PID of the caller, and there's no check at all to ensure it is unique. PIDs overrun easily, hence this is not nice at all...

and

In the meantime, you should be able to fix this by explicitly picking a randomized name for the scope using --unit=. For example, consider just adding --unit=uuidgen to your command line, and the clashes should not happen anymore

Luckily, salt centralizes the systemd-run executions in a single function, https://github.com/saltstack/salt/blob/master/salt/modules/systemd_service.py#L330 , so it wouldn't be that hard to slap in a --unit <UUID> option there.

[welcome[bot]]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[mdione-cloudian]

Another option would be to add a parameter unit-name that should be passed all the way down to _systemctl_cmd(), but that's way more work.

[mdione-cloudian]

We're currently working around it by setting systemd.scope: False in the minion config. We're not quite sure why it makes sense (from the master POV) that the service change works, while the minion that should report about it is dead or restarted. In either case the state will be marked as failed in the master and we will have no option but to rerun it. @terminalmage since you implemented this in e2d9e87 do you have any insights about it?

[terminalmage]

systemd-run was added (in part) to resolve the side effects of using Salt to upgrade itself. On package update, a service restart is triggered. When this happens, systemd sends a SIGTERM to all processes under the salt-minion main PID, causing the yum/apt/etc. update be SIGTERM'ed (and leave the package database in a bad/locked state).