[Bug]: JDK server upgrades cause unexpected `/bin/cat` errors to appear on agents

[gcoxmoz]

Is this a critical security issue?

• This is not a security issue.

Describe the Bug

Originally filed as puppetlabs/puppetserver#2900 on puppetserver-8.7.0. As of this week (2025-04-21)'s openjdk upgrade this happened again, against openvox-server-8.8.0, so, recreating issue here since I was on OSP and expect no love in the old world.

tl;dr:
When openJDK is upgraded, puppet-agents doing puppetserver runs begin complaining about Cannot run program "/bin/cat" until the puppetserver is full-restarted. The agent runs complete, but the error message looks scary and shows in foreman as an error'ed run.

Expected Behavior

"No unexplained errors" would be desirable. Whether that's "no error appears on the agent" or "a less dire one appears on the agent" or "puppetserver can detect an underlying change happened and deal with it or whine for a restart" or "explicit 'yodawg, gotta restart puppetserver'" docs/instructions somewhere, I leave up to the implementor.

Steps to Reproduce

• Rocky 9 box, downgrade to 'not-the-latest' java-17-openjdk-headless package, spin up openvox-server 8-latest.
• agent nodes (various Linux flavors, mostly CentOS7 through Rocky9) established and happy.
• dnf upgrade java-17-openjdk-headless on the openvox-server node, do not take any extraordinary action to restart the server daemon (it won't notice on its own).
• do agent runs against the openvox-server

Environment

• OpenVox 8.15 agents and 8.8 server
• Rocky 9 server, Rocky 8-9 agents

Additional Context

No response

Relevant log output

Agent nodes begin showing:

Notice: Catalog compiled by [puppetserver-compiler.name]
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Exception while executing '/bin/cat': Cannot run program "/bin/cat" (in directory "."): error=2, No such file or directory on node [node.name]

[gcoxmoz]

See https://bugs.launchpad.net/ubuntu/+source/openjdk-17/+bug/2055280/comments/5

[ffrank]

Agree that this is a scary and unfortunate result, but I don't think that openvox-server should try to handle this in any way.

Reacting to this circumstance would mean we add a heuristic that is created to match exactly one known past event (this specific JRE bug just hitting unsuspecting servers). In the future, this heuristic will likely never catch the same scenario again (putting some faith in the JRE maintainers here), but could make it harder to analyze new scenarios, that will be caught by the heuristic as false positives.

[nmburgan]

Generally, I think a best practice is when you are updating things on the server node that Puppet depends on, you should restart the services. This is what Puppet Enterprise does automatically, and it requires a bit of manual intervention or writing some code to do it with OSP or OpenVox. There are likely other deps that might cause problems if you upgrade them out from underneath a puppetserver process. We could possibly do something to make this process a little more seamless.

[ffrank]

Can it work automatically also in the case of unattended upgrades (of Java)? (Should it? 🙈 )