Does agent automatically recover after a crash?

[kubamaruszczak]

I've recently discovered that some of my agents have crash logs in their meshagent.log and I wonder if they auto-heal itself after crash or do I need to restart the meshagent service? I don't really know how to reproduce crash. Here is example of crash log:

I'm asking because I'm trying to build auto-recovery solution for agents in my system. What would be the best practice to detect if agent is healthy and operational from agent perspective so I can restart it if needed? I tested that ./meshagent -state is quite good way to tell if I can communicate to agent and if it's connected to the server or not.

[si458]

I believe it should auto restart because the systemd that gets created tells it too, the only thing I can suggest is u need to add the extra values in the systemd that watch for high memory usage and if it goes say about 200mb then u have a leak so restart the service! This is critical at the moment until we can find the disconnect/reconnect memory leak problem

[kubamaruszczak]

Ok, this is quite important. I didn't know about leak memory problem. Thanks, will add that. And what about ways to detect if agent is healthy? Is there anything beside ./meshagent -state and maybe adding controlChannelDebug=1 in meshagent.msh I could do from agent perspective?

[si458]

The controlchanneldebug just simply logs connects/disconnects and when commands get sent to the remote device (u dont see the command just how big the bytes where) into a log file that's all, so can't see how that's really helpful unless u are seeing lots of disconnects/reconnects then u will experience the memory leak