newport-conex-agp: troubleshooting scan freezes

[ddkohler]

I am in the process of understanding an issue where client calls freeze when talking to the conex-agp daemons. Symptoms are:

• clients are frozen when a message is sent to the daemon
• newport controller box keeps status light green (referenced, not moving)
• client freeze often happens when the daemon is not actively moving (only get_position/get_units calls)
• problem is not resolved by restarting daemons; power cycling the agp controller and restarting daemons does resolve the issue.

[ddkohler]

To diagnose, we have added logging to the serial commands to watch and timestamp messages being sent/recieved. Here is one record of calls right before the daemon became unresponsive:

<7> DEBUG : 2023-03-09T14:51:46-0600 : COM13 : writing b'1TP\r\n'
<7> DEBUG : 2023-03-09T14:51:46-0600 : COM13 : writing b'1TE\r\n'
<7> DEBUG : 2023-03-09T14:51:46-0600 : COM13 : writing b'1TS\r\n'
<7> DEBUG : 2023-03-09T14:51:46-0600 : COM13 : writing b'1TE\r\n'
<7> DEBUG : 2023-03-09T14:51:46-0600 : COM13 : readline: b'1TP124.5003'
<7> DEBUG : 2023-03-09T14:51:46-0600 : COM13 : readline: b'1TE@'
<7> DEBUG : 2023-03-09T14:51:46-0600 : COM13 : readline: b'1TS000033'
<7> DEBUG : 2023-03-09T14:51:46-0600 : COM13 : readline: b'1TE@'
<7> DEBUG : 2023-03-09T14:51:46-0600 : COM14 : readline: b'1TP132.0002'
<7> DEBUG : 2023-03-09T14:51:46-0600 : COM14 : readline: b'1TE@'
<7> DEBUG : 2023-03-09T14:51:46-0600 : COM14 : readline: b'1TS000033'
<7> DEBUG : 2023-03-09T14:51:46-0600 : COM14 : readline: b'1TE@'
<7> DEBUG : 2023-03-09T14:51:47-0600 : COM13 : writing b'1TP\r\n'
<7> DEBUG : 2023-03-09T14:51:47-0600 : COM13 : writing b'1TE\r\n'
<7> DEBUG : 2023-03-09T14:51:47-0600 : COM13 : writing b'1TS\r\n'
<7> DEBUG : 2023-03-09T14:51:47-0600 : COM13 : writing b'1TE\r\n'
<7> DEBUG : 2023-03-09T14:51:47-0600 : COM13 : readline: b'1TP124.5003'
<7> DEBUG : 2023-03-09T14:51:47-0600 : COM13 : readline: b'1TE@'
<7> DEBUG : 2023-03-09T14:51:47-0600 : COM13 : readline: b'1TS000033'
<7> DEBUG : 2023-03-09T14:51:47-0600 : COM13 : readline: b'1TE@'
<7> DEBUG : 2023-03-09T14:51:47-0600 : COM14 : writing b'1TP\r\n'
<7> DEBUG : 2023-03-09T14:51:47-0600 : COM14 : writing b'1TE\r\n'
<7> DEBUG : 2023-03-09T14:51:47-0600 : COM14 : writing b'1TS\r\n'
<7> DEBUG : 2023-03-09T14:51:47-0600 : COM14 : writing b'1TE\r\n'
<7> DEBUG : 2023-03-09T14:51:47-0600 : COM14 : readline: b'1TP132.0002'
<7> DEBUG : 2023-03-09T14:51:47-0600 : COM14 : readline: b'1TE@'
<7> DEBUG : 2023-03-09T14:51:47-0600 : COM14 : readline: b'1TS000033'
<7> DEBUG : 2023-03-09T14:51:47-0600 : COM14 : readline: b'1TE@'
<7> DEBUG : 2023-03-09T14:51:48-0600 : COM13 : writing b'1TP\r\n'
<7> DEBUG : 2023-03-09T14:52:28-0600 : COM13 : writing b'1TE\r\n'

(debug commands related to client communication have been omitted for clarity). No takeaways yet, but note that there is a ~30 sec delay between the last and 2nd to last write command.