Investigate sudden disconnects #8
Description
Sometimes we see disconnects in the C++ Agents, especially in transformers. What I've learned so far from investigating this issue:
Conditions:
- It always happens on the data connection
- It only happens with a good chunk of throughput
- The node setup doesn't seem to influence the setup. I had source connections to both nodes in the occurrence below.
Procedure:
- It seems to happen, whenever there are no more data messages coming from the rabbitmq
- The data connection switches over to receiving and sending heartbeats
- After a few minutes, the rabbitmq complains about missing heartbeats and closes the connection
- The client receives an EndOfFile on the data connection socket and throws exceptions until it dies.