Proper connection monitoring and reconnect

[jmarshall9120]

I'm currently doing the basic "polling the server" route to test connection, but I'd like to move my connection monitoring over to the OPCUA standard. I've not noticed any successful examples of this, despite many people asking for it.

OPCUA Specification
KeepAlive Info

IDEAS
I'm sure the subscription object must hold onto a socket and am guessing the way to do this is going to be

1. Get SessionID
2. Create a subscription to some variable node.
3. Get a reference to the underlying socket.
4. Wrap it in a try catch, because disconnects will often create socket errors (saving you the trouble of checking)
5. Send a KeepAlive every t seconds.
6. Throw error if KeepAlive fails
7. Reconnect on error.
8. Pass saved SesssionID to server
9. Reestablish Session.
10. Repeat starting at step 2.

I think this should really be an example in the repo as its core to the operation of OPCUA.

If anyone has experience with this, examples or advice would be great. Either way, I've got to figure this out and I'll post my work when I'm done.

Update 1
My investigation so far:
Digging into the client session implimentation I see the connect object has this basic workflow available from inside the Client object:

So, this is doing most of what I expect from the spec, but why is not detecting the disconnects:

1. _renew_channel_task only tries to renew the connection when it gets 75% of the way to a timeout and it does this via:
   params.RequestType = ua.SecurityTokenRequestType.Renew
   So this does a KeepAlive based on timeout, but the timeout the server sets is not probably the best interval for quickly detecting connection health.
   Also, this is a future thought, but since the KeepAlive runs in an asyncio.Task and not a separate thread, it could get blocked for so long that it misses the refresh and kills the connection. Would probably be a good idea to use the asyncio.to_thread flavor of task running.
2. activate_session is tied to user credentials and not session_id, Also, create_task is called every time connect is called and we should be just activating the last session after failure, not creating a new one.
3. I'm not sure renew_channel_task will open a closed connection, more likely it errors out if the connection is closed, otherwise I don't see how we'd ever see an unexpected connection closed message.