Quality of datatransfer

[atobiese]

Hi,
Thanks for a great software implementation.

I am testing running several clients working against a common UA-server.
The client is set up such that all available nodes on the server are subscribed to.
Data is sent back using the ua_node.set_value(val) method.
I am running about 500 tags.

My big question is how we can assure that data is transferred to server? I have tried to understand how this is implemented in the protocol but it is still not clear.
https://reference.opcfoundation.org/v104/Core/docs/Part4/5.13.1/
Using mqtt, for example, there is a QoS (quality of service) system, that gives a feedback handshake to confirm delivery.
I am loosing some data as packets are lost. If a packet gets lost, the sender should be responsible to retransmit the message within a reasonable amount of time.
Is there an example of a common way of ensuring this delivery?

Thanks again for the work on this project!
regards

[AndreasHeine]

OPC UA Protocol:
opc ua uses request / response mechanisems
e.g. Write a Node value:
https://reference.opcfoundation.org/v104/Core/docs/Part4/5.10.4/
so for each request you will get a response with a statuscode

you are referencing the opc ua subscription in the docs -> a opc ua subscription does not work the same way like a mqtt subscription work!

TCP Transport:
opc ua binary uses tcp so a opc ua request will be send in chunks (according to the maxchunksize and maxmessagesize parameter) and reconstructed on the other side
https://de.wikipedia.org/wiki/Transmission_Control_Protocol#/media/Datei:Tcp_transfer.png

[atobiese]

Hi Andreas and Oroulet, thanks basically immediate response, and taking the time. I am aware of the fundamental difference between the protocols in that respect, although I am not well versed in the OPC-protocol. I wanted to check if there were logics on a higher level that ensures delivery if socket falls out. I have a wrapper that restarts the client if connection is broken. It simply listens to both and restarts. If data is sent within the timeframe of socket-loss, the data is gone, but a failed response will be delivered over the http request application protocol. Any ideas of how I can implement such a "checker"? One other thing, I am using the legacy version w/o asyncio, but should probably move over asap.

[oroulet]

What are you afraid off? in case of a write, the write mght work or fail even if you do not get any ack in the unlickely even the networks breaks exactly inbetween.
In case of DataChangeEvents, if you lose a connection you will loose all event until you reconnect. In theory you could use a persistent session, but this is not implemented. What you can do is use history and fetch all event, since the last even you have. In fact I implemented such a solution 10 years ago so it works

[atobiese]

Yes, I suppose a persistent system or a short cache would be required to resend if delivery fails due to socket disconnect. I agree that this is not something to worry too much about generally. The problems seems to be due to my poor implementation on the notification side when data is coming in. I am aware that you warn of running heavy jobs at the data_notification level, such as network stuff.
I want to forward data as they come in, with possibility also to forward to a different server. To be efficient and avoid setting attributes to objects etc as data comes in, I therefore save the nodes locally as a named tuple.
tag = self.obj.identity[_node.nodeid.Identifier].tag \ if _node.nodeid.NamespaceIndex == self.obj.identity[ _node.nodeid.Identifier].ns else _LOG.debug(f"error in api call to local UA object")

This "identity" id is stored when the client connects first time to server and resolves all nodes.
_identity[id] = Identity(tag, ns)

Although most things are done only once before connection, the default timeout of less than 5 ms may be violated. It too some time for me to realise this, as i could not recover the exception. This forces a socket protocol error and the auto-reconnect takes some time. The issue I had doing operations running on a separate thread is that one then has to cycle the given updates of the whole list, as far as I can see. I have therefore not found a better way of doing it and felt like the separate thread running independently is more heavy (although less on the response side).
So I increased the default response timeout and things are more stable.
Is this a doable implementation or is there a better way?
Thanks again for the great work.

[zerox1212]

I think you need to explain your use case better... Are you saying that you have clients subscribed to a common server, and when a client receives a data_change_notification it immediately writes back using node.set_val? What is the point of something like that?

[atobiese]

Yes, but of course only in principle:) There are numerous possible use cases that may require network handling upon a callback.
One case could be to run some operation on the data coming in, a numercial filter, or a model that runs in parallel with the sensor the client is subscribed to, then return the result to a different node or forward in some other way. This requires than the event is fired upon the incoming notification. I understand one cannot just do a lot of time consuming operations during the data_change_notification as this has a set timeout given by the loads of incoming data, and that this really must be done in an independent operation. I was looking at ways of doing this, or if there is a common way of doing this (without having to set up a separate bus and worker, which again requires separate cycling etc). Thanks again