Centralised data logging to database

[Semptum]

Hi,
We’re currently setting up a new data‐acquisition system in our lab and have decided to use LECO for instrument coordination. Rather than using LECO’s built-in DataLogger, I’d like to take a different approach, since it offers additional flexibility. I’d appreciate advice on how best to integrate this approach with LECO.

I set up a database (postgresql/timescaledb) to be able to log events in the lab. It's set up to be able to store:

1. lab events (motor moved, pressure polled, shutter opened, detector saved image)
2. data points (timestamps when a set of detectors was triggered, plus a JSONB field describing the experimental parameters for that point, along with relational metadata linking it to other data in sequences)
3. A bunch of other tables, some for live monitoring using Grafana, some more traditional relational tables to structure everything... In total I have more than a dozen tables and the whole structure is out of the scope of the question. Suffice it to say, it should provide a structure quite well suited to the kind of data acquisition we run.

Just to be clear, we would not be sending any binary data from the detectors over LECO. Instead, whenever a detector finishes saving its data, it should send a “file‐saved” message (including the file path and associated metadata, which it received over LECO earlier) to the database. Similarly, whenever a motion starts or finishes, the actuator should send that event to the database

I was wondering how best to proceed with actually tying this into LECO, preferably without distributing the task of database writing to every actor.
The way I see it, It makes little sense to use the control protocol coordinators since they are not centralised. So my current plan is:

1. Set up a proxy_server
2. Connect any device that wants to take part in the logging to the proxy_server
3. Whenever an actuator starts or ends a motion, it sends a LECO packet (describing the event) to the same proxy/server.
4. Whenever a detector finishes saving a file, it sends a LECO packet (containing file info + metadata) to this proxy/server.
5. A single “logger” process on the proxy/server reads those packets and writes to PostgreSQL

I'm still new to LECO so I might have mixed up a few concepts (for instance, I'm not sure if you can send a message to a proxy_server or if you have to set up a Listener connected to it and have that log those messages).

My questions are:

1. Is this a reasonable way to centralize event‐logging in LECO?
2. If not, how would you recommend sending every event (file saved, motion started/ended, etc.) to a single computer for logging? I prefer to centralize it, to avoid having the logging code in a bunch of different places especially since it would contain a password and username to the database (though without DROP privileges)

I know it's a fairly vague question, if anyone has some advice on how to integrate a centralized logger with LECO (and on whether “proxy_server + listener” is the right pattern) would be greatly appreciated.

Thanks in advance for any pointers!

[BenediktBurger]

Hi @Semptum ,

thanks for your question.

That is a typical case for the Data Protocol:

• whenever you want to log an event, you use the DataPublisher to publish that information. (The Actor already contains such a publisher).
• You have a single proxy_server running (available as a command) to which every data source publishes (default config is to localhost). Every Subscriber will connect to that as well.
• You have a single Listener (which you can base on the existing Listener, which subscribed to "" (that is to everything). This listener takes all the events and logs them to the appropriate table of the database. This one is the only one writing to the database.

You have to think about the content of the DataMessages, what you want to send and how to interpret it at the Logging instance.

The nice thing about this approach is, that you can add additional listeners, which could listen only to a part or to show information directly etc.

[Semptum]

Hi @BenediktBurger ,
Thank you for the information, I will try it with the DataPublisher in the coming days..
Best regards,
Andre

[BenediktBurger]

If you have questions, feel free to ask.

One thing I forgot: You may transmit the binary data via LECO, which is no problem at all instead of saving to files which then get included into a database anyways.
LECO protocol allows binary payloads. For PyMoDAQ I transferred camera images at maximum fps from one program to another one.

[Semptum]

Thanks!
I've actually encountered some difficulty trying to implement it yesterday. I'm quite sure I misunderstood how Actors should be implemented. Below is an example of a device class and a thin actor wrapper around it.

from pyleco.actors.actor import Actor
from pyleco.utils.data_publisher import DataPublisher

class Shooter:
    """ Device class """
    def __init__(self):
        pass

    def shoot(self):
        print("Device is shooting!")
        return "success"

class ShootActor(Actor):
    def __init__(self, name: str, **kwargs):
        super().__init__(name=name, device_class=Shooter,  **kwargs)
        self.publisher = DataPublisher(full_name="SN2.Shooter", host="192.168.1.238", port=11100)
        self.connect()
        self.register_rpc_method(self.shoot)
        self.listen()
    
    def shoot(self):
        print("Actor is shooting!")
        result = self.device.shoot()
        data={"shot":result}
        self.publisher.send_data(data)
        return data
    
if __name__ == "__main__":
    actor = ShootActor(name="Shooter", host="192.168.1.238", port=12300, namespace="SN2")

I can connect a director to it:

from pyleco.directors.director import Director

class ShootDirector(Director):
    def __init__(self, actor: str, name:str, **kwargs):
        super().__init__(actor=actor, name=name, **kwargs)

    def shoot(self):
        response_id = self.call_action(action="shoot")
        return response_id

if __name__ == "__main__":
    director = ShootDirector(actor="Shooter", name="ShooterDirector", host="192.168.1.238", port=12300, namespace="SN2")
    director.shoot()

1. I expected that shooting via the director would first go into the shoot method of the Actor (which calls the shoot method of the device). Instead, the shoot method of the device is called directly.
   I saw that in pyleco-extras, an actor is implemented directly from a MesageHandler rather than Actor. Is that the appropriate way to have some additional functionality such as sending data?

2. For sending binary data, we're actually considering using it as well. But I still intend to save all the metadata into the database. It would be very useful since it allows replaying the events in the lab even years later. Still, my colleague has attempted sending camera data to a listener from pymodaq (copying the code found in pyleco-extras). I didn't look at the code yet, I'm not sure what it uses to serialise numpy arrays. But according to my colleague, the performance drop from sending the frame data is significant.

[BenediktBurger]

Thanks for your examples, they show me some deficiencies in the explanation. I'd be glad, if you helped me improve the documentation. GETTING_STARTED.md should be the starting point.

The idea of the Actor class is to wrap any instrument class, such that you do not have to write your own actor again and the Director offers three methods to set device properties, get properties, and to call device methods:

actor = Actor(name, device_class=Shooter)
...
director = Director(actor=name)
director.set_parameters({'property1: 5'})
director.get_parameters(["property1"]) == {"property1: 5})
director.call_action("shoot") == "success"

It is tedious to always write the property name in that strange way, therefore I wrote two directors:

• TransparentDirector (here in pyleco) allows to use director properties as if they were the device properties: transparentDirector.device.property1 will read the property1 of the device instantiated by the actor or will set it (if ...property1 = 5). Methods do not work automatically!
• AnalyzingDirector (here in pyleco-extras) will analyze a given driver file and its device property will have that type (allowing an IDE to offer autocompletion and type checking!) and even works with channels, if they are of pymeasure's Channel class. This works with methods and properties. For example director.device.shoot() will return "success".

Due to these directors/actors, I did not need to write several actors for the many devices.

self.publisher = DataPublisher(full_name="SN2.Shooter", host="192.168.1.238", port=11100)

That line is superfluous, as the Actor has already a DataPublisher set up.

I expected that shooting via the director would first go into the shoot method of the Actor (which calls the shoot method of the device). Instead, the shoot method of the device is called directly.

If you call director.call_action("shoot"), the actor will execute self.device.shoot(), regardless, whether it had been registered as rpc method or not.

Outside these special Actor/Director pairs, you have to create a method on the actor, add it to the available methods via register_rpc_method such that the director can call ask_rpc(method="method_name"). This is the plain LECO definition according to JSON-RPC.
In fact, the three special director method (get/set_parameters and call_action) call corresponding methods in the Actor, which resolve the name to some device property/method.
Therefore, your calling did not work as expected.

I saw that in pyleco-extras, an actor is implemented directly from a MesageHandler rather than Actor. Is that the appropriate way to have some additional functionality such as sending data?

The Actor is specialized to wrap around a device definition class (e.g. an instrument). If you want to add some functionality, which cannot described well in that way (as your service is self-contained and does not have some other class), the MessageHandler would be suited better.

That is the reason I used the MessageHandler and not the Actor for some of actors in pyleco-extras.

By the way, the Actor is a subclass of the MessageHandler with the addition of the Publisher and with the three special methods (get/set Parameters and call action).

For sending binary data, we're actually considering using it as well. But I still intend to save all the metadata into the database. It would be very useful since it allows replaying the events in the lab even years later.

You could also send the metadata via the message.

Still, my colleague has attempted sending camera data to a listener from pymodaq (copying the code found in pyleco-extras). I didn't look at the code yet, I'm not sure what it uses to serialise numpy arrays. But according to my colleague, the performance drop from sending the frame data is significant.

I did not use Camera images via LECO in my setup (no need to), so you have to investigate it. It was just a hint at the possibilites. However, serializing the data in order to write it and later to read it, should be the same as to send it.

[BenediktBurger]

Did you read my description of my setup here: #46 ?

Here are the different Actor usages (for the Starter, which starts them in their own thread): https://git.rwth-aachen.de/nloqo/thg/thg-scripts/-/tree/master/thg-tasks?ref_type=heads . They are mostly an adaptation of the pymeasure_actor in the examples folder.