Long-term Group Monitor using InfluxDB

[brenner-tobias]

Inspired by this project, I would love to reach a similar thing with this integration.
Unfortunately, the limit for messages in the GroupMonitor is 25.000, which I assume is due to a not optimised way of storing the messages.

My suggestion is to (optionally) store the messages in an InfluxDB (depending on the installation of e.g. this addon or an external DB) and by that enabling an almost endless storage.

The data would need to be stored after decoding it with the project file to enable proper analysis and visualisation in Grafana.

[farmio]

Hi 👋!
Imho storing data in 3rd party databases is out of scope of any HA integration. This is responsibility of the according db-integration. HA is built to enable interoperability between them.

In case of influx, there is https://www.home-assistant.io/integrations/influxdb/ which can automatically write entity state changes.
Or you can always create a simple automation leveraging knx.telegram triggering writes to influx.

The group monitor is not really meant to provide long term data. It's more of a quick and dirty debug tool.

On the other hand, optimizing the current store for memory use would probably be straightforward. Disk space wise, the data would be very compressible. It's just json holding timestamp, source and destination (address and name - both as strings) value and some metadata.
But since those 25k lines are only ~10 mb, I guess it's not that critical - given the default is only 1K lines.
Anyway, moving them to the frontend and rendering there might be a bottleneck.
If you are interested in contributing optimizations for that, just let me know.

[brenner-tobias]

Thanks a lot for your quick reply. I actually already thought about using the InfluxDB integration, the issue that I see is that we do need to enrich / decode the traffic using the KNX Project file, which is one of the things already done in the KNX frontend.
One option would be to recreate the full KNX-Structure in HA as entities, but I do not think this makes a lot of sense for logging purposes. Please correct me if there is another way using knx.telegram including the proper GA and Entity Names and write to InfluxDB.

While I agree that storing data in a 3rd party DB is generally out of scope of any HA integration, I do see a very valid use-case here which is specific to KNX, so some sort of option would not hurt.

Having thought about this further, I also think that this should be separated from the current frontend, which makes a lot of sense as you mentioned for quick and dirty debugging. Mixing this with long-term logging and the necessary frontend tools for the handling of the data would make things a lot more comfortable, while we do already have the proper tools for it in Grafana. Since this is the case, optimising the storage data does indeed not make too much sense and I do prefer simple and maintainable over unnecessary compression and decompression steps for the frontend.

So to summarise: My approach would be to add another tab in the frontend "Long-Term Monitoring" or something like that, where there is only a disclaimer to set-up the InfluxDB integration following a link and below that a toggle that will include the KNX-Data with all the other sensor data that is automatically written to influx by the integration. A separate db schema homeassistant.knx or something like that would probably make sense.

I get that this is a lot of work and there are other more broadly used / requested things and I am also fine with you not wanting to include this in the integration at all. If this is the case, I would love to hear your ideas on how to best utilise what you already have to write all the data to influx.

[farmio]

knx.telegram already provides project file data and decoded values. You can use rest_command or something else to send a HTTP request writing to the database in an automation action.

Sure, it would be nice if the influx integration would provide a custom service action for that - so you don't have to template the http request yourself, but that's a feature request for that integration, not for the knx integration.