CAN frames with different format causing false readings?

[moritzj29]

Hi, first of all thanks for this awesome library! I have been tinkering with it for some time already, getting all the sensor readings from my heatpumps into Home Assistant for monitoring purposes.

I recognized, that for some sensors spikes with absurdly high/low values are observed and tried to identify the root cause. The issue appears to be due to some differently formatted CAN frames appearing from time to time:

Regular CAN frame:

[13:07:45.662][D][tt:360]:      - Can-ID: 0x1F400FFF Start of message with id: 150 with length 2
[13:07:45.666][D][tt:369]:      - Can-ID: 0x1E800FFF Part of message with id: 150 with remaining length 0
[13:07:45.677][D][tt:430]: [EXS] Can-ID: 0x1E800FFF (511709183), Device ID: 1, Data: 0x423cfe002f00005c66, Datapoint: 0x002F (47), ID: 0x2FFE3C (3145276), function_group: 0x3C (60), function_number: 0xFE (254)
[13:07:45.682][D][sensor:135]: 'Wärmemenge Heizen': Sending state 23.65400 MWh with 3 decimals of accuracy

Irregular CAN frame:

[13:08:08.006][D][tt:369]:      - Can-ID: 0x00000FFF Part of message with id: 209 with remaining length 0
[13:08:08.017][D][tt:430]: [EXS] Can-ID: 0x00000FFF (4095), Device ID: 1, Data: 0x423cfe002f00008f040104, Datapoint: 0x002F (47), ID: 0x2FFE3C (3145276), function_group: 0x3C (60), function_number: 0xFE (254)
[13:08:08.022][D][sensor:135]: 'Wärmemenge Heizen': Sending state 2399404.50000 MWh with 3 decimals of accuracy

As you can see, I already modified the code a little bit to generate these log messages of existing sensors.
Both CAN IDs translate into the same device/function_group/function_number/datapoint. Therefore setting the same Home Assistant Sensor. But the data is different, as well as the first part of the CAN ID.

Does anyone have observed similar messages on the Hoval CAN bus? In my case it is a setup consisting of two heatpumps incl. control displays and a modbus gateway.

Do you have any idea on how to interpret those messages correctly?
Based on the current implementation of TopTronic::parse_frame, these CAN frames do not have msg_id = 0x1f set, which should indicate the start of a message. They are currently parsed as an additional frame, without an explicit starting frame.

Quick fix would be to exclude these kind of CAN IDs completely from processing, of course, but I would really like to know what their purpose is...

I hope it's not too pushy, but if you have a minute I would really like to get your opinion @dkarv and @abichinger, since you contributed heavily to this part of the code.

[dkarv]

Interesting. I checked my HA history and found similar spikes about twice per year for each sensor.
It seems the first frame is totally missing, but this should be caught by the check here. And then the message should be ignored.

How often does this happen in your setup?
Maybe it happens due to two heat pumps, and them using the same message ids?

[moritzj29]

Thanks for your fast reply! Twice a year would be actually quite ok 😆:

It seems the first frame is totally missing, but this should be caught by the check here. And then the message should be ignored.

I actually found the first frame. It is received ~30s before (identified by msg_id). Therefore the message is not ignored.

=> It appears to be very consistent that every false value results from a multi-frame message, with the first message being received ~30s (or more) before. Usually, the frames are sent within few seconds and the message is completed.

Meanwhile I spent quite some time going through the logs and trying to tweak the parsing. Without success.

Anyhow, let me summarize what I have tried and learned so far:

Hypothesis 1

The common pattern appears to be the second message to have msg_id = 0x00. For "regular" frames the following frames have a 0x1e, 0x1d, 0x1c etc. as msg_id.

=> not the root cause, since I observed wrong values with both msg_id patterns meanwhile.

Hypothesis 2

Maybe it happens due to two heat pumps, and them using the same message ids?

The current code relies on the uniqueness of msg_id = can_id >> 24 parsed directly from the can_id. Given multiple Hoval devices sending messages, this requirement may not be met all the time. E.g. device 1 starting message with id: 1. Before the next frame is sent by device 1, device 2 starts a message with the same id. The starting message from device 1 in the pending_messages_ dict is overwritten. This could result in a mix up of frames from all connected devices (device_id is only filtered in later stage).

=> I thought this to be a hot candidate. I implemented a pending_key to namespace the msg_id to each device:

uint8_t msg_id = can_id >> 24;
uint32_t device_id = (can_id >> 11) & 0x7FF;
...
u_int8_t msg_header = data[1];
uint32_t pending_key = (device_id << 8) | msg_header;

But the false values persist.

Hypothesis 3

I observed the issue only with datapoints consisting of multiple can frames. The identification of the datapoint is always given in the payload of the first frame. I observed that the second frame sometimes varies in length (when values are wrong). I therefore tried to check the size of the payload against the size of the expected data type.

=> while filtering a couple of messages, false values still persist.

Hypothesis 4

Hardware Issue? Since I observe this issue so consistently, but no one else is reporting it, I plan on checking the Hardware once again (termination etc.). Having no experience with CAN, I don't really believe that some hardware issue could result in such a fine grained malfunction (not observed for single frame messages for example). But who knows...

I'm using a plain ESP32 ethernet module with a sn65hvd230 CAN module, by the way. (Side note: MCP2515 is not sufficient for this application, the input buffer is too small / the read-out too slow for the rate of messages on the bus. Also experienced on a single HomeVent / Touchpanel combination. It will somehow work but some frames will be lost (warning messages in log))

Gateway

I have a Modbus gateway present in the setup. Reading the same datapoints via the gateway does not show the false readings, but the gateway actively polls and does not report every matching CAN frame.

For testing purposes I also disabled polling via the gateway, without any impact. But cloud functionality is still active.

Next step would be to disconnect the gateway completely. Does any one of you run this CAN gateway together with the Hoval gateway?