Message Parsing

[tanneberger]

Problem

Our current parsing method is to receive and re-try parsing the message every time we receive new data. This has a couple of down-sides:

1. Energy Inefficient, because many retries are necessary before it was successful
2. Poised or corrupted data will cause mayor problems with decoding

Possible Solutions

1. Prepend messages with the size of the message. This would allow us to precisely tell us when we should try to decode.
2. Only try to decode when we detect the delimiter bytes from protobuf.

Option 1.) Reference Implementation

I would suggest that the size is in big-endian order because thats the industry standard (see RFC1700). For converting from and to big-endien, linux provides the htonl and ntohl functions.

if (ntohl(*((int*)self->receive_buffer)) >= self->receive_buffer_index) {
    // ... decode 
}

Related Problems and Ideas

• Error correction for noisy channels (UART/IEEE802.11/I2C)
• Detecting received corrupted data
• Cleaning received corrupted data

[schoeberl]

AFAIK there are no delimiter bytes in protobuf, it is a binary encoding. +1 for the length prepended and also good to use network order for the length. For error correction on serial (UART) channels one could delegate this to the Internet protocol and use IP over serial (SLIP).

[erlingrj]

Our protobuf messages are actually delimited by a varint (that is a variable-sized integer which is how protobuf encodes data) with the size of the subsequent message. So we could use nanopb to decode the first received byte, record the size, and then read the entire message before calling decode on it. This is also supported by Googles protobuf libraries.

For unreliable channels, such as various serial ports, we of course run the risk of receiving a corrupted delimiter byte. Ideally error detection/correction (e.g. by checksum or something) would be delegated to the channel implementation, because it is already provided by Tcp and Coap channels