Improve flex decoder by enabling lua code for validation and/or decoding

[pjsg]

This PR adds support in the flex decoder for using LUA code to validate received messages and/or improve decoding (for those cases where 'get' is insufficient). The LUA extension is only built if LUA 5.4 development libraries can be found when building rtl_433.

A document describing the structure and functionality of the LUA code is at LUA docs.

I needed this in order to validate the message integrity checks of a remote control. I also wanted the decoded message that gets sent over MQTT to include the name of the button that was pressed.

[zuckschwerdt]

Interesting proposal. We were thinking about some kind of DSL to improve the flex syntax for a long time now. Central would be easy syntax to address and extract (ranges of) bits from the bitbuffer. I'm not sure how well this fits Lua.
This is the right direction but I'd like to see better support for that central action of pulling bits from the buffer.
Do you think some kind of preprocessing transform could introduce better syntax to Lua?

A sketch of such an operator was:

Single bit (e.g. bit 6):
[6]
A few bits in order (e.g. a byte starting at bit 6):
[6:8]
A few bits in reflected order (e.g. a byte starting at bit 13 going to 6):
[6:-8]
[13:-8]
Composite bits out of order (e.g. 12 bits with the upper 4 being in the next byte):
[20:4][8:8]
Bits padded (i.e. left shifted 4):
[6:8]0000
Composite bits out of order (e.g. 12 bits with the upper 4 being in the next byte), also some transformations:
~[20:4][8:8]
~[20:4 8:8]
~[0 2 4 6 8]
not [20:4 8:8]
signed [20:4 8:8]
[0:4] *10 + [4:4] + [8:4] * 0.1

[pjsg]

I was thinking about that -- my initial thought was to add support for an enhanced version of http://www.inf.puc-rio.br/~roberto/struct/ which added support for bit fields. For example, allow "I:n" and "i:n" to be unsinged and signed n bit integers, and "X:n" to be an n bit skip. All the existing format characters would (I think) trigger a skip to the next byte boundary. I.e. it would like like a C structure definition with bit fields and regular fields.

I would need to think about reversed bit fields.

[pjsg]

I'm working on this -- but taking a slightly different approach -- adding some useful methods to extract a bitfield from consecutive bits in various little/big endian styles.

[pjsg]

The equivalent of your example [0:4] *10 + [4:4] + [8:4] * 0.1 is

function decode(packets)
    -- assume we have validated checksum etc
    local p = packets[1]
    return {val=p[{0,4}] *10 + p[{4,4}] + p[{8,4}] * 0.1}
end

This assumes stuff about bit ordering, but that is easily settable.

I think that I'll add support for negative widths and omitted widths. For negative widths: if the buffer is 01101010 11000101 then [13:-6] is 100011 => 35 (decimal). Does that align with your understanding? @zuckschwerdt

[zuckschwerdt]

Does that align with your understanding?

Yes, the syntax looks clean and useable. Very nice!

[zuckschwerdt]

I think that we should use a cmake property for Lua to offer the same AUTO ON OFF options as for OPENSSL, RTLSDR, SOAPYSDR.

[pjsg]

Quick question -- which do you think is more natural -- starting with bot 0 as the least significant bit of the first byte, or bit 0 is the most significant bit of the first byte? I'd just set the default to the natural approach.

[gdt]

That's an age old debate. Having been raised in a PDP-11 household, it's obvious that bit 0 is the least significant bit. My view is that computer instructions lean that way, even if IEEE networking specs lean the other.

I think rtl_433 should have a global style rule about this, and I'd vote for "bit 0 is LSB" and "bit 7 is MSB" for 8-bit bytes.

[zuckschwerdt]

Yes, rtl_433 is special here: we fill the bits starting from "the left". I.e. the first decoded symbol goes to 0x80.
So for the "little endian" approach "numbers work even if you get the size wrong" we need to have index 0 point to MSB of the first byte.

[pjsg]

Thanks -- I'll leave the default as the first bit is the 0x80 bit of the first byte.

[pjsg]

Is there anything more that I need to do?

[gdt]

How many different operating systems has this been tested on?
Also, while building lua if lua is present is developer-friendly, it's not good for packaging systems. Or rather, there needs to be a (documented) build instruction not to build lua, even if it's present, because packaging systems control dependencies and need to ensure that only declared dependencies are used.

[zuckschwerdt]

I'd opt to keep it 7-bit ASCII and not add emoji's to CMakeLists.txt.

[gdt]

I'd opt to keep it 7-bit ASCII and not add emoji's to CMakeLists.txt.

Agreed. Sources should remain ASCII, with UTF-8 as needed for contributor's names.

[pjsg]

I've tested operation on linux (rpi). It builds on MacOS (and passes tests), and I'm trying to get a Windows box to figure out how to do the various build/install steps.