ideas for how to extend auto_analyse with support for fixed length encodings ?

[NiKiZe]

Looking for input on what the best approach to implement this. (or at least pro/cons)
Currently auto_analyse_raw_data.py only has support for space encoded messages, and it is great at it.
Now (mostly for fun) I was thinking off if there is any way to extend it with support for fixed length messages as well. (Such as the Amcor fixed length bit encoding)

Is there any smarter way to do this, than to first determine header, footers and gaps (can that be made to a separate function maybe) and than after that go over the remaining mark+space pairs again?

Another part is, can we make it better at displaying repeated messages. (maybe just say that it is repeated, instead of printing the same data sections multiple times, but the question then is, how to determine that it is repeated.)

Some faked 8 bit messages

// space encoded, no trailing gap
uint16_t rawData[] = {2300, 3400,  400, 1300,  400, 400,  400, 400,  400, 1300,  400, 464,  400, 400,  400, 1300,  400, 1300,  400};
// space encoded, with trailing gap
uint16_t rawData[] = {2300, 3400,  400, 1300,  400, 400,  400, 400,  400, 1300,  400, 464,  400, 400,  400, 1300,  400, 1300,  400, 6800};
// space encoded, repeated
uint16_t rawData[] = {2300, 3400,  400, 1300,  400, 400,  400, 400,  400, 1300,  400, 464,  400, 400,  400, 1300,  400, 1300,  400, 6800,  2300, 3400,  400, 1300,  400, 400,  400, 400,  400, 1300,  400, 464,  400, 400,  400, 1300,  400, 1300,  400, 6800,  2300, 3400,  400, 1300,  400, 400,  400, 400,  400, 1300,  400, 464,  400, 400,  400, 1300,  400, 1300,  400};


// fixed length encoded, no trailing gap
uint16_t rawData[] = {2300, 3400,  400, 1300,  1300, 400,  1300, 400,  400, 1300,  1300, 400,  1300, 400,  400, 1300,  400, 1300,  400};
// fixed length encoded, with trailing gap
uint16_t rawData[] = {2300, 3400,  400, 1300,  1300, 400,  1300, 400,  400, 1300,  1300, 400,  1300, 400,  400, 1300,  400, 1300,  400, 6800};
// fixed length encoded, repeated
uint16_t rawData[] = {2300, 3400,  400, 1300,  1300, 400,  1300, 400,  400, 1300,  1300, 400,  1300, 400,  400, 1300,  400, 1300,  400, 6800,  2300, 3400,  400, 1300,  1300, 400,  1300, 400,  400, 1300,  1300, 400,  1300, 400,  400, 1300,  400, 1300,  400, 6800,  2300, 3400,  400, 1300,  1300, 400,  1300, 400,  400, 1300,  1300, 400,  1300, 400,  400, 1300,  400, 1300,  400};

[crankyoldgit]

(mostly for fun)

Haha. Glutton for punishment!

Another part is, can we make it better at displaying repeated messages. (maybe just say that it is repeated, instead of printing the same data sections multiple times, but the question then is, how to determine that it is repeated.)

Let me answer these in reverse.
If you look at the history of the file (and the original bash shell script) it used to do that.
I think I ditched it doing that when I moved to python or when I "improved" it.
I remember it was when I had a mess of global variables (thus I want to say it was back when it was a shell script) that I had it doing it. It's fairly simple to do, just remember what the previous data segment(s) was/were. When you've finished a data segment, go back over the history of other data segments. If there was a match, say the current one is a repeat. You could probably add an inverted bits check as well as a "same" bits check.

if there is any way to extend it with support for fixed length messages as well.

I gave this a couple of very half-arsed attempts yonks ago. I did space encoding first because it was the most common, and the most pressing issue/reported thing.

IIRC, it currently goes over the data to try to put things into buckets of "likely the same" timings, both for marks & then for spaces.

Assuming "space" encoding meant I didn't need to backtrack. In order to do the other common protocols, we probably need to do a proper syntax parser, now that I think about it.

If we parse the raw string into lexical tokens, we could then parse it with some grammar analysis.
e.g.
mesg := leader{0,1} data-blocks footer
leader := ldrmark (ldrspace | hdrspace | ftrspace)
header := hdrmark hdrspace
footer: ftrmark ftrspace{0,1}
data-blocks := data-block{1,+}
block-footer: ftrmark ftrspace
data-block := header{0,+} bit-pairs block-footer{0,+}
bit-pairs := space_enc_pairs | equal_enc_pairs | mark_enc_pairs
space_enc_pairs := (bitmark zero) | bitmark one){1,+}
equal_enc_pairs := (zero one) | one zero){1,+}
mark_enc_pairs := (zero bitspace) | one bitspace){1,+}

Obviously, that parser grammar isn't tested, it's just off the top of my head. (AND in no particular parser specification language ;-)

[crankyoldgit]

For the record, "duplicate" detection was in the original bash script.
https://github.com/crankyoldgit/IRremoteESP8266/pull/454/files#diff-76e01453a39c52695cfbadec9bd09ddcL160-L163