Occasional Nonsense Data Received - Would you add range-check & reject (Vevor 7in1)

[tyglik]

Hi guys! First of all, thank you for this great work!
I have a Vevor 7in1 and I can read its data these last few days thanks to your addon, it's awesome.

However I noticed, that occasionally (2-3times in no more than a week) the data I receive is utter nonsense... Luminance >200kLux, Humidity 188%, Temperature 61.2 degrees Celsius, wind direction in thousands of degrees, wind speeds in the huricane range...

I think it is the occasional packet that is incorrect throughout...

Can you add an option to enable simple min max check condition on a few stats, like Humidity (can not be more than 100%, right) and/or Luminance, that if outside the bounds, the whole packet would get rejected?

If there is a way to implement some better filtering, that would be even better, as in reject out-of-distribution spike-like data, but for starters a simple "min<state.xyz<max" type of config/rule for a couple of (say three) stats may be sufficient...

Thanks again, I'm happy to provide any more info if you need.
Best regards

[ProfBoc75]

@tyglik I did the decoder and unfortunately the message validation if a simple checksum , add bytes, and not a CRC. But yes we can add few other conditions to filter out abnormal / out of range values.

To clarify your nonsense values, are all out of range in the same message or sometime the humidity, then wind speeds, and so on. I try to understand if we need to exclude out of range values only or exclude all the message which could be totally wrong.

[zuckschwerdt]

To track the source of errors it could be helpful to see the codes. E.g. run with -R 263:vv and note the codes shown after a bad transmission.

[zuckschwerdt]

As a general guideline rtl_433 will try to always report successful decoded messages. If the message is obviously broken then filtering it out is okay, e.g. if it's a likely false positive.
But if just some data field is implausible then reporting the decode has value. I.e. not receiving anything might be more of a problem for users then seeing that the sensor is received but measured bad values, perhaps due to some malfunction.
And the bad values can in some cases be yet unknown error codes, we have seen this with raw values that have a hex of e.g. 0xfffe or such.

[tyglik]

@ProfBoc75 I generally prefer open options - possibly in the config - i.e. let the user decide depending on their case and importance of data: 3x 3 input fields ( [field] [min] [max] ), 1 check-box [AND/or/OR], and 1 selector (or a second checkbox, because it only has 2 options: reject invalid value/reject whole packet)

In my case, It's meteostation data, coming in every 30sec, so discarding the whole packet is probably safest, if it can not be left customizable.

As for is the whole packet malformed or just the single data-field, I do not know, but looking into my 5min aggregate statistics:
Humidity = [ (!)188%@1219.174056, 95%@1219.2125, 94%@1220.0030]
Temp = [ (!)61.2C@1219.174056, 6.3C@1219.2125, 7.0C@1220.0030]
Wind_dir = [ (!)3838deg@1219.174057, (!)621deg@1219.2125, (!)705deg@1220.0030, (!)611deg@14:25:39],
luminance = [deleted, ... , ... , (!out of distribution spike!)5306lux@14:25:39]
UV_Index = [ deleted, 1@1219.2125, 1@1220.0030]

SNR and Noise stats seem OK (simliar to nearby samples) at these times. I manually removed from the SQL tables (states and stats) the offending entries for UV index and wind speeds yesterday before I thought of openning the ticket here. From what I have, it seems that sometimes it is a full packet, sometimes it is just a single field? The first listed error above, Hum error received 1s before the Temp & Wind_dir - I do not know the comms protocol, but if complete data is sent in 2 separate messages, it could be that it's the Weatherstation that goes into a weird state and sends garbled data with correct CRC. -
As I said, I'd leave it up to the user - Whether to consider them as all at once (AND) or any of the above (OR) as well as dispose of the value/packet... and what these protected attributes may be...
Anyway, That's as much as I have now, and I'll see if I can find something better using the SQLite and update this.

@zuckschwerdt I'm happy to help, however I run this as a "Home Assistant" plugin and there's no way I can "-R 263:vv" to anything - if you tell me how it translates to a line in the "rtl_433.conf.template" I'll add it in no problem.

coincidentally 3838 == 0x0EFE (12bit)

Also, maybe related, but possibly a different issue, the rtl_433 stops receiving for no reason. I tried resets, re-positioning antena, unplugging and plugging the USB dongle back in, it helped on 2 occasion, but I couldn't get it going yesterday, so it was dead for approx 8 hours. This morning, LLM(google) suggested to FIX the gain to 20, which doesn't look like it's working (from reading the logs) but makes the thing work, as opposed to removing it (stops it working) ..

I have it for 3-4 days only, so still getting used to it, apologies if that was OT.

Thanks guys!

[zuckschwerdt]

You might want a custom processing for your data, e.g. deriving rain rate values is a common use case here.
See https://github.com/merbanan/rtl_433/blob/master/examples/rtl_433_custom.py or https://github.com/merbanan/rtl_433/blob/master/examples/rtl_433_mqtt_relay.py

[LtqxWYEG]

I only get garbage. It does not matter if RSSI is -16 or 1: #3432

[kbahey]

I have a Vevor 7n1 as well, that has been running for about six months or so.
Like you, I do see spurious values occasionally.

Sometimes it is just Sunlight.
Other times it is several values together (wind gusts, humidity, UV).

I did implement some sanity checking in Home Assistant, as I detailed with code snippets in this comment on #1964.

But it does not cover all cases. For example, rainfall sometimes dips below previous levels then goes back to the value prior.

The values are not the same every time, so it is not straightforward to just filter on fixed values.

But it is still very annoying to have to go and delete the bad values.

P.S. Can we change the title of this issue to include the Vevor 7n1, so others can easily find it?