Blips in MS5611 sensor data

[hpmax]

When I turn on my OpenVario, I will see a lot of +/- blips on the vario, every few seconds, with a magnitude in the neighborhood of .5 m/s.

I decided to record the data out of the MS5611s and found that upon boot up, I'd see a spike high (of about 35-40 Pascals) followed by an immediate low spike of perhaps 10-15 Pascals. The high pulse lasted about 2 samples, and the low pulse lasted perhaps 10 samples.

Interestingly, the effect occurred on BOTH MS5611s simultaneously. and it repeated every 92-94 cycles (it varied).

I killed the daemon and restarted it, and the blips mostly or completely went away. I recorded twice, once for over 20 minutes and never saw a blip, once for over a minute and only saw one blip. This seems almost completely repeatable for me.

Blue trace is the restart, the other 3 were what it does on bootup.
blips.zip

[mihu-ov]

These blips (and incorrect pressure reading spikes) are an unsolved mystery and further investigation would be more than welcome. One known issue is RF noise from the VHF transmitter but this seems to be something different.

[hpmax]

@mihu-ov My glider has a Dittel FSG-2T but the unit was off. It is also equipped with a Stratux (which has no transmitter), and a SoftRF unit, both of which were on. The SoftRF is capable of transmitting at 900-ish MHz although its a pretty low power output. I also have multiple switching converters going from 12V to 5V to among other things power the SoftRF and Stratux... But their state should have been unaffected by stopping an re-starting the daemon, so this seems unlikely.

So you are saying this is a known issue? I'm surprised more hasn't been done by this, this seems like a pretty significant problem. The blips do get through the Kalman filter and are quite noticeable.

[mihu-ov]

In my setup I have no issue with blips and only minimal effects when transmitting on VHF. In one of our club gliders (DG300) there is an occasional blip and vario goes wild when transmitting. Your traces are probably an extreme case - that´s great for investigating the problem (RF issues or anything else).

[hpmax]

Again, I see absolutely no reason to believe this is RF (at least from outside the OV) related. My VHF transceiver was off at the time. Those waveforms are 100% repeatable. When I start the OS, the red, orange, and purple traces are what happen. You can see it is regular. They are occurring every 92 to 94 samples. When I kill the process and restart the blips either mostly or completely go away. I do not see how an external device could be impacting that.

Blue is TE, Orange is static... out of the same data file.

[mihu-ov]

I agree, I don´t believe your case is RF induced.
PX4/PX4-Autopilot#7311 descibes a similar effect
https://px4.github.io/Firmware-Doxygen/d6/d17/ms5611_8cpp_source.html is the code used in PX4
Maybe a comparison what they do differently would make sense?
Is there any difference in timing of sensor reads during the blip? What is the sensor temperature readout at each sample?

[hpmax]

There are some signs that point to this, but others that don't:

1. If the problem was inadequate wait between conversion and read out, its likely it would affect both sensors. It does.
2. If a temperature conversion was corrupted that temperature data is used for something like 10 pressure samples before it's re-sampled. That's about the length of the blip.

None of this explain why only temperature conversions are corrupted or why it's periodic, or why restarting makes the problem go away.

But here's the other thing... Even if the data is somehow corrupted, why is it consistently increasing for a couple samples and then consistently decreasing for more samples. I mean, okay, maybe we have a bad temperature conversion, this results in bad compensation value, and the next pressure read is consistently high, and the temperature compensation corruption is consistently decreasing it until the next temp conversion... but there are multiple high samples prior to multiple low samples, and the shape isn't very consistent.

Anyway, I'll try a few experiments:

1. modify the code to dump out the measurement counter, and the temperature
2. increasing the usleep time to say 20ms.
3. Record time between conversion and read.

[hpmax]

Okay, here's what I found:

I tried everything listed above and found nothing of use. Increasing the usleep time may have decreased the magnitude of the problem. The problem occurred at random meas_counter values, and was not tied to a temperature. I tried adding a usleep at the beginning, and tried resetting in the middle of the run. And while the time between conversion and read was not terribly consistent, it was always longer than the requested usleep value. In other words, that whole exploration was a bust.

But, I did find something very interesting out.

If sensord is started with: systemctl start sensord.service (or restart) it has the blips
If sensord is started with: /opt/bin/sensord -c /opt/conf/sensord.conf it does NOT have the blips.

This seems 100% repeatable. Doesn't matter if the service is disabled or enabled, doesn't matter if it's started, stopped, started again, or restarted, or whatever. Similarly, when I run it from the command line, it appears to run clean.

If I edit the /lib/systemd/system/sensord.service to change it from Type = forking to Type = simple, and change the command line to /opt/bin/sensord -f -c /opt/conf/sensord.conf it does NOT have the blips.

I disabled the service and added: "/opt/bin/sensord -c /opt/conf/sensrord.conf" to the ovmenu-ng.sh script and it appears that is a workaround.

So something about systemctl type=forking is causing the problem. My solution is clearly a hack work-around, but hopefully someone who knows Linux better than me might have some ideas.

[kedder]

Is sensord printing anything to stdout periodically?

[hpmax]

I'm not sure what you mean. I didn't pipe stdout to anything, and if it's run through systemctl I won't see stdout. I don't recall seeing anything when running it from the command line, and wouldn't think I would unless I put in -f mode, which I didn't try. What were you expecting/looking for?

[kedder]

Just trying to think up anything that might be different between running from systemd and console.

[hpmax]

I don't know, I was talking to a friend who is way more Linux knowledgeable than me, and he was stumped too, but thought maybe its run with different privileges or environmental variables or something. But since it only fails as "type=forking" that doesn't make much sense... And I'm completely stumped why I'm the only person who seems to be having this problem.

[hpmax]

If you can think of something specific to try, I can certainly perform an experiment, but I went through everything obvious. I still have no idea what is causing the erroneous readings let alone why systemctl has anything to do with it.

[hpmax]

I figured something out...

The foreground code writes all the log data to stdout which is the terminal:
// open console again, but as file_pointer
fp_console = stdout;
stderr = stdout;
setbuf(fp_console, NULL);
setbuf(stderr, NULL);

	// close the standard file descriptors
	close(STDIN_FILENO);
	//close(STDOUT_FILENO);
	close(STDERR_FILENO);	

The daemonizing code writes all the log data to sensord.log:
// close the standard file descriptors
close(STDIN_FILENO);
close(STDOUT_FILENO);
close(STDERR_FILENO);

	//open file for log output
	fp_console = fopen("sensord.log","w+");
	stderr = fp_console;
	setbuf(fp_console, NULL);

I tried replacing the open file for log output from the daemonizing version to the foreground version and then ran it as a daemon through systemctl (type=forking) and... it looks like it worked.

---

Unfortunately, I also noticed a different kind of blip that I hadn't seen before. It only affects the static sensor, always seems to occur on both meas_counter==6 and 26 (and only those two) and its much larger. In fact, the result almost seems random although more often than not its over 123000 and gets ignored by the pressure_read routine.

[mihu-ov]

Wow, great investigation, thank you! Ignoring obvious outliers was a rough hack but finding the root cause would be much better. There is some interesting reading on the effects of sample rate jitter on https://www.freesteel.co.uk/wpblog/2014/11/26/ms5611-altitude-sensor-very-sensitive-to-own-working-temperature/ and also http://blueflyvario.blogspot.com/2012/06/latency-and-timing.html

[hpmax]

I didn't get much out of the bluesky thing. A lot of his blog seems to be him noticing that filters can cause latency. Stuff like that is well understood, and manageable. The freesteel one on the other hand was definitely interesting. That doesn't look exactly like I am seeing, but its definitely similar. When I was looking at timing data I was looking for a short time, not an irregular time. All the times are irregular (although typically by less than 1ms) and by looking at the data I've already collected I can't see an obvious pattern (but the data I have is time from conversion to read, not from conversion to conversion).

I think others have pointed out that the way we are accessing the sensors isn't great. It's very difficult to precisely control timing on a non-real time system, and to do this properly the conversions would have to be done in hardware. Despite that, I think we could better improve timing control if that was necessary.

One obvious thing would be to monitor the time between samples and adjust usleep to better target a 12.5 ms interval. In other words, check the time immediately after waking up from usleep, and then check the time right before going to usleep, and adjust the usleep time to be target time-awake time.

Another possible thing would be to run this as nice -10. If for some reason timing is getting screwed up by a disk access (as might be implied by the fact that not writing to the log seemed to fix the problem), I'm not sure that changing the nice level will help since the disk access might have higher priority than even nice -19 (or whatever maximum un-niceness it).

What really bothers me is that I seem to be the only person who is affected. It's hard to believe... I'm using the Stefly hardware and the stock 20193 release and seeing the problem.