After reboot, pressure data is missing for several seconds

[kedder]

Apparently #25 introduced a regression: after the boot, sensord does not emit any actual pressure or vario data to the NMEA stream for several seconds. Sometimes the period is quite long - I've seen 20 or 30 seconds.

NMEA stream consist of repeated sentences like

$POV,E,+0.00*39
$POV,P,+994.33,Q,+0.00*4C

This is evident by looking at vario needle in xcsoar: it is frozen at 0 m/s for several seconds after the boot. Reverting 6fa4005 fixes the issue: the pressure/vario data starts coming immediately.

This is also reported in Openvario/meta-openvario#107

@hpmax, can you have a look?

[hpmax]

@kedder Sure... Can you or @Blaubart provide more detail:

1. Does the delay always happen?
2. How long is the delay? Is it consistent or does it vary? Min/Max/Average?
3. Are those sentences repeated every time? Or is E always 0 and P some value that stays constant during the delay period? Is Q value constantly 0?
4. Does it always go back to normal?

Any additional helpful details? How reproducible?

Honestly, I have a new version that I'm getting very close to releasing. I can probably release it tomorrow. I'm not really interested in debugging the current version at this point. So, I might prefer to make the new release and then if the problem still exists try to debug it.

[kedder]

1. Yes, it happens consistently, every time
2. I just did 3 reboots and delay was 33, 27 and 40 seconds.
3. $POV,P differs slightly between reboots, but while the delay happens, the value is always the same. $POV,E is always 0.00.
4. For me it has always started working normally after the initial delay after the reboot

It is easy to reproduce:

1. build the latest image,
2. boot, observe xcsoar vario needle stuck, despite the input to the sensors (I'm pressing on the Ptek hose nozzle with my finger).

With 6fa4005 reverted, sensors become responsive immediately.

[hpmax]

Okay... If for some reason the pressure readings were constant, that might explain why the vario is zero. I may have increased the startup time, but I wouldn't expect it to be outputting anything during that time.

What about the pitot/airspeed reading? Is that 0 the whole time (did you try blowing into the pitot to force it to read something?

[hpmax]

I think I may know the cause and think it's been corrected in the new version. There's an "if (glitch<2)" in the pressure_measurement_handler(). When I was looking at the code during the latest (not yet released) update, I couldn't figure out why it was there and removed it. It could definitely suppress the output on start up.

[hpmax]

I have pushed a new version into hpmax/sensord. Please try it and see if it fixes it and if everything seems to be working. Also, please try "compdata"... Usage is: "compdata -s -c /opt/conf/sensord.conf" It'll ask how many thousands of data points. It'll take about one minute per thousand data points, so start off with a few to see how it goes. I am now suspecting there is a temperature and/or pressure dependent on the compensation. I'm going to experiment with that to try to improve things in the next week or two.

[kedder]

@hpmax the version in master branch of https://github.com/hpmax/sensord does not build cleanly:

# make
mkdir -p obj
cc -DVERSION_GIT=\"0.3.3-30-g9677f99\" -c -o obj/ms5611.o ms5611.c -g -Wall
mkdir -p obj
cc -DVERSION_GIT=\"0.3.3-30-g9677f99\" -c -o obj/main.o main.c -g -Wall
main.c: In function 'main':
main.c:428:6: warning: unused variable 'j' [-Wunused-variable]
  int j=0;
      ^
mkdir -p obj
cc -DVERSION_GIT=\"0.3.3-30-g9677f99\" -c -o obj/nmea.o nmea.c -g -Wall
mkdir -p obj
cc -DVERSION_GIT=\"0.3.3-30-g9677f99\" -c -o obj/cmdline_parser.o cmdline_parser.c -g -Wall
mkdir -p obj
cc -DVERSION_GIT=\"0.3.3-30-g9677f99\" -c -o obj/configfile_parser.o configfile_parser.c -g -Wall
cc -g -o sensord obj/ms5611.o obj/ams5915.o obj/ads1110.o obj/main.o obj/nmea.o obj/timer.o obj/KalmanFilter1d.o obj/cmdline_parser.o obj/configfile_parser.o obj/vario.o obj/AirDensity.o obj/24c16.o -lrt -lm
make: *** No rule to make target 'cal', needed by 'all'.  Stop.

[kedder]

The sensord binary gets built though. with it XCsoar shows the vario movement immediately after the boot, however the vario needle does several sudden jumps between -1 and +1 in the first 30 seconds or so. After that it behaves normal (FWICT).

@hpmax do you have a way to test your version of sensord?

[hpmax]

The complaint about int j=0; doesn't bother me much -- just an extraneous variable declaration.

I'm not sure what to make of the can't make target cal (I presume that's sensorcal).

I don't know what to say about that. When analyzing data, I typically ignore the first 1000 samples (which should roughly correlate to 20 seconds). There is a lot of ugliness when the thing first starts running related to the glitch behavior. I.e. it has to "warm up" and it takes some time after you establish a constant polling rate before the ms5611s stabilize. Also, for reasons that I still don't understand, running it in the foreground does give better results. The standard deviation of the data is lower. Not a lot lower, but it is consistently lower.

It is possible that without having a stabilized value, the anti-glitch compensation is actually making things worse during startup. The glitch compensation is pretty straightforward but knowing when to turn it on and off is a big deal, because when it's turn on the D2 filter shuts off, which means if the D2 drifts too far from the filtered value it may never shut off. The new code (the one you just tried) will shut off automatically after some period of time even if it doesn't think the glitch is over. There's really no perfect solution here, but I can try to improve the startup behavior.

Right now, my audio amplifier broke... again. So I'm going to have to pull it out of the glider to fix it, but I'm working on a test fixture that I can run it in my house.

[hpmax]

I pulled my OV out of the glider and have it wired up to the power supply so I can now operate it in doors.

With OV running XCSoar, I logged into the device, killed sensord and restarted 5 times, and each time I saw the behavior:

1. Sat there for a few seconds.
2. Dropped to -1 m/s over a period of maybe 1-2 seconds.
3. Went back to 0 over a period of maybe 3-4 seconds.
4. Jumped between 0 and -0 repeatedly, with occasionally blips to .1 m/s.

Pretty sure I'm using the code I gave you.

Am I missing something?

[Blaubart]

Sorry, I'm not a programmer and not able to understand everything you are talking about. Is the problem solved?

Thanks!
Dirk

[hpmax]

As stated, I am unable to replicate the problem in hpmax/sensord. I am working on a new version to try to deal with the fact that glitch compensation is pressure dependent -- once complete, I will do a PR and hopefully get everything caught up.

[Blaubart]

thanks

[kedder]

@hpmax, can you replicate the problem with sensord/master (e1ecf57) though?