Singles counts for missing detectors are 0 instead of -1 or -999

[tomkooij]

This might not be a datastore issue, but perhaps a station software issue, but since it affects raw data, I'm reporting it here.

singles counts for 2 detector stations (no slave) are apparently reported as zero (0) instead of -1 or -999 for the missing detectors.
IMHO, this is wrong.

Sample TSV downloaded from my publicdb test VM:

# Event Summary Data - Singles
#
# Station: (203) College Hageveld
#
(...)
#
2017-01-02	00:00:00	1483315200	162	90	122	67	0	0	0	0
2017-01-02	00:00:01	1483315201	139	80	151	98	0	0	0	0
2017-01-02	00:00:02	1483315202	145	86	164	104	0	0	0	0

I can fix this at the import into the ESD by simply setting the missing channels to all '-999'. But that still leaves the raw data to be ambiguous.

Then again, for singles data we might be safe in assuming 0 means "not connected" or "missing detector".

Possible solutions:

• Fix in station software. Change existing raw data.
• Fix in datastore. Change existing raw data.
• Leave raw data as is. Change at import into ESD.
• Leave raw data as is. Import into ESD unchanged. Leave slave channels out of histograms for missing slave.

Any thoughts?

[davidfokkema]

It probably is a datastore issue. The station software probably does not send slave counts if the slave is not present. In the datastore, in writer/storage.py, there should be a default value in the HisparcSingle class. Currently, there is none, and that probably means 0. For the baselines etc. the default value is -1 and is overridden by actual data. We should fix that since a real 0 is an interesting value.

I'm not sure about changing raw data. We've done some manipulation before, but you really don't want to mess things up. Maybe fix the datastore and let the ESD change 0 to -1 only for old data.

[tomkooij]

Thanks for the insight!

I will:

• Propose a PR for the datastore, setting default values to -1.
• Handle -1 in the ESD/publicdb histograms part.
• Write a script that changes ESD data after import. Leave raw data untouched.

[tomkooij]

Ouch. This changed from 'wrong, because inconsistent with HiSPARC standards' to 'inconsistent'.

The missing slave data is not set to zero by the datastore. It arrives as such.

See HiSPARC/station-software#43.

The current status is:

• All data before that was in the datastore before the feb 13st 2017 11:12 UTC migration: slave columns set to -1 for missing slaves
• All data that arrived after the migration: slave columns set to 0 for missing slave.

singles data for station 203 (from the ESD):

2017-02-13	11:11:32	1486984292	158	94	165	81	-1	-1	-1	-1
2017-02-13	11:11:33	1486984293	149	75	144	82	-1	-1	-1	-1
2017-02-13	11:11:34	1486984294	179	95	142	74	-1	-1	-1	-1
2017-02-13	11:11:35	1486984295	180	91	121	56	-1	-1	-1	-1
2017-02-13	11:11:36	1486984296	160	94	121	77	0	0	0	0
2017-02-13	11:11:37	1486984297	178	104	162	81	0	0	0	0
2017-02-13	11:11:38	1486984298	149	76	138	72	0	0	0	0