"ground truth" discussion

[freeman-lab]

There's been lots of discussion of the "ground truth" labels currently used for NeuroFinder, so we wanted to consolidate that discussion in one place, and get feedback on some new ideas for moving forward.

current concerns

The labels used now reflect a mix of approaches, including activity-independent nuclear labeling, hand labeling using the raw data, hand labeling using various summary statistics, and hand curation of semi-automated methods.

All have advantages and disadvantages, but the inconsistency has been a source of confusion for both algorithm developers and those trying to interpret the results (see for example #15 and #16). A particular concern is that the variability in performance across algorithms reflects not only differences in algorithms but also differences in how ground truth is defined.

moving forward

Ideally, we should have a ground truth definition that (1) can be arrived at by following a clearly specified procedure (2) would yield similar answers if multiple people followed those instructions and (3) is applied consistently to all training and testing datasets.

Here's one proposal:

1. Provide each of several independent labelers (at least 3-5) with a mean and local correlation image
2. Also provide several examples of what individual neurons look like to the labelers
3. Have them label all datasets and aggregate the results via some consensus procedure

What do people think of this idea? Or other ideas?

cc @marius10p @agiovann @epnev @Selmaan @aaronkerlin @sofroniewn @svoboda314 @boazmohar @syncrostone

[agiovann]

Very good point. I would say it is also necessary to inspect a down-sampled and zero-mean version of the movie to be sure that spikes are not missed and/or that overlapping neurons are properly segmented.

cc @marius10p @agiovann @epnev @Selmaan @aaronkerlin @sofroniewn @svoboda314 @boazmohar @syncrostone

[sofroniewn]

@agiovann agreed - a great way to draw rois would be to have a mode where such a movie could be made to play in loop in the background as you are drawing on top of it

[Selmaan]

I'd agree with the idea about actually playing the (heavily temporally down-sampled) movie in the background, or some other means to check hand circling against the timeseries data. I don't think the mean or a local correlation image are a replacement for this if we want ground truth.

The ROIs will probably have some relatively inactive neurons selected because of a mean image, and active neurons not visible in the mean image. It would be useful in evaluating an algorithm's performance to do some minimal processing of the traces (e.g. %frames above 1 dF/F) to classify cells as active or inactive for this purpose.

[freeman-lab]

@Selmaan great point, if we do this redo of the ground truth, we could compute some measure of activity as you suggest, and then add a toggle for two versions of the results table, one for all neurons, and one for all neurons with activity above some threshold, @marius10p had suggested something similar.

[marius10p]

@freeman-lab sounds like a great plan to me, and definitely worth it for understanding the behavior of the algorithms. It would also allow you to compare the algorithms to inter-human agreement, which is useful information.

Not sure how you compute the local correlation map, but we find it useful to highpass filter the images at ~50um (like a background subtraction), and bandpass filter the timecourses of each pixel between 0.1 and 2 Hz (or something like that).