Gaussian smoothing kernels of OASIS

[AustinNgs]

Hi Peter,

I was wondering where I can see the result of optimal Gaussian smoothing kernels searching, since I didn't find out any analysis or the acutal number for the method part: "For evaluation of spike rate inference results with OASIS, we tested Gaussian smoothing kernels of variable standard deviation……".

Thanks,
Austin

[PTRRupprecht]

Hi @AustinNgs

Thanks for the question. You’re right - we didn’t show a detailed analysis of this aspect in our GCaMP8 preprint.

For each analyzed dataset, varied the temporal shift between predictions and ground truth (for MLSpike and OASIS) and identified the shift that maximized the correlation. So it's basically a grid search when evaluating the output of the algorithm. We have mentioned this approach in our original CASCADE paper. I may have looked into the sensitivity of the outcome towards shift for a few datasets in the beginning, but I have not done it systematically in the end.

For some model-based algorithms, we found that inferred spike rates were often temporally shifted to later time points, and this delay was variable across datasets (0.16 ± 0.14 s for MLSpike, 0.03 ± 0.09 s for Peeling, 0.31 ± 0.23 s for CaImAn, 0.29 ± 0.22 s for Suite2p and 0.27 ± 0.19 s for Jewell&Witten; mean delay ± s.d. across datasets). We corrected these shifts for all analyses presented here.

Regarding your question about the optimal smoothing kernels in the GCaMP8 preprint: we applied this optimization only to OASIS and MLSpike, not to CASCADE. Since CASCADE is a supervised method, it should already learn the appropriate temporal smoothing to match the ground truth. To keep comparisons as fair as possible, we smoothed OASIS and MLSpike outputs using the same Gaussian kernel used to generate the ground truth. E.g., if a CASCADE model used smoothing_200ms parameter, we applied the same amount of smoothing also to the output of OASIS and MLSpike. In theory, we could have optimized this smoothing parameter for each dataset or each neuron separately, but this would, in our opinion, be too much overfitting to specific datasets, especially if those are small.

I hope this clarifies things. I hope you have enough details to reproduce the results with that; and if you need any help with the specific implementation, just reach out. I know that there are a couple of such details that are probably not worth discussing in a paper but potentially a bit annoying when not explicitly specified.

Best,
Peter

[AustinNgs]

Hi Peter,

Many thanks for the detailed clarification — this is very helpful for reproducing the evaluation protocol.

Best,
Austin