Getting denoised and deconvolved traces that may be pathological

[ethanbb]

Hi folks,

This is a persistent issue that I decided to look deeper into, and I'm realizing it might be a real problem with our setup (but hopefully not!) I am using GCaMP8f and imaging at ~15 Hz. Using mostly default temporal parameters (AR order = 1).

Oftentimes, all but the most active cells will show some abnormality in the denoised traces (C) where for part of the movie, it looks like it's recognizing noise as activity (see bottom left panel below). I have been assuming that these are just not good quality cells, where the deconvolution is more sensitive to small changes in the noise power because the SNR is already quite low, so I've mostly been ignoring/rejecting them. Although this is hard to do for some mice where the majority of "active" cells look like this.

When I looked into it, I realized that the values of S are way higher than I would expect - it should be on the scale of 0-1 generally, no? I'm seeing values of up to 40,000 (bottom right). Furthermore, this is true even when the denoised trace looks more normal/consistent over the course of the recording, which is what has me worried.

I've been playing with the temporal parameters a bit but don't really know where to start. Can anyone help me troubleshoot?

Edit: an example neuron that is more active but still has very high values for S:

[kushalkolar]

Looks like the detrending isn't working, try playing with the params for that?

[ethanbb]

Which detrending step? The only one I'm aware of is when calculating DF/F which happens after CNMF?

[kushalkolar]

Yup the detrending dfof

…

On Fri, Aug 8, 2025, 12:02 Ethan Blackwood ***@***.***> wrote: Which detrending step? The only one I'm aware of is when calculating DF/F which happens after CNMF? — Reply to this email directly, view it on GitHub <#1516 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACHXXRHJKA3KPPJB4JHQHDD3MTC2LAVCNFSM6AAAAACDLOPK3WVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTIMBUHAYTMNI> . You are receiving this because you commented.Message ID: ***@***.*** com>

[ethanbb]

Again, that's just called after CNMF is done so it couldn't affect C or S unless I'm misunderstanding. Also, it seems to me like it is working (top right panel)?

[kushalkolar]

Ah right doesn't affect C, but I think it does affect S?

[kushalkolar]

I think spikes should be based on dfof?

[ethanbb]

I don't think so, it's just this basically

self.F_dff = detrend_df_f(self.A, self.b, self.C, self.f, R,
                          quantileMin=quantileMin,
                          frames_window=frames_window,
                          flag_auto=flag_auto, use_fast=use_fast,
                          detrend_only=detrend_only)

[ethanbb]

Anyway I am more concerned about the noise creeping into C than S (I'm not using S)

[ethanbb]

I tried doing some hacky detrending on individual neuron footprints before CNMF. But it didn't actually change much about the statistics of C and S. Here is the second example neuron from above, with this extra detrending step:

Also, doing this causes the later dF/F calculation to be inaccurate (huge values in the top right panel are due to dividing by almost-zero); I suppose this could be worked around, though.

My hunch is that this has more to do with changes in variance over the course of the recording than baseline drift. As I understand, if parts of the recording have SD above the threshold estimated by the PSD and input to OASIS (whether through changes in noise, or, I suppose, a highly active period of the neuron), the OASIS algorithm will keep adding spikes to keep the residual SD under the threshold. (However, I see in the OASIS paper that it is not designed to deal with any non-constant baseline (it's proposed as a future direction), so that may be necessary to consider as well.)

I think my next move will be either to manually increase the noise SD and see what that does (I guess by adding another "fudge factor") or learn about/try the alternative deconvolution methods.

[ethanbb]

OK, I found a better solution for the denoised trace; I think the issue was really a wandering baseline rather than anything to do with noise variance. I am just high-pass filtering the whole movie with a cutoff of about 0.01 Hz before CNMF. Maybe worth adding as a preprocessing option?

Here is what the first example neuron from the question looks like with this preprocessing step (dF/F still has to be fixed):

[ethanbb]

I'd still like to add high-pass filtering as a feature for caiman as I imagine it'd be useful to others. I was trying to add it as an option for the preprocessing step, but I just realized this may be a bad idea as it violates the assumption that the results are a decomposition of the passed-in data in a certain way (so, for example, if you then passed the non-filtered data to evaluate_components, you could get surprising results).

Now I'm thinking I could just add it as a standalone function, I'm not sure where though. Maybe in movies.py? I wouldn't necessarily load the mmap containing Y as a movie, but I guess it would help in case you're feeding data from another format into CNMF. Does this make sense to you @pgunn?