Hardware recommendations for running CaImAn on 2P datasets

[annegeorge12]

I’m planning to purchase a new workstation and would appreciate guidance on hardware specifications that work well for running CaImAn efficiently.

Typical use case:
-Two-photon calcium imaging data
-10k-30k frames per movie
-Large FOV (≈1024 × 1024)
-Primary use: CNMF-based source extraction and standard downstream analysis

Looking for guidance on:
-CPU vs GPU
-Recommended RAM (e.g., 64 vs 128 GB)
-Number of CPU cores / clock speed tradeoffs
-Storage considerations (SSD vs NVMe, I/O bottlenecks)

Thank you so much for your help!

[pgunn]

Hi,
It's hard for me to guarantee that large FOVs will perform well with any given movie format and hardware; if at all possible you'll want to see if you can find someone at your institution you can try these things out on. A few smaller points:

• Right now GPUs are more of a nice-to-have than necessary
• Our strongest recommendation on RAM is to never go under 32G, but having more RAM (and CPUs) will let you scale out to more CPUs
• It's rare that storage speed has been the bottleneck with Caiman, in my experience supporting people. You probably don't need to focus too much on that angle (a large slow shared filesystem may struggle, in theory, but I haven't seen that in practice)
• As for number of cores, that's hard to say, but for a workstation or server-class CPU you're probably going to do fine with most mid-range configs; Caiman's parallelism also only scales well up to a certain point and would need a redesign (maybe around MapReduce principles) to scale higher

Still, actually figuring out RAM and cores is something better done empirically if at all possible (n_processes in the clustering calls, maybe a few other things, are what you'll want to try if you can borrow such hardware for a bit).

Wishing I could give more concrete advice.

[pgunn]

@ethanbb If you have any thoughts to add..

[ethanbb]

• I would definitely go with 128 GB RAM if not more, this will usually be your limiting factor even if it's still possible to process with less RAM (you would only run into diminishing returns once you get to ~2-3x the size of your largest movie).
• More cores is helpful too, I have 56 on my main machine (and 512 GB RAM). You don't have to go that high, but I think the way to think about this is if you divide your RAM by the number of cores, is that still a significant chunk of memory? Once it gets small enough, adding more cores doesn't help as much due to overhead. So if you had more like 128 GB RAM, 16 or 32 cores is probably enough.
• A GPU can be useful for visualizing results (particularly with mesmerize-core), but you don't need a top-of-the line one.
• Can confirm storage speed is not much of an issue; I store all my data including intermediate results on a filesystem mounted from a NAS and it's not a problem. Although probably if you have less RAM, storage speed becomes more important because you will have to be loading and saving smaller chunks more frequently.