Kernel keeps dying when I run cnmf_refit.estimates.evaluate_components()

[nikolaswalteros]

Hi,
I'm trying to use CaImAn to analyse neurons calcium transients, I'm very interested in the segmentation part of the program, however my kernel keeps dying when I run the:

cnmf_refit.estimates.evaluate_components(images, cnmf_refit.params, dview=cluster)

I'm not sure what to do as I cannot get more specific information, as I don't know why the kernel keeps dying (I am using jupyter notebook).
I doesn't seem to be a memory problem, at least for what I understood. I also used a cropped (and less heavy video) to test if the kernel kept dying, and it did. Also by using the windows task manager, there is no full memory usage.

I don't know if it is a problem of the memory map created, as I did not run the motion correction part, I created the memmap with

memmapped_prova = cm.save_memmap([movie_path], base_name='memmap_', order='C', border_to_0=0, dview=cluster)

Yr, dims, T = cm.load_memmap(memmapped_prova, mode='r+') #not sure if r+ is useful or not, but also without it the kernel dies
images = np.reshape(Yr.T, [T] + list(dims), order='F')

Moreover, I also tried to use the demo_pipeline.ipynb, with the provided data and parameters, I also did the motion correction and everything, however when i reach the:

cnmf_refit.estimates.evaluate_components(images, cnmf_refit.params, dview=cluster);

the kernel dies.

I am also not sure how to solve this problem. If you need more information I'll be happy to provide you with everything!
If you could help me it would be fantastic,
Thanks in advance!

Nik

[pgunn]

Hi,
Usually this is caused by the system running out of memory (which can happen quite quickly - sometimes you can catch it with the task manager, but not always). A few lines of inquiry:
A) Look into your swapping settings on windows, particularly the max swapfile size
B) Verify the notebooks work with the sample data - is your data much bigger?
C) This one may not be that useful: try using the online mode, or decrease the number of workers
D) See if we can get more logging of what's going on, possibly using the CLI demos

C normally would be my go-to, but I don't think usually components evaluation is where things go wrong if that's the constraint.

A should be easy, although I don't remember if Windows actually still limits swapfile size very significantly, and it may not fix things.

D might be a viable route; you can increase the logging level of caiman (I think most notebooks include examples of this), and if need be you could also run outside of jupyter - activate the environment from the commandline and modify one of the CLI demos to do what you want entirely outside of jupyter - that at least takes any logging complexity from jupyter out of the picture.

[nikolaswalteros]

Hi,
Sorry for the bother. I've attached the swap file data, in case it's helpful.

I am encountering repeated kernel crashes when running the cnmf_refit.estimates.evaluate_components(), even when using different videos of smaller sizes. the kernel consistently dies at this step.

To better understand the online mode, I also tried running the demo_OnACID_mesoscope notebook that was provided. However the kernel also dies there when running: cnm = cnmf.online_cnmf.OnACID(params=opts).

I tested on another computer, with more memory, but I'm seeing the same behavior. The kernel dies both in the demo pipeline and when running my own videos.

At this point I'm not sure what else I could try. Even using less demanding videos the kernel still dies.

I'm still learning, so I tried running the Python program in Visual Studio Code to see if Jupyter was part of the issue. I still encountered problems there as well...

If there's any additional information I can provide that would help diagnose this, please let me know. I'd really appreciate some guidance on how to proceed.

Thank you very much in advance!

Nikolas

swapfile.pdf

[pgunn]

Ah, if the kernel dies with the unmodified demos and their datasets, that helps us rule out it being particular to your data.

How much RAM do these systems have? Generally 32G should be enough to make it through the notebooks; 64G is generally a good idea if you can afford it (some people have larger for very large datasets). You're not using remote kernels or anything like that, are you?

You might also try the CLI demo that's closest to the notebook you're using, mostly because you're more likely to get a good error message out of it.

(We could set up a videoconference to try things together if need be, I'd just like to try a bit more back and forth on the issue first)

[nikolaswalteros]

The computer I am currently using has a RAM of 32 GB, moreover I'm using the Jupyter notebook launching it from the cmd on windows, so I'm pretty sure that it is not a remote kernel.

I used the CLI demo on my cmd, on the same machine and still it seems to crash when it reaches the cnm2.estimates.evaluate_components(images, cnm2.params, dview=dview) line.
(I ran the demo_pipeline.py)

This is the message I get:

C:\Users\fkt27ez\AppData\Local\miniconda3\envs\caiman\lib\site-packages\caiman\source_extraction\cnmf\deconvolution.py:1004: FutureWarning: Beginning in SciPy 1.17, multidimensional input will be treated as a batch, not raveled. To preserve the existing behavior and silence this warning, ravel arguments before passing them to toeplitz.
A = scipy.linalg.toeplitz(xc[lags + np.arange(lags)],
[temporal.py: update_iteration():380] 50 out of total 149 temporal components updated
[temporal.py: update_iteration():380] 88 out of total 149 temporal components updated
[temporal.py: update_iteration():380] 122 out of total 149 temporal components updated
[temporal.py: update_iteration():380] 143 out of total 149 temporal components updated
[temporal.py: update_iteration():380] 149 out of total 149 temporal components updated
[temporal.py: update_iteration():380] 50 out of total 149 temporal components updated
[temporal.py: update_iteration():380] 88 out of total 149 temporal components updated
[temporal.py: update_iteration():380] 122 out of total 149 temporal components updated
[temporal.py: update_iteration():380] 143 out of total 149 temporal components updated
[temporal.py: update_iteration():380] 149 out of total 149 temporal components updated
[components_evaluation.py:estimate_components_quality():791] Component evaluation in parallel
Component 20 is only active jointly with neighboring components. Space correlation calculation might be unreliable.
Component 23 is only active jointly with neighboring components. Space correlation calculation might be unreliable.
Component 40 is only active jointly with neighboring components. Space correlation calculation might be unreliable.
Component 49 is only active jointly with neighboring components. Space correlation calculation might be unreliable.
GPU run not requested, disabling use of GPUs
[components_evaluation.py:evaluate_components_CNN():280] Using Keras
USING MODEL (keras API): C:\Users\fkt27ez\caiman_data\model\cnn_model.json
2025-11-25 15:37:18.072991: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

Then the program stops, I think in the same step as in the jupyter notebook.

(Regarding the video conference, I’ll be available if you think it might be necessary)

Thank you again for your time!

[pgunn]

I wonder if this is the same issue as #1536 -- could you see if this particular trigger has the same effect for you?

#1536 (comment)

[nikolaswalteros]

I saw the issue and I think the problem might be similar, and I tried the things you were discussing there.
First, if I try to run this cell:

from tensorflow import keras
import os
import json

base = r'C:/Users/fkt27ez/caiman_data/model'

--- JSON Architecture ---

with open(os.path.join(base, 'cnn_model.json'), 'r') as f:
json_str = f.read()

try:
model = keras.models.model_from_json(json_str)
print('architecture ok')
except Exception as e:
import traceback
traceback.print_exc()

--- H5 weights ---

try:
model.load_weights(os.path.join(base, 'cnn_model.h5'))
print('weight loaded ok')
except Exception as e:
import traceback
traceback.print_exc()

model.summary()

I get:

architecture ok
weight loaded ok
Model: "sequential"

---

Layer (type) Output Shape Param #

conv2d_20 (Conv2D) (None, 48, 48, 32) 320

activation_13 (Activation) (None, 48, 48, 32) 0

conv2d_21 (Conv2D) (None, 46, 46, 32) 9248

activation_14 (Activation) (None, 46, 46, 32) 0

max_pooling2d_10 (MaxPoolin (None, 23, 23, 32) 0
g2D)

dropout_17 (Dropout) (None, 23, 23, 32) 0

conv2d_22 (Conv2D) (None, 23, 23, 64) 18496

activation_15 (Activation) (None, 23, 23, 64) 0

conv2d_23 (Conv2D) (None, 21, 21, 64) 36928

activation_16 (Activation) (None, 21, 21, 64) 0

max_pooling2d_11 (MaxPoolin (None, 10, 10, 64) 0
g2D)

dropout_18 (Dropout) (None, 10, 10, 64) 0

flatten_8 (Flatten) (None, 6400) 0

dense_15 (Dense) (None, 512) 3277312

activation_17 (Activation) (None, 512) 0

dropout_19 (Dropout) (None, 512) 0

dense_16 (Dense) (None, 2) 1026

activation_18 (Activation) (None, 2) 0

=================================================================
Total params: 3,343,330
Trainable params: 3,343,330
Non-trainable params: 0

Then, the kernel doesn't die instantly but when I run the cell with all the imports, It dies.

this is the import cell:

import bokeh.plotting as bpl
import cv2
import datetime
import glob
import holoviews as hv
from IPython import get_ipython
import logging
import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd
import psutil
from pathlib import Path

try:
cv2.setNumThreads(0)
except():
pass

try:
if IPYTHON:
get_ipython().run_line_magic('load_ext', 'autoreload')
get_ipython().run_line_magic('autoreload', '2')
except NameError:
pass

import caiman as cm
from caiman.motion_correction import MotionCorrect
from caiman.source_extraction.cnmf import cnmf, params
from caiman.utils.utils import download_demo
from caiman.utils.visualization import plot_contours, nb_view_patches, nb_plot_contour
from caiman.utils.visualization import nb_view_quilt

bpl.output_notebook()
hv.notebook_extension('bokeh')

Here the images also, for clarity

Moreover in the terminal I get:

[I 2025-11-26 11:22:48.312 ServerApp] Connecting to kernel 815b9d96-4a8c-46a3-9bcb-bfe8ec1a0fac.
[I 2025-11-26 11:22:48.312 ServerApp] Restoring connection for 815b9d96-4a8c-46a3-9bcb-bfe8ec1a0fac:eee8f0d5-ffe2-4d94-9d1d-92ce6f08bb47
2025-11-26 11:23:01.780433: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-26 11:23:01.781500: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
OMP: Error #15: Initializing libomp.dll, but found libiomp5 already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://openmp.llvm.org/
[I 2025-11-26 11:23:39.295 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), keep random ports
[W 2025-11-26 11:23:39.295 ServerApp] kernel 815b9d96-4a8c-46a3-9bcb-bfe8ec1a0fac restarted
[I 2025-11-26 11:23:39.299 ServerApp] Starting buffering for 815b9d96-4a8c-46a3-9bcb-bfe8ec1a0fac:eee8f0d5-ffe2-4d94-9d1d-92ce6f08bb47
[I 2025-11-26 11:23:39.312 ServerApp] Connecting to kernel 815b9d96-4a8c-46a3-9bcb-bfe8ec1a0fac.
[I 2025-11-26 11:23:39.312 ServerApp] Restoring connection for 815b9d96-4a8c-46a3-9bcb-bfe8ec1a0fac:eee8f0d5-ffe2-4d94-9d1d-92ce6f08bb47

Now that I see this I can try to ''ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library''.

However I'm not sure of how to do it.

[nikolaswalteros]

If I put:

pythonimport os
os.environ['KMP_DUPLICATE_LIB_OK'] = 'TRUE'

from tensorflow import keras
import json

everything works!! (at least it seems to)
the cnmf_refit.estimates.evaluate_components(images, cnmf_refit.params, dview=cluster); works!

However I don't know if it is the best and most reliable solution.

let me know what you think!

[pgunn]

Ah! That warning (and workaround) are a great start to understanding the problem; so long as Caiman can run it's probably fine, and I'll have to look into how this kind of thing might happen. Most likely it's a packaging issue.

If you can pastebin me the output of conda list with your caiman environment listed, that may give us a hint.

[nikolaswalteros]

sure!

This is the conda list in my caiman env:

DafpDskq.txt

I don't know if it can be helpful/useful, but I'll send you also the conda list output in the base environment:

efGJxCNW.txt

[pgunn]

Ah, I wonder if it's the tensorflow you brought in with pip that brought in the extra OpenMP. If it were Linux we could dig in with ldd and a few other runtime linker-type tools, but I don't know if Windows has an equivalent.

For now use the workaround it suggests so long as it works. The next caiman release will have a relatively simple install route that won't require pip, and I'm almost done testing it.

[nikolaswalteros]

Perfect!

Thank you!

[pgunn]

It may be worth trying this again with the latest release, which swaps tensorflow out.