how much memory is needed to segment a 5GB dataset

[tatianawoller]

Hi,

My colleague and I tried to run napari nninteractive on a small dataset and a larger EM dataset (5GB) on a linux workstation. We managed to segment 12 objects for the small dataset (100Mb, also an EM dataset) and save the segmentation. Then we tested it on the larger EM dataset and it crashed with the following allocation error (178 GB should be allocated).

(nnInteractive) u0163347@gbw-d-l0172:~$ napari
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
Fetching 4 files: 100%|████████████████████████| 4/4 [00:00<00:00, 75573.05it/s]
Using Model nnInteractive_v1.0 at : /home/luna.kuleuven.be/u0163347/.cache/huggingface/hub/models--nnInteractive--nnInteractive/snapshots/5c6d2ec824c5f15fce4ee71b1a6d358dc82a0d69/nnInteractive_v1.0
nnUNet_raw is not defined and nnU-Net can only be used on data for which preprocessed files are already present on your system. nnU-Net cannot be used for experiment planning and preprocessing like this. If this is not intended, please read documentation/setting_up_paths.md for information on how to set this up properly.
nnUNet_preprocessed is not defined and nnU-Net can not be used for preprocessing or training. If this is not intended, please read documentation/setting_up_paths.md for information on how to set this up.
nnUNet_results is not defined and nnU-Net cannot be used for training or inference. If this is not intended behavior, please read documentation/setting_up_paths.md for information on how to set this up.
/opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/numba/np/ufunc/parallel.py:371: NumbaWarning: The TBB threading layer requires TBB version 2021 update 6 or later i.e., TBB_INTERFACE_VERSION >= 12060. Found TBB_INTERFACE_VERSION = 12050. The TBB threading layer is disabled.!
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File /opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/vispy/app/backends/_qt.py:509, in QtBaseCanvasBackend.mouseReleaseEvent(self=<vispy.app.backends._qt.CanvasBackendDesktop object>, ev=<PyQt5.QtGui.QMouseEvent object>)
    507 if self._vispy_canvas is None:
    508     return
--> 509 vispy_event = self._vispy_mouse_release(
        self = <vispy.app.backends._qt.CanvasBackendDesktop object at 0x794ab7d90950>
        ev = <PyQt5.QtGui.QMouseEvent object at 0x794a938736e0>
        BUTTONMAP = {0: 0, 1: 1, 2: 2, 4: 3, 8: 4, 16: 5}    510     native=ev,
    511     pos=_get_event_xy(ev),
    512     button=BUTTONMAP[ev.button()],
    513     modifiers=self._modifiers(ev),
    514 )
    515 # If vispy did not handle the event, clear the accept parameter of the qt event
    516 if not vispy_event.handled:

File /opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/vispy/app/base.py:224, in BaseCanvasBackend._vispy_mouse_release(self=<vispy.app.backends._qt.CanvasBackendDesktop object>, **kwargs={'button': 1, 'buttons': [1], 'last_event': <MouseEvent blocked=False button=1 buttons=[1] d...urces=[] time=1753195358.5667675 type=mouse_move>, 'last_mouse_press': None, 'modifiers': (), 'native': <PyQt5.QtGui.QMouseEvent object>, 'pos': (612, 518), 'press_event': <MouseEvent blocked=False button=1 buttons=[1] d...rces=[] time=1753195357.3946142 type=mouse_press>})
    220 def _vispy_mouse_release(self, **kwargs):
    221     # default method for delivering mouse release events to the canvas
    222     kwargs.update(self._vispy_mouse_data)
--> 224     ev = self._vispy_canvas.events.mouse_release(**kwargs)
        self._vispy_canvas.events.mouse_release = <vispy.util.event.EventEmitter object at 0x794ab7d4d820>
        kwargs = {'native': <PyQt5.QtGui.QMouseEvent object at 0x794a938736e0>, 'pos': (612, 518), 'button': 1, 'modifiers': (), 'buttons': [1], 'press_event': <MouseEvent blocked=False button=1 buttons=[1] delta=[0. 0.] handled=False is_dragging=False last_event=MouseEvent modifiers=() native=<PyQt5.QtGui.QMouseEvent object at 0x794a938736e0> pos=[689 390] press_event=None source=None sources=[] time=1753195357.3946142 type=mouse_press>, 'last_event': <MouseEvent blocked=False button=1 buttons=[1] delta=[0. 0.] handled=False is_dragging=True last_event=MouseEvent modifiers=() native=<PyQt5.QtGui.QMouseEvent object at 0x794a938736e0> pos=[612 518] press_event=MouseEvent source=None sources=[] time=1753195358.5667675 type=mouse_move>, 'last_mouse_press': None}
        self = <vispy.app.backends._qt.CanvasBackendDesktop object at 0x794ab7d90950>
        self._vispy_canvas.events = <vispy.util.event.EmitterGroup object at 0x794ab98adfd0>
        self._vispy_canvas = <NapariSceneCanvas (PyQt5) at 0x794ab7d42bd0>    225     if (self._vispy_mouse_data['press_event']
    226             and self._vispy_mouse_data['press_event'].button == ev.button):
    227         self._vispy_mouse_data['press_event'] = None

File /opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/vispy/util/event.py:453, in EventEmitter.__call__(self=<vispy.util.event.EventEmitter object>, *args=(), **kwargs={'button': 1, 'buttons': [1], 'last_event': <MouseEvent blocked=False button=1 buttons=[1] d...urces=[] time=1753195358.5667675 type=mouse_move>, 'last_mouse_press': None, 'modifiers': (), 'native': <PyQt5.QtGui.QMouseEvent object>, 'pos': (612, 518), 'press_event': <MouseEvent blocked=False button=1 buttons=[1] d...rces=[] time=1753195357.3946142 type=mouse_press>})
    450 if self._emitting > 1:
    451     raise RuntimeError('EventEmitter loop detected!')
--> 453 self._invoke_callback(cb, event)
        event = <MouseEvent blocked=False button=1 buttons=[1] delta=[0. 0.] handled=False is_dragging=True last_event=MouseEvent modifiers=() native=<PyQt5.QtGui.QMouseEvent object at 0x794a938736e0> pos=[612 518] press_event=MouseEvent source=None sources=[] time=1753195358.7034779 type=mouse_release>
        self = <vispy.util.event.EventEmitter object at 0x794ab7d4d820>
        cb = <bound method VispyCanvas._on_mouse_release of <napari._vispy.canvas.VispyCanvas object at 0x794ab7d42a80>>    454 if event.blocked:
    455     break

File /opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/vispy/util/event.py:471, in EventEmitter._invoke_callback(self=<vispy.util.event.EventEmitter object>, cb=<bound method VispyCanvas._on_mouse_release of <napari._vispy.canvas.VispyCanvas object>>, event=<MouseEvent blocked=False button=1 buttons=[1] d...es=[] time=1753195358.7034779 type=mouse_release>)
    469     cb(event)
    470 except Exception:
--> 471     _handle_exception(self.ignore_callback_errors,
        self = <vispy.util.event.EventEmitter object at 0x794ab7d4d820>
        cb = <bound method VispyCanvas._on_mouse_release of <napari._vispy.canvas.VispyCanvas object at 0x794ab7d42a80>>
        event = <MouseEvent blocked=False button=1 buttons=[1] delta=[0. 0.] handled=False is_dragging=True last_event=MouseEvent modifiers=() native=<PyQt5.QtGui.QMouseEvent object at 0x794a938736e0> pos=[612 518] press_event=MouseEvent source=None sources=[] time=1753195358.7034779 type=mouse_release>
        (cb, event) = (<bound method VispyCanvas._on_mouse_release of <napari._vispy.canvas.VispyCanvas object at 0x794ab7d42a80>>, <MouseEvent blocked=False button=1 buttons=[1] delta=[0. 0.] handled=False is_dragging=True last_event=MouseEvent modifiers=() native=<PyQt5.QtGui.QMouseEvent object at 0x794a938736e0> pos=[612 518] press_event=MouseEvent source=None sources=[] time=1753195358.7034779 type=mouse_release>)    472                       self.print_callback_errors,
    473                       self, cb_event=(cb, event))

File /opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/vispy/util/event.py:469, in EventEmitter._invoke_callback(self=<vispy.util.event.EventEmitter object>, cb=<bound method VispyCanvas._on_mouse_release of <napari._vispy.canvas.VispyCanvas object>>, event=<MouseEvent blocked=False button=1 buttons=[1] d...es=[] time=1753195358.7034779 type=mouse_release>)
    467 def _invoke_callback(self, cb, event):
    468     try:
--> 469         cb(event)
        cb = <bound method VispyCanvas._on_mouse_release of <napari._vispy.canvas.VispyCanvas object at 0x794ab7d42a80>>
        event = <MouseEvent blocked=False button=1 buttons=[1] delta=[0. 0.] handled=False is_dragging=True last_event=MouseEvent modifiers=() native=<PyQt5.QtGui.QMouseEvent object at 0x794a938736e0> pos=[612 518] press_event=MouseEvent source=None sources=[] time=1753195358.7034779 type=mouse_release>    470     except Exception:
    471         _handle_exception(self.ignore_callback_errors,
    472                           self.print_callback_errors,
    473                           self, cb_event=(cb, event))

File /opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/napari/_vispy/canvas.py:557, in VispyCanvas._on_mouse_release(self=<napari._vispy.canvas.VispyCanvas object>, event=<MouseEvent blocked=False button=1 buttons=[1] d...es=[] time=1753195358.7034779 type=mouse_release>)
    545 def _on_mouse_release(self, event: MouseEvent) -> None:
    546     """Called whenever mouse released in canvas.
    547 
    548     Parameters
   (...)    555     None
    556     """
--> 557     self._process_mouse_event(mouse_release_callbacks, event)
        event = <MouseEvent blocked=False button=1 buttons=[1] delta=[0. 0.] handled=False is_dragging=True last_event=MouseEvent modifiers=() native=<PyQt5.QtGui.QMouseEvent object at 0x794a938736e0> pos=[612 518] press_event=MouseEvent source=None sources=[] time=1753195358.7034779 type=mouse_release>
        self = <napari._vispy.canvas.VispyCanvas object at 0x794ab7d42a80>
File /opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/napari/_vispy/canvas.py:488, in VispyCanvas._process_mouse_event(self=<napari._vispy.canvas.VispyCanvas object>, mouse_callbacks=<function mouse_release_callbacks>, event=<MouseEvent blocked=False button=1 buttons=[1] d...es=[] time=1753195358.7034779 type=mouse_release>)
    486 layer = self.viewer.layers.selection.active
    487 if layer is not None:
--> 488     mouse_callbacks(layer, read_only_event)
        read_only_event = <ReadOnlyWrapper at 0x79460270d000 for NapariMouseEvent at 0x79460c1a2870>
        layer = <ScribbleLayer layer 'nnInteractive - Scribble Layer' at 0x7947b8e7f230>
        mouse_callbacks = <function mouse_release_callbacks at 0x794abc8f9760>    490 event.handled = napari_event.handled

File /opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/napari/utils/interactions.py:220, in mouse_release_callbacks(obj=<ScribbleLayer layer 'nnInteractive - Scribble Layer'>, event=<ReadOnlyWrapper at 0x79460270d000 for NapariMouseEvent>)
    217 obj._persisted_mouse_event[gen].__wrapped__ = event
    218 with contextlib.suppress(StopIteration):
    219     # Run last part of the function to trigger release event
--> 220     next(gen)
        gen = <generator object draw at 0x79460c381a40>    221 # Finally delete the generator and stored event
    222 del obj._mouse_drag_gen[func]

File /opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/napari/layers/labels/_labels_mouse_bindings.py:37, in draw(layer=<ScribbleLayer layer 'nnInteractive - Scribble Layer'>, event=<ReadOnlyWrapper at 0x79460c1dae80 for ReadOnlyWrapper>)
     34     new_label = layer.selected_label
     36 # on press
---> 37 with layer.block_history():
        layer = <ScribbleLayer layer 'nnInteractive - Scribble Layer' at 0x7947b8e7f230>     38     layer._draw(new_label, coordinates, coordinates)
     39     yield

File /opt/miniforge3/envs/nnInteractive/lib/python3.12/contextlib.py:144, in _GeneratorContextManager.__exit__(self=<contextlib._GeneratorContextManager object>, typ=None, value=None, traceback=None)
    142 if typ is None:
    143     try:
--> 144         next(self.gen)
        self.gen = <generator object Labels.block_history at 0x79460c1b6810>
        self = <contextlib._GeneratorContextManager object at 0x79460c1a1310>    145     except StopIteration:
    146         return False

File /opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/napari/layers/labels/labels.py:1017, in Labels.block_history(self=<ScribbleLayer layer 'nnInteractive - Scribble Layer'>)
   1015 try:
   1016     yield
-> 1017     self._commit_staged_history()
        self = <ScribbleLayer layer 'nnInteractive - Scribble Layer' at 0x7947b8e7f230>   1018 finally:
   1019     self._block_history = prev

File /opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/napari_nninteractive/layers/scribble_layer.py:65, in ScribbleLayer._commit_staged_history(self=<ScribbleLayer layer 'nnInteractive - Scribble Layer'>)
     63 super()._commit_staged_history()
     64 self._is_free = False
---> 65 self.events.finished(action=ActionType.ADDED, value="ADD")
        self = <ScribbleLayer layer 'nnInteractive - Scribble Layer' at 0x7947b8e7f230>
        self.events.finished = <napari.utils.events.event.EventEmitter object at 0x79465d117500>
        self.events = <napari.utils.events.event.EmitterGroup object at 0x7946553f6360>
        ActionType.ADDED = <ActionType.ADDED: 'added'>
        ActionType = <enum 'ActionType'>
File /opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/napari/utils/events/event.py:753, in EventEmitter.__call__(self=<napari.utils.events.event.EventEmitter object>, *args=(), **kwargs={'action': <ActionType.ADDED: 'added'>, 'value': 'ADD'})
    750     self._block_counter.update([cb])
    751     continue
--> 753 self._invoke_callback(cb, event if pass_event else None)
        event = <Event blocked=False handled=False native=None source=None sources=[] type='finished'>
        self = <napari.utils.events.event.EventEmitter object at 0x79465d117500>
        cb = <bound method LayerControls.on_interaction of <napari_nninteractive.widget_main.nnInteractiveWidget object at 0x7949c6a00950>>
        pass_event = True    754 if event.blocked:
    755     break

File /opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/napari/utils/events/event.py:791, in EventEmitter._invoke_callback(self=<napari.utils.events.event.EventEmitter object>, cb=<bound method LayerControls.on_interaction of <n...eractive.widget_main.nnInteractiveWidget object>>, event=<Event blocked=False handled=False native=None source=None sources=[] type='finished'>)
    789     self.disconnect(cb)
    790     return
--> 791 _handle_exception(
        self = <napari.utils.events.event.EventEmitter object at 0x79465d117500>
        event = <Event blocked=False handled=False native=None source=None sources=[] type='finished'>
        cb = <bound method LayerControls.on_interaction of <napari_nninteractive.widget_main.nnInteractiveWidget object at 0x7949c6a00950>>
        (cb, event) = (<bound method LayerControls.on_interaction of <napari_nninteractive.widget_main.nnInteractiveWidget object at 0x7949c6a00950>>, <Event blocked=False handled=False native=None source=None sources=[] type='finished'>)    792     self.ignore_callback_errors,
    793     self.print_callback_errors,
    794     self,
    795     cb_event=(cb, event),
    796 )

File /opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/napari/utils/events/event.py:778, in EventEmitter._invoke_callback(self=<napari.utils.events.event.EventEmitter object>, cb=<bound method LayerControls.on_interaction of <n...eractive.widget_main.nnInteractiveWidget object>>, event=<Event blocked=False handled=False native=None source=None sources=[] type='finished'>)
    776 try:
    777     if event is not None:
--> 778         cb(event)
        event = <Event blocked=False handled=False native=None source=None sources=[] type='finished'>
        cb = <bound method LayerControls.on_interaction of <napari_nninteractive.widget_main.nnInteractiveWidget object at 0x7949c6a00950>>    779     else:
    780         cb()

File /opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/napari_nninteractive/widget_controls.py:434, in LayerControls.on_interaction(self=<napari_nninteractive.widget_main.nnInteractiveWidget object>, event=<Event blocked=False handled=False native=None source=None sources=[] type='finished'>)
    427 if (
    428     self.add_ckbx.isChecked()
    429     and event.action == ActionType.ADDED
    430     and not self._viewer.layers[event.source.name].is_free()
    431 ):
    432     self._viewer.layers[event.source.name].refresh()
--> 434     self.add_interaction()
        self = <napari_nninteractive.widget_main.nnInteractiveWidget object at 0x7949c6a00950>
File /opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/napari_nninteractive/widget_main.py:205, in nnInteractiveWidget.add_interaction(self=<napari_nninteractive.widget_main.nnInteractiveWidget object>)
    203     self.session.add_bbox_interaction(bbox, _prompt, _auto_run)
    204 elif _index == 2:
--> 205     self.session.add_scribble_interaction(data, _prompt, _auto_run)
        data = array([[[0, 0, ..., 0, 0],
        [0, 0, ..., 0, 0],
        ...,
        [0, 0, ..., 0, 0],
        [0, 0, ..., 0, 0]],

       [[0, 0, ..., 0, 0],
        [0, 0, ..., 0, 0],
        ...,
        [0, 0, ..., 0, 0],
        [0, 0, ..., 0, 0]],

       ...,

       [[0, 0, ..., 0, 0],
        [0, 0, ..., 0, 0],
        ...,
        [0, 0, ..., 0, 0],
        [0, 0, ..., 0, 0]],

       [[0, 0, ..., 0, 0],
        [0, 0, ..., 0, 0],
        ...,
        [0, 0, ..., 0, 0],
        [0, 0, ..., 0, 0]]], dtype=uint8)
        self = <napari_nninteractive.widget_main.nnInteractiveWidget object at 0x7949c6a00950>
        _prompt = True
        _auto_run = True
        self.session = <nnInteractive.inference.inference_session.nnInteractiveInferenceSession object at 0x794aaed1f1a0>    206 elif _index == 3:
    207     self.session.add_lasso_interaction(data, _prompt, _auto_run)

File /opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/nnInteractive/inference/inference_session.py:277, in nnInteractiveInferenceSession.add_scribble_interaction(self=<nnInteractive.inference.inference_session.nnInteractiveInferenceSession object>, scribble_image=array([[[0, 0, ..., 0, 0],
        [0, 0, ..., 0... 0, 0],
        [0, 0, ..., 0, 0]]], dtype=uint8), include_interaction=True, run_prediction=True)
    275 def add_scribble_interaction(self, scribble_image: np.ndarray,  include_interaction: bool, run_prediction: bool = True):
    276     assert all([i == j for i, j in zip(self.original_image_shape[1:], scribble_image.shape)]), f'Given scribble image must match input image shape. Input image was: {self.original_image_shape[1:]}, given: {scribble_image.shape}'
--> 277     self._finish_preprocessing_and_initialize_interactions()
        self = <nnInteractive.inference.inference_session.nnInteractiveInferenceSession object at 0x794aaed1f1a0>    279     scribble_image = torch.from_numpy(scribble_image)
    281     # crop (as in preprocessing)

File /opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/nnInteractive/inference/inference_session.py:107, in nnInteractiveInferenceSession._finish_preprocessing_and_initialize_interactions(self=<nnInteractive.inference.inference_session.nnInteractiveInferenceSession object>)
    101 """
    102 Block until both the image preprocessing and the interactions tensor initialization
    103 are finished.
    104 """
    105 if self.preprocess_future is not None:
    106     # Wait for image preprocessing to complete.
--> 107     self.preprocess_future.result()
        self.preprocess_future = <Future at 0x794655351280 state=finished raised RuntimeError>
        self = <nnInteractive.inference.inference_session.nnInteractiveInferenceSession object at 0x794aaed1f1a0>    108     del self.preprocess_future
    109     self.preprocess_future = None

File /opt/miniforge3/envs/nnInteractive/lib/python3.12/concurrent/futures/_base.py:449, in Future.result(self=None, timeout=None)
    447     raise CancelledError()
    448 elif self._state == FINISHED:
--> 449     return self.__get_result()
        self = None    451 self._condition.wait(timeout)
    453 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:

File /opt/miniforge3/envs/nnInteractive/lib/python3.12/concurrent/futures/_base.py:401, in Future.__get_result(self=None)
    399 if self._exception:
    400     try:
--> 401         raise self._exception
        self = None    402     finally:
    403         # Break a reference cycle with the exception in self._exception
    404         self = None

File /opt/miniforge3/envs/nnInteractive/lib/python3.12/concurrent/futures/thread.py:59, in _WorkItem.run(self=None)
     56     return
     58 try:
---> 59     result = self.fn(*self.args, **self.kwargs)
        self = None     60 except BaseException as exc:
     61     self.future.set_exception(exc)

File /opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/nnInteractive/inference/inference_session.py:154, in nnInteractiveInferenceSession._background_set_image(self=<nnInteractive.inference.inference_session.nnInteractiveInferenceSession object>, image=array([[[[255, 255, ..., 255, 255],
         [25...       [255, 255, ..., 255, 255]]]], dtype=uint8), image_properties={'spacing': array([1., 1., 1.])})
    152 if self.verbose:
    153     print('Cropping input image to nonzero region')
--> 154 nonzero_idx = torch.where(image_torch != 0)
        image_torch = tensor([[[[255, 255, 255,  ..., 255, 255, 255],
          [255, 255, 255,  ..., 255, 255, 255],
          [255, 255, 255,  ..., 255, 255, 255],
          ...,
          [255, 255, 255,  ..., 160, 156, 167],
          [255, 255, 255,  ..., 166, 164, 171],
          [255, 255, 255,  ..., 161, 152, 160]],

         [[255, 255, 255,  ..., 255, 255, 255],
          [255, 255, 255,  ..., 255, 255, 255],
          [255, 255, 255,  ..., 255, 255, 255],
          ...,
          [255, 255, 255,  ..., 169, 162, 158],
          [255, 255, 255,  ..., 157, 156, 164],
          [255, 255, 255,  ..., 157, 155, 165]],

         [[255, 255, 255,  ..., 255, 255, 255],
          [255, 255, 255,  ..., 255, 255, 255],
          [255, 255, 255,  ..., 255, 255, 255],
          ...,
          [255, 255, 255,  ..., 169, 164, 159],
          [255, 255, 255,  ..., 160, 159, 160],
          [255, 255, 255,  ..., 156, 155, 157]],

         ...,

         [[255, 255, 255,  ..., 255, 255, 255],
          [255, 255, 255,  ..., 255, 255, 255],
          [255, 255, 255,  ..., 255, 255, 255],
          ...,
          [255, 255, 255,  ..., 255, 255, 255],
          [255, 255, 255,  ..., 255, 255, 255],
          [255, 255, 255,  ..., 255, 255, 255]],

         [[255, 255, 255,  ..., 255, 255, 255],
          [255, 255, 255,  ..., 255, 255, 255],
          [255, 255, 255,  ..., 255, 255, 255],
          ...,
          [255, 255, 255,  ..., 255, 255, 255],
          [255, 255, 255,  ..., 255, 255, 255],
          [255, 255, 255,  ..., 255, 255, 255]],

         [[255, 255, 255,  ..., 255, 255, 255],
          [255, 255, 255,  ..., 255, 255, 255],
          [255, 255, 255,  ..., 255, 255, 255],
          ...,
          [255, 255, 255,  ..., 255, 255, 255],
          [255, 255, 255,  ..., 255, 255, 255],
          [255, 255, 255,  ..., 255, 255, 255]]]], dtype=torch.uint8)    155 # Create bounding box: for each dimension, get the min and max (plus one) of the nonzero indices.
    156 bbox = [[i.min().item(), i.max().item() + 1] for i in nonzero_idx]

RuntimeError: [enforce fail at alloc_cpu.cpp:118] err == 0. DefaultCPUAllocator: can't allocate memory: you tried to allocate 177968337600 bytes. Error code 12 (Cannot allocate memory)
^CTraceback (most recent call last):
  File "/opt/miniforge3/envs/nnInteractive/bin/napari", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/napari/__main__.py", line 583, in main
    _run()
  File "/opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/napari/__main__.py", line 402, in _run
    run(gui_exceptions=True)
  File "/opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/napari/_qt/qt_event_loop.py", line 410, in run
    _maybe_allow_interrupt(app),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniforge3/envs/nnInteractive/lib/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/opt/miniforge3/envs/nnInteractive/lib/python3.12/site-packages/napari/_qt/utils.py", line 394, in _maybe_allow_interrupt
    old_sigint_handler(*handler_args)
KeyboardInterrupt

1. Is it correct that it is an allocation error and if yes is there an upper threshold to which one can use napari nninteractive as a GUI?
2. Do you have any suggestion how to reduce the needed RAM or should we switch to command line for larger images?
3. We can share the dataset if you would like to see how reproducible the error is.
4. Can/ will napari nninteractive or nninteractive be able to do some distributed segmentation?

Thank you in advance and kind regards,

Tatiana

[Lars-Kraemer]

Hey @tatianawoller

Indeed, it looks like you're running out of RAM. While there's no strict limit, the current version of nnInteractive in Napari loads the entire image into memory, which can be a bottleneck for large data like your 5GB EM volume (especially after tensor conversion and buffering, which significantly increase memory usage).
I recommend downscaling or cropping the image to reduce its size and testing to find the point at which it fits into memory without crashing.

Let me know if this helps or if you run into any further issues.

Best,
Lars

[FabianIsensee]

Agreed. Please try nnInteractive on a smaller crop of the image. It is currently not set up to handle very large images well.
We unfortunately have not been able to train nnInteractive intensively on EM data, mainly due to a lack of datastets with ground truth that we could access. Do you happen to have ideas for other datasets that we should include in our training?
Best,
Fabian

[tatianawoller]

Hi, thank you for your answer. For EM data, I only know this dataset: https://www.ebi.ac.uk/empiar/EMPIAR-11037/

[FabianIsensee]

Unfortunately we need 3D data for training and this seems to be 2d

[lukmuk]

Hi there,

to contribute to this issue with some EM feedback as well: The current model works already quite well for my volume EM data of brain tissue (FIB-SEM tomography). I use it for neuron instance segmentation.
I binned my volume by 4x in x and y and now it is about 1.2 GB on disk. It works fast with a 5090 with 32 GB VRAM.

Edit: I tested it also on a 5 GB file and ended up at around 170-180 GB RAM usage and GPU at some point at 17 GB VRAM. That was close to the limit of the workstation (192 GB RAM, Intel 285K Core Ultra), and I had one crash. Inference was done (I think) on CPU and quite usable.

Regarding model size: I am no expert, but would it be possible to quantize the model to reduce VRAM requirements (of course at the cost of accuracy)?

Regarding datasets, this paper has a table with annotations (instance or semantic marked), maybe interesting for future model training?
https://www.sciencedirect.com/science/article/pii/S1361841523001809?via%3Dihub

Unrelated question: Would it be beneficial to set the voxel size in napari for anisotropic data (typically z > x,y in EM) for nnUnet?

Thank you for this amazing tool!

[FabianIsensee]

Hey, thanks for your input and detailed information on how you are using nnInteractive. I believe in your situation it would be beneficial to do mitigation strategies for reduced RAM requirements. Since nnInteractive can only operate on a limited field of view and will not be able to produce segmentations spanning your entire volume without you effectively chunking it through repeated interactions, the most straightforward + lowest effort approach would be to cut your volume into patches externally and then annotating these patches one at a time. There is some overhead to be resolved regarding aligning objects of interest with the patches and stitching predictions together, but this seems to be the least effort solution to your problem. With images of that size, the GUI is probably quite laggy as well!
I was considering working on an alternative implementation of the backend last week but have to prioritize other things at the moment. The idea was to use in-memory compression for the interaction tensors. They should compress very nicely (since they are sparsely populated) and thus this approach should be a lot less memory intensive. But we would be giving up some interactivity in the process because for each nnInteractive call the patch given to the model must be uncompressed. If RAM limitations are an issue for many users we can try moving up the timeline for this idea but it would still take some time + has no guarantee to work as well as we might think it will.
VRAM consumption is primarily driven by the autozoom zoom out and should in principle not change with image size. Are you making different observations?
Best,
Fabian

[FabianIsensee]

I felt like vibe-coding today so there is now a new branch of the nninteractive inference backend which uses in-memory compression for the interactions tensor: https://github.com/MIC-DKFZ/nnInteractive/tree/feature/compressed_interactions
If you install that (instead of the default) you will use A LOT less RAM but everything will also be A LOT slower
Some of this is because of the compression, but some of it is also because of 'pad design' on our part (and partly because napari wants it this way):

• we always give a full 3D image as lasso or scribble interaction even though these interactions only live in a 2D plane
• elementwise maximum for pasting new interactions requires large decompressions and compressions
• probably some more

So this is definitely labeled as experimental. Use at your own discretion!
Lars (our napari plugin expert) is currently on vacation. I will talk to him once he is back if we can streamline things some more

[FabianIsensee]

Main speed-related issue is the global interaction decay which requires uncompression and re-compression of entire interaction channels. I don't think this can be circumvented

[FabianIsensee]

codex had a great idea on how to address this. Feeling more obsolete by the day...
It's quite a lot faster now with in-memory compression. Feel free to try and give feedback.
Again ... not fully tested + may have bugs!

[lukmuk]

Hey Fabian, thanks for your replies and work!
I will test it in another Python env with the same image files and report back on RAM usage/speed when I find time.

For my use case (and the fortunate situation to have access to a 5090 GPU) the current version works already quite good as mentioned above. I think nnInteractive is very good for labeling small subvolumes (neurons, organelles) for then training models for automatic segmentation of larger volumes.

[FabianIsensee]

Any update? How is that working for you?
I have continued the implementation internally (not publicly shared yet) and save about 15GB of RAM on a 700MB nifti (700 x 400 x 3500 voxels). Will probably make this the default in the next release