Fail to load or create a SortingAnalyzer

[vncntprvst]

I ran the ephys-pipeline, using KS4, mostly successfully (see #36).
However, I've had trouble loading or creating a SortingAnalyzer with the results.
I tried different approaches:

sorting_analyzer = si.load_sorting_analyzer("<data_path>/sorting_output/postprocessed/experiment1_Record Node 101#Neuropix-PXI-100.ProbeA_recording1.zarr", with_recording=False)

This fails with: TypeError: load_sorting_analyzer() got an unexpected keyword argument 'with_recording'.
This works initially:

sorting_analyzer = si.load_sorting_analyzer("<data_path>/sorting_output/postprocessed/experiment1_Record Node 101#Neuropix-PXI-100.ProbeA_recording1.zarr")

But then that fails:

sorting_analyzer.compute([
    "random_spikes",
    "templates",
    "template_similarity",
    "unit_locations",
    "spike_amplitudes",
    "correlograms"
])

AssertionError: Extension templates requires the recording.

Trying to create a sorting analyzer also fails

sorting_analyzer = create_sorting_analyzer(sorting=sorting_curated , recording=preproc_recording)

I got various error type when trying to create either the sorting, the recording, or the sorting_analyzer object, such as:

FileNotFoundError: [Errno 2] No such file or directory: '<data_path>/Record Node 101/experiment1/recording1/continuous/Neuropix-PXI-100.ProbeA/timestamps.npy'
...
UnboundLocalError: local variable 'si_file' referenced before assignment
...
AssertionError: Extension templates requires the recording
...
AttributeError: 'NoneType' object has no attribute 'get_data'

Additional documentation on using the SortingAnalyzer to load and visualize outputs would be appreciated.
Thanks!

[alejoe91]

Hi @vncntprvst

Thanks for the feedback.

sorting_analyzer = si.load_sorting_analyzer("<data_path>/sorting_output/postprocessed/experiment1_Record Node 101#Neuropix-PXI-100.ProbeA_recording1.zarr", with_recording=False)

with_recordng is deprecated and is not an argument. The analyzer should automatically find the recording (if it hasn't been moved), but since the pipeline does some re-organization of the output at the end, I might have remapped it wrongly. I'll check on my end. What you can do is:

sorting_analyzer = si.load_sorting_analyzer("<data_path>/sorting_output/postprocessed/experiment1_Record Node 101#Neuropix-PXI-100.ProbeA_recording1.zarr")
recording_preproc = si.load("<data_path>/sorting_output/preprocessed/experiment1_Record Node 101#Neuropix-PXI-100.ProbeA_recording1.json", bade_folder="<data_path>/sorting_output")

sorting_analyzer.set_temporary_recording(recording_preproc)

sorting_analyzer.compute([
    "random_spikes",
    "templates",
    "template_similarity",
    "unit_locations",
    "spike_amplitudes",
    "correlograms"
])

Can I ask you why you are recomputing extensions? All available extensions are already computed and ready to use!

[vncntprvst]

I had tried with recording first because it was the in the pipeline's docs ;)

The recording object is the one creating the most issues here.

recording_preproc = si.load("<data_path>/sorting_output/preprocessed/experiment1_Record Node 101#Neuropix-PXI-100.ProbeA_recording1.json", base_folder="<data_path>")

returns
FileNotFoundError: [Errno 2] No such file or directory: '<data_path>/Record Node 101/experiment1/recording1/continuous/Neuropix-PXI-100.ProbeA/timestamps.npy', while the parent directory to that timestamps file exists (but is empty).
Same with:

recording_preproc = si.load("<data_path>/sorting_output/preprocessed/experiment1_Record Node 101#Neuropix-PXI-100.ProbeA_recording1.json", base_folder="<data_path>/sorting_output")

The only reason why I recomputed extension is because it was in my code :) I thought it wouldn't hurt to do so.

[alejoe91]

Mmm...strange that the timestamps.npy is not there, since it's produced by the OpenEphys. Is the continuous.dat file there?

[vncntprvst]

I just noticed that the path in the FileNotFoundError: [Errno 2] No such file or directory: '<data_path>/Record Node 101/experiment1/recording1/continuous/Neuropix-PXI-100.ProbeA/timestamps.npy' is actually a different path (a recording in a different subject).
However, the path in recording_preproc = si.load("<data_path>/sorting_output/preprocessed/experiment1_Record Node 101#Neuropix-PXI-100.ProbeA_recording1.json", base_folder="<data_path>") is correct. No idea what happened here.
Full output:

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[3], line 2
      1 # %%
----> 2 recording_preproc = si.load("/orcd/scratch/bcs/002/Vincent/LO_03/2025-04-01/LO_03_002/sorting_output/preprocessed/experiment1_Record Node 101#Neuropix-PXI-100.ProbeA_recording1.json", base_folder="/orcd/scratch/bcs/002/Vincent/LO_03/2025-04-01/LO_03_002")

File ~/.conda/envs/aind-ephys-pipeline/lib/python3.10/site-packages/spikeinterface/core/loading.py:100, in load(file_or_folder_or_dict, **kwargs)
     98     if object_type is None:
     99         raise ValueError(_error_msg.format(file_path=file_path))
--> 100     return _load_object_from_dict(d, object_type, base_folder=base_folder)
    102 elif is_local and file_path.is_dir():
    104     folder = file_path

File ~/.conda/envs/aind-ephys-pipeline/lib/python3.10/site-packages/spikeinterface/core/loading.py:175, in _load_object_from_dict(d, object_type, base_folder)
    173 def _load_object_from_dict(d, object_type, base_folder=None):
    174     if object_type in ("Recording", "Sorting", "Recording|Sorting"):
--> 175         return BaseExtractor.from_dict(d, base_folder=base_folder)
    177     elif object_type == "Templates":
    178         from spikeinterface.core import Templates

File ~/.conda/envs/aind-ephys-pipeline/lib/python3.10/site-packages/spikeinterface/core/base.py:571, in BaseExtractor.from_dict(dictionary, base_folder)
    569     assert base_folder is not None, "When  relative_paths=True, need to provide base_folder"
    570     dictionary = make_paths_absolute(dictionary, base_folder)
--> 571 extractor = _load_extractor_from_dict(dictionary)
    572 folder_metadata = dictionary.get("folder_metadata", None)
    573 if folder_metadata is not None:

File ~/.conda/envs/aind-ephys-pipeline/lib/python3.10/site-packages/spikeinterface/core/base.py:1108, in _load_extractor_from_dict(dic)
   1106 for name, value in dic["kwargs"].items():
   1107     if is_dict_extractor(value):
-> 1108         new_kwargs[name] = _load_extractor_from_dict(value)
   1109     elif isinstance(value, dict):
   1110         new_kwargs[name] = {k: transform_dict_to_extractor(v) for k, v in value.items()}

File ~/.conda/envs/aind-ephys-pipeline/lib/python3.10/site-packages/spikeinterface/core/base.py:1108, in _load_extractor_from_dict(dic)
   1106 for name, value in dic["kwargs"].items():
   1107     if is_dict_extractor(value):
-> 1108         new_kwargs[name] = _load_extractor_from_dict(value)
   1109     elif isinstance(value, dict):
   1110         new_kwargs[name] = {k: transform_dict_to_extractor(v) for k, v in value.items()}

    [... skipping similar frames: _load_extractor_from_dict at line 1108 (3 times)]

File ~/.conda/envs/aind-ephys-pipeline/lib/python3.10/site-packages/spikeinterface/core/base.py:1108, in _load_extractor_from_dict(dic)
   1106 for name, value in dic["kwargs"].items():
   1107     if is_dict_extractor(value):
-> 1108         new_kwargs[name] = _load_extractor_from_dict(value)
   1109     elif isinstance(value, dict):
   1110         new_kwargs[name] = {k: transform_dict_to_extractor(v) for k, v in value.items()}

File ~/.conda/envs/aind-ephys-pipeline/lib/python3.10/site-packages/spikeinterface/core/base.py:1127, in _load_extractor_from_dict(dic)
   1121     warnings.warn(
   1122         f"Versions are not the same. This might lead to compatibility errors. "
   1123         f"Using {class_name.split('.')[0]}=={dic['version']} is recommended"
   1124     )
   1126 # Initialize the extractor
-> 1127 extractor = extractor_class(**new_kwargs)
   1129 extractor._annotations.update(dic["annotations"])
   1130 for k, v in dic["properties"].items():

File ~/.conda/envs/aind-ephys-pipeline/lib/python3.10/site-packages/spikeinterface/extractors/neoextractors/openephys.py:158, in OpenEphysBinaryRecordingExtractor.__init__(self, folder_path, load_sync_channel, load_sync_timestamps, experiment_names, stream_id, stream_name, block_index, all_annotations)
    146 def __init__(
    147     self,
    148     folder_path,
   (...)
    155     all_annotations=False,
    156 ):
    157     neo_kwargs = self.map_to_neo_kwargs(folder_path, load_sync_channel, experiment_names)
--> 158     NeoBaseRecordingExtractor.__init__(
    159         self,
    160         stream_id=stream_id,
    161         stream_name=stream_name,
    162         block_index=block_index,
    163         all_annotations=all_annotations,
    164         **neo_kwargs,
    165     )
    166     # get streams to find correct probe
    167     stream_names, stream_ids = self.get_streams(folder_path, load_sync_channel, experiment_names)

File ~/.conda/envs/aind-ephys-pipeline/lib/python3.10/site-packages/spikeinterface/extractors/neoextractors/neobaseextractor.py:188, in NeoBaseRecordingExtractor.__init__(self, stream_id, stream_name, block_index, all_annotations, use_names_as_ids, **neo_kwargs)
    158 def __init__(
    159     self,
    160     stream_id: Optional[str] = None,
   (...)
    165     **neo_kwargs: Dict[str, Any],
    166 ) -> None:
    167     """
    168     Initialize a NeoBaseRecordingExtractor instance.
    169 
   (...)
    185 
    186     """
--> 188     _NeoBaseExtractor.__init__(self, block_index, **neo_kwargs)
    190     kwargs = dict(all_annotations=all_annotations)
    191     if block_index is not None:

File ~/.conda/envs/aind-ephys-pipeline/lib/python3.10/site-packages/spikeinterface/extractors/neoextractors/neobaseextractor.py:27, in _NeoBaseExtractor.__init__(self, block_index, **neo_kwargs)
     23 def __init__(self, block_index, **neo_kwargs):
     24 
     25     # Avoids double initiation of the neo reader if it was already done in the __init__ of the child class
     26     if not hasattr(self, "neo_reader"):
---> 27         self.neo_reader = self.get_neo_io_reader(self.NeoRawIOClass, **neo_kwargs)
     29     if self.neo_reader.block_count() > 1 and block_index is None:
     30         raise Exception(
     31             "This dataset is multi-block. Spikeinterface can load one block at a time. "
     32             "Use 'block_index' to select the block to be loaded."
     33         )

File ~/.conda/envs/aind-ephys-pipeline/lib/python3.10/site-packages/spikeinterface/extractors/neoextractors/neobaseextractor.py:66, in _NeoBaseExtractor.get_neo_io_reader(cls, raw_class, **neo_kwargs)
     64 neoIOclass = getattr(rawio_module, raw_class)
     65 neo_reader = neoIOclass(**neo_kwargs)
---> 66 neo_reader.parse_header()
     68 return neo_reader

File ~/.conda/envs/aind-ephys-pipeline/lib/python3.10/site-packages/neo/rawio/baserawio.py:211, in BaseRawIO.parse_header(self)
    197 """
    198 Parses the header of the file(s) to allow for faster computations
    199 for all other functions
    200 
    201 """
    202 # this must create
    203 # self.header['nb_block']
    204 # self.header['nb_segment']
   (...)
    208 # self.header['spike_channels']
    209 # self.header['event_channels']
--> 211 self._parse_header()
    212 self._check_stream_signal_channel_characteristics()
    213 self.is_header_parsed = True

File ~/.conda/envs/aind-ephys-pipeline/lib/python3.10/site-packages/neo/rawio/openephysbinaryrawio.py:82, in OpenEphysBinaryRawIO._parse_header(self)
     81 def _parse_header(self):
---> 82     folder_structure, all_streams, nb_block, nb_segment_per_block, possible_experiments = explore_folder(
     83         self.dirname, self.experiment_names
     84     )
     85     check_folder_consistency(folder_structure, possible_experiments)
     86     self.folder_structure = folder_structure

File ~/.conda/envs/aind-ephys-pipeline/lib/python3.10/site-packages/neo/rawio/openephysbinaryrawio.py:626, in explore_folder(dirname, experiment_names)
    624 else:
    625     timestamp_file = recording_folder / "continuous" / info["folder_name"] / "timestamps.npy"
--> 626 timestamps = np.load(str(timestamp_file), mmap_mode="r")
    627 timestamp0 = timestamps[0]
    628 t_start = timestamp0 / info["sample_rate"]

File ~/.conda/envs/aind-ephys-pipeline/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py:459, in load(file, mmap_mode, allow_pickle, fix_imports, encoding, max_header_size)
    457     own_fid = False
    458 else:
--> 459     fid = stack.enter_context(open(os.fspath(file), "rb"))
    460     own_fid = True
    462 # Code to distinguish from NumPy binary files and pickles.

FileNotFoundError: [Errno 2] No such file or directory: '/orcd/scratch/bcs/002/Vincent/LO_02/2025-02-05/LO_02_001/Record Node 101/experiment1/recording1/continuous/Neuropix-PXI-100.ProbeA/timestamps.npy'

Note the difference between LO_03/2025-04-01/LO_03_002 in the argument, and LO_02/2025-02-05/LO_02_001 in the output.

Also, a quick point about recomputing extension. Another reason that I did that was that
plot_sorting_summary(sorting_analyzer=sorting_analyzer, curation=True, backend='sortingview') returns this:

Exception: The spike_amplitudes sorting analyzer extension is required for this widget. Run the `sorting_analyzer.compute('spike_amplitudes', ...)` to compute it.
The unit_locations sorting analyzer extension is required for this widget. Run the `sorting_analyzer.compute('unit_locations', ...)` to compute it.
The template_similarity sorting analyzer extension is required for this widget. Run the `sorting_analyzer.compute('template_similarity', ...)` to compute it.

(with sorting_analyzer defined by sorting_analyzer = si.load_sorting_analyzer("<data_path>/sorting_output/postprocessed/experiment1_Record Node 101#Neuropix-PXI-100.ProbeA_recording1.zarr").

[alejoe91]

I see! I think that some extensions might have been deleted when you attempted to recompute them. Is it possible?

The different mapping is probably due to a wrong name in the data_description.json...Can this be the case?

[vncntprvst]

It is quite possible that some extensions were deleted when attempting to recompute them.
The wrong mapping is not due to an incorrect entry in the data_description.json, or any other file as far as I can see (I searched the whole data folder).

[vncntprvst]

I'll close this issue because the sorting analyzer issue is likely solved. I'll get back to you about the subsequent errors that persist.