Nlx single file mode (#957)

PeterNSteinmetz · Peter N. Steinmetz · web-flow · commit 8dfb07934b91 · 2021-04-10T21:12:25.000+02:00
* Move constants into class so accessible for testing. * Test of building NcsBlocks. * Handle old files with truncated frequency in header and test. * Change interface on parse versus build. * Tests for PRE4 type and code corrections. Tests for v5.5.1 still failing. * Fix initializer, update loop vars. * Add additional tests on v5.5.1 with 2 blocks * Tests of block construction for incomplete blocks * Fix up single block case. * Remove unneeded classes. Clean up style. * Use private dtype by new private name. * Add test of side effects of read_ncs_files * Use NcsBlocksFactory and logical or. * Fix off by one in range for list. Comments. * Use standard time calculation for last time of block. * Remove test with tolerance over whole length. Fix microsPerSampUsed assignement. Using a tolerance over a longer experiment is not sensitive enough to detect blocks where perhaps a large amount of samples are dropped and there is a small gap afterwards. * Tests of raw io for incomplete records multiple block case. * Update stop times to include time for samples in partially filled records. * PEP and style cleanup. Corrected gap comment. * Line shortening for PEP8. * More PEP8 items. * Correct column error in data tested. Strange this hasn’t caused issues before with other file sets. * Update comment on PRE4 file limitations. * Tests for PRE4 file type in Cheetah v4.0.2. * PEP8 indent corrections. * Small PEP8 correctin. * Proper time opened header for BML and use of frequency. * Download files and activate tests for BML style files. * record start and end times in NcsBlocks for verification later * Code to handle digitallynx with cheetah v1 Like files from Dr. Florian Mormann’s lab. * Fixes to handle split block. * Tests NcsBlockFactory with split blocks and unfilled records. * Tests of data read in ncs files with partial records and split blocks. * Handle testing less data than available in all segments. * Read and test with partial blocks. Test all types in test_neuralyxio. * Read and test with partial blocks. Test all types in test_neuralyxio. * PEP8 fix ups. * Language and function renaming. # Conflicts: # neo/rawio/neuralynxrawio.py * Language and function renaming. * Move constants into class so accessible for testing. * Test of building NcsBlocks. * Handle old files with truncated frequency in header and test. * Change interface on parse versus build. # Conflicts: # neo/rawio/neuralynxrawio.py * Tests for PRE4 type and code corrections. Tests for v5.5.1 still failing. # Conflicts: # neo/rawio/neuralynxrawio.py * Fix initializer, update loop vars. * Add additional tests on v5.5.1 with 2 blocks * Tests of block construction for incomplete blocks * Fix up single block case. * Remove unneeded classes. Clean up style. # Conflicts: # neo/rawio/neuralynxrawio.py * Add test of side effects of read_ncs_files * Use NcsBlocksFactory and logical or. # Conflicts: # neo/rawio/neuralynxrawio.py * Fix off by one in range for list. Comments. * Use standard time calculation for last time of block. * Remove test with tolerance over whole length. Fix microsPerSampUsed assignement. Using a tolerance over a longer experiment is not sensitive enough to detect blocks where perhaps a large amount of samples are dropped and there is a small gap afterwards. * Tests of raw io for incomplete records multiple block case. * Update stop times to include time for samples in partially filled records. * PEP and style cleanup. Corrected gap comment. * Line shortening for PEP8. * More PEP8 items. # Conflicts: # neo/test/rawiotest/test_neuralynxrawio.py * Remove conflict markers. * More conflict resolution for rebase. * verify block function and test * Call renamed functions. * use _verifyBlockStructure to ensure same structure This is hopefully a bit faster than building NcsBlocks for each file. * Properly limit samples compared. * Temporarily remove data test. This matches apdavison’s from commit 2ac9596. I can’t find and revert the commit where I undid that change. * Remove duplication of some older tests. * Comments on structure of segment defining fields. * Factor out NcsBlock object. This to facilitate temporal comparisions needed for segment definitions when channels may have different recording lengths. * Change factory methods to use NcsBlock. * Rewrite tests for NcsBlocks refactoring. * More refactoring of NcsBlocks. * Gather all NcsBlocks first, then build segements. Builds NcsBlocks for all unique structures in all files. Then sets up segment structure based on these. A precursor to more sophisticated segments for record block structures. * change name to _scan_ncs_files rather than read * Proper formatting of number of NcsBlock structures. * Move neuralynxrawio to subdirectory to factor our other classes. * Eliminate WholeTimesPositionBlock as class. * Factor helper classes out in package. * Define data returned by scan_ncs_files and rename. * Return results from scan_ncs_files and unpack them into private atttributes. * Test returned results. * Fix for version 5.4.0 headers without : * Allow gap which is fraction of sample time, rather than fixed micros. This important when sampling rates are lower. * Test changes for Cheetah v5.4.0. * change wording about ambiguity * Add docstring about required signature with sets of channels. * Add comment about restrictions on selected set of channels. * Single file mode and tests. * Clarify use of dirname or filename, but not both. * Test no nev filenames. Co-authored-by: Peter N. Steinmetz <PeterNSteinmetz@steinmetz.org>
diff --git a/neo/rawio/neuralynxrawio/neuralynxrawio.py b/neo/rawio/neuralynxrawio/neuralynxrawio.py
@@ -49,22 +49,39 @@ class NeuralynxRawIO(BaseRawIO):
     _ncs_dtype = [('timestamp', 'uint64'), ('channel_id', 'uint32'), ('sample_rate', 'uint32'),
                   ('nb_valid', 'uint32'), ('samples', 'int16', (NcsSection._RECORD_SIZE))]
 
-    def __init__(self, dirname='', keep_original_times=False, **kargs):
+    def __init__(self, dirname='', filename='', keep_original_times=False, **kargs):
         """
+        Initialize io for either a directory of Ncs files or a single Ncs file.
+
         Parameters
         ----------
         dirname: str
-            name of directory containing all files for dataset
+            name of directory containing all files for dataset. If provided, filename is
+            ignored.
+        filename: str
+            name of a single ncs, nse, nev, or ntt file to include in dataset. If used,
+            dirname must not be provided.
         keep_original_times:
             if True, keep original start time as in files,
             otherwise set 0 of time to first time in dataset
         """
-        self.dirname = dirname
+        if dirname != '':
+            self.dirname = dirname
+            self.rawmode = 'one-dir'
+        elif filename != '':
+            self.filename = filename
+            self.rawmode = 'one-file'
+        else:
+            raise ValueError("One of dirname or filename must be provided.")
+
         self.keep_original_times = keep_original_times
         BaseRawIO.__init__(self, **kargs)
 
     def _source_name(self):
-        return self.dirname
+        if self.rawmode == 'one-file':
+            return self.filename
+        else:
+            return self.dirname
 
     def _parse_header(self):
 
@@ -91,8 +108,15 @@ def _parse_header(self):
         unit_annotations = []
         event_annotations = []
 
-        for filename in sorted(os.listdir(self.dirname)):
-            filename = os.path.join(self.dirname, filename)
+        if self.rawmode == 'one-dir':
+            filenames = sorted(os.listdir(self.dirname))
+            dirname = self.dirname
+        else:
+            dirname, fname = os.path.split(self.filename)
+            filenames = [fname]
+
+        for filename in filenames:
+            filename = os.path.join(dirname, filename)
 
             _, ext = os.path.splitext(filename)
             ext = ext[1:]  # remove dot
diff --git a/neo/rawio/neuralynxrawio/nlxheader.py b/neo/rawio/neuralynxrawio/nlxheader.py
@@ -89,6 +89,14 @@ def _to_bool(txt):
                             r'  At Time: (?P<time>\S+)',
             filename_regex=r'## File Name: (?P<filename>\S+)',
             datetimeformat='%m/%d/%Y %H:%M:%S.%f'),
+        # Cheetah version 5.4.0
+        'v5.4.0': dict(
+            datetime1_regex=r'## Time Opened \(m/d/y\): (?P<date>\S+)'
+                            r'  At Time: (?P<time>\S+)',
+            datetime2_regex=r'## Time Closed \(m/d/y\): (?P<date>\S+)'
+                            r'  At Time: (?P<time>\S+)',
+            filename_regex=r'## File Name: (?P<filename>\S+)',
+            datetimeformat='%m/%d/%Y %H:%M:%S.%f'),
         # Cheetah version 5 before and including v 5.6.4 as well as version 1
         'bv5.6.4': dict(
             datetime1_regex=r'## Time Opened \(m/d/y\): (?P<date>\S+)'
@@ -199,6 +207,8 @@ def __init__(self, filename):
                 hpd = NlxHeader.header_pattern_dicts['bv5.6.4']
             elif av < '5':
                 hpd = NlxHeader.header_pattern_dicts['bv5']
+            elif av <= '5.4.0':
+                hpd = NlxHeader.header_pattern_dicts['v5.4.0']
             elif av <= '5.6.4':
                 hpd = NlxHeader.header_pattern_dicts['bv5.6.4']
             else:
diff --git a/neo/test/iotest/test_neuralynxio.py b/neo/test/iotest/test_neuralynxio.py
@@ -309,7 +309,7 @@ def _load_plaindata(self, filename, numSamps):
             #     np.testing.assert_allclose(plain_data[:numToTest],
             #                                anasig.magnitude[:numToTest, 0] * gain_factor_0,
             #                                rtol=0.01, err_msg=" for file " + filename)
-    @unittest.skip
+    @unittest.skip("nse failing for now as per issue #907")
     def test_keep_original_spike_times(self):
         for session in self.files_to_test:
             dirname = self.get_filename_path(session)
@@ -347,9 +347,11 @@ def test_incomplete_block_handling_v632(self):
         self.assertEqual(len(block.segments), n_gaps + 1)
         # self.assertEqual(len(block.channel_indexes[0].analogsignals), n_gaps + 1)
 
-        for t, gt in zip(nio._sigs_t_start, [8408.806811, 8427.832053, 8487.768561]):
+        for t, gt in zip(nio._ncs_seg_timestamp_limits.t_start, [8408.806811, 8427.832053,
+                                                                 8487.768561]):
             self.assertEqual(np.round(t, 4), np.round(gt, 4))
-        for t, gt in zip(nio._sigs_t_stop, [8427.831990, 8487.768498, 8515.816549]):
+        for t, gt in zip(nio._ncs_seg_timestamp_limits.t_stop, [8427.831990, 8487.768498,
+                                                                8515.816549]):
             self.assertEqual(np.round(t, 4), np.round(gt, 4))
 
 
diff --git a/neo/test/rawiotest/test_neuralynxrawio.py b/neo/test/rawiotest/test_neuralynxrawio.py
@@ -124,7 +124,8 @@ def test_read_ncs_files_sideeffects(self):
         # three blocks of records. Gaps are on the order of 60 microseconds or so.
         rawio = NeuralynxRawIO(self.get_filename_path('Cheetah_v6.3.2/incomplete_blocks'))
         rawio.parse_header()
-        # test values here from direct inspection of .ncs file
+        # test values here from direct inspection of .ncs file, except for 3rd block
+        # t_stop, which is extended due to events past the last block of ncs records.
         self.assertEqual(rawio._nb_segment, 3)
         self.assertListEqual(rawio._timestamp_limits, [(8408806811, 8427831990),
                                                        (8427832053, 8487768498),
@@ -134,6 +135,41 @@ def test_read_ncs_files_sideeffects(self):
         self.assertListEqual(rawio._sigs_t_start, [8408.806811, 8427.832053, 8487.768561])
         self.assertEqual(len(rawio._sigs_memmaps), 3)  # check only that there are 3 memmaps
 
+    def test_single_file_mode(self):
+        """
+        Tests reading of single files.
+        """
+
+        # test single analog signal channel
+        fname = self.get_filename_path('Cheetah_v5.6.3/original_data/CSC1.ncs')
+        rawio = NeuralynxRawIO(filename=fname)
+        rawio.parse_header()
+
+        self.assertEqual(rawio._nb_segment, 2)
+        self.assertEqual(len(rawio.ncs_filenames), 1)
+        self.assertEqual(len(rawio.nev_filenames), 0)
+        sigHdrs = rawio.header['signal_channels']
+        self.assertEqual(sigHdrs.size, 1)
+        self.assertEqual(sigHdrs[0][0], 'CSC1')
+        self.assertEqual(sigHdrs[0][1], 58)
+        self.assertEqual(len(rawio.header['unit_channels']), 0)
+        self.assertEqual(len(rawio.header['event_channels']), 0)
+
+        # test one single electrode channel
+        fname = self.get_filename_path('Cheetah_v5.5.1/original_data/STet3a.nse')
+        rawio = NeuralynxRawIO(filename=fname)
+        rawio.parse_header()
+
+        self.assertEqual(rawio._nb_segment, 1)
+        self.assertEqual(len(rawio.ncs_filenames), 0)
+        self.assertEqual(len(rawio.nev_filenames), 0)
+        seHdrs = rawio.header['unit_channels']
+        self.assertEqual(len(seHdrs), 1)
+        self.assertEqual(seHdrs[0][0], 'chSTet3a#8#0')
+        self.assertEqual(seHdrs[0][1], '0')
+        self.assertEqual(len(rawio.header['signal_channels']), 0)
+        self.assertEqual(len(rawio.header['event_channels']), 0)
+
 
 class TestNcsRecordingType(TestNeuralynxRawIO, unittest.TestCase):
     """
