Fixes for cheetah v5.4.0 (#913)

PeterNSteinmetz · Peter N. Steinmetz · web-flow · commit 4cce40fcb854 · 2021-04-10T23:36:55.000+02:00
* Move constants into class so accessible for testing. * Test of building NcsBlocks. * Handle old files with truncated frequency in header and test. * Change interface on parse versus build. * Tests for PRE4 type and code corrections. Tests for v5.5.1 still failing. * Fix initializer, update loop vars. * Add additional tests on v5.5.1 with 2 blocks * Tests of block construction for incomplete blocks * Fix up single block case. * Remove unneeded classes. Clean up style. * Use private dtype by new private name. * Add test of side effects of read_ncs_files * Use NcsBlocksFactory and logical or. * Fix off by one in range for list. Comments. * Use standard time calculation for last time of block. * Remove test with tolerance over whole length. Fix microsPerSampUsed assignement. Using a tolerance over a longer experiment is not sensitive enough to detect blocks where perhaps a large amount of samples are dropped and there is a small gap afterwards. * Tests of raw io for incomplete records multiple block case. * Update stop times to include time for samples in partially filled records. * PEP and style cleanup. Corrected gap comment. * Line shortening for PEP8. * More PEP8 items. * Correct column error in data tested. Strange this hasn’t caused issues before with other file sets. * Update comment on PRE4 file limitations. * Tests for PRE4 file type in Cheetah v4.0.2. * PEP8 indent corrections. * Small PEP8 correctin. * Proper time opened header for BML and use of frequency. * Download files and activate tests for BML style files. * record start and end times in NcsBlocks for verification later * Code to handle digitallynx with cheetah v1 Like files from Dr. Florian Mormann’s lab. * Fixes to handle split block. * Tests NcsBlockFactory with split blocks and unfilled records. * Tests of data read in ncs files with partial records and split blocks. * Handle testing less data than available in all segments. * Read and test with partial blocks. Test all types in test_neuralyxio. * Read and test with partial blocks. Test all types in test_neuralyxio. * PEP8 fix ups. * Language and function renaming. # Conflicts: # neo/rawio/neuralynxrawio.py * Language and function renaming. * Move constants into class so accessible for testing. * Test of building NcsBlocks. * Handle old files with truncated frequency in header and test. * Change interface on parse versus build. # Conflicts: # neo/rawio/neuralynxrawio.py * Tests for PRE4 type and code corrections. Tests for v5.5.1 still failing. # Conflicts: # neo/rawio/neuralynxrawio.py * Fix initializer, update loop vars. * Add additional tests on v5.5.1 with 2 blocks * Tests of block construction for incomplete blocks * Fix up single block case. * Remove unneeded classes. Clean up style. # Conflicts: # neo/rawio/neuralynxrawio.py * Add test of side effects of read_ncs_files * Use NcsBlocksFactory and logical or. # Conflicts: # neo/rawio/neuralynxrawio.py * Fix off by one in range for list. Comments. * Use standard time calculation for last time of block. * Remove test with tolerance over whole length. Fix microsPerSampUsed assignement. Using a tolerance over a longer experiment is not sensitive enough to detect blocks where perhaps a large amount of samples are dropped and there is a small gap afterwards. * Tests of raw io for incomplete records multiple block case. * Update stop times to include time for samples in partially filled records. * PEP and style cleanup. Corrected gap comment. * Line shortening for PEP8. * More PEP8 items. # Conflicts: # neo/test/rawiotest/test_neuralynxrawio.py * Remove conflict markers. * More conflict resolution for rebase. * verify block function and test * Call renamed functions. * use _verifyBlockStructure to ensure same structure This is hopefully a bit faster than building NcsBlocks for each file. * Properly limit samples compared. * Temporarily remove data test. This matches apdavison’s from commit 2ac9596. I can’t find and revert the commit where I undid that change. * Remove duplication of some older tests. * Comments on structure of segment defining fields. * Factor out NcsBlock object. This to facilitate temporal comparisions needed for segment definitions when channels may have different recording lengths. * Change factory methods to use NcsBlock. * Rewrite tests for NcsBlocks refactoring. * More refactoring of NcsBlocks. * Gather all NcsBlocks first, then build segements. Builds NcsBlocks for all unique structures in all files. Then sets up segment structure based on these. A precursor to more sophisticated segments for record block structures. * change name to _scan_ncs_files rather than read * Proper formatting of number of NcsBlock structures. * Move neuralynxrawio to subdirectory to factor our other classes. * Eliminate WholeTimesPositionBlock as class. * Factor helper classes out in package. * Define data returned by scan_ncs_files and rename. * Return results from scan_ncs_files and unpack them into private atttributes. * Test returned results. * Fix for version 5.4.0 headers without : * Allow gap which is fraction of sample time, rather than fixed micros. This important when sampling rates are lower. * Test changes for Cheetah v5.4.0. * Clean up some comments. * Better error message if header/pattern mismatch. * PEP8 clean ups. * Changes to address comments by Julia Sprenger # Conflicts: # neo/rawio/neuralynxrawio/__init__.py # neo/rawio/neuralynxrawio/ncssections.py # neo/rawio/neuralynxrawio/nlxheader.py # Conflicts: # neo/rawio/neuralynxrawio/ncssections.py * Remove merge conflicts. # Conflicts: # neo/rawio/neuralynxrawio/__init__.py # neo/rawio/neuralynxrawio/ncssections.py # neo/rawio/neuralynxrawio/nlxheader.py * Fix up some PEP8 comments. # Conflicts: # neo/test/rawiotest/test_neuralynxrawio.py * Roll back inadvertent commit of local gin repository. * Fixup improperly resolved conflicts. These were created by inadvertendly committed the required changes on another branch and needing to cherry-pick off that branch. # Conflicts: # neo/rawio/neuralynxrawio/__init__.py # neo/rawio/neuralynxrawio/ncssections.py # neo/rawio/neuralynxrawio/nlxheader.py # neo/test/rawiotest/test_neuralynxrawio.py * More PEP8 fixups. * Replace inadvertently deleted package file. * Repaired inadvertent modification of blocks variable in nixio * Take 6.3.2 io tests out. Attempt to see what is happening with CircleCI testing. * Specify no entitites to test. * Retain tracking of maximum block length. # Conflicts: # neo/rawio/neuralynxrawio/ncssections.py * PEP8 clean ups. * Download Cheetah 5.4.0 test data. # Conflicts: # neo/test/iotest/test_neuralynxio.py # neo/test/rawiotest/test_neuralynxrawio.py # Conflicts: # neo/test/iotest/test_neuralynxio.py # neo/test/rawiotest/test_neuralynxrawio.py * use self for results Co-authored-by: Peter N. Steinmetz <PeterNSteinmetz@steinmetz.org>
diff --git a/neo/rawio/neuralynxrawio/nlxheader.py b/neo/rawio/neuralynxrawio/nlxheader.py
@@ -215,19 +215,32 @@ def __init__(self, filename):
                 hpd = NlxHeader.header_pattern_dicts['def']
         elif an == 'BML':
             hpd = NlxHeader.header_pattern_dicts['bml']
+            av = "2"
         else:
+            an = "Unknown"
+            av = "NA"
             hpd = NlxHeader.header_pattern_dicts['def']
 
         # opening time
-        dt1 = re.search(hpd['datetime1_regex'], txt_header).groupdict()
-        self['recording_opened'] = datetime.datetime.strptime(
-            dt1['date'] + ' ' + dt1['time'], hpd['datetimeformat'])
+        sr = re.search(hpd['datetime1_regex'], txt_header)
+        if not sr:
+            raise IOError("No matching header open date/time for application {} " +
+                          "version {}. Please contact developers.".format(an, av))
+        else:
+            dt1 = sr.groupdict()
+            self['recording_opened'] = datetime.datetime.strptime(
+                dt1['date'] + ' ' + dt1['time'], hpd['datetimeformat'])
 
         # close time, if available
         if 'datetime2_regex' in hpd:
-            dt2 = re.search(hpd['datetime2_regex'], txt_header).groupdict()
-            self['recording_closed'] = datetime.datetime.strptime(
-                dt2['date'] + ' ' + dt2['time'], hpd['datetimeformat'])
+            sr = re.search(hpd['datetime2_regex'], txt_header)
+            if not sr:
+                raise IOError("No matching header close date/time for application {} " +
+                              "version {}. Please contact developers.".format(an, av))
+            else:
+                dt2 = sr.groupdict()
+                self['recording_closed'] = datetime.datetime.strptime(
+                    dt2['date'] + ' ' + dt2['time'], hpd['datetimeformat'])
 
     def type_of_recording(self):
         """
diff --git a/neo/test/iotest/test_neuralynxio.py b/neo/test/iotest/test_neuralynxio.py
@@ -24,6 +24,7 @@ class CommonNeuralynxIOTest(BaseTestIO, unittest.TestCase, ):
         'BML_unfilledsplit/original_data',
         'Cheetah_v1.1.0/original_data',
         'Cheetah_v4.0.2/original_data',
+        'Cheetah_v5.4.0/original_data',
         'Cheetah_v5.5.1/original_data',
         'Cheetah_v5.6.3/original_data',
         'Cheetah_v5.7.4/original_data',
@@ -42,6 +43,8 @@ class CommonNeuralynxIOTest(BaseTestIO, unittest.TestCase, ):
         'Cheetah_v4.0.2/original_data/CSC14_trunc.Ncs',
         'Cheetah_v4.0.2/plain_data/CSC14_trunc.txt',
         'Cheetah_v4.0.2/README.txt',
+        'Cheetah_v5.4.0/original_data/CSC5_trunc.Ncs',
+        'Cheetah_v5.4.0/plain_data/CSC5_trunc.txt',
         'Cheetah_v5.5.1/original_data/CheetahLogFile.txt',
         'Cheetah_v5.5.1/original_data/CheetahLostADRecords.txt',
         'Cheetah_v5.5.1/original_data/Events.nev',
@@ -320,7 +323,7 @@ def test_keep_original_spike_times(self):
                 filename = st.file_origin.replace('original_data', 'plain_data')
                 if '.nse' in st.file_origin:
                     filename = filename.replace('.nse', '.txt')
-                    times_column = 0
+                    times_column = 1
                     plain_data = np.loadtxt(filename)[:, times_column]
                 elif '.ntt' in st.file_origin:
                     filename = filename.replace('.ntt', '.txt')
diff --git a/neo/test/rawiotest/test_neuralynxrawio.py b/neo/test/rawiotest/test_neuralynxrawio.py
@@ -19,6 +19,7 @@ class TestNeuralynxRawIO(BaseTestRawIO, unittest.TestCase, ):
         'BML_unfilledsplit/original_data',
         'Cheetah_v1.1.0/original_data',
         'Cheetah_v4.0.2/original_data',
+        'Cheetah_v5.4.0/original_data',
         'Cheetah_v5.5.1/original_data',
         'Cheetah_v5.6.3/original_data',
         'Cheetah_v5.7.4/original_data',
@@ -36,6 +37,8 @@ class TestNeuralynxRawIO(BaseTestRawIO, unittest.TestCase, ):
         'Cheetah_v4.0.2/original_data/CSC14_trunc.Ncs',
         'Cheetah_v4.0.2/plain_data/CSC14_trunc.txt',
         'Cheetah_v4.0.2/README.txt',
+        'Cheetah_v5.4.0/original_data/CSC5_trunc.Ncs',
+        'Cheetah_v5.4.0/plain_data/CSC5_trunc.txt',
         'Cheetah_v5.5.1/original_data/CheetahLogFile.txt',
         'Cheetah_v5.5.1/original_data/CheetahLostADRecords.txt',
         'Cheetah_v5.5.1/original_data/Events.nev',
@@ -79,7 +82,7 @@ class TestNeuralynxRawIO(BaseTestRawIO, unittest.TestCase, ):
         'Cheetah_v6.3.2/incomplete_blocks/Events.nev',
         'Cheetah_v6.3.2/incomplete_blocks/README.txt']
 
-    def test_read_ncs_files_sideeffects(self):
+    def test_scan_ncs_files(self):
 
         # Test BML style of Ncs files, similar to PRE4 but with fractional frequency
         # in the header and fractional microsPerSamp, which is then rounded as appropriate
@@ -179,6 +182,7 @@ class TestNcsRecordingType(TestNeuralynxRawIO, unittest.TestCase):
 
     ncsTypeTestFiles = [
         ('Cheetah_v4.0.2/original_data/CSC14_trunc.Ncs', 'PRE4'),
+        ('Cheetah_v5.4.0/original_data/CSC5_trunc.Ncs', 'DIGITALLYNX'),
         ('Cheetah_v5.5.1/original_data/STet3a.nse', 'DIGITALLYNXSX'),
         ('Cheetah_v5.5.1/original_data/Tet3a.ncs', 'DIGITALLYNXSX'),
         ('Cheetah_v5.6.3/original_data/CSC1.ncs', 'DIGITALLYNXSX'),
