RecordingExtractors

[pauladkisson]

Recording Extractors Architecture

Fixes #170

Overview

This refactor replaces monolithic format detection and reading logic with a modular extractor architecture. Each data format (TDT, Doric, CSV, NPM) now has its own dedicated class implementing a common interface.

Benefits: Modularity, extensibility for new formats, consistent API, and isolated testability.

Architecture

classDiagram
    class BaseRecordingExtractor {
        <>
        +discover_events_and_flags()* tuple~list, list~
        +read(events, outputPath)* list~dict~
        +save(output_dicts, outputPath)* None
        #_write_hdf5(data, storename, output_path, key) None
    }
    
    class TdtRecordingExtractor
    class DoricRecordingExtractor
    class CsvRecordingExtractor
    class NpmRecordingExtractor
    
    BaseRecordingExtractor <|-- TdtRecordingExtractor
    BaseRecordingExtractor <|-- DoricRecordingExtractor
    BaseRecordingExtractor <|-- CsvRecordingExtractor
    CsvRecordingExtractor <|-- NpmRecordingExtractor

Loading

API Contract

All extractors implement three methods:

Method	Purpose
discover_events_and_flags()	Class method to find available events in data files
read(*, events, outputPath)	Extract data for specified events → returns list of dicts
save(*, output_dicts, outputPath)	Write extracted data to HDF5

Note: discover_events_and_flags() has a flexible signature—NPM requires additional num_ch and inputParameters arguments for channel configuration.

NPM Configuration Pattern: Tkinter GUI code has been moved out of the extractor and into saveStoresList.py. The extractor provides helper methods (has_multiple_event_ttls(), needs_ts_unit()) to determine what configuration is needed, while the GUI layer collects user input and passes it to discover_events_and_flags() via inputParameters. This keeps the extractor free of GUI dependencies.

Pipeline Integration

1. Step 2 (saveStoresList.py): Calls discover_events_and_flags() to find events, presents GUI for user to create friendly name mappings → outputs storesList.csv

2. Step 3 (readTevTsq.py): Creates appropriate extractor, reads storesList.csv for event list, processes all events in parallel via read_and_save_all_events() → outputs HDF5 files

Doric note: Uses storesList.csv to build the required event_name_to_event_type mapping.

Data Flow

flowchart TB
    A[Raw Data Files] --> B[Step 2: saveStoresList.py]
    B --> C[discover_events_and_flags]
    C --> D[GUI: User Maps Events]
    D --> E[storesList.csv]
    E --> F[Step 3: readTevTsq.py]
    A --> F
    F --> G[Create Extractor]
    G --> H[read_and_save_all_events]
    H --> I[HDF5 Files]

Loading