Add XDF importer for multi-stream Lab Streaming Layer files#36
Add XDF importer for multi-stream Lab Streaming Layer files#36neuromechanist merged 16 commits intomainfrom
Conversation
- Add XDFImporter class for loading XDF files - Add summarize_xdf function to explore XDF contents - Support stream selection by name, type, or ID - Handle multi-stream files with different sampling rates - Add pyxdf as dependency - Add comprehensive tests with real XDF data - Update EMG.from_file to support XDF format Closes #35
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #36 +/- ##
==========================================
+ Coverage 90.65% 90.87% +0.22%
==========================================
Files 22 24 +2
Lines 3038 3573 +535
==========================================
+ Hits 2754 3247 +493
- Misses 284 326 +42
🚀 New features to boost your workflow:
|
- Create synthetic 4-stream XDF: EEG, EMG, Mocap, Markers - Add 8 tests for multi-stream import functionality - Test stream selection by type and name - Test sampling rate handling and channel labels
- xdf_writer.py: Python XDF writer based on XDF spec - create_multistream_test_xdf.py: Script to create test file
There was a problem hiding this comment.
Pull request overview
This PR adds comprehensive support for importing XDF (Extensible Data Format) files, which are used by the Lab Streaming Layer (LSL) ecosystem for multi-stream physiological data recording. The implementation includes both a summary/exploration function and a full importer with stream selection capabilities.
Key changes:
- XDF file format support with selective stream import by name, type, or ID
summarize_xdf()function for exploring multi-stream files before loading- Automatic format detection for
.xdfand.xdfzextensions - Support for handling both numeric streams (arrays) and marker streams (lists)
Reviewed changes
Copilot reviewed 6 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| emgio/importers/xdf.py | Core XDF importer with stream selection, synchronization, and metadata extraction capabilities |
| emgio/tests/test_xdf_importer.py | Comprehensive test suite covering single/multi-stream files, stream selection, and roundtrip conversion |
| emgio/core/emg.py | Updated type hints and registered XDF format in the from_file method |
| scripts/xdf_writer.py | Test utility for creating synthetic multi-stream XDF files for testing |
| scripts/create_multistream_test_xdf.py | Script to extract and replay XDF data segments using LSL and LabRecorder |
| pyproject.toml | Added pyxdf>=1.16.0 as a core dependency |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Add XDF format docs with multi-stream examples - Add XDF importer API reference - Add XDF examples page - Update index and README with XDF support
- Add include_timestamps parameter to preserve LSL timestamps
- Each stream gets a {stream_name}_LSL_timestamps channel
- Timestamp channels marked as MISC type with seconds unit
- Add 5 tests for timestamp functionality
- Add timestamp preservation docs to format page - Update API reference with include_timestamps parameter - Add timestamp examples to examples page - Update README and index with feature mention
- Integrate _determine_channel_type_from_label into _extract_channel_info - Guard against division by zero in time index calculation - Use relative paths in xdf_writer.py - Use env vars for configurable paths in create_multistream_test_xdf.py
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 14 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Check for empty stream_starts before calling max() - Handle missing format key with sensible defaults - Catch TimeoutExpired when stopping recorder process
- Check timestamps length before accessing first element in resampling - Handle empty timestamps in LSL timestamp channel creation with NaN fill
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 15 changed files in this pull request and generated 12 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Rewrite _load_streams to find highest sampling rate stream first - Use highest rate stream as time base to avoid data loss from downsampling - Add reference_stream parameter to allow user override - Add tests verifying highest rate behavior and reference_stream parameter - Fix bounds check for channel_types access
- Change default unit from uV to a.u. (arbitrary units) for XDF channels
without explicit unit metadata
- Remove unused local_clock() import and call from replay script
- Fix _validate_paths to properly check env var strings before creating
Path objects (Path('') is truthy, so explicit None check needed)
- Add robust metadata extraction with try-except handling for malformed
XDF channel info, handling both list and string value types
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 15 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 15 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Use isoformat() for proper ISO 8601 datetime in XDF writer - Broaden error pattern matching in test for cross-platform compatibility - Note: duplicate code in xdf_writer is acceptable for test utility clarity - Note: channel name test is intentionally specific to verify documented behavior
- Add clarifying comment for duration calculation edge case - Raise ValueError when data length mismatches and no timestamps available for interpolation
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 15 changed files in this pull request and generated 10 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Use -1 as sentinel for missing stream_id instead of 0 - Improve docstring for _load_synchronized_streams wrapper - Clarify comment about default a.u. units
Summary
summarize_xdf()function to explore XDF file contents before importFeatures
summarize_xdf(filepath): Returns XDFSummary with stream info (name, type, channels, srate, duration)XDFImporter.load(filepath, stream_names=..., stream_types=..., stream_ids=...): Load specific streamsEMG.from_file("recording.xdf"): Auto-detect XDF formatTest plan
Closes #35