Frequency detec #123
Merged
Frequency detec #123
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Features: - Add lazy frequency detection from temporal coordinates using xarray/Dask - Validate frequency consistency across concatenated input files - Configurable validation with tolerance settings for slight differences - Custom FrequencyMismatchError for clear error reporting - integrate validation into all CMORiser classes with user control - Comprehensive unit tests covering various frequency scenarios This prevents issues when concatenating files with different temporal frequencies and provides early detection of data inconsistencies without loading full datasets into memory. The implementation is performance-oriented using lazy evaluation to check only the first few time points of each file. Usage: - Validation enabled by default: ACCESS_ESM_CMORiser(...) - Disable validation: ACCESS_ESM_CMORiser(..., validate_frequency=False) - Tolerance configurable via validate_consistent_frequency() function Files modified: - utilities.py: Core frequency detection functions - base.py: Integration with load_dataset method - driver.py: User-facing validate_frequency parameter - ocean.py: Updated constructor to pass through parameter - tests/unit/test_frequency_detection.py: Comprehensive test suite
… new utility functions for frequency parsing and compatibility checks
- Added support for detecting temporal frequency from CF-compliant time bounds in `detect_time_frequency_lazy`. - Implemented a new utility function `_detect_frequency_from_bounds` for improved frequency detection. - Introduced `validate_and_resample_if_needed` to validate dataset frequency and resample if necessary for CMIP6 compatibility. - Enhanced `resample_dataset_temporal` to handle various resampling methods based on variable metadata. - Updated tests to cover new frequency detection and resampling features, including edge cases for single time points and bounds detection. - Created a new test suite for temporal resampling functionality, ensuring robust validation and error handling.
…calendar month variations
… for calendar month variations. Implement helper functions to validate monthly files and adjust compatibility checks in CMIP6 frequency validation.
…hout resampling. Special handling for calendar month variations added to improve accuracy in CMIP6 frequency checks.
…tenation; add fallback to individual file analysis. Remove outdated test scripts for exact error simulation and monthly calendar variations.
This reverts commit 936f1c6.
…pdate related tests
…nd update integration tests accordingly
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces comprehensive support for temporal frequency validation and automatic resampling in the ACCESS-MOPPy workflow, ensuring input datasets conform to CMIP6 requirements. The changes add new configuration options for frequency validation and resampling, integrate these options into the main classes, and provide robust error handling. Additionally, a thorough suite of unit tests has been added to verify the new functionality.
Temporal frequency validation and resampling (core logic):
validate_frequency,enable_resampling, andresampling_methodoptions toCMIP6_CMORiser,CMIP6_Ocean_CMORiser, and driver classes, allowing users to control frequency validation and resampling behavior. These options propagate through the workflow for consistent handling. [1] [2] [3] [4] [5]Integration and configuration propagation:
Testing and robustness:
tests/unit/test_temporal_resampling.py) covering detection of aggregation methods, frequency string conversion, actual resampling operations, integrated validation/resampling workflow, and error handling for edge cases.These changes significantly improve the reliability and flexibility of temporal frequency handling in the ACCESS-MOPPy pipeline, making it easier to ensure CMIP6 compliance and diagnose issues with input data.