-
Notifications
You must be signed in to change notification settings - Fork 122
Fix timestamp handling, expand migrations systems #711
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Convert numerical values to and from datetime objects. Includes a new data migration feature and a migration to convert numerical TAG fields to NUMERIC. Moves OM CLI commands into a new top-level `om` command while preserving backwards compat for old `migrate` command.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR implements timestamp handling for datetime fields in Redis-OM Python by converting datetime objects to Unix timestamps for storage, enabling proper NUMERIC indexing in RediSearch. It includes a comprehensive data migration system and moves CLI commands under a unified om
command structure.
- Convert datetime fields from string/object storage to Unix timestamp storage for proper indexing
- Add data migration framework with dependency management and rollback support
- Create unified
om
CLI interface while maintaining backward compatibility
Reviewed Changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 7 comments.
Show a summary per file
File | Description |
---|---|
tests/test_datetime_fix.py | Tests datetime timestamp conversion for HashModel and JsonModel |
tests/test_datetime_date_fix.py | Tests date field conversion to timestamps |
tests/test_data_migrations.py | Comprehensive test suite for the data migration system |
pyproject.toml | Adds new om CLI entry point with backward compatibility |
docs/migrations.md | Complete documentation for schema and data migration systems |
aredis_om/util.py | Extends NUMERIC_TYPES to include datetime types |
aredis_om/model/model.py | Core timestamp conversion logic and model save/get modifications |
aredis_om/model/migrations/datetime_migration.py | Built-in migration to convert existing datetime data |
aredis_om/model/migrations/data_migrator.py | Framework for managing data migrations |
aredis_om/model/cli/migrate_data.py | CLI commands for data migration operations |
aredis_om/cli/main.py | Unified CLI entry point |
aredis_om/cli/init.py | CLI package initialization |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Replace nested exception blocks with cleaner helper functions to improve maintainability and debuggability. Eliminates broad exception catching that could mask real bugs while preserving datetime conversion functionality. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Remove duplicate 'om' command prefixes in migration documentation. Commands should be 'om migrate-data' not 'om om migrate-data'. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
The refactored datetime conversion helper functions introduced subtle timezone handling differences that broke model equality comparisons in tests. Restoring the original working implementation to maintain compatibility. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Add missing importlib.util import in data_migrator.py - Use type(None) instead of string comparison for type checking - Remove debug print statements from test files (security concern) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Add missing importlib.util import in data_migrator.py - Fix type checking with proper noqa for E721 - Remove debug print statements from tests 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
# Get model data and apply transformations in the correct order | ||
data = self.model_dump() | ||
# Convert datetime objects to timestamps for proper indexing | ||
data = convert_datetime_to_timestamp(data) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This allows users to use normal type annotations for datetime fields, instead of a special field type we provide, even if it's messier "on the backend."
- Remove run_async() calls from sync CLI commands to prevent coroutine errors - Add AsyncMock -> Mock transformation in unasync configuration - Fix test_create_and_status_empty to use clean_redis fixture 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Replace Python 3.10+ union syntax (str | None) with Optional[str] to ensure compatibility with Python 3.9 used in CI 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Changed instance access of `self._meta.database` to class access using `self.__class__._meta.database` to resolve MyPy type checking issues. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Fixed async CLI functions that were incorrectly wrapped with run_async() calls, which caused "a coroutine was expected" errors in CI tests. Changed all CLI command functions to be properly async and use await instead of run_async() wrapper. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Fixed the proper async/sync pattern for CLI commands: - CLI command functions are sync (required by Click) - Inner functions are async and called with run_async() wrapper - Proper imports to use async migrators in async CLI, sync in sync CLI - Fixed unasync transformation issues for CLI execution 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Added trailing newline to pass flake8 linting requirements. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
The async-to-sync transformation was incomplete for CLI commands, causing "coroutine was expected" errors in tests. Added proper transformation rules to convert run_async() wrapper calls to direct function calls in sync versions. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Restructure async CLI functions to use run_async() around individual method calls - Add post-processing in make_sync.py to remove run_async() wrappers from sync versions - Resolves 'coroutine expected' errors in CLI tests
- Only mark migrations as unapplied after successful rollback - Handle NotImplementedError properly to maintain applied migration state - Add better exception handling for other rollback failures - Resolves test failures in test_rollback_not_supported and related tests
Add worker-specific Redis keys and index names to prevent race conditions when running schema migration tests with pytest-xdist (-n auto). - Use PYTEST_XDIST_WORKER environment variable for worker isolation - Create _WorkerAwareSchemaMigrator with worker-specific Redis keys - Update all test functions to use worker-specific index names - Fix test helper classes to use worker-isolated schema tracking keys This allows schema migration tests to run reliably in parallel CI execution. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Simplify the worker isolation approach by overriding the class constant APPLIED_MIGRATIONS_KEY instead of overriding individual methods. This ensures all methods that use the constant (including status()) use worker-specific keys. The previous approach missed that status() -> get_applied() -> uses APPLIED_MIGRATIONS_KEY causing cross-worker contamination in migration state tracking. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
The legacy 'migrate' command now uses automatic migrations with deprecation warnings pointing users to 'om migrate' for the new file-based system. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Remove placeholder dm1.py migration file that shouldn't be committed - Update migrations.md to clarify migrate vs om migrate differences - Document deprecation of standalone migrate command 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Remove test migration file and fix formatting issues found by flake8. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Fix line length and formatting issues caught by black linter. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
This PR makes a few changes, all to support datetime fields better.
Store datetime fields as NUMERIC data in Redis
Previously, we stored datetimes as TAGs (why, I don't know). Now, we convert datetime fields on models to and from NUMERIC fields in Redis.
New data migrations system
Introduces a data migrations system and CLI command (
om migrate-data
). The first built-in data migration will migrate datetime fields indexed as TAG data to NUMERIC data. Important: You must run this migration in order to use this version of Redis OM with datetime fields, as the models will now expect to get NUMERIC data for datetimes fields.New top-level
om
CLI commandMoves OM CLI commands into a new top-level
om
command while preserving backwards compatibility for the oldmigrate
command. This makes room for normal migration and now data migration commands.Expanded schema migrations system
To match the new
om migrate-data
sub-commands, such asrun
andcreate
, we expand the schema migrations system to support these features through file-based migrations. This introduces the possibility to roll back migrations, as well as practical concerns like deploying migrations to different environments and checking whether or not they have applied.