Skip to content

Conversation

abrookins
Copy link
Collaborator

@abrookins abrookins commented Aug 20, 2025

This PR makes a few changes, all to support datetime fields better.

Store datetime fields as NUMERIC data in Redis

Previously, we stored datetimes as TAGs (why, I don't know). Now, we convert datetime fields on models to and from NUMERIC fields in Redis.

New data migrations system

Introduces a data migrations system and CLI command (om migrate-data). The first built-in data migration will migrate datetime fields indexed as TAG data to NUMERIC data. Important: You must run this migration in order to use this version of Redis OM with datetime fields, as the models will now expect to get NUMERIC data for datetimes fields.

New top-level om CLI command

Moves OM CLI commands into a new top-level om command while preserving backwards compatibility for the old migrate command. This makes room for normal migration and now data migration commands.

Expanded schema migrations system

To match the new om migrate-data sub-commands, such as run and create, we expand the schema migrations system to support these features through file-based migrations. This introduces the possibility to roll back migrations, as well as practical concerns like deploying migrations to different environments and checking whether or not they have applied.

Convert numerical values to and from datetime objects.
Includes a new data migration feature and a migration to
convert numerical TAG fields to NUMERIC.

Moves OM CLI commands into a new top-level `om` command
while preserving backwards compat for old `migrate` command.
@abrookins abrookins requested a review from Copilot August 20, 2025 00:45
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements timestamp handling for datetime fields in Redis-OM Python by converting datetime objects to Unix timestamps for storage, enabling proper NUMERIC indexing in RediSearch. It includes a comprehensive data migration system and moves CLI commands under a unified om command structure.

  • Convert datetime fields from string/object storage to Unix timestamp storage for proper indexing
  • Add data migration framework with dependency management and rollback support
  • Create unified om CLI interface while maintaining backward compatibility

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
tests/test_datetime_fix.py Tests datetime timestamp conversion for HashModel and JsonModel
tests/test_datetime_date_fix.py Tests date field conversion to timestamps
tests/test_data_migrations.py Comprehensive test suite for the data migration system
pyproject.toml Adds new om CLI entry point with backward compatibility
docs/migrations.md Complete documentation for schema and data migration systems
aredis_om/util.py Extends NUMERIC_TYPES to include datetime types
aredis_om/model/model.py Core timestamp conversion logic and model save/get modifications
aredis_om/model/migrations/datetime_migration.py Built-in migration to convert existing datetime data
aredis_om/model/migrations/data_migrator.py Framework for managing data migrations
aredis_om/model/cli/migrate_data.py CLI commands for data migration operations
aredis_om/cli/main.py Unified CLI entry point
aredis_om/cli/init.py CLI package initialization

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

abrookins and others added 6 commits August 20, 2025 09:34
Replace nested exception blocks with cleaner helper functions to improve
maintainability and debuggability. Eliminates broad exception catching
that could mask real bugs while preserving datetime conversion functionality.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Remove duplicate 'om' command prefixes in migration documentation.
Commands should be 'om migrate-data' not 'om om migrate-data'.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
The refactored datetime conversion helper functions introduced subtle timezone
handling differences that broke model equality comparisons in tests.
Restoring the original working implementation to maintain compatibility.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
- Add missing importlib.util import in data_migrator.py
- Use type(None) instead of string comparison for type checking
- Remove debug print statements from test files (security concern)

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
- Add missing importlib.util import in data_migrator.py
- Fix type checking with proper noqa for E721
- Remove debug print statements from tests

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
@abrookins abrookins marked this pull request as ready for review August 21, 2025 23:09
# Get model data and apply transformations in the correct order
data = self.model_dump()
# Convert datetime objects to timestamps for proper indexing
data = convert_datetime_to_timestamp(data)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This allows users to use normal type annotations for datetime fields, instead of a special field type we provide, even if it's messier "on the backend."

@abrookins abrookins changed the title Fix timestamp handling Fix timestamp handling, expand migrations systems Aug 25, 2025
abrookins and others added 16 commits August 27, 2025 15:11
- Remove run_async() calls from sync CLI commands to prevent coroutine errors
- Add AsyncMock -> Mock transformation in unasync configuration
- Fix test_create_and_status_empty to use clean_redis fixture

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Replace Python 3.10+ union syntax (str | None) with Optional[str]
to ensure compatibility with Python 3.9 used in CI

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Changed instance access of `self._meta.database` to class access using
`self.__class__._meta.database` to resolve MyPy type checking issues.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Fixed async CLI functions that were incorrectly wrapped with run_async()
calls, which caused "a coroutine was expected" errors in CI tests.
Changed all CLI command functions to be properly async and use await
instead of run_async() wrapper.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Fixed the proper async/sync pattern for CLI commands:
- CLI command functions are sync (required by Click)
- Inner functions are async and called with run_async() wrapper
- Proper imports to use async migrators in async CLI, sync in sync CLI
- Fixed unasync transformation issues for CLI execution

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Added trailing newline to pass flake8 linting requirements.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
The async-to-sync transformation was incomplete for CLI commands,
causing "coroutine was expected" errors in tests. Added proper
transformation rules to convert run_async() wrapper calls to
direct function calls in sync versions.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
- Restructure async CLI functions to use run_async() around individual method calls
- Add post-processing in make_sync.py to remove run_async() wrappers from sync versions
- Resolves 'coroutine expected' errors in CLI tests
- Only mark migrations as unapplied after successful rollback
- Handle NotImplementedError properly to maintain applied migration state
- Add better exception handling for other rollback failures
- Resolves test failures in test_rollback_not_supported and related tests
Add worker-specific Redis keys and index names to prevent race conditions
when running schema migration tests with pytest-xdist (-n auto).

- Use PYTEST_XDIST_WORKER environment variable for worker isolation
- Create _WorkerAwareSchemaMigrator with worker-specific Redis keys
- Update all test functions to use worker-specific index names
- Fix test helper classes to use worker-isolated schema tracking keys

This allows schema migration tests to run reliably in parallel CI execution.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Simplify the worker isolation approach by overriding the class constant
APPLIED_MIGRATIONS_KEY instead of overriding individual methods. This ensures
all methods that use the constant (including status()) use worker-specific keys.

The previous approach missed that status() -> get_applied() -> uses APPLIED_MIGRATIONS_KEY
causing cross-worker contamination in migration state tracking.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
The legacy 'migrate' command now uses automatic migrations with deprecation
warnings pointing users to 'om migrate' for the new file-based system.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
abrookins and others added 3 commits August 29, 2025 13:47
- Remove placeholder dm1.py migration file that shouldn't be committed
- Update migrations.md to clarify migrate vs om migrate differences
- Document deprecation of standalone migrate command

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Remove test migration file and fix formatting issues found by flake8.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Fix line length and formatting issues caught by black linter.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant