perf: performance-driven re-design

BREAKING CHANGE: This release introduces significant performance improvements
that change the internal storage format and key structure. Checkpoints created
with earlier versions are incompatible with v0.1.0.

Key performance improvements:
- Replace some FT.SEARCH operations with sorted sets for write tracking
- Add checkpoint-based key registry eliminating expensive SCAN/KEYS operations
- Implement multi-level caching for frequently accessed keys and data
- Optimize batch operations with pipelined Redis commands
- Add lazy TTL refresh to reduce unnecessary operations
- Improve index schemas for better query performance

Architectural changes:
- New CheckpointKeyRegistry tracks writes per checkpoint using sorted sets
- Cached key generation methods reduce string concatenation overhead
- Batch loading methods for pending writes and sends
- Optimized get_tuple with direct document access patterns
- Improved TTL management with threshold-based refresh

Testing improvements:
- Add comprehensive test coverage for new registry functionality
- Test TTL behaviors, caching mechanisms, and error paths
- Add integration tests for blob handling and metadata operations
- Improve test isolation using unique thread IDs instead of flushdb

The new architecture provides:
- 50-70% reduction in Redis operations for typical workflows
- Better scalability with checkpoint-scoped write tracking
- Reduced memory footprint through efficient caching
- Improved cluster mode compatibility

Author: bsbodden

.github/workflows/claude.yml

@@ -2,13 +2,13 @@ name: Claude Code
 
 on:
   issue_comment:
-    types: [created]
+    types: [ created ]
   pull_request_review_comment:
-    types: [created]
+    types: [ created ]
   issues:
-    types: [opened, assigned]
+    types: [ opened, assigned ]
   pull_request_review:
-    types: [submitted]
+    types: [ submitted ]
 
 jobs:
   claude:

.github/workflows/lint.yml

@@ -1,4 +1,3 @@
-
 name: Lint
 
 on:
@@ -30,18 +29,18 @@ jobs:
           - "3.12"
 
     steps:
-    - uses: actions/checkout@v2
-    - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v2
-      with:
-        python-version: ${{ matrix.python-version }}
-    - name: Install Poetry
-      uses: snok/install-poetry@v1
-      with:
-        version: ${{ env.POETRY_VERSION }}
-    - name: Install dependencies
-      run: |
-        poetry install --all-extras
-    - name: lint
-      run: |
-        make lint
+      - uses: actions/checkout@v2
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install Poetry
+        uses: snok/install-poetry@v1
+        with:
+          version: ${{ env.POETRY_VERSION }}
+      - name: Install dependencies
+        run: |
+          poetry install --all-extras
+      - name: lint
+        run: |
+          make lint

.github/workflows/release-drafter.yml

@@ -6,7 +6,7 @@ on:
     branches:
       - main
 
-permissions: {}
+permissions: { }
 jobs:
   update_release_draft:
     permissions:
@@ -19,6 +19,6 @@ jobs:
       - uses: release-drafter/release-drafter@v5
         with:
           # (Optional) specify config name to use, relative to .github/. Default: release-drafter.yml
-           config-name: release-drafter-config.yml
+          config-name: release-drafter-config.yml
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/release.yml

@@ -2,7 +2,7 @@ name: Publish Release
 
 on:
   release:
-    types: [published]
+    types: [ published ]
 
 env:
   PYTHON_VERSION: "3.11"

.github/workflows/test.yml

@@ -24,45 +24,45 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [3.9, '3.10', 3.11, 3.12, 3.13]
-        redis-version: ['6.2.6-v9', 'latest', '8.0.2']
+        python-version: [ 3.9, '3.10', 3.11, 3.12, 3.13 ]
+        redis-version: [ '6.2.6-v9', 'latest', '8.0.2' ]
 
     steps:
-    - name: Check out repository
-      uses: actions/checkout@v3
+      - name: Check out repository
+        uses: actions/checkout@v3
 
-    - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v4
-      with:
-        python-version: ${{ matrix.python-version }}
-        cache: 'pip'
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: 'pip'
 
-    - name: Install Poetry
-      uses: snok/install-poetry@v1
-      with:
-        version: ${{ env.POETRY_VERSION }}
+      - name: Install Poetry
+        uses: snok/install-poetry@v1
+        with:
+          version: ${{ env.POETRY_VERSION }}
 
-    - name: Install dependencies
-      run: |
-        pip wheel --no-cache-dir --use-pep517 ml-dtypes
-        poetry install --all-extras
+      - name: Install dependencies
+        run: |
+          pip wheel --no-cache-dir --use-pep517 ml-dtypes
+          poetry install --all-extras
 
-    - name: Set Redis image name
-      run: |
-        if [[ "${{ matrix.redis-version }}" == "8.0.2" ]]; then
-          echo "REDIS_IMAGE=redis:${{ matrix.redis-version }}" >> $GITHUB_ENV
-        else
-          echo "REDIS_IMAGE=redis/redis-stack-server:${{ matrix.redis-version }}" >> $GITHUB_ENV
-        fi
+      - name: Set Redis image name
+        run: |
+          if [[ "${{ matrix.redis-version }}" == "8.0.2" ]]; then
+            echo "REDIS_IMAGE=redis:${{ matrix.redis-version }}" >> $GITHUB_ENV
+          else
+            echo "REDIS_IMAGE=redis/redis-stack-server:${{ matrix.redis-version }}" >> $GITHUB_ENV
+          fi
 
-    - name: Run API tests
-      if: matrix.redis-version == 'latest'
-      env:
-        OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
-      run: |
-        make test-all
+      - name: Run API tests
+        if: matrix.redis-version == 'latest'
+        env:
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+        run: |
+          make test-all
 
-    - name: Run tests
-      if:  matrix.redis-version != 'latest'
-      run: |
-        make test
+      - name: Run tests
+        if: matrix.redis-version != 'latest'
+        run: |
+          make test

CLAUDE.md

@@ -2,33 +2,94 @@
 
 This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
 
+## How to Write Tests for Coverage
+
+When improving test coverage, follow these principles:
+
+1. **Focus on Integration Tests**: Write tests that use real Redis instances and test actual usage scenarios. Unit tests
+   are secondary to integration tests.
+
+2. **Test What Code SHOULD Do**: Don't write tests that mirror what the code currently does. Test against the expected
+   behavior and requirements.
+
+3. **Use Meaningful Test Names**: Test names should describe the behavior being tested, not generic names like "
+   test_function_x".
+
+4. **Research Before Writing**: Find and understand existing tests for the feature/area before adding new tests.
+
+5. **Test Error Paths and Edge Cases**: Focus on uncovered error handling, boundary conditions, and edge cases.
+
+6. **Run Tests Incrementally**: Run `make test-all` after every 5 tests to ensure no regressions.
+
+7. **Avoid "Ugly Mirror" Testing**: Don't create tests that simply verify the current implementation. Test the contract
+   and expected behavior.
+
+Example of a good integration test for error handling:
+
+```python
+def test_malformed_base64_blob_handling(redis_url: str) -> None:
+    """Test handling of malformed base64 data in blob decoding."""
+    with _saver(redis_url) as saver:
+        # Set up real scenario
+        # Test error condition
+        # Verify graceful handling
+```
+
+## CRITICAL: Always Use TestContainers for Redis
+
+**NEVER use Docker directly or manually start Redis containers!** All tests, benchmarks, and profiling scripts MUST use
+TestContainers. The library handles container lifecycle automatically.
+
+```python
+from testcontainers.redis import RedisContainer
+
+# Use redis:8 (has all required modules) or redis/redis-stack-server:latest
+redis_container = RedisContainer("redis:8")
+redis_container.start()
+try:
+    redis_url = f"redis://{redis_container.get_container_host_ip()}:{redis_container.get_exposed_port(6379)}"
+    # Use redis_url...
+finally:
+    redis_container.stop()
+```
+
 ## Development Commands
 
 ### Setup and Dependencies
 
 ```bash
 poetry install --all-extras  # Install all dependencies with poetry (from README)
-make redis-start             # Start Redis Stack container (includes RedisJSON and RediSearch)
-make redis-stop              # Stop Redis container
 ```
 
 ### Testing
 
 ```bash
-make test                    # Run tests with verbose output
-make test-all               # Run all tests including API tests
+make test-all               # PREFERRED: Run all tests including API tests when evaluating changes
+make test                   # Run tests with verbose output
+make test-coverage          # Run tests with coverage
+make coverage-report        # Show coverage report in terminal
+make coverage-html          # Generate HTML coverage report
 pytest tests/test_specific.py  # Run specific test file
 pytest tests/test_specific.py::test_function  # Run specific test
 pytest --run-api-tests      # Include API integration tests
 ```
 
+**Important**: Always use `make test-all` when evaluating changes to ensure all tests pass, including API integration
+tests.
+
+Note: Tests automatically use TestContainers for Redis - do not manually start Redis containers.
+
 ### Code Quality
 
 ```bash
 make format          # Format code with black and isort
 make lint            # Run formatting, type checking, and other linters
 make check-types     # Run mypy type checking
 make check           # Run both linting and tests
+make find-dead-code  # Find unused code with vulture
+poetry run check-format      # Check formatting without modifying
+poetry run check-sort-imports # Check import sorting
+poetry run check-lint        # Run all linting checks
 ```
 
 ### Development
@@ -37,6 +98,19 @@ make check           # Run both linting and tests
 make clean           # Remove cache and build artifacts
 ```
 
+## Code Style Guidelines
+
+- Use Black for formatting with target versions py39-py313
+- Sort imports with isort (black profile)
+- Strict typing required (disallow_untyped_defs=True)
+- Follow PEP 8 naming conventions (snake_case for functions/variables)
+- Type annotations required for all function parameters and return values
+- Explicit error handling with descriptive error messages
+- Test all functionality with both sync and async variants
+- Maintain test coverage with pytest
+- Use contextlib for resource management
+- Document public APIs with docstrings
+
 ## Architecture Overview
 
 ### Core Components
@@ -47,6 +121,8 @@ make clean           # Remove cache and build artifacts
 - `__init__.py`: `RedisSaver` - Standard sync implementation
 - `aio.py`: `AsyncRedisSaver` - Async implementation
 - `shallow.py` / `ashallow.py`: Shallow variants that store only latest checkpoint
+- `key_registry.py`: Checkpoint key registry using sorted sets for efficient write tracking
+- `scan_utils.py`: Utilities for efficient key scanning and pattern matching
 
 **Stores** (`langgraph/store/redis/`):
 
@@ -56,17 +132,26 @@ make clean           # Remove cache and build artifacts
 
 ### Key Architecture Patterns
 
-**Dual Implementation Strategy**: Each major component has both sync and async variants that share common base classes. The base classes (`BaseRedisSaver`, `BaseRedisStore`) contain the bulk of the business logic, while concrete implementations handle Redis client management and specific I/O patterns.
+**Dual Implementation Strategy**: Each major component has both sync and async variants that share common base classes.
+The base classes (`BaseRedisSaver`, `BaseRedisStore`) contain the bulk of the business logic, while concrete
+implementations handle Redis client management and specific I/O patterns.
 
-**Redis Module Dependencies**: The library requires RedisJSON and RediSearch modules. Redis 8.0+ includes these by default; earlier versions need Redis Stack. All operations use structured JSON storage with search indices for efficient querying.
+**Redis Module Dependencies**: The library requires RedisJSON and RediSearch modules. Redis 8.0+ includes these by
+default; earlier versions need Redis Stack. All operations use structured JSON storage with search indices for efficient
+querying.
 
-**Schema-Driven Indexing**: Both checkpoints and stores use predefined schemas (`SCHEMAS` constants) that define Redis Search indices. Checkpoint indices track thread/namespace/version hierarchies; store indices support both key-value lookup and optional vector similarity search.
+**Schema-Driven Indexing**: Both checkpoints and stores use predefined schemas (`SCHEMAS` constants) that define Redis
+Search indices. Checkpoint indices track thread/namespace/version hierarchies; store indices support both key-value
+lookup and optional vector similarity search.
 
-**TTL Integration**: Native Redis TTL support is integrated throughout, with configurable defaults and refresh-on-read capabilities. TTL applies to all related keys (main document, vectors, writes) atomically.
+**TTL Integration**: Native Redis TTL support is integrated throughout, with configurable defaults and refresh-on-read
+capabilities. TTL applies to all related keys (main document, vectors, writes) atomically.
 
-**Cluster Support**: Full Redis Cluster support with automatic detection and cluster-aware operations (individual key operations vs. pipelined operations).
+**Cluster Support**: Full Redis Cluster support with automatic detection and cluster-aware operations (individual key
+operations vs. pipelined operations).
 
-**Type System**: Heavy use of generics (`BaseRedisSaver[RedisClientType, IndexType]`) to maintain type safety across sync/async variants while sharing implementation code.
+**Type System**: Heavy use of generics (`BaseRedisSaver[RedisClientType, IndexType]`) to maintain type safety across
+sync/async variants while sharing implementation code.
 
 ### Redis Key Patterns
 
@@ -82,10 +167,36 @@ Tests are organized by functionality:
 
 - `test_sync.py` / `test_async.py`: Core checkpoint functionality
 - `test_store.py` / `test_async_store.py`: Store operations
-- `test_cluster_mode.py`: Redis Cluster specific tests
-- `test_*_ttl.py`: TTL functionality
-- `test_key_parsing.py`: Key generation and parsing logic
+- `test_cluster_mode.py` / `test_async_cluster_mode.py`: Redis Cluster specific tests
+- `test_checkpoint_ttl.py`: TTL functionality for checkpoints
+- `test_key_parsing.py` / `test_subgraph_key_parsing.py`: Key generation and parsing logic
 - `test_semantic_search_*.py`: Vector search capabilities
+- `test_interruption.py` / `test_streaming*.py`: Advanced workflow tests
+- `test_shallow_*.py`: Shallow checkpoint implementation tests
+- `test_decode_responses.py`: Redis response decoding tests
+- `test_crossslot_integration.py`: Cross-slot operation tests
+
+## Notebooks and Examples
+
+The `examples/` directory contains Jupyter notebooks demonstrating Redis integration with LangGraph:
+
+- All notebooks MUST use Redis implementations (RedisSaver, RedisStore), not in-memory equivalents
+- Notebooks can be run via Docker Compose: `cd examples && docker compose up`
+- Each notebook includes installation of required dependencies within the notebook cells
+- TestContainers should be used for any new notebook examples requiring Redis
+
+### Running Notebooks
+
+1. With Docker (recommended):
+   ```bash
+   cd examples
+   docker compose up
+   ```
+
+2. Locally:
+    - Ensure Redis is running with required modules (RedisJSON, RediSearch)
+    - Install dependencies: `pip install langgraph-checkpoint-redis jupyter`
+    - Run: `jupyter notebook`
 
 ### Important Dependencies