Phase 3: Fields and References Async Support (#3)

* Add Phase 3 planning: Fields and References async support

- Detailed implementation plan for ReferenceField async support
- AsyncReferenceProxy design for lazy loading in async contexts
- LazyReferenceField async fetch methods
- GridFS async operations for file handling
- Cascade operations async support
- Comprehensive usage examples and testing strategy

🤖 Generated with Claude Code (https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* Update GridFS implementation to use PyMongo's native async API

- Replace Motor references with PyMongo's gridfs.asynchronous module
- Use AsyncGridFSBucket from gridfs.asynchronous
- Add note about PyMongo's built-in async support
- Include example of direct PyMongo GridFS usage

Thanks to user feedback about current PyMongo API

🤖 Generated with Claude Code (https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* Add PyMongo GridFS async API tutorial documentation

- Comprehensive guide for PyMongo's native GridFS async support
- Covers AsyncGridFS and AsyncGridFSBucket usage
- Includes practical examples and best practices
- Documents key differences between legacy and modern APIs

This documentation supports Phase 3 implementation.

🤖 Generated with Claude Code (https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* Implement Phase 3: Fields and References async support

- ReferenceField async support with AsyncReferenceProxy
  - Async dereferencing via fetch() method
  - Connection type detection in __get__
  - _async_lazy_load_ref() for database lookups

- LazyReferenceField async_fetch() method
  - Async version of fetch() for manual dereferencing
  - Maintains compatibility with sync behavior

- GridFS async operations
  - AsyncGridFSProxy for file operations
  - FileField async_put() and async_get() methods
  - Support for file metadata and custom collections
  - Streaming support for large files

- Comprehensive test coverage
  - 8 tests for async reference fields
  - 9 tests for async GridFS operations
  - All tests passing with proper error handling

Known limitations:
- ListField with ReferenceField doesn't auto-convert to AsyncReferenceProxy
- This is tracked as a future enhancement

🤖 Generated with Claude Code (https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* docs: Update progress tracking for Phase 3 completion

- Mark all Phase 3 tasks as completed in PROGRESS_FIELDS.md
- Update PROGRESS.md with Phase 3 completion summary
- Document deferred items moved to Phase 4
- Note known limitations and test status

* chore: Remove PROGRESS_FIELDS.md after Phase 3 completion

Phase 3 has been successfully completed with all tasks either done or deferred to Phase 4

* docs: Add Phase 3 learnings and GridFS tutorial

- Add Phase 3 implementation learnings to CLAUDE.md
- Include GridFS async tutorial documentation
- Document design decisions and known limitations
- Provide guidance for future development

---------

Co-authored-by: Claude <[email protected]>

Author: jeonghanjoo

CLAUDE.md

@@ -235,4 +235,56 @@ When working on async support implementation, follow this workflow:
 #### Migration Strategy
 - Projects can use both sync and async QuerySets in same codebase
 - Connection type determines which methods are available
-- Clear error messages guide users to correct method usage
+- Clear error messages guide users to correct method usage
+
+### Phase 3 Implementation Learnings
+
+#### ReferenceField Async Design
+- **AsyncReferenceProxy Pattern**: In async context, ReferenceField returns a proxy object requiring explicit `await proxy.fetch()`
+- This prevents accidental sync operations in async code and makes async dereferencing explicit
+- The proxy caches fetched values to avoid redundant database calls
+
+#### Field-Level Async Methods
+- Async methods should be on the field class, not on proxy instances:
+  ```python
+  # Correct - call on field class
+  await AsyncFileDoc.file.async_put(file_obj, instance=doc)
+  
+  # Incorrect - don't call on proxy instance
+  await doc.file.async_put(file_obj)
+  ```
+- This pattern maintains consistency and avoids confusion with instance methods
+
+#### GridFS Async Implementation
+- Use PyMongo's native `gridfs.asynchronous` module instead of Motor
+- Key imports: `from gridfs.asynchronous import AsyncGridFSBucket`
+- AsyncGridFSProxy handles async file operations (read, delete, replace)
+- File operations return sync GridFSProxy for storage in document to maintain compatibility
+
+#### LazyReferenceField Enhancement
+- Added `async_fetch()` method directly to LazyReference class
+- Maintains same caching behavior as sync version
+- Works seamlessly with existing passthrough mode
+
+#### Error Handling Patterns
+- GridFS operations need careful error handling for missing files
+- Stream position management critical for file reads (always seek(0) after write)
+- Grid ID extraction from different proxy types requires type checking
+
+#### Testing Async Fields
+- Use separate test classes for each field type for clarity
+- Test both positive cases and error conditions
+- Always clean up GridFS collections in teardown to avoid test pollution
+- Verify proxy behavior separately from actual async operations
+
+#### Known Limitations
+- **ListField with ReferenceField**: Currently doesn't auto-convert to AsyncReferenceProxy
+  - This is a complex case requiring deeper changes to ListField
+  - Documented as limitation - users need manual async dereferencing for now
+  - Could be addressed in future enhancement
+
+#### Design Decisions
+- **Explicit over Implicit**: Async dereferencing must be explicit via `fetch()` method
+- **Proxy Pattern**: Provides clear indication when async operation needed
+- **Field-Level Methods**: Consistency with sync API while maintaining async safety
+- **Native PyMongo**: Leverage PyMongo's built-in async support rather than external libraries

PROGRESS.md

@@ -153,12 +153,12 @@ async def async_run_in_transaction():
 - [x] async_create(), async_update(), async_delete() 벌크 작업
 - [x] 비동기 커서 관리 및 최적화
 
-### Phase 3: 필드 및 참조 (2-3주)
-- [ ] ReferenceField에 async_fetch() 메서드 추가
-- [ ] AsyncReferenceProxy 구현
-- [ ] LazyReferenceField 비동기 지원
-- [ ] GridFS 비동기 작업 (async_put, async_get)
-- [ ] 캐스케이드 작업 비동기화
+### Phase 3: 필드 및 참조 (2-3주) ✅ **완료** (2025-07-31)
+- [x] ReferenceField에 async_fetch() 메서드 추가
+- [x] AsyncReferenceProxy 구현
+- [x] LazyReferenceField 비동기 지원
+- [x] GridFS 비동기 작업 (async_put, async_get)
+- [ ] 캐스케이드 작업 비동기화 (Phase 4로 이동)
 
 ### Phase 4: 고급 기능 (3-4주)
 - [ ] 하이브리드 신호 시스템 구현
@@ -304,4 +304,32 @@ author = await post.author.async_fetch()
 - `async_aggregate()`, `async_distinct()` - 고급 집계 기능
 - `async_values()`, `async_values_list()` - 필드 프로젝션
 - `async_explain()`, `async_hint()` - 쿼리 최적화
-- 이들은 기본 인프라 구축 후 필요시 추가 가능
+- 이들은 기본 인프라 구축 후 필요시 추가 가능
+
+### Phase 3: Fields and References Async Support (2025-07-31 완료)
+
+#### 구현 내용
+- **ReferenceField 비동기 지원**: AsyncReferenceProxy 패턴으로 안전한 비동기 참조 처리
+- **LazyReferenceField 개선**: LazyReference 클래스에 async_fetch() 메서드 추가
+- **GridFS 비동기 작업**: PyMongo의 native async API 사용 (gridfs.asynchronous.AsyncGridFSBucket)
+- **필드 레벨 비동기 메서드**: async_put(), async_get(), async_read(), async_delete(), async_replace()
+
+#### 주요 성과
+- 17개 새로운 async 테스트 추가 (참조: 8개, GridFS: 9개)
+- 총 54개 async 테스트 모두 통과 (Phase 1: 23, Phase 2: 14, Phase 3: 17)
+- PyMongo의 native GridFS async API 완벽 통합
+- 명시적 async dereferencing으로 안전한 참조 처리
+
+#### 기술적 세부사항
+- AsyncReferenceProxy 클래스로 async context에서 명시적 fetch() 필요
+- FileField.__get__이 GridFSProxy 반환, async 메서드는 field class에서 호출
+- async_read() 시 stream position reset 처리
+- GridFSProxy 인스턴스에서 grid_id 추출 로직 개선
+
+#### 알려진 제한사항
+- ListField 내 ReferenceField는 AsyncReferenceProxy로 자동 변환되지 않음
+- 이는 low priority로 문서화되어 있으며 필요시 향후 개선 가능
+
+#### Phase 4로 이동된 항목
+- 캐스케이드 작업 (CASCADE, NULLIFY, PULL, DENY) 비동기화
+- 복잡한 참조 관계의 비동기 처리

mongoengine/async_utils.py

@@ -1,5 +1,7 @@
 """Async utility functions for MongoEngine async support."""
 
+import contextvars
+
 from mongoengine.connection import (
     DEFAULT_CONNECTION_NAME,
     ConnectionFailure,
@@ -10,6 +12,9 @@
     is_async_connection,
 )
 
+# Context variable for async sessions
+_async_session_context = contextvars.ContextVar('mongoengine_async_session', default=None)
+
 
 async def get_async_collection(collection_name, alias=DEFAULT_CONNECTION_NAME):
     """Get an async collection for the given name and alias.
@@ -49,6 +54,22 @@ def ensure_sync_connection(alias=DEFAULT_CONNECTION_NAME):
         )
 
 
+async def _get_async_session():
+    """Get the current async session if any.
+    
+    :return: Current async session or None
+    """
+    return _async_session_context.get()
+
+
+async def _set_async_session(session):
+    """Set the current async session.
+    
+    :param session: The async session to set
+    """
+    _async_session_context.set(session)
+
+
 async def async_exec_js(code, *args, **kwargs):
     """Execute JavaScript code asynchronously in MongoDB.
     
@@ -94,4 +115,6 @@ async def __aexit__(self, exc_type, exc_val, exc_tb):
     'ensure_sync_connection',
     'async_exec_js',
     'AsyncContextManager',
+    '_get_async_session',
+    '_set_async_session',
 ]

mongoengine/base/datastructures.py

@@ -11,6 +11,7 @@
     "BaseList",
     "EmbeddedDocumentList",
     "LazyReference",
+    "AsyncReferenceProxy",
 )
 
 
@@ -472,3 +473,35 @@ def __getattr__(self, name):
 
     def __repr__(self):
         return f"<LazyReference({self.document_type}, {self.pk!r})>"
+    
+    async def async_fetch(self, force=False):
+        """Async version of fetch()."""
+        if not self._cached_doc or force:
+            self._cached_doc = await self.document_type.objects.async_get(pk=self.pk)
+            if not self._cached_doc:
+                raise DoesNotExist("Trying to dereference unknown document %s" % (self))
+        return self._cached_doc
+
+
+class AsyncReferenceProxy:
+    """Proxy object for async reference field access.
+    
+    This proxy is returned when accessing a ReferenceField in an async context,
+    requiring explicit async dereferencing via fetch() method.
+    """
+    
+    __slots__ = ("field", "instance", "_cached_value")
+    
+    def __init__(self, field, instance):
+        self.field = field
+        self.instance = instance
+        self._cached_value = None
+    
+    async def fetch(self):
+        """Explicitly fetch the referenced document."""
+        if self._cached_value is None:
+            self._cached_value = await self.field.async_fetch(self.instance)
+        return self._cached_value
+    
+    def __repr__(self):
+        return f"<AsyncReferenceProxy: {self.field.name} (unfetched)>"