Skip to content

Conversation

@TexasCoding
Copy link
Owner

Full Async Re-Write

TexasCoding and others added 30 commits July 30, 2025 18:32
- Add AsyncProjectX client with full async/await support
- Implement HTTP/2 enabled httpx client with connection pooling
- Add comprehensive error handling with exponential backoff retry logic
- Create basic async methods: authenticate, get_positions, get_instrument, get_health_status
- Add complete test suite with 9 passing tests
- Update dependencies: add httpx[http2] for async HTTP support
- Add example async usage script
- Update CHANGELOG with version 1.2.0 changes

This completes Phase 1 of the async refactoring plan from issue #12
- Implement AsyncRateLimiter with sliding window algorithm
- Add account management methods: list_accounts, search_open_positions
- Implement market data retrieval: get_bars with timezone conversion and caching
- Add instrument search: search_instruments with live filter support
- Implement trade history: search_trades with date range filtering
- Add 5 new comprehensive tests for all new methods
- All 14 async client tests now passing

Phase 2 features:
- Rate limiting prevents API overload
- Market data caching reduces redundant API calls
- Timezone-aware data processing for all timestamps
- Complete async/await patterns for all operations

This completes Phase 2 of the async refactoring plan from issue #12
- Created AsyncOrderManager with comprehensive async order operations
- Supports market, limit, stop, and bracket orders with async/await
- Implements async-safe locking for thread safety
- Adds order search, modification, and cancellation functionality
- Includes full test suite with 12 passing tests
- Fixed deadlock issues in bracket orders by removing nested locks
- Added stub for AsyncProjectXRealtimeClient (to be implemented in Phase 4)
- Created example script demonstrating async order manager usage

This completes the OrderManager conversion to async as part of Phase 3 of
the async refactoring plan from issue #12.
- Created AsyncPositionManager with complete async/await support
- Implemented all position tracking and management operations
- Added async portfolio P&L calculation and risk metrics
- Converted position closure operations (direct, partial, bulk) to async
- Implemented async position monitoring with alerts
- Full test suite with 17 passing tests covering all functionality
- Proper validation for ProjectX position payload formats
- Async-safe operations with asyncio locks
- Added comprehensive example usage script

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
…derBook

- Add AsyncRealtimeDataManager with full async/await support
  - Multi-timeframe OHLCV data management
  - Real-time tick processing with async callbacks
  - Memory management with automatic cleanup
  - Comprehensive test suite (16 tests)

- Add AsyncOrderBook with async market depth analysis
  - Level 2 market depth processing
  - Trade flow analysis and iceberg detection
  - Async-safe operations with asyncio.Lock
  - Proper timezone handling for Polars DataFrames
  - Comprehensive test suite (17 tests)

- Update __init__.py exports for new async components
- Add example usage scripts for both components

Part of async/await refactoring (#12)
…ry functions

- Implemented AsyncProjectXRealtimeClient with full async/await support
  - Created async wrapper around synchronous SignalR client
  - Added support for both sync and async callbacks
  - Implemented non-blocking event forwarding with asyncio.create_task()
  - Proper JWT token refresh and reconnection support
  - Thread-safe operations using asyncio.Lock
  - Runs synchronous SignalR operations in executor

- Created comprehensive async factory functions:
  - create_async_client() - AsyncProjectX client creation
  - create_async_realtime_client() - Real-time WebSocket client
  - create_async_order_manager() - Order management
  - create_async_position_manager() - Position tracking
  - create_async_data_manager() - OHLCV data management
  - create_async_orderbook() - Market depth analysis
  - create_async_trading_suite() - Complete trading toolkit

- Added example scripts demonstrating:
  - AsyncProjectXRealtimeClient usage
  - Integrated trading suite with shared connections
  - Factory function usage patterns

- Created comprehensive test suite (20 tests) for AsyncProjectXRealtimeClient
- Updated async_refactoring_issue.md with completion status

All async managers now share a single AsyncProjectXRealtimeClient instance,
ensuring efficient resource usage and coordinated event handling.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
- Created comprehensive async test suites covering all major components
- test_async_comprehensive.py: Core AsyncProjectX client functionality
- test_async_integration_comprehensive.py: End-to-end async workflows
- test_async_order_manager_comprehensive.py: Async order management testing
- test_async_utils_comprehensive.py: Utility functions in async context

Tests demonstrate:
- Concurrent API operations and performance benefits
- Error handling in async contexts
- Compatibility between sync and async components
- Rate limiting and resource management
- Thread safety in async environments

Completes Phase 5 async refactoring: comprehensive test coverage
ensuring reliability of async/await patterns across the SDK.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
This commit resolves multiple issues with the async real-time client to make it
work as reliably as the sync version:

WebSocket Event Loop Fixes:
- Store event loop reference in AsyncProjectXRealtimeClient constructor
- Use asyncio.run_coroutine_threadsafe() for cross-thread event scheduling
- Fix 'no running event loop' errors when callbacks fire from SignalR threads

Historical Data Retrieval:
- Change from incorrect GET /market/bars to POST /History/retrieveBars endpoint
- Add column renaming from ProjectX short format (t,o,h,l,c,v) to full names
- Match sync client's API usage patterns exactly

Callback Structure Alignment:
- Update callback signatures from tuple args to dict format
- Pass data as {'contract_id': id, 'data': data} to match sync version
- Fix data extraction to use callback_data.get('data', {})

Real-time Data Processing:
- Copy exact bar time calculation logic from sync version
- Add _update_timeframe_data method matching sync architecture
- Fix timestamp advancement for proper bar updates
- Ensure bars advance to new timeframe boundaries correctly

SignalR Method Calls:
- Replace deprecated invoke() with send() for hub method calls
- Update subscribe/unsubscribe methods to use correct SignalR API

Testing and Examples:
- Remove temporary 20-second timeout from examples
- Set logging to INFO level for cleaner output
- Verify all async examples work correctly

The async client now provides the same reliability and functionality as the
sync version, with proper event loop handling for WebSocket callbacks.

Closes #12
…tions

- Fixed SignalR subscription parameter formats in AsyncProjectXRealtimeClient
  - SubscribeAccounts now sends empty list instead of account ID
  - Other subscriptions now send list with int account ID
- Fixed order data extraction in AsyncOrderManager._on_order_update
  - Handle nested SignalR format: {'action': 1, 'data': {...}}
  - Extract actual order data from nested structure
- Updated async_02_order_management.py example
  - Pass realtime_client to order_manager.initialize()
  - Script no longer hangs waiting for real-time cache
  - Order modifications now properly reflected in cache
- Changed debug logging to info level for received events

The async order management now works correctly with real-time updates,
properly tracking orders through their lifecycle without hanging.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
- Add point_value parameter to calculate_position_pnl for accurate dollar P&L
- Fix market data retrieval to use real-time data manager when available
- Remove hardcoded fallback price that caused unrealistic price jumps
- Use entry price for P&L when no market data available (shows /bin/zsh P&L)
- Fix get_data -> get_bars method name for AsyncProjectX
- Fix account_id type conversion for realtime client
- Add proper error handling and informative messages for data availability
- Change create_async_realtime_data_manager to create_async_data_manager
- This matches the actual export from __init__.py
…ce updates

- Fix AsyncRealtimeDataManager to properly append tick data to current_tick_data
- This allows get_current_price() to return the latest real-time price
- Without this fix, get_current_price() would always fall back to historical bar data
- Add call to start_realtime_feed() after initialize() in async example
- Add info logging to debug quote/trade update callbacks
- This ensures market data subscriptions are activated
- Matches the pattern used in sync version
…bose logging

- Fixed async_03_position_management.py to properly demonstrate position lifecycle
  - Added actual position opening with 1 MNQ contract for testing
  - Fixed P&L calculations using instrument tick values (tickSize, tickValue)
  - Fixed real-time price updates by properly starting the feed
  - Fixed AsyncRealtimeDataManager to append tick data for get_current_price()
  - Used entry price fallback when no market data available
  - Cleaned up import (removed unused Optional)

- Reduced verbose logging across async components
  - Changed quote/trade update logs from INFO to DEBUG in AsyncRealtimeDataManager
  - Changed general event received logs from INFO to DEBUG in AsyncProjectXRealtimeClient
  - Output is now much cleaner, showing only important status messages

The position management example now properly:
1. Opens a test position with user confirmation
2. Monitors the position with accurate P&L calculations
3. Shows real-time price updates when market is open
4. Closes positions at the end of the demo
5. Displays comprehensive position statistics
…mple

- Fixed async reentrancy deadlock in _process_trade method
- _process_trade was calling get_best_bid_ask() while already holding orderbook_lock
- Now directly accesses orderbook data since it's already inside the lock
- Added comprehensive async_05_orderbook_analysis.py example matching sync version
- Fixed quote update handler to use correct 'contract_id' field name
- Updated _symbol_matches_instrument to match sync version's simplified logic
- All async orderbook functionality now working correctly with real-time data

The deadlock occurred because process_market_depth holds the lock while processing
trades, and _process_trade tried to acquire the same lock via get_best_bid_ask().

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
…ok_snapshot

- get_orderbook_snapshot was calling async methods while holding orderbook_lock
- Fixed by directly accessing orderbook data instead of calling:
  - get_orderbook_bids()
  - get_orderbook_asks()
  - get_best_bid_ask()
- All these methods try to acquire the same lock, causing deadlock
- Now the async orderbook example runs without hanging
- Changed logging level from INFO to DEBUG for frequent update messages

This completes the async orderbook implementation with all deadlock issues resolved.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
…g fixes

Major refactoring of async orderbook implementation:

- Moved from single file (async_orderbook.py) to modular structure:
  - base.py: Core orderbook functionality and data structures
  - analytics.py: Market analytics and liquidity analysis
  - detection.py: Order detection algorithms (iceberg, clusters)
  - profile.py: Volume profile and support/resistance analysis
  - realtime.py: Real-time WebSocket data handling
  - memory.py: Memory management and cleanup strategies
  - types.py: Type definitions and configuration

- Fixed all critical issues:
  - Resolved deadlock in get_orderbook_depth() by using unlocked methods
  - Fixed detect_order_clusters() return type mismatch
  - Fixed iceberg detection field name (last_update vs last_seen)
  - Added missing get_order_type_statistics() method
  - Fixed cumulative delta calculations
  - Fixed trade flow summary calculations

- Fixed all type annotation and linting issues:
  - Added proper type hints throughout all modules
  - Fixed Polars aggregation type conversions
  - Added missing return type annotations
  - Fixed generic dict type parameters
  - Resolved all mypy and ruff errors

- Updated example async_05_orderbook_analysis.py:
  - Fixed all method calls to match new implementation
  - Added proper error handling
  - Demonstrates all 21 async orderbook methods

This completes the async orderbook refactoring with 100% linting compliance
and all functionality working correctly.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
- Fix import errors: BBANDS instead of BollingerBands, STOCH instead of Stochastic
- Fix column name case sensitivity: indicators create lowercase columns (rsi_14, sma_20)
- Fix AttributeError: use get_bars() instead of get_data() for AsyncProjectX
- Fix data manager method: use get_data() instead of get_bars_for_timeframe()
- Add proper type hints for Polars DataFrames
- Improve error handling and add debug output for real-time monitoring
- Fix account_info null checks and type conversions
- Update to use instrument.id instead of activeContract

The async technical indicators example now works properly with real-time
indicator calculations across multiple timeframes (5sec, 1min, 5min)
…atic cleanup

- Create async_08_order_and_position_tracking.py demonstrating order cleanup
- Uses proper async components (AsyncOrderManager, AsyncPositionManager, etc)
- Fix AsyncOrderBook initialization parameters in create_async_trading_suite
- Demonstrates automatic order cancellation when positions close
- Includes proper async/await patterns and concurrent operations
- Add graceful shutdown handling with cleanup

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
- Create async_09_get_check_available_instruments.py using AsyncProjectX
- Demonstrates async search_instruments() and get_instrument() methods
- Includes non-blocking user input with asyncio executor
- Shows background performance stats monitoring
- Proper async/await patterns throughout
- Interactive loop for testing different instrument symbols
- Change '1hour' to '1hr' and '4hour' to '4hr' to match valid timeframes
- Update timeframe references in create_async_trading_suite call
- Update get_data() calls to use correct timeframe format
- Fixes ValueError: Invalid timeframe error
- Replace setup_logging(__name__) with logging.getLogger(__name__)
- Add missing logging import
- Remove unused setup_logging import
- Fixes AttributeError: module 'logging' has no attribute '__MAIN__'
- Replace setup_logging() with logging.basicConfig() and getLogger()
- Convert account_id to string as required by create_async_trading_suite
- Fixes NameError and type mismatch issues
…ame strategy

- Fix SMA calls to use 'column' parameter instead of 'column_name'
- Use proper indicator output column names (SMA_N, RSI_N)
- Replace get_order_imbalance with get_market_imbalance
- Use get_best_bid_ask instead of get_bid_ask_spread
- Handle portfolio_pnl dict format correctly
- Add safe defaults for orderbook data
- Remove all column_name parameters from SMA and RSI calls
- Update RSI column reference from 'rsi' to 'rsi_14'
- Indicators now use their default column naming (sma_N, rsi_N)
- Change SMA_10 and SMA_20 to lowercase sma_10 and sma_20
- Matches actual column names created by indicators
- Fixes 'SMA_10 not found' error
- Add check for CompletionMessage type in _on_connection_error
- Log as debug instead of error for normal protocol messages
- Only increment error counter for actual errors
- Fixes spurious 'Market hub error' messages in logs
- Add 'live' parameter to AsyncProjectX.get_instrument() for sync compatibility
- Create migration plan documentation (sync_to_async_migration_plan.md)
- Document migration gaps and action items (async_migration_gaps.md)
- Identify missing methods and API differences
- All async examples verified working
BREAKING CHANGE: Complete removal of synchronous API in favor of async-only implementation

- Remove all sync implementation files (client.py, order_manager.py, etc.)
- Remove all sync examples
- Remove all sync test files
- Update __init__.py to export async classes with original names
- Update all examples to use new import names (ProjectX instead of AsyncProjectX)
- Update factory functions to be async-only
- Update README.md with async-first documentation and migration guide
- Set version to 2.0.0

This is a major breaking change. All APIs now require async/await patterns.
See README.md for migration guide from v1.x to v2.0.0.
@TexasCoding TexasCoding merged commit 42db26d into main Aug 1, 2025
2 checks passed
@claude
Copy link

claude bot commented Aug 1, 2025

Claude finished @TexasCoding's task —— View job


PR Review: Full Async Re-Write

Comprehensive Code Review Analysis

This is an exceptionally well-executed async transformation of the entire codebase. Here's my detailed analysis:

Code Quality & Architecture - Excellent

Consistent async patterns throughout:

  • Proper async/await usage: All I/O operations correctly use await
  • Thread-safe operations: Comprehensive use of asyncio.Lock() for shared state
  • Clean separation of concerns: Modular design with specialized async managers
  • Smart factory functions: Excellent use of dependency injection patterns

Notable architecture improvements:

  • HTTP/2 support via httpx for better performance
  • Proper async context managers (async with) for resource cleanup
  • Connection pooling and rate limiting implemented correctly

Error Handling & Resource Management - Robust

Comprehensive error handling patterns found:

# Example from async_client.py:403-413
if response.status_code == 429:
    if retry_count < self.config.retry_attempts:
        retry_after = int(response.headers.get("Retry-After", "5"))
        await asyncio.sleep(retry_after)
        return await self._make_request(...)
    raise ProjectXRateLimitError("Rate limit exceeded after retries")

Strong points:

  • Granular exception handling: Specific catches for timeouts, auth errors, rate limits
  • Retry logic with backoff: Exponential backoff for failed requests
  • Resource cleanup: Proper cleanup in except blocks and context managers
  • Async-safe locking: All shared state protected with async locks

Test Coverage - Comprehensive

Excellent test infrastructure:

  • 11 dedicated async test files covering all components
  • Integration testing: End-to-end scenarios with real async workflows
  • Proper async testing: Using pytest-asyncio, AsyncMock, aioresponses
  • Edge case coverage: Error conditions, timeouts, connection failures

Performance Considerations - Well Optimized

Smart performance patterns:

  • Concurrent operations: asyncio.gather() for parallel API calls
  • Connection pooling: Reused HTTP connections reduce overhead
  • Memory management: Sliding windows in OrderBook, cleanup routines
  • Caching strategies: Instrument data cached to reduce API calls

Benchmark infrastructure added: benchmarks/async_vs_sync_benchmark.py for measuring improvements

🔒 Security Analysis - Secure

No security concerns identified:

  • No hardcoded credentials: Uses environment variables correctly
  • JWT token handling: Proper token refresh and secure URL construction
  • No sensitive data logging: Tokens not exposed in log messages
  • Input validation: API responses properly validated before processing

Token management example:

# JWT tokens passed via query params (standard SignalR pattern)
self.user_hub_url = f"{final_user_url}?access_token={jwt_token}"

Documentation & Examples - Excellent

Comprehensive migration support:

  • Migration guide: ASYNC_MIGRATION_GUIDE.md with before/after examples
  • Updated examples: All 9 examples converted to async patterns
  • Clear documentation: Extensive docstrings with usage examples
  • Breaking changes: Properly documented in changelog

🚨 Minor Issues & Recommendations

  1. Rate limiter improvements: The AsyncRateLimiter class could benefit from sliding window cleanup optimization
  2. Memory cleanup timing: Consider more aggressive cleanup intervals for long-running applications
  3. Error context: Some generic except Exception blocks could provide more context

📊 Overall Assessment

Rating: 9.5/10 - This is a masterclass in async migration

Strengths:

  • ✅ Clean, consistent async patterns throughout
  • ✅ Comprehensive error handling and recovery
  • ✅ Excellent test coverage and documentation
  • ✅ Smart performance optimizations
  • ✅ Security best practices followed
  • ✅ Breaking changes properly handled (v2.0.0)

This PR represents a complete, professional async transformation that maintains API consistency while dramatically improving performance and scalability.


@TexasCoding TexasCoding deleted the TexasCoding/issue12 branch August 1, 2025 20:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants