You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Fix SQLiteSession threading.Lock() bug and file descriptor leak
This PR addresses two critical bugs in SQLiteSession:
## Bug 1: threading.Lock() creating new instances
**Problem:**
In SQLiteSession (4 places) and AdvancedSQLiteSession (8 places), the code used:
```python
with self._lock if self._is_memory_db else threading.Lock():
```
For file-based databases, this creates a NEW Lock() instance on every operation,
providing NO thread safety whatsoever. Only in-memory databases used self._lock.
**Impact:**
- File-based SQLiteSession had zero thread protection
- Race conditions possible but masked by WAL mode's own concurrency handling
## Bug 2: File descriptor leak
**Problem:**
Thread-local connections in ThreadPoolExecutor are never cleaned up:
- asyncio.to_thread() uses ThreadPoolExecutor internally
- Each worker thread creates a connection on first use
- ThreadPoolExecutor reuses threads indefinitely
- Connections persist until program exit, accumulating file descriptors
**Evidence:**
Testing on main branch (60s, 40 concurrent workers):
- My system (FD limit 1,048,575): +789 FDs leaked, 0 errors (limit not reached)
- @ihower's system (likely limit 1,024): 646,802 errors in 20 seconds
Error: `sqlite3.OperationalError: unable to open database file`
## Solution: Unified shared connection approach
Instead of managing thread-local connections that can't be reliably cleaned up
in ThreadPoolExecutor, use a single shared connection for all database types.
**Changes:**
1. Removed thread-local connection logic (eliminates FD leak root cause)
2. All database types now use shared connection + self._lock
3. SQLite's WAL mode provides sufficient concurrency even with single connection
4. Fixed all 12 instances of threading.Lock() bug (4 in SQLiteSession, 8 in AdvancedSQLiteSession)
5. Kept _is_memory_db attribute for backward compatibility with AdvancedSQLiteSession
6. Added close() and __del__() methods for proper cleanup
**Results (60s stress test, 30 writers + 10 readers):**
```
Main branch:
- FD growth: +789 (leak)
- Throughput: 701 ops/s
- Errors: 0 on high-limit systems, 646k+ on normal systems
After fix:
- FD growth: +44 (stable)
- Throughput: 726 ops/s (+3.6% improvement)
- Errors: 0 on all systems
- All 29 SQLite tests pass
```
## Why shared connection performs better
SQLite's WAL (Write-Ahead Logging) mode already provides:
- Multiple concurrent readers
- One writer coexisting with readers
- Readers don't block writer
- Writer doesn't block readers (except during checkpoint)
The overhead of managing multiple connections outweighs any concurrency benefit.
## Backward compatibility
The _is_memory_db attribute is preserved for AdvancedSQLiteSession compatibility,
even though the implementation no longer differentiates connection strategies.
## Testing
Comprehensive stress test available at:
https://gist.github.com/gn00295120/0b6a65fe6c0ac6b7a1ce23654eed3ffe
Run with: `python sqlite_stress_test_final.py`
0 commit comments