Skip to content

Commit fed055c

Browse files
committed
chore: remove deprecated docker-compose and environment configuration files
1 parent 01caf14 commit fed055c

File tree

7 files changed

+17
-399
lines changed

7 files changed

+17
-399
lines changed

README.md

Lines changed: 1 addition & 294 deletions
Original file line numberDiff line numberDiff line change
@@ -148,297 +148,4 @@
148148
- If port 5678 is in use, specify a different port with `--debug-port`
149149
- Ensure VS Code Python extension is installed
150150
- Check that breakpoints are set in the correct files
151-
- Verify the debug server shows "Debug server listening on port 5678"
152-
153-
154-
# Dual-Write System: MongoDB to PostgreSQL
155-
156-
## 🎯 Overview
157-
158-
This feature implements a comprehensive dual-write system that ensures all data written to MongoDB is automatically synchronized to PostgreSQL. The system is designed to enable future migration from MongoDB to PostgreSQL with minimal operational risk and code changes.
159-
160-
## ✨ Key Features
161-
162-
- **Dual-Write Operations**: Every MongoDB write is automatically mirrored to PostgreSQL
163-
- **Synchronous Operations**: Immediate consistency with both databases
164-
- **Comprehensive Error Handling**: Automatic retry mechanisms and failure tracking
165-
- **Future-Ready Architecture**: Abstract repository pattern for seamless database switching
166-
- **Real-time Monitoring**: Health checks and sync metrics
167-
- **Production Ready**: Includes Docker setup, migrations, and comprehensive documentation
168-
169-
## 🏗️ Architecture
170-
171-
```
172-
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
173-
│ Application │───▶│ Dual-Write │───▶│ PostgreSQL │
174-
│ (MongoDB) │ │ Service │ │ Database │
175-
└─────────────────┘ └──────────────────┘ └─────────────────┘
176-
```
177-
178-
## 🚀 Quick Start
179-
180-
### 1. Prerequisites
181-
182-
- Python 3.9+
183-
- PostgreSQL 15+
184-
- MongoDB 7+ (existing)
185-
186-
### 2. Setup Development Environment
187-
188-
```bash
189-
# Start required services
190-
docker-compose -f docker-compose.dev.yml up -d
191-
192-
# Install dependencies
193-
pip install -r requirements.txt
194-
195-
# Copy environment configuration
196-
cp env.example .env
197-
# Edit .env with your configuration
198-
199-
# Run Django migrations
200-
python manage.py migrate
201-
```
202-
203-
### 3. Basic Usage
204-
205-
```python
206-
from todo.services.enhanced_dual_write_service import EnhancedDualWriteService
207-
208-
# Initialize the service
209-
dual_write_service = EnhancedDualWriteService()
210-
211-
# Create a document (automatically syncs to both databases)
212-
success = dual_write_service.create_document(
213-
collection_name='users',
214-
data={'name': 'John Doe', 'email': '[email protected]'},
215-
mongo_id='507f1f77bcf86cd799439011'
216-
)
217-
```
218-
219-
## 📊 Data Mapping
220-
221-
The system automatically maps MongoDB collections to PostgreSQL tables:
222-
223-
| MongoDB Collection | PostgreSQL Table | Description |
224-
|-------------------|------------------|-------------|
225-
| `users` | `postgres_users` | User accounts and profiles |
226-
| `tasks` | `postgres_tasks` | Task management |
227-
| `teams` | `postgres_teams` | Team organization |
228-
| `labels` | `postgres_labels` | Task labels and categories |
229-
| `roles` | `postgres_roles` | User roles and permissions |
230-
| `task_assignments` | `postgres_task_assignments` | Task assignments |
231-
| `watchlists` | `postgres_watchlists` | User watchlists |
232-
| `audit_logs` | `postgres_audit_logs` | System audit trail |
233-
234-
## ⚙️ Configuration
235-
236-
### Environment Variables
237-
238-
```bash
239-
# Enable/disable dual-write
240-
DUAL_WRITE_ENABLED=True
241-
242-
# Retry configuration
243-
DUAL_WRITE_RETRY_ATTEMPTS=3
244-
DUAL_WRITE_RETRY_DELAY=5
245-
246-
# Database connections
247-
POSTGRES_HOST=localhost
248-
POSTGRES_PORT=5432
249-
POSTGRES_NAME=todo_postgres
250-
POSTGRES_USER=todo_user
251-
POSTGRES_PASSWORD=todo_password
252-
```
253-
254-
### Django Settings
255-
256-
The system automatically configures Django to use PostgreSQL while maintaining MongoDB connectivity.
257-
258-
## 📈 Monitoring and Health Checks
259-
260-
### Sync Metrics
261-
262-
```python
263-
# Get comprehensive sync metrics
264-
metrics = dual_write_service.get_sync_metrics()
265-
print(f"Total failures: {metrics['total_failures']}")
266-
```
267-
268-
### Health Monitoring
269-
270-
```python
271-
# Check sync status of specific document
272-
status = dual_write_service.get_sync_status('users', str(user_id))
273-
print(f"Sync status: {status}")
274-
```
275-
276-
## 🧪 Testing
277-
278-
### Unit Tests
279-
280-
```bash
281-
# Run unit tests
282-
python manage.py test todo.tests.unit
283-
284-
# Run specific test file
285-
python manage.py test todo.tests.unit.services.test_dual_write_service
286-
```
287-
288-
### Integration Tests
289-
290-
```bash
291-
# Run integration tests
292-
python manage.py test todo.tests.integration
293-
```
294-
295-
### Manual Testing
296-
297-
```python
298-
# Test sync operations
299-
from todo.services.enhanced_dual_write_service import EnhancedDualWriteService
300-
301-
service = EnhancedDualWriteService()
302-
303-
# Test create
304-
success = service.create_document('users', user_data, str(user_id))
305-
assert success == True
306-
307-
# Test update
308-
success = service.update_document('users', str(user_id), updated_data)
309-
assert success == True
310-
311-
# Test delete
312-
success = service.delete_document('users', str(user_id))
313-
assert success == True
314-
```
315-
316-
## 🚨 Error Handling
317-
318-
### Automatic Retries
319-
320-
Failed sync operations are automatically retried with configurable attempts and delays.
321-
322-
### Failure Tracking
323-
324-
```python
325-
# Get recent sync failures
326-
failures = dual_write_service.get_sync_failures()
327-
for failure in failures:
328-
print(f"Collection: {failure['collection']}")
329-
print(f"Document ID: {failure['mongo_id']}")
330-
print(f"Error: {failure['error']}")
331-
```
332-
333-
### Manual Retry
334-
335-
```python
336-
# Retry a specific failed sync
337-
success = dual_write_service.retry_failed_sync('users', str(user_id))
338-
```
339-
340-
## 🔮 Future Migration Path
341-
342-
### Phase 1: Dual-Write (Current)
343-
- ✅ All writes go to both databases
344-
- ✅ Reads continue from MongoDB
345-
- ✅ Postgres schema validated
346-
347-
### Phase 2: Read Migration
348-
- 🔄 Gradually shift reads to PostgreSQL
349-
- 🔄 Use feature flags for control
350-
- 🔄 Monitor performance
351-
352-
### Phase 3: Full Migration
353-
- 🎯 All operations use PostgreSQL
354-
- 🎯 MongoDB becomes backup
355-
- 🎯 Eventually decommission MongoDB
356-
357-
### Code Changes Required
358-
359-
The abstract repository pattern minimizes code changes:
360-
361-
```python
362-
# Current: MongoDB
363-
from todo.repositories.user_repository import UserRepository
364-
user_repo = UserRepository()
365-
366-
# Future: PostgreSQL (minimal change)
367-
from todo.repositories.postgres_repository import PostgresUserRepository
368-
user_repo = PostgresUserRepository()
369-
370-
# Same interface!
371-
user = user_repo.get_by_email("[email protected]")
372-
```
373-
374-
## 🐳 Docker Development
375-
376-
### Start Services
377-
378-
```bash
379-
# Start all services
380-
docker-compose -f docker-compose.dev.yml up -d
381-
382-
# Check service status
383-
docker-compose -f docker-compose.dev.yml ps
384-
385-
# View logs
386-
docker-compose -f docker-compose.dev.yml logs postgres
387-
```
388-
389-
### Database Access
390-
391-
```bash
392-
# Connect to PostgreSQL
393-
docker exec -it todo_postgres_dev psql -U todo_user -d todo_postgres
394-
395-
# Connect to MongoDB
396-
docker exec -it todo_mongodb_dev mongosh -u admin -p password
397-
```
398-
399-
## 📚 Documentation
400-
401-
- **README**: [README_DUAL_WRITE.md](README_DUAL_WRITE.md)
402-
- **System Docs**: [docs/DUAL_WRITE_SYSTEM.md](docs/DUAL_WRITE_SYSTEM.md)
403-
- **Environment Config**: [env.example](env.example)
404-
- **Docker Setup**: [docker-compose.dev.yml](docker-compose.dev.yml)
405-
406-
## 🤝 Contributing
407-
408-
1. Fork the repository
409-
2. Create a feature branch
410-
3. Make your changes
411-
4. Add tests
412-
5. Submit a pull request
413-
414-
## 🐛 Troubleshooting
415-
416-
### Common Issues
417-
418-
1. **PostgreSQL Connection Failed**
419-
- Check if PostgreSQL is running
420-
- Verify credentials in `.env`
421-
- Check network connectivity
422-
423-
2. **Sync Failures**
424-
- Review sync error logs
425-
- Check data transformation logic
426-
- Verify PostgreSQL schema
427-
428-
3. **Performance Issues**
429-
- Monitor sync latency
430-
- Optimize batch operation sizes
431-
- Monitor database performance
432-
433-
### Debug Commands
434-
435-
```python
436-
# Enable debug logging
437-
import logging
438-
logging.getLogger('todo.services.dual_write_service').setLevel(logging.DEBUG)
439-
440-
# Check sync status
441-
status = dual_write_service.get_sync_status('users', str(user_id))
442-
print(f"Sync status: {status}")
443-
```
444-
151+
- Verify the debug server shows "Debug server listening on port 5678"

docker-compose.dev.yml

Lines changed: 0 additions & 49 deletions
This file was deleted.

env.example

Lines changed: 0 additions & 48 deletions
This file was deleted.

0 commit comments

Comments
 (0)