|
148 | 148 | - If port 5678 is in use, specify a different port with `--debug-port`
|
149 | 149 | - Ensure VS Code Python extension is installed
|
150 | 150 | - Check that breakpoints are set in the correct files
|
151 |
| -- Verify the debug server shows "Debug server listening on port 5678" |
152 |
| - |
153 |
| - |
154 |
| -# Dual-Write System: MongoDB to PostgreSQL |
155 |
| - |
156 |
| -## 🎯 Overview |
157 |
| - |
158 |
| -This feature implements a comprehensive dual-write system that ensures all data written to MongoDB is automatically synchronized to PostgreSQL. The system is designed to enable future migration from MongoDB to PostgreSQL with minimal operational risk and code changes. |
159 |
| - |
160 |
| -## ✨ Key Features |
161 |
| - |
162 |
| -- **Dual-Write Operations**: Every MongoDB write is automatically mirrored to PostgreSQL |
163 |
| -- **Synchronous Operations**: Immediate consistency with both databases |
164 |
| -- **Comprehensive Error Handling**: Automatic retry mechanisms and failure tracking |
165 |
| -- **Future-Ready Architecture**: Abstract repository pattern for seamless database switching |
166 |
| -- **Real-time Monitoring**: Health checks and sync metrics |
167 |
| -- **Production Ready**: Includes Docker setup, migrations, and comprehensive documentation |
168 |
| - |
169 |
| -## 🏗️ Architecture |
170 |
| - |
171 |
| -``` |
172 |
| -┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ |
173 |
| -│ Application │───▶│ Dual-Write │───▶│ PostgreSQL │ |
174 |
| -│ (MongoDB) │ │ Service │ │ Database │ |
175 |
| -└─────────────────┘ └──────────────────┘ └─────────────────┘ |
176 |
| -``` |
177 |
| - |
178 |
| -## 🚀 Quick Start |
179 |
| - |
180 |
| -### 1. Prerequisites |
181 |
| - |
182 |
| -- Python 3.9+ |
183 |
| -- PostgreSQL 15+ |
184 |
| -- MongoDB 7+ (existing) |
185 |
| - |
186 |
| -### 2. Setup Development Environment |
187 |
| - |
188 |
| -```bash |
189 |
| -# Start required services |
190 |
| -docker-compose -f docker-compose.dev.yml up -d |
191 |
| - |
192 |
| -# Install dependencies |
193 |
| -pip install -r requirements.txt |
194 |
| - |
195 |
| -# Copy environment configuration |
196 |
| -cp env.example .env |
197 |
| -# Edit .env with your configuration |
198 |
| - |
199 |
| -# Run Django migrations |
200 |
| -python manage.py migrate |
201 |
| -``` |
202 |
| - |
203 |
| -### 3. Basic Usage |
204 |
| - |
205 |
| -```python |
206 |
| -from todo.services.enhanced_dual_write_service import EnhancedDualWriteService |
207 |
| - |
208 |
| -# Initialize the service |
209 |
| -dual_write_service = EnhancedDualWriteService() |
210 |
| - |
211 |
| -# Create a document (automatically syncs to both databases) |
212 |
| -success = dual_write_service.create_document( |
213 |
| - collection_name='users', |
214 |
| - data={ 'name': 'John Doe', 'email': '[email protected]'}, |
215 |
| - mongo_id='507f1f77bcf86cd799439011' |
216 |
| -) |
217 |
| -``` |
218 |
| - |
219 |
| -## 📊 Data Mapping |
220 |
| - |
221 |
| -The system automatically maps MongoDB collections to PostgreSQL tables: |
222 |
| - |
223 |
| -| MongoDB Collection | PostgreSQL Table | Description | |
224 |
| -|-------------------|------------------|-------------| |
225 |
| -| `users` | `postgres_users` | User accounts and profiles | |
226 |
| -| `tasks` | `postgres_tasks` | Task management | |
227 |
| -| `teams` | `postgres_teams` | Team organization | |
228 |
| -| `labels` | `postgres_labels` | Task labels and categories | |
229 |
| -| `roles` | `postgres_roles` | User roles and permissions | |
230 |
| -| `task_assignments` | `postgres_task_assignments` | Task assignments | |
231 |
| -| `watchlists` | `postgres_watchlists` | User watchlists | |
232 |
| -| `audit_logs` | `postgres_audit_logs` | System audit trail | |
233 |
| - |
234 |
| -## ⚙️ Configuration |
235 |
| - |
236 |
| -### Environment Variables |
237 |
| - |
238 |
| -```bash |
239 |
| -# Enable/disable dual-write |
240 |
| -DUAL_WRITE_ENABLED=True |
241 |
| - |
242 |
| -# Retry configuration |
243 |
| -DUAL_WRITE_RETRY_ATTEMPTS=3 |
244 |
| -DUAL_WRITE_RETRY_DELAY=5 |
245 |
| - |
246 |
| -# Database connections |
247 |
| -POSTGRES_HOST=localhost |
248 |
| -POSTGRES_PORT=5432 |
249 |
| -POSTGRES_NAME=todo_postgres |
250 |
| -POSTGRES_USER=todo_user |
251 |
| -POSTGRES_PASSWORD=todo_password |
252 |
| -``` |
253 |
| - |
254 |
| -### Django Settings |
255 |
| - |
256 |
| -The system automatically configures Django to use PostgreSQL while maintaining MongoDB connectivity. |
257 |
| - |
258 |
| -## 📈 Monitoring and Health Checks |
259 |
| - |
260 |
| -### Sync Metrics |
261 |
| - |
262 |
| -```python |
263 |
| -# Get comprehensive sync metrics |
264 |
| -metrics = dual_write_service.get_sync_metrics() |
265 |
| -print(f"Total failures: {metrics['total_failures']}") |
266 |
| -``` |
267 |
| - |
268 |
| -### Health Monitoring |
269 |
| - |
270 |
| -```python |
271 |
| -# Check sync status of specific document |
272 |
| -status = dual_write_service.get_sync_status('users', str(user_id)) |
273 |
| -print(f"Sync status: {status}") |
274 |
| -``` |
275 |
| - |
276 |
| -## 🧪 Testing |
277 |
| - |
278 |
| -### Unit Tests |
279 |
| - |
280 |
| -```bash |
281 |
| -# Run unit tests |
282 |
| -python manage.py test todo.tests.unit |
283 |
| - |
284 |
| -# Run specific test file |
285 |
| -python manage.py test todo.tests.unit.services.test_dual_write_service |
286 |
| -``` |
287 |
| - |
288 |
| -### Integration Tests |
289 |
| - |
290 |
| -```bash |
291 |
| -# Run integration tests |
292 |
| -python manage.py test todo.tests.integration |
293 |
| -``` |
294 |
| - |
295 |
| -### Manual Testing |
296 |
| - |
297 |
| -```python |
298 |
| -# Test sync operations |
299 |
| -from todo.services.enhanced_dual_write_service import EnhancedDualWriteService |
300 |
| - |
301 |
| -service = EnhancedDualWriteService() |
302 |
| - |
303 |
| -# Test create |
304 |
| -success = service.create_document('users', user_data, str(user_id)) |
305 |
| -assert success == True |
306 |
| - |
307 |
| -# Test update |
308 |
| -success = service.update_document('users', str(user_id), updated_data) |
309 |
| -assert success == True |
310 |
| - |
311 |
| -# Test delete |
312 |
| -success = service.delete_document('users', str(user_id)) |
313 |
| -assert success == True |
314 |
| -``` |
315 |
| - |
316 |
| -## 🚨 Error Handling |
317 |
| - |
318 |
| -### Automatic Retries |
319 |
| - |
320 |
| -Failed sync operations are automatically retried with configurable attempts and delays. |
321 |
| - |
322 |
| -### Failure Tracking |
323 |
| - |
324 |
| -```python |
325 |
| -# Get recent sync failures |
326 |
| -failures = dual_write_service.get_sync_failures() |
327 |
| -for failure in failures: |
328 |
| - print(f"Collection: {failure['collection']}") |
329 |
| - print(f"Document ID: {failure['mongo_id']}") |
330 |
| - print(f"Error: {failure['error']}") |
331 |
| -``` |
332 |
| - |
333 |
| -### Manual Retry |
334 |
| - |
335 |
| -```python |
336 |
| -# Retry a specific failed sync |
337 |
| -success = dual_write_service.retry_failed_sync('users', str(user_id)) |
338 |
| -``` |
339 |
| - |
340 |
| -## 🔮 Future Migration Path |
341 |
| - |
342 |
| -### Phase 1: Dual-Write (Current) |
343 |
| -- ✅ All writes go to both databases |
344 |
| -- ✅ Reads continue from MongoDB |
345 |
| -- ✅ Postgres schema validated |
346 |
| - |
347 |
| -### Phase 2: Read Migration |
348 |
| -- 🔄 Gradually shift reads to PostgreSQL |
349 |
| -- 🔄 Use feature flags for control |
350 |
| -- 🔄 Monitor performance |
351 |
| - |
352 |
| -### Phase 3: Full Migration |
353 |
| -- 🎯 All operations use PostgreSQL |
354 |
| -- 🎯 MongoDB becomes backup |
355 |
| -- 🎯 Eventually decommission MongoDB |
356 |
| - |
357 |
| -### Code Changes Required |
358 |
| - |
359 |
| -The abstract repository pattern minimizes code changes: |
360 |
| - |
361 |
| -```python |
362 |
| -# Current: MongoDB |
363 |
| -from todo.repositories.user_repository import UserRepository |
364 |
| -user_repo = UserRepository() |
365 |
| - |
366 |
| -# Future: PostgreSQL (minimal change) |
367 |
| -from todo.repositories.postgres_repository import PostgresUserRepository |
368 |
| -user_repo = PostgresUserRepository() |
369 |
| - |
370 |
| -# Same interface! |
371 |
| -user = user_repo.get_by_email( "[email protected]") |
372 |
| -``` |
373 |
| - |
374 |
| -## 🐳 Docker Development |
375 |
| - |
376 |
| -### Start Services |
377 |
| - |
378 |
| -```bash |
379 |
| -# Start all services |
380 |
| -docker-compose -f docker-compose.dev.yml up -d |
381 |
| - |
382 |
| -# Check service status |
383 |
| -docker-compose -f docker-compose.dev.yml ps |
384 |
| - |
385 |
| -# View logs |
386 |
| -docker-compose -f docker-compose.dev.yml logs postgres |
387 |
| -``` |
388 |
| - |
389 |
| -### Database Access |
390 |
| - |
391 |
| -```bash |
392 |
| -# Connect to PostgreSQL |
393 |
| -docker exec -it todo_postgres_dev psql -U todo_user -d todo_postgres |
394 |
| - |
395 |
| -# Connect to MongoDB |
396 |
| -docker exec -it todo_mongodb_dev mongosh -u admin -p password |
397 |
| -``` |
398 |
| - |
399 |
| -## 📚 Documentation |
400 |
| - |
401 |
| -- **README**: [README_DUAL_WRITE.md](README_DUAL_WRITE.md) |
402 |
| -- **System Docs**: [docs/DUAL_WRITE_SYSTEM.md](docs/DUAL_WRITE_SYSTEM.md) |
403 |
| -- **Environment Config**: [env.example](env.example) |
404 |
| -- **Docker Setup**: [docker-compose.dev.yml](docker-compose.dev.yml) |
405 |
| - |
406 |
| -## 🤝 Contributing |
407 |
| - |
408 |
| -1. Fork the repository |
409 |
| -2. Create a feature branch |
410 |
| -3. Make your changes |
411 |
| -4. Add tests |
412 |
| -5. Submit a pull request |
413 |
| - |
414 |
| -## 🐛 Troubleshooting |
415 |
| - |
416 |
| -### Common Issues |
417 |
| - |
418 |
| -1. **PostgreSQL Connection Failed** |
419 |
| - - Check if PostgreSQL is running |
420 |
| - - Verify credentials in `.env` |
421 |
| - - Check network connectivity |
422 |
| - |
423 |
| -2. **Sync Failures** |
424 |
| - - Review sync error logs |
425 |
| - - Check data transformation logic |
426 |
| - - Verify PostgreSQL schema |
427 |
| - |
428 |
| -3. **Performance Issues** |
429 |
| - - Monitor sync latency |
430 |
| - - Optimize batch operation sizes |
431 |
| - - Monitor database performance |
432 |
| - |
433 |
| -### Debug Commands |
434 |
| - |
435 |
| -```python |
436 |
| -# Enable debug logging |
437 |
| -import logging |
438 |
| -logging.getLogger('todo.services.dual_write_service').setLevel(logging.DEBUG) |
439 |
| - |
440 |
| -# Check sync status |
441 |
| -status = dual_write_service.get_sync_status('users', str(user_id)) |
442 |
| -print(f"Sync status: {status}") |
443 |
| -``` |
444 |
| - |
| 151 | +- Verify the debug server shows "Debug server listening on port 5678" |
0 commit comments