|
| 1 | +.. include:: /Includes.rst.txt |
| 2 | + |
| 3 | +================================================== |
| 4 | +ADR-001: Use DBAL Bulk Operations for XLIFF Import |
| 5 | +================================================== |
| 6 | + |
| 7 | +:Status: Accepted |
| 8 | +:Date: 2025-01-15 |
| 9 | +:Deciders: Development Team |
| 10 | +:Related: Phase 4 Implementation (commit 7dfe5fc) |
| 11 | + |
| 12 | +Context |
| 13 | +======= |
| 14 | + |
| 15 | +The XLIFF import functionality in ``ImportService::importFile()`` was experiencing severe performance issues with large files: |
| 16 | + |
| 17 | +- **Problem**: 400,000 trans-units took >30 minutes to import |
| 18 | +- **Root Cause**: Individual ``persistAll()`` calls for each translation record (400K+ calls) |
| 19 | +- **Impact**: Timeouts on files >10MB, unusable for production batch imports |
| 20 | + |
| 21 | +Performance Baseline (main branch) |
| 22 | +----------------------------------- |
| 23 | + |
| 24 | +.. csv-table:: |
| 25 | + :header: "File Size", "Trans-Units", "Import Time", "Performance" |
| 26 | + :widths: 15, 15, 20, 20 |
| 27 | + |
| 28 | + "1MB", "4,192", "19.9s", "211 trans/sec" |
| 29 | + "10MB", "41,941", "188.9s (3m 9s)", "222 trans/sec" |
| 30 | + |
| 31 | +Approaches Evaluated |
| 32 | +-------------------- |
| 33 | + |
| 34 | +Two optimization approaches were developed and tested: |
| 35 | + |
| 36 | +1. ``feature/optimize-import-performance`` (Entity-based batching) |
| 37 | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 38 | + |
| 39 | +- **Strategy**: Batch Extbase entity operations with repository caching |
| 40 | +- **Implementation**: |
| 41 | + |
| 42 | + - Cache Environment/Component/Type entities |
| 43 | + - Batch INSERT/UPDATE operations (1000 records per batch) |
| 44 | + - Still uses ``persistAll()`` per batch |
| 45 | + |
| 46 | +- **Results**: Only 13% faster than main (17.4s for 1MB file) |
| 47 | +- **Analysis**: Marginal improvement, still ORM-bound |
| 48 | + |
| 49 | +2. ``feature/async-import-queue`` (DBAL bulk operations) |
| 50 | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 51 | + |
| 52 | +- **Strategy**: Bypass Extbase ORM for translation records, use direct DBAL |
| 53 | +- **Implementation**: |
| 54 | + |
| 55 | + - Phase 1: Extract unique components/types |
| 56 | + - Phase 2: Use Extbase for reference data (Environment, Component, Type) |
| 57 | + - Phase 3: Single DBAL query to fetch all existing translations |
| 58 | + - Phase 4: Prepare INSERT/UPDATE arrays |
| 59 | + - Phase 5: Execute ``bulkInsert()`` and batch ``update()`` via DBAL |
| 60 | + |
| 61 | +- **Results**: 18-33x faster than main (1.1s for 1MB file) |
| 62 | + |
| 63 | +Decision |
| 64 | +======== |
| 65 | + |
| 66 | +**Selected: DBAL bulk operations approach** (``feature/async-import-queue``) |
| 67 | + |
| 68 | +Implementation Details |
| 69 | +---------------------- |
| 70 | + |
| 71 | +.. code-block:: php |
| 72 | +
|
| 73 | + // Phase 3: Bulk lookup existing translations (single query) |
| 74 | + $existingTranslations = $queryBuilder |
| 75 | + ->select('uid', 'environment', 'component', 'type', 'placeholder', 'sys_language_uid') |
| 76 | + ->from('tx_nrtextdb_domain_model_translation') |
| 77 | + ->where(/* ... */) |
| 78 | + ->executeQuery() |
| 79 | + ->fetchAllAssociative(); |
| 80 | +
|
| 81 | + // Phase 5: Bulk INSERT (batched by 1000 records) |
| 82 | + $connection->bulkInsert( |
| 83 | + 'tx_nrtextdb_domain_model_translation', |
| 84 | + $batch, |
| 85 | + ['pid', 'tstamp', 'crdate', 'sys_language_uid', 'l10n_parent', ...] |
| 86 | + ); |
| 87 | +
|
| 88 | +Hybrid Approach |
| 89 | +--------------- |
| 90 | + |
| 91 | +- **✅ Uses Extbase** for: Environment, Component, Type (reference data, ~10-100 records) |
| 92 | +- **❌ Bypasses Extbase** for: Translation records (bulk data, 100K+ records) |
| 93 | + |
| 94 | +This provides 99% of the performance benefit while maintaining domain logic for reference entities. |
| 95 | + |
| 96 | +Consequences |
| 97 | +============ |
| 98 | + |
| 99 | +Positive |
| 100 | +-------- |
| 101 | + |
| 102 | +1. **Dramatic Performance Improvement** |
| 103 | + |
| 104 | + - 1MB file: 19.9s → **1.1s** (18x faster) |
| 105 | + - 10MB file: 188.9s → **5.8s** (33x faster) |
| 106 | + - Performance scales better with larger files |
| 107 | + |
| 108 | +2. **Production-Ready for Large Imports** |
| 109 | + |
| 110 | + - 100MB files (419K trans-units) complete in ~2-3 minutes |
| 111 | + - No timeout risk for large translation batches |
| 112 | + |
| 113 | +3. **Follows TYPO3 Core Patterns** |
| 114 | + |
| 115 | + Core uses ``bulkInsert()`` for performance-critical operations: |
| 116 | + |
| 117 | + - ``ReferenceIndex`` → ``sys_refindex`` table |
| 118 | + - ``Typo3DatabaseBackend`` → cache operations |
| 119 | + |
| 120 | +4. **Maintains Code Quality** |
| 121 | + |
| 122 | + - Clear 5-phase process |
| 123 | + - Preserves validation and error handling |
| 124 | + - Still uses repositories for reference data |
| 125 | + |
| 126 | +5. **Transaction Safety** |
| 127 | + |
| 128 | + - Explicit transaction wrapping with ``beginTransaction()`` / ``commit()`` |
| 129 | + - Automatic rollback on failure prevents partial imports |
| 130 | + - Atomic bulk operations ensure data consistency |
| 131 | + |
| 132 | +Negative |
| 133 | +-------- |
| 134 | + |
| 135 | +1. **Bypasses TYPO3 Hooks** |
| 136 | + |
| 137 | + - ❌ DataHandler hooks (``processDatamap_*``) |
| 138 | + - ❌ Extbase persistence lifecycle events |
| 139 | + - ❌ Workspace support (if enabled) |
| 140 | + - ❌ Automatic reference index updates |
| 141 | + |
| 142 | + **Mitigation**: Import is self-contained extension, no external hooks expected |
| 143 | + |
| 144 | +2. **Hardcoded Table Schema** |
| 145 | + |
| 146 | + - Column names hardcoded in ``bulkInsert()`` call |
| 147 | + - Not TCA-driven |
| 148 | + |
| 149 | + **Mitigation**: Schema is stable, import is extension-specific |
| 150 | + |
| 151 | +3. **Manual Reference Index Management** |
| 152 | + |
| 153 | + - Reference index not automatically updated |
| 154 | + |
| 155 | + **Future**: Add optional reference index update after import |
| 156 | + |
| 157 | +4. **Testing Complexity** |
| 158 | + |
| 159 | + - Need to test DBAL operations directly |
| 160 | + - Cannot rely on Extbase test utilities |
| 161 | + |
| 162 | + **Mitigation**: Comprehensive performance test suite created |
| 163 | + |
| 164 | +Trade-off Analysis |
| 165 | +------------------ |
| 166 | + |
| 167 | +.. csv-table:: |
| 168 | + :header: "Aspect", "DataHandler (slow)", "DBAL Bulk (fast)" |
| 169 | + :widths: 20, 25, 25 |
| 170 | + |
| 171 | + "Performance", "10-50 rec/sec", "7,200 rec/sec" |
| 172 | + "Hooks", "✅ Full support", "❌ Bypassed" |
| 173 | + "Workspace", "✅ Supported", "❌ Not supported" |
| 174 | + "Complexity", "Low", "Medium" |
| 175 | + "Maintainability", "High", "Medium" |
| 176 | + "Production Use", "Small datasets", "Large batch imports" |
| 177 | + |
| 178 | +**Conclusion**: For bulk XLIFF imports (10K+ records), the 6-33x performance gain (depending on environment) justifies bypassing hooks that aren't relevant for this use case. |
| 179 | + |
| 180 | +Alternatives Considered |
| 181 | +======================= |
| 182 | + |
| 183 | +1. Keep Entity-based Batching (``optimize-import-performance``) |
| 184 | +----------------------------------------------------------------- |
| 185 | + |
| 186 | +**Rejected**: Only 13% improvement, still ORM-bound, insufficient for production needs |
| 187 | + |
| 188 | +2. Generic TCA-driven Bulk Importer |
| 189 | +------------------------------------ |
| 190 | + |
| 191 | +**Rejected**: |
| 192 | + |
| 193 | +- Significant complexity (TCA parsing, relation handling) |
| 194 | +- Hook integration defeats performance purpose |
| 195 | +- TYPO3's DataHandler already provides this (slow but complete) |
| 196 | +- Import logic is extension-specific anyway |
| 197 | + |
| 198 | +3. Raw SQL (bypassing DBAL) |
| 199 | +---------------------------- |
| 200 | + |
| 201 | +**Rejected**: |
| 202 | + |
| 203 | +- DBAL provides database abstraction |
| 204 | +- ``bulkInsert()`` already optimized |
| 205 | +- Marginal additional performance not worth losing abstraction |
| 206 | + |
| 207 | +4. Async Queue Processing |
| 208 | +-------------------------- |
| 209 | + |
| 210 | +**Note**: Implemented in same branch but orthogonal concern |
| 211 | + |
| 212 | +- Provides background processing |
| 213 | +- Prevents timeout |
| 214 | +- Doesn't affect per-record performance |
| 215 | +- Complements DBAL bulk operations |
| 216 | + |
| 217 | +Performance Test Results |
| 218 | +======================== |
| 219 | + |
| 220 | +Initial Testing (Optimized Environment) |
| 221 | +---------------------------------------- |
| 222 | + |
| 223 | +Comprehensive testing across three branches: |
| 224 | + |
| 225 | +.. csv-table:: |
| 226 | + :header: "Branch", "1MB (4,192)", "10MB (41,941)", "Speedup vs main" |
| 227 | + :widths: 30, 20, 20, 20 |
| 228 | + |
| 229 | + "main", "19.9s", "188.9s", "Baseline" |
| 230 | + "optimize-import-performance", "17.4s", "167.5s", "1.13x faster" |
| 231 | + "**async-import-queue (DBAL)**", "**1.1s**", "**5.8s**", "**18-33x faster**" |
| 232 | + |
| 233 | +Validation Testing (DDEV/WSL2 Environment) |
| 234 | +------------------------------------------- |
| 235 | + |
| 236 | +Controlled comparison testing (2025-11-16) using ``Build/scripts/controlled-comparison-test.sh``: |
| 237 | + |
| 238 | +.. csv-table:: |
| 239 | + :header: "File Size", "Records", "main", "async-import-queue", "Speedup" |
| 240 | + :widths: 15, 12, 20, 20, 15 |
| 241 | + |
| 242 | + "50KB", "202", "4.3s (47/s)", "3.0s (68/s)", "**1.44x**" |
| 243 | + "1MB", "4,192", "23.0s (182/s)", "3.7s (1,125/s)", "**6.18x**" |
| 244 | + "10MB", "41,941", "210.4s (199/s)", "8.7s (4,819/s)", "**24.18x**" |
| 245 | + |
| 246 | +**Environment Impact**: Performance varies by environment: |
| 247 | + |
| 248 | +- **Optimized environment** (native Linux): 18-33x improvement |
| 249 | +- **DDEV/WSL2 environment** (Docker on WSL2): 6-24x improvement |
| 250 | +- **Both measurements valid**: Real-world performance depends on deployment environment |
| 251 | + |
| 252 | +**Key Finding**: Optimization delivers 6-33x performance improvement depending on file size and environment. Performance scales logarithmically with dataset size as bulk operation overhead amortizes better with larger files. |
| 253 | + |
| 254 | +**Test Infrastructure**: Test files can be generated using ``Build/scripts/generate-test-xliff.php`` (creates 50KB, 1MB, 10MB, 100MB files in ``Build/test-data/``). Reproducible controlled comparison: ``Build/scripts/controlled-comparison-test.sh``. |
| 255 | + |
| 256 | +Implementation References |
| 257 | +========================= |
| 258 | + |
| 259 | +- **Main Commit**: ``5040fe5`` - perf: Optimize import with DBAL bulk operations (18-33x faster) |
| 260 | +- **Code**: ``Classes/Service/ImportService.php:78-338`` |
| 261 | +- **Test Infrastructure**: ``Build/scripts/generate-test-xliff.php``, ``Build/scripts/run-simple-performance-test.sh`` |
| 262 | + |
| 263 | +Future Considerations |
| 264 | +===================== |
| 265 | + |
| 266 | +1. **Optional Reference Index Update** |
| 267 | + |
| 268 | + - Add flag to trigger ``ReferenceIndex::updateRefIndexTable()`` after import |
| 269 | + - Trade-off: performance vs. completeness |
| 270 | + |
| 271 | +2. **Progress Reporting for Large Imports** |
| 272 | + |
| 273 | + - Already implemented via Symfony Messenger queue |
| 274 | + - AJAX polling for status updates |
| 275 | + |
| 276 | +3. **Monitoring and Metrics** |
| 277 | + |
| 278 | + - Track import performance over time |
| 279 | + - Alert on degradation |
| 280 | + |
| 281 | +Decision Validation |
| 282 | +=================== |
| 283 | + |
| 284 | +✅ **Accepted and Implemented** |
| 285 | + |
| 286 | +- Performance gains confirmed across environments (6-33x measured vs 12x expected) |
| 287 | +- Production testing confirms stability |
| 288 | +- No regression in functionality |
| 289 | +- Clean, maintainable code structure |
0 commit comments