Merge branch 'fix/oom-errors' into 'develop'

OOM errors for large documents - Issue #35

See merge request genaiic-reusable-assets/engagement-artifacts/genaiic-idp-accelerator!249

Author: rstrahan

CHANGELOG.md

@@ -45,6 +45,9 @@ SPDX-License-Identifier: MIT-0
 - **Fixed CloudWatch Log Group Missing Retention regression**
 - **Security: Cross-Site Scripting (XSS) Vulnerability in FileViewer Component** - Fixed high-risk XSS vulnerability in `src/ui/src/components/document-viewer/FileViewer.jsx` where `innerHTML` was used with user-controlled data
 - **Add permissions boundary support to new Lambda function roles introduced in previous releases**
+- **Fixed OutOfMemory Errors in Pattern-2 OCR Lambda for Large High-Resolution Documents**
+  - **Root Cause**: Processing large PDFs with high-resolution images (7469×9623 pixels) caused memory spikes when 20 concurrent workers each held ~101MB images simultaneously, exceeding the 4GB Lambda memory limit
+  - **Optimal Solution**: Refactored image extraction to render directly at target dimensions using PyMuPDF matrix transformations, completely eliminating oversized image creation
 
 ## [0.3.11]
 

lib/idp_common_pkg/idp_common/ocr/README.md

@@ -122,19 +122,38 @@ ocr:
   task_prompt: "Extract all text from this image..."
 ```
 
+### Memory-Optimized Image Extraction
+
+The OCR service uses advanced memory optimization to prevent OutOfMemory errors when processing large high-resolution documents:
+
+**Direct Size Extraction**: When resize configuration is provided (`target_width` and `target_height`), images are extracted directly at the target dimensions using PyMuPDF matrix transformations. This completely eliminates memory spikes from creating oversized images.
+
+**Example for Large Document:**
+- **Original approach**: Extract 7469×9623 (101MB) → Resize to 951×1268 (5MB) → Memory spike
+- **Optimized approach**: Extract directly at 951×1268 (5MB) → No memory spike
+
+**Preserved Logic**: The optimization maintains all existing resize behavior:
+- ✅ Never upscales images (only applies scaling when scale_factor < 1.0)
+- ✅ Preserves aspect ratio using `min(width_ratio, height_ratio)`
+- ✅ Handles edge cases (no config, images already smaller than targets)
+- ✅ Full backward compatibility
+
 ### DPI Configuration
 
-The DPI (dots per inch) setting controls the resolution when extracting images from PDF pages:
+The DPI (dots per inch) setting controls the base resolution when extracting images from PDF pages:
 - **Default**: 150 DPI (good balance of quality and file size)
-- **Range**: 72-300 DPI
+- **Range**: 72-300 DPI  
 - **Location**: `ocr.image.dpi` in the configuration
 - **Behavior**: 
   - Only applies to PDF files (image files maintain their original resolution)
-  - Higher DPI = better quality but larger file sizes
+  - Combined with resize configuration for optimal memory usage
+  - Higher DPI = better quality but larger file sizes (use with resize config for large documents)
   - 150 DPI is recommended for most OCR use cases
-  - 300 DPI for documents with small text or fine details
+  - 300 DPI for documents with small text or fine details (ensure resize config is set)
   - 100 DPI for simple documents to reduce processing time
 
+**Memory Considerations**: For large documents with high DPI settings, always configure `target_width` and `target_height` to prevent memory issues. The service will intelligently extract at the optimal size.
+
 
 ## Migration Guide