Merge branch 'develop' v0.3.15

Author: Bob Strahan

.gitattributes

@@ -0,0 +1,5 @@
+* text=auto eol=lf
+*.py text eol=lf
+*.sh text eol=lf
+*.yaml text eol=lf
+*.yml text eol=lf

.gitlab-ci.yml

@@ -56,14 +56,15 @@ integration_tests:
   #   AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION}
   #   IDP_ACCOUNT_ID: ${IDP_ACCOUNT_ID}
 
-  # Add rules to only run on develop branch
+ # Add rules to only run on develop branch
+ # Add rules to only run on develop branch
   rules:
     - if: $CI_COMMIT_BRANCH == "develop"
-      when: manual # always # When idp-accelerator CICD is reconfigured
+      when: always # always # When idp-accelerator CICD is reconfigured
     - if: $CI_COMMIT_BRANCH =~ /^feature\/.*/
-      when: manual
+      when: always
     - if: $CI_COMMIT_BRANCH =~ /^fix\/.*/
-      when: manual
+      when: always
     - if: $CI_COMMIT_BRANCH =~ /^hotfix\/.*/
       when: manual
     - if: $CI_COMMIT_BRANCH =~ /^release\/.*/

CHANGELOG.md

@@ -5,6 +5,62 @@ SPDX-License-Identifier: MIT-0
 
 ## [Unreleased]
 
+## [0.3.15]
+
+### Added
+
+- **Intelligent Document Discovery Module for Automated Configuration Generation**
+  - Added Discovery module that automatically analyzes document samples to identify structure, field types, and organizational patterns
+  - **Pattern-Neutral Design**: Works across all processing patterns (1, 2, 3) with unified discovery process and pattern-specific implementations
+  - **Dual Discovery Methods**: Discovery without ground truth (exploratory analysis) and with ground truth (optimization using labeled data)
+  - **Automated Blueprint Creation**: Pattern 1 includes zero-touch BDA blueprint generation with intelligent change detection and version management
+  - **Web UI Integration**: Real-time discovery job monitoring, interactive results review, and seamless configuration integration
+  - **Advanced Features**: Multi-model support (Nova, Claude), customizable prompts, configurable parameters, ground truth processing, schema conversion, and lifecycle management
+  - **Key Benefits**: Rapid new document type onboarding, reduced time-to-production, configuration optimization, and automated workflow bootstrapping
+  - **Use Cases**: New document exploration, configuration improvement, rapid prototyping, and document understanding
+  - **Documentation**: Guide in `docs/discovery.md` with architecture details, best practices, and troubleshooting
+
+- **Optional Pattern-2 Regex-Based Classification for Enhanced Performance**
+  - Added support for optional regex patterns in document class definitions for performance optimization
+  - **Document Name Regex**: Match against document ID/name to classify all pages without LLM processing when all pages should be the same class
+  - **Document Page Content Regex**: Match against page text content during multi-modal page-level classification for fast page classification
+  - **Key Benefits**: Significant performance improvements and cost savings by bypassing LLM calls for pattern-matched documents, deterministic classification results for known document patterns, seamless fallback to existing LLM classification when regex patterns don't match
+  - **Configuration**: Optional `document_name_regex` and `document_page_content_regex` fields in class definitions with automatic regex compilation and validation
+  - **Logging**: Comprehensive info-level logging when regex patterns match for observability and debugging
+  - **CloudFormation Integration**: Updated Pattern-2 schema to support regex configuration through the Web UI
+  - **Demonstration**: New `step2_classification_with_regex.ipynb` notebook showcasing regex configuration and performance comparisons
+  - **Documentation**: Enhanced classification module README and main documentation with regex usage examples and best practices
+  
+- **Windows WSL Development Environment Setup Guide**
+  - Added WSL-based development environment setup guide for Windows developers in `docs/setup-development-env-WSL.md`
+  - **Key Features**: Automated setup script (`wsl_setup.sh`) for quick installation of Git, Python, Node.js, AWS CLI, and SAM CLI
+  - **Integrated Workflow**: Development setup combining Windows tools (VS Code, browsers) with native Linux environment
+  - **Target Use Cases**: Windows developers needing Linux compatibility without Docker Desktop or VM overhead
+
+### Fixed
+- **Throttling Error Detection and Retry Logic for Assessment Functions** - [GitHub Issue #45](https://github.com/aws-solutions-library-samples/accelerated-intelligent-document-processing-on-aws/issues/45)
+  - **Assessment Function**: Enhanced throttling detection to check for throttling errors returned in `document.errors` field in addition to thrown exceptions, raising `ThrottlingException` to trigger Step Functions retry when throttling is detected
+  - **Granular Assessment Task Caching**: Fixed caching logic to properly cache successful assessment tasks when there are ANY failed tasks (both exception-based and result-based failures), enabling efficient retry optimization by only reprocessing failed tasks while preserving successful results
+  - **Impact**: Improved resilience for throttling scenarios, reduced redundant processing during retries, and better Step Functions retry behavior
+
+- **Security Vulnerability Mitigation - Package Updates**
+
+- **GovCloud Compatibility - Hardcoded Service Domain References**
+  - Fixed hardcoded `amazonaws.com` references in CloudFormation templates that prevented GovCloud deployment
+  - Updated all service principals and endpoints to use dynamic `${AWS::URLSuffix}` expressions for automatic region-based resolution
+  - **Templates Updated**: `template.yaml` (main template), `patterns/pattern-3/sagemaker_classifier_endpoint.yaml`
+  - **Services Fixed**: EventBridge, Cognito, SageMaker, ECR, CloudFront, CodeBuild, AppSync, Lambda, DynamoDB, CloudWatch Logs, Glue
+  - Resolves GitHub Issue #50 - templates now deploy correctly in both standard AWS and GovCloud regions
+
+- **Bug Fixes and Code Improvements**
+  - Fixed HITL processing errors in both Pattern-1 (DynamoDB validation with empty strings) and Pattern-2 (string indices error in A2I output processing)
+  - Fixed Step Function UI issues including auto-refresh button auto-disable and fetch failures for failed executions with datetime serialization errors
+  - Cleaned up unused Step Function subscription infrastructure and removed duplicate code in Pattern-2 HITL function
+  - Expanded UI Visual Editor bounding box size with padding for better visibility and user interaction
+  - Fixed bug in list of models supporting cache points - previously claude 4 sonnet and opus had been excluded.
+  - Validations added at the assessment step for checking valid json response. The validation fails after extraction/assessment is complete if json parsing issues are encountered.
+
+
 ## [0.3.14]
 
 ### Added

Makefile

@@ -40,26 +40,34 @@ lint-cicd:
 	fi
 	@echo -e "$(GREEN)All code quality checks passed!$(NC)"
 
-# Check CloudFormation templates for hardcoded AWS partition ARNs
+# Check CloudFormation templates for hardcoded AWS partition ARNs and service principals
 check-arn-partitions:
-	@echo "Checking CloudFormation templates for hardcoded ARN partitions..."
+	@echo "Checking CloudFormation templates for hardcoded ARN partitions and service principals..."
 	@FOUND_ISSUES=0; \
 	for template in template.yaml patterns/*/template.yaml patterns/*/sagemaker_classifier_endpoint.yaml options/*/template.yaml; do \
 		if [ -f "$$template" ]; then \
 			echo "Checking $$template..."; \
-			MATCHES=$$(grep -n "arn:aws:" "$$template" | grep -v "arn:\$${AWS::Partition}:" || true); \
-			if [ -n "$$MATCHES" ]; then \
+			ARN_MATCHES=$$(grep -n "arn:aws:" "$$template" | grep -v "arn:\$${AWS::Partition}:" || true); \
+			if [ -n "$$ARN_MATCHES" ]; then \
 				echo -e "$(RED)ERROR: Found hardcoded 'arn:aws:' references in $$template:$(NC)"; \
-				echo "$$MATCHES" | sed 's/^/  /'; \
+				echo "$$ARN_MATCHES" | sed 's/^/  /'; \
 				echo -e "$(YELLOW)  These should use 'arn:\$${AWS::Partition}:' instead for GovCloud compatibility$(NC)"; \
 				FOUND_ISSUES=1; \
 			fi; \
+			SERVICE_MATCHES=$$(grep -n "\.amazonaws\.com" "$$template" | grep -v "\$${AWS::URLSuffix}" | grep -v "^[[:space:]]*#" | grep -v "Description:" | grep -v "Comment:" | grep -v "cognito" | grep -v "ContentSecurityPolicy" || true); \
+			if [ -n "$$SERVICE_MATCHES" ]; then \
+				echo -e "$(RED)ERROR: Found hardcoded service principal references in $$template:$(NC)"; \
+				echo "$$SERVICE_MATCHES" | sed 's/^/  /'; \
+				echo -e "$(YELLOW)  These should use '\$${AWS::URLSuffix}' instead of 'amazonaws.com' for GovCloud compatibility$(NC)"; \
+				echo -e "$(YELLOW)  Example: 'lambda.amazonaws.com' should be 'lambda.\$${AWS::URLSuffix}'$(NC)"; \
+				FOUND_ISSUES=1; \
+			fi; \
 		fi; \
 	done; \
 	if [ $$FOUND_ISSUES -eq 0 ]; then \
-		echo -e "$(GREEN)✅ No hardcoded ARN partition references found!$(NC)"; \
+		echo -e "$(GREEN)✅ No hardcoded ARN partition or service principal references found!$(NC)"; \
 	else \
-		echo -e "$(RED)❌ Found hardcoded ARN partition references that need to be fixed$(NC)"; \
+		echo -e "$(RED)❌ Found hardcoded references that need to be fixed for GovCloud compatibility$(NC)"; \
 		exit 1; \
 	fi
 

README.md

@@ -128,6 +128,7 @@ For detailed deployment and testing instructions, see the [Deployment Guide](./d
 - [Agent Analysis](./docs/agent-analysis.md) - Natural language analytics and data visualization feature
 - [Custom MCP Agent](./docs/custom-MCP-agent.md) - Integrating external MCP servers for custom tools and capabilities
 - [Configuration](./docs/configuration.md) - Configuration and customization options
+- [Discovery](./docs/discovery.md) - Pattern-neutral discovery process and BDA blueprint automation
 - [Classification](./docs/classification.md) - Customizing document classification
 - [Extraction](./docs/extraction.md) - Customizing information extraction
 - [Human-in-the-Loop Review](./docs/human-review.md) - Human review workflows with Amazon A2I

VERSION

@@ -1 +1 @@
-0.3.14
+0.3.15

docs/README.md

@@ -12,6 +12,7 @@ This folder contains detailed documentation on various aspects of the GenAI Inte
 - [Web UI](./web-ui.md) - Web interface features and usage
 - [Agent Analysis](./agent-analysis.md) - Natural language analytics and data visualization feature
 - [Knowledge Base](./knowledge-base.md) - Document knowledge base query feature
+- [Post-Processing Lambda Hook](./post-processing-lambda-hook.md) - Custom downstream processing integration
 - [Evaluation Framework](./evaluation.md) - Accuracy assessment system
 - [Assessment Feature](./assessment.md) - Extraction confidence evaluation using LLMs
 - [Configuration](./configuration.md) - Configuration and customization options

docs/architecture.md

@@ -277,6 +277,8 @@ The solution supports an optional post-processing Lambda hook integration:
   - Custom notification systems
 - Receives the document processing details and output location
 
+For comprehensive implementation guidance, use cases, and code examples, see [post-processing-lambda-hook.md](./post-processing-lambda-hook.md).
+
 ## Additional Documentation
 
 - [classification.md](./classification.md) - Details on document classification capabilities