Merge branch 'feature/enable-disable-steps' into 'develop'

Feature/enable disable steps

See merge request genaiic-reusable-assets/engagement-artifacts/genaiic-idp-accelerator!245

Author: rstrahan

CHANGELOG.md

@@ -12,6 +12,33 @@ SPDX-License-Identifier: MIT-0
   - Dynamic pricing configuration loading from configuration
   - Enhanced cost analysis capabilities with comprehensive Athena queries for cost tracking, trend analysis, and efficiency metrics
   - Automatic cost calculation as `estimated_cost = value × unit_cost` for all metering records
+  
+- **Configuration-Based Summarization Control**
+  - Summarization can now be enabled/disabled via configuration file `summarization.enabled` property instead of CloudFormation stack parameter
+  - **Key Benefits**: Runtime control without stack redeployment, zero LLM costs when disabled, simplified state machine architecture, backward compatible defaults
+  - **Implementation**: Always calls SummarizationStep but service skips processing when `enabled: false`
+  - **Cost Optimization**: When disabled, no LLM API calls or S3 operations are performed
+  - **Configuration Example**: Set `summarization.enabled: false` to disable, `enabled: true` to enable (default)
+
+- **Configuration-Based Assessment Control**
+  - Assessment can now be enabled/disabled via configuration file `assessment.enabled` property instead of CloudFormation stack parameter
+  - **Key Benefits**: Runtime control without stack redeployment, zero LLM costs when disabled, simplified state machine architecture, backward compatible defaults
+  - **Implementation**: Always calls AssessmentStep but service skips processing when `enabled: false`
+  - **Cost Optimization**: When disabled, no LLM API calls or S3 operations are performed
+  - **Configuration Example**: Set `assessment.enabled: false` to disable, `enabled: true` to enable (default)
+
+### Removed
+- **CloudFormation Parameters**: Removed `IsSummarizationEnabled` and `IsAssessmentEnabled` parameters from all pattern templates
+- **Related Conditions**: Removed parameter conditions and state machine definition substitutions for both features
+- **Conditional Logic**: Eliminated complex conditional logic from state machine definitions for summarization and assessment steps
+
+### ⚠️ Breaking Changes
+- **Configuration Migration Required**: When updating a stack that previously had `IsSummarizationEnabled` or `IsAssessmentEnabled` set to `false`, these features will now default to `enabled: true` after the update. To maintain the disabled behavior:
+  1. Update your configuration file to set `summarization.enabled: false` and/or `assessment.enabled: false` as needed
+  2. Save the configuration changes immediately after the stack update
+  3. This ensures continued cost optimization by preventing unexpected LLM API calls
+- **Action Required**: Review your current CloudFormation parameter settings before updating and update your configuration accordingly to preserve existing behavior
+
 
 ## [0.3.11]
 

VERSION

@@ -1 +1 @@
-0.3.11
+0.3.12-wip

config_library/pattern-1/lending-package-sample/config.yaml

@@ -5,6 +5,7 @@ notes: Processing configuration in BDA project.
 assessment:
   default_confidence_threshold: '0.8'
 summarization:
+  enabled: true
   top_p: '0.1'
   max_tokens: '4096'
   top_k: '5'

config_library/pattern-2/bank-statement-sample/config.yaml

@@ -307,6 +307,7 @@ extraction:
   system_prompt: >-
     You are a document assistant. Respond only with JSON. Never make up data, only provide data found in the document being provided.
 summarization:
+  enabled: true
   top_p: '0.1'
   max_tokens: '4096'
   top_k: '5'
@@ -368,6 +369,7 @@ summarization:
   system_prompt: >-
     You are a document summarization expert who can analyze and summarize documents from various domains including medical, financial, legal, and general business documents. Your task is to create a summary that captures the key information, main points, and important details from the document. Your output must be in valid JSON format. \nSummarization Style: Balanced\\nCreate a balanced summary that provides a moderate level of detail. Include the main points and key supporting information, while maintaining the document's overall structure. Aim for a comprehensive yet concise summary.\n Your output MUST be in valid JSON format with markdown content. You MUST strictly adhere to the output format specified in the instructions.
 assessment:
+  enabled: true
   image:
     target_height: ''
     target_width: ''

config_library/pattern-2/lending-package-sample/config.yaml

@@ -1082,6 +1082,7 @@ extraction:
   system_prompt: >-
     You are a document assistant. Respond only with JSON. Never make up data, only provide data found in the document being provided.
 summarization:
+  enabled: true
   top_p: '0.1'
   max_tokens: '4096'
   top_k: '5'
@@ -1143,6 +1144,7 @@ summarization:
   system_prompt: >-
     You are a document summarization expert who can analyze and summarize documents from various domains including medical, financial, legal, and general business documents. Your task is to create a summary that captures the key information, main points, and important details from the document. Your output must be in valid JSON format. \nSummarization Style: Balanced\\nCreate a balanced summary that provides a moderate level of detail. Include the main points and key supporting information, while maintaining the document's overall structure. Aim for a comprehensive yet concise summary.\n Your output MUST be in valid JSON format with markdown content. You MUST strictly adhere to the output format specified in the instructions.
 assessment:
+  enabled: true
   image:
     target_height: ''
     target_width: ''

config_library/pattern-2/rvl-cdip-package-sample-with-few-shot-examples/config.yaml

@@ -886,6 +886,7 @@ pricing:
       - name: cacheWriteInputTokens
         price: '3.75E-6'
 assessment:
+  enabled: true
   image:
     target_height: ''
     target_width: ''
@@ -1069,6 +1070,7 @@ evaluation:
       values match for document attribute extraction. You will consider the
       context and meaning rather than just exact string matching.
 summarization:
+  enabled: true
   top_p: '0.1'
   max_tokens: '4096'
   top_k: '5'

config_library/pattern-2/rvl-cdip-package-sample/config.yaml

@@ -547,6 +547,7 @@ extraction:
   system_prompt: >-
     You are a document assistant. Respond only with JSON. Never make up data, only provide data found in the document being provided.
 summarization:
+  enabled: true
   top_p: '0.1'
   max_tokens: '4096'
   top_k: '5'
@@ -608,6 +609,7 @@ summarization:
   system_prompt: >-
     You are a document summarization expert who can analyze and summarize documents from various domains including medical, financial, legal, and general business documents. Your task is to create a summary that captures the key information, main points, and important details from the document. Your output must be in valid JSON format. \nSummarization Style: Balanced\\nCreate a balanced summary that provides a moderate level of detail. Include the main points and key supporting information, while maintaining the document's overall structure. Aim for a comprehensive yet concise summary.\n Your output MUST be in valid JSON format with markdown content. You MUST strictly adhere to the output format specified in the instructions.
 assessment:
+  enabled: true
   image:
     target_height: ''
     target_width: ''

config_library/pattern-3/rvl-cdip-package-sample/config.yaml

@@ -407,6 +407,7 @@ extraction:
   system_prompt: >-
     You are a document assistant. Respond only with JSON. Never make up data, only provide data found in the document being provided.
 summarization:
+  enabled: true
   top_p: '0.1'
   max_tokens: '4096'
   top_k: '5'
@@ -468,6 +469,7 @@ summarization:
   system_prompt: >-
     You are a document summarization expert who can analyze and summarize documents from various domains including medical, financial, legal, and general business documents. Your task is to create a summary that captures the key information, main points, and important details from the document. Your output must be in valid JSON format. \nSummarization Style: Balanced\\nCreate a balanced summary that provides a moderate level of detail. Include the main points and key supporting information, while maintaining the document's overall structure. Aim for a comprehensive yet concise summary.\n Your output MUST be in valid JSON format with markdown content. You MUST strictly adhere to the output format specified in the instructions.
 assessment:
+  enabled: true
   image:
     target_width: ''
     target_height: ''

docs/architecture.md

@@ -40,7 +40,7 @@ The solution uses a modular architecture with nested CloudFormation stacks to su
 - Centralized monitoring and management across all patterns
 - Pattern-specific optimizations and configurations
 - Optional features that can be enabled across all patterns:
-  - Document summarization (controlled by `IsSummarizationEnabled` parameter)
+  - Document summarization (controlled by configuration `summarization.enabled` property)
     - This feature also enables the "Chat with Document" functionality
     - This feature does not use the Bedrock Knowledge Base but stores a full-text text file in S3
   - Document Knowledge Base (using Amazon Bedrock)
@@ -215,7 +215,7 @@ For detailed information about the Web UI, its features, and usage, see [web-ui.
 
 ### Document Summarization
 
-When enabled via the `IsSummarizationEnabled` parameter (default: true), the solution provides document summarization across all patterns:
+When enabled via the configuration `summarization.enabled` property (default: true), the solution provides document summarization across all patterns:
 
 - All patterns use a dedicated summarization step with Bedrock models
 - Summarization provides a concise overview of the document content

docs/assessment.md

@@ -54,19 +54,33 @@ The assessment step is conditionally integrated into Pattern-2's ProcessSections
 
 ## Configuration
 
-### Deployment Parameter
+### Configuration-Based Control
 
-Enable assessment during stack deployment:
+Assessment can now be controlled via the configuration file rather than CloudFormation stack parameters. This provides more flexibility and eliminates the need for stack redeployment when changing assessment behavior.
 
+**Configuration-based Control (Recommended):**
 ```yaml
-Parameters:
-  IsAssessmentEnabled:
-    Type: String
-    Default: "false"
-    AllowedValues: ["true", "false"]
-    Description: Enable assessment functionality for extraction confidence evaluation
+assessment:
+  enabled: true  # Set to false to disable assessment
+  model: us.amazon.nova-lite-v1:0
+  temperature: 0.0
+  # ... other assessment settings
 ```
 
+**Key Benefits:**
+- **Runtime Control**: Enable/disable without stack redeployment  
+- **Cost Optimization**: Zero LLM costs when disabled (`enabled: false`)
+- **Simplified Architecture**: No conditional logic in state machines
+- **Backward Compatible**: Defaults to `enabled: true` when property is missing
+
+**Behavior When Disabled:**
+- Assessment lambda is still called (minimal overhead)
+- Service immediately returns with logging: "Assessment is disabled via configuration"
+- No LLM API calls or S3 operations are performed
+- Document processing continues to completion
+
+**Migration Note**: The previous `IsAssessmentEnabled` CloudFormation parameter has been removed in favor of this configuration-based approach.
+
 ### Assessment Configuration Section
 
 Add the assessment section to your configuration YAML:
@@ -808,7 +822,7 @@ The assessment feature implements several cost optimization techniques:
 
 1. **Text Confidence Data**: Uses condensed OCR confidence information instead of full raw OCR results (80-90% token reduction)
 2. **Conditional Image Processing**: Images only processed when `{DOCUMENT_IMAGE}` placeholder is present
-3. **Optional Deployment**: Assessment infrastructure only deployed when `IsAssessmentEnabled=true`
+3. **Configuration-Based Control**: Assessment can be enabled/disabled via configuration `enabled` property for flexible deployment
 4. **Efficient Prompting**: Optimized prompt templates minimize token usage while maintaining accuracy
 5. **Configurable Image Dimensions**: Adjust image resolution to balance assessment quality and processing costs
 6. **Granular Assessment with Caching**: For complex documents, use granular assessment with prompt caching for 60-80% cost reduction
@@ -889,7 +903,7 @@ ValueError: "Assessment prompt template formatting failed: missing required plac
 ### Common Issues
 
 1. **Assessment Not Running**
-   - Verify `IsAssessmentEnabled=true` in deployment
+   - Verify `assessment.enabled: true` in configuration file
    - Check state machine definition includes assessment step
    - Confirm assessment Lambda function deployed successfully