Add configuration-based summarization control with runtime enable/disable

Bob Strahan · Bob Strahan · commit dc12559ca5d9 · 2025-08-18T20:15:16.000Z
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,6 +5,27 @@ SPDX-License-Identifier: MIT-0
 
 ## [Unreleased]
 
+### Added
+- **Configuration-Based Summarization Control**
+  - Summarization can now be enabled/disabled via configuration file `summarization.enabled` property instead of CloudFormation stack parameter
+  - **Key Benefits**: Runtime control without stack redeployment, zero LLM costs when disabled, simplified state machine architecture, backward compatible defaults
+  - **Implementation**: Always calls SummarizationStep but service skips processing when `enabled: false`
+  - **Cost Optimization**: When disabled, no LLM API calls or S3 operations are performed
+  - **Configuration Example**: Set `summarization.enabled: false` to disable, `enabled: true` to enable (default)
+
+### Changed
+- **State Machine Simplification**: Removed `SummarizationChoice` conditional states from all patterns (Pattern 1, 2, 3) for cleaner workflows
+- **Service Logic Enhancement**: SummarizationService now checks configuration `enabled` flag at the beginning of `process_document()`
+- **Configuration Schema Updates**: Added `enabled` boolean property to summarization sections in all CloudFormation template schemas
+
+### Removed
+- **CloudFormation Parameter**: Removed `IsSummarizationEnabled` parameter from all pattern templates (patterns/pattern-1, pattern-2, pattern-3)
+- **Related Conditions**: Removed `IsSummarizationEnabled` conditions and state machine definition substitutions
+
+### Documentation
+- **Updated Documentation**: Enhanced docs/configuration.md, docs/architecture.md, and all pattern-specific docs (pattern-1.md, pattern-2.md, pattern-3.md)
+- **Service Documentation**: Updated lib/idp_common_pkg/idp_common/summarization/README.md with configuration examples and behavior details
+- **Migration Guidance**: Added migration notes about the CloudFormation parameter removal
 
 ## [0.3.11]
 
diff --git a/VERSION b/VERSION
@@ -1 +1 @@
-0.3.11
+0.3.12-wip
diff --git a/config_library/pattern-1/lending-package-sample/config.yaml b/config_library/pattern-1/lending-package-sample/config.yaml
@@ -5,6 +5,7 @@ notes: Processing configuration in BDA project.
 assessment:
   default_confidence_threshold: '0.8'
 summarization:
+  enabled: true
   top_p: '0.1'
   max_tokens: '4096'
   top_k: '5'
diff --git a/config_library/pattern-2/bank-statement-sample/config.yaml b/config_library/pattern-2/bank-statement-sample/config.yaml
@@ -307,6 +307,7 @@ extraction:
   system_prompt: >-
     You are a document assistant. Respond only with JSON. Never make up data, only provide data found in the document being provided.
 summarization:
+  enabled: true
   top_p: '0.1'
   max_tokens: '4096'
   top_k: '5'
diff --git a/config_library/pattern-2/lending-package-sample/config.yaml b/config_library/pattern-2/lending-package-sample/config.yaml
@@ -1082,6 +1082,7 @@ extraction:
   system_prompt: >-
     You are a document assistant. Respond only with JSON. Never make up data, only provide data found in the document being provided.
 summarization:
+  enabled: true
   top_p: '0.1'
   max_tokens: '4096'
   top_k: '5'
diff --git a/config_library/pattern-2/rvl-cdip-package-sample-with-few-shot-examples/config.yaml b/config_library/pattern-2/rvl-cdip-package-sample-with-few-shot-examples/config.yaml
@@ -1069,6 +1069,7 @@ evaluation:
       values match for document attribute extraction. You will consider the
       context and meaning rather than just exact string matching.
 summarization:
+  enabled: true
   top_p: '0.1'
   max_tokens: '4096'
   top_k: '5'
diff --git a/config_library/pattern-2/rvl-cdip-package-sample/config.yaml b/config_library/pattern-2/rvl-cdip-package-sample/config.yaml
@@ -547,6 +547,7 @@ extraction:
   system_prompt: >-
     You are a document assistant. Respond only with JSON. Never make up data, only provide data found in the document being provided.
 summarization:
+  enabled: true
   top_p: '0.1'
   max_tokens: '4096'
   top_k: '5'
diff --git a/config_library/pattern-3/rvl-cdip-package-sample/config.yaml b/config_library/pattern-3/rvl-cdip-package-sample/config.yaml
@@ -407,6 +407,7 @@ extraction:
   system_prompt: >-
     You are a document assistant. Respond only with JSON. Never make up data, only provide data found in the document being provided.
 summarization:
+  enabled: true
   top_p: '0.1'
   max_tokens: '4096'
   top_k: '5'
diff --git a/docs/architecture.md b/docs/architecture.md
@@ -40,7 +40,7 @@ The solution uses a modular architecture with nested CloudFormation stacks to su
 - Centralized monitoring and management across all patterns
 - Pattern-specific optimizations and configurations
 - Optional features that can be enabled across all patterns:
-  - Document summarization (controlled by `IsSummarizationEnabled` parameter)
+  - Document summarization (controlled by configuration `summarization.enabled` property)
     - This feature also enables the "Chat with Document" functionality
     - This feature does not use the Bedrock Knowledge Base but stores a full-text text file in S3
   - Document Knowledge Base (using Amazon Bedrock)
@@ -215,7 +215,7 @@ For detailed information about the Web UI, its features, and usage, see [web-ui.
 
 ### Document Summarization
 
-When enabled via the `IsSummarizationEnabled` parameter (default: true), the solution provides document summarization across all patterns:
+When enabled via the configuration `summarization.enabled` property (default: true), the solution provides document summarization across all patterns:
 
 - All patterns use a dedicated summarization step with Bedrock models
 - Summarization provides a concise overview of the document content
diff --git a/docs/configuration.md b/docs/configuration.md
@@ -16,7 +16,7 @@ The web interface allows real-time configuration updates without stack redeploym
 - **Prompt Engineering**: Customize system and task prompts for optimal results
 - **OCR Features**: Configure Textract features (TABLES, FORMS, SIGNATURES, LAYOUT) for enhanced data capture
 - **Evaluation Methods**: Set evaluation methods and thresholds for each attribute
-- **Summarization**: Configure model, prompts, and parameters for document summarization (when `IsSummarizationEnabled` is true)
+- **Summarization**: Configure model, prompts, parameters, and enable/disable document summarization via the `enabled` property
 
 ### Configuration Management Features
 
@@ -27,6 +27,35 @@ The web interface allows real-time configuration updates without stack redeploym
 
 Configuration changes are validated and applied immediately, with rollback capability if issues arise. See [web-ui.md](web-ui.md) for details on using the administration interface.
 
+## Summarization Configuration
+
+### Enable/Disable Summarization
+
+Summarization can be controlled via the configuration file rather than CloudFormation stack parameters. This provides more flexibility and eliminates the need for stack redeployment when changing summarization behavior.
+
+**Configuration-based Control (Recommended):**
+```yaml
+summarization:
+  enabled: true  # Set to false to disable summarization
+  model: us.anthropic.claude-3-7-sonnet-20250219-v1:0
+  temperature: 0.0
+  # ... other summarization settings
+```
+
+**Key Benefits:**
+- **Runtime Control**: Enable/disable without stack redeployment
+- **Cost Optimization**: Zero LLM costs when disabled (`enabled: false`)
+- **Simplified Architecture**: No conditional logic in state machines
+- **Backward Compatible**: Defaults to `enabled: true` when property is missing
+
+**Behavior When Disabled:**
+- Summarization lambda is still called (minimal overhead)
+- Service immediately returns with logging: "Summarization is disabled in configuration"
+- No LLM API calls or S3 operations are performed
+- Document processing continues to completion
+
+**Migration Note**: The previous `IsSummarizationEnabled` CloudFormation parameter has been removed in favor of this configuration-based approach.
+
 ## Stack Parameters
 
 Key parameters that can be configured during CloudFormation deployment:
@@ -64,7 +93,6 @@ Key parameters that can be configured during CloudFormation deployment:
   - `Pattern3Configuration`: Configuration preset to use
 
 ### Optional Features
-- `IsSummarizationEnabled`: Enable/disable document summarization (default: true)
 - `EvaluationBaselineBucketName`: Optional existing bucket for ground truth data
 - `EvaluationAutoEnabled`: Enable automatic accuracy evaluation (default: true)
 - `DocumentKnowledgeBase`: Enable document knowledge base functionality
diff --git a/docs/pattern-1.md b/docs/pattern-1.md
@@ -68,7 +68,7 @@ InvokeDataAutomation (with waitForTaskToken)
 
 **Stack Deployment Parameters:**
 - `BDAProjectArn`: ARN of your Bedrock Data Automation project
-- `IsSummarizationEnabled`: Boolean to enable/disable summarization functionality (true|false)
+- **Summarization**: Control summarization via configuration file `summarization.enabled` property (replaces `IsSummarizationEnabled` parameter)
 - `ConfigurationDefaultS3Uri`: Optional S3 URI to custom configuration (uses default configuration if not specified)
 - `InputBucket`: S3 bucket for input documents
 - `WorkingBucket`: S3 bucket for temporary BDA job output
@@ -84,7 +84,7 @@ InvokeDataAutomation (with waitForTaskToken)
 **Configuration Management:**
 - Configuration now supports multiple presets per pattern (e.g., default, checkboxed_attributes_extraction, medical_records_summarization)
 - Configuration can be updated through the Web UI without stack redeployment
-- Summarization functionality is controlled through the centralized `IsSummarizationEnabled` parameter rather than model-specific settings
+- Summarization functionality is controlled through the configuration file `summarization.enabled` property rather than CloudFormation parameters
 - BDA-specific configuration is handled within the Bedrock Data Automation project rather than the IDP stack configuration
 
 **Note on BDA Configuration:**
diff --git a/docs/pattern-2.md b/docs/pattern-2.md
@@ -213,7 +213,7 @@ The pattern exports these outputs to the parent stack:
 
 **Stack Deployment Parameters:**
 - `ClassificationMethod`: Classification methodology to use (options: 'multimodalPageLevelClassification' or 'textbasedHolisticClassification')
-- `IsSummarizationEnabled`: Boolean to enable/disable summarization functionality (true|false)
+- **Summarization**: Control summarization via configuration file `summarization.enabled` property (replaces `IsSummarizationEnabled` parameter)
 - `ConfigurationDefaultS3Uri`: Optional S3 URI to custom configuration (uses default configuration if not specified)
 - `MaxConcurrentWorkflows`: Workflow concurrency limit
 - `LogRetentionDays`: CloudWatch log retention period
diff --git a/docs/pattern-3.md b/docs/pattern-3.md
@@ -209,7 +209,7 @@ The pattern exports these outputs to the parent stack:
 
 **Stack Deployment Parameters:**
 - `UDOPModelArtifactPath`: S3 path to UDOP model artifacts (see [Fine tuning a UDOP model](#fine-tuning-a-udop-model-for-classification))
-- `IsSummarizationEnabled`: Boolean to enable/disable summarization functionality (true|false)
+- **Summarization**: Control summarization via configuration file `summarization.enabled` property (replaces `IsSummarizationEnabled` parameter)
 - `ConfigurationDefaultS3Uri`: Optional S3 URI to custom configuration (uses default configuration if not specified)
 - `MaxConcurrentWorkflows`: Workflow concurrency limit
 - `LogRetentionDays`: CloudWatch log retention period
diff --git a/lib/idp_common_pkg/idp_common/assessment/service.py b/lib/idp_common_pkg/idp_common/assessment/service.py
@@ -598,6 +598,12 @@ def process_document_section(self, document: Document, section_id: str) -> Docum
         Returns:
             Document: Updated Document object with assessment results appended to extraction results
         """
+        # Check if assessment is enabled in configuration
+        assessment_config = self.config.get("assessment", {})
+        if not assessment_config.get("enabled", True):
+            logger.info("Assessment is disabled via configuration")
+            return document
+
         # Validate input document
         if not document:
             logger.error("No document provided")
diff --git a/lib/idp_common_pkg/idp_common/summarization/README.md b/lib/idp_common_pkg/idp_common/summarization/README.md
@@ -92,6 +92,7 @@ class SummarizationService:
         store_results: bool = True
     ) -> Document:
         # Process a Document object and update it with summary information
+        # Checks config.summarization.enabled first - returns early if disabled
         # Automatically detects whether to use section-based or whole document summarization
         # store_results parameter controls whether to create and store the markdown report
 ```
@@ -164,6 +165,7 @@ The service requires configuration with the following structure:
 ```json
 {
   "summarization": {
+    "enabled": true,
     "model": "us.amazon.nova-pro-v1:0",
     "temperature": 0,
     "top_k": 5,
@@ -173,6 +175,26 @@ The service requires configuration with the following structure:
 }
 ```
 
+### Configuration Properties
+
+#### `enabled` (boolean)
+- **Purpose**: Controls whether summarization processing is performed
+- **Default**: `true` (for backward compatibility)
+- **Behavior**:
+  - `true`: Summarization processing proceeds normally
+  - `false`: Summarization is skipped entirely with minimal overhead
+
+**Cost Optimization**: When `enabled: false`, no LLM API calls are made, resulting in zero summarization costs.
+
+**Example - Disabling Summarization:**
+```yaml
+summarization:
+  enabled: false  # Disables all summarization processing
+  # Other properties can remain but will be ignored
+  model: us.anthropic.claude-3-7-sonnet-20250219-v1:0
+  temperature: 0.0
+```
+
 The service can handle any JSON structure returned by the model. You can use any field names in your prompt template:
 
 ```json
diff --git a/lib/idp_common_pkg/idp_common/summarization/service.py b/lib/idp_common_pkg/idp_common/summarization/service.py
@@ -393,6 +393,19 @@ def process_document(
         Returns:
             Document: Updated Document object with summary and summarization_result
         """
+        # Check if summarization is enabled in configuration
+        summarization_config = self.config.get("summarization", {})
+        if not summarization_config.get(
+            "enabled", True
+        ):  # Default to True for backward compatibility
+            logger.info(
+                f"Summarization is disabled in configuration for document {document.id}, skipping processing"
+            )
+            # Update document status to completed if not already failed
+            if document.status != Status.FAILED:
+                document.status = Status.COMPLETED
+            return document
+
         if not document.pages:
             logger.warning("Document has no pages to summarize")
             return self._update_document_status(
diff --git a/lib/idp_common_pkg/tests/unit/summarization/test_summarization_service.py b/lib/idp_common_pkg/tests/unit/summarization/test_summarization_service.py
diff --git a/patterns/pattern-1/statemachine/workflow.asl.json b/patterns/pattern-1/statemachine/workflow.asl.json
diff --git a/patterns/pattern-1/template.yaml b/patterns/pattern-1/template.yaml
diff --git a/patterns/pattern-2/statemachine/workflow.asl.json b/patterns/pattern-2/statemachine/workflow.asl.json
diff --git a/patterns/pattern-2/template.yaml b/patterns/pattern-2/template.yaml
diff --git a/patterns/pattern-3/statemachine/workflow.asl.json b/patterns/pattern-3/statemachine/workflow.asl.json
diff --git a/patterns/pattern-3/template.yaml b/patterns/pattern-3/template.yaml
diff --git a/template.yaml b/template.yaml
