Merge pull request #80 from kazmer97/feature/json-schema-extraction-model

Feature/json schema extraction model

Author: rstrahan

ANALYSIS_JSON_SCHEMA_LIBRARIES.md

@@ -0,0 +1,236 @@
+# JSON Schema Library Utilization Analysis
+
+## Current State
+
+### Backend (Python)
+**Library Available:** `jsonschema` (Draft202012Validator)
+**Currently Used In:**
+- ✅ `src/lambda/update_configuration/index.py:88-135` - Validates extractionSchema on upload
+- ✅ `lib/idp_common_pkg/idp_common/extraction/agentic_idp.py` - Some validation
+
+**Schema Definitions Available:**
+- ✅ `EXTRACTION_CLASS_SCHEMA` in `schema_definition.py`
+- ✅ `EXTRACTION_SCHEMA_ARRAY` (referenced but need to verify)
+- ✅ Comprehensive schema with AWS extensions defined
+
+### Frontend (JavaScript)
+**Library Available:** `ajv` + `ajv-formats`
+**Currently Used In:**
+- ✅ `src/ui/src/hooks/useSchemaValidation.js` - Custom validation logic
+- ⚠️  Duplicates validation that could use AJV's built-in features
+
+## Opportunities for Improvement
+
+### HIGH PRIORITY: Backend Migration Validation
+
+**File:** `lib/idp_common_pkg/idp_common/config_schema/migration.py`
+**Issue:** NO validation after migration
+**Risk:** Can produce invalid schemas that break downstream
+
+**Current:**
+```python
+def migrate_legacy_to_schema(legacy_classes):
+    # ... migration logic ...
+    return _convert_classes_to_json_schema(migrated_classes)
+    # NO VALIDATION!
+```
+
+**Proposed:**
+```python
+def migrate_legacy_to_schema(legacy_classes, validate=True):
+    result = _convert_classes_to_json_schema(migrated_classes)
+    
+    if validate:
+        from jsonschema import Draft202012Validator, ValidationError
+        from .schema_definition import EXTRACTION_CLASS_SCHEMA
+        
+        validator = Draft202012Validator(EXTRACTION_CLASS_SCHEMA)
+        try:
+            if isinstance(result, list):
+                for schema in result:
+                    validator.validate(schema)
+            else:
+                validator.validate(result)
+        except ValidationError as e:
+            raise ValueError(f"Migration produced invalid schema: {e.message}")
+    
+    return result
+```
+
+**Impact:** Prevents invalid schemas from being created
+
+---
+
+### HIGH PRIORITY: Validate AWS Extensions
+
+**File:** `lib/idp_common_pkg/idp_common/config_schema/migration.py:56-67`
+**Issue:** No validation of AWS extension values
+**Risk:** Invalid evaluation_method, confidence_threshold can be stored
+
+**Current:**
+```python
+if "evaluation_method" in attr:
+    schema_attr["x-aws-idp-evaluation-method"] = attr["evaluation_method"]
+    # No validation!
+
+if "confidence_threshold" in attr:
+    threshold = attr["confidence_threshold"]
+    # Weak string-to-float conversion
+```
+
+**Proposed:** Use schema definition to validate
+```python
+def _validate_aws_extensions(extensions: Dict[str, Any]) -> None:
+    """Validate AWS IDP extensions against schema."""
+    from jsonschema import Draft202012Validator, ValidationError
+    
+    # Extract AWS extension schema from EXTRACTION_CLASS_SCHEMA
+    aws_extension_schema = {
+        "type": "object",
+        "properties": {
+            "x-aws-idp-evaluation-method": {
+                "type": "string",
+                "enum": ["EXACT", "NUMERIC_EXACT", "FUZZY", "SEMANTIC"]
+            },
+            "x-aws-idp-confidence-threshold": {
+                "type": "number",
+                "minimum": 0,
+                "maximum": 1
+            }
+        }
+    }
+    
+    validator = Draft202012Validator(aws_extension_schema)
+    try:
+        validator.validate(extensions)
+    except ValidationError as e:
+        raise ValueError(f"Invalid AWS extension: {e.message}")
+```
+
+---
+
+### MEDIUM PRIORITY: Frontend - Use AJV for All Validation
+
+**File:** `src/ui/src/hooks/useSchemaValidation.js:67-133`
+**Issue:** Manual validation logic duplicates what AJV can do
+
+**Current:** Manual checks for minLength/maxLength, etc.
+```javascript
+if (attribute.type === 'string') {
+  if (attribute.minLength !== undefined && attribute.maxLength !== undefined) {
+    if (attribute.minLength > attribute.maxLength) {
+      errors.push({ path: '/minLength', message: 'minLength cannot be greater than maxLength' });
+    }
+  }
+}
+```
+
+**Proposed:** Define meta-schema and use AJV
+```javascript
+const ATTRIBUTE_META_SCHEMA = {
+  type: 'object',
+  properties: {
+    type: { enum: ['string', 'number', 'integer', 'boolean', 'object', 'array', 'null'] },
+    // AJV will validate all JSON Schema keywords automatically
+  },
+  // Add custom formats for AWS extensions
+  if: { properties: { type: { const: 'string' } } },
+  then: {
+    properties: {
+      minLength: { type: 'integer', minimum: 0 },
+      maxLength: { type: 'integer', minimum: 0 }
+    },
+    // AJV can validate this relationship:
+    if: { 
+      required: ['minLength', 'maxLength']
+    },
+    then: {
+      // Custom keyword or use ajv-keywords plugin
+    }
+  }
+};
+
+const validateAttribute = useCallback((attribute) => {
+  const validate = ajv.compile(ATTRIBUTE_META_SCHEMA);
+  const valid = validate(attribute);
+  
+  if (!valid) {
+    return {
+      valid: false,
+      errors: validate.errors.map(err => ({
+        path: err.instancePath,
+        message: err.message
+      }))
+    };
+  }
+  
+  return { valid: true, errors: [] };
+}, [ajv]);
+```
+
+**Benefits:**
+- Eliminate 70+ lines of manual validation
+- Leverage AJV's optimized validation
+- Automatically handle new JSON Schema keywords
+
+---
+
+### MEDIUM PRIORITY: Backend - Validate Configuration on Read
+
+**File:** `lib/idp_common_pkg/idp_common/config/configuration_manager.py`
+**Issue:** No validation when reading from DynamoDB
+
+**Proposed:**
+```python
+def get_configuration(self, configuration_type: str, validate=True) -> Dict[str, Any]:
+    """Get configuration with optional validation."""
+    config = # ... fetch from DynamoDB ...
+    
+    if validate and 'classes' in config:
+        from idp_common.config_schema import validate_extraction_schema
+        try:
+            validate_extraction_schema(config['classes'])
+        except Exception as e:
+            logger.error(f"Invalid config in DynamoDB: {e}")
+            # Could return default or raise
+    
+    return config
+```
+
+---
+
+### LOW PRIORITY: Frontend - Share Schema Definition
+
+**Issue:** Schema definition duplicated between frontend and backend
+
+**Current:**
+- Backend: `lib/idp_common_pkg/idp_common/config_schema/schema_definition.py`
+- Frontend: Partial schema in `useSchemaValidation.js`
+
+**Proposed:**
+- Generate JSON file from Python schema definition
+- Import in both frontend and backend
+- Single source of truth
+
+---
+
+## Implementation Priority
+
+### Phase 1 (HIGH - Immediate)
+1. ✅ Add validation to `migrate_legacy_to_schema()` 
+2. ✅ Add AWS extension validation in migration
+3. ✅ Validate after migration in configuration_resolver
+
+### Phase 2 (MEDIUM - This Sprint)
+4. ⚠️  Use AJV meta-schema in useSchemaValidation
+5. ⚠️  Add validation on config read in ConfigurationManager
+
+### Phase 3 (LOW - Future)
+6. ⬜ Share schema definition between frontend/backend
+7. ⬜ Add JSON Schema $ref resolution using library features
+
+## Code Savings Estimate
+
+- Backend validation: +50 lines (new code for safety)
+- Frontend AJV improvements: -70 lines (eliminate manual validation)
+- **Net:** -20 lines, +much better validation coverage

CHANGELOG.md

@@ -13,22 +13,15 @@ SPDX-License-Identifier: MIT-0
   - Added support for Claude Haiku 4.5
   - Available for configuration across all document processing steps
 
-- **X-Ray Integration for Error Analyzer Agent**
-  - Integrated AWS X-Ray tracing tools to enhance diagnostic capabilities of the error analyzer agent
-  - X-Ray context enables better distinction between infrastructure issues and application logic failures
-  - Added trace ID persistence in DynamoDB alongside document status for complete traceability
-  - Enhanced CloudWatch error log filtering for more targeted error analysis
-  - Simplified CloudWatch results structure for improved readability and analysis
-  - Updated error analyzer recommendations to leverage X-Ray insights for more accurate root cause identification
-
-- **EU Region Support with Automatic Model Mapping**
-  - Added support for deploying the solution in EU regions (eu-central-1, eu-west-1, etc.)
-  - Automatic model endpoint mapping between US and EU regions for seamless deployment
-  - Comprehensive model mapping table covering Amazon Nova and Anthropic Claude models
-  - Intelligent fallback mappings when direct EU equivalents are unavailable
-  - Quick Launch button for eu-central-1 region in README and deployment documentation
-  - IDP CLI now supports eu-central-1 deployment with automatic template URL selection
-  - Complete technical documentation in `docs/eu-region-model-support.md` with best practices and troubleshooting
+- **JSON Schema Format for Class Definitions** - [docs/json-schema-migration.md](./docs/json-schema-migration.md)
+  - Document class definitions now use industry-standard JSON Schema Draft 2020-12 format for improved flexibility and tooling integration
+  - **Standards-Based Validation**: Leverage standard JSON Schema validators and tooling ecosystem for better configuration validation
+  - **Enhanced Extensibility**: Custom IDP properties use standard JSON Schema extension pattern (`x-aws-idp-*` prefix) for clean separation of concerns
+  - **Modern Data Contract**: Define document structures using widely-adopted JSON Schema format with robust type system (`string`, `number`, `boolean`, `object`, `array`)
+  - **Nested Structure Support**: Natural representation of complex documents with nested objects and arrays using JSON Schema's native `properties` and `items` keywords
+  - **Automatic Migration**: Existing legacy configurations automatically migrate to JSON Schema format on first load - completely transparent to users
+  - **Backward Compatible**: Legacy format remains supported through automatic migration - no manual configuration updates required
+  - **Comprehensive Documentation**: New migration guide with format comparison, field mapping table, and best practices
 
 ### Changed
 
@@ -42,9 +35,9 @@ SPDX-License-Identifier: MIT-0
 
 
 - **Migrated UI Build System from Create React App to Vite**
-  - Upgraded to Vite 7 for faster build times
+  - Upgraded to Vite 7 for faster build times and improved developer experience
   - Updated to React 18, AWS Amplify v6, react-router-dom v6, and Cloudscape Design System
-  - Reduced dependencies and node_modules size
+  - Reduced dependencies and node_modules size for faster installs
   - Implemented strategic code splitting for improved performance
   - Environment variables now use `VITE_` prefix instead of `REACT_APP_` for local development
 

README.md

@@ -161,6 +161,7 @@ For detailed deployment and testing instructions, see the [Deployment Guide](./d
 - [Agent Analysis](./docs/agent-analysis.md) - Natural language analytics and data visualization feature
 - [Custom MCP Agent](./docs/custom-MCP-agent.md) - Integrating external MCP servers for custom tools and capabilities
 - [Configuration](./docs/configuration.md) - Configuration and customization options
+- [JSON Schema Migration](./docs/json-schema-migration.md) - JSON Schema format guide and legacy migration details
 - [Discovery](./docs/discovery.md) - Pattern-neutral discovery process and BDA blueprint automation
 - [Classification](./docs/classification.md) - Customizing document classification
 - [Extraction](./docs/extraction.md) - Customizing information extraction

docs/configuration.md

@@ -5,12 +5,14 @@ SPDX-License-Identifier: MIT-0
 
 The GenAIIDP solution provides multiple configuration approaches to customize document processing behavior to suit your specific needs.
 
+> **📝 Note:** Starting with version 0.3.21, document class definitions use **JSON Schema** format instead of the legacy custom format. See [json-schema-migration.md](json-schema-migration.md) for migration details and format comparison. Legacy configurations are automatically migrated on first use.
+
 ## Pattern Configuration via Web UI
 
 The web interface allows real-time configuration updates without stack redeployment:
 
-- **Document Classes**: Define and modify document categories and their descriptions
-- **Extraction Attributes**: Configure fields to extract for each document class
+- **Document Classes**: Define and modify document categories and their descriptions (using JSON Schema format)
+- **Extraction Attributes**: Configure fields to extract for each document class (defined as JSON Schema properties)
 - **Few Shot Examples**: Upload and configure example documents to improve accuracy (supported in Pattern 2)
 - **Model Selection**: Choose between available Bedrock models for classification and extraction
 - **Prompt Engineering**: Customize system and task prompts for optimal results