Merge remote-tracking branch 'origin/develop' into feature/chat-with-document

Author: ilmarmur

CHANGELOG.md

@@ -6,6 +6,23 @@ SPDX-License-Identifier: MIT-0
 ## [Unreleased]
 
 ### Added
+
+## [0.3.16]
+
+### Added
+
+- **S3 Vectors Support for Cost-Optimized Knowledge Base Storage**
+  - Added S3 Vectors as alternative vector store option to OpenSearch Serverless for Bedrock Knowledge Base with lower storage costs
+  - Custom resource Lambda implementation for S3 vector bucket and index management (using boto3 s3vectors client) with proper IAM permissions and resource cleanup
+  - Unified Knowledge Base interface supporting both vector store types with automatic resource provisioning based on user selection
+
+- **Page Limit Configuration for Classification Control**
+  - Added `maxPagesForClassification` configuration option to control how many pages are used during document classification
+  - **Default Behavior**: `"ALL"` - uses all pages for classification (existing behavior)
+  - **Limited Page Classification**: Set to numeric value (e.g., `"1"`, `"2"`, `"3"`) to classify only the first N pages
+  - **Important**: When using numeric limit, the classification result from the first N pages is applied to ALL pages in the document, effectively forcing the entire document to be assigned a single class with one section
+  - **Use Cases**: Performance optimization for large documents, cost reduction for documents with consistent classification patterns, simplified processing for homogeneous document types
+
 - **CloudFormation Service Role for Delegated Deployment Access**
   - Added example CloudFormation service role template that enables non-administrator users to deploy and maintain IDP stacks without requiring ongoing administrator permissions
   - Administrators can provision the service role once with elevated privileges, then delegate deployment capabilities to developer/DevOps teams
@@ -18,6 +35,7 @@ SPDX-License-Identifier: MIT-0
 - Fix occasional UI error 'Failed to get document details - please try again later' - resolves #58
 - Fixed UI zipfile creation to exclude .aws-sam directories and .env files from deployment package
 - Added security recommendation to set LogLevel parameter to WARN or ERROR (not INFO) for production deployments to prevent logging of sensitive information including PII data, document contents, and S3 presigned URLs
+- Hardened several aspects of the new Discovery feature
 
 ## [0.3.15]
 

VERSION

@@ -1 +1 @@
-0.3.16-wip3
+0.3.16

docs/knowledge-base.md

@@ -5,12 +5,18 @@ SPDX-License-Identifier: MIT-0
 
 The GenAIIDP solution includes an integrated Document Knowledge Base query feature that enables you to interactively ask questions about your processed document collection using natural language. This feature leverages the processed data to create a searchable knowledge base.
 
+
+https://github.com/user-attachments/assets/991b4112-0fc9-4e4d-98ab-ef4e3cbae04a
+
+
+
 ## How It Works
 
-1. **Document Indexing**
+1. **Document Processing & Indexing**
    - Processed documents are automatically indexed in a vector database
    - Documents are chunked into semantic segments for efficient retrieval
    - Each chunk maintains reference to its source document
+   - **Ingestion Schedule**: Documents are ingested into the knowledge base every 30 minutes, so newly processed documents may not be immediately available for querying
 
 2. **Interactive Query Interface**
    - Access through the Web UI via the "Knowledge Base" section
@@ -33,6 +39,25 @@ The GenAIIDP solution includes an integrated Document Knowledge Base query featu
 - **Markdown Formatting**: Responses support rich formatting for better readability
 - **Real-time Processing**: Get answers in seconds, even across large document collections
 
+## Architecture & Vector Storage Options
+
+The Knowledge Base feature supports two vector storage backends to optimize for different performance and cost requirements:
+
+### Vector Store Comparison
+
+| Aspect | OpenSearch Serverless | S3 Vectors |
+|--------|----------------------|------------|
+| **Query Latency** | Sub-millisecond | Sub-second |
+| **Pricing Model** | Always On (continuous capacity costs) | On Demand (pay-per-query) |
+| **Storage Cost** | Higher | 40-60% lower |
+| **Best For** | Real-time applications | Cost-sensitive deployments |
+| **Features** | Full-text search, advanced filtering | Native S3 integration |
+
+### Choosing Your Vector Store
+
+- **OpenSearch Serverless** (Default): Choose for applications requiring ultra-fast retrieval and real-time performance
+- **S3 Vectors**: Choose for cost optimization when query latency is acceptable
+
 ## Configuration
 
 The Document Knowledge Base Query feature can be configured during stack deployment:
@@ -46,14 +71,29 @@ ShouldUseDocumentKnowledgeBase:
     - "false"
   Description: Enable/disable the Document Knowledge Base feature
 
+KnowledgeBaseVectorStore:
+  Type: String
+  Default: "OPENSEARCH_SERVERLESS"
+  AllowedValues:
+    - "OPENSEARCH_SERVERLESS"
+    - "S3_VECTORS"
+  Description: Vector storage backend for the knowledge base
+
 DocumentKnowledgeBaseModel:
   Type: String
   Default: "us.amazon.nova-pro-v1:0"
   Description: Bedrock model to use for knowledge base queries (e.g., "us.anthropic.claude-3-7-sonnet-20250219-v1:0")
 ```
 
+### Supported Embedding Models
+
+Both vector store types support the same embedding models:
+- `amazon.titan-embed-text-v2:0` (default)
+- `cohere.embed-english-v3`  (disabled by default)
+- `cohere.embed-multilingual-v3` (disabled by default)
+
 When the feature is enabled, the solution:
-- Creates necessary OpenSearch resources for document indexing
+- Creates the selected vector storage resources (OpenSearch or S3 Vectors)
 - Configures API endpoints for querying the knowledge base
 - Adds the query interface to the Web UI
 
@@ -111,3 +151,14 @@ The Knowledge Base feature maintains the security controls of the overall soluti
 - Document visibility respects user permissions
 - Questions and answers are processed securely within your AWS account
 - No data is sent to external services beyond the configured Bedrock models
+
+## Future Enhancements
+
+### Potential Improvements & Community Contributions
+- **CloudFormation Support**: When S3 Vectors gains native CloudFormation support
+- **Migration Tools**: Utilities to migrate between vector store types
+- **Hybrid Deployment**: Support for multiple Knowledge Bases with different vector stores
+- **Document Chunking Options**: The system currently uses default chunking strategies, with additional chunking methods available for optimization based on document types and use cases
+- Performance optimization suggestions
+- Additional embedding model support
+- Enhanced monitoring and alerting