Merge branch 'fix/quality-database-table-partitions' into 'develop'

Improved S3 Partition Key Format for Better Date Range Filtering

See merge request genaiic-reusable-assets/engagement-artifacts/genaiic-idp-accelerator!187

Author: rstrahan

CHANGELOG.md

@@ -61,6 +61,10 @@ SPDX-License-Identifier: MIT-0
 - Defend against non-numeric confidence_threshold values in the configuration - avoid float conversion or numeric comparison exceptions in Assessement step
 - Prevent creation of empty configuration fields in UI
 - Firefox browser issues with signed URLs (PR #14)
+- Improved S3 Partition Key Format for Better Date Range Filtering:
+  - Updated reporting data partition keys to use YYYY-MM format for month and YYYY-MM-DD format for day
+  - Enables easier date range filtering in analytics queries across different months and years
+  - Partition structure now: `year=2024/month=2024-03/day=2024-03-15/` instead of `year=2024/month=03/day=15/`
 
 ## [0.3.3]
 

docs/reporting-database.md

@@ -37,7 +37,7 @@ The `document_evaluations` table contains document-level evaluation metrics:
 | false_discovery_rate | double | False discovery rate (0-1) |
 | execution_time | double | Time taken to evaluate (seconds) |
 
-This table is partitioned by year, month, day, and document ID.
+This table is partitioned by year, month (YYYY-MM format), day (YYYY-MM-DD format), and document ID.
 
 ### Section Evaluations
 
@@ -56,7 +56,7 @@ The `section_evaluations` table contains section-level evaluation metrics:
 | false_discovery_rate | double | Section false discovery rate (0-1) |
 | evaluation_date | timestamp | When the evaluation was performed |
 
-This table is partitioned by year, month, day, and document ID.
+This table is partitioned by year, month (YYYY-MM format), day (YYYY-MM-DD format), and document ID.
 
 ### Attribute Evaluations
 
@@ -78,7 +78,7 @@ The `attribute_evaluations` table contains attribute-level evaluation metrics:
 | confidence_threshold | string | Confidence threshold used |
 | evaluation_date | timestamp | When the evaluation was performed |
 
-This table is partitioned by year, month, day, and document ID.
+This table is partitioned by year, month (YYYY-MM format), day (YYYY-MM-DD format), and document ID.
 
 ## Metering Table
 
@@ -94,7 +94,7 @@ The `metering` table captures detailed usage metrics for each document processin
 | number_of_pages | int | Number of pages in the document |
 | timestamp | timestamp | When the operation was performed |
 
-This table is partitioned by year, month, day, and document ID.
+This table is partitioned by year, month (YYYY-MM format), day (YYYY-MM-DD format), and document ID.
 
 The metering table is particularly valuable for:
 - Cost analysis and allocation

lib/idp_common_pkg/idp_common/reporting/save_reporting_data.py

@@ -250,8 +250,8 @@ def save_evaluation_results(self, document: Document) -> Optional[Dict[str, Any]
                 evaluation_date = doc_time
                 year, month, day = (
                     doc_time.strftime("%Y"),
-                    doc_time.strftime("%m"),
-                    doc_time.strftime("%d"),
+                    doc_time.strftime("%Y-%m"),
+                    doc_time.strftime("%Y-%m-%d"),
                 )
                 logger.info(
                     f"Using document initial_event_time: {document.initial_event_time} for partitioning"
@@ -263,8 +263,8 @@ def save_evaluation_results(self, document: Document) -> Optional[Dict[str, Any]
                 evaluation_date = datetime.datetime.now()
                 year, month, day = (
                     evaluation_date.strftime("%Y"),
-                    evaluation_date.strftime("%m"),
-                    evaluation_date.strftime("%d"),
+                    evaluation_date.strftime("%Y-%m"),
+                    evaluation_date.strftime("%Y-%m-%d"),
                 )
         else:
             logger.warning(
@@ -273,8 +273,8 @@ def save_evaluation_results(self, document: Document) -> Optional[Dict[str, Any]
             evaluation_date = datetime.datetime.now()
             year, month, day = (
                 evaluation_date.strftime("%Y"),
-                evaluation_date.strftime("%m"),
-                evaluation_date.strftime("%d"),
+                evaluation_date.strftime("%Y-%m"),
+                evaluation_date.strftime("%Y-%m-%d"),
             )
 
         # Escape document ID by replacing slashes with underscores
@@ -435,8 +435,8 @@ def save_metering_data(self, document: Document) -> Optional[Dict[str, Any]]:
                 timestamp = doc_time
                 year, month, day = (
                     doc_time.strftime("%Y"),
-                    doc_time.strftime("%m"),
-                    doc_time.strftime("%d"),
+                    doc_time.strftime("%Y-%m"),
+                    doc_time.strftime("%Y-%m-%d"),
                 )
                 logger.info(
                     f"Using document initial_event_time: {document.initial_event_time} for partitioning"
@@ -448,8 +448,8 @@ def save_metering_data(self, document: Document) -> Optional[Dict[str, Any]]:
                 timestamp = datetime.datetime.now()
                 year, month, day = (
                     timestamp.strftime("%Y"),
-                    timestamp.strftime("%m"),
-                    timestamp.strftime("%d"),
+                    timestamp.strftime("%Y-%m"),
+                    timestamp.strftime("%Y-%m-%d"),
                 )
         else:
             logger.warning(
@@ -458,8 +458,8 @@ def save_metering_data(self, document: Document) -> Optional[Dict[str, Any]]:
             timestamp = datetime.datetime.now()
             year, month, day = (
                 timestamp.strftime("%Y"),
-                timestamp.strftime("%m"),
-                timestamp.strftime("%d"),
+                timestamp.strftime("%Y-%m"),
+                timestamp.strftime("%Y-%m-%d"),
             )
 
         # Escape document ID by replacing slashes with underscores