docs: reflect schema-driven completeness engine delivery

Author: Cataldir

docs/architecture/components/apps/product-management-consistency-validation.md

@@ -2,11 +2,15 @@
 
 **Path**: `apps/product-management-consistency-validation/`  
 **Domain**: Product Management  
-**Purpose**: Validate catalog completeness and consistency
+**Purpose**: Evaluate schema-driven catalog completeness and publish enrichment triggers for missing enrichable attributes
 
 ## Overview
 
-Checks product data for missing fields, invalid pricing, and incomplete media.
+Implements a schema-driven Completeness Engine that:
+- evaluates products against category field definitions,
+- computes weighted completeness scores,
+- stores gap reports,
+- triggers enrichment workflows for enrichable gaps below threshold.
 
 ## Architecture
 
@@ -15,7 +19,11 @@ graph LR
     Client[Validation Request] -->|POST /invoke| API[FastAPI App]
     API --> Agent[Consistency Agent]
     Agent --> Products[Product Adapter]
-    Agent --> Validator[Consistency Rules]
+    Agent --> Validator[Legacy Validator]
+    EH[Event Hub completeness-jobs] --> Consumer[Completeness Event Consumer]
+    Consumer --> Engine[Completeness Engine]
+    Engine --> Cosmos[Completeness Storage]
+    Engine --> EH2[Event Hub enrichment-jobs]
 ```
 
 ## Components
@@ -34,28 +42,50 @@ graph LR
 
 Orchestrates:
 - Product retrieval
-- Consistency validation
+- Legacy consistency validation (`/invoke` flow)
+
+### 3. Completeness Engine (`completeness_engine.py`)
+
+Provides:
+- `CategorySchema`, `FieldDefinition`, `FieldGap`, `GapReport`
+- weighted completeness score computation
+- nested dot-path field evaluation
+- enrichable gap extraction
+
+### 4. Completeness Event Consumer (`event_consumer.py`)
+
+Consumes `completeness-jobs` and executes:
+- load product
+- load category schema
+- evaluate completeness
+- persist gap report
+- publish `enrichment_requested` to `enrichment-jobs` when score is below `COMPLETENESS_THRESHOLD`
 
 **Current Status**: ✅ **IMPLEMENTED (mock adapters)**
 
-### 3. Adapters
+### 5. Adapters
 
 **Product Adapter**: Catalog product retrieval  
-**Validator**: Completeness rules
+**Validator**: Legacy consistency rules
+**Completeness Storage**: Cosmos-backed schema/gap-report adapter with in-memory fallback
 
-**Current Status**: ⚠️ **PARTIAL** — Mock adapters return deterministic data
+**Current Status**: ✅ **IMPLEMENTED** — Completeness engine pipeline is active; storage supports local in-memory fallback for tests/dev
 
 ## What's Implemented
 
 ✅ MCP tool registration  
-✅ Consistency validation agent orchestration  
+✅ Legacy consistency validation agent orchestration  
+✅ Schema-driven completeness scoring pipeline  
+✅ Completeness Event Hub consumer (`completeness-jobs`)  
+✅ Enrichment trigger publishing (`enrichment-jobs`)  
+✅ Cosmos/in-memory completeness storage adapter  
+✅ Unit + integration tests for completeness engine and event flow  
 ✅ Dockerfile + Bicep module
 
 ## What's NOT Implemented
 
-❌ Real product integrations  
-❌ Foundry model integration for remediation guidance  
-❌ Observability dashboards for data quality
+❌ Dedicated remediation model orchestration (currently only trigger publication)  
+❌ Dedicated observability dashboards for completeness quality trends
 
 ## Operational Playbooks
 

docs/implementation/truth-layer-api.md

@@ -6,7 +6,7 @@ This document describes currently available truth-layer endpoints and planned se
 
 - Implemented services in the current repo/deployment topology:
   - `truth-ingestion` (custom REST ingestion routes + standard service endpoints)
-  - `product-management-consistency-validation` (completeness checks via `/invoke`)
+  - `product-management-consistency-validation` (legacy `/invoke` + event-driven completeness engine)
   - `ecommerce-product-detail-enrichment` (enrichment via `/invoke`)
   - `product-management-acp-transformation` (ACP export via `/invoke`)
   - `crud-service` (transactional APIs, including review endpoints used as interim review flow)
@@ -83,7 +83,24 @@ Base URL: `http://<consistency-host>`
 | --- | --- | --- |
 | GET | `/health` | Liveness |
 | GET | `/ready` | Readiness |
-| POST | `/invoke` | Validate SKU consistency/completeness |
+| POST | `/invoke` | Legacy consistency/completeness validation |
+
+### Event-driven completeness flow (implemented)
+
+- **Consumes**: Event Hub topic `completeness-jobs` (consumer group: `completeness-engine`)
+- **Loads**: product + category schema
+- **Evaluates**: weighted completeness score and field-level gaps
+- **Stores**: gap report via completeness storage adapter (Cosmos-backed with local/test fallback)
+- **Publishes**: `enrichment_requested` to `enrichment-jobs` when:
+  - completeness score `< COMPLETENESS_THRESHOLD` (default `0.7`)
+  - enrichable gaps are present
+
+### Completeness report model highlights
+
+- `entity_id`, `category_id`, `schema_version`
+- `completeness_score` (`0.0`–`1.0`)
+- `gaps[]` with gap type (`missing` / `invalid`)
+- `enrichable_gaps[]`
 
 ### Example request: `POST /invoke` (completeness)
 

docs/project-status.md

@@ -130,7 +130,7 @@ Remaining quality issues to address.
 | [#94](https://github.com/Azure-Samples/holiday-peak-hub/issues/94) | Phase 1: Event Hub helpers | 1 | PR #150 (Draft) |
 | [#96](https://github.com/Azure-Samples/holiday-peak-hub/issues/96) | Phase 2: Generic REST PIM connector | 2 | PR #151 (Draft) |
 | [#98](https://github.com/Azure-Samples/holiday-peak-hub/issues/98) | Phase 2: Truth Ingestion service | 2 | ✅ Closed (PR #146) |
-| [#99](https://github.com/Azure-Samples/holiday-peak-hub/issues/99) | Phase 2: Completeness Engine refactor | 2 | Open |
+| [#99](https://github.com/Azure-Samples/holiday-peak-hub/issues/99) | Phase 2: Completeness Engine refactor | 2 | ✅ Closed (PR #123) |
 | [#101](https://github.com/Azure-Samples/holiday-peak-hub/issues/101) | Phase 3: Truth Enrichment service | 3 | PR #125 (Draft) |
 | [#102](https://github.com/Azure-Samples/holiday-peak-hub/issues/102) | Phase 3: Truth HITL service (Human-in-the-Loop) | 3 |
 | [#103](https://github.com/Azure-Samples/holiday-peak-hub/issues/103) | Phase 3: HITL Staff Review UI pages | 3 |

docs/roadmap/012-product-truth-layer-plan.md

@@ -84,7 +84,7 @@ The spec defines a **deployable reference implementation** for retailers to inge
 | Connector contracts | `PIMConnectorBase`, `DAMConnectorBase` + 7 other ABCs in `integrations/contracts.py` | **Good** — Need concrete implementations |
 | Connector registry | `ConnectorRegistry` in `integrations/registry.py` | **Good** — Runtime registry with health monitoring |
 | App factory | `build_service_app()` in `app_factory.py` | **Strong** — Standard service bootstrap |
-| Consistency validation | `product-management-consistency-validation` agent | **Weak** — Only checks 4 fields (name, price sign, currency, image); no completeness scoring |
+| Consistency validation | `product-management-consistency-validation` agent | **Implemented** — Schema-driven completeness scoring, gap reporting, and enrichment trigger integration (PR #123) |
 | Product enrichment | `ecommerce-product-detail-enrichment` agent | **Wrong target** — Enriches PDP display, not PIM attributes |
 | Product schemas | `CatalogProduct`, `ProductContext` in `schemas/product.py` | **Partial** — Missing style/variant split, provenance, share policy |
 | IaC | Full Bicep stack: AKS, ACR, Cosmos DB, Event Hubs, Redis, Storage, APIM, Key Vault, App Insights, VNet, AI Foundry | **Strong** — All infra provisioned; Cosmos containers empty, Service Bus absent |
@@ -120,7 +120,7 @@ Latest on `origin/main`: commit `74d56c6` — "Fix APIM routing, postdeploy hook
 | G1 | §3.1 Core Data Plane | No Cosmos DB containers for product graph, candidates, schemas, audit. Cosmos container array is `[]` in Bicep. | **CRITICAL** |
 | G2 | §7 Data Model | No `ProductStyle`, `ProductVariant`, `TruthAttribute`, `ProposedAttribute` models. Only flat `CatalogProduct`. | **CRITICAL** |
 | G3 | §3.1 Ingestion | No ingestion service. PIM/DAM connector ABCs exist but no concrete connectors or scheduled pull. | **CRITICAL** |
-| G4 | §3.1 Completeness Engine | No completeness scoring. Existing validation checks 4 fields. No category schema definitions. | **CRITICAL** |
+| G4 | §3.1 Completeness Engine | **Resolved in PR #123** — weighted completeness scoring, schema-driven gap analysis, and Event Hub enrichment trigger are implemented. | **CLOSED** |
 | G5 | §3.1 Enrichment Orchestrator | No PIM enrichment agent. Existing enrichment targets e-commerce PDP, not product graph. | **CRITICAL** |
 | G6 | §3.1 HITL Workflow | Zero implementation. No approval endpoints, no review queue, no UI. | **CRITICAL** |
 | G7 | §8 Category Schemas | No `/schemas/` directory. No category-level required attributes or validation rules. | **CRITICAL** |
@@ -186,20 +186,21 @@ GapReport          — entityId, missingKeys[], invalidKeys[], completenessScore
 **Add**: Optional parameter to register truth-layer specific middleware (audit logging, provenance tracking).
 **Add**: Truth-layer Event Hub consumer groups and lifespan helpers for job topics.
 
-### 4.7 `apps/product-management-consistency-validation/` — MAJOR REFACTOR
-
-**Current**: Checks 4 fields (name, price, currency, image).
-**Target**: Becomes the **Completeness Engine** (§3.1.3, §8). Must:
-- Load category schemas from Cosmos `schemas` container
-- Compute `missing = required - truthAttributesPresent`
-- Compute `invalid = truthAttributesFailValidation`
-- Generate `GapReport` with `completenessScore`
-- Publish enrichment job events to Event Hub
-
-Keep existing MCP tools and add new ones:
-- `/product/completeness/check` — full gap analysis
-- `/product/completeness/score` — score-only endpoint
-- `/product/completeness/batch` — batch job trigger
+### 4.7 `apps/product-management-consistency-validation/` — COMPLETED (PR #123)
+
+Delivered capabilities:
+- Added schema-driven completeness engine (`completeness_engine.py`) with:
+  - weighted scoring (`0.0`–`1.0`)
+  - nested field-path evaluation
+  - per-field gap typing (`missing`, `invalid`)
+  - enrichable gap extraction
+- Added completeness job consumer (`event_consumer.py`) for `completeness-jobs`.
+- Added enrichment trigger publishing to `enrichment-jobs` when:
+  - completeness score is below `COMPLETENESS_THRESHOLD`
+  - enrichable gaps exist.
+- Added Cosmos-backed completeness storage adapter with in-memory fallback for local/test.
+- Preserved backward compatibility of existing validator pathways.
+- Added unit and integration test coverage for scoring and event flow.
 
 ### 4.8 `apps/product-management-acp-transformation/` — EXTEND