[LOW] Implement Data Governance Features (Classification, Lineage, Retention)

[frankbria]

🎯 Overview

Add enterprise data governance features for data classification, lineage tracking, and retention policies.

✅ Acceptance Criteria

• Data classification system (Public, Internal, Confidential, Restricted)
• Automated data classification suggestions
• Data lineage tracking (data flow visualization)
• Retention policy configuration
• Automated data deletion based on retention
• Consent management for PII
• Compliance reporting (GDPR, CCPA)
• Data access audit trails

🏗️ Technical Requirements

• Backend: GovernanceService
• Data classification engine
• Lineage graph storage and visualization
• Retention policy scheduler
• Compliance report generator
• Frontend: Governance dashboard

🏷️ Labels

`low-priority`, `backend`, `frontend`, `security`, `compliance`, `governance`

⏱️ Estimated Effort

5-6 weeks

[traycerai]

Plan

Observations

The codebase has a solid foundation for governance features with existing PII detection in DatasetMetadata, versioning models (DatasetVersion, TransformationLineage), and audit fields. The architecture follows a service layer pattern with BaseService for CRUD operations, FastAPI backend with Beanie ODM (MongoDB), and Next.js frontend with TypeScript. Background task processing uses FastAPI's BackgroundTasks and asyncio.Task. The frontend uses React Flow (@xyflow/react) which is perfect for lineage visualization, and shadcn/ui components for consistent UI patterns.

Approach

The implementation will extend existing models and services rather than creating entirely new systems. Data classification will be added to DatasetMetadata with an AI-powered classification engine. Lineage visualization will leverage existing TransformationLineage and DatasetVersion models. Retention policies will use a new RetentionPolicy model with a scheduler service for automated deletion. Consent management will extend PII tracking with user consent records. Compliance reporting will aggregate data from classification, lineage, and audit trails. The governance dashboard will be a new frontend page using existing component patterns.

Implementation Steps

1. Backend Models - Data Governance Core

1.1 Create Data Classification Model

Create file:apps/backend/app/models/governance.py with the following models:

• DataClassification enum: PUBLIC, INTERNAL, CONFIDENTIAL, RESTRICTED
• ClassificationRule model: Pattern-based rules for auto-classification
• DataClassificationRecord document:
  • dataset_id: Indexed string
  • user_id: Indexed string
  • classification: DataClassification enum
  • classification_method: AUTO or MANUAL
  • confidence_score: Float (0-1) for auto-classification
  • classified_by: User ID or "system"
  • classified_at: DateTime
  • justification: String explaining classification
  • sensitive_fields: List of field names with sensitivity
  • review_required: Boolean flag
  • reviewed_by: Optional user ID
  • reviewed_at: Optional datetime

1.2 Create Retention Policy Model

Add to file:apps/backend/app/models/governance.py:

• RetentionPolicy document:

  • policy_id: Indexed unique string
  • user_id: Indexed string (owner)
  • name: Policy name
  • description: Policy description
  • classification_level: Optional DataClassification (apply to specific classification)
  • retention_days: Integer (days to retain)
  • auto_delete: Boolean (enable automated deletion)
  • applies_to_datasets: List of dataset IDs (empty = all)
  • exclude_datasets: List of dataset IDs to exclude
  • created_at: DateTime
  • updated_at: DateTime
  • is_active: Boolean

• RetentionSchedule document:

  • schedule_id: Indexed unique string
  • dataset_id: Indexed string
  • policy_id: String reference
  • retention_expires_at: Indexed datetime
  • deletion_scheduled: Boolean
  • deletion_executed: Boolean
  • deleted_at: Optional datetime
  • created_at: DateTime

1.3 Create Consent Management Model

Add to file:apps/backend/app/models/governance.py:

• ConsentRecord document:
  • consent_id: Indexed unique string
  • dataset_id: Indexed string
  • user_id: Indexed string (data owner)
  • data_subject_id: Optional string (person whose data it is)
  • consent_type: Enum (PROCESSING, STORAGE, SHARING, ANALYTICS)
  • consent_given: Boolean
  • consent_date: DateTime
  • expiry_date: Optional datetime
  • withdrawn: Boolean
  • withdrawn_at: Optional datetime
  • purpose: String describing data use purpose
  • legal_basis: String (GDPR Article 6 basis)
  • metadata: Dict for additional context

1.4 Create Audit Trail Model

Add to file:apps/backend/app/models/governance.py:

• DataAccessAudit document:
  • audit_id: Indexed unique string
  • dataset_id: Indexed string
  • user_id: Indexed string
  • access_type: Enum (READ, WRITE, DELETE, EXPORT, TRANSFORM)
  • accessed_at: Indexed datetime
  • ip_address: Optional string
  • user_agent: Optional string
  • action_details: Dict with specific action info
  • classification_at_access: Optional DataClassification
  • compliance_flags: List of strings (GDPR, CCPA, etc.)

1.5 Update DatasetMetadata Model

Extend file:apps/backend/app/models/dataset.py:

• Add classification: Optional DataClassification field
• Add classification_record_id: Optional string reference
• Add retention_policy_id: Optional string reference
• Add retention_expires_at: Optional datetime
• Add consent_required: Boolean (default False)
• Add consent_records: List of consent IDs

1.6 Register Models in Beanie

Update file:apps/backend/app/main.py in the lifespan function to include new models in init_beanie:

• DataClassificationRecord
• RetentionPolicy
• RetentionSchedule
• ConsentRecord
• DataAccessAudit

2. Backend Services - Governance Logic

2.1 Create Classification Service

Create file:apps/backend/app/services/governance/classification_service.py:

ClassificationService class extending BaseService[DataClassificationRecord]:

• classify_dataset(dataset_id, user_id, manual_classification=None): Main classification method
• _auto_classify(dataset): AI-powered classification using OpenAI
  • Analyze schema, PII presence, field names, data sensitivity
  • Return classification with confidence score
• _apply_classification_rules(dataset): Pattern-based rule matching
• suggest_classification(dataset_id): Return suggestions without saving
• update_classification(dataset_id, classification, user_id): Manual override
• get_classification_history(dataset_id): Audit trail of classification changes
• bulk_classify(user_id, filters): Classify multiple datasets

AI Classification Prompt Strategy:

• Analyze PII report, field names, data types, statistics
• Consider industry standards (NIST, ISO 27001)
• Return JSON with classification, confidence, reasoning, sensitive_fields

2.2 Create Lineage Service

Create file:apps/backend/app/services/governance/lineage_service.py:

LineageService class:

• get_dataset_lineage(dataset_id, depth=5): Build lineage graph
  • Query DatasetVersion and TransformationLineage models
  • Build parent-child relationships
  • Include transformation details
• get_lineage_graph(dataset_id): Return graph structure for visualization
  • Nodes: datasets/versions with metadata
  • Edges: transformations with details
  • Format compatible with React Flow
• get_downstream_datasets(dataset_id): Find all derived datasets
• get_upstream_datasets(dataset_id): Find all source datasets
• get_impact_analysis(dataset_id): Analyze impact of changes
• export_lineage(dataset_id, format='json'): Export lineage data

2.3 Create Retention Service

Create file:apps/backend/app/services/governance/retention_service.py:

RetentionService class extending BaseService[RetentionPolicy]:

• create_policy(user_id, policy_data): Create retention policy
• apply_policy_to_dataset(dataset_id, policy_id): Apply policy
• calculate_expiry_date(dataset, policy): Calculate expiration
• schedule_deletion(dataset_id, policy_id): Create retention schedule
• get_expiring_datasets(days_ahead=30): Find datasets expiring soon
• execute_scheduled_deletions(): Background task to delete expired data
• extend_retention(dataset_id, additional_days): Extend retention period
• get_retention_report(user_id): Report on retention status

RetentionScheduler class:

• start_scheduler(): Initialize background scheduler
• _check_expirations(): Periodic check (run every 6 hours)
• _execute_deletion(schedule_id): Safely delete dataset and related data
• _notify_before_deletion(dataset_id, days_until): Send notifications

2.4 Create Consent Service

Create file:apps/backend/app/services/governance/consent_service.py:

ConsentService class extending BaseService[ConsentRecord]:

• record_consent(dataset_id, user_id, consent_data): Record consent
• withdraw_consent(consent_id, user_id): Withdraw consent
• check_consent(dataset_id, purpose): Verify consent exists
• get_active_consents(dataset_id): Get all active consents
• get_expired_consents(user_id): Find expired consents
• get_consent_report(user_id): Consent management report

2.5 Create Audit Service

Create file:apps/backend/app/services/governance/audit_service.py:

AuditService class extending BaseService[DataAccessAudit]:

• log_access(dataset_id, user_id, access_type, details): Log data access
• get_access_history(dataset_id, start_date, end_date): Query audit trail
• get_user_activity(user_id, start_date, end_date): User activity report
• get_compliance_report(user_id, regulation='GDPR'): Generate compliance report
• export_audit_trail(filters, format='csv'): Export audit data

2.6 Create Compliance Reporting Service

Create file:apps/backend/app/services/governance/compliance_service.py:

ComplianceService class:

• generate_gdpr_report(user_id): GDPR compliance report
  • Data inventory with classifications
  • PII processing activities
  • Consent records
  • Data retention status
  • Access audit trail
• generate_ccpa_report(user_id): CCPA compliance report
• get_data_inventory(user_id): Complete data inventory
• get_pii_processing_activities(user_id): PII processing summary
• check_compliance_status(user_id): Overall compliance health check

2.7 Create Governance Middleware

Create file:apps/backend/app/middleware/governance.py:

GovernanceMiddleware:

• Intercept dataset access requests
• Log to DataAccessAudit
• Check consent requirements
• Enforce classification-based access controls
• Add to FastAPI middleware stack in file:apps/backend/app/main.py

3. Backend API Routes - Governance Endpoints

3.1 Create Governance Router

Create file:apps/backend/app/api/routes/governance.py:

Classification Endpoints:

• POST /api/v1/governance/classify/{dataset_id}: Classify dataset
• GET /api/v1/governance/classify/{dataset_id}: Get classification
• PUT /api/v1/governance/classify/{dataset_id}: Update classification
• POST /api/v1/governance/classify/suggest/{dataset_id}: Get suggestions
• POST /api/v1/governance/classify/bulk: Bulk classify datasets

Lineage Endpoints:

• GET /api/v1/governance/lineage/{dataset_id}: Get lineage graph
• GET /api/v1/governance/lineage/{dataset_id}/upstream: Get upstream datasets
• GET /api/v1/governance/lineage/{dataset_id}/downstream: Get downstream datasets
• GET /api/v1/governance/lineage/{dataset_id}/impact: Impact analysis
• GET /api/v1/governance/lineage/{dataset_id}/export: Export lineage

Retention Endpoints:

• POST /api/v1/governance/retention/policies: Create policy
• GET /api/v1/governance/retention/policies: List policies
• PUT /api/v1/governance/retention/policies/{policy_id}: Update policy
• DELETE /api/v1/governance/retention/policies/{policy_id}: Delete policy
• POST /api/v1/governance/retention/apply/{dataset_id}: Apply policy to dataset
• GET /api/v1/governance/retention/expiring: Get expiring datasets
• POST /api/v1/governance/retention/extend/{dataset_id}: Extend retention

Consent Endpoints:

• POST /api/v1/governance/consent: Record consent
• GET /api/v1/governance/consent/{dataset_id}: Get consents for dataset
• DELETE /api/v1/governance/consent/{consent_id}: Withdraw consent
• GET /api/v1/governance/consent/expired: Get expired consents

Audit Endpoints:

• GET /api/v1/governance/audit/{dataset_id}: Get access history
• GET /api/v1/governance/audit/user/{user_id}: Get user activity
• POST /api/v1/governance/audit/export: Export audit trail

Compliance Endpoints:

• GET /api/v1/governance/compliance/gdpr: GDPR report
• GET /api/v1/governance/compliance/ccpa: CCPA report
• GET /api/v1/governance/compliance/inventory: Data inventory
• GET /api/v1/governance/compliance/status: Compliance status

3.2 Register Router

Update file:apps/backend/app/main.py to include governance router:

from app.api.routes import governance
app.include_router(
    governance.router,
    prefix=f"{settings.API_V1_STR}/governance",
    tags=["governance"]
)

4. Backend Schemas - Request/Response Models

Create file:apps/backend/app/schemas/governance.py:

Request Schemas:

• ClassifyDatasetRequest: manual_classification, justification
• CreateRetentionPolicyRequest: name, description, retention_days, classification_level, auto_delete
• RecordConsentRequest: consent_type, purpose, legal_basis, expiry_date
• AuditQueryRequest: start_date, end_date, access_types, user_ids

Response Schemas:

• ClassificationResponse: classification, confidence, sensitive_fields, classified_at
• LineageGraphResponse: nodes, edges, metadata
• RetentionPolicyResponse: policy details with applied datasets count
• ConsentResponse: consent details with status
• AuditRecordResponse: audit entry with user and dataset info
• ComplianceReportResponse: comprehensive compliance data

5. Frontend Components - Governance UI

5.1 Create Governance Dashboard Page

Create file:apps/frontend/app/governance/page.tsx:

Main Dashboard Layout:

• Overview cards: Total datasets, Classified datasets, Active policies, Consent records
• Quick actions: Classify datasets, Create policy, View lineage, Generate report
• Recent activity: Latest classifications, policy applications, consent changes
• Compliance status indicators: GDPR, CCPA compliance scores
• Navigation to sub-pages: Classification, Lineage, Retention, Consent, Audit

5.2 Create Classification Management Page

Create file:apps/frontend/app/governance/classification/page.tsx:

Features:

• Dataset list with classification status
• Bulk classification action
• Filter by classification level
• Manual classification dialog
• Classification suggestions display
• Confidence score visualization
• Sensitive fields highlighting

5.3 Create Lineage Visualization Page

Create file:apps/frontend/app/governance/lineage/[datasetId]/page.tsx:

Features:

• React Flow graph visualization using @xyflow/react
• Interactive nodes: datasets with metadata tooltips
• Edges: transformations with details on hover
• Controls: Zoom, pan, fit view, layout options
• Filters: Show/hide transformation types, depth control
• Export: PNG, SVG, JSON formats
• Impact analysis panel: Downstream datasets affected

Node Types:

• Dataset node: Name, classification badge, row/column count
• Transformation edge: Type, date, data loss percentage

5.4 Create Retention Policy Management Page

Create file:apps/frontend/app/governance/retention/page.tsx:

Features:

• Policy list with active/inactive status
• Create/edit policy form
• Apply policy to datasets (multi-select)
• Expiring datasets table with countdown
• Extend retention action
• Policy impact preview
• Deletion schedule calendar view

5.5 Create Consent Management Page

Create file:apps/frontend/app/governance/consent/page.tsx:

Features:

• Consent records table
• Record new consent form
• Withdraw consent action
• Expired consents alert
• Consent status by dataset
• Legal basis documentation

5.6 Create Audit Trail Page

Create file:apps/frontend/app/governance/audit/page.tsx:

Features:

• Audit log table with filters
• Date range picker
• Access type filter
• User filter
• Dataset filter
• Export to CSV
• Activity timeline visualization
• User activity heatmap

5.7 Create Compliance Reports Page

Create file:apps/frontend/app/governance/compliance/page.tsx:

Features:

• Report type selector: GDPR, CCPA, Custom
• Generate report button
• Report sections: Data inventory, PII processing, Consent status, Retention compliance, Access audit
• Export report: PDF, CSV, JSON
• Compliance score dashboard
• Recommendations for improvement

5.8 Create Shared Components

Create file:apps/frontend/components/governance/:

• ClassificationBadge.tsx: Color-coded classification badge
• LineageGraph.tsx: Reusable React Flow lineage component
• RetentionPolicyCard.tsx: Policy display card
• ConsentStatusIndicator.tsx: Consent status icon
• AuditLogTable.tsx: Reusable audit table
• ComplianceScoreCard.tsx: Compliance score visualization
• DataInventoryTable.tsx: Data inventory display

6. Frontend Services - API Integration

Create file:apps/frontend/lib/services/governance.ts:

API Client Functions:

• classifyDataset(datasetId, classification?): Classify dataset
• getClassification(datasetId): Get classification
• getLineageGraph(datasetId): Fetch lineage data
• createRetentionPolicy(policyData): Create policy
• applyRetentionPolicy(datasetId, policyId): Apply policy
• recordConsent(consentData): Record consent
• getAuditTrail(filters): Fetch audit logs
• generateComplianceReport(type): Generate report
• exportLineage(datasetId, format): Export lineage

7. Background Tasks - Automated Governance

7.1 Create Retention Scheduler

Create file:apps/backend/app/services/governance/scheduler.py:

RetentionScheduler:

• Use asyncio background task
• Start in file:apps/backend/app/main.py lifespan
• Run every 6 hours
• Check RetentionSchedule for expired datasets
• Execute deletions with proper cleanup
• Send notifications before deletion (7 days, 1 day)
• Log all actions to audit trail

7.2 Create Classification Suggestion Task

Add to file:apps/backend/app/services/governance/scheduler.py:

ClassificationScheduler:

• Run daily
• Find unclassified datasets
• Generate classification suggestions
• Store suggestions for user review
• Notify users of pending classifications

8. Database Migrations and Indexes

8.1 Add Indexes

Ensure indexes in model Settings classes:

• DataClassificationRecord: dataset_id, user_id, classified_at
• RetentionPolicy: user_id, is_active
• RetentionSchedule: dataset_id, retention_expires_at, deletion_scheduled
• ConsentRecord: dataset_id, user_id, consent_type, expiry_date
• DataAccessAudit: dataset_id, user_id, accessed_at, access_type

8.2 Update Existing Data

Create migration script file:apps/backend/scripts/migrate_governance.py:

• Add default classification to existing datasets (INTERNAL)
• Create default retention policy (365 days)
• Backfill audit records from existing access logs

9. Testing Strategy

9.1 Backend Unit Tests

Create test files:

• file:apps/backend/tests/test_services/test_classification_service.py
• file:apps/backend/tests/test_services/test_lineage_service.py
• file:apps/backend/tests/test_services/test_retention_service.py
• file:apps/backend/tests/test_services/test_consent_service.py
• file:apps/backend/tests/test_services/test_audit_service.py
• file:apps/backend/tests/test_api/test_governance.py

9.2 Frontend Component Tests

Create test files:

• file:apps/frontend/__tests__/governance/ClassificationPage.test.tsx
• file:apps/frontend/__tests__/governance/LineageGraph.test.tsx
• file:apps/frontend/__tests__/governance/RetentionPage.test.tsx

9.3 Integration Tests

• End-to-end classification workflow
• Lineage graph generation and visualization
• Retention policy application and deletion
• Consent recording and withdrawal
• Compliance report generation

10. Documentation

10.1 API Documentation

Update file:apps/backend/app/services/api_documentation.py:

• Add governance endpoints to OpenAPI spec
• Include request/response examples
• Document authentication requirements
• Add compliance-related notes

10.2 User Documentation

Create file:GOVERNANCE_GUIDE.md:

• Data classification guidelines
• Retention policy best practices
• Consent management procedures
• Compliance reporting instructions
• Lineage visualization guide

Architecture Diagram

graph TD
    subgraph Frontend
        A[Governance Dashboard] --> B[Classification UI]
        A --> C[Lineage Visualization]
        A --> D[Retention Management]
        A --> E[Consent Management]
        A --> F[Audit Trail]
        A --> G[Compliance Reports]
    end
    
    subgraph Backend Services
        H[ClassificationService] --> M[OpenAI API]
        I[LineageService]
        J[RetentionService]
        K[ConsentService]
        L[AuditService]
        N[ComplianceService]
    end
    
    subgraph Background Tasks
        O[RetentionScheduler]
        P[ClassificationScheduler]
    end
    
    subgraph Database
        Q[(DataClassificationRecord)]
        R[(RetentionPolicy)]
        S[(ConsentRecord)]
        T[(DataAccessAudit)]
        U[(DatasetMetadata)]
        V[(DatasetVersion)]
        W[(TransformationLineage)]
    end
    
    B --> H
    C --> I
    D --> J
    E --> K
    F --> L
    G --> N
    
    H --> Q
    H --> U
    I --> V
    I --> W
    J --> R
    J --> U
    K --> S
    L --> T
    N --> Q
    N --> S
    N --> T
    
    O --> J
    P --> H

Loading

Data Flow - Classification Process

sequenceDiagram
    participant User
    participant UI as Classification UI
    participant API as Governance API
    participant CS as ClassificationService
    participant AI as OpenAI
    participant DB as MongoDB
    
    User->>UI: Request classification
    UI->>API: POST /governance/classify/{dataset_id}
    API->>CS: classify_dataset(dataset_id)
    CS->>DB: Fetch dataset metadata
    DB-->>CS: Dataset with PII report
    CS->>AI: Analyze dataset for classification
    AI-->>CS: Classification + confidence
    CS->>DB: Save DataClassificationRecord
    CS->>DB: Update DatasetMetadata
    DB-->>CS: Success
    CS-->>API: Classification result
    API-->>UI: Classification response
    UI-->>User: Display classification badge

Loading

Compliance Report Structure

Section	Data Source	Content
Data Inventory	DatasetMetadata, DataClassificationRecord	All datasets with classifications
PII Processing	PIIReport, DataClassificationRecord	Datasets containing PII
Consent Records	ConsentRecord	Active and expired consents
Retention Status	RetentionPolicy, RetentionSchedule	Retention compliance
Access Audit	DataAccessAudit	Data access history
Recommendations	ComplianceService	Compliance improvements

Implementation Timeline

Week	Focus Area	Deliverables
1	Backend Models & Services	Classification, Retention, Consent models and services
2	Lineage & Audit Services	Lineage graph generation, audit logging
3	API Routes & Schemas	All governance endpoints
4	Frontend Dashboard & Classification	Main dashboard, classification UI
5	Frontend Lineage & Retention	Lineage visualization, retention management
6	Frontend Consent & Compliance	Consent management, compliance reports, testing

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase has a solid foundation for governance features with existing PII detection in `DatasetMetadata`, versioning models (`DatasetVersion`, `TransformationLineage`), and audit fields. The architecture follows a service layer pattern with `BaseService` for CRUD operations, FastAPI backend with Beanie ODM (MongoDB), and Next.js frontend with TypeScript. Background task processing uses FastAPI's `BackgroundTasks` and `asyncio.Task`. The frontend uses React Flow (@xyflow/react) which is perfect for lineage visualization, and shadcn/ui components for consistent UI patterns.

## Approach

The implementation will extend existing models and services rather than creating entirely new systems. Data classification will be added to `DatasetMetadata` with an AI-powered classification engine. Lineage visualization will leverage existing `TransformationLineage` and `DatasetVersion` models. Retention policies will use a new `RetentionPolicy` model with a scheduler service for automated deletion. Consent management will extend PII tracking with user consent records. Compliance reporting will aggregate data from classification, lineage, and audit trails. The governance dashboard will be a new frontend page using existing component patterns.

## Implementation Steps

### 1. Backend Models - Data Governance Core

#### 1.1 Create Data Classification Model
Create `file:apps/backend/app/models/governance.py` with the following models:

- **`DataClassification`** enum: PUBLIC, INTERNAL, CONFIDENTIAL, RESTRICTED
- **`ClassificationRule`** model: Pattern-based rules for auto-classification
- **`DataClassificationRecord`** document:
  - `dataset_id`: Indexed string
  - `user_id`: Indexed string
  - `classification`: DataClassification enum
  - `classification_method`: AUTO or MANUAL
  - `confidence_score`: Float (0-1) for auto-classification
  - `classified_by`: User ID or "system"
  - `classified_at`: DateTime
  - `justification`: String explaining classification
  - `sensitive_fields`: List of field names with sensitivity
  - `review_required`: Boolean flag
  - `reviewed_by`: Optional user ID
  - `reviewed_at`: Optional datetime

#### 1.2 Create Retention Policy Model
Add to `file:apps/backend/app/models/governance.py`:

- **`RetentionPolicy`** document:
  - `policy_id`: Indexed unique string
  - `user_id`: Indexed string (owner)
  - `name`: Policy name
  - `description`: Policy description
  - `classification_level`: Optional DataClassification (apply to specific classification)
  - `retention_days`: Integer (days to retain)
  - `auto_delete`: Boolean (enable automated deletion)
  - `applies_to_datasets`: List of dataset IDs (empty = all)
  - `exclude_datasets`: List of dataset IDs to exclude
  - `created_at`: DateTime
  - `updated_at`: DateTime
  - `is_active`: Boolean

- **`RetentionSchedule`** document:
  - `schedule_id`: Indexed unique string
  - `dataset_id`: Indexed string
  - `policy_id`: String reference
  - `retention_expires_at`: Indexed datetime
  - `deletion_scheduled`: Boolean
  - `deletion_executed`: Boolean
  - `deleted_at`: Optional datetime
  - `created_at`: DateTime

#### 1.3 Create Consent Management Model
Add to `file:apps/backend/app/models/governance.py`:

- **`ConsentRecord`** document:
  - `consent_id`: Indexed unique string
  - `dataset_id`: Indexed string
  - `user_id`: Indexed string (data owner)
  - `data_subject_id`: Optional string (person whose data it is)
  - `consent_type`: Enum (PROCESSING, STORAGE, SHARING, ANALYTICS)
  - `consent_given`: Boolean
  - `consent_date`: DateTime
  - `expiry_date`: Optional datetime
  - `withdrawn`: Boolean
  - `withdrawn_at`: Optional datetime
  - `purpose`: String describing data use purpose
  - `legal_basis`: String (GDPR Article 6 basis)
  - `metadata`: Dict for additional context

#### 1.4 Create Audit Trail Model
Add to `file:apps/backend/app/models/governance.py`:

- **`DataAccessAudit`** document:
  - `audit_id`: Indexed unique string
  - `dataset_id`: Indexed string
  - `user_id`: Indexed string
  - `access_type`: Enum (READ, WRITE, DELETE, EXPORT, TRANSFORM)
  - `accessed_at`: Indexed datetime
  - `ip_address`: Optional string
  - `user_agent`: Optional string
  - `action_details`: Dict with specific action info
  - `classification_at_access`: Optional DataClassification
  - `compliance_flags`: List of strings (GDPR, CCPA, etc.)

#### 1.5 Update DatasetMetadata Model
Extend `file:apps/backend/app/models/dataset.py`:

- Add `classification`: Optional DataClassification field
- Add `classification_record_id`: Optional string reference
- Add `retention_policy_id`: Optional string reference
- Add `retention_expires_at`: Optional datetime
- Add `consent_required`: Boolean (default False)
- Add `consent_records`: List of consent IDs

#### 1.6 Register Models in Beanie
Update `file:apps/backend/app/main.py` in the `lifespan` function to include new models in `init_beanie`:
- `DataClassificationRecord`
- `RetentionPolicy`
- `RetentionSchedule`
- `ConsentRecord`
- `DataAccessAudit`

### 2. Backend Services - Governance Logic

#### 2.1 Create Classification Service
Create `file:apps/backend/app/services/governance/classification_service.py`:

**ClassificationService** class extending `BaseService[DataClassificationRecord]`:
- `classify_dataset(dataset_id, user_id, manual_classification=None)`: Main classification method
- `_auto_classify(dataset)`: AI-powered classification using OpenAI
  - Analyze schema, PII presence, field names, data sensitivity
  - Return classification with confidence score
- `_apply_classification_rules(dataset)`: Pattern-based rule matching
- `suggest_classification(dataset_id)`: Return suggestions without saving
- `update_classification(dataset_id, classification, user_id)`: Manual override
- `get_classification_history(dataset_id)`: Audit trail of classification changes
- `bulk_classify(user_id, filters)`: Classify multiple datasets

**AI Classification Prompt Strategy**:
- Analyze PII report, field names, data types, statistics
- Consider industry standards (NIST, ISO 27001)
- Return JSON with classification, confidence, reasoning, sensitive_fields

#### 2.2 Create Lineage Service
Create `file:apps/backend/app/services/governance/lineage_service.py`:

**LineageService** class:
- `get_dataset_lineage(dataset_id, depth=5)`: Build lineage graph
  - Query `DatasetVersion` and `TransformationLineage` models
  - Build parent-child relationships
  - Include transformation details
- `get_lineage_graph(dataset_id)`: Return graph structure for visualization
  - Nodes: datasets/versions with metadata
  - Edges: transformations with details
  - Format compatible with React Flow
- `get_downstream_datasets(dataset_id)`: Find all derived datasets
- `get_upstream_datasets(dataset_id)`: Find all source datasets
- `get_impact_analysis(dataset_id)`: Analyze impact of changes
- `export_lineage(dataset_id, format='json')`: Export lineage data

#### 2.3 Create Retention Service
Create `file:apps/backend/app/services/governance/retention_service.py`:

**RetentionService** class extending `BaseService[RetentionPolicy]`:
- `create_policy(user_id, policy_data)`: Create retention policy
- `apply_policy_to_dataset(dataset_id, policy_id)`: Apply policy
- `calculate_expiry_date(dataset, policy)`: Calculate expiration
- `schedule_deletion(dataset_id, policy_id)`: Create retention schedule
- `get_expiring_datasets(days_ahead=30)`: Find datasets expiring soon
- `execute_scheduled_deletions()`: Background task to delete expired data
- `extend_retention(dataset_id, additional_days)`: Extend retention period
- `get_retention_report(user_id)`: Report on retention status

**RetentionScheduler** class:
- `start_scheduler()`: Initialize background scheduler
- `_check_expirations()`: Periodic check (run every 6 hours)
- `_execute_deletion(schedule_id)`: Safely delete dataset and related data
- `_notify_before_deletion(dataset_id, days_until)`: Send notifications

#### 2.4 Create Consent Service
Create `file:apps/backend/app/services/governance/consent_service.py`:

**ConsentService** class extending `BaseService[ConsentRecord]`:
- `record_consent(dataset_id, user_id, consent_data)`: Record consent
- `withdraw_consent(consent_id, user_id)`: Withdraw consent
- `check_consent(dataset_id, purpose)`: Verify consent exists
- `get_active_consents(dataset_id)`: Get all active consents
- `get_expired_consents(user_id)`: Find expired consents
- `get_consent_report(user_id)`: Consent management report

#### 2.5 Create Audit Service
Create `file:apps/backend/app/services/governance/audit_service.py`:

**AuditService** class extending `BaseService[DataAccessAudit]`:
- `log_access(dataset_id, user_id, access_type, details)`: Log data access
- `get_access_history(dataset_id, start_date, end_date)`: Query audit trail
- `get_user_activity(user_id, start_date, end_date)`: User activity report
- `get_compliance_report(user_id, regulation='GDPR')`: Generate compliance report
- `export_audit_trail(filters, format='csv')`: Export audit data

#### 2.6 Create Compliance Reporting Service
Create `file:apps/backend/app/services/governance/compliance_service.py`:

**ComplianceService** class:
- `generate_gdpr_report(user_id)`: GDPR compliance report
  - Data inventory with classifications
  - PII processing activities
  - Consent records
  - Data retention status
  - Access audit trail
- `generate_ccpa_report(user_id)`: CCPA compliance report
- `get_data_inventory(user_id)`: Complete data inventory
- `get_pii_processing_activities(user_id)`: PII processing summary
- `check_compliance_status(user_id)`: Overall compliance health check

#### 2.7 Create Governance Middleware
Create `file:apps/backend/app/middleware/governance.py`:

**GovernanceMiddleware**:
- Intercept dataset access requests
- Log to `DataAccessAudit`
- Check consent requirements
- Enforce classification-based access controls
- Add to FastAPI middleware stack in `file:apps/backend/app/main.py`

### 3. Backend API Routes - Governance Endpoints

#### 3.1 Create Governance Router
Create `file:apps/backend/app/api/routes/governance.py`:

**Classification Endpoints**:
- `POST /api/v1/governance/classify/{dataset_id}`: Classify dataset
- `GET /api/v1/governance/classify/{dataset_id}`: Get classification
- `PUT /api/v1/governance/classify/{dataset_id}`: Update classification
- `POST /api/v1/governance/classify/suggest/{dataset_id}`: Get suggestions
- `POST /api/v1/governance/classify/bulk`: Bulk classify datasets

**Lineage Endpoints**:
- `GET /api/v1/governance/lineage/{dataset_id}`: Get lineage graph
- `GET /api/v1/governance/lineage/{dataset_id}/upstream`: Get upstream datasets
- `GET /api/v1/governance/lineage/{dataset_id}/downstream`: Get downstream datasets
- `GET /api/v1/governance/lineage/{dataset_id}/impact`: Impact analysis
- `GET /api/v1/governance/lineage/{dataset_id}/export`: Export lineage

**Retention Endpoints**:
- `POST /api/v1/governance/retention/policies`: Create policy
- `GET /api/v1/governance/retention/policies`: List policies
- `PUT /api/v1/governance/retention/policies/{policy_id}`: Update policy
- `DELETE /api/v1/governance/retention/policies/{policy_id}`: Delete policy
- `POST /api/v1/governance/retention/apply/{dataset_id}`: Apply policy to dataset
- `GET /api/v1/governance/retention/expiring`: Get expiring datasets
- `POST /api/v1/governance/retention/extend/{dataset_id}`: Extend retention

**Consent Endpoints**:
- `POST /api/v1/governance/consent`: Record consent
- `GET /api/v1/governance/consent/{dataset_id}`: Get consents for dataset
- `DELETE /api/v1/governance/consent/{consent_id}`: Withdraw consent
- `GET /api/v1/governance/consent/expired`: Get expired consents

**Audit Endpoints**:
- `GET /api/v1/governance/audit/{dataset_id}`: Get access history
- `GET /api/v1/governance/audit/user/{user_id}`: Get user activity
- `POST /api/v1/governance/audit/export`: Export audit trail

**Compliance Endpoints**:
- `GET /api/v1/governance/compliance/gdpr`: GDPR report
- `GET /api/v1/governance/compliance/ccpa`: CCPA report
- `GET /api/v1/governance/compliance/inventory`: Data inventory
- `GET /api/v1/governance/compliance/status`: Compliance status

#### 3.2 Register Router
Update `file:apps/backend/app/main.py` to include governance router:
```python
from app.api.routes import governance
app.include_router(
    governance.router,
    prefix=f"{settings.API_V1_STR}/governance",
    tags=["governance"]
)
```

### 4. Backend Schemas - Request/Response Models

Create `file:apps/backend/app/schemas/governance.py`:

**Request Schemas**:
- `ClassifyDatasetRequest`: manual_classification, justification
- `CreateRetentionPolicyRequest`: name, description, retention_days, classification_level, auto_delete
- `RecordConsentRequest`: consent_type, purpose, legal_basis, expiry_date
- `AuditQueryRequest`: start_date, end_date, access_types, user_ids

**Response Schemas**:
- `ClassificationResponse`: classification, confidence, sensitive_fields, classified_at
- `LineageGraphResponse`: nodes, edges, metadata
- `RetentionPolicyResponse`: policy details with applied datasets count
- `ConsentResponse`: consent details with status
- `AuditRecordResponse`: audit entry with user and dataset info
- `ComplianceReportResponse`: comprehensive compliance data

### 5. Frontend Components - Governance UI

#### 5.1 Create Governance Dashboard Page
Create `file:apps/frontend/app/governance/page.tsx`:

**Main Dashboard Layout**:
- Overview cards: Total datasets, Classified datasets, Active policies, Consent records
- Quick actions: Classify datasets, Create policy, View lineage, Generate report
- Recent activity: Latest classifications, policy applications, consent changes
- Compliance status indicators: GDPR, CCPA compliance scores
- Navigation to sub-pages: Classification, Lineage, Retention, Consent, Audit

#### 5.2 Create Classification Management Page
Create `file:apps/frontend/app/governance/classification/page.tsx`:

**Features**:
- Dataset list with classification status
- Bulk classification action
- Filter by classification level
- Manual classification dialog
- Classification suggestions display
- Confidence score visualization
- Sensitive fields highlighting

#### 5.3 Create Lineage Visualization Page
Create `file:apps/frontend/app/governance/lineage/[datasetId]/page.tsx`:

**Features**:
- React Flow graph visualization using `@xyflow/react`
- Interactive nodes: datasets with metadata tooltips
- Edges: transformations with details on hover
- Controls: Zoom, pan, fit view, layout options
- Filters: Show/hide transformation types, depth control
- Export: PNG, SVG, JSON formats
- Impact analysis panel: Downstream datasets affected

**Node Types**:
- Dataset node: Name, classification badge, row/column count
- Transformation edge: Type, date, data loss percentage

#### 5.4 Create Retention Policy Management Page
Create `file:apps/frontend/app/governance/retention/page.tsx`:

**Features**:
- Policy list with active/inactive status
- Create/edit policy form
- Apply policy to datasets (multi-select)
- Expiring datasets table with countdown
- Extend retention action
- Policy impact preview
- Deletion schedule calendar view

#### 5.5 Create Consent Management Page
Create `file:apps/frontend/app/governance/consent/page.tsx`:

**Features**:
- Consent records table
- Record new consent form
- Withdraw consent action
- Expired consents alert
- Consent status by dataset
- Legal basis documentation

#### 5.6 Create Audit Trail Page
Create `file:apps/frontend/app/governance/audit/page.tsx`:

**Features**:
- Audit log table with filters
- Date range picker
- Access type filter
- User filter
- Dataset filter
- Export to CSV
- Activity timeline visualization
- User activity heatmap

#### 5.7 Create Compliance Reports Page
Create `file:apps/frontend/app/governance/compliance/page.tsx`:

**Features**:
- Report type selector: GDPR, CCPA, Custom
- Generate report button
- Report sections: Data inventory, PII processing, Consent status, Retention compliance, Access audit
- Export report: PDF, CSV, JSON
- Compliance score dashboard
- Recommendations for improvement

#### 5.8 Create Shared Components
Create `file:apps/frontend/components/governance/`:

- `ClassificationBadge.tsx`: Color-coded classification badge
- `LineageGraph.tsx`: Reusable React Flow lineage component
- `RetentionPolicyCard.tsx`: Policy display card
- `ConsentStatusIndicator.tsx`: Consent status icon
- `AuditLogTable.tsx`: Reusable audit table
- `ComplianceScoreCard.tsx`: Compliance score visualization
- `DataInventoryTable.tsx`: Data inventory display

### 6. Frontend Services - API Integration

Create `file:apps/frontend/lib/services/governance.ts`:

**API Client Functions**:
- `classifyDataset(datasetId, classification?)`: Classify dataset
- `getClassification(datasetId)`: Get classification
- `getLineageGraph(datasetId)`: Fetch lineage data
- `createRetentionPolicy(policyData)`: Create policy
- `applyRetentionPolicy(datasetId, policyId)`: Apply policy
- `recordConsent(consentData)`: Record consent
- `getAuditTrail(filters)`: Fetch audit logs
- `generateComplianceReport(type)`: Generate report
- `exportLineage(datasetId, format)`: Export lineage

### 7. Background Tasks - Automated Governance

#### 7.1 Create Retention Scheduler
Create `file:apps/backend/app/services/governance/scheduler.py`:

**RetentionScheduler**:
- Use `asyncio` background task
- Start in `file:apps/backend/app/main.py` lifespan
- Run every 6 hours
- Check `RetentionSchedule` for expired datasets
- Execute deletions with proper cleanup
- Send notifications before deletion (7 days, 1 day)
- Log all actions to audit trail

#### 7.2 Create Classification Suggestion Task
Add to `file:apps/backend/app/services/governance/scheduler.py`:

**ClassificationScheduler**:
- Run daily
- Find unclassified datasets
- Generate classification suggestions
- Store suggestions for user review
- Notify users of pending classifications

### 8. Database Migrations and Indexes

#### 8.1 Add Indexes
Ensure indexes in model `Settings` classes:
- `DataClassificationRecord`: dataset_id, user_id, classified_at
- `RetentionPolicy`: user_id, is_active
- `RetentionSchedule`: dataset_id, retention_expires_at, deletion_scheduled
- `ConsentRecord`: dataset_id, user_id, consent_type, expiry_date
- `DataAccessAudit`: dataset_id, user_id, accessed_at, access_type

#### 8.2 Update Existing Data
Create migration script `file:apps/backend/scripts/migrate_governance.py`:
- Add default classification to existing datasets (INTERNAL)
- Create default retention policy (365 days)
- Backfill audit records from existing access logs

### 9. Testing Strategy

#### 9.1 Backend Unit Tests
Create test files:
- `file:apps/backend/tests/test_services/test_classification_service.py`
- `file:apps/backend/tests/test_services/test_lineage_service.py`
- `file:apps/backend/tests/test_services/test_retention_service.py`
- `file:apps/backend/tests/test_services/test_consent_service.py`
- `file:apps/backend/tests/test_services/test_audit_service.py`
- `file:apps/backend/tests/test_api/test_governance.py`

#### 9.2 Frontend Component Tests
Create test files:
- `file:apps/frontend/__tests__/governance/ClassificationPage.test.tsx`
- `file:apps/frontend/__tests__/governance/LineageGraph.test.tsx`
- `file:apps/frontend/__tests__/governance/RetentionPage.test.tsx`

#### 9.3 Integration Tests
- End-to-end classification workflow
- Lineage graph generation and visualization
- Retention policy application and deletion
- Consent recording and withdrawal
- Compliance report generation

### 10. Documentation

#### 10.1 API Documentation
Update `file:apps/backend/app/services/api_documentation.py`:
- Add governance endpoints to OpenAPI spec
- Include request/response examples
- Document authentication requirements
- Add compliance-related notes

#### 10.2 User Documentation
Create `file:GOVERNANCE_GUIDE.md`:
- Data classification guidelines
- Retention policy best practices
- Consent management procedures
- Compliance reporting instructions
- Lineage visualization guide

## Architecture Diagram

```mermaid
graph TD
    subgraph Frontend
        A[Governance Dashboard] --> B[Classification UI]
        A --> C[Lineage Visualization]
        A --> D[Retention Management]
        A --> E[Consent Management]
        A --> F[Audit Trail]
        A --> G[Compliance Reports]
    end
    
    subgraph Backend Services
        H[ClassificationService] --> M[OpenAI API]
        I[LineageService]
        J[RetentionService]
        K[ConsentService]
        L[AuditService]
        N[ComplianceService]
    end
    
    subgraph Background Tasks
        O[RetentionScheduler]
        P[ClassificationScheduler]
    end
    
    subgraph Database
        Q[(DataClassificationRecord)]
        R[(RetentionPolicy)]
        S[(ConsentRecord)]
        T[(DataAccessAudit)]
        U[(DatasetMetadata)]
        V[(DatasetVersion)]
        W[(TransformationLineage)]
    end
    
    B --> H
    C --> I
    D --> J
    E --> K
    F --> L
    G --> N
    
    H --> Q
    H --> U
    I --> V
    I --> W
    J --> R
    J --> U
    K --> S
    L --> T
    N --> Q
    N --> S
    N --> T
    
    O --> J
    P --> H
```

## Data Flow - Classification Process

```mermaid
sequenceDiagram
    participant User
    participant UI as Classification UI
    participant API as Governance API
    participant CS as ClassificationService
    participant AI as OpenAI
    participant DB as MongoDB
    
    User->>UI: Request classification
    UI->>API: POST /governance/classify/{dataset_id}
    API->>CS: classify_dataset(dataset_id)
    CS->>DB: Fetch dataset metadata
    DB-->>CS: Dataset with PII report
    CS->>AI: Analyze dataset for classification
    AI-->>CS: Classification + confidence
    CS->>DB: Save DataClassificationRecord
    CS->>DB: Update DatasetMetadata
    DB-->>CS: Success
    CS-->>API: Classification result
    API-->>UI: Classification response
    UI-->>User: Display classification badge
```

## Compliance Report Structure

| Section | Data Source | Content |
|---------|-------------|---------|
| Data Inventory | DatasetMetadata, DataClassificationRecord | All datasets with classifications |
| PII Processing | PIIReport, DataClassificationRecord | Datasets containing PII |
| Consent Records | ConsentRecord | Active and expired consents |
| Retention Status | RetentionPolicy, RetentionSchedule | Retention compliance |
| Access Audit | DataAccessAudit | Data access history |
| Recommendations | ComplianceService | Compliance improvements |

## Implementation Timeline

| Week | Focus Area | Deliverables |
|------|-----------|--------------|
| 1 | Backend Models & Services | Classification, Retention, Consent models and services |
| 2 | Lineage & Audit Services | Lineage graph generation, audit logging |
| 3 | API Routes & Schemas | All governance endpoints |
| 4 | Frontend Dashboard & Classification | Main dashboard, classification UI |
| 5 | Frontend Lineage & Retention | Lineage visualization, retention management |
| 6 | Frontend Consent & Compliance | Consent management, compliance reports, testing |

---

Execution Information

Branch: main
Commit: 1963a18

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

_{You can configure auto-planning by selecting labels in the issue_enrichment configuration.}

🔗 Similar Issues

Related Issues

• [LOW] Implement Comprehensive Audit Logging System #92
• [LOW] Implement Role-Based Access Control (RBAC) for Team Collaboration #91
• [BLOCKER] Implement AI-Powered Data Issue Detection and Suggested Fixes #66
• [BLOCKER] Implement Comprehensive Model Evaluation Dashboard with Performance Metrics #79
• [BLOCKER] Implement Visual Data Cleaning Interface for Stage 3: Data Preparation #65

👤 Suggested Assignees

• frankbria

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!

[traycerai]

I have updated the Implementation Plan to reflect the recent changes in the ticket.

New Plan

Observations

The codebase has a solid foundation for governance features with existing PII detection in DatasetMetadata, versioning models (DatasetVersion, TransformationLineage), and audit fields. The architecture follows a service layer pattern with BaseService for CRUD operations, FastAPI backend with Beanie ODM (MongoDB), and Next.js frontend with TypeScript. Background task processing uses FastAPI's BackgroundTasks and asyncio.Task. The frontend uses React Flow (@xyflow/react) which is perfect for lineage visualization, and shadcn/ui components for consistent UI patterns.

Approach

The implementation will extend existing models and services rather than creating entirely new systems. Data classification will be added to DatasetMetadata with an AI-powered classification engine. Lineage visualization will leverage existing TransformationLineage and DatasetVersion models. Retention policies will use a new RetentionPolicy model with a scheduler service for automated deletion. Consent management will extend PII tracking with user consent records. Compliance reporting will aggregate data from classification, lineage, and audit trails. The governance dashboard will be a new frontend page using existing component patterns.

Implementation Steps

1. Backend Models - Data Governance Core

1.1 Create Data Classification Model

Create file:apps/backend/app/models/governance.py with the following models:

• DataClassification enum: PUBLIC, INTERNAL, CONFIDENTIAL, RESTRICTED
• ClassificationRule model: Pattern-based rules for auto-classification
• DataClassificationRecord document:
  • dataset_id: Indexed string
  • user_id: Indexed string
  • classification: DataClassification enum
  • classification_method: AUTO or MANUAL
  • confidence_score: Float (0-1) for auto-classification
  • classified_by: User ID or "system"
  • classified_at: DateTime
  • justification: String explaining classification
  • sensitive_fields: List of field names with sensitivity
  • review_required: Boolean flag
  • reviewed_by: Optional user ID
  • reviewed_at: Optional datetime

1.2 Create Retention Policy Model

Add to file:apps/backend/app/models/governance.py:

• RetentionPolicy document:

  • policy_id: Indexed unique string
  • user_id: Indexed string (owner)
  • name: Policy name
  • description: Policy description
  • classification_level: Optional DataClassification (apply to specific classification)
  • retention_days: Integer (days to retain)
  • auto_delete: Boolean (enable automated deletion)
  • applies_to_datasets: List of dataset IDs (empty = all)
  • exclude_datasets: List of dataset IDs to exclude
  • created_at: DateTime
  • updated_at: DateTime
  • is_active: Boolean

• RetentionSchedule document:

  • schedule_id: Indexed unique string
  • dataset_id: Indexed string
  • policy_id: String reference
  • retention_expires_at: Indexed datetime
  • deletion_scheduled: Boolean
  • deletion_executed: Boolean
  • deleted_at: Optional datetime
  • created_at: DateTime

1.3 Create Consent Management Model

Add to file:apps/backend/app/models/governance.py:

• ConsentRecord document:
  • consent_id: Indexed unique string
  • dataset_id: Indexed string
  • user_id: Indexed string (data owner)
  • data_subject_id: Optional string (person whose data it is)
  • consent_type: Enum (PROCESSING, STORAGE, SHARING, ANALYTICS)
  • consent_given: Boolean
  • consent_date: DateTime
  • expiry_date: Optional datetime
  • withdrawn: Boolean
  • withdrawn_at: Optional datetime
  • purpose: String describing data use purpose
  • legal_basis: String (GDPR Article 6 basis)
  • metadata: Dict for additional context

1.4 Create Audit Trail Model

Add to file:apps/backend/app/models/governance.py:

• DataAccessAudit document:
  • audit_id: Indexed unique string
  • dataset_id: Indexed string
  • user_id: Indexed string
  • access_type: Enum (READ, WRITE, DELETE, EXPORT, TRANSFORM)
  • accessed_at: Indexed datetime
  • ip_address: Optional string
  • user_agent: Optional string
  • action_details: Dict with specific action info
  • classification_at_access: Optional DataClassification
  • compliance_flags: List of strings (GDPR, CCPA, etc.)

1.5 Update DatasetMetadata Model

Extend file:apps/backend/app/models/dataset.py:

• Add classification: Optional DataClassification field
• Add classification_record_id: Optional string reference
• Add retention_policy_id: Optional string reference
• Add retention_expires_at: Optional datetime
• Add consent_required: Boolean (default False)
• Add consent_records: List of consent IDs

1.6 Register Models in Beanie

Update file:apps/backend/app/main.py in the lifespan function to include new models in init_beanie:

• DataClassificationRecord
• RetentionPolicy
• RetentionSchedule
• ConsentRecord
• DataAccessAudit

2. Backend Services - Governance Logic

2.1 Create Classification Service

Create file:apps/backend/app/services/governance/classification_service.py:

ClassificationService class extending BaseService[DataClassificationRecord]:

• classify_dataset(dataset_id, user_id, manual_classification=None): Main classification method
• _auto_classify(dataset): AI-powered classification using OpenAI
  • Analyze schema, PII presence, field names, data sensitivity
  • Return classification with confidence score
• _apply_classification_rules(dataset): Pattern-based rule matching
• suggest_classification(dataset_id): Return suggestions without saving
• update_classification(dataset_id, classification, user_id): Manual override
• get_classification_history(dataset_id): Audit trail of classification changes
• bulk_classify(user_id, filters): Classify multiple datasets

AI Classification Prompt Strategy:

• Analyze PII report, field names, data types, statistics
• Consider industry standards (NIST, ISO 27001)
• Return JSON with classification, confidence, reasoning, sensitive_fields

2.2 Create Lineage Service

Create file:apps/backend/app/services/governance/lineage_service.py:

LineageService class:

• get_dataset_lineage(dataset_id, depth=5): Build lineage graph
  • Query DatasetVersion and TransformationLineage models
  • Build parent-child relationships
  • Include transformation details
• get_lineage_graph(dataset_id): Return graph structure for visualization
  • Nodes: datasets/versions with metadata
  • Edges: transformations with details
  • Format compatible with React Flow
• get_downstream_datasets(dataset_id): Find all derived datasets
• get_upstream_datasets(dataset_id): Find all source datasets
• get_impact_analysis(dataset_id): Analyze impact of changes
• export_lineage(dataset_id, format='json'): Export lineage data

2.3 Create Retention Service

Create file:apps/backend/app/services/governance/retention_service.py:

RetentionService class extending BaseService[RetentionPolicy]:

• create_policy(user_id, policy_data): Create retention policy
• apply_policy_to_dataset(dataset_id, policy_id): Apply policy
• calculate_expiry_date(dataset, policy): Calculate expiration
• schedule_deletion(dataset_id, policy_id): Create retention schedule
• get_expiring_datasets(days_ahead=30): Find datasets expiring soon
• execute_scheduled_deletions(): Background task to delete expired data
• extend_retention(dataset_id, additional_days): Extend retention period
• get_retention_report(user_id): Report on retention status

RetentionScheduler class:

• start_scheduler(): Initialize background scheduler
• _check_expirations(): Periodic check (run every 6 hours)
• _execute_deletion(schedule_id): Safely delete dataset and related data
• _notify_before_deletion(dataset_id, days_until): Send notifications

2.4 Create Consent Service

Create file:apps/backend/app/services/governance/consent_service.py:

ConsentService class extending BaseService[ConsentRecord]:

• record_consent(dataset_id, user_id, consent_data): Record consent
• withdraw_consent(consent_id, user_id): Withdraw consent
• check_consent(dataset_id, purpose): Verify consent exists
• get_active_consents(dataset_id): Get all active consents
• get_expired_consents(user_id): Find expired consents
• get_consent_report(user_id): Consent management report

2.5 Create Audit Service

Create file:apps/backend/app/services/governance/audit_service.py:

AuditService class extending BaseService[DataAccessAudit]:

• log_access(dataset_id, user_id, access_type, details): Log data access
• get_access_history(dataset_id, start_date, end_date): Query audit trail
• get_user_activity(user_id, start_date, end_date): User activity report
• get_compliance_report(user_id, regulation='GDPR'): Generate compliance report
• export_audit_trail(filters, format='csv'): Export audit data

2.6 Create Compliance Reporting Service

Create file:apps/backend/app/services/governance/compliance_service.py:

ComplianceService class:

• generate_gdpr_report(user_id): GDPR compliance report
  • Data inventory with classifications
  • PII processing activities
  • Consent records
  • Data retention status
  • Access audit trail
• generate_ccpa_report(user_id): CCPA compliance report
• get_data_inventory(user_id): Complete data inventory
• get_pii_processing_activities(user_id): PII processing summary
• check_compliance_status(user_id): Overall compliance health check

2.7 Create Governance Middleware

Create file:apps/backend/app/middleware/governance.py:

GovernanceMiddleware:

• Intercept dataset access requests
• Log to DataAccessAudit
• Check consent requirements
• Enforce classification-based access controls
• Add to FastAPI middleware stack in file:apps/backend/app/main.py

3. Backend API Routes - Governance Endpoints

3.1 Create Governance Router

Create file:apps/backend/app/api/routes/governance.py:

Classification Endpoints:

• POST /api/v1/governance/classify/{dataset_id}: Classify dataset
• GET /api/v1/governance/classify/{dataset_id}: Get classification
• PUT /api/v1/governance/classify/{dataset_id}: Update classification
• POST /api/v1/governance/classify/suggest/{dataset_id}: Get suggestions
• POST /api/v1/governance/classify/bulk: Bulk classify datasets

Lineage Endpoints:

• GET /api/v1/governance/lineage/{dataset_id}: Get lineage graph
• GET /api/v1/governance/lineage/{dataset_id}/upstream: Get upstream datasets
• GET /api/v1/governance/lineage/{dataset_id}/downstream: Get downstream datasets
• GET /api/v1/governance/lineage/{dataset_id}/impact: Impact analysis
• GET /api/v1/governance/lineage/{dataset_id}/export: Export lineage

Retention Endpoints:

• POST /api/v1/governance/retention/policies: Create policy
• GET /api/v1/governance/retention/policies: List policies
• PUT /api/v1/governance/retention/policies/{policy_id}: Update policy
• DELETE /api/v1/governance/retention/policies/{policy_id}: Delete policy
• POST /api/v1/governance/retention/apply/{dataset_id}: Apply policy to dataset
• GET /api/v1/governance/retention/expiring: Get expiring datasets
• POST /api/v1/governance/retention/extend/{dataset_id}: Extend retention

Consent Endpoints:

• POST /api/v1/governance/consent: Record consent
• GET /api/v1/governance/consent/{dataset_id}: Get consents for dataset
• DELETE /api/v1/governance/consent/{consent_id}: Withdraw consent
• GET /api/v1/governance/consent/expired: Get expired consents

Audit Endpoints:

• GET /api/v1/governance/audit/{dataset_id}: Get access history
• GET /api/v1/governance/audit/user/{user_id}: Get user activity
• POST /api/v1/governance/audit/export: Export audit trail

Compliance Endpoints:

• GET /api/v1/governance/compliance/gdpr: GDPR report
• GET /api/v1/governance/compliance/ccpa: CCPA report
• GET /api/v1/governance/compliance/inventory: Data inventory
• GET /api/v1/governance/compliance/status: Compliance status

3.2 Register Router

Update file:apps/backend/app/main.py to include governance router:

from app.api.routes import governance
app.include_router(
    governance.router,
    prefix=f"{settings.API_V1_STR}/governance",
    tags=["governance"]
)

4. Backend Schemas - Request/Response Models

Create file:apps/backend/app/schemas/governance.py:

Request Schemas:

• ClassifyDatasetRequest: manual_classification, justification
• CreateRetentionPolicyRequest: name, description, retention_days, classification_level, auto_delete
• RecordConsentRequest: consent_type, purpose, legal_basis, expiry_date
• AuditQueryRequest: start_date, end_date, access_types, user_ids

Response Schemas:

• ClassificationResponse: classification, confidence, sensitive_fields, classified_at
• LineageGraphResponse: nodes, edges, metadata
• RetentionPolicyResponse: policy details with applied datasets count
• ConsentResponse: consent details with status
• AuditRecordResponse: audit entry with user and dataset info
• ComplianceReportResponse: comprehensive compliance data

5. Frontend Components - Governance UI

5.1 Create Governance Dashboard Page

Create file:apps/frontend/app/governance/page.tsx:

Main Dashboard Layout:

• Overview cards: Total datasets, Classified datasets, Active policies, Consent records
• Quick actions: Classify datasets, Create policy, View lineage, Generate report
• Recent activity: Latest classifications, policy applications, consent changes
• Compliance status indicators: GDPR, CCPA compliance scores
• Navigation to sub-pages: Classification, Lineage, Retention, Consent, Audit

5.2 Create Classification Management Page

Create file:apps/frontend/app/governance/classification/page.tsx:

Features:

• Dataset list with classification status
• Bulk classification action
• Filter by classification level
• Manual classification dialog
• Classification suggestions display
• Confidence score visualization
• Sensitive fields highlighting

5.3 Create Lineage Visualization Page

Create file:apps/frontend/app/governance/lineage/[datasetId]/page.tsx:

Features:

• React Flow graph visualization using @xyflow/react
• Interactive nodes: datasets with metadata tooltips
• Edges: transformations with details on hover
• Controls: Zoom, pan, fit view, layout options
• Filters: Show/hide transformation types, depth control
• Export: PNG, SVG, JSON formats
• Impact analysis panel: Downstream datasets affected

Node Types:

• Dataset node: Name, classification badge, row/column count
• Transformation edge: Type, date, data loss percentage

5.4 Create Retention Policy Management Page

Create file:apps/frontend/app/governance/retention/page.tsx:

Features:

• Policy list with active/inactive status
• Create/edit policy form
• Apply policy to datasets (multi-select)
• Expiring datasets table with countdown
• Extend retention action
• Policy impact preview
• Deletion schedule calendar view

5.5 Create Consent Management Page

Create file:apps/frontend/app/governance/consent/page.tsx:

Features:

• Consent records table
• Record new consent form
• Withdraw consent action
• Expired consents alert
• Consent status by dataset
• Legal basis documentation

5.6 Create Audit Trail Page

Create file:apps/frontend/app/governance/audit/page.tsx:

Features:

• Audit log table with filters
• Date range picker
• Access type filter
• User filter
• Dataset filter
• Export to CSV
• Activity timeline visualization
• User activity heatmap

5.7 Create Compliance Reports Page

Create file:apps/frontend/app/governance/compliance/page.tsx:

Features:

• Report type selector: GDPR, CCPA, Custom
• Generate report button
• Report sections: Data inventory, PII processing, Consent status, Retention compliance, Access audit
• Export report: PDF, CSV, JSON
• Compliance score dashboard
• Recommendations for improvement

5.8 Create Shared Components

Create file:apps/frontend/components/governance/:

• ClassificationBadge.tsx: Color-coded classification badge
• LineageGraph.tsx: Reusable React Flow lineage component
• RetentionPolicyCard.tsx: Policy display card
• ConsentStatusIndicator.tsx: Consent status icon
• AuditLogTable.tsx: Reusable audit table
• ComplianceScoreCard.tsx: Compliance score visualization
• DataInventoryTable.tsx: Data inventory display

6. Frontend Services - API Integration

Create file:apps/frontend/lib/services/governance.ts:

API Client Functions:

• classifyDataset(datasetId, classification?): Classify dataset
• getClassification(datasetId): Get classification
• getLineageGraph(datasetId): Fetch lineage data
• createRetentionPolicy(policyData): Create policy
• applyRetentionPolicy(datasetId, policyId): Apply policy
• recordConsent(consentData): Record consent
• getAuditTrail(filters): Fetch audit logs
• generateComplianceReport(type): Generate report
• exportLineage(datasetId, format): Export lineage

7. Background Tasks - Automated Governance

7.1 Create Retention Scheduler

Create file:apps/backend/app/services/governance/scheduler.py:

RetentionScheduler:

• Use asyncio background task
• Start in file:apps/backend/app/main.py lifespan
• Run every 6 hours
• Check RetentionSchedule for expired datasets
• Execute deletions with proper cleanup
• Send notifications before deletion (7 days, 1 day)
• Log all actions to audit trail

7.2 Create Classification Suggestion Task

Add to file:apps/backend/app/services/governance/scheduler.py:

ClassificationScheduler:

• Run daily
• Find unclassified datasets
• Generate classification suggestions
• Store suggestions for user review
• Notify users of pending classifications

8. Database Migrations and Indexes

8.1 Add Indexes

Ensure indexes in model Settings classes:

• DataClassificationRecord: dataset_id, user_id, classified_at
• RetentionPolicy: user_id, is_active
• RetentionSchedule: dataset_id, retention_expires_at, deletion_scheduled
• ConsentRecord: dataset_id, user_id, consent_type, expiry_date
• DataAccessAudit: dataset_id, user_id, accessed_at, access_type

8.2 Update Existing Data

Create migration script file:apps/backend/scripts/migrate_governance.py:

• Add default classification to existing datasets (INTERNAL)
• Create default retention policy (365 days)
• Backfill audit records from existing access logs

9. Testing Strategy

9.1 Backend Unit Tests

Create test files:

• file:apps/backend/tests/test_services/test_classification_service.py
• file:apps/backend/tests/test_services/test_lineage_service.py
• file:apps/backend/tests/test_services/test_retention_service.py
• file:apps/backend/tests/test_services/test_consent_service.py
• file:apps/backend/tests/test_services/test_audit_service.py
• file:apps/backend/tests/test_api/test_governance.py

9.2 Frontend Component Tests

Create test files:

• file:apps/frontend/__tests__/governance/ClassificationPage.test.tsx
• file:apps/frontend/__tests__/governance/LineageGraph.test.tsx
• file:apps/frontend/__tests__/governance/RetentionPage.test.tsx

9.3 Integration Tests

• End-to-end classification workflow
• Lineage graph generation and visualization
• Retention policy application and deletion
• Consent recording and withdrawal
• Compliance report generation

10. Documentation

10.1 API Documentation

Update file:apps/backend/app/services/api_documentation.py:

• Add governance endpoints to OpenAPI spec
• Include request/response examples
• Document authentication requirements
• Add compliance-related notes

10.2 User Documentation

Create file:GOVERNANCE_GUIDE.md:

• Data classification guidelines
• Retention policy best practices
• Consent management procedures
• Compliance reporting instructions
• Lineage visualization guide

Architecture Diagram

graph TD
    subgraph Frontend
        A[Governance Dashboard] --> B[Classification UI]
        A --> C[Lineage Visualization]
        A --> D[Retention Management]
        A --> E[Consent Management]
        A --> F[Audit Trail]
        A --> G[Compliance Reports]
    end
    
    subgraph Backend Services
        H[ClassificationService] --> M[OpenAI API]
        I[LineageService]
        J[RetentionService]
        K[ConsentService]
        L[AuditService]
        N[ComplianceService]
    end
    
    subgraph Background Tasks
        O[RetentionScheduler]
        P[ClassificationScheduler]
    end
    
    subgraph Database
        Q[(DataClassificationRecord)]
        R[(RetentionPolicy)]
        S[(ConsentRecord)]
        T[(DataAccessAudit)]
        U[(DatasetMetadata)]
        V[(DatasetVersion)]
        W[(TransformationLineage)]
    end
    
    B --> H
    C --> I
    D --> J
    E --> K
    F --> L
    G --> N
    
    H --> Q
    H --> U
    I --> V
    I --> W
    J --> R
    J --> U
    K --> S
    L --> T
    N --> Q
    N --> S
    N --> T
    
    O --> J
    P --> H

Loading

Data Flow - Classification Process

sequenceDiagram
    participant User
    participant UI as Classification UI
    participant API as Governance API
    participant CS as ClassificationService
    participant AI as OpenAI
    participant DB as MongoDB
    
    User->>UI: Request classification
    UI->>API: POST /governance/classify/{dataset_id}
    API->>CS: classify_dataset(dataset_id)
    CS->>DB: Fetch dataset metadata
    DB-->>CS: Dataset with PII report
    CS->>AI: Analyze dataset for classification
    AI-->>CS: Classification + confidence
    CS->>DB: Save DataClassificationRecord
    CS->>DB: Update DatasetMetadata
    DB-->>CS: Success
    CS-->>API: Classification result
    API-->>UI: Classification response
    UI-->>User: Display classification badge

Loading

Compliance Report Structure

Section	Data Source	Content
Data Inventory	DatasetMetadata, DataClassificationRecord	All datasets with classifications
PII Processing	PIIReport, DataClassificationRecord	Datasets containing PII
Consent Records	ConsentRecord	Active and expired consents
Retention Status	RetentionPolicy, RetentionSchedule	Retention compliance
Access Audit	DataAccessAudit	Data access history
Recommendations	ComplianceService	Compliance improvements

Implementation Timeline

Week	Focus Area	Deliverables
1	Backend Models & Services	Classification, Retention, Consent models and services
2	Lineage & Audit Services	Lineage graph generation, audit logging
3	API Routes & Schemas	All governance endpoints
4	Frontend Dashboard & Classification	Main dashboard, classification UI
5	Frontend Lineage & Retention	Lineage visualization, retention management
6	Frontend Consent & Compliance	Consent management, compliance reports, testing

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase has a solid foundation for governance features with existing PII detection in `DatasetMetadata`, versioning models (`DatasetVersion`, `TransformationLineage`), and audit fields. The architecture follows a service layer pattern with `BaseService` for CRUD operations, FastAPI backend with Beanie ODM (MongoDB), and Next.js frontend with TypeScript. Background task processing uses FastAPI's `BackgroundTasks` and `asyncio.Task`. The frontend uses React Flow (@xyflow/react) which is perfect for lineage visualization, and shadcn/ui components for consistent UI patterns.

## Approach

The implementation will extend existing models and services rather than creating entirely new systems. Data classification will be added to `DatasetMetadata` with an AI-powered classification engine. Lineage visualization will leverage existing `TransformationLineage` and `DatasetVersion` models. Retention policies will use a new `RetentionPolicy` model with a scheduler service for automated deletion. Consent management will extend PII tracking with user consent records. Compliance reporting will aggregate data from classification, lineage, and audit trails. The governance dashboard will be a new frontend page using existing component patterns.

## Implementation Steps

### 1. Backend Models - Data Governance Core

#### 1.1 Create Data Classification Model
Create `file:apps/backend/app/models/governance.py` with the following models:

- **`DataClassification`** enum: PUBLIC, INTERNAL, CONFIDENTIAL, RESTRICTED
- **`ClassificationRule`** model: Pattern-based rules for auto-classification
- **`DataClassificationRecord`** document:
  - `dataset_id`: Indexed string
  - `user_id`: Indexed string
  - `classification`: DataClassification enum
  - `classification_method`: AUTO or MANUAL
  - `confidence_score`: Float (0-1) for auto-classification
  - `classified_by`: User ID or "system"
  - `classified_at`: DateTime
  - `justification`: String explaining classification
  - `sensitive_fields`: List of field names with sensitivity
  - `review_required`: Boolean flag
  - `reviewed_by`: Optional user ID
  - `reviewed_at`: Optional datetime

#### 1.2 Create Retention Policy Model
Add to `file:apps/backend/app/models/governance.py`:

- **`RetentionPolicy`** document:
  - `policy_id`: Indexed unique string
  - `user_id`: Indexed string (owner)
  - `name`: Policy name
  - `description`: Policy description
  - `classification_level`: Optional DataClassification (apply to specific classification)
  - `retention_days`: Integer (days to retain)
  - `auto_delete`: Boolean (enable automated deletion)
  - `applies_to_datasets`: List of dataset IDs (empty = all)
  - `exclude_datasets`: List of dataset IDs to exclude
  - `created_at`: DateTime
  - `updated_at`: DateTime
  - `is_active`: Boolean

- **`RetentionSchedule`** document:
  - `schedule_id`: Indexed unique string
  - `dataset_id`: Indexed string
  - `policy_id`: String reference
  - `retention_expires_at`: Indexed datetime
  - `deletion_scheduled`: Boolean
  - `deletion_executed`: Boolean
  - `deleted_at`: Optional datetime
  - `created_at`: DateTime

#### 1.3 Create Consent Management Model
Add to `file:apps/backend/app/models/governance.py`:

- **`ConsentRecord`** document:
  - `consent_id`: Indexed unique string
  - `dataset_id`: Indexed string
  - `user_id`: Indexed string (data owner)
  - `data_subject_id`: Optional string (person whose data it is)
  - `consent_type`: Enum (PROCESSING, STORAGE, SHARING, ANALYTICS)
  - `consent_given`: Boolean
  - `consent_date`: DateTime
  - `expiry_date`: Optional datetime
  - `withdrawn`: Boolean
  - `withdrawn_at`: Optional datetime
  - `purpose`: String describing data use purpose
  - `legal_basis`: String (GDPR Article 6 basis)
  - `metadata`: Dict for additional context

#### 1.4 Create Audit Trail Model
Add to `file:apps/backend/app/models/governance.py`:

- **`DataAccessAudit`** document:
  - `audit_id`: Indexed unique string
  - `dataset_id`: Indexed string
  - `user_id`: Indexed string
  - `access_type`: Enum (READ, WRITE, DELETE, EXPORT, TRANSFORM)
  - `accessed_at`: Indexed datetime
  - `ip_address`: Optional string
  - `user_agent`: Optional string
  - `action_details`: Dict with specific action info
  - `classification_at_access`: Optional DataClassification
  - `compliance_flags`: List of strings (GDPR, CCPA, etc.)

#### 1.5 Update DatasetMetadata Model
Extend `file:apps/backend/app/models/dataset.py`:

- Add `classification`: Optional DataClassification field
- Add `classification_record_id`: Optional string reference
- Add `retention_policy_id`: Optional string reference
- Add `retention_expires_at`: Optional datetime
- Add `consent_required`: Boolean (default False)
- Add `consent_records`: List of consent IDs

#### 1.6 Register Models in Beanie
Update `file:apps/backend/app/main.py` in the `lifespan` function to include new models in `init_beanie`:
- `DataClassificationRecord`
- `RetentionPolicy`
- `RetentionSchedule`
- `ConsentRecord`
- `DataAccessAudit`

### 2. Backend Services - Governance Logic

#### 2.1 Create Classification Service
Create `file:apps/backend/app/services/governance/classification_service.py`:

**ClassificationService** class extending `BaseService[DataClassificationRecord]`:
- `classify_dataset(dataset_id, user_id, manual_classification=None)`: Main classification method
- `_auto_classify(dataset)`: AI-powered classification using OpenAI
  - Analyze schema, PII presence, field names, data sensitivity
  - Return classification with confidence score
- `_apply_classification_rules(dataset)`: Pattern-based rule matching
- `suggest_classification(dataset_id)`: Return suggestions without saving
- `update_classification(dataset_id, classification, user_id)`: Manual override
- `get_classification_history(dataset_id)`: Audit trail of classification changes
- `bulk_classify(user_id, filters)`: Classify multiple datasets

**AI Classification Prompt Strategy**:
- Analyze PII report, field names, data types, statistics
- Consider industry standards (NIST, ISO 27001)
- Return JSON with classification, confidence, reasoning, sensitive_fields

#### 2.2 Create Lineage Service
Create `file:apps/backend/app/services/governance/lineage_service.py`:

**LineageService** class:
- `get_dataset_lineage(dataset_id, depth=5)`: Build lineage graph
  - Query `DatasetVersion` and `TransformationLineage` models
  - Build parent-child relationships
  - Include transformation details
- `get_lineage_graph(dataset_id)`: Return graph structure for visualization
  - Nodes: datasets/versions with metadata
  - Edges: transformations with details
  - Format compatible with React Flow
- `get_downstream_datasets(dataset_id)`: Find all derived datasets
- `get_upstream_datasets(dataset_id)`: Find all source datasets
- `get_impact_analysis(dataset_id)`: Analyze impact of changes
- `export_lineage(dataset_id, format='json')`: Export lineage data

#### 2.3 Create Retention Service
Create `file:apps/backend/app/services/governance/retention_service.py`:

**RetentionService** class extending `BaseService[RetentionPolicy]`:
- `create_policy(user_id, policy_data)`: Create retention policy
- `apply_policy_to_dataset(dataset_id, policy_id)`: Apply policy
- `calculate_expiry_date(dataset, policy)`: Calculate expiration
- `schedule_deletion(dataset_id, policy_id)`: Create retention schedule
- `get_expiring_datasets(days_ahead=30)`: Find datasets expiring soon
- `execute_scheduled_deletions()`: Background task to delete expired data
- `extend_retention(dataset_id, additional_days)`: Extend retention period
- `get_retention_report(user_id)`: Report on retention status

**RetentionScheduler** class:
- `start_scheduler()`: Initialize background scheduler
- `_check_expirations()`: Periodic check (run every 6 hours)
- `_execute_deletion(schedule_id)`: Safely delete dataset and related data
- `_notify_before_deletion(dataset_id, days_until)`: Send notifications

#### 2.4 Create Consent Service
Create `file:apps/backend/app/services/governance/consent_service.py`:

**ConsentService** class extending `BaseService[ConsentRecord]`:
- `record_consent(dataset_id, user_id, consent_data)`: Record consent
- `withdraw_consent(consent_id, user_id)`: Withdraw consent
- `check_consent(dataset_id, purpose)`: Verify consent exists
- `get_active_consents(dataset_id)`: Get all active consents
- `get_expired_consents(user_id)`: Find expired consents
- `get_consent_report(user_id)`: Consent management report

#### 2.5 Create Audit Service
Create `file:apps/backend/app/services/governance/audit_service.py`:

**AuditService** class extending `BaseService[DataAccessAudit]`:
- `log_access(dataset_id, user_id, access_type, details)`: Log data access
- `get_access_history(dataset_id, start_date, end_date)`: Query audit trail
- `get_user_activity(user_id, start_date, end_date)`: User activity report
- `get_compliance_report(user_id, regulation='GDPR')`: Generate compliance report
- `export_audit_trail(filters, format='csv')`: Export audit data

#### 2.6 Create Compliance Reporting Service
Create `file:apps/backend/app/services/governance/compliance_service.py`:

**ComplianceService** class:
- `generate_gdpr_report(user_id)`: GDPR compliance report
  - Data inventory with classifications
  - PII processing activities
  - Consent records
  - Data retention status
  - Access audit trail
- `generate_ccpa_report(user_id)`: CCPA compliance report
- `get_data_inventory(user_id)`: Complete data inventory
- `get_pii_processing_activities(user_id)`: PII processing summary
- `check_compliance_status(user_id)`: Overall compliance health check

#### 2.7 Create Governance Middleware
Create `file:apps/backend/app/middleware/governance.py`:

**GovernanceMiddleware**:
- Intercept dataset access requests
- Log to `DataAccessAudit`
- Check consent requirements
- Enforce classification-based access controls
- Add to FastAPI middleware stack in `file:apps/backend/app/main.py`

### 3. Backend API Routes - Governance Endpoints

#### 3.1 Create Governance Router
Create `file:apps/backend/app/api/routes/governance.py`:

**Classification Endpoints**:
- `POST /api/v1/governance/classify/{dataset_id}`: Classify dataset
- `GET /api/v1/governance/classify/{dataset_id}`: Get classification
- `PUT /api/v1/governance/classify/{dataset_id}`: Update classification
- `POST /api/v1/governance/classify/suggest/{dataset_id}`: Get suggestions
- `POST /api/v1/governance/classify/bulk`: Bulk classify datasets

**Lineage Endpoints**:
- `GET /api/v1/governance/lineage/{dataset_id}`: Get lineage graph
- `GET /api/v1/governance/lineage/{dataset_id}/upstream`: Get upstream datasets
- `GET /api/v1/governance/lineage/{dataset_id}/downstream`: Get downstream datasets
- `GET /api/v1/governance/lineage/{dataset_id}/impact`: Impact analysis
- `GET /api/v1/governance/lineage/{dataset_id}/export`: Export lineage

**Retention Endpoints**:
- `POST /api/v1/governance/retention/policies`: Create policy
- `GET /api/v1/governance/retention/policies`: List policies
- `PUT /api/v1/governance/retention/policies/{policy_id}`: Update policy
- `DELETE /api/v1/governance/retention/policies/{policy_id}`: Delete policy
- `POST /api/v1/governance/retention/apply/{dataset_id}`: Apply policy to dataset
- `GET /api/v1/governance/retention/expiring`: Get expiring datasets
- `POST /api/v1/governance/retention/extend/{dataset_id}`: Extend retention

**Consent Endpoints**:
- `POST /api/v1/governance/consent`: Record consent
- `GET /api/v1/governance/consent/{dataset_id}`: Get consents for dataset
- `DELETE /api/v1/governance/consent/{consent_id}`: Withdraw consent
- `GET /api/v1/governance/consent/expired`: Get expired consents

**Audit Endpoints**:
- `GET /api/v1/governance/audit/{dataset_id}`: Get access history
- `GET /api/v1/governance/audit/user/{user_id}`: Get user activity
- `POST /api/v1/governance/audit/export`: Export audit trail

**Compliance Endpoints**:
- `GET /api/v1/governance/compliance/gdpr`: GDPR report
- `GET /api/v1/governance/compliance/ccpa`: CCPA report
- `GET /api/v1/governance/compliance/inventory`: Data inventory
- `GET /api/v1/governance/compliance/status`: Compliance status

#### 3.2 Register Router
Update `file:apps/backend/app/main.py` to include governance router:
```python
from app.api.routes import governance
app.include_router(
    governance.router,
    prefix=f"{settings.API_V1_STR}/governance",
    tags=["governance"]
)
```

### 4. Backend Schemas - Request/Response Models

Create `file:apps/backend/app/schemas/governance.py`:

**Request Schemas**:
- `ClassifyDatasetRequest`: manual_classification, justification
- `CreateRetentionPolicyRequest`: name, description, retention_days, classification_level, auto_delete
- `RecordConsentRequest`: consent_type, purpose, legal_basis, expiry_date
- `AuditQueryRequest`: start_date, end_date, access_types, user_ids

**Response Schemas**:
- `ClassificationResponse`: classification, confidence, sensitive_fields, classified_at
- `LineageGraphResponse`: nodes, edges, metadata
- `RetentionPolicyResponse`: policy details with applied datasets count
- `ConsentResponse`: consent details with status
- `AuditRecordResponse`: audit entry with user and dataset info
- `ComplianceReportResponse`: comprehensive compliance data

### 5. Frontend Components - Governance UI

#### 5.1 Create Governance Dashboard Page
Create `file:apps/frontend/app/governance/page.tsx`:

**Main Dashboard Layout**:
- Overview cards: Total datasets, Classified datasets, Active policies, Consent records
- Quick actions: Classify datasets, Create policy, View lineage, Generate report
- Recent activity: Latest classifications, policy applications, consent changes
- Compliance status indicators: GDPR, CCPA compliance scores
- Navigation to sub-pages: Classification, Lineage, Retention, Consent, Audit

#### 5.2 Create Classification Management Page
Create `file:apps/frontend/app/governance/classification/page.tsx`:

**Features**:
- Dataset list with classification status
- Bulk classification action
- Filter by classification level
- Manual classification dialog
- Classification suggestions display
- Confidence score visualization
- Sensitive fields highlighting

#### 5.3 Create Lineage Visualization Page
Create `file:apps/frontend/app/governance/lineage/[datasetId]/page.tsx`:

**Features**:
- React Flow graph visualization using `@xyflow/react`
- Interactive nodes: datasets with metadata tooltips
- Edges: transformations with details on hover
- Controls: Zoom, pan, fit view, layout options
- Filters: Show/hide transformation types, depth control
- Export: PNG, SVG, JSON formats
- Impact analysis panel: Downstream datasets affected

**Node Types**:
- Dataset node: Name, classification badge, row/column count
- Transformation edge: Type, date, data loss percentage

#### 5.4 Create Retention Policy Management Page
Create `file:apps/frontend/app/governance/retention/page.tsx`:

**Features**:
- Policy list with active/inactive status
- Create/edit policy form
- Apply policy to datasets (multi-select)
- Expiring datasets table with countdown
- Extend retention action
- Policy impact preview
- Deletion schedule calendar view

#### 5.5 Create Consent Management Page
Create `file:apps/frontend/app/governance/consent/page.tsx`:

**Features**:
- Consent records table
- Record new consent form
- Withdraw consent action
- Expired consents alert
- Consent status by dataset
- Legal basis documentation

#### 5.6 Create Audit Trail Page
Create `file:apps/frontend/app/governance/audit/page.tsx`:

**Features**:
- Audit log table with filters
- Date range picker
- Access type filter
- User filter
- Dataset filter
- Export to CSV
- Activity timeline visualization
- User activity heatmap

#### 5.7 Create Compliance Reports Page
Create `file:apps/frontend/app/governance/compliance/page.tsx`:

**Features**:
- Report type selector: GDPR, CCPA, Custom
- Generate report button
- Report sections: Data inventory, PII processing, Consent status, Retention compliance, Access audit
- Export report: PDF, CSV, JSON
- Compliance score dashboard
- Recommendations for improvement

#### 5.8 Create Shared Components
Create `file:apps/frontend/components/governance/`:

- `ClassificationBadge.tsx`: Color-coded classification badge
- `LineageGraph.tsx`: Reusable React Flow lineage component
- `RetentionPolicyCard.tsx`: Policy display card
- `ConsentStatusIndicator.tsx`: Consent status icon
- `AuditLogTable.tsx`: Reusable audit table
- `ComplianceScoreCard.tsx`: Compliance score visualization
- `DataInventoryTable.tsx`: Data inventory display

### 6. Frontend Services - API Integration

Create `file:apps/frontend/lib/services/governance.ts`:

**API Client Functions**:
- `classifyDataset(datasetId, classification?)`: Classify dataset
- `getClassification(datasetId)`: Get classification
- `getLineageGraph(datasetId)`: Fetch lineage data
- `createRetentionPolicy(policyData)`: Create policy
- `applyRetentionPolicy(datasetId, policyId)`: Apply policy
- `recordConsent(consentData)`: Record consent
- `getAuditTrail(filters)`: Fetch audit logs
- `generateComplianceReport(type)`: Generate report
- `exportLineage(datasetId, format)`: Export lineage

### 7. Background Tasks - Automated Governance

#### 7.1 Create Retention Scheduler
Create `file:apps/backend/app/services/governance/scheduler.py`:

**RetentionScheduler**:
- Use `asyncio` background task
- Start in `file:apps/backend/app/main.py` lifespan
- Run every 6 hours
- Check `RetentionSchedule` for expired datasets
- Execute deletions with proper cleanup
- Send notifications before deletion (7 days, 1 day)
- Log all actions to audit trail

#### 7.2 Create Classification Suggestion Task
Add to `file:apps/backend/app/services/governance/scheduler.py`:

**ClassificationScheduler**:
- Run daily
- Find unclassified datasets
- Generate classification suggestions
- Store suggestions for user review
- Notify users of pending classifications

### 8. Database Migrations and Indexes

#### 8.1 Add Indexes
Ensure indexes in model `Settings` classes:
- `DataClassificationRecord`: dataset_id, user_id, classified_at
- `RetentionPolicy`: user_id, is_active
- `RetentionSchedule`: dataset_id, retention_expires_at, deletion_scheduled
- `ConsentRecord`: dataset_id, user_id, consent_type, expiry_date
- `DataAccessAudit`: dataset_id, user_id, accessed_at, access_type

#### 8.2 Update Existing Data
Create migration script `file:apps/backend/scripts/migrate_governance.py`:
- Add default classification to existing datasets (INTERNAL)
- Create default retention policy (365 days)
- Backfill audit records from existing access logs

### 9. Testing Strategy

#### 9.1 Backend Unit Tests
Create test files:
- `file:apps/backend/tests/test_services/test_classification_service.py`
- `file:apps/backend/tests/test_services/test_lineage_service.py`
- `file:apps/backend/tests/test_services/test_retention_service.py`
- `file:apps/backend/tests/test_services/test_consent_service.py`
- `file:apps/backend/tests/test_services/test_audit_service.py`
- `file:apps/backend/tests/test_api/test_governance.py`

#### 9.2 Frontend Component Tests
Create test files:
- `file:apps/frontend/__tests__/governance/ClassificationPage.test.tsx`
- `file:apps/frontend/__tests__/governance/LineageGraph.test.tsx`
- `file:apps/frontend/__tests__/governance/RetentionPage.test.tsx`

#### 9.3 Integration Tests
- End-to-end classification workflow
- Lineage graph generation and visualization
- Retention policy application and deletion
- Consent recording and withdrawal
- Compliance report generation

### 10. Documentation

#### 10.1 API Documentation
Update `file:apps/backend/app/services/api_documentation.py`:
- Add governance endpoints to OpenAPI spec
- Include request/response examples
- Document authentication requirements
- Add compliance-related notes

#### 10.2 User Documentation
Create `file:GOVERNANCE_GUIDE.md`:
- Data classification guidelines
- Retention policy best practices
- Consent management procedures
- Compliance reporting instructions
- Lineage visualization guide

## Architecture Diagram

```mermaid
graph TD
    subgraph Frontend
        A[Governance Dashboard] --> B[Classification UI]
        A --> C[Lineage Visualization]
        A --> D[Retention Management]
        A --> E[Consent Management]
        A --> F[Audit Trail]
        A --> G[Compliance Reports]
    end
    
    subgraph Backend Services
        H[ClassificationService] --> M[OpenAI API]
        I[LineageService]
        J[RetentionService]
        K[ConsentService]
        L[AuditService]
        N[ComplianceService]
    end
    
    subgraph Background Tasks
        O[RetentionScheduler]
        P[ClassificationScheduler]
    end
    
    subgraph Database
        Q[(DataClassificationRecord)]
        R[(RetentionPolicy)]
        S[(ConsentRecord)]
        T[(DataAccessAudit)]
        U[(DatasetMetadata)]
        V[(DatasetVersion)]
        W[(TransformationLineage)]
    end
    
    B --> H
    C --> I
    D --> J
    E --> K
    F --> L
    G --> N
    
    H --> Q
    H --> U
    I --> V
    I --> W
    J --> R
    J --> U
    K --> S
    L --> T
    N --> Q
    N --> S
    N --> T
    
    O --> J
    P --> H
```

## Data Flow - Classification Process

```mermaid
sequenceDiagram
    participant User
    participant UI as Classification UI
    participant API as Governance API
    participant CS as ClassificationService
    participant AI as OpenAI
    participant DB as MongoDB
    
    User->>UI: Request classification
    UI->>API: POST /governance/classify/{dataset_id}
    API->>CS: classify_dataset(dataset_id)
    CS->>DB: Fetch dataset metadata
    DB-->>CS: Dataset with PII report
    CS->>AI: Analyze dataset for classification
    AI-->>CS: Classification + confidence
    CS->>DB: Save DataClassificationRecord
    CS->>DB: Update DatasetMetadata
    DB-->>CS: Success
    CS-->>API: Classification result
    API-->>UI: Classification response
    UI-->>User: Display classification badge
```

## Compliance Report Structure

| Section | Data Source | Content |
|---------|-------------|---------|
| Data Inventory | DatasetMetadata, DataClassificationRecord | All datasets with classifications |
| PII Processing | PIIReport, DataClassificationRecord | Datasets containing PII |
| Consent Records | ConsentRecord | Active and expired consents |
| Retention Status | RetentionPolicy, RetentionSchedule | Retention compliance |
| Access Audit | DataAccessAudit | Data access history |
| Recommendations | ComplianceService | Compliance improvements |

## Implementation Timeline

| Week | Focus Area | Deliverables |
|------|-----------|--------------|
| 1 | Backend Models & Services | Classification, Retention, Consent models and services |
| 2 | Lineage & Audit Services | Lineage graph generation, audit logging |
| 3 | API Routes & Schemas | All governance endpoints |
| 4 | Frontend Dashboard & Classification | Main dashboard, classification UI |
| 5 | Frontend Lineage & Retention | Lineage visualization, retention management |
| 6 | Frontend Consent & Compliance | Consent management, compliance reports, testing |

---

Execution Information

Branch: main
Commit: 1963a18