VERIFICATION_SYSTEM_GUIDE.md

Voter Verification System Guide

Overview

The Voter Verification System allows human-in-the-loop verification and editing of extracted voter records with smart suggestions based on existing campaign data.

Features

1. Progress Tracking

• Real-time progress bar showing verified/remaining/total voters
• Percentage completion display
• Visual feedback on verification progress

2. Smart Suggestions

The system provides intelligent suggestions based on:

• Village/Cell Matching: Matches village names with registered cells
• Polling Station: Suggests correct polling stations based on parish data
• Date Corrections: Fixes common OCR errors in dates (1070 → 1970, 1006 → 2006)
• Confidence Scoring: Each suggestion has a confidence score (high/medium/low)

3. One-Click Apply

• Click "Apply" button to automatically fill in suggested values
• Visual feedback when suggestions are applied
• Automatic cascade updates (e.g., selecting a cell updates ward/division)

4. Cell ID Reconciliation

• Automatically assigns cell_id when village is selected
• Reconciles village names with cell database throughout the system
• Ensures consistency across all voter records

5. Dual Update System

• Updates voters_unverified table (tracking table)
• Automatically updates main voters table with verified data
• Maintains data consistency across both tables

Setup Instructions

Step 1: Create Database Table

Run the SQL script to create the verification table:

# Option 1: Run from Supabase dashboard
- Copy contents of setup_verification_table.sql
- Paste into SQL Editor in Supabase dashboard
- Execute

# Option 2: Use psql (if you have database credentials)
psql -h [hostname] -U postgres -d postgres -f setup_verification_table.sql

Step 2: Import Voters for Verification

Import the extracted voters into the verification table:

# This will import the improved extraction (20,047 voters)
python import_voters_for_verification.py

Expected output:

================================================================================
IMPORTING VOTERS FOR VERIFICATION
================================================================================

Source CSV: extracted_voters_improved.csv

Loaded 20047 voters from CSV

Clearing existing unverified voters...
  [OK] Cleared existing records

Importing voters in batches...
  Batch 1: Imported 500 voters (500/20047)
  Batch 2: Imported 500 voters (1000/20047)
  ...
  Batch 41: Imported 47 voters (20047/20047)

================================================================================
IMPORT COMPLETE
================================================================================
Successfully imported: 20047 voters
Failed: 0 voters

Next steps:
  1. Go to the website and navigate to /verify
  2. Start verifying and editing voter records
  3. Use smart suggestions to fix common errors

================================================================================

Step 3: Access Verification Interface

1. Log into the DNM Data Center system
2. Click "Verify" in the navigation menu (✅ icon)
3. Start verifying voters

Using the Verification Interface

Layout

The verification page has three main sections:

1. Progress Section (Top)

• Verified: Number of voters you've verified
• Remaining: Number left to verify
• Total: Total voters in the system
• Progress Bar: Visual representation of completion

2. Voter Form (Middle)

Edit voter fields:

• Voter ID: National ID number
• Surname: Last name
• Given Name: First name
• Date of Birth: Format DD-MM-YYYY
• Gender: M/F dropdown
• Polling Station: Dropdown of all 54 stations
• Village/Cell: Dropdown of all 74 cells
• Ward: Auto-filled when station/cell selected
• Division: Auto-filled when station/cell selected

3. Smart Suggestions Panel (Bottom)

Shows AI-generated suggestions with:

• Type: What field the suggestion is for
• Confidence: Match percentage (0-100%)
• Reason: Why this suggestion is being made
• Apply Button: One-click to accept suggestion

Workflow

1. Review Record: Look at the current voter data
2. Check Suggestions: Review smart suggestions (if any appear)
3. Apply Suggestions: Click "Apply" on suggestions you want to accept
4. Manual Edits: Fix any remaining errors manually
5. Save & Next: Click to save and load next voter
6. Skip: If unsure, skip to review later

Keyboard Shortcuts

• Tab: Move between fields
• Enter: Submit form (Save & Next)
• Esc: Skip current voter

Smart Suggestion Examples

Example 1: Village/Cell Matching

Voter has village: "BUTARE"
System finds: Cell "BUTARE" in ward "CENTRAL"

Suggestion:
  Type: Village/Cell
  Value: BUTARE
  Reason: Matched with cell: BUTARE in CENTRAL
  Confidence: 90%
  [Apply]

Example 2: Date Correction

Voter has DOB: "11-06-1070"
System detects OCR error

Suggestion:
  Type: Date of Birth
  Value: 11-06-1970
  Reason: OCR error: 1070 → 1970
  Confidence: 95%
  [Apply]

Example 3: Polling Station

Voter has parish: "AUTOOMA"
System finds matching station

Suggestion:
  Type: Polling Station
  Value: AMIIZI MARUNGI CENTRE
  Reason: Matched based on parish: AUTOOMA
  Confidence: 80%
  [Apply]

Tips for Efficient Verification

1. Use Suggestions First: Always check and apply suggestions before manual editing
2. Batch Similar Records: Complete all records from one polling station together
3. Look for Patterns: OCR errors often repeat (e.g., "1070" instead of "1970")
4. Validate Relationships: Ensure village ↔ ward ↔ division are consistent
5. Use Skip Sparingly: Only skip truly ambiguous records

Data Quality Checks

The system automatically validates:

• ✅ Voter ID format (13+ digits + letters)
• ✅ Date format (DD-MM-YYYY)
• ✅ Gender values (M/F only)
• ✅ Polling station exists in database
• ✅ Village/cell exists in database
• ✅ Cell ID is correctly assigned from village name

Expected Timeline

Based on 20,047 voters to verify:

Speed	Time per Voter	Total Time	Daily (4 hrs)
Fast	15 seconds	83 hours	21 days
Medium	30 seconds	167 hours	42 days
Slow	60 seconds	334 hours	84 days

Recommendation: Aim for 30 seconds per voter (with suggestions, should be achievable).

After Verification

Once all voters are verified:

1. Verified data is automatically in the main voters table
2. All cell_ids are properly assigned
3. Data quality is significantly improved
4. Ready for final campaign operations

Troubleshooting

Issue: "Database not available"

• Check .env file has correct SUPABASE_URL and SUPABASE_KEY
• Verify internet connection
• Restart the Flask app

Issue: "No voters to verify"

• Run import_voters_for_verification.py again
• Check that extracted_voters_improved.csv exists

Issue: Suggestions not appearing

• This is normal for some voters
• Suggestions only appear when system finds confident matches
• Continue with manual editing

Issue: Can't find correct cell/station in dropdown

• Check if it exists in database (/structure page)
• Add missing cells/stations first
• Then return to verification

API Endpoints

For developers/automation:

GET  /api/verify/stats             - Get verification progress
GET  /api/verify/next              - Get next unverified voter
POST /api/verify/suggestions       - Get smart suggestions for a voter
POST /api/verify/save              - Save verified voter

Database Schema

voters_unverified table

id              BIGSERIAL PRIMARY KEY
voter_id        TEXT
surname         TEXT
given_name      TEXT
dob             TEXT
gender          TEXT
village         TEXT
cell_id         BIGINT (FK to cells.id)
polling_station TEXT
parish          TEXT
ward            TEXT
division        TEXT
verified        BOOLEAN DEFAULT FALSE
created_at      TIMESTAMPTZ
updated_at      TIMESTAMPTZ

Support

For issues or questions:

1. Check this guide first
2. Review error messages in browser console (F12)
3. Check Flask app logs for backend errors
4. Contact system administrator

---

Remember: Quality over speed! Take time to verify each record correctly.