Use settings.toml to configure LLM OCR reader

- Implement UV as the python build management tool

Author: kvithayathil

.github/workflows/main.yml

@@ -5,37 +5,37 @@ name: Python application
 
 on:
   push:
-    branches: [ "main" ]
+    branches: ["main"]
   pull_request:
-    branches: [ "main" ]
+    branches: ["main"]
 
 permissions:
   contents: read
 
 jobs:
   build:
-
     runs-on: ubuntu-latest
 
     steps:
-    - uses: actions/checkout@v4
-    - name: Set up Python 3.12
-      uses: actions/setup-python@v5
-      with:
-        python-version: "3.12"
-    - name: Display Python version
-      run: python -c "import sys; print(sys.version)"
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip
-        pip install flake8 pytest pytest-cov
-        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
-    - name: Lint with flake8
-      run: |
-        # stop the build if there are Python syntax errors or undefined names
-        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
-        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
-        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
-    - name: Test with pytest
-      run: |
-       pytest --cov app tests/ --cov-report xml --cov-report html --cov-report term
+      - uses: actions/checkout@v4
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+
+      - name: "Set up Python"
+        uses: actions/setup-python@v5
+        with:
+          python-version-file: "pyproject.toml"
+
+      - name: Install the project
+        run: uv sync --all-extras --dev
+
+      - name: Display Python version
+        run: python -c "import sys; print(sys.version)"
+
+      - name: Lint with ruff
+        run: |
+          uv run ruff check app tests
+
+      - name: Run tests with coverage
+        run: |
+          uv run pytest tests --cov app tests/ --cov-report term

README.md

@@ -40,7 +40,7 @@ The goal of the Ballot Initiative project is to reduce the manual labor involved
 
 ![Core Algorithm](app/ballot_initiative_schematic.png)
 
-1. **Extraction:** Forms in PDF format are processed through an OCR engine (using [gpt-4o-mini](https://platform.openai.com/docs/models/gpt-4o-mini)) to crop text sections and extract data.
+1. **Extraction:** Forms in PDF format are processed through an OCR engine (using generative AI) to crop text sections and extract data.
 
 2. **Identification:** The engine identifies and extracts key information (tailored to DC Ballot Initiatives) related to validating signatures:
 
@@ -63,10 +63,14 @@ An alternate approach to get up and running is to use [Github Codespaces](https:
 
 ### Prerequisites
 
-- Python 3.12
-- OpenAI API key[^1]
+- [Python 3.12+](https://wiki.python.org/moin/BeginnersGuide/Download)
+- [UV](https://docs.astral.sh/uv/getting-started/installation/) for building the project
+- API keys for at least one of the following[^1]:
+  - [OpenAI API key](https://help.openai.com/en/articles/4936850-where-do-i-find-my-openai-api-key)
+  - [Mistral API key](https://docs.mistral.ai/getting-started/quickstart/)
+  - [Gemini API key](https://ai.google.dev/gemini-api/docs/api-key)
 
-[^1]: The OpenAI free tier has a low rate limit. To increase the rate limit, you'll have to have a form payment on your OpenAI account. [See this page for details](https://platform.openai.com/docs/guides/rate-limits?tier=tier-one)
+[^1]: The free tiers for these services typically have a low rate limit that can cause issues. Many services require adding a payment method to your account to increase rate limits. Please verify your account settings and usage limits before running the application.
 
 - PDF files of ballot initiative signatures
   - Use fake data in [`sample_data/fake_signed_petitions.pdf`](sample_data/fake_signed_petitions.pdf) folder to test.
@@ -86,8 +90,8 @@ cd ballot-initiative
 2. Create and activate a virtual environment:
 
 ```bash
-# Create virtual environment
-python -m venv venv
+# Initalise project and install dependencies
+uv sync --all-extras --dev
 
 # Activate virtual environment
 # On Windows:
@@ -96,29 +100,20 @@ venv\Scripts\activate
 source venv/bin/activate
 ```
 
-3. Install dependencies:
-
-```bash
-pip install -r requirements.txt
-```
-
-4. Set up your environment:
-   - Create a `.env` file in the project root folder.
-   - Replicate the format shown in the `.env.example` file.
-   - [Get an OpenAI API key](https://www.howtogeek.com/885918/how-to-get-an-openai-api-key/) if you don't have one
-   - Add your OpenAI API key to the `.env` file:
-     ```
-     OPENAI_API_KEY=<YOUR_API_KEY>
-     ```
+3. Configure and save settings:
+   - Make a copy of the `settings.example.toml` file and rename it to `settings.toml`.
+   - Add your GenAI API keys to the `api_key` field of the selected model
+   - Add the name of the model to the `model` field e.g. `mistral-small-latest` or `gpt-4o-mini`
 
 ### Running the Application
 
 1. Start the Streamlit app:
 
 ```bash
-streamlit run app/Home.py
+uv run main.py
 ```
 
+
 2. Upload your files:
    - PDF of signed petitions
    - Voter records file
@@ -131,7 +126,7 @@ streamlit run app/Home.py
 3. Run the following command:
 
 ```bash
-python pytest
+uv run pytest
 ```
 
 ## Project Documentation

app/__init__.py

@@ -9,7 +9,6 @@
 import pandas as pd
 import numpy as np
 from concurrent.futures import ThreadPoolExecutor
-import streamlit as st
 import logging
 from datetime import datetime
 

app/fuzzy_match_helper.py

@@ -0,0 +1,3 @@
+from .ocr_client_factory import extract_from_encoding_async
+
+__all__ = ["extract_from_encoding_async"]

app/ocr/__init__.py

@@ -0,0 +1,117 @@
+from typing import List
+from langchain_openai import ChatOpenAI
+from langchain_mistralai import ChatMistralAI
+from langchain_google_genai import ChatGoogleGenerativeAI
+from langchain_core.runnables import (
+    Runnable,
+)
+from langchain_core.messages import HumanMessage
+from pydantic import BaseModel, Field
+from settings import (
+    load_settings,
+    OpenAiConfig,
+    MistralAiConfig,
+    GeminiAiConfig,
+)
+from utils.app_logger import logger
+import json
+
+
+###
+## OCR FUNCTIONS
+###
+class OCREntry(BaseModel):
+    """Ballot signatory data"""
+
+    Name: str = Field(description="Name of the petition signer")
+    Address: str = Field(description="Address of the petition signatory")
+    Date: str = Field(description="Date of the signed")
+    Ward: int = Field(description="The area or 'Ward' that the signer belongs to")
+
+
+class OCRData(BaseModel):
+    Data: List[OCREntry]
+
+
+def _create_ocr_client() -> Runnable:
+    """
+    Create an OpenAI client with the appropriate settings.
+
+    Returns:
+        Runnable: An AI client for OCR extraction.
+    """
+
+    ocr_config = load_settings().selected_config
+
+    client: Runnable = None
+
+    match ocr_config:
+        case OpenAiConfig():
+            client = ChatOpenAI(
+                api_key=ocr_config.api_key,
+                temperature=0.0,
+                openai_api_base="https://oai.helicone.ai/v1",
+                model=ocr_config.model,
+            ).with_structured_output(OCRData)
+        case MistralAiConfig():
+            client = ChatMistralAI(
+                api_key=ocr_config.api_key,
+                temperature=0.0,
+                model_name=ocr_config.model,
+            ).with_structured_output(OCRData)
+        case GeminiAiConfig():
+            client = ChatGoogleGenerativeAI(
+                api_key=ocr_config.api_key,
+                temperature=0.0,
+                model=ocr_config.model,
+            ).with_structured_output(OCRData)
+
+    logger.debug(f"Creating client {ocr_config}")
+
+    return client
+
+
+async def extract_from_encoding_async(base64_image: str) -> List[dict]:
+    """
+    Extracts names and addresses from single ballot image asynchronously.
+    Uses base64_image
+
+    Args:
+        base64_image: The base64 encoded image to extract data from.
+
+    Returns:
+        list: A list of dictionaries with the OCR data.
+    """
+    logger.debug("Starting OCR extraction for image")
+
+    try:
+        # AI client definition
+        client = _create_ocr_client()
+        # prompt message
+        messages = [
+            {
+                "type": "text",
+                "text": """Using the written text in the image create a list of dictionaries where each dictionary consists of keys 'Name', 'Address', 'Date', and 'Ward'. Fill in the values of each dictionary with the correct entries for each key. Write all the values of the dictionary in full. Only output the list of dictionaries. No other intro text is necessary.""",
+            },
+            {
+                "type": "text",
+                "text": """Remove the city name 'Washington, DC' and any zip codes from the 'Address' values.""",
+            },
+            {
+                "type": "image_url",
+                "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"},
+            },
+        ]
+
+        results = await client.ainvoke([HumanMessage(content=messages)])
+
+        parsed_results = results
+
+        # dictionary results
+        parsed_list = json.loads(parsed_results.json())["Data"]
+        logger.debug(f"Successfully extracted {len(parsed_list)} entries from image")
+        return parsed_list
+
+    except Exception as e:
+        logger.error(f"Error in OCR extraction: {str(e)}")
+        raise