AI Voice Agent (GP Practice)

A voice-enabled customer support agent for a general practitioner (GP) clinic. It transcribes audio to text (Whisper via OpenAI), decides which tool to call and returns the answer as text and optional speech. The project simulates a GP practice with a local SQLite database and document search over clinic policies and FAQs.

How it works

1. Transcription: Optional audio is transcribed into text.
2. Reasoning + tools: A tool-aware LLM decides what information to fetch.
3. Retrieval: Tools query the clinic database or semantic search over clinic documents.
4. Response: The final answer is returned as text and can be spoken with TTS.

The agent is orchestrated with LangGraph (agent/graph.py) and uses a state model defined in agent/state.py.

Available tools

• Document search (Chroma vector search)
  • Practice information (practice_info.md)
  • FAQ (faq.md)
  • Appointment policy (appointment_policy.md)
• Clinic database (SQLite + SQLAlchemy)
  • List services and pricing
  • Look up a service by name
  • Retrieve patient contact details
  • List patient appointments
  • Cancel or reschedule appointments
  • Change appointment doctor or service

Data model (clinic database)

Tables and fields:

• patients: id, first_name, last_name, phone_number, date_of_birth
• doctors: id, first_name, last_name
• services: id, name, duration_minutes, price
• appointments: id, patient_id, doctor_id, service_id, scheduled_at, status

The local database lives in internal_details/clinic.db and is seeded with sample data.

Setup

Prerequisites

• Python 3.11+
• API keys for:
  • OPENAI_API_KEY (transcription, embeddings, and text-to-speech)
  • ANTHROPIC_API_KEY (LLM reasoning)

Create a .env file (do not commit secrets and put the .env in the .gitignore file):

OPENAI_API_KEY=your_openai_key
ANTHROPIC_API_KEY=your_anthropic_key

Install dependencies

If you use uv (recommended when a uv.lock is present):

uv sync

Or with pip:

python -m venv .venv
source .venv/bin/activate
pip install -e .

Seed the database

python -m internal_details.seed_data

(Optional) Rebuild the document vector store

The vector store lives in internal_details/internal_files/chroma_db. To re-embed the markdown files, run from the internal_details/internal_files directory so relative paths resolve:

cd internal_details/internal_files
python embed_file.py

Usage

Text-only example

from agent.main import run

print(run("What are your opening hours?"))

Audio example (advanced)

from agent.graph import app
from agent.state import AgentState

with open("test.wav", "rb") as f:
    audio_bytes = f.read()

state = AgentState(messages=[], audio=audio_bytes)
result = app.invoke(state)
print(result["messages"][-1].content)

On macOS, the demo uses afplay to play the TTS output locally (agent/nodes.py). Replace this with your own playback or streaming logic for production.

Tests

pytest

Project layout

• agent/: LangGraph wiring, node logic, prompts, and tool bindings
• internal_details/: SQLite models, seed data, and embedded document storage
• tests/: Pytest suites and recorded cassettes for deterministic runs