hardbyte
diff --git a/‎CLAUDE.md‎
Lines changed: 57 additions & 164 deletions b/‎CLAUDE.md‎
Lines changed: 57 additions & 164 deletions
diff --git a/‎README.md‎
Lines changed: 42 additions & 0 deletions b/‎README.md‎
Lines changed: 42 additions & 0 deletions
@@ -6,210 +6,103 @@
 - **REST API Consistency**: DELETE operations should return appropriate HTTP status codes per REST conventions (typically 204 No Content for successful deletions with no response body, or 200 OK if returning meaningful response data).
 - **Declarative Database Infrastructure**: Use `alembic_utils` with Python-defined functions and triggers in `app/db/functions.py` and `app/db/triggers.py` for version-controlled, type-safe database logic.
 
-Only Professional Comments Should Be used.
+## Code Style
+
+Only professional comments should be used:
 - Remove task-focused comments like "OLD import removed"
 - Remove "NEW:" markers etc.
 - Focus on why not what
 - Remove comments that just restate the code
 
+Ruff for linting (configured in `pyproject.toml`).
+
 ## Development Commands
 
-### Dependencies
-- **Install dependencies**: `poetry install`
-- **Add new dependency**: `poetry add <package_name>`
-- **Update dependencies**: `poetry update`
+See [README.md](README.md) for full setup, testing, migration, and deployment instructions.
 
-### Database Operations
-- **Apply migrations**: `poetry run alembic upgrade head`
-- **Create new migration**: `poetry run alembic revision --autogenerate -m "Description"`
-- **Set database connection**: `export SQLALCHEMY_DATABASE_URI=postgresql://postgres:password@localhost/postgres`
+Quick reference:
 
-### Testing
-- **Run all tests**: `bash scripts/start-tests.sh` or `poetry run pytest -v app/tests`
-- **Run integration tests in Docker**: `bash scripts/integration-tests.sh` (recommended - provides proper environment)
-- **Run integration tests locally**: Direct pytest may have async fixture issues, use Docker instead
-- **Single test**: `poetry run pytest -v app/tests/integration/test_specific.py::test_function`
-
-**Important Note**: Integration tests should be run using `bash scripts/integration-tests.sh` which provides the proper Docker environment with database migrations and all dependencies. Running integration tests directly with pytest may encounter async fixture configuration issues. Ensure no conflicting postgres containers are running on port 5432.
-
-### Code Quality
-- **Lint code**: `poetry run ruff check`
-- **Fix linting issues**: `poetry run ruff check --fix`
-- **Pre-commit hooks**: `poetry run pre-commit run --all-files`
-
-### Local Development
-- **Start with Docker Compose**: `docker compose up -d`
-- **Run API directly**: `uvicorn app.main:app --reload`
-- **Run internal API**: `gunicorn --workers=1 --worker-class=uvicorn.workers.UvicornWorker app.internal_api:internal_app`
-
-### Seed Test Data (Admin UI / Chatflows)
-Use the declarative fixture + seeder to create a consistent school, users, books, CMS content, and flows.
-
-**Config**: `scripts/fixtures/admin-ui-seed.json`  
-**Seeder**: `scripts/seed_admin_ui_data.py`
-
-```bash
-# Seed data and print JWTs for each user role
-docker compose run --rm --entrypoint python \
-  -v "$PWD/scripts:/app/scripts" \
-  api /app/scripts/seed_admin_ui_data.py --emit-tokens --tokens-format json
-```
+| Task | Command |
+|------|---------|
+| Install dependencies | `poetry install` |
+| Start stack | `docker compose up -d --build` |
+| Apply migrations | `docker compose run --rm migration` |
+| Seed data | `docker compose run --rm --entrypoint python -v "$PWD/scripts:/app/scripts" api /app/scripts/seed_admin_ui_data.py --emit-tokens --tokens-format json` |
+| Unit tests | `poetry run pytest app/tests/unit/ -v` |
+| Integration tests | `bash scripts/integration-tests.sh` |
+| Single test | `poetry run pytest -v app/tests/integration/test_specific.py::test_function` |
+| Lint | `poetry run ruff check` |
+| Lint fix | `poetry run ruff check --fix` |
+| Run API directly | `uvicorn app.main:app --reload` |
+| Run internal API | `gunicorn --workers=1 --worker-class=uvicorn.workers.UvicornWorker app.internal_api:internal_app` |
+
+**Important**: Integration tests should be run using `bash scripts/integration-tests.sh` which provides the proper Docker environment. Running integration tests directly with pytest may encounter async fixture issues. Ensure no conflicting postgres containers are running on port 5432.
 
 ### Configuring Local User Permissions
-To grant admin access for testing features like the CMS/chatflow builder in the admin UI:
+
+To grant admin access for testing the CMS/chatflow builder in the admin UI:
 
 ```sql
--- Update user type to WRIVETED admin
 UPDATE users SET type = 'WRIVETED' WHERE email = 'your-email@example.com';
 
--- Create corresponding wriveted_admins record (required for joined-table inheritance)
 INSERT INTO wriveted_admins (id)
 SELECT id FROM users WHERE email = 'your-email@example.com';
 ```
 
-Or via Python:
-```python
-export SQLALCHEMY_DATABASE_URI=postgresql://postgres:password@localhost/postgres
-poetry run python -c "
-from sqlalchemy import create_engine, text
-engine = create_engine('postgresql://postgres:password@localhost/postgres')
-with engine.begin() as conn:
-    result = conn.execute(text(\"SELECT id FROM users WHERE email = 'your-email@example.com'\"))
-    user_id = result.fetchone()[0]
-    conn.execute(text(f\"UPDATE users SET type = 'WRIVETED' WHERE id = '{user_id}'\"))
-    conn.execute(text(f\"INSERT INTO wriveted_admins (id) VALUES ('{user_id}')\"))
-"
-```
-
-After updating, log out and back in to the admin UI to get a new JWT with updated permissions.
-
-## Declarative Database Pattern
-
-### Overview
-All PostgreSQL functions, triggers, and complex database objects are defined declaratively in Python using `alembic_utils`. This ensures version control, type safety, and maintainability.
-
-### Key Files
-- **`app/db/functions.py`**: PostgreSQL function definitions
-- **`app/db/triggers.py`**: Trigger definitions that reference functions
-- **Migrations**: Use `op.create_entity()` and `op.drop_entity()` for declarative objects
-
-### Example Pattern
-```python
-# In app/db/functions.py
-from alembic_utils.pg_function import PGFunction
-
-my_function = PGFunction(
-    schema="public",
-    signature="my_function_name()",
-    definition="returns trigger LANGUAGE plpgsql AS $$ ... $$"
-)
-
-# In app/db/triggers.py
-from alembic_utils.pg_trigger import PGTrigger
-from app.db.functions import my_function
-
-my_trigger = PGTrigger(
-    schema="public",
-    signature="trg_my_trigger",
-    on_entity="public.my_table",
-    definition=f"... EXECUTE FUNCTION {my_function.signature}"
-)
-
-# In migration
-def upgrade():
-    op.create_entity(my_function)
-    op.create_entity(my_trigger)
-```
-
-### Benefits
-- **Single Source of Truth**: Database logic defined once in Python
-- **Version Control**: All changes tracked in git
-- **Type Safety**: Python validates syntax before deployment
-- **Migration Safety**: Automatic up/down migration generation
-- **IDE Support**: Full Python tooling available
+After updating, log out and back in to get a new JWT with updated permissions.
 
 ## Architecture Overview
 
 ### Dual API Structure
-The application consists of two separate FastAPI applications:
 - **Public API** (`app.main:app`): External-facing REST API with authentication/authorization
 - **Internal API** (`app.internal_api:internal_app`): Background task processing, webhook handling
 
-### Database Architecture
+### Database
 - **ORM**: SQLAlchemy 2.0 with async support (asyncpg driver)
 - **Migrations**: Alembic for schema management
 - **Base Class**: Custom `Base` class with auto-generated table names
 - **User Model**: Uses joined-table inheritance for different user types (Student, Educator, Parent, etc.)
+- **Connection**: Always use `SQLALCHEMY_DATABASE_URI` environment variable
 
-### Key Domain Models
-- **Users**: Hierarchical user system (User → Student/Educator/Parent/etc.)
-- **Books**: Work → Edition → CollectionItem relationship
-- **Schools**: Schools contain ClassGroups and Users
-- **Collections**: Library collections with items and activity tracking
-- **Labels**: AI-powered book categorization system via LabelSets
-
-### Authentication & Authorization
-- **Firebase Authentication**: Users authenticate via Firebase, exchange token for Wriveted JWT
-- **RBAC**: Role-based access control with principals (user-xyz, school-1)
-- **Service Accounts**: Long-lived tokens for LMS integrations
-
-### Configuration
-- **Settings**: Pydantic-based configuration in `app.config.py`
-- **Environment Variables**: Database connection, API keys, feature flags
-- **GCP Integration**: Cloud SQL, Cloud Storage, Cloud Tasks
-
-### API Structure
-- **External API**: Routes in `app/api/` with dependencies in `app/api/dependencies/`
-- **Schemas**: Pydantic models for request/response in `app/schemas/`
-- **CRUD Operations**: Database operations in `app/crud/` (legacy pattern)
-- **Repositories**: Domain-focused data access interfaces in `app/repositories/`
+### Code Organization
+- **Routes**: `app/api/` with dependencies in `app/api/dependencies/`
+- **Schemas**: Pydantic request/response models in `app/schemas/`
+- **Repositories**: Domain-focused data access in `app/repositories/` (modern pattern)
+- **CRUD**: Legacy data access in `app/crud/` (being phased out)
 - **Services**: Business logic in `app/services/`
+- **Configuration**: Pydantic-based settings in `app/config.py`
 
-
-## Development Notes
-
-### Database Connection Patterns
-- Always use environment variable `SQLALCHEMY_DATABASE_URI` for connections
-- Local development: `postgresql://postgres:password@localhost/postgres`
-
-### Testing Environment
-- Integration tests use Docker Compose with real PostgreSQL
-- Tests are in `app/tests/integration/` and `app/tests/unit/`
-- Test configuration in `conftest.py` files
-
-### Code Style
-- Ruff for linting (configured in `pyproject.toml`)
+See [docs/architecture-service-layer.md](docs/architecture-service-layer.md) for the full service layer architecture.
 
 ### Migration Workflow
 1. Modify SQLAlchemy models in `app/models/`
 2. Add imports to `app/models/__init__.py`
 3. Generate migration: `poetry run alembic revision --autogenerate -m "Description"`
-4. Review generated migration file manually. Try ensure models are source of truth.
+4. Review generated migration file manually. Models are source of truth.
 5. Apply: `poetry run alembic upgrade head`
 
-## Integration Test Insights & Patterns
+## Common Patterns and Pitfalls
 
-### Data Access Patterns  
-- **Legacy CRUD Pattern**: Some classes still use generic `CRUDBase` (being phased out)
-- **Modern Repository Pattern**: New domain-focused repositories in `app/repositories/`
-- **Service Layer**: Business logic extracted from data access layer to `app/services/`
-- **Async Operations**: All new services use proper async/await patterns consistently
-- **Field Validation**: Pydantic schemas use consistent field names between database and API
+### Data Access
+- **Repository pattern** is the modern approach; `app/crud/` is legacy (being phased out)
+- All new services should use proper async/await patterns
+- Pydantic schemas use consistent field names between database and API
 
-### API Endpoint Patterns
-- **Authentication**: Many endpoints require service account or admin authentication
-- **Validation Endpoints**: Custom endpoints like `/flows/{id}/validate` for business logic validation
-- **Query Parameters**: Support filtering, searching, pagination consistently across list endpoints
+### API Endpoints
+- Many endpoints require service account or admin authentication
+- Custom validation endpoints like `/flows/{id}/validate` for business logic
+- List endpoints support filtering, searching, and pagination consistently
 
-### Testing Best Practices
-- **Docker Environment**: Always use `bash scripts/integration-tests.sh` for proper database setup
-- **Test Isolation**: Clean up test data in fixtures to prevent test interference
-- **Async Context**: Be careful with SQLAlchemy async operations
-
-### Performance Considerations
-- **Bulk Operations**: Implement batch create/update operations for efficiency
-- **Query Optimization**: Full-text search uses PostgreSQL tsvector and GIN indexes
-
-### Common Pitfalls
-- **Status Code Expectations**: Verify actual API behavior vs REST conventions
-- **Async/Await Consistency**: Ensure all database operations use proper async patterns
+### Testing
+- Always use `bash scripts/integration-tests.sh` for integration tests (proper Docker environment)
+- Clean up test data in fixtures to prevent test interference
+- Be careful with SQLAlchemy async session management in tests
+- Test configuration lives in `conftest.py` files
+
+### Performance
+- Bulk operations: use batch create/update for efficiency
+- Full-text search uses PostgreSQL tsvector and GIN indexes
+
+### REST Conventions
+- Verify actual API behavior vs REST conventions for status codes
+- Ensure async/await consistency across all database operations
@@ -66,6 +66,29 @@ The public API is available at `http://localhost:8000`. The seed script prints J
 
 > **Note:** The `api` service volume-mounts `./app` so code changes are live without rebuild. The `scripts/` directory is _not_ mounted by default -- the seed command above uses `-v` to mount it explicitly.
 
+### Running without Docker
+
+```bash
+# Public API
+uvicorn app.main:app --reload
+
+# Internal API
+gunicorn --workers=1 --worker-class=uvicorn.workers.UvicornWorker app.internal_api:internal_app
+```
+
+### Configuring local admin access
+
+To grant admin access for testing the CMS/chatflow builder in the admin UI:
+
+```sql
+UPDATE users SET type = 'WRIVETED' WHERE email = 'your-email@example.com';
+
+INSERT INTO wriveted_admins (id)
+SELECT id FROM users WHERE email = 'your-email@example.com';
+```
+
+After updating, log out and back in to get a new JWT with updated permissions.
+
 ## Chatflow runtime
 
 The chat runtime (`app/services/chat_runtime.py`) drives Huey's interactive reading-preference conversations. Flows are directed graphs of nodes (messages, questions, actions, conditions) defined in the admin UI and stored as JSON.
@@ -90,7 +113,10 @@ CMS content (questions, messages, jokes, facts) is managed via the API and surfa
 
 ### Unit tests (no database required)
 
+Unit tests require several environment variables to be set. Use the helper script:
+
 ```bash
+source scripts/setup-test-env.sh
 poetry run pytest app/tests/unit/ -v
 ```
 
@@ -102,8 +128,22 @@ The recommended way to run integration tests -- provides a proper environment wi
 bash scripts/integration-tests.sh
 ```
 
+If you don't have GCR credentials (e.g. first-time setup), skip remote Docker cache pulls:
+
+```bash
+LOCAL_BUILD_ONLY=1 bash scripts/integration-tests.sh
+```
+
 Ensure no conflicting PostgreSQL containers are running on port 5432.
 
+### Isolated tests
+
+Some tests require isolation from other tests (e.g. connection pool stress tests). These are marked with `@pytest.mark.isolated` and skipped during normal test runs. CI runs them separately:
+
+```bash
+bash scripts/integration-tests.sh --run-isolated-tests
+```
+
 ### Single test
 
 ```bash
@@ -118,6 +158,8 @@ Requires a running Docker stack with seeded data:
 python scripts/test_huey_flow_e2e.py
 ```
 
+See [docs/testing-credentials.md](docs/testing-credentials.md) for test data setup and authentication tokens.
+
 ## Database migrations
 
 Uses [Alembic](https://alembic.sqlalchemy.org/) with SQLAlchemy 2.0 models. PostgreSQL functions and triggers are defined declaratively in Python using `alembic_utils` (`app/db/functions.py`, `app/db/triggers.py`).