Switch to Whoosh implementation to improve speed

.cursor

@@ -46,29 +46,60 @@ python manage.py migrate                # Apply migrations
 python manage.py collectstatic         # Static files
 ```
 
+### Search Index Management
+The search system uses three indexes: SQLite FTS5, Whoosh (Haystack), and spaCy vector embeddings.
+
+```bash
+# Rebuild all search indexes (2-3 minutes, use for CI/deployment)
+pipenv run python manage.py quickindex
+
+# Lower-level commands:
+pipenv run python manage.py buildindex --v1 --v2  # Build indexes from scratch
+pipenv run python manage.py indexctl pack         # Pack indexes for distribution
+pipenv run python manage.py indexctl unpack       # Unpack indexes after clone
+```
+
+- **quickindex**: Convenience command that runs `buildindex --v1 --v2` then `indexctl pack`
+- Packed indexes stored in `search/indexes/` (committed to repo)
+- After running `quickindex`, commit `search/indexes/` and `search/index_manifest.json`
+- Background re-indexing runs ~60s after server startup (configurable via `SEARCH_INDEX_REFRESH_DELAY`)
+
 ### Fixture Data Loading
 - Fixtures are automatically loaded by quicksetup
 - Data format: JSON files with Django fixture format
 - v2 fixtures in `data/v2/publisher/document/` structure
+- Each document folder contains model-specific JSON files (e.g., `Spell.json`, `SpellCastingOption.json`)
+
+### Spell Data Structure
+- `Spell.json` - Core spell definitions with `higher_level` text field
+- `SpellCastingOption.json` - Structured casting options (slot levels, damage scaling, etc.)
+- **Rule**: Every spell with `higher_level` text must have corresponding entries in `SpellCastingOption.json`
+- Casting option types: `default`, `slot_level_1` through `slot_level_9`, `player_level_5/11/17` for cantrips
 
-## Testing System (APPROVAL TESTS)
+## Testing System
 
-### How Approval Tests Work
-The project uses **approval testing** - tests compare actual API responses against pre-approved JSON files.
+### Test Types
+1. **Approval Tests** - Compare API responses against pre-approved JSON files
+2. **Data Integrity Tests** - Validate relationships and completeness of fixture data
 
-#### Test Structure
+### Test Structure
 - Tests in: `api_v2/tests/test_objects.py`
 - Approved responses: `api_v2/tests/responses/*.approved.json`
 - Failed test responses: `api_v2/tests/responses/*.received.json`
 
-#### Running Tests
+### Running Tests
 ```bash
 # Requires running server for integration tests
-python manage.py runserver 8000 &
-python -m pytest api_v2/tests/test_objects.py -v
+pipenv run python manage.py runserver 8000 &
+pipenv run pytest api_v2/tests/ -v
 pkill -f "python manage.py runserver"  # Stop server
 ```
 
+### Data Integrity Tests (`TestSpellCastingOptions`)
+- `test_all_spells_with_higher_level_have_casting_options` - Validates every spell with `higher_level` text has casting options
+- `test_no_duplicate_casting_option_types` - Ensures no spell has duplicate casting option types
+- **When adding new spells**: If the spell has `higher_level` text, you must also add entries to `SpellCastingOption.json`
+
 #### Updating Test Expectations (IMPORTANT PATTERN)
 When API responses change (like adding new fields):
 
@@ -160,10 +191,13 @@ pkill -f "python manage.py runserver"
 
 ```bash
 # Full development reset
-python manage.py quicksetup --clean --noindex
+pipenv run python manage.py quicksetup --clean --noindex
+
+# Run all tests
+pipenv run pytest api_v2/tests/ -v
 
 # Run specific test
-python -m pytest api_v2/tests/test_objects.py::TestObjects::test_document_example -v
+pipenv run pytest api_v2/tests/test_objects.py::TestObjects::test_document_example -v
 
 # Find fixture files
 find data/v2/ -name "*.json" | grep -i document
@@ -173,6 +207,7 @@ curl -s "http://localhost:8000/v2/documents/srd-2024/" | python -m json.tool
 
 # Update all failing tests at once
 cd api_v2/tests/responses/ && for file in *.received.json; do mv "$file" "${file%.received.json}.approved.json"; done
+
 ```
 
 ## Project Conventions
@@ -185,4 +220,8 @@ cd api_v2/tests/responses/ && for file in *.received.json; do mv "$file" "${file
 ### API Design
 - RESTful endpoints: `/v2/modelname/` and `/v2/modelname/key/`
 - Consistent field naming across models
-- Rich relationship serialization (nested objects, not just IDs)
+- Rich relationship serialization (nested objects, not just IDs)
+
+### Documentation
+- Avoid obvious documentation, only document what is not immediately obvious from names
+- Include swagger spec where appropriate

.gitignore

@@ -19,8 +19,10 @@ db.sqlite3-journal
 # Django static files
 staticfiles/
 
-# Search index
+# Search indexes (unpacked - these are generated from search/indexes/)
 server/whoosh_index/
+whoosh_index/
+server/vector_index.pkl
 
 # IDE and editor configurations
 .vscode/
@@ -32,7 +34,5 @@ api/tests/approved_files/*.recieved.*
 # Generated files
 openapi-schema.yml
 
-server/vector_index.pkl
-
 # Temporary files for magic items update
 temp_magic_items/

Dockerfile

@@ -2,19 +2,24 @@ FROM python:3.11-slim
 
 RUN mkdir -p /opt/services/open5e-api
 WORKDIR /opt/services/open5e-api
-# copy our project code
 
-# install our dependencies
+# Install dependencies
 RUN pip install pipenv gunicorn
+
+# Copy project files
 COPY . /opt/services/open5e-api
 
-RUN pipenv install
+# Install all dependencies from Pipfile
+RUN pipenv install -v
+
+# Download spaCy model for semantic search
+RUN pipenv run python -m spacy download en_core_web_md
 
-# migrate the db, load content, and index it
-RUN pipenv run python manage.py quicksetup
+# Remove .env file (set your env vars via docker-compose.yml or your hosting provider)
+RUN rm -f .env
 
-# remove .env file (set your env vars via docker-compose.yml or your hosting provider)
-RUN rm .env
+# Run setup (SECRET_KEY only needed at build time for collectstatic/migrations)
+RUN SECRET_KEY=insecure-dummy-key pipenv run python manage.py quicksetup
 
-#run gunicorn.
-CMD ["pipenv", "run", "gunicorn","-b", ":8888", "server.wsgi:application"]
+# Run gunicorn
+CMD ["pipenv", "run", "gunicorn", "-b", ":8080", "--workers", "2", "--timeout", "120", "server.wsgi:application"]

Pipfile

@@ -4,7 +4,7 @@ verify_ssl = true
 name = "pypi"
 
 [packages]
-django = "5.*"
+django = "~=5.0"
 djangorestframework = "*"
 django-filter = "*"
 django-cors-headers = "*"
@@ -13,9 +13,10 @@ requests = "*"
 whitenoise = "*"
 gunicorn = "*"
 drf-spectacular = {extras = ["sidecar"], version = "*"}
-numpy = "*"
-rapidfuzz = "*"
-scikit-learn = "*"
+django-haystack = "*"
+Whoosh = "*"
+numpy = "<2.0"
+spacy = ">=3.0"
 
 [dev-packages]
 pytest = "*"