Document base-only recovery and track overlay build scripts

Author: iandorsey00

.gitignore

@@ -145,6 +145,8 @@ bin/
 scripts/*
 !scripts/
 !scripts/fetch_latest_acs.py
+!scripts/fetch_overlays.py
+!scripts/build_nibrs_crime_overlay.py
 
 # Local overlay inputs (crime/project data)
 /overlays/*

HANDOFF.md

@@ -4,74 +4,59 @@
 - Project: `geocompare`
 - Handoff date: 2026-03-08
 - Branch: `master`
-- Version: `0.6.2`
-- Last completed commit at handoff prep: `9d11bd7` (`Bump version to 0.6.2`)
+- Version: `0.6.9`
 
-## Purpose
-GeoCompare builds and queries local demographic/geographic data products (Gazetteer + ACS-derived inputs) and supports ranking, distance, nearest geographies, and profile exports.
+## Project Scope
+GeoCompare builds and queries local demographic data products from ACS/Gazetteer
+inputs and optional overlays.
 
-## Environment
-- Python: 3.9+
-- Package/deps: managed via `pyproject.toml`
-- Common setup:
-  - `python3 -m pip install -e ".[dev]"`
+Core/base overlays:
+- Crime overlay (`CRIME`)
+- Voter registration overlay (`VOTER REGISTRATION`)
 
-## Canonical CLI (Current)
-Use canonical commands and flags only.
+Optional/custom overlays:
+- User/private metrics, typically in `project_data.csv` (`PROJECT DATA` or
+  manifest-defined section)
 
-- Build data products:
+## Current Data/Overlay Model
+- Build command:
   - `python3 -m geocompare.interfaces.cli build <data_path>`
-- Search:
-  - `python3 -m geocompare.interfaces.cli query search "san francisco"`
-- Profile:
-  - `python3 -m geocompare.interfaces.cli query profile "San Francisco city, California"`
-- Top/bottom with modern filter and scope:
-  - `python3 -m geocompare.interfaces.cli query top median_year_structure_built --where 'population>=100000' --universe places --in-state ca`
-  - `python3 -m geocompare.interfaces.cli query bottom median_household_income --where 'population>=50000' --scope places+ca`
-- Nearest:
-  - `python3 -m geocompare.interfaces.cli query nearest "San Francisco city, California" --where 'population>=100000' --universe places --in-state ca -n 10`
-- Resolve:
-  - `python3 -m geocompare.interfaces.cli resolve "San Francisco, CA" --state ca -n 5`
-- Export rows:
-  - `python3 -m geocompare.interfaces.cli export rows ":population :income" --where 'population>=100000' --universe places --in-state ca`
-
-## Important Interface Decisions
-- Legacy CLI aliases were purged.
-  - Removed command aliases like `hv`, `lv`, `cg`, `dist`, `dp`, etc.
-  - Removed legacy flag names `--geofilter` and `--context`.
-- Canonical query flags are:
-  - Filter: `--where` (short: `-w`)
-  - Scope string: `--scope` (short: `-s`)
-  - Explicit scope composition: `--universe`, plus one of `--in-state|--in-county|--in-zcta`
-- Legacy tool shims were removed.
-  - Deleted: `CountyTools.py`, `StateTools.py`, `KeyTools.py`, `SummaryLevelTools.py`
-  - Canonical modules: `county_lookup.py`, `state_lookup.py`, `county_key_index.py`, `summary_level_parser.py`
-
-## Shell Caveat
-- In `zsh`, quote or escape filter expressions that contain `>` or `<`.
-  - Good: `--where 'population>=100000'`
-  - Good: `--where population\>=100000`
-
-## Data Expectations
-- Build command expects source files under the provided `<data_path>`.
-- The project has support for recent ACS table-based inputs and latest Gazetteer-era ingestion logic integrated in prior updates.
-- If refreshing download logic, verify source year discovery against the files present in `<data_path>`.
-
-## Validation / Definition of Done
-Before ACP:
-1. `ruff check geocompare/interfaces/cli.py geocompare/engine.py geocompare/tools tests`
-2. `python3 -m pytest -q`
-3. Run at least one smoke query:
-   - `python3 -m geocompare.interfaces.cli --version`
-   - `python3 -m geocompare.interfaces.cli query search "san francisco" -n 3`
-   - `python3 -m geocompare.interfaces.cli query top median_year_structure_built --where 'population>=100000' --universe places --in-state ca -n 5`
-
-## ACP Workflow
-- Stage only intended files.
-- Commit message should describe functional change precisely.
-- Push to `origin/master` unless directed otherwise.
-
-## Immediate Backlog Suggestions
-1. Add CLI integration tests that invoke argument parsing paths for `--where`, `--scope`, and explicit scope flags.
-2. Document canonical CLI examples in `README.md` and remove any remaining historical examples.
-3. Optionally add `--where` parser help examples for compound (`:c`/`:cc`) usage.
+- Base inputs are discovered from files under `<data_path>`.
+- Overlay inputs are discovered under `<data_path>/overlays`.
+- Optional manifest support:
+  - `overlay_manifest.json` (or `manifest.json`) in overlays directory.
+  - Supports per-metric metadata (`key`, `label`, `section`, `type`, `order`).
+- Overlay section placement:
+  - Base profile sections first.
+  - Overlay sections appended at bottom.
+  - Overlay rows deterministically ordered.
+
+## Base-Only Recovery (No Custom Overlay)
+To restore a clean base state without private project overlay:
+
+1. Fetch core data:
+   - `python3 scripts/fetch_latest_acs.py --out-dir <data_path> --archive-existing`
+2. Optionally build canonical base overlays:
+   - `python3 scripts/fetch_overlays.py --out-dir <data_path> --crime-source <src> --voter-source <src>`
+3. Ensure custom overlay artifacts are absent:
+   - remove/relocate `<data_path>/overlays/project_data.csv`
+   - remove/relocate `<data_path>/overlays/overlay_manifest.json`
+4. Rebuild:
+   - `python3 -m geocompare.interfaces.cli build <data_path>`
+
+## Tracked Scripts
+- `scripts/fetch_latest_acs.py`
+- `scripts/fetch_overlays.py`
+- `scripts/build_nibrs_crime_overlay.py`
+
+## Validation
+Recommended checks before ACP:
+
+1. `ruff check tests geocompare/identity geocompare/repository/sqlite_repository.py geocompare/interfaces/cli.py scripts/fetch_overlays.py`
+2. `black --check tests geocompare/identity geocompare/repository/sqlite_repository.py geocompare/interfaces/cli.py scripts/fetch_overlays.py`
+3. `mypy geocompare/identity geocompare/repository/sqlite_repository.py geocompare/interfaces/cli.py`
+4. `PYTHONPATH=. pytest -q`
+
+## License
+Repository license is MIT (`LICENSE`). This remains appropriate for the base
+project.

README.md

@@ -114,6 +114,44 @@ Recommended metadata file (`overlay_manifest.json`) in your overlay repo:
 `overlay_manifest.json` is optional today, but recommended for stable naming,
 labels, and section placement across overlay builds.
 
+## Base-Only Rebuild (No Custom Overlay)
+
+Use this path to return to a clean, shareable base geocompare state:
+
+1. Fetch ACS + Gazetteer files:
+
+```bash
+python3 scripts/fetch_latest_acs.py --out-dir /path/to/data --archive-existing
+```
+
+2. (Optional) refresh canonical built-in overlays:
+
+```bash
+python3 scripts/fetch_overlays.py \
+  --out-dir /path/to/data \
+  --crime-source /path/or/url/to/crime.csv \
+  --voter-source /path/or/url/to/voter.csv
+```
+
+3. Ensure no private overlay file is present:
+
+- Remove or relocate `/path/to/data/overlays/project_data.csv`
+- Remove or relocate `/path/to/data/overlays/overlay_manifest.json`
+
+4. Build:
+
+```bash
+python3 -m geocompare.interfaces.cli build /path/to/data
+```
+
+## Repository Scripts
+
+Tracked operational scripts:
+
+- `scripts/fetch_latest_acs.py`: download/update ACS + Gazetteer inputs.
+- `scripts/fetch_overlays.py`: normalize built-in crime/voter overlays.
+- `scripts/build_nibrs_crime_overlay.py`: build base crime overlay from NIBRS inputs.
+
 Query workflows:
 
 ```bash