Merge branch 'live' into mbarton/view-old-cohorts-2

Author: mbarton

agents.md

@@ -0,0 +1,178 @@
+# agents.md — RCPCH Audit Engine: Agent Orientation
+
+> This file is intended as a growing reference for AI agents and developers working on this codebase. It will be expanded over time to cover each major area of the project.
+
+---
+
+## Project Overview
+
+**rcpch-audit-engine** is the backend and web application for **Epilepsy12**, a national clinical audit run by the Royal College of Paediatrics and Child Health (RCPCH). It collects, validates, and reports on epilepsy care for children across England and Wales.
+
+- **Framework**: Django (Python)
+- **Database**: PostgreSQL with the PostGIS extension (spatial queries)
+- **Deployment target**: Azure Container Apps
+- **Container registry**: Azure Container Registry (ACR)
+- **Reverse proxy**: Caddy (handles HTTPS)
+- **Task queue**: Celery (celerybeat for scheduled tasks)
+- **Primary app**: `epilepsy12/` — all audit domain logic lives here
+- **Documentation**: MkDocs site, served via a separate Docker Compose service and built into the image at deploy time
+- **Full docs**: https://e12.rcpch.ac.uk/docs
+
+The main Django project config is in `rcpch-audit-engine/` (the inner directory), including `settings.py`, `urls.py`, `logging_settings.py`, and `build_info.py`.
+
+---
+
+## The `s/` Scripts Directory
+
+All developer and CI operations are driven by short shell scripts in `s/`. These exist to reduce typing, reduce errors, and ensure consistency. Scripts are plain bash; make them executable with `chmod +x s/<script>` if needed.
+
+| Script | Purpose |
+|---|---|
+| `s/up` | `docker compose up` — starts all services (caddy, django, postgis, mkdocs) |
+| `s/down` | `docker compose down` — stops services, does **not** destroy volumes or images |
+| `s/rebuild` | Destroys containers and images then calls `s/up` (runs `s/remove-containers-and-images` then `s/up`) |
+| `s/remove-containers-and-images` | Removes local containers and images without touching volumes |
+| `s/DELETE-LOCAL-DATA` | **Destructive** — prompts for confirmation, then runs `docker compose down -v --rmi local` removing volumes too. Never run on live/production. |
+| `s/start-dev` | Django entrypoint for development: `collectstatic`, `migrate`, seed groups/permissions, create dev users, then `runserver` |
+| `s/start-prod` | Django entrypoint for production |
+| `s/start-test` | Django entrypoint used during test runs: `collectstatic` then sleeps (keeps container alive for pytest) |
+| `s/seed` | Seeds 200 cases and registrations into a running django container via `manage.py seed` |
+| `s/test` | Runs `pytest -v` inside the running django container; passes all extra args through (e.g. `-m slow`) |
+| `s/pr-check` | Used in CI on PRs: spins up compose with `start-test`, runs `not slow` then `slow` test markers, tears down |
+| `s/ci` | Full deployment pipeline script (see CI section below) |
+| `s/logs` | Tails all compose service logs with timestamps |
+| `s/psql` | Opens a psql shell inside the postgis container |
+| `s/django-shell` | Opens a Django shell inside the django container |
+| `s/create-superuser` | Creates a Django superuser inside the running container |
+| `s/get-build-info` | Writes git metadata (hash, branch, etc.) to `build_info.json` for the build info page |
+| `s/push-envs-github-secret` | Pushes environment secrets to GitHub Actions secrets |
+| `s/trust-caddy-ca` | Trusts Caddy's local CA certificate for local HTTPS development |
+| `s/restart` | Restarts compose services |
+
+The `DJANGO_START_COMMAND` environment variable controls which start script the django container runs. It defaults to `s/start-dev`; CI overrides it to `s/start-test` when running tests.
+
+---
+
+## CI / Deployment Pipeline
+
+### GitHub Actions Workflows
+
+| Workflow file | Trigger | Purpose |
+|---|---|---|
+| `run-docker-compose-test-on-pr.yml` | PR to any branch | Runs the full pytest suite via `s/pr-check` |
+| `deploy.yml` | Push to `live` branch | Full build, test, and deploy to Azure via `s/ci` |
+| `staging_e12-staging-web-app-service.yml` | (see file) | Staging App Service deployment |
+| `auto-add-issues-to-project.yml` | Issue events | Automatically adds issues to the GitHub Project board |
+
+### The `s/ci` Deployment Script (called by `deploy.yml`)
+
+This is the authoritative deploy sequence executed on every push to `live`:
+
+1. **Login to Azure ACR** — `az acr login`
+2. **Download `.env` from Azure File Share** — production secrets are stored in Azure Storage, not in the repo
+3. **Burn in build info** — `s/get-build-info` writes git metadata to `build_info.json`
+4. **Build the Docker image** — `docker compose build`
+5. **Build the MkDocs documentation** — runs inside the image; docs are embedded into the static files
+6. **Rebuild the image** — a second build to embed the freshly built docs
+7. **Tag and push to ACR** — tagged with the Git SHA: `<registry>.azurecr.io/e12-django:<SHA>`
+8. **Run tests** — `s/test -m 'not slow'` then `s/test -m 'slow'` against a local Postgres container
+9. **Deploy to staging** — `az containerapp revision copy` creates a new revision on the staging Container App
+10. **Deploy to production** — same command targets the live Container App
+
+> Note: the image is pushed to ACR **before** tests run, so that an emergency deploy is possible from a known-good SHA even if tests are mid-flight.
+
+### Authentication to Azure
+
+The GitHub Actions workflow uses OIDC (`id-token: write` permission) with Azure federated credentials — no long-lived secrets for the Azure login itself. Remaining secrets (registry name, resource group, app names, storage account, etc.) are stored as GitHub Actions secrets and injected as environment variables into `s/ci`.
+
+---
+
+## Docker Compose Services
+
+| Service | Image / Build | Role |
+|---|---|---|
+| `caddy` | `caddy` (official) | Reverse proxy, TLS termination, serves static docs |
+| `django` | `e12-django:built` (local build) | Main Django application |
+| `postgis` | `postgis/postgis:15-3.3` | PostgreSQL + PostGIS |
+| `mkdocs` | `e12-django:built` | Builds and optionally serves the MkDocs documentation |
+
+All services share environment from `envs/.env` (not committed to git). Two named volumes are used: `caddy-data` and `postgis-data`.
+
+---
+
+## IMD Calculation — Design Notes
+
+### Background
+
+The **Index of Multiple Deprivation (IMD)** quintile is stored on `Case.index_of_multiple_deprivation_quintile`. The correct IMD year to use depends on the patient's **cohort**:
+
+- Cohort < 8 → 2019 IMD (England 2019 / Wales 2019, based on 2011 LSOA boundaries)
+- Cohort ≥ 8 → 2025 IMD (England 2025, based on 2021 LSOA boundaries; Wales still 2019)
+
+The cohort is stored on `Registration` and is derived from `Registration.first_paediatric_assessment_date`. The RCPCH Census Platform API was updated to **v2**, which now accepts a `year` parameter (`2019` or `2025`) in the IMD endpoint.
+
+### The problem with putting IMD in `Case.save()`
+
+`Case` and `Registration` have a 1-to-1 relation, but they can be created in either order. If IMD is calculated inside `Case.save()`, the Registration (and therefore the cohort) may not exist yet on first save, making it impossible to know the correct year. Workarounds inside `save()` grow complexity and can cause `ValueError` when filtering on an unsaved instance.
+
+### Current design: signal-driven utility
+
+IMD is calculated in a **single utility function** and triggered by **`post_save` signals** on both models:
+
+```
+epilepsy12/general_functions/index_multiple_deprivation.py
+  └── recalculate_imd_for_case(case)
+        - no-op if postcode missing or unknown
+        - no-op if Registration or cohort not yet available
+        - derives imd_year from registration.cohort
+        - calls imd_for_postcode(postcode, year=imd_year) once
+        - persists via queryset .update() to avoid re-triggering Case.save()
+```
+
+```
+epilepsy12/signals.py
+  ├── pre_save / post_save on Case
+  │     → fires recalculate_imd_for_case when postcode changes
+  └── pre_save / post_save on Registration
+        → fires recalculate_imd_for_case when first_paediatric_assessment_date changes
+```
+
+`Case.save()` itself only normalises the postcode (strip spaces/dashes, uppercase) and updates geolocation coordinates (`location_wgs84`, `location_bng`). It sets `index_of_multiple_deprivation_quintile = None` when postcode switches to an unknown/placeholder value.
+
+### Bulk recalculation
+
+To recalculate IMD for existing records (e.g. after a cohort boundary change or API update):
+
+```bash
+s/recalculate-imd          # all cohorts
+s/recalculate-imd 6        # cohort 6 only
+```
+
+This wraps `manage.py recalculate_imd --all` / `--cohort N` (`epilepsy12/management/commands/recalculate_imd.py`).
+
+### Key constants
+
+| Setting | Location |
+|---|---|
+| `RCPCH_CENSUS_PLATFORM_URL` | `settings.py` / `.env` |
+| `RCPCH_CENSUS_PLATFORM_TOKEN` | `settings.py` / `.env` |
+| `UNKNOWN_POSTCODES_NO_SPACES` | `epilepsy12/constants/postcodes.py` |
+
+---
+
+## Areas to Expand
+
+The following sections will be added over time:
+
+- `epilepsy12/` app structure (models, views, forms, KPIs, migrations)
+- `epilepsy12/models_folder/` — domain model breakdown
+- `epilepsy12/views/` — view organisation and HTMX patterns
+- `epilepsy12/constants/` — audit constants and clinical coding
+- `epilepsy12/management/commands/` — custom management commands including `seed`
+- `epilepsy12/tests/` — test structure and pytest markers
+- `epilepsy12/common_view_functions/` — shared view logic
+- KPI calculation logic (`kpi.py`, `organisational_audit.py`)
+- Permissions and decorator patterns
+- Celery / celerybeat scheduled tasks
+- REST API (serializers, DRF)
+- Template and HTMX patterns

documentation/docs/development/imd.md

@@ -13,6 +13,6 @@ Index of multiple deprivation is dependent on geography, and the underlying assu
 
 The country is first broken into areas of similar population size, known as low layer super output areas (LSOAs) and the findings for each measure summarized at this level. The LSOAs are then ranked in order by raw score, with the lower raw scores representing the least deprived. These are then broken into quantiles, depending on the size of the population / reporting priorities. It is typical to report as deciles or quintiles.
 
-The last English data was published in 2019, with Wales the same year.
+The last English data was published in 2019 and updated in 2025. Wales was published in 2019.
 
 Jersey has recently been added to Epilepsy12 and is currently not supported by the RCPCH Census Platform. Because the population is small (~100,000), IMD is reported in vingtaines, rather than quintiles or deciles. The actual data is not published and is being requested for inclusion. In Epilepsy12 currently IMD quantiles are therefore not reported.

documentation/docs/development/postcodes.md

@@ -11,4 +11,17 @@ Postcodes are used primarily to calculate indices of multiple deprivation, but a
 
 Postcodes are passed to `api.rcpch.ac.uk/postcodes`. This is an RCPCH managed instance of the `postcodes.io` and requires an api key. It reports information against postcode which include LSOA (see Indices of Multiple Deprivation) as well as longitude and latitude. These latter data points are used for scatter plots.
 
-Jersey is currently not supported as there is no open source solution for mapping currently though this is tracked in a [github issue](https://github.com/rcpch/rcpch-audit-engine/issues/1107)
+Jersey is currently not supported as there is no open source solution for mapping currently though this is tracked in a [github issue](https://github.com/rcpch/rcpch-audit-engine/issues/1107)
+
+## Postcode and IMD
+
+To calculate an IMD we need the year for England as new data was published in 2025.
+
+If the patient is in cohort 8, this will by default use 2025 data. Earlier cohorts will use 2019.
+
+To recalculate all postcodes in a given cohort use the convenience script:
+
+```console
+s/recalculate-imd        # all cohorts
+s/recalculate-imd 6      # cohort 6 only
+```

epilepsy12/general_functions/index_multiple_deprivation.py

@@ -9,29 +9,85 @@
 # Third party imports
 from django.conf import settings
 
-# RCPCH imports
-
 # Logging setup
 logger = logging.getLogger(__name__)
 
 
-def imd_for_postcode(user_postcode: str) -> int:
+def imd_for_postcode(user_postcode: str, year: int = 2019) -> int | None:
     """
-    Makes an API call to the RCPCH Census Platform with postcode and quantile_type
+    Makes an API call to the RCPCH Census Platform with postcode, quantile_type and IMD year to get the quantile for the given postcode and quantile type.
+    The English IMD have 2019 and 2025 versions, the Welsh only 2019.
     Postcode - can have spaces or not - this is processed by the API
     Quantile - this is an integer representing what quantiles are requested (eg quintile, decile etc)
+    Returns the quantile for the given postcode and quantile type, or None if there was an error
     """
-
+    if year not in [2019, 2025]:
+        logger.error("Invalid year %s for IMD. Must be 2019 or 2025", year)
+        return None
     response = requests.get(
-        url=f"{settings.RCPCH_CENSUS_PLATFORM_URL}/index_of_multiple_deprivation_quantile?postcode={user_postcode}&quantile=5",
+        url=f"{settings.RCPCH_CENSUS_PLATFORM_URL}/index_of_multiple_deprivation_quantile?postcode={user_postcode}&quantile=5&year={year}",
         headers={"Subscription-Key": f"{settings.RCPCH_CENSUS_PLATFORM_TOKEN}"},
         timeout=10,  # times out after 10 seconds
     )
 
     if response.status_code != 200:
         logger.error(
-            "Could not get deprivation score for %s. Response status %s", user_postcode, response.status_code
+            "Could not get deprivation score for %s. Response status %s",
+            user_postcode,
+            response.status_code,
         )
         return None
 
     return response.json()["result"]["data_quantile"]
+
+
+def recalculate_imd_for_case(case) -> None:
+    """
+    Recalculates the IMD quintile for a Case and persists only that field.
+
+    No-op if any of the following are true:
+    - case.postcode is absent or an unknown/placeholder postcode
+    - case has no Registration yet, or Registration.cohort is not yet set
+
+    Uses queryset .update() rather than case.save() to avoid re-triggering
+    Case.post_save signals and coordinate lookups.
+    Also updates the in-memory instance so the caller sees the new value.
+    """
+    from ..constants import UNKNOWN_POSTCODES_NO_SPACES
+
+    postcode = case.postcode
+    if not postcode:
+        return
+
+    normalised = postcode.replace(" ", "").replace("-", "").upper()
+    if normalised in UNKNOWN_POSTCODES_NO_SPACES:
+        return
+
+    try:
+        registration = case.registration
+    except Exception:
+        # No registration exists yet — IMD will be recalculated once Registration is saved.
+        return
+
+    if registration is None or registration.cohort is None:
+        return
+
+    imd_year = 2025 if registration.cohort >= 8 else 2019
+
+    try:
+        quintile = imd_for_postcode(normalised, year=imd_year)
+    except Exception as error:
+        logger.exception(
+            "Cannot calculate deprivation score for %s: %s", normalised, error
+        )
+        quintile = None
+
+    # Use queryset update to skip Case.save() and its signals entirely.
+    from django.apps import apps
+
+    Case = apps.get_model("epilepsy12", "Case")
+    Case.objects.filter(pk=case.pk).update(
+        index_of_multiple_deprivation_quintile=quintile
+    )
+    # Keep the in-memory object consistent.
+    case.index_of_multiple_deprivation_quintile = quintile

epilepsy12/management/commands/recalculate_imd.py

@@ -0,0 +1,97 @@
+"""
+Management command to recalculate the Index of Multiple Deprivation (IMD) quintile
+for all Cases in a given cohort (or all cohorts).
+
+Usage:
+    # All cases across all cohorts:
+    python manage.py recalculate_imd --all
+
+    # All cases in cohort 6:
+    python manage.py recalculate_imd --cohort 6
+
+Or via the convenience script:
+    s/recalculate-imd        # all cohorts
+    s/recalculate-imd 6      # cohort 6 only
+"""
+
+# Standard imports
+import logging
+
+# Django imports
+from django.core.management.base import BaseCommand, CommandError
+
+# RCPCH imports
+from epilepsy12.models import Case
+from epilepsy12.general_functions.index_multiple_deprivation import (
+    recalculate_imd_for_case,
+)
+
+logger = logging.getLogger(__name__)
+
+
+class Command(BaseCommand):
+    help = "Recalculate IMD quintile for all Cases in a given cohort (or all cohorts)."
+
+    def add_arguments(self, parser):
+        group = parser.add_mutually_exclusive_group(required=True)
+        group.add_argument(
+            "--cohort",
+            type=int,
+            metavar="N",
+            help="Recalculate only for cases in this cohort number.",
+        )
+        group.add_argument(
+            "--all",
+            action="store_true",
+            dest="all_cohorts",
+            help="Recalculate for all cases regardless of cohort.",
+        )
+
+    def handle(self, *args, **options):
+        if options["all_cohorts"]:
+            cases = Case.objects.select_related("registration").all()
+            self.stdout.write("Recalculating IMD for all cases…")
+        else:
+            cohort = options["cohort"]
+            cases = Case.objects.select_related("registration").filter(
+                registration__cohort=cohort
+            )
+            self.stdout.write(f"Recalculating IMD for cohort {cohort}…")
+
+        total = cases.count()
+        if total == 0:
+            self.stdout.write(self.style.WARNING("No matching cases found."))
+            return
+
+        success = 0
+        skipped = 0
+        errors = 0
+
+        for ix, case in enumerate(cases, 1):
+            try:
+                before = case.index_of_multiple_deprivation_quintile
+                recalculate_imd_for_case(case)
+                after = case.index_of_multiple_deprivation_quintile
+
+                if after is None and not case.postcode:
+                    skipped += 1
+                    self.stdout.write(
+                        f"  [{ix}/{total}] Case {case.pk}: skipped (no postcode)"
+                    )
+                else:
+                    success += 1
+                    self.stdout.write(
+                        f"  [{ix}/{total}] Case {case.pk}: {before} → {after}"
+                    )
+            except Exception as exc:
+                errors += 1
+                self.stderr.write(
+                    self.style.ERROR(f"  [{ix}/{total}] Case {case.pk}: ERROR — {exc}")
+                )
+                logger.exception("recalculate_imd failed for case %s", case.pk)
+
+        self.stdout.write(
+            self.style.SUCCESS(
+                f"\nDone. {success} updated, {skipped} skipped, {errors} errors (total {total})."
+            )
+        )