Support for Diavgeia decisions

.env.example

@@ -57,6 +57,20 @@ DO_SPACES_SECRET=your_do_spaces_secret
 DO_SPACES_BUCKET=your_do_spaces_bucket
 CDN_URL=https://data.opencouncil.gr
 
+# ---------------------------------
+# Cron Jobs
+# ---------------------------------
+# Secret for authenticating cron endpoints (e.g., /api/cron/poll-decisions)
+# Generate with: openssl rand -base64 32
+CRON_SECRET=
+
+# ---------------------------------
+# Preview Server (for --preview-db flag)
+# ---------------------------------
+# SSH target for connecting to the preview server's database.
+# Only needed when using: nix run .#dev -- --preview-db=N
+# OC_PREVIEW_SSH=root@159.89.98.26
+
 # ---------------------------------
 # Optional Variables
 # ---------------------------------

.gitignore

@@ -10,6 +10,7 @@
 /coverage
 /og-previews
 /prisma/seed_data.json
+/prisma/diavgeia_municipalities.json
 
 # next.js
 /.next/

CLAUDE.md

@@ -35,6 +35,12 @@ If you open an interactive shell session first (`nix develop`), subsequent comma
 - `npm run prisma:migrate:reset` - Reset database and re-run migrations
 - `npx prisma db seed` - Seed database with sample data
 
+**IMPORTANT**: When making schema changes, always use `--create-only` to generate the migration file without applying it:
+```
+npx prisma migrate dev --name <migration_name> --create-only
+```
+This allows testing the migration against a local database first before applying to production. Never run `npx prisma migrate dev` directly, as it both creates and applies the migration to whatever database `DATABASE_URL` points to.
+
 ### Utility Scripts
 - `npm run lint` - Run ESLint
 - `npm run email` - Test municipality email sending
@@ -90,6 +96,12 @@ src/
 - Re-export from `src/lib/db/types/index.ts`
 - Import from `@/lib/db/types` to prevent circular dependencies
 
+**CRITICAL - Before Creating New Types**:
+- **Always check if the type already exists** before defining a new one
+- When a function returns a type you need, follow the import chain to find its definition
+- If the type exists but isn't exported, export it rather than duplicating
+- Example: If `getCouncilMeeting()` returns `CouncilMeetingWithAdminBody`, check `src/lib/db/meetings.ts` for that type definition before creating your own
+
 ### Authentication Patterns
 
 **Always use methods from `src/lib/auth.ts`**:
@@ -205,6 +217,24 @@ The condition must use `process.env.NODE_ENV === 'development'` directly (not vi
 3. Extract duplicates to shared utilities
 4. Ensure all imports are at the top of files
 
+### Git Workflow
+
+**Commit organization on feature branches**:
+When working on a branch with existing commits, new changes must be categorized:
+- **Fixup commits** (`fixup! <original commit message>`): Changes that modify, improve, or clean up code introduced by an existing commit on the branch. Use the `fixup!` prefix with the exact original commit message so `git rebase -i --autosquash` can squash them automatically.
+- **New commits**: Genuinely new functionality that doesn't belong to any existing commit.
+
+Before creating commits, run `git log --oneline main..HEAD` to understand the branch's commit structure and decide which changes are fixups vs new commits. Stage files selectively to keep each commit focused.
+
+After all commits are created, run `GIT_SEQUENCE_EDITOR=true git rebase -i --autosquash main` to fold fixup commits into their targets.
+
+**Build Verification**:
+- **Always run `npm run build` after completing changes** to verify TypeScript compiles and catch errors early
+- This closes the feedback loop quickly - don't wait for the user to discover build failures
+- For schema changes: run `npm run prisma:generate` before building
+- Quick TypeScript check without full build: `npx tsc --noEmit`
+- **Full stack verification**: Run `nix run .#dev` to verify the app starts with a fresh local DB (includes running migrations)
+
 ### TypeScript
 - Strict mode is enabled
 - Use interfaces/types for data structures

README.md

@@ -28,6 +28,7 @@ OpenCouncil is developed by [Schema Labs](https://schemalabs.gr), a non-profit o
 - 📊 **Subject Analysis**: Automatic categorization of discussion subjects
 - 📢 **Notification System**: Personalized updates for citizens
 - 🎥 [**Meeting Highlights**](./docs/guides/meeting-highlights.md): Create and share custom video clips from council meeting moments, with automatic generation and editing capabilities
+- 📜 **Diavgeia Integration**: Link council decisions with their official publications on [Diavgeia](https://diavgeia.gov.gr/), Greece's government transparency portal
 - 🌐 **Open Data**: All data available through a public API
 - 🔐 **Role-Based Access**: Granular permissions for different user types
 - 🤖 **AI Chat Assistant**: Ask questions about council meetings

docs/environment-variables.md

@@ -132,6 +132,7 @@ Copy the output and set it as your `NEXTAUTH_SECRET` in your `.env` file.
 |----------|-------------|----------|---------|
 | `TASK_API_URL` | URL for the background task processing API. | Yes | - |
 | `TASK_API_KEY` | API key for task processing API. | Yes | - |
+| `CRON_SECRET` | Bearer token for authenticating cron job endpoints (e.g., `/api/cron/poll-decisions`). Generate with `openssl rand -base64 32`. | No | - |
 
 ### Google Calendar Integration
 | Variable | Description | Required | Default |

docs/guides/preview-deployments.md

@@ -356,6 +356,20 @@ ls -la /var/lib/opencouncil-previews/pr-<num>/postgres/
 
 ## Troubleshooting
 
+**Amended migration files (schema mismatch after force-push):**
+
+`prisma migrate deploy` tracks applied migrations by name. If you amend a migration file that was already applied (e.g., rename a column from `diavgeiaUnitId` to `diavgeiaUnitIds`), the isolated DB keeps the old schema — Prisma sees the migration name as already applied and skips it. Symptoms: `The column X does not exist in the current database` errors at runtime.
+
+Fix: destroy and recreate the preview to get a fresh DB with the updated migration:
+```bash
+ssh root@<droplet-ip>
+STORE_PATH=$(readlink /var/lib/opencouncil-previews/pr-<num>/app)
+sudo opencouncil-preview-destroy <num>
+sudo opencouncil-preview-create <num> "$STORE_PATH" --with-db
+```
+
+To avoid this, prefer creating additive migrations instead of amending existing ones. If you must amend, remember to reset the preview DB afterwards.
+
 **Preview not accessible:**
 1. DNS: `dig pr-123.preview.opencouncil.gr` should resolve to droplet IP
 2. Caddy: `systemctl status caddy` + check `/etc/caddy/conf.d/pr-123.conf` exists

docs/nix-usage.md

@@ -93,6 +93,48 @@ When `TASK_API_URL` is configured in `.env`, the dev runner will check if the ta
    ⚠ Task server not reachable (start it separately for E2E testing)
 ```
 
+### Preview Database (--preview-db=N)
+
+When testing against a preview deployment's database locally (e.g., running cron jobs or admin tools against a PR's data), use `--preview-db=N` where N is the **opencouncil** PR number:
+
+```bash
+nix run .#dev -- --preview-db=193
+```
+
+Can be combined with `--preview-tasks=M` to also connect to a tasks preview (where M is the **opencouncil-tasks** PR number):
+
+```bash
+nix run .#dev -- --preview-db=193 --preview-tasks=26
+```
+
+**Prerequisites:**
+- `OC_PREVIEW_SSH` must be set to the SSH target for the preview server:
+  ```bash
+  # In .env or exported
+  OC_PREVIEW_SSH=root@159.89.98.26
+  ```
+- Your SSH key must be in the server's `authorized_keys`
+
+**How it works:**
+
+The flag detects whether the PR has an isolated database (migration PRs with `.has-local-db` marker) or uses the shared staging database:
+
+- **Isolated DB**: Opens an SSH tunnel to the per-PR PostgreSQL instance on the server
+- **Shared staging DB**: Reads `DATABASE_URL` from the server's `.env` and uses it directly
+
+In both cases, the local postgres is skipped (`--db=external` mode is forced).
+
+**Startup output:**
+```
+🗄️  Connecting to preview database for PR #193...
+   ✓ SSH connection OK
+   ✓ Isolated database detected (port 5625)
+   ✓ SSH tunnel active: localhost:5625 → 127.0.0.1:5625
+   Database: postgresql://opencouncil@localhost:5625/opencouncil
+```
+
+The SSH tunnel is automatically cleaned up when the dev server exits.
+
 ### Mobile Preview (QR code for phone testing)
 
 The dev server binds to `0.0.0.0` by default, making it accessible from other devices on the same Wi-Fi. Click the **QR button** next to the DEV panel (bottom-right) to see a QR code encoding the LAN URL for the current page.

docs/task-architecture.md

@@ -7,11 +7,12 @@ This document outlines the architecture and workflow for handling asynchronous t
 
 1. [High-Level Overview](#1-high-level-overview)
 2. [Component Descriptions](#2-component-descriptions)
-3. [The Lifecycle of a Task](#3-the-lifecycle-of-a-task)
-4. [Task Handler Registry Pattern](#4-task-handler-registry-pattern)
-5. [Task Reprocessing](#5-task-reprocessing)
-6. [Error Handling](#6-error-handling)
-7. [Adding a New Task](#7-adding-a-new-task)
+3. [Automated Task Initiation (Cron)](#3-automated-task-initiation-cron)
+4. [The Lifecycle of a Task](#4-the-lifecycle-of-a-task)
+5. [Task Handler Registry Pattern](#5-task-handler-registry-pattern)
+6. [Task Reprocessing](#6-task-reprocessing)
+7. [Error Handling](#7-error-handling)
+8. [Adding a New Task](#8-adding-a-new-task)
 
 ## 1. High-Level Overview
 
@@ -97,10 +98,77 @@ graph TD;
     -   `version`: A version number for the task, allowing for reprocessing.
     -   ...and other relevant fields.
 
-## 3. The Lifecycle of a Task
+## 3. Automated Task Initiation (Cron)
+
+Some tasks are initiated automatically on a schedule rather than by a user action. These use cron-triggered API routes that authenticate via a shared secret (`CRON_SECRET`).
+
+### Diavgeia Decision Polling
+
+The `pollDecisions` task can be triggered automatically to fetch decisions from [Diavgeia](https://diavgeia.gov.gr) (the Greek government transparency portal) for recent council meetings.
+
+**Endpoint:** `GET /api/cron/poll-decisions`
+**Authentication:** `Authorization: Bearer <CRON_SECRET>`
+
+**What it does:**
+1. Finds meetings from the last 90 days in cities that have `diavgeiaUid` configured
+2. Filters to meetings that still have subjects with no linked decision
+3. Applies progressive backoff based on previous polling history (see below)
+4. Dispatches `pollDecisions` tasks to the backend task server for up to 10 meetings per invocation
+5. Only polls for subjects that don't already have a decision
+
+**Progressive backoff:**
+Some subjects may never have decisions on Diavgeia, so the cron uses time-based backoff to avoid polling forever. It derives the first and last poll dates from existing `TaskStatus` records (type `pollDecisions`, status `succeeded`) for each meeting:
+
+| Days since first poll | Minimum interval between polls |
+|-----------------------|-------------------------------|
+| 0–7 (week 1)         | None (every cron run)         |
+| 7–14 (week 2)        | 2 days                        |
+| 14–21 (week 3)       | 3 days                        |
+| 21+ (week 4+)        | 7 days                        |
+| 90+                   | Automatic polling stops        |
+
+With the cron running 2x/day, this means ~14 polls in week 1, ~3-4 in week 2, ~2-3 in week 3, then ~1/week.
+
+After automatic polling stops, users can still manually fetch decisions from the subject page. The backoff schedule is defined as `BACKOFF_SCHEDULE` at the top of `pollDecisions.ts` and is easy to adjust.
+
+**Polling stats:**
+To fine-tune the backoff schedule, use the stats endpoint:
+
+```bash
+curl -H "Authorization: Bearer $CRON_SECRET" https://your-domain.com/api/cron/poll-decisions-stats
+```
+
+This returns per-decision data showing: when Diavgeia published it, when the cron found it, how many poll attempts it took, and the delay between the meeting date and publication. Use the summary stats (average/median discovery delay, publish delay) to decide if the schedule needs tuning.
+
+**Setup:**
+1. Set the `CRON_SECRET` environment variable (generate with `openssl rand -base64 32`)
+2. Configure an external cron scheduler (e.g., Vercel Cron, GitHub Actions, or a simple crontab) to call the endpoint periodically:
+
+```bash
+# Poll every 12 hours (2x/day)
+0 0,12 * * * curl -s -H "Authorization: Bearer $CRON_SECRET" https://your-domain.com/api/cron/poll-decisions
+```
+
+**Key files:**
+- `src/app/api/cron/poll-decisions/route.ts` — Cron API route
+- `src/app/api/cron/poll-decisions-stats/route.ts` — Polling effectiveness stats
+- `src/lib/tasks/pollDecisions.ts` — `pollDecisionsForRecentMeetings()` (cron logic), `getPollingStats()` (stats), `pollDecisionsForMeeting()` (core logic shared with admin UI and cron)
+
+**Prerequisites per city:**
+- City must have `diavgeiaUid` set (configured in city settings)
+- Optionally, administrative bodies can have `diavgeiaUnitIds` for more precise matching
+
+### Adding New Cron Tasks
+
+To add a new cron-triggered task:
+1. Create a new API route under `src/app/api/cron/<task-name>/route.ts`
+2. Authenticate with `CRON_SECRET` (see the poll-decisions route for the pattern)
+3. Call `startTask()` for each unit of work — the rest of the task lifecycle (execution, callbacks, result handling) follows the standard flow described below
+
+## 4. The Lifecycle of a Task
 
 1.  **Initiation:**
-    -   A user action in the Next.js application triggers the `startTask` function.
+    -   A user action (or cron job) in the Next.js application triggers the `startTask` function.
     -   A new entry is created in the `TaskStatus` table with a status of `"pending"`.
     -   A `POST` request is sent to the backend task server's corresponding endpoint (e.g., `/transcribe`). The request body includes the task parameters and a `callbackUrl`.
 
@@ -119,7 +187,7 @@ graph TD;
     -   The `handleTaskUpdate` function updates the corresponding `TaskStatus` record in the database.
     -   If the task was successful, a task-specific result handler (e.g., `handleTranscribeResult`) is called to process the results.
 
-## 4. Task Handler Registry Pattern
+## 5. Task Handler Registry Pattern
 
 To maintain clean, scalable code, the system uses a centralized task handler registry. This pattern is defined in `src/lib/tasks/registry.ts`:
 
@@ -149,7 +217,7 @@ async function handleTaskResult(
 
 The optional `options` parameter allows handlers to accept flags like `force` for special processing modes (see Task Reprocessing section).
 
-## 5. Task Reprocessing
+## 6. Task Reprocessing
 
 A key feature of the task architecture is the ability to reprocess the results of a task without having to re-run the entire task on the backend server. This is made possible by storing the complete `responseBody` from the task server in the `TaskStatus` table.
 
@@ -210,14 +278,14 @@ The `TaskStatus` component (`src/components/meetings/admin/TaskStatus.tsx`) prov
 
 This approach provides clear feedback, prevents user confusion, and handles both idempotent and non-idempotent tasks appropriately.
 
-## 6. Error Handling
+## 7. Error Handling
 
 -   If the initial `fetch` call from the Next.js app to the task server fails, the task status is immediately set to `"failed"`.
 -   If the task fails on the backend server, it reports the `"error"` status back to the Next.js app.
 -   There is no automatic retry mechanism. Failed tasks can be reprocessed using the TaskStatus UI component.
 -   Result processing errors are caught and logged, with the task status updated to `"failed"` and Discord alerts sent to admins.
 
-## 7. Adding a New Task
+## 8. Adding a New Task
 
 Thanks to the task handler registry pattern, adding a new task type is straightforward. Follow these steps: