Skip to content

Support for Diavgeia decisions#193

Open
kouloumos wants to merge 19 commits intoschemalabz:mainfrom
kouloumos:diavgeia-decisions
Open

Support for Diavgeia decisions#193
kouloumos wants to merge 19 commits intoschemalabz:mainfrom
kouloumos:diavgeia-decisions

Conversation

@kouloumos
Copy link
Member

@kouloumos kouloumos commented Feb 9, 2026

Fixes #103

Decisions polled from Diavgeia (support for manual addition)
image

Subject's Page now shows the Decision iff available

  • on the title:
    image
  • separate card
    image

Allows to manually fetch if Decision not available
decision-search

Admin dashboard for Polling Stats
Screenshot from 2026-02-25 19-38-18


Note

Medium Risk
Introduces a new persisted Decision model with Prisma migration and multiple new polling/management endpoints (cron + admin + public trigger), so regressions could affect meeting/subject data integrity and background task load. Risk is mitigated by auth checks on admin routes and CRON_SECRET gating for cron, but rollout requires correct env configuration and monitoring.

Overview
Adds end-to-end support for linking meeting Subjects to official Diavgeia decisions, including a new Decision table (1:1 with Subject) plus configuration fields City.diavgeiaUid and AdministrativeBody.diavgeiaUnitIds.

Implements automated polling via new cron routes GET /api/cron/poll-decisions and GET /api/cron/poll-decisions-stats (Bearer CRON_SECRET), plus admin/public workflows to fetch and manage decisions: an admin DecisionsPanel, a meeting decisions API (GET/PUT/DELETE /api/cities/[cityId]/meetings/[meetingId]/decisions), decision status surfacing in admin meetings lists, and decision display + on-demand fetch on the subject page.

Adds a new /admin/diavgeia polling stats dashboard, expands env/docs (CRON_SECRET, preview DB SSH), updates Nix dev tooling with --preview-db/--preview-tasks (ngrok + SSH tunneling), and includes small UI stability tweaks (dropdown focus handling, hash-linked CollapsibleCard).

Written by Cursor Bugbot for commit 4f3bbf6. This will update automatically on new commits. Configure here.

Greptile Summary

This PR adds comprehensive support for linking council meeting subjects to their official Diavgeia decisions (Greek government transparency platform). The implementation includes:

Database Schema: New Decision table (1:1 with Subject) stores decision metadata (ada, protocolNumber, title, pdfUrl, issueDate). Configuration fields added: City.diavgeiaUid and AdministrativeBody.diavgeiaUnitIds for Diavgeia API matching.

Automated Polling: Cron job (/api/cron/poll-decisions) runs 2x/day to poll recent meetings (90-day window) for decisions. Uses progressive backoff schedule (daily→weekly) to reduce load as meetings age. Rate limiting prevents duplicate concurrent tasks via TaskStatus queries filtering non-terminal states.

Manual Workflows: Admins can bulk-manage decisions via DecisionsPanel UI and /api/cities/[cityId]/meetings/[meetingId]/decisions API. Public users can trigger on-demand polling from subject pages with simple 5-minute rate limiting. All mutations validate subject ownership within the meeting before upserting.

Monitoring: New /admin/diavgeia dashboard displays polling effectiveness stats (discovery delays, publish delays, meetings still polling) to tune the backoff schedule. Task error handling improved to persist diagnostic messages in responseBody.

Dev Tooling: Nix dev runner enhanced with --preview-db and --tasks-preview flags to connect to remote preview environments via ngrok tunnels. Import script provided to seed Diavgeia configuration from external sources.

All previous thread concerns have been addressed: auth is properly enforced via withUserAuthorizedToEdit, upsert is idempotent on subjectId, PostgreSQL UNIQUE on nullable ada works correctly, rate limiting covers all non-terminal task states, and subject validation prevents cross-meeting pollution.

Confidence Score: 4/5

  • This PR is safe to merge with minor risks around configuration and cron scheduling
  • The implementation is well-structured with proper authorization, idempotent operations, and defensive validation. Auth checks are correctly placed and awaited. Database schema changes are sound with appropriate constraints and indices. Task polling logic includes backoff and rate limiting. Subject ownership validation prevents cross-tenant data pollution. Error handling improvements preserve diagnostic context. Previous review concerns were all addressed or clarified. The main risk areas are: (1) CRON_SECRET is optional, so cron endpoints will return 503 until configured in production, (2) backoff schedule effectiveness depends on real-world Diavgeia publishing patterns and may need tuning, (3) the 60-second task status polling loop on subject pages creates sustained load during user-triggered polls but is rate-limited.
  • Check that CRON_SECRET is configured in production environment before first cron run. Monitor /admin/diavgeia stats after deployment to verify backoff schedule matches actual Diavgeia publishing delays.

Important Files Changed

Filename Overview
prisma/schema.prisma Adds Decision table with 1:1 relation to Subject, plus City.diavgeiaUid and AdministrativeBody.diavgeiaUnitIds fields for Diavgeia integration
prisma/migrations/20260202164839_add_diavgeia_decisions/migration.sql Migration creates Decision table with unique indices on subjectId and ada, adds Diavgeia configuration fields to City and AdministrativeBody
src/lib/tasks/pollDecisions.ts Core polling logic with progressive backoff, rate limiting via task status checks, validates subjectIds before upsert to prevent cross-meeting pollution
src/lib/tasks/pollDecisionsBackoff.ts Configurable backoff schedule for cron polling (daily→weekly over 90 days), pure helper functions for backoff state calculation
src/app/api/cron/poll-decisions/route.ts Cron endpoint authenticated via CRON_SECRET bearer token, dispatches polling for recent meetings via pollDecisionsForRecentMeetings
src/app/api/cities/[cityId]/meetings/[meetingId]/decisions/route.ts Admin-only API (GET/PUT/DELETE) for manual decision management, validates subject ownership before upsert/delete
src/lib/db/decisions.ts DB helpers for decision CRUD, upsert by subjectId, preserves original taskId/createdById on updates
src/components/meetings/subject/subject.tsx Subject page displays decision card, allows public users to trigger polling with 60-second task status poll loop
src/components/meetings/admin/DecisionsPanel.tsx Admin UI for bulk decision management, supports manual entry with PDF upload, shows polling history and backoff status
src/lib/tasks/tasks.ts Improved error handling: persists error messages in responseBody for failed tasks instead of losing diagnostic info

Sequence Diagram

sequenceDiagram
    participant Cron as Cron Job
    participant API as Poll API
    participant DB as Database
    participant Tasks as Task Backend
    participant Admin as Admin UI
    participant User as Subject Page

    Note over Cron,Tasks: Automated Polling Flow
    Cron->>API: GET /api/cron/poll-decisions
    API->>DB: Find meetings (90d, unlinked decisions)
    DB-->>API: Recent meetings list
    API->>API: Apply backoff filter
    API->>DB: Create TaskStatus (pollDecisions)
    API->>Tasks: POST task to TASK_API_URL
    Tasks-->>API: Task accepted
    API-->>Cron: Result (dispatched count)
    
    Note over Tasks,DB: Task Processing
    Tasks->>Tasks: Fetch Diavgeia API
    Tasks->>API: POST /api/taskStatuses/{id}
    API->>DB: Validate subjectIds
    API->>DB: Upsert Decision records
    DB-->>API: Success
    API-->>Tasks: Callback acknowledged

    Note over Admin,User: Manual Workflows
    Admin->>API: PUT /api/cities/{id}/meetings/{id}/decisions
    API->>DB: Validate subject ownership
    API->>DB: Upsert decision (createdById)
    DB-->>Admin: Decision saved
    
    User->>API: requestPollDecisionForSubject
    API->>DB: Check rate limit (5min)
    API->>DB: Create TaskStatus
    API->>Tasks: POST task
    User->>User: Poll taskStatus (60s loop)
    User->>API: GET /api/cities/{id}/meetings/{id}/taskStatuses/{id}
    API-->>User: Task status
Loading

Last reviewed commit: 4f3bbf6

@greptile-apps
Copy link

greptile-apps bot commented Feb 9, 2026

Greptile Overview

Greptile Summary

Adds comprehensive Diavgeia decision integration to OpenCouncil, allowing admins to link official government decisions to meeting subjects. Introduces a new Decision model (1:1 with Subject) and extends City and AdministrativeBody with Diavgeia configuration fields (diavgeiaUid and diavgeiaUnitId).

Key changes:

  • Database schema: New Decision table with unique constraints on subjectId and nullable ada, foreign keys to Subject, TaskStatus, and User
  • API routes: New /api/cities/{cityId}/meetings/{meetingId}/decisions endpoint with GET/PUT/DELETE handlers, properly authorized via withUserAuthorizedToEdit
  • Async polling: pollDecisions task queries Diavgeia API via external task server, validates returned subjectIds against meeting ownership, and upserts matched decisions
  • Admin UI: DecisionsPanel component for viewing, filtering, manually linking, and triggering automated polling of decisions
  • Subject queries: All subject fetching now includes decision relation to display linked decisions
  • Forms: Updated CityForm and AdministrativeBodiesList to edit Diavgeia configuration

Important notes:

  • Migration correctly uses PostgreSQL-compatible unique index on nullable ada (allows multiple NULLs)
  • Authorization properly enforced at API boundaries (previous thread comments were false positives)
  • Subject ownership validated before decision linking
  • One potential issue: duplicate ada values from backend not deduplicated before upserting (see inline comment)

Confidence Score: 4/5

  • This PR is safe to merge with one minor edge case to address
  • The implementation is solid with proper authorization, validation, and data integrity checks. Database schema is well-designed with appropriate constraints. The one issue is potential duplicate ada handling in the polling task that could cause runtime errors if the backend returns multiple matches with the same ada
  • Pay attention to src/lib/tasks/pollDecisions.ts - add deduplication for ada values before upserting to avoid unique constraint violations

Important Files Changed

Filename Overview
prisma/schema.prisma Added Decision model with 1:1 relationship to Subject, plus diavgeiaUid on City and diavgeiaUnitId on AdministrativeBody. Schema properly defines relationships and constraints.
src/app/api/cities/[cityId]/meetings/[meetingId]/decisions/route.ts Well-implemented API route with proper authorization checks, subject ownership validation, and clean CRUD operations. All handlers properly use withUserAuthorizedToEdit.
src/lib/db/decisions.ts Clean database helper with proper typing. Upsert uses subjectId as key (ensures idempotency), preserves source tracking on updates.
src/lib/tasks/pollDecisions.ts Task handler with proper auth checks and subject validation. Validates returned subjectIds against meeting ownership before upserting. Missing validation that ada values are unique across the meeting.
src/components/meetings/admin/DecisionsPanel.tsx Comprehensive UI component with filtering, manual entry, and polling features. Good UX with loading states, validation, and source attribution.

Sequence Diagram

sequenceDiagram
    participant Admin as Admin User
    participant UI as DecisionsPanel
    participant API as /api/decisions
    participant Task as pollDecisions
    participant Backend as Task Server
    participant Diavgeia as Diavgeia API
    participant DB as Database

    Note over Admin,DB: Manual Decision Linking Flow
    Admin->>UI: Open Decisions Panel
    UI->>API: GET /decisions
    API->>DB: getDecisionsForMeeting()
    DB-->>API: Return existing decisions
    API-->>UI: Display decisions by subject
    Admin->>UI: Fill form (pdfUrl, ada, etc.)
    UI->>API: PUT /decisions
    API->>DB: Verify subject ownership
    DB-->>API: Subject validated
    API->>DB: upsertDecision(subjectId, data, createdById)
    DB-->>API: Decision created/updated
    API-->>UI: Success
    UI-->>Admin: Show linked decision

    Note over Admin,DB: Automated Polling Flow
    Admin->>UI: Click "Poll Decisions"
    UI->>Task: requestPollDecisions(cityId, meetingId)
    Task->>DB: Fetch meeting + subjects
    DB-->>Task: Meeting data with subjects
    Task->>DB: Create TaskStatus record
    DB-->>Task: Task created
    Task->>Backend: POST to TASK_API_URL
    Backend-->>Task: Task queued
    Task-->>UI: Task started
    UI-->>Admin: Show "polling in progress"

    Note over Backend,DB: Background Processing
    Backend->>Diavgeia: Search decisions (by date, unitId)
    Diavgeia-->>Backend: Return decision list
    Backend->>Backend: Match decisions to subjects
    Backend->>API: POST /api/taskStatuses/{id} (callback)
    API->>Task: handlePollDecisionsResult(result)
    Task->>DB: Validate subjectIds belong to meeting
    DB-->>Task: Validation complete
    loop For each matched decision
        Task->>DB: upsertDecision(subjectId, ada, taskId)
        DB-->>Task: Decision linked
    end
    Task-->>API: Processing complete
    API-->>Backend: Success

    Note over Admin,DB: View Updated Decisions
    Admin->>UI: Refresh panel
    UI->>API: GET /decisions
    API->>DB: getDecisionsForMeeting()
    DB-->>API: Decisions (with task/user source)
    API-->>UI: Display with source badges
    UI-->>Admin: Show automated + manual decisions
Loading

Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

10 files reviewed, 3 comments

Edit Code Review Agent Settings | Greptile

@greptile-apps
Copy link

greptile-apps bot commented Feb 9, 2026

Additional Comments (1)

src/components/cities/CityForm.tsx
Missing diavgeiaUnitId in type

CityForm’s local administrativeBodies state type doesn’t include diavgeiaUnitId, but the API now returns it and AdministrativeBodiesList expects it. TypeScript will fail when calling setAdministrativeBodies(data) (data includes extra field) / passing administrativeBodies to AdministrativeBodiesList.

    const [administrativeBodies, setAdministrativeBodies] = useState<Array<{
        id: string;
        name: string;
        name_en: string;
        type: AdministrativeBodyType;
        youtubeChannelUrl?: string | null;
        notificationBehavior?: NotificationBehavior | null;
        diavgeiaUnitId?: string | null;
    }>>([])
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/components/cities/CityForm.tsx
Line: 54:61

Comment:
**Missing diavgeiaUnitId in type**

`CityForm`’s local `administrativeBodies` state type doesn’t include `diavgeiaUnitId`, but the API now returns it and `AdministrativeBodiesList` expects it. TypeScript will fail when calling `setAdministrativeBodies(data)` (data includes extra field) / passing `administrativeBodies` to `AdministrativeBodiesList`.

```suggestion
    const [administrativeBodies, setAdministrativeBodies] = useState<Array<{
        id: string;
        name: string;
        name_en: string;
        type: AdministrativeBodyType;
        youtubeChannelUrl?: string | null;
        notificationBehavior?: NotificationBehavior | null;
        diavgeiaUnitId?: string | null;
    }>>([])
```

How can I resolve this? If you propose a fix, please make it concise.

@github-actions
Copy link

github-actions bot commented Feb 9, 2026

🚀 Preview deployment ready!

Preview URL: https://pr-193.preview.opencouncil.gr
Commit: 4f3bbf6
Database: Isolated (PostGIS 3.3.5, seed data)
Tasks API: https://pr-26.tasks.opencouncil.gr

The preview will be automatically updated when you push new commits.
It will be destroyed when this PR is closed or merged.


This preview uses an isolated database with seed data (migrations were applied).

@kouloumos
Copy link
Member Author

I force-pushed to address bot review comment(s) (1, 2):

  • Added subject ownership validation to PUT/DELETE decision endpoints to prevent cross-city/meeting authorization bypass
  • Added diavgeiaUnitId field to CityForm's administrativeBodies state type

@kouloumos
Copy link
Member Author

I force-pushed to address bot review comment(s) (1, 2):

  • Added subject ownership validation in handlePollDecisionsResult to verify subjectIds belong to the task's meeting
  • Fixed city creation to use data.diavgeiaUid instead of hardcoded null

Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

27 files reviewed, 3 comments

Edit Code Review Agent Settings | Greptile

Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5 files reviewed, 1 comment

Edit Code Review Agent Settings | Greptile

@kouloumos
Copy link
Member Author

I force-pushed to address bot review comment(s) (1, 2):

  • Show "Manage Decisions" button regardless of diavgeiaUid config (manual entry still works)
  • Reset form state when dialog opens (clear expanded entries, filter tab, form values)

Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.

Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

53 files reviewed, 4 comments

Edit Code Review Agent Settings | Greptile

Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

53 files reviewed, 3 comments

Edit Code Review Agent Settings | Greptile

@kouloumos
Copy link
Member Author

I force-pushed to address bot review comment(s) (1, 2):

  • Rate limit in requestPollDecisionForSubject now checks status: { notIn: ['failed', 'succeeded'] } instead of { in: ['pending'] }, matching the checkTaskIdempotency pattern
  • Fixed agendaItemIndex truthy check in subject header to agendaItemIndex != null for consistency with the rest of the file

Add support for linking meeting subjects to decisions from the Greek
Government Transparency portal (Diavgeia).

Schema changes:
- Add Decision model with ada, protocolNumber, title, pdfUrl, issueDate
- Add City.diavgeiaUid for organization identifier
- Add AdministrativeBody.diavgeiaUnitId for unit filtering

UI additions:
- Add DecisionsPanel for managing decision links in meeting admin
- Add diavgeiaUid field to CityForm
- Add diavgeiaUnitId field to AdministrativeBodiesList
- Add export functionality for manual matching

Task integration:
- Add pollDecisions task type and handler
- Register in task registry for automatic callback handling
- Support filtering by diavgeiaUnitId for more accurate matching

Also update CLAUDE.md with guidance on checking for existing types
before creating new ones.
Adds event propagation stop to DropdownMenuContent to prevent infinite
focus loops when used inside Dialogs. Mirrors the fix applied to
SelectContent in 889da63.
Display linked Diavgeia decisions as a collapsible card on the subject
page, showing the title, ADA, protocol number, issue date, and a link
to the PDF.
Add CRON_SECRET env var and GET /api/cron/poll-decisions endpoint
that polls Diavgeia for decisions on recent meetings (last 90 days)
with unlinked subjects. Processes up to 10 meetings per invocation.
Authenticated via Bearer token.
Add requestPollDecisionForSubject() server action that any user can
trigger from the subject page, with 5-minute rate limiting. Show a
'Fetch from Diavgeia' button when no decision is linked, with loading
state and confirmation message.
Document the team convention for using fixup! commits when modifying
existing branch commits, and autosquash rebase to fold them in.
Add automated task initiation section to task-architecture.md covering
the Diavgeia decision polling cron endpoint, setup instructions, and
pattern for adding new cron tasks. Add CRON_SECRET to env docs and
.env.example.
Add /admin/diavgeia page with summary stat cards (total discoveries,
meetings still polling, avg discovery/publish delays), backoff schedule
table, and sortable discoveries table. Includes progressive backoff for
cron polling and a stats API endpoint for monitoring.
Add getPollingHistoryForMeeting() to derive current backoff tier and
next eligible poll time. DecisionsPanel now shows poll count, start
date, tier label, and next auto-poll date below the toolbar.
Add decision counts column (X/Y linked/eligible) to the meetings table
and a Decisions card in expanded row content with lazy-loaded polling
history (tier, last poll, next eligible poll).
Standalone script to link cities and administrative bodies to their
Diavgeia counterparts by matching names with accent-stripped comparison,
substring matching, and known abbreviations (ΔΣ, ΔΕ). Supports --dry-run
to preview and --force to overwrite existing UIDs.
Allows connecting to an opencouncil-tasks preview deployment for a
given PR number. Starts an ngrok tunnel so the remote tasks server
can POST callbacks back to localhost, and overrides NEXTAUTH_URL to
the ngrok URL for correct URL construction.
Click the Eye button on any poll row to open a Sheet sidebar showing
full metadata (status, time, city, meeting, results summary), a link
to the meeting admin page, and copyable request/response JSON bodies.
Two failure paths in startTask and handleTaskUpdate were setting status
to 'failed' without storing the error in responseBody, making it
impossible to debug failures from the admin UI.
Add server-side URL param filters to the Diavgeia admin polling stats
page. City filter narrows polls by city; meeting filter appears once a
city is selected and narrows further by council meeting ID. Changing the
city automatically clears the meeting selection.

Also adds updateParams() to useUrlParams hook for atomic multi-param
URL updates.
Connects to the database used by an opencouncil preview deployment.
Detects whether the PR has an isolated DB (SSH tunnel) or uses
the shared staging DB (reads URL from server), and forces external
DB mode to skip local postgres.

Requires OC_PREVIEW_SSH to be set to the preview server SSH target.
Can be used independently or alongside --preview-tasks=M.
…a admin

Extract getBackoffState() helper from inline tier calculation logic so
it can be reused. Expand getPollingStats() to return per-meeting details
(unlinked subjects, backoff state, poll history) instead of just a count.

Add a collapsible table showing each still-polling meeting with city,
date, unlinked/eligible counts, first/last poll dates (with timestamp
tooltips on hover), backoff tier, and a details sidebar listing the
unlinked subject names with a link to the meeting admin page.

Refactor getPollingHistoryForMeeting() to use the shared helper.
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.

currentTierLabel,
nextPollEligible,
};
});
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Diavgeia stats filters ignored for discoveries and meetings

Medium Severity

getPollingStats accepts cityId and councilMeetingId but never applies them to the discoveries query, discoveryDetails, stillPollingMeetings, or meetingsStillPolling. Only recentPollTasks and pollMeetings are filtered. As a result, when a city or meeting is selected in the Diavgeia admin UI, the Discoveries table, Meetings Still Polling table, and summary stats (total discoveries, discovery delay, publish delay) still show global data instead of the filtered subset.

Fix in Cursor Fix in Web

}

console.log(`Poll decisions completed: ${processedCount} processed, ${result.unmatchedSubjects.length} unmatched, ${result.ambiguousSubjects.length} ambiguous`);
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

handlePollDecisionsResult crashes on malformed task result

Medium Severity

handlePollDecisionsResult assumes result.matches, result.unmatchedSubjects, and result.ambiguousSubjects are always defined. If the task server sends a malformed callback (e.g. result: {} or result: { matches: undefined }), the handler throws on result.matches.map, for...of result.matches, or result.unmatchedSubjects.length, causing a 500 and failed callback processing.

Fix in Cursor Fix in Web

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Poll Diavgeia decisions and link them to meeting subjects

1 participant