Feature/total progress (#19)

* Improve progress logging in basisprofiel app

Add total counts for missing and outdated KvK numbers at the start of
each processing step, so it's clear how far behind the mirror is.
Add get_outdated_kvk_nummers_count() to BasisProfielReader.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Improve progress logging in vestigingen app

Add total counts for missing and outdated vestigingen at the start of
each processing step. Add get_missing_kvk_nummers_count() and
get_outdated_vestigingen_count() to KvKVestigingenReader.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Improve progress logging in vestigingsprofiel app

Add total counts for missing and outdated vestigingsprofielen at the
start of each processing step. Add three count methods to
VestigingsProfielReader. Also fix W1203 violations (f-strings in
logger.info calls).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* .claude in .gitignore voor nu.

---------

Co-authored-by: Rob Verkuijlen <rob.verkuijlen@outlook.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: 3 people

.gitignore

@@ -4,6 +4,7 @@
 *.pyproj.user
 settings.json
 .idea/
+.claude/
 
 #Remove test input/output
 *.zip

CLAUDE.md

@@ -0,0 +1,83 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+KVK-Connect is a Python library and Docker microservice suite for integrating with the Dutch Chamber of Commerce (KVK) API. It serves three purposes:
+1. A pip-installable package for fetching KVK data without deep API knowledge
+2. A local mirror of KVK data (basisprofiel, vestigingen, vestigingsprofiel) kept up-to-date in a database
+3. An optional mutation service that polls KVK API for company changes and syncs them to the local mirror
+
+## Commands
+
+Before every commit, always run `just check-all` twice. The first run may auto-fix files (ruff); the second run validates the result is clean. Only commit if the second run passes fully.
+
+```bash
+# Install dependencies
+just install          # uv sync
+
+# Quality checks
+just test             # pytest
+just cov              # pytest with coverage report
+just lint             # ruff check + format
+just typing           # pyright type checking
+just check-all        # all checks (lint, cov, typing, pre-commit)
+
+# Run a single test
+uv run pytest tests/path/to/test_file.py::test_function_name
+
+# Docker
+just docker-build     # build containers
+just docker-up        # start all services
+just docker-down      # stop all services
+```
+
+## Architecture
+
+### Three Model Layers
+
+Data flows through three distinct model layers (all in `src/kvk_connect/models/`):
+- **API models** (`models/api/`): Raw dataclasses deserialized from KVK API JSON responses. Each has `from_dict()` and `to_dict()` methods.
+- **Domain models** (`models/domain/`): Business logic representations, also dataclasses with `from_dict()`/`to_dict()`.
+- **ORM models** (`models/orm/`): SQLAlchemy mapped tables for database persistence.
+
+### Mappers and Services
+
+`src/kvk_connect/mappers/` contains functions that convert API models → Domain models. `src/kvk_connect/services/record_service.py` (`KVKRecordService`) is the public-facing high-level API that orchestrates fetching and mapping. Both `KVKApiClient` and `KVKRecordService` are exported from `kvk_connect.__init__` as the library's public API.
+
+### Database Layer
+
+`src/kvk_connect/db/` has separate reader (`*_reader.py`) and writer (`*_writer.py`) classes. Schema is auto-initialized via `ensure_database_initialized()` in `db/init.py` — no Alembic. Use direct SQL for any schema changes.
+
+### Docker Apps
+
+Five Docker services in `apps/`, each with a `main.py` and `Dockerfile`:
+- `gateway`: NGINX rate limiter (port 8080) — all apps route API calls through this
+- `basisprofiel`, `vestigingen`, `vestigingsprofiel`: Fetch and persist respective KVK data types
+- `mutatie-reader`: Polls KVK mutation API and writes change signals to DB
+
+Apps depend on each other in order: mutatie-reader → basisprofiel → vestigingen → vestigingsprofiel. Compose files: `docker-compose.local.yaml` (SQLite/local) and `docker-compose.db.yaml` (PostgreSQL).
+
+### Error Handling Pattern
+
+`src/kvk_connect/exceptions.py` defines two error types:
+- `KVKPermanentError`: Company doesn't exist (e.g., error code IPD0005) — long retry delay (24h default)
+- `KVKTemporaryError`: Temporary unavailability (e.g., IPD1002, IPD1003) — short retry delay (10m)
+
+`KVKApiClient` uses `@global_rate_limit()` decorator (`utils/rate_limit.py`) on all API methods.
+
+## Coding Standards
+
+- **Python 3.13+**, PEP 8, full type hints everywhere
+- Use `Mapped[T]` for SQLAlchemy ORM models
+- All dataclasses must have `from_dict()` static method and `to_dict()` using `asdict()`
+- Use `from __future__ import annotations` for forward references in dataclasses
+- Log with lazy `%s` formatting, not f-strings: `logging.info('Fetched %d records', count)` (Pylint W1203)
+- Use `uv` for package management, not `pip`
+- Database agnostic via SQLAlchemy — no Redis, no Alembic
+- Business domain terms in comments/docstrings may use Dutch; all code (variables, functions) in English
+
+### Git Worktrees
+
+When using git worktrees, create them **inside the project folder** (e.g., `.worktrees/`).

apps/basisprofiel/main.py

@@ -152,24 +152,19 @@ def process_csv(csv_path: str, kvk_client: KVKApiClient, writer: BasisProfielWri
 
 
 def process_missing(kvk_client: KVKApiClient, writer: BasisProfielWriter, reader: BasisProfielReader) -> int:
-    logger.info("Finding missing KvK nummers...")
-
     count_missing = reader.get_missing_kvk_nummers_count()
-    logger.info("Total missing KvK nummers: %s", count_missing)
-
     missing_kvk_nummers = reader.get_missing_kvk_nummers(KVK_FETCH_LIMIT)
-    description = f"{len(missing_kvk_nummers)} missing KvK nummer(s) (limit: {KVK_FETCH_LIMIT})"
+    logger.info("Missing KvK nummers: %s total, processing %s", count_missing, len(missing_kvk_nummers))
 
-    return process_kvk_nummers(missing_kvk_nummers, description, kvk_client, writer)
+    return process_kvk_nummers(missing_kvk_nummers, "missing", kvk_client, writer)
 
 
 def process_outdated(kvk_client: KVKApiClient, writer: BasisProfielWriter, reader: BasisProfielReader) -> int:
-    logger.info("Finding outdated KvK nummers...")
+    count_outdated = reader.get_outdated_kvk_nummers_count()
     outdated_kvk_nummers = reader.get_outdated_kvk_nummers()
-    logger.debug("Found outdated kvk records: %s", outdated_kvk_nummers)
+    logger.info("Outdated KvK nummers: %s total, processing %s", count_outdated, len(outdated_kvk_nummers))
 
-    description = f"{len(outdated_kvk_nummers)} outdated KvK nummer(s)"
-    return process_kvk_nummers(outdated_kvk_nummers, description, kvk_client, writer)
+    return process_kvk_nummers(outdated_kvk_nummers, "outdated", kvk_client, writer)
 
 
 def run_daemon(kvk_client: KVKApiClient, engine, batch_size: int, interval: int) -> None:

apps/vestigingen/main.py

@@ -69,15 +69,17 @@ def process_csv_kvk(csv_path: str, kvk_client: KVKApiClient, writer: KvKVestigin
 
 
 def process_missing(kvk_client: KVKApiClient, writer: KvKVestigingenWriter, reader: KvKVestigingenReader) -> int:
+    count_missing = reader.get_missing_kvk_nummers_count()
     missing_kvk_nummers = reader.get_missing_kvk_nummers()
-    description = f"{len(missing_kvk_nummers)} missing KvK nummer(s)"
-    return process_kvk_nummers(missing_kvk_nummers, description, kvk_client, writer)
+    logger.info("Missing vestigingen: %s total, processing %s", count_missing, len(missing_kvk_nummers))
+    return process_kvk_nummers(missing_kvk_nummers, "missing", kvk_client, writer)
 
 
 def process_outdated(kvk_client: KVKApiClient, writer: KvKVestigingenWriter, reader: KvKVestigingenReader) -> int:
+    count_outdated = reader.get_outdated_vestigingen_count()
     outdated_kvk_nummers = reader.get_outdated_vestigingen()
-    description = f"{len(outdated_kvk_nummers)} outdated vestigingen"
-    return process_kvk_nummers(outdated_kvk_nummers, description, kvk_client, writer)
+    logger.info("Outdated vestigingen: %s total, processing %s", count_outdated, len(outdated_kvk_nummers))
+    return process_kvk_nummers(outdated_kvk_nummers, "outdated", kvk_client, writer)
 
 
 def get_kvk_vestigingen(kvk_nummer: str, kvk_client: KVKApiClient) -> KvKVestigingsNummersDomain | None:
@@ -98,10 +100,7 @@ def run_daemon(kvk_client: KVKApiClient, engine, batch_size: int, interval: int)
             with KvKVestigingenWriter(engine, batch_size=batch_size) as writer:
                 reader = KvKVestigingenReader(engine)
 
-                logger.info("Updating known Vestigingen...")
                 count += process_outdated(kvk_client, writer, reader)
-
-                logger.info("Updating missing Vestigingen...")
                 count += process_missing(kvk_client, writer, reader)
 
                 writer.flush()

apps/vestigingsprofiel/main.py

@@ -96,21 +96,23 @@ def process_csv_vestiging(csv_path: str, kvk_client: KVKApiClient, writer: Vesti
 
 
 def process_missing(kvk_client: KVKApiClient, writer: VestigingsProfielWriter, reader: VestigingsProfielReader) -> int:
+    count_missing = reader.get_vestigingen_zonder_vestigingsprofielen_count()
     missing_profielen = reader.get_vestigingen_zonder_vestigingsprofielen()
-    description = f"{len(missing_profielen)} vestigingen without VestigingsProfielen"
-    return process_vestigingen(missing_profielen, description, kvk_client, writer)
+    logger.info("Missing vestigingsprofielen: %s total, processing %s", count_missing, len(missing_profielen))
+    return process_vestigingen(missing_profielen, "missing", kvk_client, writer)
 
 
 def process_outdated(kvk_client: KVKApiClient, writer: VestigingsProfielWriter, reader: VestigingsProfielReader) -> int:
-    outdated_vestigingen = reader.get_outdated_vestigingen()
-    description = f"{len(outdated_vestigingen)} outdated vestigingsprofielen (from kvk vestigingen)"
-    logger.info(description)
-
+    count_outdated = reader.get_outdated_vestigingen_count()
+    count_outdated_signaal = reader.get_outdated_vestigingen_signaal_count()
     outdated_vestigingen_signaal = reader.get_outdated_vestigingen_signaal()
-    description = f"{len(outdated_vestigingen_signaal)} outdated vestigingsprofielen (from signaal)"
-    logger.info(description)
-
-    return process_vestigingen(outdated_vestigingen_signaal, description, kvk_client, writer)
+    logger.info("Outdated vestigingsprofielen from vestigingen: %s total", count_outdated)
+    logger.info(
+        "Outdated vestigingsprofielen from signaal: %s total, processing %s",
+        count_outdated_signaal,
+        len(outdated_vestigingen_signaal),
+    )
+    return process_vestigingen(outdated_vestigingen_signaal, "outdated", kvk_client, writer)
 
 
 def run_daemon(kvk_client: KVKApiClient, engine, batch_size: int, interval: int) -> None:

src/kvk_connect/db/basisprofiel_reader.py

@@ -79,7 +79,8 @@ def get_outdated_kvk_nummers(self, limit: int = 1000) -> list[str]:
                     SignaalORM.timestamp > BasisProfielORM.last_updated,
                     SignaalORM.vestigingsnummer.is_(None),  # Alleen basisprofiel updates, geen vestigingsprofielen
                     BasisProfielORM.niet_leverbaar_code.is_(None),  # Geen tombstones updaten
-                    BasisProfielORM.retry_after.is_(None) | (BasisProfielORM.retry_after <= func.now()),  # Geen actieve blokkade
+                    BasisProfielORM.retry_after.is_(None)
+                    | (BasisProfielORM.retry_after <= func.now()),  # Geen actieve blokkade
                 )
                 .distinct()
                 .limit(limit)  # maximaal limit nieuwe per keer ophalen
@@ -88,6 +89,22 @@ def get_outdated_kvk_nummers(self, limit: int = 1000) -> list[str]:
             result = session.execute(stmt).scalars().all()
             return list(result)
 
+    def get_outdated_kvk_nummers_count(self) -> int:
+        """Retourneert het totaal aantal KVK nummers met een nieuwer signaal dan het opgeslagen basisprofiel."""
+        with Session(self.engine) as session:
+            stmt = (
+                select(func.count(func.distinct(SignaalORM.kvknummer)))
+                .join(BasisProfielORM, SignaalORM.kvknummer == BasisProfielORM.kvk_nummer)
+                .where(
+                    SignaalORM.timestamp > BasisProfielORM.last_updated,
+                    SignaalORM.vestigingsnummer.is_(None),
+                    BasisProfielORM.niet_leverbaar_code.is_(None),
+                    BasisProfielORM.retry_after.is_(None) | (BasisProfielORM.retry_after <= func.now()),
+                )
+            )
+            result = session.execute(stmt).scalar()
+            return result or 0
+
     def kvk_nummer_exists(self, kvk_nummer: str) -> bool:
         """Check if KVK number exists in basisprofiel.
 

src/kvk_connect/db/kvkvestigingen_reader.py

@@ -1,4 +1,4 @@
-from sqlalchemy import select
+from sqlalchemy import func, select
 from sqlalchemy.engine import Engine
 from sqlalchemy.orm import Session
 
@@ -25,6 +25,29 @@ def get_missing_kvk_nummers(self, limit: int = 1000) -> list[str]:
             result = session.execute(stmt).scalars().all()
             return list(result)
 
+    def get_missing_kvk_nummers_count(self) -> int:
+        """Retourneert het totaal aantal KVK nummers die wel in basisprofielen staan maar nog niet in kvkvestigingen."""
+        with Session(self.engine) as session:
+            stmt = (
+                select(func.count(func.distinct(BasisProfielORM.kvk_nummer)))
+                .select_from(BasisProfielORM)
+                .outerjoin(VestigingenORM, BasisProfielORM.kvk_nummer == VestigingenORM.kvk_nummer)
+                .where(VestigingenORM.kvk_nummer.is_(None))
+            )
+            result = session.execute(stmt).scalar()
+            return result or 0
+
+    def get_outdated_vestigingen_count(self) -> int:
+        """Retourneert het totaal aantal KVK nummers waarvan de vestigingen verouderd zijn."""
+        with Session(self.engine) as session:
+            stmt = (
+                select(func.count(func.distinct(BasisProfielORM.kvk_nummer)))
+                .join(VestigingenORM, BasisProfielORM.kvk_nummer == VestigingenORM.kvk_nummer)
+                .where(BasisProfielORM.last_updated > VestigingenORM.last_updated)
+            )
+            result = session.execute(stmt).scalar()
+            return result or 0
+
     def get_outdated_vestigingen(self, limit: int = 1000) -> list[str]:
         """Geen een lijst van unieke kvknummers terug waarvan de vestigingen verouderd zijn.
 

src/kvk_connect/db/vestigingenprofiel_reader.py

@@ -1,4 +1,4 @@
-from sqlalchemy import select
+from sqlalchemy import func, select
 from sqlalchemy.orm import Session
 
 from kvk_connect.models.orm.signaal_orm import SignaalORM
@@ -31,6 +31,45 @@ def get_vestigingen_zonder_vestigingsprofielen(self, limit: int = 1000) -> list[
             result = session.execute(stmt)
             return [row[0] for row in result.fetchall()]
 
+    def get_vestigingen_zonder_vestigingsprofielen_count(self) -> int:
+        """Retourneert het totaal aantal vestigingsnummers zonder vestigingsprofiel."""
+        with Session(self.engine) as session:
+            stmt = (
+                select(func.count(func.distinct(VestigingenORM.vestigingsnummer)))
+                .outerjoin(
+                    VestigingsProfielORM, VestigingenORM.vestigingsnummer == VestigingsProfielORM.vestigingsnummer
+                )
+                .where(VestigingsProfielORM.vestigingsnummer.is_(None))
+                .where(VestigingenORM.vestigingsnummer != VestigingenORM.SENTINEL_VESTIGINGSNUMMER)
+            )
+            result = session.execute(stmt).scalar()
+            return result or 0
+
+    def get_outdated_vestigingen_count(self) -> int:
+        """Retourneert het totaal aantal vestigingsnummers met vestiging nieuwer dan het vestigingsprofiel."""
+        with Session(self.engine) as session:
+            stmt = (
+                select(func.count(func.distinct(VestigingenORM.vestigingsnummer)))
+                .join(VestigingsProfielORM, VestigingenORM.vestigingsnummer == VestigingsProfielORM.vestigingsnummer)
+                .where(VestigingenORM.last_updated > VestigingsProfielORM.last_updated)
+                .where(VestigingenORM.vestigingsnummer != VestigingenORM.SENTINEL_VESTIGINGSNUMMER)
+            )
+            result = session.execute(stmt).scalar()
+            return result or 0
+
+    def get_outdated_vestigingen_signaal_count(self) -> int:
+        """Retourneert het totaal aantal vestigingsnummers met signaal nieuwer dan het vestigingsprofiel."""
+        with Session(self.engine) as session:
+            stmt = (
+                select(func.count(func.distinct(SignaalORM.vestigingsnummer)))
+                .join(VestigingsProfielORM, SignaalORM.vestigingsnummer == VestigingsProfielORM.vestigingsnummer)
+                .where(
+                    SignaalORM.timestamp > VestigingsProfielORM.last_updated, SignaalORM.vestigingsnummer.is_not(None)
+                )
+            )
+            result = session.execute(stmt).scalar()
+            return result or 0
+
     def get_outdated_vestigingen(self, limit: int = 1000) -> list[str]:
         """Return lijst van vestigingsnummers met vestiging nieuwer dan de lastupdated van het vestigingenprofiel."""