Merge pull request #221 from FreeshardBase/feature/clayde/verify-owner-email

Verify the owner email before asserting email_verified

Author: max-tet

README.md

@@ -123,6 +123,14 @@ Shard Core logs in to that registry at startup, using the credentials checked in
 
 If you would rather not depend on our infrastructure, note that this is what you would have to replace.
 
+### The owner's email address
+
+The owner's address lives on their user row and is verified by definition: a new address is only a candidate until somebody opens a confirmation link that was delivered to it. On a hosted shard the controller does that delivery, and it is also what mails the owner about disk space, billing and service notices.
+
+A self-hosted shard has no controller and therefore no way to send mail at all, so it cannot run that round trip. Set `email.enabled = false` in `local_config.toml` and the address is taken as given, with no candidate step and no confirmation mail — acceptable because you control the machine. Leave it at its default of `true` and setting an address fails with HTTP 502 — the address is kept as a candidate, but the confirmation mail that would promote it cannot be delivered.
+
+The address is what the built-in OIDC provider emits as the `email` claim, alongside `email_verified`. With no confirmed address it emits neither, and apps you log into through it will create fresh accounts rather than linking to an existing one by address.
+
 ### Localhost
 
 In order to test freeshard, you might want to launch it on localhost first.

agents.md

@@ -29,6 +29,7 @@ shard_core/
     memory_pressure.py  PSI parsing (/host/pressure/memory), cgroup v2 memory.reclaim page-out
     pause_metrics.py    In-memory pause-tier telemetry accumulators (transitions, latencies, PSI snapshots)
     pairing.py          Terminal pairing (JWT creation, code generation)
+    owner_email.py      Owner address: candidate, confirmation token, promotion
     backup.py           Azure Blob Storage backup via rclone
     peer.py             Peer shard management
     crypto.py           RSA-4096 key generation, signing, verification (PSS padding)
@@ -41,7 +42,9 @@ shard_core/
   database/           → PostgreSQL access layer (per-entity modules, conn-first-arg pattern)
   data_model/         → Pydantic v2 models
     app_meta.py         App metadata, Status enum, VMSize enum
-    identity.py         Shard identity (keys, domain, short_id)
+    identity.py         Shard identity (keys, domain, short_id) — the shard's
+                        public profile, published unauthenticated; holds no address
+    user.py             People on the shard (owner, later members) and their address
     terminal.py         Paired device models
     peer.py             Peer shard models
     backend/            Models copied from freeshard-controller (via `just get-types`)
@@ -71,7 +74,7 @@ async with db_conn() as conn:
     await db_installed_apps.update_status(conn, "myapp", Status.RUNNING)
 ```
 
-Tables: `identities`, `terminals`, `installed_apps`, `peers`, `backups`, `tours`, `app_usage_tracks`, `kv_store`.
+Tables: `identities`, `users`, `terminals`, `installed_apps`, `peers`, `backups`, `tours`, `app_usage_tracks`, `kv_store`, `oidc_clients`, `oidc_codes`, `oidc_tokens`.
 
 Postgres data is not part of the rclone backup set (which only syncs `core/`/`user_data/`). To keep it, `database/db_snapshot.py` dumps all application tables to `core/db_snapshot.json` before each backup, and `init_database()` restores it on a fresh shard (before the default identity is generated, so the restored identity survives). Pre-0.38 backups are restored from TinyDB by `tinydb_migration.py` instead.
 
@@ -90,6 +93,14 @@ Started at app lifespan startup, stopped at shutdown:
 - `CronTask(docker_prune_images, daily)` — image cleanup
 - Various telemetry and peer key refresh tasks
 
+### The Owner's Email Address
+`users.email` is the single home for a person's address and is **verified by definition**; `users.pending_email` is an unverified candidate, at most one in flight per user. That invariant is the only reason the OIDC provider may assert `email_verified: true` — with no verified address it emits neither claim, and there is deliberately no synthetic fallback.
+
+- `service/owner_email.py` owns every transition. Anything that writes `pending_email` must retire the token with it, or the token promotes an address it was never sent to.
+- On confirmation, in this order: notify the old address, promote, mirror to `shards.owner_email`, notify the new one. The first step must precede the mirror — the controller's relay only ever reaches the address it currently has on file.
+- `POST /public/users/confirm-email` is unauthenticated by design and the token is its only credential. There is no `GET`: mail scanners and link prefetchers would burn a single-use token.
+- `email.enabled` (default `true`) says whether the shard can send mail at all, and is the explicit self-hosted signal — no controller means no mail, so the address is set directly. Never infer it from a failed delivery; a controller outage would silently downgrade a security control.
+
 ### Signals (Event System)
 Blinker-based async signals defined in `util/signals.py`. DB-writing handlers are async and called via `await signal.send_async()`:
 - `on_apps_update` — app state changed

config.toml

@@ -76,6 +76,9 @@ send_interval_seconds = 300
 [oidc]
 enabled = false
 
+[email]
+enabled = true
+
 [management]
 api_url = "https://ptlfunctionapp.azurewebsites.net/api/management"
 

migrations/shard-core-0005-owner-email-verification.sql

@@ -0,0 +1,31 @@
+-- shard-core-0005-owner-email-verification
+-- depends: shard-core-0004-oidc
+
+-- users.email becomes the single home for a person's address, and it is
+-- verified by definition. pending_email holds an unverified candidate, at most
+-- one per user, together with the digest and expiry of its confirmation token.
+ALTER TABLE users ADD COLUMN pending_email TEXT;
+ALTER TABLE users ADD COLUMN email_token_hash TEXT;
+ALTER TABLE users ADD COLUMN email_token_expires TIMESTAMPTZ;
+
+-- the two token columns are only ever written together
+ALTER TABLE users ADD CONSTRAINT users_email_token_paired
+    CHECK ((email_token_hash IS NULL) = (email_token_expires IS NULL));
+
+-- identities.email was editable on the Public page and never verified, so it
+-- carries over as a candidate rather than as the owner's verified address.
+-- NULLIF because the dropped column defaulted to '' rather than NULL on the
+-- Public page, and an empty candidate is one nobody can ever confirm.
+UPDATE users
+SET pending_email = NULLIF(
+    (SELECT email FROM identities WHERE is_default = TRUE LIMIT 1), '')
+WHERE role = 'owner';
+
+-- Every users.email written so far is either the synthetic owner@<domain> or an
+-- unverified copy of identities.email. Neither may be asserted as verified, so
+-- every shard lands with no verified address.
+UPDATE users SET email = NULL WHERE role = 'owner';
+
+-- An identity is the shard's public profile, published unauthenticated by
+-- GET /public/meta/whoareyou; a personal address has no business there.
+ALTER TABLE identities DROP COLUMN email;

shard_core/data_model/identity.py

@@ -1,7 +1,6 @@
 from typing import Optional
 
-from email_validator import validate_email, EmailNotValidError
-from pydantic import field_validator, BaseModel, computed_field
+from pydantic import BaseModel, computed_field
 
 from shard_core.service import crypto
 from shard_core.settings import settings
@@ -10,34 +9,20 @@
 class Identity(BaseModel):
     id: str
     name: str
-    email: Optional[str] = None
     description: Optional[str] = None
     private_key: str
     is_default: bool = False
 
     def __str__(self):
         return f"Identity[{self.short_id}, {self.name}]"
 
-    @field_validator("email")
     @classmethod
-    def validate_email(cls, v):
-        if v:
-            try:
-                validate_email(v)
-            except EmailNotValidError as e:
-                raise ValueError(f"invalid email: {e}") from e
-        return v
-
-    @classmethod
-    def create(
-        cls, name: str, description: str = None, email: str = None
-    ) -> "Identity":
+    def create(cls, name: str, description: str = None) -> "Identity":
         private_key = crypto.PrivateKey()
         return Identity(
             id=private_key.get_public_key().to_hash_id(),
             name=name,
             description=description,
-            email=email,
             private_key=private_key.to_bytes().decode(),
         )
 
@@ -86,7 +71,6 @@ def from_identity(cls, identity: Identity):
 class OutputIdentity(BaseModel):
     id: str
     name: str
-    email: Optional[str] = None
     description: Optional[str] = None
     is_default: bool
     public_key_pem: str
@@ -96,15 +80,4 @@ class OutputIdentity(BaseModel):
 class InputIdentity(BaseModel):
     id: Optional[str] = None
     name: Optional[str] = ""
-    email: Optional[str] = ""
     description: Optional[str] = ""
-
-    @field_validator("email")
-    @classmethod
-    def validate_email(cls, v):
-        if v:
-            try:
-                validate_email(v)
-            except EmailNotValidError as e:
-                raise ValueError(f"invalid email: {e}") from e
-        return v

shard_core/data_model/user.py

@@ -2,7 +2,8 @@
 from enum import Enum
 from typing import Optional
 
-from pydantic import BaseModel
+from email_validator import validate_email, EmailNotValidError
+from pydantic import BaseModel, Field, field_validator
 
 
 class Role(str, Enum):
@@ -16,9 +17,52 @@ class User(BaseModel):
     username: str
     display_name: str
     email: Optional[str] = None
+    pending_email: Optional[str] = None
+    email_token_hash: Optional[str] = None
+    email_token_expires: Optional[datetime] = None
     role: Role = Role.MEMBER
     disabled: bool = False
     created: Optional[datetime] = None
 
     def __str__(self):
         return f"User[{self.id}, {self.username}]"
+
+
+class OutputUser(BaseModel):
+    id: int
+    username: str
+    display_name: str
+    email: Optional[str] = None
+    pending_email: Optional[str] = None
+    role: Role
+
+    @classmethod
+    def from_user(cls, user: User) -> "OutputUser":
+        return cls(**user.model_dump())
+
+
+class InputUser(BaseModel):
+    display_name: Optional[str] = Field(default=None, max_length=200)
+    email: Optional[str] = Field(default=None, max_length=254)
+
+    @field_validator("email")
+    @classmethod
+    def validate_email(cls, v):
+        """Reject anything that is not an address; `null` is how you clear one.
+
+        An empty string is a rejection, not a clear — a form that blanks its
+        field must not take the notify-then-clear path by accident.
+        """
+        if v is None:
+            return None
+        try:
+            # No deliverability check: it is a blocking DNS query on the event
+            # loop, and an MX record says nothing about who reads the mailbox.
+            validated = validate_email(
+                v, check_deliverability=False, allow_smtputf8=False
+            )
+        except EmailNotValidError as e:
+            raise ValueError(f"invalid email: {e}") from e
+        # the normalized form is what gets asserted as verified to third
+        # parties, so store that rather than whatever casing was typed
+        return validated.normalized

shard_core/database/db_snapshot.py

@@ -101,13 +101,41 @@ async def restore_db_snapshot():
             if not rows:
                 continue
             col_types = await _column_types(conn, table)
+            _warn_about_dropped_columns(table, rows[0], col_types)
             for row in rows:
                 await _insert_row(conn, table, row, col_types)
             await _reset_sequences(conn, table, list(col_types))
             restored += len(rows)
+        await _demote_unverified_owner_email(conn, snapshot)
     log.info(f"restored {restored} rows from DB snapshot")
 
 
+async def _demote_unverified_owner_email(conn: AsyncConnection, snapshot: dict):
+    """Apply the 0005 migration's rule to a snapshot that predates it.
+
+    Before 0005 the owner's users.email was the synthetic owner@<domain> or an
+    unverified copy of identities.email, and restoring it as-is would hand the
+    OIDC provider an address it would assert as verified. A snapshot from that
+    era is recognisable by its identity rows still carrying an email column.
+    """
+    identities = snapshot.get("identities") or []
+    if not any("email" in row for row in identities):
+        return
+    candidate = next(
+        (
+            row["email"]
+            for row in identities
+            if row.get("is_default") and row.get("email")
+        ),
+        None,
+    )
+    await conn.execute(
+        "UPDATE users SET email = NULL, pending_email = %s WHERE role = 'owner'",
+        (candidate,),
+    )
+    log.info("restored owner address from a pre-0005 snapshot as unverified")
+
+
 async def _list_data_tables(conn: AsyncConnection) -> list[str]:
     async with conn.cursor() as cur:
         await cur.execute(
@@ -172,7 +200,9 @@ async def _column_types(conn: AsyncConnection, table: str) -> dict[str, str]:
 async def _insert_row(
     conn: AsyncConnection, table: str, row: dict, col_types: dict[str, str]
 ):
-    values = {c: _adapt_value(v, col_types.get(c)) for c, v in row.items()}
+    values = {
+        c: _adapt_value(v, col_types[c]) for c, v in row.items() if c in col_types
+    }
     columns = list(values)
     query = sql.SQL(
         "INSERT INTO {table} ({columns}) VALUES ({placeholders}) "
@@ -185,6 +215,15 @@ async def _insert_row(
     await conn.execute(query, values)
 
 
+def _warn_about_dropped_columns(table: str, row: dict, col_types: dict[str, str]):
+    """A snapshot written by an older version carries columns this schema has
+    since dropped. _insert_row skips them — inserting them would abort the whole
+    restore — but a silent skip on a disaster-recovery path deserves a line."""
+    dropped = [c for c in row if c not in col_types]
+    if dropped:
+        log.warning(f"ignoring columns no longer in {table}: {', '.join(dropped)}")
+
+
 def _adapt_value(value, data_type: str | None):
     if value is None:
         return None

shard_core/database/identities.py

@@ -3,7 +3,7 @@
 from psycopg import AsyncConnection
 from psycopg.rows import dict_row
 
-_UPDATABLE_COLUMNS = {"name", "email", "description", "private_key", "is_default"}
+_UPDATABLE_COLUMNS = {"name", "description", "private_key", "is_default"}
 
 
 async def get_all(conn: AsyncConnection) -> list[dict]:
@@ -35,8 +35,8 @@ async def search_by_name(conn: AsyncConnection, name: str) -> list[dict]:
 
 
 async def insert(conn: AsyncConnection, identity: dict) -> dict:
-    sql: LiteralString = """INSERT INTO identities (id, name, email, description, private_key, is_default)
-        VALUES (%(id)s, %(name)s, %(email)s, %(description)s, %(private_key)s, %(is_default)s)
+    sql: LiteralString = """INSERT INTO identities (id, name, description, private_key, is_default)
+        VALUES (%(id)s, %(name)s, %(description)s, %(private_key)s, %(is_default)s)
         RETURNING *"""
     async with conn.cursor(row_factory=dict_row) as cur:
         await cur.execute(sql, identity)

shard_core/database/tinydb_migration.py

@@ -29,7 +29,7 @@ def _jsonb(value):
 _TINYDATE_PREFIX = "{TinyDate}:"
 
 # Columns that exist in the DB for each table — used to filter out computed fields
-_IDENTITY_COLUMNS = {"id", "name", "email", "description", "private_key", "is_default"}
+_IDENTITY_COLUMNS = {"id", "name", "description", "private_key", "is_default"}
 _INSTALLED_APP_COLUMNS = {"name", "installation_reason", "status", "last_access"}
 _TERMINAL_COLUMNS = {"id", "name", "icon", "last_connection"}
 _PEER_COLUMNS = {"id", "name", "public_bytes_b64", "is_reachable"}
@@ -71,8 +71,8 @@ async def migrate_tinydb_data():
 
     async with db_conn() as conn:
         await _migrate_kv_store(conn, data.get("_default", {}))
-        await _migrate_identities(conn, data.get("identities", {}))
-        owner_id = await _ensure_owner_user(conn)
+        pending_email = await _migrate_identities(conn, data.get("identities", {}))
+        owner_id = await _ensure_owner_user(conn, pending_email)
         await _migrate_installed_apps(conn, data.get("installed_apps", {}))
         await _migrate_terminals(conn, data.get("terminals", {}), owner_id)
         await _migrate_peers(conn, data.get("peers", {}))
@@ -96,16 +96,25 @@ async def _migrate_kv_store(conn: AsyncConnection, records: dict):
     log.info(f"migrated {len(records)} kv_store entries")
 
 
-async def _migrate_identities(conn: AsyncConnection, records: dict):
+async def _migrate_identities(conn: AsyncConnection, records: dict) -> str | None:
+    """Insert the identities and return the default one's TinyDB-era address.
+
+    That address was never verified, so it is handed to the owner user as a
+    candidate rather than written to the identity, which no longer holds one.
+    """
+    default_email = None
     for record in records.values():
+        if record.get("is_default") and record.get("email"):
+            default_email = record["email"]
         filtered = _filter_keys(record, _IDENTITY_COLUMNS)
         await conn.execute(
-            """INSERT INTO identities (id, name, email, description, private_key, is_default)
-               VALUES (%(id)s, %(name)s, %(email)s, %(description)s, %(private_key)s, %(is_default)s)
+            """INSERT INTO identities (id, name, description, private_key, is_default)
+               VALUES (%(id)s, %(name)s, %(description)s, %(private_key)s, %(is_default)s)
                ON CONFLICT (id) DO NOTHING""",
             filtered,
         )
     log.info(f"migrated {len(records)} identities")
+    return default_email
 
 
 async def _migrate_installed_apps(conn: AsyncConnection, records: dict):
@@ -122,17 +131,22 @@ async def _migrate_installed_apps(conn: AsyncConnection, records: dict):
     log.info(f"migrated {len(records)} installed apps")
 
 
-async def _ensure_owner_user(conn: AsyncConnection) -> int | None:
+async def _ensure_owner_user(
+    conn: AsyncConnection, pending_email: str | None
+) -> int | None:
     """Terminals require a user (NOT NULL); create the owner from the just-
     migrated default identity, mirroring the 0002 migration's backfill."""
     cur = await conn.execute("SELECT id FROM users WHERE role = 'owner'")
     row = await cur.fetchone()
     if row:
         return row[0]
-    cur = await conn.execute("""INSERT INTO users (username, display_name, email, role)
-           SELECT 'owner', COALESCE(name, 'Shard Owner'), email, 'owner'
+    cur = await conn.execute(
+        """INSERT INTO users (username, display_name, pending_email, role)
+           SELECT 'owner', COALESCE(name, 'Shard Owner'), %s, 'owner'
            FROM identities WHERE is_default = TRUE
-           RETURNING id""")
+           RETURNING id""",
+        (pending_email,),
+    )
     row = await cur.fetchone()
     return row[0] if row else None