New endpoints: info, polling, fetching, permissions, pinning

Implements a bunch of new sogs endpoints, mostly as described by the api
spec (with some changes applied to the spec, where needed):

- `/rooms` -- returns list of readable rooms, including various room
  metadata plus derived permission of the user in the rooms

- `/room/<token>` -- same as above, but for a single room

- `/room/<token>/pollInfo/<id>` -- polls a room for any metadata updates
  applied since `<id>`.  If the room's current metadata tracker
  (`info_updates`) is unchanged from `<id>` then this returns a light
  poll response with just active_users and current user permissions in
  the room; otherwise it includes that plus a `details` key containing
  the full room metadata (as would be returned by `/room/<token>`).

Message fetching endpoints:
- `/room/<token>/message/1234` retrieves a single message with the given
  id.
- `/room/<token>/messages/recent` retrieves the most recent 100 messages
  in a room.
- `/room/<token>/messages/before/123` retrieves the most recent 100
  message events (i.e. post, edits, deletions) immediately preceding the
  *sequence value* 123.  (This is not a message id, but rather the
  messages `seqno` value).
- `/room/<token>/messages/since/456` retrieves the next 100 message
  events following `seqno=456`.
- all of the above except for the single message endpoint support a
  `?limit=42` parameter that allows values up to 256.

Message pinning (these all require *admin* privileges, not just
moderator permissions):
- POST `/room/<token>/pin/123` pins the message with id=123 to this
  room.  (Note that this adds a pinned; if there are other pinned
  messages you end up with multiple pinned messages).
- POST `/room/<token>/unpin/123` does what you'd expect.
- POST `/room/<token>/unpin/all` does what you'd expect.

- `/room/<token>/pollInfo/<id>` and `/room/<token>/messages/since/456`
  together replace legacy SOGS compact_poll *and* the need for periodic
  room info fetching.

  Ideally you'd poll everything in once by throwing these into a
  sequence request such as:

      [{'method': 'GET', 'path': '/room/room1/pollInfo/123'},
       {'method': 'GET', 'path': '/room/room2/pollInfo/456'},
       {'method': 'GET', 'path': '/room/room1/messages/since/789'},
       {'method': 'GET', 'path': '/room/room2/messages/since/1011'}]

    The first two give you room information such as the number of active
    users and your current permissions, and when room details change,
    the full room metadata (name, description, pinned_messages, image,
    moderators list, etc.)

    The second two give you new post updates (i.e. new posts, edits,
    deletions) for the two rooms.

    (It may sometimes be necessary to submit extra fetch requests if the
    messages-since endpoints give back a full set of 100 message
    updates).

Other notes:

- both the polling and full version of room details include the current
  permission keys,
  `read`/`write`/`upload`/`moderator`/`admin`/`global_moderator`/`global_admin`,
  which clients can use to update the UI as appropriate.

- Room "updates" and post "updated" are renamed to "message_sequence"
  and "seqno", respectively, to better describe what they are.  That is,
  these define the sequence of updates that a client needs: a new post
  counts as an update, but so does an edit to and existing post and a
  deletion.  The old name was rather confusing since there is also an
  `info_updates` on rooms which tracks changes to room details like
  name/image/etc.

- Simplified the active_users to report a single value plus the cutoff
  for that value (which is sogs-admin configurable).  Previously the API
  specified that it would return a default plus several different
  values, but that seems needlessly complex.  The cutoff, however, might
  be useful for an advanced page to know the time frame beyond which
  users are no longer considered "active".

- Added notes to file expiry that the short expiry until association
  with a post isn't implemented yet.

- Fix the user_permissions view not returning true for read/write/upload
  for moderator permissions in rooms if the permissions were not
  specifically applied to the moderator.  (Moderators/admins always have
  those permissions even if the defaults are false).

- Split up the moderators/admins list in room info so that regular users
  can distinguish between room moderators and admins, and so that
  mods/admins can distinguish between hidden mods/admins.

- Add extensive unit tests for everything above

Author: jagerman

api.yaml

@@ -118,6 +118,7 @@ def database_init():
         update_message_views,
         create_message_details_deleter,
         check_for_hacks,
+        seqno_etc_updates,
     ):
         if migrate(conn):
             changes = True
@@ -168,6 +169,7 @@ def add_new_columns(conn):
     for table, cols in new_table_cols.items():
         for name, definition in cols.items():
             if name not in metadata.tables[table].c:
+                logging.warning(f"DB migration: Adding new column {table}.{name}")
                 conn.execute(f"ALTER TABLE {table} ADD COLUMN {name} {definition}")
                 added = True
 
@@ -177,6 +179,7 @@ def add_new_columns(conn):
 def add_new_tables(conn):
     added = False
     if 'user_request_nonces' not in metadata.tables:
+        logging.warning("DB migration: Adding new table user_request_nonces")
         if engine.name == 'sqlite':
             conn.execute(
                 """
@@ -212,6 +215,7 @@ def add_new_tables(conn):
 def update_message_views(conn):
     if engine.name != "sqlite":
         if any(x not in metadata.tables['message_metadata'].c for x in ('whisper_to', 'filtered')):
+            logging.warning("DB migration: replacing message_metadata/message_details views")
             conn.execute("DROP VIEW IF EXISTS message_metadata")
             conn.execute("DROP VIEW IF EXISTS message_details")
             conn.execute(
@@ -233,6 +237,7 @@ def update_message_views(conn):
             )
 
             return True
+        # else: don't worry about this for postgresql because initial pg support had the fix
 
     return False
 
@@ -271,8 +276,10 @@ def check_for_hacks(conn):
         # drop it.
         n_fid_hacks = conn.execute("SELECT COUNT(*) FROM file_id_hacks").first()[0]
         if n_fid_hacks == 0:
+            logging.warning("Dropping file_id_hacks old sogs import table (no longer required)")
             metadata.tables['file_id_hacks'].drop(engine)
         else:
+            logging.warning("Keeping file_id_hacks old sogs import table (still required)")
             global HAVE_FILE_ID_HACKS
             HAVE_FILE_ID_HACKS = True
 
@@ -286,6 +293,187 @@ def check_for_hacks(conn):
         pass
 
 
+def seqno_etc_updates(conn):
+    """
+    Rename rooms.updates/messages.updated to rooms.message_sequence/messages.seqno for better
+    disambiguation with rooms.info_updates.
+
+    This also does various other changes/fixes that came at the same time as the column rename:
+
+    - remove "updated" from and add "pinned_by"/"pinned_at" to pinned_messages
+    - recreate the pinned_messages table and triggers because we need several changes:
+        - add trigger to unpin a message when the message is deleted
+        - remove "updates" (now message_sequence) updates from room metadata update trigger
+        - add AFTER UPDATE trigger to properly update room metadata counter when re-pinning an
+          existing pinned message
+    - fix user_permissions view to return true for read/write/upload to true for moderators
+    """
+
+    if 'seqno' in metadata.tables['messages'].c:
+        return False
+
+    with transaction(dbconn=conn):
+        logging.warning("Applying message_sequence renames")
+        conn.execute("ALTER TABLE rooms RENAME COLUMN updates TO message_sequence")
+        conn.execute("ALTER TABLE messages RENAME COLUMN updated TO seqno")
+
+        # We can't insert the required pinned_messages because we don't have the pinned_by user, but
+        # that isn't a big deal since we didn't have any endpoints for pinned messsages before this
+        # anyway, so we just recreate the whole thing (along with triggers which we also need to
+        # update/fix)
+        logging.warning("Recreating pinned_messages table")
+        conn.execute("DROP TABLE pinned_messages")
+        if engine.name == 'sqlite':
+            conn.execute(
+                """
+CREATE TABLE pinned_messages (
+    room INTEGER NOT NULL REFERENCES rooms(id) ON DELETE CASCADE,
+    message INTEGER NOT NULL REFERENCES rooms(id) ON DELETE CASCADE,
+    pinned_by INTEGER NOT NULL REFERENCES users(id),
+    pinned_at FLOAT NOT NULL DEFAULT ((julianday('now') - 2440587.5)*86400.0), /* unix epoch when pinned */
+    PRIMARY KEY(room, message)
+)
+"""  # noqa: E501
+            )
+            conn.execute(
+                """
+CREATE TRIGGER messages_after_delete AFTER UPDATE OF data ON messages
+FOR EACH ROW WHEN NEW.data IS NULL AND OLD.data IS NOT NULL
+BEGIN
+    -- Unpin if we deleted a pinned message:
+    DELETE FROM pinned_messages WHERE message = OLD.id;
+END
+"""
+            )
+            conn.execute(
+                """
+CREATE TRIGGER room_metadata_pinned_add AFTER INSERT ON pinned_messages
+FOR EACH ROW
+BEGIN
+    UPDATE rooms SET info_updates = info_updates + 1 WHERE id = NEW.room;
+END
+"""
+            )
+            conn.execute(
+                """
+CREATE TRIGGER room_metadata_pinned_update AFTER UPDATE ON pinned_messages
+FOR EACH ROW
+BEGIN
+    UPDATE rooms SET info_updates = info_updates + 1 WHERE id = NEW.room;
+END
+"""
+            )
+            conn.execute(
+                """
+CREATE TRIGGER room_metadata_pinned_remove AFTER DELETE ON pinned_messages
+FOR EACH ROW
+BEGIN
+    UPDATE rooms SET info_updates = info_updates + 1 WHERE id = OLD.room;
+END
+"""
+            )
+
+            logging.warning("Fixing user_permissions view")
+            conn.execute("DROP VIEW IF EXISTS user_permissions")
+            conn.execute(
+                """
+CREATE VIEW user_permissions AS
+SELECT
+    rooms.id AS room,
+    users.id AS user,
+    users.session_id,
+    CASE WHEN users.banned THEN TRUE ELSE COALESCE(user_permission_overrides.banned, FALSE) END AS banned,
+    CASE WHEN users.moderator THEN TRUE ELSE COALESCE(user_permission_overrides.read, rooms.read) END AS read,
+    CASE WHEN users.moderator THEN TRUE ELSE COALESCE(user_permission_overrides.write, rooms.write) END AS write,
+    CASE WHEN users.moderator THEN TRUE ELSE COALESCE(user_permission_overrides.upload, rooms.upload) END AS upload,
+    CASE WHEN users.moderator THEN TRUE ELSE COALESCE(user_permission_overrides.moderator, FALSE) END AS moderator,
+    CASE WHEN users.admin THEN TRUE ELSE COALESCE(user_permission_overrides.admin, FALSE) END AS admin,
+    -- room_moderator will be TRUE if the user is specifically listed as a moderator of the room
+    COALESCE(user_permission_overrides.moderator OR user_permission_overrides.admin, FALSE) AS room_moderator,
+    -- global_moderator will be TRUE if the user is a global moderator/admin (note that this is
+    -- *not* exclusive of room_moderator: a moderator/admin could be listed in both).
+    COALESCE(users.moderator OR users.admin, FALSE) as global_moderator,
+    -- visible_mod will be TRUE if this mod is a publicly viewable moderator of the room
+    CASE
+        WHEN user_permission_overrides.moderator OR user_permission_overrides.admin THEN user_permission_overrides.visible_mod
+        WHEN users.moderator OR users.admin THEN users.visible_mod
+        ELSE FALSE
+    END AS visible_mod
+FROM
+    users JOIN rooms LEFT OUTER JOIN user_permission_overrides ON
+        users.id = user_permission_overrides.user AND rooms.id = user_permission_overrides.room
+"""  # noqa: E501
+            )
+
+        else:  # postgresql
+            logging.warning("Recreating pinned_messages table")
+            conn.execute(
+                """
+CREATE TABLE pinned_messages (
+    room BIGINT NOT NULL REFERENCES rooms ON DELETE CASCADE,
+    message BIGINT NOT NULL REFERENCES rooms ON DELETE CASCADE,
+    pinned_by BIGINT NOT NULL REFERENCES users,
+    pinned_at FLOAT NOT NULL DEFAULT (extract(epoch from now())),
+    PRIMARY KEY(room, message)
+);
+
+
+-- Trigger to handle required updates after a message gets deleted (in the SOGS context: that is,
+-- has data set to NULL)
+CREATE OR REPLACE FUNCTION trigger_messages_after_delete()
+RETURNS TRIGGER LANGUAGE PLPGSQL AS $$BEGIN
+    -- Unpin if we deleted a pinned message:
+    DELETE FROM pinned_messages WHERE message = OLD.id;
+    RETURN NULL;
+END;$$;
+CREATE TRIGGER messages_insert_history AFTER UPDATE OF data ON messages
+FOR EACH ROW WHEN (NEW.data IS DISTINCT FROM OLD.data)
+EXECUTE PROCEDURE trigger_messages_insert_history();
+
+CREATE TRIGGER room_metadata_pinned_add AFTER INSERT OR UPDATE ON pinned_messages
+FOR EACH ROW
+EXECUTE PROCEDURE trigger_room_metadata_info_update_new();
+
+CREATE TRIGGER room_metadata_pinned_remove AFTER DELETE ON pinned_messages
+FOR EACH ROW
+EXECUTE PROCEDURE trigger_room_metadata_info_update_old();
+"""
+            )
+
+            logging.warning("Fixing user_permissions view")
+            conn.execute(
+                """
+CREATE OR REPLACE VIEW user_permissions AS
+SELECT
+    rooms.id AS room,
+    users.id AS "user",
+    users.session_id,
+    CASE WHEN users.banned THEN TRUE ELSE COALESCE(user_permission_overrides.banned, FALSE) END AS banned,
+    CASE WHEN users.moderator THEN TRUE ELSE COALESCE(user_permission_overrides.read, rooms.read) END AS read,
+    CASE WHEN users.moderator THEN TRUE ELSE COALESCE(user_permission_overrides.write, rooms.write) END AS write,
+    CASE WHEN users.moderator THEN TRUE ELSE COALESCE(user_permission_overrides.upload, rooms.upload) END AS upload,
+    CASE WHEN users.moderator THEN TRUE ELSE COALESCE(user_permission_overrides.moderator, FALSE) END AS moderator,
+    CASE WHEN users.admin THEN TRUE ELSE COALESCE(user_permission_overrides.admin, FALSE) END AS admin,
+    -- room_moderator will be TRUE if the user is specifically listed as a moderator of the room
+    COALESCE(user_permission_overrides.moderator OR user_permission_overrides.admin, FALSE) AS room_moderator,
+    -- global_moderator will be TRUE if the user is a global moderator/admin (note that this is
+    -- *not* exclusive of room_moderator: a moderator/admin could be listed in both).
+    COALESCE(users.moderator OR users.admin, FALSE) as global_moderator,
+    -- visible_mod will be TRUE if this mod is a publicly viewable moderator of the room
+    CASE
+        WHEN user_permission_overrides.moderator OR user_permission_overrides.admin THEN user_permission_overrides.visible_mod
+        WHEN users.moderator OR users.admin THEN users.visible_mod
+        ELSE FALSE
+    END AS visible_mod
+FROM
+    users CROSS JOIN rooms LEFT OUTER JOIN user_permission_overrides ON
+        (users.id = user_permission_overrides."user" AND rooms.id = user_permission_overrides.room);
+"""  # noqa: E501
+            )
+
+    return True
+
+
 def create_admin_user(dbconn):
     """
     We create a dummy user (with id 0) for system tasks such as changing moderators from
@@ -305,7 +493,7 @@ def create_admin_user(dbconn):
 
 
 if config.DB_URL.startswith('postgresql'):
-    # room.token is a 'citext' (case-insensitive text), which sqlalchemy doesn't recognize out of
+    # rooms.token is a 'citext' (case-insensitive text), which sqlalchemy doesn't recognize out of
     # the box.  Map it to a plain TEXT which is good enough for what we need (if we actually needed
     # to generate this wouldn't suffice: we'd have to use something like the sqlalchemy-citext
     # module).
@@ -342,6 +530,12 @@ def _init_engine(*args, **kwargs):
     if engine.name == "sqlite":
         import sqlite3
 
+        if sqlite3.sqlite_version_info < (3, 25, 0):
+            raise RuntimeError(
+                f"SQLite3 library version {'.'.join(sqlite3.sqlite_version_info)} "
+                "is too old for pysogs (3.25.0+ required)!"
+            )
+
         have_returning = sqlite3.sqlite_version_info >= (3, 35, 0)
 
         @sqlalchemy.event.listens_for(engine, "connect")

sogs/db.py

@@ -34,6 +34,14 @@ def __init__(self, session_id):
         super().__init__(f"No such user: {session_id}")
 
 
+class NoSuchPost(NotFound):
+    """Thrown when attempting to retrieve or reference a post that doesn't exist"""
+
+    def __init__(self, id):
+        self.id = id
+        super().__init__(f"No such post: {id}")
+
+
 class AlreadyExists(RuntimeError):
     """
     Thrown when attempting to create a record (e.g. a Room) that already exists.