filter responses, overhaul filter configuration

This adds an ability to send responses to filtered messages either back
to (just) the user who sent them (via a whisper), or to the whole room.

This allows configuration of the message (or messages, for random
selection), profile name, and uses a (simple) templating mechanism for
the message so that you can (somewhat) personalize it for the user.

This builds off PR #129 for language filtering, but revamps how
languages are specified and overridden with a mechanism that we can use
in the future for other room-specific settings, and that doesn't rely on
room ids (but rather tokens).

Other changes included here:

- You can now control whether or not mods/admins are affected by
  filtering via a new `filter_mods` setting.

- profanity filtering can be room specific.

- alphabet filtering can be room specific (revamping how PR #129 did
  room-specific settings).

- alphabet filtering can be configurably silent or not, like profanity
  filtering.

- added a new, separate example .ini file showing how to set things up
  with room-specific settings.

Author: jagerman

sogs.ini.filter-sample

@@ -0,0 +1,119 @@
+; This file describes the settings you can use for advanced filtering controls.
+;
+; Note that when configuring this, it does *not* go in a separate file but rather in your active
+; sogs.ini configuration file.  (Since everything goes here in a separate section, it doesn't matter
+; where in sogs.ini you add it).
+
+
+;
+; Room-specific filtering
+;
+; To set filtration rules for a specific room you add a [room:TOKEN] section and then set the
+; rules that should apply to this specific room.  For example, to enable the profanity filter and
+; disallow (only) cyrillic characters in the room with token 'sudoku' you would add:
+;
+;[room:sudoku]
+;profanity_filter=yes
+;profanity_silent=yes
+;alphabet_filters=cyrillic
+;
+; This overrides the default from the main [messages] config section for any given keys, so it can
+; be used to replace or change the rules that apply to a given room.  Currently only the
+; profanity_filter, profanity_silent, alphabet_filters can be overridden in this way.
+
+;
+; Filtration responses
+;
+; When a message is filtered because of the profanity or alphabet filtrations SOGS can optionally
+; send a reply in the room; this reply can either be visible to everyone, or just to the specific
+; user.  To enable such a reply, add a filter section here: the section name consists of
+; 'filter:TYPE:ROOM' where TYPE and ROOM are the filtration type and room token, or '*' to match all
+; types/rooms.
+;
+; Section names for all filtered messages:
+;[filter:*:*]
+;
+; Section names for a particular filtration type:
+;[filter:*:profanity]
+;[filter:*:alphabet]
+;
+; The "type" can also be a specific language:
+;[filter:*:arabic]
+;[filter:*:cyrillic]
+; etc.
+;
+; Room-specific filtration section names:
+;
+;[filter:fishing:*]
+;[filter:sudoku:profanity]
+;
+; If using both '*' and specific values, the value from the more specific section will be used where
+; present.
+;
+; Within this section there are currently three settings:
+;
+; - reply -- the body of a reply to send (see details below).  If omitted or empty then no reply
+;   will be sent.
+; - profile_name -- the profile name to use in that reply.
+; - public -- whether the reply should be seen by everyone or just the poster.  The default is 'no'
+;   (i.e. only the user will see the reply).
+;
+; The `reply` value should be specified on a single line of the config, and supports the following
+; substitutions:
+;
+; \@ - the profile name, in @tag form, of the poster whose message was declined.
+; \p - the profile name in plain text.
+; \r - the name of the room
+; \t - the token of the room
+; \n - a line break
+; \\ - a literal \ character
+;
+; You can also randomize among multiple responses by specifying multiple lines in the config: each
+; additional line must be indented in the .ini file to be properly recognized.
+;
+; For example if you use this config:
+;
+
+[messages]
+profanity_filter=yes
+profanity_silent=yes
+alphabet_filters=arabic cyrillic
+alphabet_silent=yes
+
+[room:sailors]
+profanity_filter=no
+
+[filter:*:*]
+profile_name=LanguagePolice
+reply=Hi \@, I'm afraid your message couldn't be sent: \r is English-only!
+
+[filter:*:profanity]
+profile_name=Swear Jar
+reply=Whoa there, \@!  That language is too strong for the \r group!  Try the Sailors group instead.
+
+[filter:sudoku:profanity]
+profile_name=Bot45
+public=yes
+reply=\@ got a little too enthusiastic today with their solve.  Maybe someone can assist?
+ Uh oh, I think \@ has two ３s in the same row!
+ I think \@'s sudoku broke 😦
+
+; then arabic/cyrillic/person would be blocked everywhere, profanity would be blocked everywhere
+; except the 'sailors' room, and when a message is blocked you would get a message such as one of
+; the following depending on the room and the rule applied:
+;
+;
+; (LanguagePolice)
+; Hi @Foreignsailor1988, I'm afraid your message couldn't be set: Salty Sailors is English-only!
+;
+;
+; (Swear Jar)
+; Whoa there @87yearoldgrandma!  That language is too strong for the Cuddly Kittens group!  Try the Sailors group instead.
+;
+;
+; (Bot45); [one of the following would be sent randomly, visible to everyone in the group]
+; @87yearoldgrandma got a little too enthusiastic today with their solve.  Maybe someone can assist?
+;
+; Uh oh, I think @87yearoldgrandma has two ３s in the same row!
+;
+; I think @87yearoldgrandma's sudoku broke 😦

sogs.ini.sample

@@ -93,8 +93,7 @@
 
 
 ; Whether we should pass words through a profanity filter, rejecting messages that contain profane
-; words (and common permutations of those words).  (Note that profanity filters are never applied to
-; room or server admins).
+; words (and common permutations of those words).
 ;
 ;profanity_filter = no
 
@@ -115,19 +114,31 @@
 ;
 ;profanity_custom =
 
-; Whether we should reject messages that use a particular alphabet.
+; Whether we should reject messages that use a particular alphabet.  This is a space or
+; comma-separated list of alphabet names; posts with characters in the given language ranges will be
+; blocked (unless posted by a mod/admin).  Currently supported are: arabic, cyrillic, and persian
+; (note that persian is included within arabic).
 ;
-;alphabet_filters = [ arabic, cyrillic, persian ]
-;alphabet_filters = []
+; *This* setting is the default setting for all rooms, but you can also make room-specific filtering
+; (see sogs.ini.filter-sample for details).
+;
+;alphabet_filters =
+
+
+; Whether the alphabet filter should silently drop (true) or return an error (false).
+;
+;alphabet_silent = yes
 
 
-; If we reject messages written in a given alphabet, we should still allow them in
-; specific rooms. A list of whitelisted room ids can be given here, as returned by
-; `SELECT * FROM rooms;`. An empty list means to reject from all rooms.
+; Whether the above filters should be applied to moderators and admins (=yes) or not (=no).  The
+; default is no, that is, mods and admins messages are not filtered by default.
 ;
-;alphabet_whitelist_arabic = []
-;alphabet_whitelist_cyrillic = []
-;alphabet_whitelist_persian = []
+;filter_mods = no
+
+
+; The profanity and alphabet filters can be controlled on a per-room setting (which overrides the
+; global default set above) and can have automated responses sent by the SOGS server.  For details
+; and examples see the sogs.ini.filter-sample file.
 
 
 [web]

sogs/config.py

@@ -32,14 +32,15 @@
 PROFANITY_FILTER = False
 PROFANITY_SILENT = True
 PROFANITY_CUSTOM = None
-ALPHABET_FILTERS = []
-ALPHABET_WHITELIST_ARABIC = []
-ALPHABET_WHITELIST_CYRILLIC = []
-ALPHABET_WHITELIST_PERSIAN = []
+ALPHABET_FILTERS = set()
+ALPHABET_SILENT = True
+FILTER_MODS = False
 REQUIRE_BLIND_KEYS = False
 TEMPLATE_PATH = 'templates'
 STATIC_PATH = 'static'
 UPLOAD_PATH = 'uploads'
+ROOM_OVERRIDES = {}
+FILTER_SETTINGS = {}
 
 # Will be true if we're running as a uwsgi app, false otherwise; used where we need to do things
 # only in one case or another (e.g. database initialization only via app mode).
@@ -87,11 +88,8 @@ def days_to_seconds(v):
     def days_to_seconds_or_none(v):
         return days_to_seconds(v) if v else None
 
-    def list_of_strs(v):
-        return re.split('[,\s]+', value[1:-1].strip())
-
-    def list_of_ints(v):
-        return [int(i) for i in list_of_strs(v)]
+    def set_of_strs(v):
+        return {s for s in re.split('[,\\s]+', v) if s != ''}
 
     truthy = ('y', 'yes', 'Y', 'Yes', 'true', 'True', 'on', 'On', '1')
     falsey = ('n', 'no', 'N', 'No', 'false', 'False', 'off', 'Off', '0')
@@ -100,6 +98,25 @@ def list_of_ints(v):
     def bool_opt(name):
         return (name, lambda x: x in booly, lambda x: x in truthy)
 
+    reply_fields = {
+        r'\@': '{profile_at}',
+        r'\p': '{profile_name}',
+        r'\r': '{room_name}',
+        r'\t': '{room token}',
+        '{': '{{',
+        '}': '}}',
+        r'\\': '\\',
+        r'\n': '\n',
+    }
+    reply_fields_re = '(?:' + '|'.join(re.escape(k) for k in reply_fields.keys()) + ')'
+
+    def reply_to_format(v):
+        return [
+            re.sub(reply_fields_re, lambda x: reply_fields[x.group(0)], reply)
+            for reply in v.split("\n")
+            if reply != ''
+        ]
+
     # Map of: section => { param => ('GLOBAL', test lambda, value lambda) }
     # global is the string name of the global variable to set
     # test lambda returns True/False for validation (if None/omitted, accept anything)
@@ -136,10 +153,9 @@ def bool_opt(name):
             'profanity_filter': bool_opt('PROFANITY_FILTER'),
             'profanity_silent': bool_opt('PROFANITY_SILENT'),
             'profanity_custom': ('PROFANITY_CUSTOM', path_exists, val_or_none),
-            'alphabet_filters': ('ALPHABET_FILTERS', None, list_of_strs),
-            'alphabet_whitelist_arabic': ('ALPHABET_WHITELIST_ARABIC', None, list_of_ints),
-            'alphabet_whitelist_cyrillic': ('ALPHABET_WHITELIST_CYRILLIC', None, list_of_ints),
-            'alphabet_whitelist_persian': ('ALPHABET_WHITELIST_PERSIAN', None, list_of_ints),
+            'alphabet_filters': ('ALPHABET_FILTERS', None, set_of_strs),
+            'alphabet_silent': bool_opt('ALPHABET_SILENT'),
+            'filter_mods': bool_opt('FILTER_MODS'),
         },
         'web': {
             'template_path': ('TEMPLATE_PATH', path_exists, val_or_none),
@@ -148,33 +164,73 @@ def bool_opt(name):
         'log': {'level': ('LOG_LEVEL',)},
     }
 
-    for s in cp.sections():
-        if s not in setting_map:
-            logger.warning(f"Ignoring unknown section [{s}] in {conf_ini}")
-            continue
-        for opt in cp[s]:
-            if opt not in setting_map[s]:
-                logger.warning(f"Ignoring unknown config setting [{s}].{opt} in {conf_ini}")
-                continue
+    room_setting_map = {
+        'profanity_filter': bool_opt('profanity_filter'),
+        'profanity_silent': bool_opt('profanity_silent'),
+        'alphabet_filters': ('alphabet_filters', None, set_of_strs),
+    }
+
+    filter_setting_map = {
+        'public': bool_opt('public'),
+        'profile_name': ('profile_name',),
+        'reply': ('reply', None, reply_to_format),
+    }
 
-            value = cp[s][opt]
-            conf = setting_map[s][opt]
+    def parse_option(fields, s, opt, *, room=None, filt=None):
+        conf_type = 'room-specific ' if room else 'filter ' if filt else ''
+        if opt not in fields:
+            logger.warning(f"Ignoring unknown {conf_type} config setting [{s}].{opt} in {conf_ini}")
+            return
+        conf = fields[opt]
+        value = cp[s][opt]
 
-            assert isinstance(conf, tuple) and 1 <= len(conf) <= 3
+        assert isinstance(conf, tuple) and 1 <= len(conf) <= 3
+        if not room and not filt:
             assert conf[0] in globals()
 
-            logger.debug(f"Loaded config setting [{s}].{opt}={value}")
+        logger.debug(f"Loaded {'room-specific ' if room else ''}config setting [{s}].{opt}={value}")
 
-            if len(conf) >= 2 and conf[1]:
-                if not conf[1](value):
-                    raise RuntimeError(f"Invalid value [{s}].{opt}={value} in {conf_ini}")
+        if len(conf) >= 2 and conf[1]:
+            if not conf[1](value):
+                raise RuntimeError(f"Invalid value [{s}].{opt}={value} in {conf_ini}")
 
-            if len(conf) >= 3 and conf[2]:
-                value = conf[2](value)
+        if len(conf) >= 3 and conf[2]:
+            value = conf[2](value)
 
+        if room:
+            logger.debug(f"Set config.ROOM_OVERRIDES[{room}][{conf[0]}] = {value}")
+            ROOM_OVERRIDES[room][conf[0]] = value
+        elif filt:
+            logger.debug(f"Set config.FILTER_SETTINGS[{filt[0]}][{filt[1]}][{conf[0]}] = {value}")
+            FILTER_SETTINGS.setdefault(filt[0], {}).setdefault(filt[1], {})[conf[0]] = value
+        else:
             logger.debug(f"Set config.{conf[0]} = {value}")
             globals()[conf[0]] = value
 
+    for s in cp.sections():
+        if len(s) > 5 and s.startswith('room:'):
+            token = s[5:]
+            if token not in ROOM_OVERRIDES:
+                ROOM_OVERRIDES[token] = {}
+            for opt in cp[s]:
+                parse_option(room_setting_map, s, opt, room=token)
+
+        elif s.startswith('filter:'):
+            filt = s.split(':')[1:]
+            if len(filt) != 2:
+                raise RuntimeError(
+                    f"Invalid filter section [{s}] in {conf_ini}: expected [filter:TYPE:ROOM]"
+                )
+            for opt in cp[s]:
+                parse_option(filter_setting_map, s, opt, filt=filt)
+
+        elif s in setting_map:
+            for opt in cp[s]:
+                parse_option(setting_map[s], s, opt)
+
+        else:
+            logger.warning(f"Ignoring unknown section [{s}] in {conf_ini}")
+
 
 try:
     load_config()

sogs/crypto.py

@@ -72,11 +72,11 @@ def verify_sig_from_pk(data, sig, pk):
     return VerifyKey(pk).verify(data, sig)
 
 
-_server_signkey = SigningKey(_privkey_bytes)
+server_signkey = SigningKey(_privkey_bytes)
+server_verifykey = server_signkey.verify_key
 
-server_verify = _server_signkey.verify_key.verify
-
-server_sign = _server_signkey.sign
+server_verify = server_verifykey.verify
+server_sign = server_signkey.sign
 
 
 def server_encrypt(pk, data):