sqlite: Remove -DSQLITE_ENABLE_FTS3_TOKENIZER and add tests for compile options (#791)

As noted in the discussion in #562, compiling SQLite with the
-DSQLITE_ENABLE_FTS3_TOKENIZER flag is equivalent to using
`connection.setconfig(sqlite3.SQLITE_DBCONFIG_ENABLE_FTS3_TOKENIZER)` at
runtime. The purpose of this option, in either syntax, is to disable a
security measure to provide backwards compatibility for older code.
Specifically, the `fts3_tokenizer()` function can accept or return a
native-code pointer to a structure containing callback functions, which
makes it an attractive target for SQL injection attacks to escalate to
arbitrary native code execution. The more-secure behavior is to require
the use of bound parameters with this function; the backwards-compatible
behavior allows the function to be called with blob literals or computed
values. Because of a documentation shortcoming, some applications
thought they needed this option on at compile time, and so Debian's
SQLite build, used by e.g. the `python` container on Dockerhub, has it
on. But there is no functionality that is only enabled by having this
option on at compile time. Ideally, applications should use bound
parameters when calling this function. If that code change is hard, they
can alternatively set the option themselves at runtime to preserve
compatibility with existing code, but that still doesn't need anything
turned on at compile time. So the right decision for us is not to enable
this flag at compile time and preserve the secure behavior.

Add a test that `fts3_tokenizer()` is usable with bound parameters but
not with blob literals, and also add tests for a couple of other
preivously-requested SQLite flags for compatibility with other
implementations:

* #309: -DSQLITE_ENABLE_DBSTAT_VTAB
* #449: serialize/deserialize (on by default, was just a compile-time
detection issue)
* #550: -DSQLITE_ENABLE_FTS3_PARENTHESIS

Author: geofft

cpython-unix/build-sqlite.sh

@@ -32,7 +32,6 @@ CFLAGS="${EXTRA_TARGET_CFLAGS} \
     -DSQLITE_ENABLE_DBSTAT_VTAB \
     -DSQLITE_ENABLE_FTS3 \
     -DSQLITE_ENABLE_FTS3_PARENTHESIS \
-    -DSQLITE_ENABLE_FTS3_TOKENIZER \
     -DSQLITE_ENABLE_FTS4 \
     -DSQLITE_ENABLE_FTS5 \
     -DSQLITE_ENABLE_GEOPOLY \

src/verify_distribution.py

@@ -3,6 +3,7 @@
 # file, You can obtain one at https://mozilla.org/MPL/2.0/.
 
 import os
+import struct
 import sys
 import unittest
 
@@ -137,6 +138,45 @@ def test_sqlite(self):
                 cursor.execute(
                     f"CREATE VIRTUAL TABLE test{extension} USING {extension}(a, b, c);"
                 )
+
+        # Test various SQLite flags and features requested / expected by users.
+        # The DBSTAT virtual table shows some metadata about disk usage.
+        # https://www.sqlite.org/dbstat.html
+        self.assertNotEqual(
+            cursor.execute("SELECT COUNT(*) FROM dbstat;").fetchone()[0],
+            0,
+        )
+
+        # The serialize/deserialize API is configurable at compile time.
+        if sys.version_info[0:2] >= (3, 11):
+            self.assertEqual(conn.serialize()[:15], b"SQLite format 3")
+
+        # The "enhanced query syntax" (-DSQLITE_ENABLE_FTS3_PARENTHESIS) allows parenthesizable
+        # AND, OR, and NOT operations. The "standard query syntax" only has OR as a keyword, so we
+        # can test for the difference with a query using AND.
+        # https://www.sqlite.org/fts3.html#_set_operations_using_the_enhanced_query_syntax
+        cursor.execute("INSERT INTO testfts3 VALUES('hello world', '', '');")
+        self.assertEqual(
+            cursor.execute(
+                "SELECT COUNT(*) FROM testfts3 WHERE a MATCH 'hello AND world';"
+            ).fetchone()[0],
+            1,
+        )
+
+        # fts3_tokenizer() takes/returns native pointers. Newer SQLite versions require the use of
+        # bound parameters with this function to avoid the risk of a SQL injection esclating into a
+        # full RCE. This requirement can be disabled at either compile time or runtime for
+        # backwards compatibility. Ensure that the check is enabled (more secure) by default but
+        # applications can still use fts3_tokenize with a bound parameter. See discussion at
+        # https://github.com/astral-sh/python-build-standalone/pull/562#issuecomment-3254522958
+        wild_pointer = struct.pack("P", 0xDEADBEEF)
+        with self.assertRaises(sqlite3.OperationalError) as caught:
+            cursor.execute(
+                f"SELECT fts3_tokenizer('mytokenizer', x'{wild_pointer.hex()}')"
+            )
+        self.assertEqual(str(caught.exception), "fts3tokenize disabled")
+        cursor.execute("SELECT fts3_tokenizer('mytokenizer', ?)", (wild_pointer,))
+
         conn.close()
 
     def test_ssl(self):