Add Lua binding with IEEE 754 compliant float serialization

/examples/simple-lua/ contains a demo script and runner.

Incorporates upstream PR asg017#237 with the following bugfixes:

Extension loading:
- Fix return value check: lsqlite3's load_extension returns true on
  success, not sqlite3.OK (which is 0). Changed from `if ok then` to
  `if ok and result then` to properly detect successful loads.
- Add vec0 naming paths alongside sqlite-vec paths for this fork.

IEEE 754 float serialization (float_to_bytes):
- Switch from half-round-up to round-half-to-even (banker's rounding)
  for IEEE 754 compliance. This prevents systematic bias when
  processing large datasets where half-values accumulate.
- Handle special cases: NaN, Inf, -Inf, and -0.0 which the original
  implementation did not support.
- Fix subnormal number encoding: corrected formula from 2^(exp+126)
  to 2^(exp+127) so minimum subnormal 2^(-149) encodes correctly.
- Add mantissa overflow carry: when rounding causes mantissa >= 2^23,
  carry into exponent field.
- Add exponent overflow handling: values too large now return ±Inf
  instead of producing corrupted output.
- Use epsilon comparison (1e-9) for 0.5 tie detection to handle
  floating-point precision issues.

JSON serialization (serialize_json):
- Error on NaN and Infinity values which are not valid JSON.
- Convert -0.0 to 0.0 for JSON compatibility.

Co-Authored-By: asr1 <[email protected]>
Co-Authored-By: Claude Opus 4.5 <[email protected]>

Author: 3 people

bindings/lua/sqlite_vec.lua

@@ -0,0 +1,201 @@
+-- sqlite_vec.lua Lua 5.1 compatible version with JSON fallback
+local sqlite3 = require("lsqlite3")
+
+local M = {}
+
+-- Function to load extension
+function M.load(db)
+    local possible_paths = {
+        -- vec0 naming (this fork)
+        "../../dist/vec0.so",       -- Linux
+        "../../dist/vec0.dll",      -- Windows
+        "../../dist/vec0.dylib",    -- macOS
+        "./dist/vec0.so",
+        "./dist/vec0.dll",
+        "./dist/vec0.dylib",
+        "../dist/vec0.so",
+        "../dist/vec0.dll",
+        "../dist/vec0.dylib",
+        "vec0",
+        -- sqlite-vec naming (upstream)
+        "../../sqlite-vec.so",
+        "../../sqlite-vec.dll",
+        "../../sqlite-vec.dylib",
+        "./sqlite-vec.so",
+        "./sqlite-vec.dll",
+        "./sqlite-vec.dylib",
+        "../sqlite-vec.so",
+        "../sqlite-vec.dll",
+        "../sqlite-vec.dylib",
+        "sqlite-vec",
+    }
+
+    local entry_point = "sqlite3_vec_init"
+
+    if db.enable_load_extension then
+        db:enable_load_extension(true)
+        for _, path in ipairs(possible_paths) do
+            local ok, result = pcall(function()
+                return db:load_extension(path, entry_point)
+            end)
+            -- lsqlite3 load_extension returns true on success
+            if ok and result then
+                db:enable_load_extension(false)
+                return true
+            end
+        end
+        db:enable_load_extension(false)
+        error("Failed to load extension from all paths")
+    else
+        for _, path in ipairs(possible_paths) do
+            local ok, result = pcall(function()
+                return db:load_extension(path, entry_point)
+            end)
+            -- lsqlite3 load_extension returns true on success
+            if ok and result then
+                return true
+            end
+        end
+        error("Failed to load extension from all paths")
+    end
+end
+
+-- Lua 5.1 compatible float to binary conversion function (IEEE 754 single precision, little-endian)
+local function float_to_bytes(f)
+    -- Handle special cases: NaN, Inf, -Inf, -0.0
+    if f ~= f then
+        -- NaN: exponent=255, mantissa!=0, sign=0 (quiet NaN)
+        return string.char(0, 0, 192, 127)
+    elseif f == math.huge then
+        -- +Inf: exponent=255, mantissa=0, sign=0
+        return string.char(0, 0, 128, 127)
+    elseif f == -math.huge then
+        -- -Inf: exponent=255, mantissa=0, sign=1
+        return string.char(0, 0, 128, 255)
+    elseif f == 0 then
+        -- Check for -0.0 vs +0.0
+        if 1/f == -math.huge then
+            -- -0.0: sign=1, exponent=0, mantissa=0
+            return string.char(0, 0, 0, 128)
+        else
+            -- +0.0
+            return string.char(0, 0, 0, 0)
+        end
+    end
+
+    local sign = 0
+    if f < 0 then
+        sign = 1
+        f = -f
+    end
+
+    local mantissa, exponent = math.frexp(f)
+    -- math.frexp returns mantissa in [0.5, 1), we need [1, 2) for IEEE 754
+    exponent = exponent - 1
+
+    local is_subnormal = exponent < -126
+    if is_subnormal then
+        -- Subnormal number: exponent field is 0, mantissa is denormalized
+        -- Formula: mantissa_stored = value * 2^149 = m * 2^(e + 149)
+        -- Since exponent = e - 1, we need: m * 2^(exponent + 1 + 149) = m * 2^(exponent + 150)
+        -- After multiplying by 2^23 later: m * 2^(exponent + 150) becomes the stored mantissa
+        -- Simplified: mantissa = m * 2^(exponent + 127) before the 2^23 scaling
+        mantissa = mantissa * 2^(exponent + 127)
+        exponent = 0
+    else
+        -- Normal number: remove implicit leading 1
+        -- frexp returns mantissa in [0.5, 1), convert to [0, 1) for IEEE 754
+        mantissa = (mantissa - 0.5) * 2
+        exponent = exponent + 127
+    end
+
+    -- Round half to even (banker's rounding) for IEEE 754 compliance
+    local scaled = mantissa * 2^23
+    local floor_val = math.floor(scaled)
+    local frac = scaled - floor_val
+    -- Use epsilon comparison for 0.5 to handle floating-point precision issues
+    local is_half = math.abs(frac - 0.5) < 1e-9
+    if frac > 0.5 + 1e-9 or (is_half and floor_val % 2 == 1) then
+        mantissa = floor_val + 1
+    else
+        mantissa = floor_val
+    end
+
+    -- Handle mantissa overflow from rounding (mantissa >= 2^23)
+    if mantissa >= 2^23 then
+        if is_subnormal then
+            -- Subnormal rounded up to smallest normal
+            mantissa = 0
+            exponent = 1
+        else
+            -- Normal number: carry into exponent
+            mantissa = 0
+            exponent = exponent + 1
+        end
+    end
+
+    -- Handle exponent overflow -> Infinity
+    if exponent >= 255 then
+        -- Return ±Infinity
+        if sign == 1 then
+            return string.char(0, 0, 128, 255)  -- -Inf
+        else
+            return string.char(0, 0, 128, 127)  -- +Inf
+        end
+    end
+
+    -- Encode as little-endian IEEE 754 single precision
+    local bytes = {}
+    bytes[1] = mantissa % 256
+    mantissa = math.floor(mantissa / 256)
+    bytes[2] = mantissa % 256
+    mantissa = math.floor(mantissa / 256)
+    bytes[3] = (mantissa % 128) + (exponent % 2) * 128
+    exponent = math.floor(exponent / 2)
+    bytes[4] = exponent + sign * 128
+
+    return string.char(bytes[1], bytes[2], bytes[3], bytes[4])
+end
+
+-- Helper function: serialize float vector to binary format (little-endian IEEE 754)
+function M.serialize_f32(vector)
+    local buffer = {}
+
+    if string.pack then
+        -- Use "<f" for little-endian float (Lua 5.3+)
+        for _, v in ipairs(vector) do
+            table.insert(buffer, string.pack("<f", v))
+        end
+    else
+        -- Lua 5.1/5.2 fallback
+        for _, v in ipairs(vector) do
+            table.insert(buffer, float_to_bytes(v))
+        end
+    end
+
+    return table.concat(buffer)
+end
+
+-- JSON format vector serialization
+-- Note: JSON does not support NaN, Inf, or -0.0, so these will error
+function M.serialize_json(vector)
+    local values = {}
+    for i, v in ipairs(vector) do
+        -- Check for NaN
+        if v ~= v then
+            error("serialize_json: NaN at index " .. i .. " is not valid JSON")
+        end
+        -- Check for Inf/-Inf
+        if v == math.huge or v == -math.huge then
+            error("serialize_json: Infinity at index " .. i .. " is not valid JSON")
+        end
+        -- Handle -0.0 (convert to 0.0 for JSON compatibility)
+        if v == 0 and 1/v == -math.huge then
+            v = 0.0
+        end
+        table.insert(values, tostring(v))
+    end
+    return "[" .. table.concat(values, ",") .. "]"
+end
+
+return M

examples/simple-lua/.gitignore

@@ -0,0 +1,7 @@
+# Lua bytecode
+*.luac
+
+# SQLite databases
+*.db
+*.sqlite
+*.sqlite3

examples/simple-lua/README.md

@@ -0,0 +1,83 @@
+# SQLite-Vec Simple Lua Example
+
+This example demonstrates how to use sqlite-vec with Lua and the lsqlite3 binding.
+
+## Prerequisites
+
+1. **Lua 5.1+** - The example is compatible with Lua 5.1 and later
+2. **lsqlite3** - Lua SQLite3 binding
+3. **sqlite-vec extension** - Built for your platform
+
+## Installation
+
+### Install lsqlite3
+
+Using LuaRocks:
+```bash
+luarocks install lsqlite3
+```
+
+Or on Ubuntu/Debian:
+```bash
+apt install lua-sql-sqlite3
+```
+
+### Build sqlite-vec
+
+From the repository root:
+```bash
+make loadable
+```
+
+This creates `dist/vec0.so` (Linux), `dist/vec0.dylib` (macOS), or `dist/vec0.dll` (Windows).
+
+## Running the Example
+
+From this directory:
+```bash
+lua demo.lua
+```
+
+Or using the run script:
+```bash
+./run.sh
+```
+
+## Expected Output
+
+```
+=== SQLite-Vec Simple Lua Example ===
+sqlite_version=3.x.x, vec_version=v0.x.x
+sqlite-vec extension loaded successfully
+Inserting vector data...
+Inserted 5 vectors
+Executing KNN query...
+Results (closest to [0.3, 0.3, 0.3, 0.3]):
+  rowid=3 distance=0.000000
+  rowid=2 distance=0.200000
+  rowid=4 distance=0.200000
+
+Testing binary serialization...
+  Binary round-trip: rowid=1 distance=0.000000
+
+Demo completed successfully
+```
+
+## Using the Binding in Your Project
+
+```lua
+local sqlite3 = require("lsqlite3")
+local sqlite_vec = require("sqlite_vec")
+
+local db = sqlite3.open_memory()
+
+-- Option 1: Auto-detect extension path
+sqlite_vec.load(db)
+
+-- Option 2: Explicit path
+sqlite_vec.load(db, "/path/to/vec0.so")
+
+-- Serialize vectors
+local json_vec = sqlite_vec.serialize_json({1.0, 2.0, 3.0})  -- "[1.0,2.0,3.0]"
+local binary_vec = sqlite_vec.serialize_f32({1.0, 2.0, 3.0}) -- 12 bytes
+```