feat(metadata): add filtering and counting by metadata

- Implement `filter_by_metadata()` for efficient key-value filtering using JSON_EXTRACT.
- Add `count_by_metadata()` to count records matching metadata filters.
- Introduce `similarity_search_with_filter()` for combined similarity and metadata filtering.
- Enhance validation for metadata filters to prevent SQL injection and ensure valid keys.
- Update documentation and examples for new metadata filtering features.
- Add comprehensive tests for metadata filtering functionality.

Author: atasoglu

CHANGELOG.md

@@ -5,6 +5,30 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.2.0] - 2025-02-01
+
+### Added
+- **Metadata filtering**: New `filter_by_metadata()` method for efficient key-value filtering using SQLite's JSON_EXTRACT
+- **Metadata counting**: New `count_by_metadata()` method to count records matching metadata filters
+- **Combined search**: New `similarity_search_with_filter()` method combining vector similarity with metadata filtering
+- Support for nested JSON paths in metadata filters (e.g., `{"author.name": "Alice"}`)
+- Pagination support in `filter_by_metadata()` with `limit` and `offset` parameters
+- New validation function `validate_metadata_filters()` for secure filter validation
+- New utility function `build_metadata_where_clause()` for safe SQL generation
+- Comprehensive test coverage for metadata filtering (unit, integration, and security tests)
+- New example `advanced_metadata_queries.py` demonstrating nested paths and complex queries
+- Updated `metadata_filtering.py` example with new filtering methods
+
+### Security
+- SQL injection prevention in metadata filter keys
+- Validation of JSON paths to prevent malicious queries
+- Parameterized queries for all metadata filtering operations
+
+### Documentation
+- Added "Metadata Filtering" section to README with examples
+- Updated examples list in README
+- Added comprehensive docstrings for new methods
+
 ## [2.1.1] - 2025-01-31
 
 ### Changed
@@ -166,6 +190,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## Version History
 
+- **2.2.0** - Added metadata filtering with JSON_EXTRACT support
+- **2.1.1** - Moved table name validation to create_table()
+- **2.1.0** - Added connection pooling support
 - **2.0.0** - Major refactor: simplified API, removed niche methods, cleaner naming
 - **1.2.0** - Added benchmarks module
 - **1.0.0** - First stable release

README.md

@@ -66,6 +66,40 @@ rows = client.get_many(rowids)
 client.close()
 ```
 
+## Metadata Filtering
+
+Efficiently filter records by metadata fields using SQLite's JSON functions:
+
+```python
+# Filter by single field
+results = client.filter_by_metadata({"category": "python"})
+
+# Filter by multiple fields
+results = client.filter_by_metadata({"category": "python", "year": 2024})
+
+# Nested JSON paths
+results = client.filter_by_metadata({"author.name": "Alice"})
+
+# Count matching records
+count = client.count_by_metadata({"category": "python"})
+
+# Combined similarity search + metadata filtering
+hits = client.similarity_search_with_filter(
+    embedding=query_vector,
+    filters={"category": "python"},
+    top_k=5
+)
+
+# Pagination
+results = client.filter_by_metadata(
+    {"category": "python"},
+    limit=10,
+    offset=0
+)
+```
+
+See [examples/metadata_filtering.py](examples/metadata_filtering.py) and [examples/advanced_metadata_queries.py](examples/advanced_metadata_queries.py) for more examples.
+
 ## Bulk Operations
 
 The client provides optimized methods for bulk operations:
@@ -219,6 +253,8 @@ Edit [benchmarks/config.yaml](benchmarks/config.yaml) to customize:
 - [TESTING.md](TESTING.md) - Testing documentation
 - [Examples](examples/) - Usage examples
   - [basic_usage.py](examples/basic_usage.py) - Basic CRUD operations
+  - [metadata_filtering.py](examples/metadata_filtering.py) - Metadata filtering and queries
+  - [advanced_metadata_queries.py](examples/advanced_metadata_queries.py) - Advanced metadata filtering with nested paths
   - [transaction_example.py](examples/transaction_example.py) - Transaction management with all CRUD operations
   - [batch_operations.py](examples/batch_operations.py) - Bulk operations
   - [logging_example.py](examples/logging_example.py) - Logging configuration

TODO

@@ -69,8 +69,8 @@
 - [x] Benchmark tests
 
 ### New Features
-- [ ] Partial search on JSON metadata (JSON_EXTRACT)
-- [ ] Metadata field filtering (key-value based)
+- [x] Partial search on JSON metadata (JSON_EXTRACT)
+- [x] Metadata field filtering (key-value based)
 - [x] Transaction context manager
 - [ ] Async/await support (aiosqlite)
 - [ ] Export/import functions (JSON, CSV)

examples/advanced_metadata_queries.py

@@ -0,0 +1,109 @@
+"""Advanced metadata querying examples for sqlite-vec-client.
+
+Demonstrates:
+- Nested JSON path queries
+- Complex filtering scenarios
+- Performance comparison: filter vs manual iteration
+- Real-world use cases
+"""
+
+from sqlite_vec_client import SQLiteVecClient
+
+
+def main():
+    client = SQLiteVecClient(table="documents", db_path=":memory:")
+    client.create_table(dim=128, distance="cosine")
+
+    # Add documents with nested metadata
+    texts = [
+        "Introduction to Python",
+        "Advanced JavaScript",
+        "Python for Data Science",
+        "Java Programming Guide",
+        "Machine Learning with Python",
+    ]
+
+    embeddings = [[0.1 * i] * 128 for i in range(len(texts))]
+
+    metadata = [
+        {"author": {"name": "Alice", "country": "US"}, "tags": ["python", "beginner"]},
+        {
+            "author": {"name": "Bob", "country": "UK"},
+            "tags": ["javascript", "advanced"],
+        },
+        {"author": {"name": "Alice", "country": "US"}, "tags": ["python", "data"]},
+        {"author": {"name": "Charlie", "country": "CA"}, "tags": ["java", "beginner"]},
+        {"author": {"name": "Alice", "country": "US"}, "tags": ["python", "ml"]},
+    ]
+
+    rowids = client.add(texts=texts, embeddings=embeddings, metadata=metadata)
+    print(f"Added {len(rowids)} documents\n")
+
+    # Example 1: Nested JSON path queries
+    print("=== Nested JSON Path Queries ===")
+    print("\nDocuments by Alice (nested path):")
+    results = client.filter_by_metadata({"author.name": "Alice"})
+    for rowid, text, meta, _ in results:
+        print(f"  [{rowid}] {text}")
+    print(f"Total: {len(results)} documents")
+
+    # Example 2: Filter by country
+    print("\n\nDocuments from US authors:")
+    results = client.filter_by_metadata({"author.country": "US"})
+    for rowid, text, meta, _ in results:
+        print(f"  [{rowid}] {text} by {meta['author']['name']}")
+
+    # Example 3: Count by author
+    print("\n\n=== Count by Author ===")
+    for author in ["Alice", "Bob", "Charlie"]:
+        count = client.count_by_metadata({"author.name": author})
+        print(f"  {author}: {count} documents")
+
+    # Example 4: Combined similarity + metadata filtering
+    print("\n\n=== Combined Similarity + Metadata Filtering ===")
+    query_emb = [0.15] * 128
+    print("\nSimilar documents by Alice (top 10 candidates, filtered):")
+    hits = client.similarity_search_with_filter(
+        embedding=query_emb, filters={"author.name": "Alice"}, top_k=10
+    )
+    if hits:
+        for rowid, text, distance in hits:
+            dist_str = f"{distance:.4f}" if distance is not None else "N/A"
+            print(f"  [{rowid}] {text} (distance: {dist_str})")
+    else:
+        print("  No results found (filters may be too restrictive)")
+
+    # Example 5: Pagination for large result sets
+    print("\n\n=== Pagination Example ===")
+    all_us_docs = client.count_by_metadata({"author.country": "US"})
+    print(f"Total US documents: {all_us_docs}")
+    print("\nFetching in pages of 2:")
+    for page in range(0, all_us_docs, 2):
+        results = client.filter_by_metadata(
+            {"author.country": "US"}, limit=2, offset=page
+        )
+        print(f"  Page {page//2 + 1}: {[r[1] for r in results]}")
+
+    # Example 6: Alternative - regular similarity search
+    print("\n\n=== Regular Similarity Search (no filter) ===")
+    hits = client.similarity_search(embedding=query_emb, top_k=3)
+    print("Top 3 similar documents:")
+    for rowid, text, distance in hits:
+        if distance is not None:
+            print(f"  [{rowid}] {text} (distance: {distance:.4f})")
+        else:
+            print(f"  [{rowid}] {text}")
+
+    # Performance note
+    print("\n\n=== Performance Note ===")
+    print("filter_by_metadata() uses SQLite's json_extract() for efficient queries.")
+    print("This is much faster than manually iterating with get_all().")
+    print("\nFor frequently queried fields, consider:")
+    print("  1. Using filter_by_metadata() for ad-hoc queries")
+    print("  2. Creating computed columns for indexed queries (advanced)")
+
+    client.close()
+
+
+if __name__ == "__main__":
+    main()

examples/metadata_filtering.py

@@ -2,7 +2,9 @@
 
 Demonstrates:
 - Adding records with metadata
-- Querying records with get_all
+- Filtering by metadata fields
+- Counting records by metadata
+- Combined similarity search with metadata filtering
 - Updating metadata
 """
 
@@ -35,19 +37,32 @@ def main():
     rowids = client.add(texts=texts, embeddings=embeddings, metadata=metadata)
     print(f"Added {len(rowids)} articles")
 
-    # Query all articles and filter by author
-    print("\nAlice's articles:")
-    for rowid, text, meta, _ in client.get_all():
-        if meta.get("author") == "Alice":
-            print(f"  [{rowid}] {text} - {meta}")
+    # Filter by metadata - efficient JSON_EXTRACT queries
+    print("\nAlice's articles (using filter_by_metadata):")
+    results = client.filter_by_metadata({"author": "Alice"})
+    for rowid, text, meta, _ in results:
+        print(f"  [{rowid}] {text} - {meta}")
 
-    # Query all articles and filter by text
-    print("\nPython-related articles:")
-    for rowid, text, meta, _ in client.get_all():
-        if "Python" in text:
-            print(f"  [{rowid}] {text}")
+    # Filter by multiple fields
+    print("\nArticles from 2024:")
+    results = client.filter_by_metadata({"year": 2024})
+    for rowid, text, meta, _ in results:
+        print(f"  [{rowid}] {text} - Year: {meta['year']}")
 
-    # Update metadata
+    # Count records by metadata
+    count = client.count_by_metadata({"author": "Alice"})
+    print(f"\nTotal articles by Alice: {count}")
+
+    # Combined similarity search with metadata filtering
+    print("\nSimilar to 'Python' in category 'programming':")
+    query_emb = [0.1] * 128
+    hits = client.similarity_search_with_filter(
+        embedding=query_emb, filters={"category": "programming"}, top_k=5
+    )
+    for rowid, text, distance in hits:
+        print(f"  [{rowid}] {text} (distance: {distance:.4f})")
+
+    # Update metadata and verify with filter
     if rowids:
         client.update(
             rowids[0],
@@ -58,9 +73,16 @@ def main():
                 "updated": True,
             },
         )
-        updated = client.get(rowids[0])
-        if updated:
-            print(f"\nUpdated metadata: {updated[2]}")
+        # Find updated records
+        updated_records = client.filter_by_metadata({"updated": True})
+        print(f"\nUpdated records: {len(updated_records)}")
+        if updated_records:
+            print(f"  Metadata: {updated_records[0][2]}")
+
+    # Pagination example
+    print("\nPagination example (limit=2):")
+    page1 = client.filter_by_metadata({"year": 2024}, limit=2, offset=0)
+    print(f"  Page 1: {len(page1)} results")
 
     client.close()
 

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "sqlite-vec-client"
-version = "2.1.1"
+version = "2.2.0"
 description = "A lightweight Python client around sqlite-vec for CRUD and similarity search."
 readme = "README.md"
 requires-python = ">=3.9"