Full text search across all providers - Bump cosmos DB version

Author: aritchie

Directory.Packages.props

@@ -5,7 +5,7 @@
     </PropertyGroup>
     <ItemGroup>
         <PackageVersion Include="Nerdbank.GitVersioning" Version="3.9.50"/>
-        <PackageVersion Include="Microsoft.Azure.Cosmos" Version="3.46.1"/>
+        <PackageVersion Include="Microsoft.Azure.Cosmos" Version="3.61.0"/>
         <PackageVersion Include="Newtonsoft.Json" Version="13.0.3"/>
         <PackageVersion Include="Microsoft.Data.Sqlite" Version="10.0.3"/>
         <PackageVersion Include="Microsoft.Data.Sqlite.Core" Version="10.0.3"/>

readme.md

@@ -53,6 +53,7 @@ A lightweight, multi-provider document store for .NET that turns relational data
 - **Batch upsert / update / remove** — `store.BatchUpsert(items)`, `store.BatchUpdate(items)`, and `store.BatchRemove<T>(ids)` apply many writes as one set operation — a single multi-row `INSERT … ON CONFLICT` deep-merge on SQLite/DuckDB, one `BulkWrite`/`DeleteMany` on MongoDB, parallel request waves on Cosmos, and a single `DELETE … IN (…)` for `BatchRemove` on every relational provider. All-or-nothing: the first version conflict rolls the whole batch back.
 - **Spatial / geo queries** — `WithinRadius`, `WithinBoundingBox`, and `NearestNeighbors` methods with `GeoPoint` support. SQLite uses R*Tree virtual tables; CosmosDB uses native `ST_DISTANCE`/`ST_WITHIN`. Configure with `MapSpatialProperty<T>(x => x.Location)`.
 - **Vector / ANN search** — `MapVectorProperty<T>(x => x.Embedding, dimensions: 1536)` + `store.Query<T>().NearestVectors(queryEmbedding, k: 10)` for cross-provider ANN over `ReadOnlyMemory<float>` embeddings. Provider-native indexes: pgvector (PostgreSQL), `VECTOR_DISTANCE` (SQL Server 2025 and Oracle 23ai), DiskANN (CosmosDB), `$vectorSearch` (MongoDB Atlas), `vss` (DuckDB), `sqlite-vec` (SQLite). Cosine / Euclidean / DotProduct everywhere; Hamming on pgvector. Pre-filter via `Where(...)` where the engine supports it. Auto-embed text properties on insert via `Shiny.DocumentDb.Extensions.AI`'s `AutoEmbedOnInsert<T>` hook + `Microsoft.Extensions.AI.IEmbeddingGenerator`.
+- **Full-text search (all providers)** — `MapFullTextProperty<T>(a => a.Body)` (or `[a => a.Title, a => a.Body]`) + `store.FullTextSearch<T>("orleans persistence")` for relevance-ranked text search, returning `FullTextResult<T>` (`Document` + normalized `Score`, higher = better) ordered by relevance, with an optional pre-filter predicate and a fluent `store.Query<T>().Where(...).FullTextMatch("...")` form. The native index is created for you and engine-maintained: FTS5 (SQLite), `tsvector`+GIN (PostgreSQL), `FULLTEXT` (MySQL), Oracle Text (Oracle), Full-Text Index (SQL Server), the `fts` extension (DuckDB), full-text policy (CosmosDB), `$text` (MongoDB), and an in-memory TF-IDF fallback on LiteDB / IndexedDB. Declarative: a type must be mapped before it can be searched. Oracle Text and SQL Server Full-Text Search are optional server components; CosmosDB full-text needs `Microsoft.Azure.Cosmos` 3.61.0+.
 - **Telemetry & observability (`Shiny.DocumentDb.Diagnostics`)** — `services.AddDocumentStoreInstrumentation()` wraps any provider in a decorator that emits OpenTelemetry-native metrics (`db.client.operation.duration` + an operations counter + a returned-rows histogram, tagged per the OTel DB semantic conventions) and an `ActivitySource` client span per operation. Covers CRUD, the fluent-query terminals, the temporal `ITemporalDocumentStore` ops, and `UnitOfWork.SaveChanges` (inner ops become child spans). Built on `System.Diagnostics.Metrics`/`IMeterFactory`; subscribe with `.AddMeter("Shiny.DocumentDb")` / `.AddSource("Shiny.DocumentDb")`. Zero-cost when nobody is listening; never records document bodies or ids.
 - **Hot backup** — `store.Backup("/path/to/backup.db")` copies the database to a file. Available on `SqliteDocumentStore`, `SqlCipherDocumentStore`, and `LiteDbDocumentStore` (not on the `IDocumentStore` interface).
 - **Clear the whole store (`IDocumentMaintenance.ClearAll`)** — `((IDocumentMaintenance)store).ClearAll()` wipes every document type (plus temporal-history, spatial, and vector sidecars) for test/dev resets. A whole-store wipe — not type- or tenant-scoped (use `Clear<T>()` for one type) — that targets only user tables in the current database and never the system catalogs. Implemented on the relational `DocumentStore` (SQLite, SQL Server, PostgreSQL, MySQL, DuckDB, Oracle), MongoDB, and CosmosDB; `SqliteDocumentStore.ClearAllAsync()` still works and delegates to it.

skills/shiny-documentdb/SKILL.md

@@ -23,6 +23,14 @@ triggers:
   - vector search
   - NearestVectors
   - MapVectorProperty
+  - full-text search
+  - FullTextSearch
+  - FullTextMatch
+  - MapFullTextProperty
+  - FullTextResult
+  - FullTextLanguage
+  - FTS5
+  - tsvector
   - sqlite-vec
   - VectorExtensionPreloaded
   - EnableVectorExtension
@@ -1442,6 +1450,30 @@ If you supply your own binary, two mutually complementary flags on `SqliteDataba
 
 Either flag (or the package helper) makes `SupportsVector` return `true`. Without one, `NearestVectors` throws `NotSupportedException`. vec0 is flat-scan (no HNSW); when a `.Where(...)` filter is combined with the search, the library over-fetches `k * postFilterMultiplier` (default 4) candidates.
 
+## Full-Text Search
+
+Relevance-ranked text search over one or more string properties. **Declarative and up-front**: map the searchable property with `MapFullTextProperty<T>(...)` and the library creates the native index for you at startup. A type **must be mapped before it can be searched** — there is no ad-hoc full-text (unlike `.Where(x => x.Body.Contains(...))`, which works on any field). Supported on **every provider**: FTS5 (SQLite), `tsvector`+GIN (PostgreSQL), `FULLTEXT` (MySQL), Oracle Text (Oracle), Full-Text Index (SQL Server), the `fts` extension (DuckDB), full-text policy (Cosmos), `$text` (MongoDB), and an in-memory TF-IDF scan on LiteDB / IndexedDB.
+
+```csharp
+// single field, or several combined into one index
+options.MapFullTextProperty<Article>(a => a.Body);
+options.MapFullTextProperty<Article>([a => a.Title, a => a.Body]);
+
+// terminal API — ordered by relevance descending, each with a Score (higher = better)
+IReadOnlyList<FullTextResult<Article>> hits =
+    await store.FullTextSearch<Article>("orleans persistence", maxResults: 20);
+
+// optional pre-filter predicate (tenant/category scoping)
+var tech = await store.FullTextSearch<Article>("orleans", filter: a => a.Category == "tech");
+
+// fluent form — folds the query's Where predicates into the pre-filter
+var hits2 = await store.Query<Article>()
+    .Where(a => a.Category == "tech")
+    .FullTextMatch("orleans", maxResults: 10);
+```
+
+`FullTextResult<T>` carries `Document` and a normalized `double Score` (higher = more relevant; absolute scale is provider-specific — compare only within one result set). `MapFullTextProperty` also has an AOT-safe overload taking `propertyNames` + a `Func<T, IEnumerable<string?>>` selector (for combining fields or indexing a string collection), and an optional `FullTextLanguage` (controls stemming where the backend supports it). The index is engine-maintained, so `Insert`/`Update`/`Remove`/`Clear` keep it in sync automatically. Notes: engines with one full-text index per table (SQL Server, MongoDB) support a single mapped type per table/collection; **Oracle Text** and **SQL Server Full-Text Search** are optional server components that must be installed; Cosmos full-text needs `Microsoft.Azure.Cosmos` 3.61.0+.
+
 ## Fluent Query Builder (IDocumentQuery<T>)
 
 The fluent query builder is the primary way to query documents. Start with `store.Query<T>()` and chain builder methods, then terminate with a materialization method.

src/Shiny.DocumentDb.CosmosDb/CosmosDbDocumentQuery.cs

@@ -359,6 +359,15 @@ public IAsyncEnumerable<DocumentChange<T>> NotifyOnChange(CancellationToken ct =
         => throw new NotSupportedException(
             "Per-query change observation is not supported by CosmosDbDocumentStore. " +
             "Use SubscribeChanges<T>() to consume the native Cosmos change feed.");
+
+    public Task<IReadOnlyList<FullTextResult<T>>> FullTextMatch(string searchText, int maxResults = 50, CancellationToken ct = default)
+    {
+        var effective = this.GetEffectivePredicateExpressions().ToList();
+        Expression<Func<T, bool>>? filter = effective.Count == 0
+            ? null
+            : DocumentQuery<T>.CombinePredicates(effective);
+        return this.store.FullTextSearch(searchText, maxResults, filter, ct);
+    }
 }
 
 internal class CosmosDbProjectedDocumentQuery<TSource, TResult> : IDocumentQuery<TResult>

src/Shiny.DocumentDb.CosmosDb/CosmosDbDocumentStore.cs

@@ -96,10 +96,12 @@ public CosmosDbDocumentStore(CosmosDbDocumentStoreOptions options)
         options.ResolveVersionJsonPaths(this.jsonOptions);
         options.ResolveSpatialJsonPaths(this.jsonOptions);
         options.ResolveVectorJsonPaths(this.jsonOptions);
+        options.ResolveFullTextJsonPaths(this.jsonOptions);
     }
 
     public bool SupportsSpatial => this.options.spatialMappings.Count > 0;
     public bool SupportsVector => this.options.vectorMappings.Count > 0;
+    public bool SupportsFullText => this.options.fullTextMappings.Count > 0;
 
     public void Dispose()
     {
@@ -226,6 +228,27 @@ async Task<Container> EnsureContainerAsync(string containerName, CancellationTok
                 containerProperties.VectorEmbeddingPolicy = new VectorEmbeddingPolicy(embeddings);
             }
 
+            // Full-text policy + indexes for mapped full-text properties.
+            if (this.options.fullTextMappings.Count > 0)
+            {
+                var ftPaths = new System.Collections.ObjectModel.Collection<FullTextPath>();
+                foreach (var mapping in this.options.fullTextMappings.Values)
+                {
+                    var lang = CosmosFullTextLanguage(mapping.Language);
+                    foreach (var jsonPath in mapping.JsonPaths)
+                    {
+                        ftPaths.Add(new FullTextPath { Path = $"/data/{jsonPath}", Language = lang });
+                        containerProperties.IndexingPolicy.FullTextIndexes.Add(
+                            new FullTextIndexPath { Path = $"/data/{jsonPath}" });
+                    }
+                }
+                containerProperties.FullTextPolicy = new FullTextPolicy
+                {
+                    DefaultLanguage = "en-US",
+                    FullTextPaths = ftPaths
+                };
+            }
+
             await this.database.CreateContainerIfNotExistsAsync(
                 containerProperties, this.options.DefaultThroughput, cancellationToken: ct).ConfigureAwait(false);
 
@@ -238,6 +261,77 @@ await this.database.CreateContainerIfNotExistsAsync(
         }
     }
 
+    static string CosmosFullTextLanguage(FullTextLanguage language) => language switch
+    {
+        FullTextLanguage.German => "de-DE",
+        FullTextLanguage.Spanish => "es-ES",
+        FullTextLanguage.French => "fr-FR",
+        FullTextLanguage.Italian => "it-IT",
+        FullTextLanguage.Portuguese => "pt-BR",
+        FullTextLanguage.Dutch => "nl-NL",
+        FullTextLanguage.Russian => "ru-RU",
+        _ => "en-US"
+    };
+
+    // ── Full-text search (Cosmos DB full-text policy + FullTextScore RANK) ──
+
+    public async Task<IReadOnlyList<FullTextResult<T>>> FullTextSearch<T>(
+        string searchText,
+        int maxResults = 50,
+        Expression<Func<T, bool>>? filter = null,
+        CancellationToken cancellationToken = default) where T : class
+    {
+        var mapping = this.options.ResolveFullTextMapping(typeof(T))
+            ?? throw new NotSupportedException(
+                $"No full-text property mapped for type '{typeof(T).Name}'. Call MapFullTextProperty<{typeof(T).Name}>() in options.");
+        ArgumentException.ThrowIfNullOrEmpty(searchText);
+        if (maxResults <= 0)
+            throw new ArgumentOutOfRangeException(nameof(maxResults));
+
+        var typeInfo = this.FindTypeInfo<T>(null);
+        var typeName = this.ResolveTypeName<T>();
+        var container = await this.EnsureContainerAsync(this.ResolveContainerName<T>(), cancellationToken).ConfigureAwait(false);
+
+        var terms = FullTextMappingFactory.Tokenize(searchText);
+        if (terms.Count == 0)
+            return Array.Empty<FullTextResult<T>>();
+
+        // Tokens are alphanumeric → safe to embed as Cosmos string literals.
+        var termLiterals = string.Join(", ", terms.Select(t => "\"" + t + "\""));
+        var paths = mapping.JsonPaths.Select(p => $"c.data.{p}").ToList();
+        var contains = string.Join(" OR ", paths.Select(p => $"FullTextContainsAny({p}, {termLiterals})"));
+        var scores = paths.Select(p => $"FullTextScore({p}, {termLiterals})").ToList();
+        // FullTextScore is only valid in ORDER BY RANK; combine multiple fields with reciprocal rank fusion.
+        var rank = scores.Count == 1 ? scores[0] : $"RRF({string.Join(", ", scores)})";
+
+        // FullTextScore cannot be projected, so the score is synthesized from rank order; over-fetch
+        // when a post-filter is present so it doesn't starve the top-N.
+        var fetch = filter == null ? maxResults : maxResults * 4;
+        var sql = $"SELECT TOP {fetch} c.data FROM c WHERE c.typeName = @typeName AND ({contains}) ORDER BY RANK {rank}";
+        var queryDef = new QueryDefinition(sql).WithParameter("@typeName", typeName);
+
+        var postFilter = filter == null ? null : ExpressionInterpreter.Interpret(filter);
+        var results = new List<FullTextResult<T>>();
+        var position = 0;
+        using var iterator = container.GetItemQueryIterator<CosmosDocument>(queryDef, requestOptions: new QueryRequestOptions
+        {
+            PartitionKey = new PartitionKey(typeName)
+        });
+        while (iterator.HasMoreResults && results.Count < maxResults)
+        {
+            var response = await iterator.ReadNextAsync(cancellationToken).ConfigureAwait(false);
+            foreach (var doc in response)
+            {
+                var obj = Deserialize(doc.Data, typeInfo, this.jsonOptions);
+                if (obj == null) continue;
+                if (postFilter != null && !postFilter(obj)) continue;
+                results.Add(new FullTextResult<T> { Document = obj, Score = 1.0 / ++position });
+                if (results.Count >= maxResults) break;
+            }
+        }
+        return results;
+    }
+
     string GenerateId<T>(IdAccessor<T> accessor) where T : class
     {
         return accessor.Kind switch

src/Shiny.DocumentDb.CosmosDb/CosmosDbDocumentStoreOptions.cs

@@ -16,6 +16,7 @@ public class CosmosDbDocumentStoreOptions
     internal readonly Dictionary<Type, VersionMapping> versionMappings = new();
     internal readonly Dictionary<Type, CosmosDbSpatialMapping> spatialMappings = new();
     internal readonly Dictionary<Type, VectorMapping> vectorMappings = new();
+    internal readonly Dictionary<Type, FullTextMapping> fullTextMappings = new();
     internal readonly Dictionary<Type, TemporalMapping> temporalMappings = new();
 
     public required string ConnectionString { get; set; }
@@ -397,6 +398,46 @@ internal void ResolveVectorJsonPaths(JsonSerializerOptions jsonOptions)
         }
     }
 
+    /// <summary>
+    /// Declares a string property as full-text searchable via Cosmos DB full-text search (full-text
+    /// policy + index, queried with <c>FullTextScore</c>/<c>FullTextContainsAny</c>). See
+    /// <see cref="DocumentStoreOptions.MapFullTextProperty{T}(Expression{Func{T, string}}, FullTextLanguage)"/>.
+    /// </summary>
+    public CosmosDbDocumentStoreOptions MapFullTextProperty<T>(
+        Expression<Func<T, string?>> property,
+        FullTextLanguage language = FullTextLanguage.English) where T : class
+    {
+        ArgumentNullException.ThrowIfNull(property);
+        this.fullTextMappings[typeof(T)] = FullTextMappingFactory.FromExpressions([property], language);
+        return this;
+    }
+
+    /// <summary>Declares several string properties combined into one full-text index.</summary>
+    public CosmosDbDocumentStoreOptions MapFullTextProperty<T>(
+        IReadOnlyList<Expression<Func<T, string?>>> properties,
+        FullTextLanguage language = FullTextLanguage.English) where T : class
+    {
+        ArgumentNullException.ThrowIfNull(properties);
+        this.fullTextMappings[typeof(T)] = FullTextMappingFactory.FromExpressions(properties, language);
+        return this;
+    }
+
+    /// <summary>AOT-safe overload mapping full-text to a direct text selector (combine fields or index a string collection).</summary>
+    public CosmosDbDocumentStoreOptions MapFullTextProperty<T>(
+        IReadOnlyList<string> propertyNames,
+        Func<T, IEnumerable<string?>> textSelector,
+        FullTextLanguage language = FullTextLanguage.English) where T : class
+    {
+        this.fullTextMappings[typeof(T)] = FullTextMappingFactory.FromAccessor(propertyNames, textSelector, language);
+        return this;
+    }
+
+    internal FullTextMapping? ResolveFullTextMapping(Type type) =>
+        this.fullTextMappings.TryGetValue(type, out var mapping) ? mapping : null;
+
+    internal void ResolveFullTextJsonPaths(JsonSerializerOptions jsonOptions)
+        => FullTextMappingFactory.ResolveJsonPaths(this.fullTextMappings.Values, jsonOptions);
+
     static string ExtractPropertyName<T>(Expression<Func<T, object>> expression)
     {
         var body = expression.Body;

src/Shiny.DocumentDb.Diagnostics/InstrumentedDocumentStore.cs

@@ -131,6 +131,11 @@ public Task<IReadOnlyList<SpatialResult<T>>> NearestNeighbors<T>(GeoPoint center
     public Task<IReadOnlyList<VectorResult<T>>> NearestVectors<T>(ReadOnlyMemory<float> query, int k, Expression<Func<T, bool>>? filter = null, CancellationToken cancellationToken = default) where T : class
         => this.tracker.Track("nearest_vectors", Coll<T>(), () => this.inner.NearestVectors(query, k, filter, cancellationToken), r => r.Count);
 
+    public bool SupportsFullText => this.inner.SupportsFullText;
+
+    public Task<IReadOnlyList<FullTextResult<T>>> FullTextSearch<T>(string searchText, int maxResults = 50, Expression<Func<T, bool>>? filter = null, CancellationToken cancellationToken = default) where T : class
+        => this.tracker.Track("full_text_search", Coll<T>(), () => this.inner.FullTextSearch(searchText, maxResults, filter, cancellationToken), r => r.Count);
+
     // ── ITemporalDocumentStore ──────────────────────────────────────────
 
     public Task<IReadOnlyList<DocumentVersion<T>>> History<T>(object id, JsonTypeInfo<T>? jsonTypeInfo = null, CancellationToken cancellationToken = default) where T : class