Merge branch 'main' into exclude_vector_source

Author: jimczi

docs/changelog/132845.yaml

@@ -0,0 +1,5 @@
+pr: 132845
+summary: Expose existing DLS cache x-pack usage statistics
+area: Authorization
+type: enhancement
+issues: []

docs/reference/elasticsearch-plugins/plugin-management.md

@@ -116,7 +116,7 @@ If Elasticsearch was installed using the deb or rpm package then run `/usr/share
 
 For detailed instructions on installing, managing, and configuring plugins, see the following:
 
-* [Intalling Plugings](./installation.md)
+* [Installing Plugins](./installation.md)
 * [Custom URL or file system](./plugin-management-custom-url.md)
 * [Installing multiple plugins](./installing-multiple-plugins.md)
 * [Mandatory plugins](./mandatory-plugins.md)

docs/reference/elasticsearch/index-settings/bbq.md

@@ -0,0 +1,162 @@
+---
+navigation_title: Better Binary Quantization (BBQ)
+applies_to:
+  stack: all
+  serverless: all
+---
+
+# Better Binary Quantization (BBQ) [bbq]
+
+Better Binary Quantization (BBQ) is an advanced vector quantization method, designed for large-scale similarity search. BBQ is a form of lossy compression for [`dense_vector` fields](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/dense-vector) that enables efficient storage and retrieval of large numbers of vectors, while keeping results close to those from the original uncompressed vectors.
+
+BBQ offers significant improvements over scalar quantization by relying on optimized `bit` level computations to reduce memory usage and computational costs while maintaining high search relevance using pre-computed corrective factors. BBQ is designed to work in combination with [oversampling](#bbq-oversampling) and reranking, and is compatible with various [vector search algorithms](#bbq-vector-search-algorithms), such as [HNSW](#bbq-hnsw) and [brute force (flat)](#bbq-flat).
+
+## How BBQ works [bbq-how-it-works]
+
+BBQ retains the original vector’s dimensionality but transforms the datatype of the dimensions from the original `float32` to `bit` effectively compressing each vector by 32x plus an additional 14 bytes of corrective data per vector. BBQ uses these pre-computed corrective factors as partial distance calculations to help realize impressively robust approximations of the original vector. 
+
+Measuring vector similarity with BBQ vectors requires much less computing effort, allowing more candidates to be considered when using the HNSW algorithm. This often results in better ranking quality and improved relevance compared to the original `float32` vectors.
+
+## Supported vector search algorithms [bbq-vector-search-algorithms]
+
+BBQ currently supports two vector search algorithms, each suited to different scenarios. You can configure them by setting the dense vector field’s `index_type`.
+
+### `bbq_hnsw` [bbq-hnsw]
+
+When you set a dense vector field’s `index_options` parameter to `type: bbq_hnsw`, {{es}} uses the HNSW algorithm for fast [kNN search](https://www.elastic.co/docs//solutions/search/vector/knn) on compressed vectors. With the default [oversampling](#bbq-oversampling) applied, it delivers better cost efficiency, lower latency, and improved relevance ranking, making it the best choice for large-scale similarity search.
+
+:::{note}
+Starting in version 9.1, `bbq_hnsw` is the default indexing method for new `dense_vector` fields with greater than 384 dimensions, so you typically don’t need to specify it explicitly when creating an index.  
+
+Datasets with less than 384 dimensions may see less accuracy and incur a higher overhead cost related to the corrective factors, but we have observed some production datasets perform well even at fairly low dimensions including [tests on e5-small](https://www.elastic.co/search-labs/blog/better-binary-quantization-lucene-elasticsearch).
+:::
+
+The following example creates an index with a `dense_vector` field configured to use the `bbq_hnsw` algorithm.
+
+```console
+PUT bbq_hnsw-index
+{
+  "mappings": {
+    "properties": {
+      "my_vector": {
+        "type": "dense_vector",
+        "dims": 64,
+        "index": true,
+        "index_options": {
+          "type": "bbq_hnsw"
+        }
+      }
+    }
+  }
+}
+```
+
+To change an existing index to use `bbq_hnsw`, update the field mapping:
+
+```console
+PUT bbq_hnsw-index/_mapping
+{
+  "properties": {
+    "my_vector": {
+      "type": "dense_vector",
+      "dims": 64,
+      "index": true,
+      "index_options": {
+        "type": "bbq_hnsw"
+      }
+    }
+  }
+}
+```
+
+After this change, all newly created segments will use the `bbq_hnsw` algorithm. As you add or update documents, the index will gradually convert to `bbq_hnsw`. 
+
+To apply `bbq_hnsw` to all vectors at once, reindex them into a new index where the `index_options` parameter's `type` is set to `bbq_hnsw`:
+
+:::::{stepper}
+::::{step} Create a destination index
+```console
+PUT my-index-bbq
+{
+  "mappings": {
+    "properties": {
+      "my_vector": {
+        "type": "dense_vector",
+        "dims": 64,
+        "index": true,
+        "index_options": {
+          "type": "bbq_hnsw"
+        }
+      }
+    }
+  }
+}
+```
+::::
+
+::::{step} Reindex the data
+```console
+POST _reindex
+{
+  "source": { "index": "my-index" }, <1>
+  "dest":   { "index": "my-index-bbq" }
+}
+```
+1. The existing index to be reindexed into the newly created index with the `bbq_hnsw` algorithm.
+::::
+
+:::::
+
+### `bbq_flat` [bbq-flat]
+
+When you set a dense vector field’s `index_options` parameter to `type: bbq_flat`, {{es}} uses the BBQ algorithm without HNSW. This option generally requires fewer computing resources and works best when the number of vectors being searched is relatively low.
+
+The following example creates an index with a `dense_vector` field configured to use the `bbq_flat` algorithm.
+
+```console
+PUT bbq_flat-index
+{
+  "mappings": {
+    "properties": {
+      "my_vector": {
+        "type": "dense_vector",
+        "dims": 64,
+        "index": true,
+        "index_options": {
+          "type": "bbq_flat"
+        }
+      }
+    }
+  }
+}
+```
+
+## Oversampling [bbq-oversampling]
+
+Oversampling is a technique used with BBQ searches to reduce the accuracy loss from compression. Compression lowers the memory footprint by over 95% and improves query latency, at the cost of decreased result accuracy. This decrease can be mitigated by oversampling during query time and reranking the top results using the full vector. 
+
+When you run a kNN search on a BBQ-indexed field, {{es}} automatically retrieves more candidate vectors than the number of results you request. This oversampling improves accuracy by giving the system more vectors to re-rank using their full-precision values before returning the top results.
+
+```console
+GET bbq-index/_search
+{
+  "knn": {
+    "field": "my_vector",
+    "query_vector": [0.12, -0.45, ...],
+    "k": 10,
+    "num_candidates": 100
+  }
+}
+```
+
+By default, oversampling is set to 3×, meaning if you request k:10, {{es}} retrieves 30 candidates for re-ranking. You don’t need to configure this behavior; it’s applied automatically for BBQ searches.
+
+:::{note}
+You can change oversampling from the default 3× to another value. Refer to [Oversampling and rescoring for quantized vectors](https://www.elastic.co/docs/solutions/search/vector/knn#dense-vector-knn-search-rescoring) for details.
+:::
+
+## Learn more [bbq-learn-more]
+
+- [Better Binary Quantization (BBQ) in Lucene and {{es}}](https://www.elastic.co/search-labs/blog/better-binary-quantization-lucene-elasticsearch) - Learn how BBQ works, its benefits, and how it reduces memory usage while preserving search accuracy.
+- [Dense vector field type](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/dense-vector) - Find code examples for using `bbq_hnsw` `index_type`.
+- [kNN search](https://www.elastic.co/docs/solutions/search/vector/knn) - Learn about the search algorithm that BBQ works with.

docs/reference/elasticsearch/index-settings/source.md

@@ -0,0 +1,24 @@
+---
+mapped_pages:
+  - https://www.elastic.co/guide/en/elasticsearch/reference/current/source-index-settings.html
+navigation_title: Source settings
+---
+
+# Source index settings [source-index-settings]
+
+All settings around the _source metadata field.
+
+$$$source-mode$$$
+
+`index.source.mode`
+: (Static, string) The source mode for the index. Valid values are [`synthetic`](/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source), [`disabled`](/reference/elasticsearch/mapping-reference/mapping-source-field.md#disable-source-field) or `stored`. Defaults to `stored`. The `stored` source mode always stores the source metadata field on disk.
+
+$$$recovery-use_synthetic_source$$$
+
+`index.recovery.use_synthetic_source`
+: (Static, boolean) If synthetic source mode is used, whether the recovery source should also be synthesized instead of stored to disk. Defaults to `true`. This setting can only be configured if synthetic source mode is enabled.
+
+$$$synthetic-source-keep$$$
+
+`index.mapping.synthetic_source_keep`
+: (Static, string) Controls how to retain accuracy of fields at the index level. Valid values are `none` or `arrays`.This is a subset of [synthetic source keep mapping attribute](/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-keep). Defaults to `arrays` if `index.mode` is `logsdb` or otherwise `none`.

docs/reference/elasticsearch/toc.yml

@@ -37,6 +37,7 @@ toc:
   - file: index-settings/index.md
     children:
         - file: index-settings/serverless.md
+        - file: index-settings/bbq.md
         - file: index-settings/index-modules.md
         - file: index-settings/shard-allocation.md
           children:
@@ -56,6 +57,7 @@ toc:
           children:
             - file: index-settings/preloading-data-into-file-system-cache.md
         - file: index-settings/time-series.md
+        - file: index-settings/source.md
         - file: index-settings/translog.md
         - file: index-settings/pressure.md
         - file: index-settings/path.md
@@ -214,4 +216,4 @@ toc:
       - file: command-line-tools/setup-passwords.md
       - file: command-line-tools/shard-tool.md
       - file: command-line-tools/syskeygen.md
-      - file: command-line-tools/users-command.md
+      - file: command-line-tools/users-command.md

muted-tests.yml

@@ -576,6 +576,12 @@ tests:
 - class: org.elasticsearch.test.rest.yaml.CcsCommonYamlTestSuiteIT
   method: test {p0=search.highlight/50_synthetic_source/text multi fvh source order}
   issue: https://github.com/elastic/elasticsearch/issues/133056
+- class: org.elasticsearch.upgrades.SyntheticSourceRollingUpgradeIT
+  method: testIndexing {upgradedNodes=1}
+  issue: https://github.com/elastic/elasticsearch/issues/133060
+- class: org.elasticsearch.upgrades.SyntheticSourceRollingUpgradeIT
+  method: testIndexing {upgradedNodes=0}
+  issue: https://github.com/elastic/elasticsearch/issues/133061
 
 # Examples:
 #

server/src/main/java/org/elasticsearch/index/mapper/DynamicFieldsBuilder.java

@@ -334,11 +334,17 @@ public boolean newDynamicStringField(DocumentParserContext context, String name)
                     mapperBuilderContext
                 );
             } else {
+                var indexSettings = context.indexSettings();
                 return createDynamicField(
-                    new TextFieldMapper.Builder(name, context.indexAnalyzers(), SourceFieldMapper.isSynthetic(context.indexSettings()))
-                        .addMultiField(
-                            new KeywordFieldMapper.Builder("keyword", context.indexSettings().getIndexVersionCreated()).ignoreAbove(256)
-                        ),
+                    new TextFieldMapper.Builder(
+                        name,
+                        indexSettings.getIndexVersionCreated(),
+                        context.indexAnalyzers(),
+                        SourceFieldMapper.isSynthetic(indexSettings),
+                        false
+                    ).addMultiField(
+                        new KeywordFieldMapper.Builder("keyword", context.indexSettings().getIndexVersionCreated()).ignoreAbove(256)
+                    ),
                     context
                 );
             }

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/accesscontrol/DocumentSubsetBitsetCache.java

@@ -40,6 +40,8 @@
 
 import java.io.Closeable;
 import java.io.IOException;
+import java.util.Collections;
+import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
@@ -320,7 +322,16 @@ public static List<Setting<?>> getSettings() {
 
     public Map<String, Object> usageStats() {
         final ByteSizeValue ram = ByteSizeValue.ofBytes(ramBytesUsed());
-        return Map.of("count", entryCount(), "memory", ram.toString(), "memory_in_bytes", ram.getBytes());
+        final Cache.Stats cacheStats = bitsetCache.stats();
+
+        final Map<String, Object> stats = new LinkedHashMap<>();
+        stats.put("count", entryCount());
+        stats.put("memory", ram.toString());
+        stats.put("memory_in_bytes", ram.getBytes());
+        stats.put("hits", cacheStats.getHits());
+        stats.put("misses", cacheStats.getMisses());
+        stats.put("evictions", cacheStats.getEvictions());
+        return Collections.unmodifiableMap(stats);
     }
 
     private static final class BitsetCacheKey {

x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authz/accesscontrol/DocumentSubsetBitsetCacheTests.java

@@ -53,6 +53,7 @@
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.IdentityHashMap;
+import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
@@ -62,8 +63,10 @@
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicInteger;
 import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Supplier;
 
 import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.greaterThan;
 import static org.hamcrest.Matchers.is;
 import static org.hamcrest.Matchers.not;
 import static org.hamcrest.Matchers.notNullValue;
@@ -396,9 +399,9 @@ public void testCacheUnderConcurrentAccess() throws Exception {
                 cache.verifyInternalConsistency();
 
                 // Due to cache evictions, we must get more bitsets than fields
-                assertThat(uniqueBitSets.size(), Matchers.greaterThan(FIELD_COUNT));
+                assertThat(uniqueBitSets.size(), greaterThan(FIELD_COUNT));
                 // Due to cache evictions, we must have seen more bitsets than the cache currently holds
-                assertThat(uniqueBitSets.size(), Matchers.greaterThan(cache.entryCount()));
+                assertThat(uniqueBitSets.size(), greaterThan(cache.entryCount()));
                 // Even under concurrent pressure, the cache should hit the expected size
                 assertThat(cache.entryCount(), is(maxCacheCount));
                 assertThat(cache.ramBytesUsed(), is(maxCacheBytes));
@@ -517,6 +520,64 @@ public void testEquivalentMatchAllDocsQuery() {
         assertFalse(DocumentSubsetBitsetCache.isEffectiveMatchAllDocsQuery(new TermQuery(new Term("term"))));
     }
 
+    public void testHitsMissesAndEvictionsStats() throws Exception {
+        // cache that will evict all-but-one element, to test evictions
+        final long maxCacheBytes = EXPECTED_BYTES_PER_BIT_SET + (EXPECTED_BYTES_PER_BIT_SET / 2);
+        final Settings settings = Settings.builder()
+            .put(DocumentSubsetBitsetCache.CACHE_SIZE_SETTING.getKey(), maxCacheBytes + "b")
+            .build();
+        final DocumentSubsetBitsetCache cache = newCache(settings);
+
+        final Supplier<Map<String, Object>> emptyStatsSupplier = () -> {
+            final Map<String, Object> stats = new LinkedHashMap<>();
+            stats.put("count", 0);
+            stats.put("memory", "0b");
+            stats.put("memory_in_bytes", 0L);
+            stats.put("hits", 0L);
+            stats.put("misses", 0L);
+            stats.put("evictions", 0L);
+            return stats;
+        };
+
+        final Map<String, Object> expectedStats = emptyStatsSupplier.get();
+        assertThat(cache.usageStats(), equalTo(expectedStats));
+
+        runTestOnIndex((searchExecutionContext, leafContext) -> {
+            // first lookup - miss
+            final Query query1 = QueryBuilders.termQuery("field-1", "value-1").toQuery(searchExecutionContext);
+            final BitSet bitSet1 = cache.getBitSet(query1, leafContext);
+            assertThat(bitSet1, notNullValue());
+            expectedStats.put("count", 1);
+            expectedStats.put("misses", 1L);
+            expectedStats.put("memory", EXPECTED_BYTES_PER_BIT_SET + "b");
+            expectedStats.put("memory_in_bytes", EXPECTED_BYTES_PER_BIT_SET);
+            assertThat(cache.usageStats(), equalTo(expectedStats));
+
+            // second same lookup - hit
+            final BitSet bitSet1Again = cache.getBitSet(query1, leafContext);
+            assertThat(bitSet1Again, sameInstance(bitSet1));
+            expectedStats.put("hits", 1L);
+            assertThat(cache.usageStats(), equalTo(expectedStats));
+
+            // second query - miss, should evict the first one
+            final Query query2 = QueryBuilders.termQuery("field-2", "value-2").toQuery(searchExecutionContext);
+            final BitSet bitSet2 = cache.getBitSet(query2, leafContext);
+            assertThat(bitSet2, notNullValue());
+            // surprisingly, the eviction callback can call `get` on the cache (asynchronously) which causes another miss (or hit)
+            // so this assertion is about the current state of the code, rather than the expected or desired state.
+            // see https://github.com/elastic/elasticsearch/issues/132842
+            expectedStats.put("misses", 3L);
+            expectedStats.put("evictions", 1L);
+            assertBusy(() -> { assertThat(cache.usageStats(), equalTo(expectedStats)); }, 200, TimeUnit.MILLISECONDS);
+        });
+
+        final Map<String, Object> finalStats = emptyStatsSupplier.get();
+        finalStats.put("hits", 1L);
+        finalStats.put("misses", 3L);
+        finalStats.put("evictions", 2L);
+        assertThat(cache.usageStats(), equalTo(finalStats));
+    }
+
     private void runTestOnIndex(CheckedBiConsumer<SearchExecutionContext, LeafReaderContext, Exception> body) throws Exception {
         runTestOnIndices(1, ctx -> {
             final TestIndexContext indexContext = ctx.get(0);
@@ -638,5 +699,4 @@ private void runTestOnIndices(int numberIndices, CheckedConsumer<List<TestIndexC
     private DocumentSubsetBitsetCache newCache(Settings settings) {
         return new DocumentSubsetBitsetCache(settings, singleThreadExecutor);
     }
-
 }