Fixups, pt. III

Author: rschu1ze

docs/en/engines/table-engines/mergetree-family/annindexes.md

@@ -78,7 +78,7 @@ ClickHouse provides a special "vector similarity" index to perform approximate n
 :::note
 Vector similarity indexes are currently experimental.
 To enable them, please first run `SET allow_experimental_vector_similarity_index = 1`.
-If you run into problems, kindly open an issue at github.com/clickhouse/clickhouse/issues.
+If you run into problems, kindly open an issue in the [ClickHouse repository](github.com/clickhouse/clickhouse/issues).
 :::
 
 ### Creating a Vector Similarity Index {#creating-a-vector-similarity-index}
@@ -174,13 +174,13 @@ LIMIT <N>
 ClickHouse's query optimizer tries to match above query template and make use of available vector similarity indexes.
 A query can only use a vector similarity index if the distance function in the SELECT query is the same as the distance function in the index definition.
 
-Advanced users may provide a custom value for setting [hnsw_candidate_list_size_for_search](../../../operations/settings/settings.md#hnsw_candidate_list_size_for_search) (also know as HNSW hyperparameter `ef_search`) to tune the size of the candidate list during search (e.g.  `SELECT [...] SETTINGS hnsw_candidate_list_size_for_search = <value>`).
+Advanced users may provide a custom value for setting [hnsw_candidate_list_size_for_search](../../../operations/settings/settings.md#hnsw_candidate_list_size_for_search) (also know as HNSW hyperparameter "ef_search") to tune the size of the candidate list during search (e.g.  `SELECT [...] SETTINGS hnsw_candidate_list_size_for_search = <value>`).
 The default value of the setting 256 works well in the majority of use cases.
 Higher setting values mean better accuracy at the cost of slower performance.
 
 If the query can use a vector similarity index, ClickHouse checks that the LIMIT `<N>` provided in SELECT queries is within reasonable bounds.
 More specifically, an error is returned if `<N>` is bigger than the value of setting [max_limit_for_vector_search_queries](../../../operations/settings/settings.md#max_limit_for_vector_search_queries) with default value 100.
-Too large LIMITs can slow down searches and usually indicate a usage error.
+Too large LIMIT values can slow down searches and usually indicate a usage error.
 
 To check if a SELECT query uses a vector similarity index, you can prefix the query with `EXPLAIN indexes = 1`.
 
@@ -231,21 +231,22 @@ To enforce index usage, you can run the SELECT query with setting [force_data_sk
 
 **Post-filtering and Pre-filtering**
 
-Users may optionally specify a `WHERE` clause with additional filter conditions in SELECT queries.
-Depending on these filter conditions, ClickHouse will utilize post-filtering or pre-filtering.
-These two strategies determine the order in which the filters are evaluated:
-- With post-filtering, the vector similarity index is evaluated first, afterwards ClickHouse evaluates the additional filter(s) specified of the `WHERE` clause.
-- With pre-filtering, the filter evaluation order is the other way round.
+Users may optionally specify a `WHERE` clause with additional filter conditions for the SELECT query.
+ClickHouse will evaluate these filter conditions using post-filtering or pre-filtering strategy.
+In short, both strategies determine the order in which the filters are evaluated:
+- Post-filtering means that the vector similarity index is evaluated first, afterwards ClickHouse evaluates the additional filter(s) specified in the `WHERE` clause.
+- Pre-filtering means that the filter evaluation order is the other way round.
 
-Both strategies have different trade-offs:
-- Post-filtering has the general problem that it may return less than the number of rows requested in the `LIMIT <N>` clause. This situation happens when at least one of the result rows returned by the vector similarity index fails to satisfy the additional filters.
-- Pre-filtering is generally an unsolved problem. Some specialized vector databases implement it but most databases including ClickHouse will fall back to exact neighbor search, i.e., a brute-force scan without index.
+The strategies have different trade-offs:
+- Post-filtering has the general problem that it may return less than the number of rows requested in the `LIMIT <N>` clause. This situation happens when one or more result rows returned by the vector similarity index fails to satisfy the additional filters.
+- Pre-filtering is generally an unsolved problem. Certain specialized vector databases provide pre-filtering algorithms but most relational databases (including ClickHouse) will fall back to exact neighbor search, i.e., a brute-force scan without index.
 
-What strategy is used comes down to whether ClickHouse can use indexes for the additional filter conditions.
-If no index can be used, post-filtering will be applied.
+What strategy is used depends on the filter condition.
+
+*Additional filters are part of the partition key*
 
 If the additional filter condition is part of the partition key, then ClickHouse will apply partition pruning.
-For example, assuming that the table is range-partitioned by `year`:
+As an example, a table is range-partitioned by column `year` and the following query is run:
 
 ```sql
 WITH [0., 2.] AS reference_vec
@@ -256,20 +257,30 @@ ORDER BY L2Distance(vec, reference_vec) ASC
 LIMIT 3;
 ```
 
-ClickHouse will ignore all partitions but the one for year 2025.
-Within this partition, a post-filtering strategy will be applied.
+ClickHouse will prune all partitions except the 2025 one.
+
+*Additional filters cannot be evaluated using indexes*
+
+If additional filter conditions cannot be evaluated using indexes (primary key index, skipping index), ClickHouse will apply post-filtering.
+
+*Additional filters can be evaluated using the primary key index*
 
-If the additional filter condition is on the primary key columns and the filter selects some but not all ranges of a part, then Clickhouse will fall back to exact neighbour search (brute-force scan without index) on the selected ranges of the part.
-If the primary key filter selects entire parts, Clickhouse will use the vector similarity index on those parts to retrieve results.
+If additional filter conditions can be evaluated using the [primary key](mergetree.md#primary-key) (i.e., they form a prefix of the primary key) and
+- the filter condition eliminates at least one row within a part, the ClickHouse will fall back to pre-filtering for the "surviving" ranges within the part,
+- the filter condition eliminates no rows within a part, the ClickHouse will perform post-filtering for the part.
 
-In case additional filter conditions on columns can make use of skip indexes (minmax, set etc), Clickhouse by default chooses a post-filtering strategy.
-Clickhouse gives higher priority to the vector similarity index because the vector index is expected to deliver business value by accelerating semantic search response times.
+In practical use cases, the latter case is rather unlikely.
 
-Clickhouse provides 2 settings for finer control on post-filtering and pre-filtering:
+*Additional filters can be evaluated using skipping index*
 
-When the additional filter conditions are extremely selective, it is possible that brute force search on a small filtered set of rows gives better results then post-filtering using the vector search.
-Users can force pre-filtering by setting [vector_search_filter_strategy](../../../operations/settings/settings#vector_search_filter_strategy) to `prefilter` (default is `auto` which is equivalent to `postfilter`).
-An example query where pre-filtering could be a good choice is
+If additional filter conditions can be evaluated using [skipping indexes](mergetree.md#table_engine-mergetree-data_skipping-indexes) (minmax index, set index, etc.), Clickhouse performs post-filtering.
+In such cases, the vector similarity index is evaluated first as it is expected to remove the most rows relative to other skipping indexes.
+
+For finer control over post-filtering vs. pre-filtering, two settings can be used:
+
+Setting [vector_search_filter_strategy](../../../operations/settings/settings#vector_search_filter_strategy) (default: `auto` which implements above heuristics) may be set to `prefilter`.
+This is useful to force pre-filtering in cases where the additional filter conditions are extremely selective.
+As an example, the following query may benefit from pre-filtering:
 
 ```sql
 SELECT bookid, author, title
@@ -279,24 +290,25 @@ ORDER BY cosineDistance(book_vector, getEmbedding('Books on ancient Asian empire
 LIMIT 10
 ```
 
-Assuming that only very few books cost less than $2, post-filtering may return zero rows because the top 10 matches returned by the vector index could all be priced above $2.
-By forcing pre-filtering (add `SETTINGS vector_search_filter_strategy = 'prefilter'` to the query), ClickHouse first finds all books with a price of less than $2 and then executes a brute-force vector search on the matches.
+Assuming that only a very small number of books cost less than $2, post-filtering may return zero rows because the top 10 matches returned by the vector index could all be priced above $2.
+By forcing pre-filtering (add `SETTINGS vector_search_filter_strategy = 'prefilter'` to the query), ClickHouse first finds all books with a price of less than $2 and then executes a brute-force vector search for the found books.
 
-As mentioned above, post-filtering may return less matches then specified in the `LIMIT <N>` clause.
-Consider query
+As an alternative approach to resolve above issue, setting [vector_search_postfilter_multiplier](../../../operations/settings/settings#vector_search_postfilter_multiplier) (default: `1.0`) may be configured to a value > `1.0` (for example, `2.0`).
+The number of nearest neighbors fetched from the vector index is multiplied by the setting value and then the additional filter to be applied on those rows to return LIMIT-many rows.
+As an example, we can query again but with multiplier `3.0`:
 
 ```sql
 SELECT bookid, author, title
 FROM books
-WHERE published_year <= 2000
+WHERE price < 2.00
 ORDER BY cosineDistance(book_vector, getEmbedding('Books on ancient Asian empires'))
 LIMIT 10
+SETTING vector_search_postfilter_multiplier = 3.0;
 ```
 
-With post-filtering, some of the 10 nearest matching books returned by the vector index may be pruned from the result because they were published later than in the year 2000.
-As a result, the query may return less rows than the user requested.
-For such cases, you can set parameter [vector_search_postfilter_multiplier](../../../operations/settings/settings#vector_search_postfilter_multiplier) to a value > 1.0 (for example, 2.0) to indicate that N times this factor many matches should be returned by the vector index and then the additional filter to be applied on those rows to return the result of 10 rows.
-We note that this method can mitigate the problem with post-filtering but in extreme cases (extremely selective WHERE condition), there may still less than N requested rows returned.
+ClickHouse will fetch 3.0 x 10 = 30 nearest neighbors from the vector index in each part and afterwards evaluate the additional filters.
+Only the ten closest neighbors will be returned.
+We note that setting `vector_search_postfilter_multiplier` can mitigate the problem but in extreme cases (very selective WHERE condition), it is still possible that less than N requested rows returned.
 
 ### Performance Tuning {#performance-tuning}
 

src/Core/Settings.cpp

@@ -6567,10 +6567,7 @@ Enable experimental hash functions
 Allow the obsolete Object data type
 )", EXPERIMENTAL) \
     DECLARE(Bool, allow_experimental_time_series_table, false, R"(
-Allows creation of tables with the [TimeSeries](../../engines/table-engines/integrations/time-series.md) table engine.
-
-Possible values:
-
+Allows creation of tables with the [TimeSeries](../../engines/table-engines/integrations/time-series.md) table engine. Possible values:
 - 0 — the [TimeSeries](../../engines/table-engines/integrations/time-series.md) table engine is disabled.
 - 1 — the [TimeSeries](../../engines/table-engines/integrations/time-series.md) table engine is enabled.
 )", EXPERIMENTAL) \
@@ -6587,13 +6584,10 @@ SELECT queries with LIMIT bigger than this setting cannot use vector similarity
 The size of the dynamic candidate list when searching the vector similarity index, also known as 'ef_search'.
 )", EXPERIMENTAL) \
     DECLARE(VectorSearchFilterStrategy, vector_search_filter_strategy, VectorSearchFilterStrategy::AUTO, R"(
-If a vector search query has a WHERE clause, this setting determines if it is evaluated first (pre-filtering) OR if the vector similarity index is checked first (post-filtering).
-
-Possible values:
-
-'auto' - Postfiltering (the exact semantics may change in future).
-'postfilter' - Use vector similarity index to identify the nearest neighbours, then apply other filters
-'prefilter' - Evaluate other filters first, then perform brute-force search to identify neighbours.
+If a vector search query has a WHERE clause, this setting determines if it is evaluated first (pre-filtering) OR if the vector similarity index is checked first (post-filtering). Possible values:
+- 'auto' - Postfiltering (the exact semantics may change in future).
+- 'postfilter' - Use vector similarity index to identify the nearest neighbours, then apply other filters
+- 'prefilter' - Evaluate other filters first, then perform brute-force search to identify neighbours.
 )", EXPERIMENTAL) \
     DECLARE(Float, vector_search_postfilter_multiplier, 1.0, R"(
 Multiply the fetched nearest neighbors from the vector similarity index by this number before performing post-filtering on other predicates.

src/Storages/MergeTree/MergeTreeIndexVectorSimilarity.cpp

@@ -467,7 +467,7 @@ std::vector<UInt64> MergeTreeIndexConditionVectorSimilarity::calculateApproximat
 
     size_t limit = parameters->limit;
     if (parameters->additional_filters_present)
-        /// Post-filters may remove matches. Allow to fetch more rows by a factor to compensate.
+        /// Additional filters mean post-filtering which means that matches may be removed. To compensate, allow to fetch more rows by a factor.
         limit = std::min(static_cast<size_t>(limit * postfilter_multiplier), max_limit);
 
     /// We want to run the search with the user-provided value for setting hnsw_candidate_list_size_for_search (aka. expansion_search).

src/Storages/MergeTree/MergeTreeIndices.h

@@ -105,7 +105,7 @@ struct VectorSearchParameters
     std::vector<Float64> reference_vector;
 
     /// Other metadata
-    bool additional_filters_present;
+    bool additional_filters_present; /// SELECT contains a WHERE or PREWHERE clause
 };
 
 /// Stores some info about a single block of data.

tests/queries/0_stateless/02354_vector_search_postfiltering_bug.reference

@@ -0,0 +1,20 @@
+-- Tags: no-fasttest, long, no-asan, no-ubsan, no-debug
+-- Test for Bug 78161
+
+SET allow_experimental_vector_similarity_index = 1;
+SET enable_analyzer = 1;
+
+CREATE TABLE tab (id Int32, vec Array(Float32)) ENGINE = MergeTree() ORDER BY id SETTINGS index_granularity = 128;
+INSERT INTO tab SELECT number, [randCanonical(), randCanonical()] FROM numbers(100000);
+
+-- Create index
+ALTER TABLE tab ADD INDEX idx_vec vec TYPE vector_similarity('hnsw', 'cosineDistance', 2, 'f32', 64, 400);
+ALTER TABLE tab MATERIALIZE INDEX idx_vec SETTINGS mutations_sync=2;
+
+WITH [1., 2.] AS reference_vec
+SELECT *
+FROM tab
+PREWHERE id < 5000
+ORDER BY cosineDistance(vec, reference_vec) ASC
+LIMIT 10
+FORMAT Null;

tests/queries/0_stateless/02354_vector_search_postfiltering_bug.sql

@@ -22,3 +22,4 @@ The first 3 neighbours returned by vector index dont pass the attr2 >= 1008 filt
 10
 11
 12
+-- Negative parameter values throw an exception

tests/queries/0_stateless/02354_vector_search_pre_and_post_filtering.reference

@@ -1,6 +1,6 @@
 -- Tags: no-fasttest, no-ordinary-database
 
--- Tests for vector search and pre-filtering/post-filtering.
+-- Tests pre vs. post-filtering for vector search.
 
 SET allow_experimental_vector_similarity_index = 1;
 SET enable_analyzer = 1;
@@ -11,15 +11,15 @@ DROP TABLE IF EXISTS tab;
 CREATE TABLE tab
 (
     id Int32,
-    dt Date,
+    date Date,
     attr1 Int32,
     attr2 Int32,
-    vector Array(Float32),
-    INDEX attr1_index attr1 TYPE minmax,
-    INDEX vector_index vector TYPE vector_similarity('hnsw', 'L2Distance', 2) GRANULARITY 10000
+    vec Array(Float32),
+    INDEX idx_attr1 attr1 TYPE minmax,
+    INDEX idx_vec vec TYPE vector_similarity('hnsw', 'L2Distance', 2) GRANULARITY 10000
 )
 ENGINE = MergeTree
-PARTITION BY dt
+PARTITION BY date
 ORDER BY id
 SETTINGS index_granularity = 3;
 
@@ -44,7 +44,7 @@ SELECT trimLeft(explain) FROM (
     EXPLAIN indexes = 1
     SELECT id
     FROM tab
-    ORDER BY L2Distance(vector, [1.0, 1.0])
+    ORDER BY L2Distance(vec, [1.0, 1.0])
     LIMIT 2
     SETTINGS vector_search_filter_strategy = 'prefilter'
 )
@@ -56,7 +56,7 @@ SELECT trimLeft(explain) FROM (
     SELECT id
     FROM tab
     WHERE attr2 >= 1006
-    ORDER BY L2Distance(vector, [1.0, 1.0])
+    ORDER BY L2Distance(vec, [1.0, 1.0])
     LIMIT 2
     SETTINGS vector_search_filter_strategy = 'prefilter'
 )
@@ -68,7 +68,7 @@ SELECT trimLeft(explain) FROM (
     SELECT id
     FROM tab
     WHERE attr1 <= 105
-    ORDER BY L2Distance(vector, [1.0, 1.0])
+    ORDER BY L2Distance(vec, [1.0, 1.0])
     LIMIT 2
     SETTINGS vector_search_filter_strategy = 'prefilter'
 )
@@ -80,7 +80,7 @@ SELECT trimLeft(explain) FROM (
     SELECT id
     FROM tab
     WHERE id <= 6
-    ORDER BY L2Distance(vector, [1.0, 1.0])
+    ORDER BY L2Distance(vec, [1.0, 1.0])
     LIMIT 2
     SETTINGS vector_search_filter_strategy = 'prefilter'
 )
@@ -93,7 +93,7 @@ SELECT trimLeft(explain) FROM (
     EXPLAIN indexes = 1
     SELECT id
     FROM tab
-    ORDER BY L2Distance(vector, [1.0, 1.0])
+    ORDER BY L2Distance(vec, [1.0, 1.0])
     LIMIT 2
     SETTINGS vector_search_filter_strategy = 'postfilter'
 )
@@ -104,8 +104,8 @@ SELECT trimLeft(explain) FROM (
     EXPLAIN indexes = 1
     SELECT id
     FROM tab
-    WHERE dt <= '2025-01-02'
-    ORDER BY L2Distance(vector, [1.0, 1.0])
+    WHERE date <= '2025-01-02'
+    ORDER BY L2Distance(vec, [1.0, 1.0])
     LIMIT 2
     SETTINGS vector_search_filter_strategy = 'postfilter'
 )
@@ -116,9 +116,9 @@ SELECT trimLeft(explain) FROM (
     EXPLAIN indexes = 1
     SELECT id
     FROM tab
-    WHERE dt = '2025-01-03'
+    WHERE date = '2025-01-03'
     AND attr1 = 110
-    ORDER BY L2Distance(vector, [1.0, 1.0])
+    ORDER BY L2Distance(vec, [1.0, 1.0])
     LIMIT 2
     SETTINGS vector_search_filter_strategy = 'postfilter'
 )
@@ -127,8 +127,8 @@ WHERE explain LIKE '%vector_similarity%';
 SELECT '-- Additional WHERE clauses present, 2 full parts selected by partition key / 1 part partially selected by PK, index usage not expected';
 SELECT id
 FROM tab
-WHERE dt = '2025-01-03' AND id <= 9
-ORDER BY L2Distance(vector, [1.0, 1.0])
+WHERE date = '2025-01-03' AND id <= 9
+ORDER BY L2Distance(vec, [1.0, 1.0])
 LIMIT 2
 SETTINGS log_comment = '02354_vector_search_post_filter_strategy_query1';
 
@@ -143,16 +143,24 @@ AND type = 'QueryFinish';
 SELECT 'The first 3 neighbours returned by vector index dont pass the attr2 >= 1008 filter. Hence no rows returned by the query...';
 SELECT id
 FROM tab
-WHERE dt = '2025-01-03' AND attr2 >= 1008
-ORDER BY L2Distance(vector, [1.0, 1.0])
+WHERE date = '2025-01-03' AND attr2 >= 1008
+ORDER BY L2Distance(vec, [1.0, 1.0])
 LIMIT 3;
 
 SELECT '... but there are results for the same query with postfilter multiplier = 2.0';
 SELECT id
 FROM tab
-WHERE dt = '2025-01-03' AND attr2 >= 1008
-ORDER BY L2Distance(vector, [1.0, 1.0])
+WHERE date = '2025-01-03' AND attr2 >= 1008
+ORDER BY L2Distance(vec, [1.0, 1.0])
 LIMIT 3
 SETTINGS vector_search_postfilter_multiplier = 2.0;
 
+SELECT '-- Negative parameter values throw an exception';
+SELECT id
+FROM tab
+WHERE date = '2025-01-03' AND attr2 >= 1008
+ORDER BY L2Distance(vec, [1.0, 1.0])
+LIMIT 3
+SETTINGS vector_search_postfilter_multiplier = -1.0; -- { serverError INVALID_SETTING_VALUE }
+
 DROP TABLE tab;