You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/reference/query-languages/query-dsl/query-dsl-sparse-vector-query.md
+9-1Lines changed: 9 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -83,7 +83,15 @@ GET _search
83
83
The default values for `tokens_freq_ratio_threshold` and `tokens_weight_threshold` were chosen based on tests using ELSERv2 that provided the most optimal results.
84
84
::::
85
85
86
-
86
+
When token pruning is applied, non-significant tokens will be pruned from the query.
87
+
Non-significant tokens can be defined as tokens that meet both of the following criteria:
88
+
* The token appears much more frequently than most tokens, indicating that it is a very common word and may not benefit the overall search results much.
89
+
* The weight/score is so low that the token is likely not very relevant to the original term
90
+
91
+
Both the token frequency threshold and weight threshold must show the token is non-significant in order for the token to be pruned.
92
+
This ensures that:
93
+
* The tokens that are kept are frequent enough and have significant scoring.
94
+
* Very infrequent tokens that may not have as high of a score are removed.
87
95
88
96
## Example ELSER query [sparse-vector-query-example]
0 commit comments