Skip to content

Commit 707b3c2

Browse files
carlosdelestcarlosdelest
authored
Merge branch 'main' into non-issue/esql-vector-search-functions-basics
2 parents 81f26ec + baf4f2f commit 707b3c2

File tree

13 files changed

+256
-44
lines changed

13 files changed

+256
-44
lines changed

‎docs/changelog/130705.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 130705
2+
summary: Fix `BytesRef2BlockHash`
3+
area: ES|QL
4+
type: bug
5+
issues: []

‎docs/reference/elasticsearch/mapping-reference/sparse-vector.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ Parameters for `index_options` are:
8383
: (Optional, float) Tokens whose weight is less than `tokens_weight_threshold` are considered insignificant and pruned. This value must be between 0 and 1. Default: `0.4`.
8484

8585
::::{note}
86-
The default values for `tokens_freq_ratio_threshold` and `tokens_weight_threshold` were chosen based on tests using ELSERv2 that provided the most optimal results.
86+
The default values for `tokens_freq_ratio_threshold` and `tokens_weight_threshold` were chosen based on tests using ELSERv2 that provided the optimal results.
8787
::::
8888

8989
When token pruning is applied, non-significant tokens will be pruned from the query.

‎docs/reference/elasticsearch/rest-apis/retrievers.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1200,8 +1200,8 @@ Note, however, that wildcard field patterns will only resolve to fields that eit
12001200
12011201
### Examples
12021202
1203-
<!-- - [RRF with the multi-field query format](docs-content://solutions/search/retrievers-examples.md#retrievers-examples-rrf-multi-field-query-format) -->
1204-
<!-- - [Linear retriever with the multi-field query format](docs-content://solutions/search/retrievers-examples.md#retrievers-examples-linear-multi-field-query-format) -->
1203+
- [RRF with the multi-field query format](docs-content://solutions/search/retrievers-examples.md#retrievers-examples-rrf-multi-field-query-format)
1204+
- [Linear retriever with the multi-field query format](docs-content://solutions/search/retrievers-examples.md#retrievers-examples-linear-multi-field-query-format)
12051205
12061206
## Common usage guidelines [retriever-common-parameters]
12071207

‎docs/reference/query-languages/query-dsl/query-dsl-sparse-vector-query.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ GET _search
8080
: (Optional, boolean) If `true` we only input pruned tokens into scoring, and discard non-pruned tokens. It is strongly recommended to set this to `false` for the main query, but this can be set to `true` for a rescore query to get more relevant results. Default: `false`.
8181

8282
::::{note}
83-
The default values for `tokens_freq_ratio_threshold` and `tokens_weight_threshold` were chosen based on tests using ELSERv2 that provided the most optimal results.
83+
The default values for `tokens_freq_ratio_threshold` and `tokens_weight_threshold` were chosen based on tests using ELSERv2 that provided the optimal results.
8484
::::
8585

8686
When token pruning is applied, non-significant tokens will be pruned from the query.

‎docs/reference/query-languages/query-dsl/query-dsl-text-expansion-query.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,7 @@ GET _search
6969
: (Optional, boolean) [preview] If `true` we only input pruned tokens into scoring, and discard non-pruned tokens. It is strongly recommended to set this to `false` for the main query, but this can be set to `true` for a rescore query to get more relevant results. Default: `false`.
7070

7171
::::{note}
72-
The default values for `tokens_freq_ratio_threshold` and `tokens_weight_threshold` were chosen based on tests using ELSER that provided the most optimal results.
72+
The default values for `tokens_freq_ratio_threshold` and `tokens_weight_threshold` were chosen based on tests using ELSER that provided the optimal results.
7373
::::
7474

7575

‎docs/reference/query-languages/query-dsl/query-dsl-weighted-tokens-query.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ POST _search
6666
: (Optional, boolean) If `true` we only input pruned tokens into scoring, and discard non-pruned tokens. It is strongly recommended to set this to `false` for the main query, but this can be set to `true` for a rescore query to get more relevant results. Default: `false`.
6767

6868
::::{note}
69-
The default values for `tokens_freq_ratio_threshold` and `tokens_weight_threshold` were chosen based on tests using ELSER that provided the most optimal results.
69+
The default values for `tokens_freq_ratio_threshold` and `tokens_weight_threshold` were chosen based on tests using ELSER that provided the optimal results.
7070
::::
7171

7272

‎modules/transport-netty4/src/main/java/org/elasticsearch/http/netty4/Netty4HttpServerTransport.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -165,7 +165,7 @@ public Settings settings() {
165165
}
166166

167167
@Override
168-
protected void doStart() {
168+
protected void startInternal() {
169169
boolean success = false;
170170
try {
171171
sharedGroup = sharedGroupFactory.getHttpGroup();

‎server/src/internalClusterTest/java/org/elasticsearch/search/aggregations/bucket/FiltersCancellationIT.java‎

Lines changed: 28 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@
2828
import org.elasticsearch.test.ESIntegTestCase;
2929
import org.elasticsearch.xcontent.XContentBuilder;
3030
import org.elasticsearch.xcontent.json.JsonXContent;
31+
import org.junit.Before;
3132

3233
import java.util.Collection;
3334
import java.util.List;
@@ -39,9 +40,22 @@
3940
import static org.elasticsearch.search.aggregations.AggregationBuilders.filters;
4041
import static org.elasticsearch.search.aggregations.AggregationBuilders.terms;
4142
import static org.hamcrest.Matchers.empty;
43+
import static org.hamcrest.Matchers.equalTo;
4244
import static org.hamcrest.Matchers.greaterThan;
4345
import static org.hamcrest.Matchers.not;
4446

47+
/**
48+
* Ensures the filters aggregation checks task cancellation, by ensuring it doesn't process all the docs.
49+
* <p>
50+
* The CancellableBulkScorer we use to break the execution is called per search thread in the query.
51+
* It currently breaks the "for each doc" into blocks of 4096 docs (x2 every iteration), and checks for cancellation between blocks.
52+
* This test creates N docs and releases N - X permits, to ensure the search request gets cancelled before grabbing all the permits.
53+
* </p>
54+
* <p>
55+
* Also, if the search thread pool size is too high, it can lead to them trying to process too many documents anyway (pool size * 4096),
56+
* eventually blocking the threads (And failing the test). So it's explicitly set to a small number to avoid this.
57+
* </p>
58+
*/
4559
@ESIntegTestCase.SuiteScopeTestCase
4660
public class FiltersCancellationIT extends ESIntegTestCase {
4761

@@ -55,11 +69,12 @@ public class FiltersCancellationIT extends ESIntegTestCase {
5569

5670
@Override
5771
protected Collection<Class<? extends Plugin>> nodePlugins() {
58-
return CollectionUtils.appendToCopy(super.nodePlugins(), pausableFieldPluginClass());
72+
return CollectionUtils.appendToCopy(super.nodePlugins(), PauseScriptPlugin.class);
5973
}
6074

61-
protected Class<? extends Plugin> pausableFieldPluginClass() {
62-
return PauseScriptPlugin.class;
75+
@Override
76+
public Settings nodeSettings(int nodeOrdinal, Settings otherSettings) {
77+
return Settings.builder().put(super.nodeSettings(nodeOrdinal, otherSettings)).put("thread_pool.search.size", 4).build();
6378
}
6479

6580
@Override
@@ -99,6 +114,11 @@ public void setupSuiteScopeCluster() throws Exception {
99114
client().admin().indices().prepareForceMerge(INDEX).setMaxNumSegments(1).get();
100115
}
101116

117+
@Before
118+
public void reset() {
119+
SCRIPT_SEMAPHORE.drainPermits();
120+
}
121+
102122
public void testFiltersCountCancellation() throws Exception {
103123
ensureProperCancellation(
104124
client().prepareSearch(INDEX)
@@ -129,14 +149,14 @@ public void testFiltersSubAggsCancellation() throws Exception {
129149

130150
private void ensureProperCancellation(SearchRequestBuilder searchRequestBuilder) throws Exception {
131151
var searchRequestFuture = searchRequestBuilder.setTimeout(TimeValue.timeValueSeconds(1)).execute();
132-
assertFalse(searchRequestFuture.isCancelled());
133-
assertFalse(searchRequestFuture.isDone());
152+
assertThat(searchRequestFuture.isCancelled(), equalTo(false));
153+
assertThat(searchRequestFuture.isDone(), equalTo(false));
134154

135155
// Check that there are search tasks running
136156
assertThat(getSearchTasks(), not(empty()));
137157

138158
// Wait for the script field to get blocked
139-
assertBusy(() -> { assertThat(SCRIPT_SEMAPHORE.getQueueLength(), greaterThan(0)); });
159+
assertBusy(() -> assertThat(SCRIPT_SEMAPHORE.getQueueLength(), greaterThan(0)));
140160

141161
// Cancel the tasks
142162
// Warning: Adding a waitForCompletion(true)/execute() here sometimes causes tasks to not get canceled and threads to get stuck
@@ -146,8 +166,8 @@ private void ensureProperCancellation(SearchRequestBuilder searchRequestBuilder)
146166

147167
// Ensure the search request finished and that there are no more search tasks
148168
assertBusy(() -> {
149-
assertTrue(searchRequestFuture.isDone());
150-
assertThat(getSearchTasks(), empty());
169+
assertThat("Search request didn't finish", searchRequestFuture.isDone(), equalTo(true));
170+
assertThat("There are dangling search tasks", getSearchTasks(), empty());
151171
});
152172
}
153173

‎server/src/main/java/org/elasticsearch/http/AbstractHttpServerTransport.java‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -235,6 +235,13 @@ private TransportAddress bindAddress(final InetAddress hostAddress) {
235235

236236
protected abstract HttpServerChannel bind(InetSocketAddress hostAddress) throws Exception;
237237

238+
@Override
239+
protected final void doStart() {
240+
startInternal();
241+
}
242+
243+
protected abstract void startInternal();
244+
238245
/**
239246
* Gracefully shut down. If {@link HttpTransportSettings#SETTING_HTTP_SERVER_SHUTDOWN_GRACE_PERIOD} is zero, the default, then
240247
* forcefully close all open connections immediately.
@@ -251,7 +258,7 @@ private TransportAddress bindAddress(final InetAddress hostAddress) {
251258
* </ol>
252259
*/
253260
@Override
254-
protected void doStop() {
261+
protected final void doStop() {
255262
synchronized (httpServerChannels) {
256263
if (httpServerChannels.isEmpty() == false) {
257264
try {

‎server/src/test/java/org/elasticsearch/http/AbstractHttpServerTransportTests.java‎

Lines changed: 12 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -211,14 +211,10 @@ protected HttpServerChannel bind(InetSocketAddress hostAddress) {
211211
}
212212

213213
@Override
214-
protected void doStart() {
215-
216-
}
214+
protected void startInternal() {}
217215

218216
@Override
219-
protected void stopInternal() {
220-
221-
}
217+
protected void stopInternal() {}
222218

223219
@Override
224220
public HttpStats stats() {
@@ -295,14 +291,10 @@ protected HttpServerChannel bind(InetSocketAddress hostAddress) {
295291
}
296292

297293
@Override
298-
protected void doStart() {
299-
300-
}
294+
protected void startInternal() {}
301295

302296
@Override
303-
protected void stopInternal() {
304-
305-
}
297+
protected void stopInternal() {}
306298

307299
@Override
308300
public HttpStats stats() {
@@ -385,7 +377,7 @@ protected HttpServerChannel bind(InetSocketAddress hostAddress) {
385377
}
386378

387379
@Override
388-
protected void doStart() {}
380+
protected void startInternal() {}
389381

390382
@Override
391383
protected void stopInternal() {}
@@ -516,7 +508,7 @@ protected HttpServerChannel bind(InetSocketAddress hostAddress) {
516508
}
517509

518510
@Override
519-
protected void doStart() {}
511+
protected void startInternal() {}
520512

521513
@Override
522514
protected void stopInternal() {}
@@ -567,9 +559,7 @@ protected HttpServerChannel bind(InetSocketAddress hostAddress) {
567559
}
568560

569561
@Override
570-
protected void doStart() {
571-
572-
}
562+
protected void startInternal() {}
573563

574564
@Override
575565
protected void stopInternal() {
@@ -710,14 +700,10 @@ protected HttpServerChannel bind(InetSocketAddress hostAddress) {
710700
}
711701

712702
@Override
713-
protected void doStart() {
714-
715-
}
703+
protected void startInternal() {}
716704

717705
@Override
718-
protected void stopInternal() {
719-
720-
}
706+
protected void stopInternal() {}
721707

722708
@Override
723709
public HttpStats stats() {
@@ -774,7 +760,7 @@ protected HttpServerChannel bind(InetSocketAddress hostAddress) {
774760
}
775761

776762
@Override
777-
protected void doStart() {}
763+
protected void startInternal() {}
778764

779765
@Override
780766
protected void stopInternal() {}
@@ -856,7 +842,7 @@ protected HttpServerChannel bind(InetSocketAddress hostAddress) {
856842
}
857843

858844
@Override
859-
protected void doStart() {}
845+
protected void startInternal() {}
860846

861847
@Override
862848
protected void stopInternal() {}
@@ -1226,7 +1212,7 @@ protected HttpServerChannel bind(InetSocketAddress hostAddress) {
12261212
}
12271213

12281214
@Override
1229-
protected void doStart() {
1215+
protected void startInternal() {
12301216
bindServer();
12311217
}
12321218

0 commit comments

Comments
 (0)