Skip to content

Commit fc5a3cb

Browse files
henningandersenhenningandersen
authored
Merge branch 'main' into add_blob_cache_epoch_metric
2 parents 5a333d7 + 0dbf9f7 commit fc5a3cb

File tree

25 files changed

+1457
-1235
lines changed

25 files changed

+1457
-1235
lines changed

‎docs/reference/query-languages/esql/_snippets/commands/examples/rerank.csv-spec/combine.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do not edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
FROM books METADATA _score
5+
| WHERE MATCH(description, "hobbit") OR MATCH(author, "Tolkien")
6+
| SORT _score DESC
7+
| LIMIT 100
8+
| RERANK rerank_score = "hobbit" ON description, author WITH { "inference_id" : "test_reranker" }
9+
| EVAL original_score = _score, _score = rerank_score + original_score
10+
| SORT _score
11+
| LIMIT 3
12+
| KEEP title, original_score, rerank_score, _score
13+
```
14+
15+
| title:text | _score:double | rerank_score:double | rerank_score:double |
16+
| --- | --- | --- | --- |
17+
| Poems from the Hobbit | 4.012462615966797 | 0.001396648003719747 | 0.001396648003719747 |
18+
| The Lord of the Rings - Boxed Set | 3.768855094909668 | 0.0010020040208473802 | 0.001396648003719747 |
19+
| Return of the King Being the Third Part of The Lord of the Rings | 3.6248698234558105 | 9.000900317914784E-4 | 0.001396648003719747 |

‎docs/reference/query-languages/esql/_snippets/commands/examples/rerank.csv-spec/simple-query.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do not edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
FROM books METADATA _score
5+
| WHERE MATCH(description, "hobbit")
6+
| SORT _score DESC
7+
| LIMIT 100
8+
| RERANK "hobbit" ON description WITH { "inference_id" : "test_reranker" }
9+
| LIMIT 3
10+
| KEEP title, _score
11+
```
12+
13+
| title:text | _score:double |
14+
| --- | --- |
15+
| Poems from the Hobbit | 0.0015673980815336108 |
16+
| A Tolkien Compass: Including J. R. R. Tolkien's Guide to the Names in The Lord of the Rings | 0.007936508394777775 |
17+
| Return of the King Being the Third Part of The Lord of the Rings | 9.960159659385681E-4 |

‎docs/reference/query-languages/esql/_snippets/commands/examples/rerank.csv-spec/two-queries.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
% This is generated by ESQL's AbstractFunctionTestCase. Do not edit it. See ../README.md for how to regenerate it.
2+
3+
```esql
4+
FROM books METADATA _score
5+
| WHERE MATCH(description, "hobbit") OR MATCH(author, "Tolkien")
6+
| SORT _score DESC
7+
| LIMIT 100
8+
| RERANK rerank_score = "hobbit" ON description, author WITH { "inference_id" : "test_reranker" }
9+
| SORT rerank_score
10+
| LIMIT 3
11+
| KEEP title, _score, rerank_score
12+
```
13+
14+
| title:text | _score:double | rerank_score:double |
15+
| --- | --- | --- |
16+
| Return of the Shadow | 2.8181066513061523 | 5.740527994930744E-4 |
17+
| Return of the King Being the Third Part of The Lord of the Rings | 3.6248698234558105 | 9.000900317914784E-4 |
18+
| The Lays of Beleriand | 1.3002015352249146 | 9.36329597607255E-4 |

‎docs/reference/query-languages/esql/_snippets/commands/layout/rerank.md‎

Lines changed: 6 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -100,61 +100,17 @@ If you don't want to increase the timeout limit, try the following:
100100

101101
Rerank search results using a simple query and a single field:
102102

103-
```esql
104-
FROM books
105-
| WHERE MATCH(title, "science fiction")
106-
| SORT _score DESC
107-
| LIMIT 100
108-
| RERANK "science fiction" ON (title) WITH { "inference_id" : "my_reranker" }
109-
| LIMIT 3
110-
| KEEP title, _score
111-
```
112103

113-
| title:keyword | _score:double |
114-
|---------------|---------------|
115-
| Neuromancer | 0.98 |
116-
| Dune | 0.95 |
117-
| Foundation | 0.92 |
104+
:::{include} ../examples/rerank.csv-spec/simple-query.md
105+
:::
118106

119107
Rerank search results using a query and multiple fields, and store the new score
120108
in a column named `rerank_score`:
121109

122-
```esql
123-
FROM movies
124-
| WHERE MATCH(title, "dystopian future") OR MATCH(synopsis, "dystopian future")
125-
| SORT _score DESC
126-
| LIMIT 100
127-
| RERANK rerank_score = "dystopian future" ON (title, synopsis) WITH { "inference_id" : "my_reranker" }
128-
| SORT rerank_score DESC
129-
| LIMIT 5
130-
| KEEP title, _score, rerank_score
131-
```
132-
133-
| title:keyword | _score:double | rerank_score:double |
134-
|-----------------|---------------|---------------------|
135-
| Blade Runner | 8.75 | 0.99 |
136-
| The Matrix | 9.12 | 0.97 |
137-
| Children of Men | 8.50 | 0.96 |
138-
| Akira | 8.99 | 0.94 |
139-
| Gattaca | 8.65 | 0.91 |
110+
:::{include} ../examples/rerank.csv-spec/two-queries.md
111+
:::
140112

141113
Combine the original score with the reranked score:
142114

143-
```esql
144-
FROM movies
145-
| WHERE MATCH(title, "dystopian future") OR MATCH(synopsis, "dystopian future")
146-
| SORT _score DESC
147-
| LIMIT 100
148-
| RERANK rerank_score = "dystopian future" ON (title, synopsis) WITH { "inference_id" : "my_reranker" }
149-
| EVAL original_score = _score, _score = rerank_score + original_score
150-
| SORT _score DESC
151-
| LIMIT 2
152-
| KEEP title, original_score, rerank_score, _score
153-
```
154-
155-
| title:keyword | original_score:double | rerank_score:double | _score:double |
156-
|---------------|-----------------------|---------------------|---------------|
157-
| The Matrix | 9.12 | 0.97 | 10.09 |
158-
| Akira | 8.99 | 0.94 | 9.93 |
159-
160-
115+
:::{include} ../examples/rerank.csv-spec/combine.md
116+
:::

‎docs/reference/query-languages/esql/_snippets/functions/layout/copy_sign.md‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎modules/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3Repository.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@
4242
import org.elasticsearch.repositories.blobstore.MeteredBlobStoreRepository;
4343
import org.elasticsearch.snapshots.SnapshotId;
4444
import org.elasticsearch.snapshots.SnapshotsService;
45+
import org.elasticsearch.snapshots.SnapshotsServiceUtils;
4546
import org.elasticsearch.threadpool.Scheduler;
4647
import org.elasticsearch.threadpool.ThreadPool;
4748
import org.elasticsearch.xcontent.NamedXContentRegistry;
@@ -388,7 +389,7 @@ private static ByteSizeValue objectSizeLimit(ByteSizeValue chunkSize, ByteSizeVa
388389
@Override
389390
public void finalizeSnapshot(final FinalizeSnapshotContext finalizeSnapshotContext) {
390391
final FinalizeSnapshotContext wrappedFinalizeContext;
391-
if (SnapshotsService.useShardGenerations(finalizeSnapshotContext.repositoryMetaVersion()) == false) {
392+
if (SnapshotsServiceUtils.useShardGenerations(finalizeSnapshotContext.repositoryMetaVersion()) == false) {
392393
final ListenableFuture<Void> metadataDone = new ListenableFuture<>();
393394
wrappedFinalizeContext = new FinalizeSnapshotContext(
394395
finalizeSnapshotContext.serializeProjectMetadata(),

‎qa/repository-multi-version/src/test/java/org/elasticsearch/upgrades/MultiVersionRepositoryAccessIT.java‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818
import org.elasticsearch.common.settings.Settings;
1919
import org.elasticsearch.index.IndexVersion;
2020
import org.elasticsearch.index.IndexVersions;
21-
import org.elasticsearch.snapshots.SnapshotsService;
21+
import org.elasticsearch.snapshots.SnapshotsServiceUtils;
2222
import org.elasticsearch.test.rest.ESRestTestCase;
2323
import org.elasticsearch.test.rest.ObjectPath;
2424
import org.elasticsearch.xcontent.XContentParser;
@@ -183,7 +183,7 @@ public void testUpgradeMovesRepoToNewMetaVersion() throws IOException {
183183
// incompatibility in the downgrade test step. We verify that it is impossible here and then create the repo using verify=false
184184
// to check behavior on other operations below.
185185
final boolean verify = TEST_STEP != TestStep.STEP3_OLD_CLUSTER
186-
|| SnapshotsService.includesUUIDs(minNodeVersion)
186+
|| SnapshotsServiceUtils.includesUUIDs(minNodeVersion)
187187
|| minNodeVersion.before(IndexVersions.V_7_12_0);
188188
if (verify == false) {
189189
expectThrowsAnyOf(EXPECTED_BWC_EXCEPTIONS, () -> createRepository(repoName, false, true));
@@ -208,7 +208,7 @@ public void testUpgradeMovesRepoToNewMetaVersion() throws IOException {
208208
ensureSnapshotRestoreWorks(repoName, "snapshot-2", shards, index);
209209
}
210210
} else {
211-
if (SnapshotsService.includesUUIDs(minNodeVersion) == false) {
211+
if (SnapshotsServiceUtils.includesUUIDs(minNodeVersion) == false) {
212212
assertThat(TEST_STEP, is(TestStep.STEP3_OLD_CLUSTER));
213213
expectThrowsAnyOf(EXPECTED_BWC_EXCEPTIONS, () -> listSnapshots(repoName));
214214
expectThrowsAnyOf(EXPECTED_BWC_EXCEPTIONS, () -> deleteSnapshot(repoName, "snapshot-1"));

‎server/src/internalClusterTest/java/org/elasticsearch/snapshots/CorruptedBlobStoreRepositoryIT.java‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -327,7 +327,7 @@ public void testHandlingMissingRootLevelSnapshotMetadata() throws Exception {
327327

328328
logger.info("--> verify that repo is assumed in old metadata format");
329329
assertThat(
330-
SnapshotsService.minCompatibleVersion(IndexVersion.current(), getRepositoryData(repoName), null),
330+
SnapshotsServiceUtils.minCompatibleVersion(IndexVersion.current(), getRepositoryData(repoName), null),
331331
is(SnapshotsService.OLD_SNAPSHOT_FORMAT)
332332
);
333333

@@ -336,7 +336,7 @@ public void testHandlingMissingRootLevelSnapshotMetadata() throws Exception {
336336

337337
logger.info("--> verify that repository is assumed in new metadata format after removing corrupted snapshot");
338338
assertThat(
339-
SnapshotsService.minCompatibleVersion(IndexVersion.current(), getRepositoryData(repoName), null),
339+
SnapshotsServiceUtils.minCompatibleVersion(IndexVersion.current(), getRepositoryData(repoName), null),
340340
is(IndexVersion.current())
341341
);
342342
final RepositoryData finalRepositoryData = getRepositoryData(repoName);

‎server/src/main/java/org/elasticsearch/action/admin/cluster/repositories/cleanup/TransportCleanupRepositoryAction.java‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@
3737
import org.elasticsearch.repositories.RepositoryCleanupResult;
3838
import org.elasticsearch.repositories.RepositoryData;
3939
import org.elasticsearch.repositories.blobstore.BlobStoreRepository;
40-
import org.elasticsearch.snapshots.SnapshotsService;
40+
import org.elasticsearch.snapshots.SnapshotsServiceUtils;
4141
import org.elasticsearch.tasks.Task;
4242
import org.elasticsearch.threadpool.ThreadPool;
4343
import org.elasticsearch.transport.TransportService;
@@ -178,8 +178,8 @@ private void cleanupRepo(ProjectId projectId, String repositoryName, ActionListe
178178
@Override
179179
public ClusterState execute(ClusterState currentState) {
180180
final ProjectMetadata projectMetadata = currentState.metadata().getProject(projectId);
181-
SnapshotsService.ensureRepositoryExists(repositoryName, projectMetadata);
182-
SnapshotsService.ensureNotReadOnly(projectMetadata, repositoryName);
181+
SnapshotsServiceUtils.ensureRepositoryExists(repositoryName, projectMetadata);
182+
SnapshotsServiceUtils.ensureNotReadOnly(projectMetadata, repositoryName);
183183
// Repository cleanup is intentionally cluster wide exclusive
184184
final RepositoryCleanupInProgress repositoryCleanupInProgress = RepositoryCleanupInProgress.get(currentState);
185185
if (repositoryCleanupInProgress.hasCleanupInProgress()) {

‎server/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/status/TransportSnapshotsStatusAction.java‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@
4646
import org.elasticsearch.snapshots.SnapshotShardFailure;
4747
import org.elasticsearch.snapshots.SnapshotShardsService;
4848
import org.elasticsearch.snapshots.SnapshotState;
49-
import org.elasticsearch.snapshots.SnapshotsService;
49+
import org.elasticsearch.snapshots.SnapshotsServiceUtils;
5050
import org.elasticsearch.tasks.CancellableTask;
5151
import org.elasticsearch.tasks.Task;
5252
import org.elasticsearch.threadpool.ThreadPool;
@@ -121,7 +121,7 @@ protected void masterOperation(
121121

122122
final SnapshotsInProgress snapshotsInProgress = SnapshotsInProgress.get(state);
123123
final ProjectId projectId = projectResolver.getProjectId();
124-
List<SnapshotsInProgress.Entry> currentSnapshots = SnapshotsService.currentSnapshots(
124+
List<SnapshotsInProgress.Entry> currentSnapshots = SnapshotsServiceUtils.currentSnapshots(
125125
snapshotsInProgress,
126126
projectId,
127127
request.repository(),

0 commit comments

Comments
 (0)