Skip to content

Commit 928c85d

Browse files
julian-elasticjulian-elastic
committed
Merge branch 'main' into RLikeList_v4
2 parents c0c6bad + 952b62e commit 928c85d

File tree

32 files changed

+373
-202
lines changed

32 files changed

+373
-202
lines changed

‎docs/changelog/130544.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
pr: 130544
2+
summary: Sync Inference with Trained Model stats
3+
area: Machine Learning
4+
type: bug
5+
issues:
6+
- 130339

‎docs/internal/GeneralArchitectureGuide.md‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -182,12 +182,12 @@ capabilities.
182182

183183
## Serializations
184184

185-
## Settings
185+
# Settings
186186

187187
Elasticsearch supports [cluster-level settings][] and [index-level settings][], configurable via [node-level file settings][]
188188
(e.g. `elasticsearch.yml` file), command line arguments and REST APIs.
189189

190-
### Declaring a Setting
190+
## Declaring a Setting
191191

192192
[cluster-level settings]: https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-update-settings.html
193193
[index-level settings]: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html
@@ -222,7 +222,7 @@ settings.
222222
[SettingsModule constructor]: https://github.com/elastic/elasticsearch/blob/v8.13.2/server/src/main/java/org/elasticsearch/node/NodeConstruction.java#L491-L495
223223
[getSettings()]: https://github.com/elastic/elasticsearch/blob/v8.13.2/server/src/main/java/org/elasticsearch/plugins/Plugin.java#L203-L208
224224

225-
### Dynamically updating a Setting
225+
## Dynamically updating a Setting
226226

227227
Externally, [TransportClusterUpdateSettingsAction][] and [TransportUpdateSettingsAction][] (and the corresponding REST endpoints)
228228
allow users to dynamically change cluster and index settings, respectively. Internally, `AbstractScopedSettings` (parent class
@@ -244,9 +244,9 @@ state must ever be reloaded from persisted state.
244244
[Metadata]: https://github.com/elastic/elasticsearch/blob/v8.13.2/server/src/main/java/org/elasticsearch/cluster/metadata/Metadata.java#L212-L213
245245
[applied here]: https://github.com/elastic/elasticsearch/blob/v8.13.2/server/src/main/java/org/elasticsearch/cluster/metadata/Metadata.java#L2437
246246

247-
## Deprecations
247+
# Deprecations
248248

249-
## Backwards Compatibility
249+
# Backwards Compatibility
250250

251251
major releases are mostly about breaking compatibility and dropping deprecated functionality.
252252

@@ -292,18 +292,18 @@ See the [public upgrade docs][] for the upgrade process.
292292

293293
[public upgrade docs]: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html
294294

295-
## Plugins
295+
# Plugins
296296

297297
(what warrants a plugin?)
298298

299299
(what plugins do we have?)
300300

301-
## Testing
301+
# Testing
302302

303303
(Overview of our testing frameworks. Discuss base test classes.)
304304

305-
### Unit Testing
305+
## Unit Testing
306306

307-
### REST Testing
307+
## REST Testing
308308

309-
### Integration Testing
309+
## Integration Testing

‎muted-tests.yml‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -502,9 +502,6 @@ tests:
502502
- class: org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT
503503
method: test {p0=mtermvectors/10_basic/Tests catching other exceptions per item}
504504
issue: https://github.com/elastic/elasticsearch/issues/122414
505-
- class: org.elasticsearch.xpack.slm.SLMFileSettingsIT
506-
method: testSettingsApplied
507-
issue: https://github.com/elastic/elasticsearch/issues/130853
508505
- class: org.elasticsearch.cluster.ClusterStateSerializationTests
509506
method: testSerializationPreMultiProject
510507
issue: https://github.com/elastic/elasticsearch/issues/130872
@@ -520,9 +517,6 @@ tests:
520517
- class: org.elasticsearch.xpack.esql.action.EsqlActionBreakerIT
521518
method: testRowStatsProjectGroupByInt
522519
issue: https://github.com/elastic/elasticsearch/issues/131024
523-
- class: org.elasticsearch.common.util.concurrent.TaskExecutionTimeTrackingEsThreadPoolExecutorTests
524-
method: testMaxQueueLatency
525-
issue: https://github.com/elastic/elasticsearch/issues/131093
526520
- class: org.elasticsearch.xpack.esql.action.CrossClusterAsyncQueryStopIT
527521
method: testStopQueryLocal
528522
issue: https://github.com/elastic/elasticsearch/issues/121672
@@ -541,6 +535,12 @@ tests:
541535
- class: org.elasticsearch.test.rest.yaml.RcsCcsCommonYamlTestSuiteIT
542536
method: test {p0=field_caps/40_time_series/Get simple time series field caps}
543537
issue: https://github.com/elastic/elasticsearch/issues/131225
538+
- class: org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitorIT
539+
method: testFloodStageExceeded
540+
issue: https://github.com/elastic/elasticsearch/issues/131146
541+
- class: org.elasticsearch.packaging.test.DockerTests
542+
method: test090SecurityCliPackaging
543+
issue: https://github.com/elastic/elasticsearch/issues/131107
544544

545545
# Examples:
546546
#

‎rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.vectors/135_knn_query_nested_search_ivf.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
setup:
22
- requires:
3-
cluster_features: "mapper.ivf_nested_support"
4-
reason: 'ivf nested support required'
3+
cluster_features: "mapper.bbq_disk_support"
4+
reason: 'bbq disk support required'
55
- do:
66
indices.create:
77
index: test
@@ -24,7 +24,7 @@ setup:
2424
index: true
2525
similarity: l2_norm
2626
index_options:
27-
type: bbq_ivf
27+
type: bbq_disk
2828

2929
aliases:
3030
my_alias:

‎rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.vectors/46_knn_search_bbq_ivf.yml‎

Lines changed: 24 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
setup:
22
- requires:
3-
cluster_features: ["mapper.ivf_format_cluster_feature"]
4-
reason: Needs mapper.ivf_format_cluster_feature feature
3+
cluster_features: ["mapper.bbq_disk_support"]
4+
reason: Needs mapper.bbq_disk_support feature
55
- skip:
66
features: "headers"
77
- do:
88
indices.create:
9-
index: bbq_ivf
9+
index: bbq_disk
1010
body:
1111
settings:
1212
index:
@@ -19,11 +19,11 @@ setup:
1919
index: true
2020
similarity: max_inner_product
2121
index_options:
22-
type: bbq_ivf
22+
type: bbq_disk
2323

2424
- do:
2525
index:
26-
index: bbq_ivf
26+
index: bbq_disk
2727
id: "1"
2828
body:
2929
vector: [0.077, 0.32 , -0.205, 0.63 , 0.032, 0.201, 0.167, -0.313,
@@ -37,11 +37,11 @@ setup:
3737
# Flush in order to provoke a merge later
3838
- do:
3939
indices.flush:
40-
index: bbq_ivf
40+
index: bbq_disk
4141

4242
- do:
4343
index:
44-
index: bbq_ivf
44+
index: bbq_disk
4545
id: "2"
4646
body:
4747
vector: [0.196, 0.514, 0.039, 0.555, -0.042, 0.242, 0.463, -0.348,
@@ -55,11 +55,11 @@ setup:
5555
# Flush in order to provoke a merge later
5656
- do:
5757
indices.flush:
58-
index: bbq_ivf
58+
index: bbq_disk
5959

6060
- do:
6161
index:
62-
index: bbq_ivf
62+
index: bbq_disk
6363
id: "3"
6464
body:
6565
name: rabbit.jpg
@@ -74,11 +74,11 @@ setup:
7474
# Flush in order to provoke a merge later
7575
- do:
7676
indices.flush:
77-
index: bbq_ivf
77+
index: bbq_disk
7878

7979
- do:
8080
indices.forcemerge:
81-
index: bbq_ivf
81+
index: bbq_disk
8282
max_num_segments: 1
8383

8484
- do:
@@ -87,7 +87,7 @@ setup:
8787
"Test knn search":
8888
- do:
8989
search:
90-
index: bbq_ivf
90+
index: bbq_disk
9191
body:
9292
knn:
9393
field: vector
@@ -116,7 +116,7 @@ setup:
116116
Content-Type: application/json
117117
search:
118118
rest_total_hits_as_int: true
119-
index: bbq_ivf
119+
index: bbq_disk
120120
body:
121121
knn:
122122
field: vector
@@ -182,7 +182,7 @@ setup:
182182
element_type: byte
183183
index: true
184184
index_options:
185-
type: bbq_ivf
185+
type: bbq_disk
186186

187187
- do:
188188
catch: bad_request
@@ -196,7 +196,7 @@ setup:
196196
dims: 64
197197
index: false
198198
index_options:
199-
type: bbq_ivf
199+
type: bbq_disk
200200
---
201201
"Test index configured rescore vector":
202202
- skip:
@@ -216,7 +216,7 @@ setup:
216216
index: true
217217
similarity: max_inner_product
218218
index_options:
219-
type: bbq_ivf
219+
type: bbq_disk
220220
rescore_vector:
221221
oversample: 1.5
222222

@@ -298,7 +298,7 @@ setup:
298298
vector:
299299
type: dense_vector
300300
index_options:
301-
type: bbq_ivf
301+
type: bbq_disk
302302
rescore_vector:
303303
oversample: 0
304304

@@ -314,7 +314,7 @@ setup:
314314
vector:
315315
type: dense_vector
316316
index_options:
317-
type: bbq_ivf
317+
type: bbq_disk
318318
rescore_vector:
319319
oversample: 1
320320

@@ -326,7 +326,7 @@ setup:
326326
vector:
327327
type: dense_vector
328328
index_options:
329-
type: bbq_ivf
329+
type: bbq_disk
330330
rescore_vector:
331331
oversample: 0
332332

@@ -354,7 +354,7 @@ setup:
354354
index: true
355355
similarity: max_inner_product
356356
index_options:
357-
type: bbq_ivf
357+
type: bbq_disk
358358
rescore_vector:
359359
oversample: 0
360360

@@ -430,7 +430,7 @@ setup:
430430
index: true
431431
similarity: max_inner_product
432432
index_options:
433-
type: bbq_ivf
433+
type: bbq_disk
434434
rescore_vector:
435435
oversample: 2
436436

@@ -470,7 +470,7 @@ setup:
470470
index: true
471471
similarity: max_inner_product
472472
index_options:
473-
type: bbq_ivf
473+
type: bbq_disk
474474
rescore_vector:
475475
oversample: 0
476476

@@ -509,6 +509,6 @@ setup:
509509
"default oversample value":
510510
- do:
511511
indices.get_mapping:
512-
index: bbq_ivf
512+
index: bbq_disk
513513

514-
- match: { bbq_ivf.mappings.properties.vector.index_options.rescore_vector.oversample: 3.0 }
514+
- match: { bbq_disk.mappings.properties.vector.index_options.rescore_vector.oversample: 3.0 }

‎server/src/main/java/org/elasticsearch/cluster/metadata/Metadata.java‎

Lines changed: 5 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1230,6 +1230,7 @@ public static Metadata readFrom(StreamInput in) throws IOException {
12301230
}
12311231

12321232
private static void readBwcCustoms(StreamInput in, Builder builder) throws IOException {
1233+
final ProjectMetadata.Builder projectBuilder = builder.getProject(ProjectId.DEFAULT);
12331234
final Set<String> clusterScopedNames = in.namedWriteableRegistry().getReaders(ClusterCustom.class).keySet();
12341235
final Set<String> projectScopedNames = in.namedWriteableRegistry().getReaders(ProjectCustom.class).keySet();
12351236
final int count = in.readVInt();
@@ -1245,9 +1246,9 @@ private static void readBwcCustoms(StreamInput in, Builder builder) throws IOExc
12451246
if (custom instanceof PersistentTasksCustomMetadata persistentTasksCustomMetadata) {
12461247
final var tuple = persistentTasksCustomMetadata.split();
12471248
builder.putCustom(tuple.v1().getWriteableName(), tuple.v1());
1248-
builder.putProjectCustom(tuple.v2().getWriteableName(), tuple.v2());
1249+
projectBuilder.putCustom(tuple.v2().getWriteableName(), tuple.v2());
12491250
} else {
1250-
builder.putProjectCustom(custom.getWriteableName(), custom);
1251+
projectBuilder.putCustom(custom.getWriteableName(), custom);
12511252
}
12521253
} else {
12531254
throw new IllegalArgumentException("Unknown custom name [" + name + "]");
@@ -1552,7 +1553,8 @@ public Builder putCustom(String type, ClusterCustom custom) {
15521553

15531554
@Deprecated(forRemoval = true)
15541555
public Builder putCustom(String type, ProjectCustom custom) {
1555-
return putProjectCustom(type, custom);
1556+
getSingleProject().putCustom(type, Objects.requireNonNull(custom, type));
1557+
return this;
15561558
}
15571559

15581560
public ClusterCustom getCustom(String type) {
@@ -1575,12 +1577,6 @@ public Builder customs(Map<String, ClusterCustom> clusterCustoms) {
15751577
return this;
15761578
}
15771579

1578-
@Deprecated(forRemoval = true)
1579-
public Builder putProjectCustom(String type, ProjectCustom custom) {
1580-
getSingleProject().putCustom(type, Objects.requireNonNull(custom, type));
1581-
return this;
1582-
}
1583-
15841580
@Deprecated(forRemoval = true)
15851581
public Builder projectCustoms(Map<String, ProjectCustom> projectCustoms) {
15861582
projectCustoms.forEach((key, value) -> Objects.requireNonNull(value, key));

‎server/src/main/java/org/elasticsearch/index/mapper/MapperFeatures.java‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@ public class MapperFeatures implements FeatureSpecification {
4444
static final NodeFeature NPE_ON_DIMS_UPDATE_FIX = new NodeFeature("mapper.npe_on_dims_update_fix");
4545
static final NodeFeature IVF_FORMAT_CLUSTER_FEATURE = new NodeFeature("mapper.ivf_format_cluster_feature");
4646
static final NodeFeature IVF_NESTED_SUPPORT = new NodeFeature("mapper.ivf_nested_support");
47+
static final NodeFeature BBQ_DISK_SUPPORT = new NodeFeature("mapper.bbq_disk_support");
4748
static final NodeFeature SEARCH_LOAD_PER_SHARD = new NodeFeature("mapper.search_load_per_shard");
4849
static final NodeFeature PATTERNED_TEXT = new NodeFeature("mapper.patterned_text");
4950

@@ -76,6 +77,7 @@ public Set<NodeFeature> getTestFeatures() {
7677
USE_DEFAULT_OVERSAMPLE_VALUE_FOR_BBQ,
7778
IVF_FORMAT_CLUSTER_FEATURE,
7879
IVF_NESTED_SUPPORT,
80+
BBQ_DISK_SUPPORT,
7981
SEARCH_LOAD_PER_SHARD,
8082
SPARSE_VECTOR_INDEX_OPTIONS_FEATURE,
8183
PATTERNED_TEXT

‎server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1675,7 +1675,7 @@ public boolean supportsDimension(int dims) {
16751675
return dims >= BBQ_MIN_DIMS;
16761676
}
16771677
},
1678-
BBQ_IVF("bbq_ivf", true) {
1678+
BBQ_DISK("bbq_disk", true) {
16791679
@Override
16801680
public DenseVectorIndexOptions parseIndexOptions(String fieldName, Map<String, ?> indexOptionsMap, IndexVersion indexVersion) {
16811681
Object clusterSizeNode = indexOptionsMap.remove("cluster_size");
@@ -2295,7 +2295,7 @@ static class BBQIVFIndexOptions extends QuantizedIndexOptions {
22952295
final int defaultNProbe;
22962296

22972297
BBQIVFIndexOptions(int clusterSize, int defaultNProbe, RescoreVector rescoreVector) {
2298-
super(VectorIndexType.BBQ_IVF, rescoreVector);
2298+
super(VectorIndexType.BBQ_DISK, rescoreVector);
22992299
this.clusterSize = clusterSize;
23002300
this.defaultNProbe = defaultNProbe;
23012301
}

‎server/src/test/java/org/elasticsearch/cluster/metadata/MetadataPersistentTasksTests.java‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -306,13 +306,16 @@ private Tuple<Metadata, Metadata> randomMetadataAndUpdate() {
306306
ClusterPersistentTasksCustomMetadata::new
307307
)
308308
)
309-
.putProjectCustom(
310-
PersistentTasksCustomMetadata.TYPE,
311-
mutatePersistentTasks(
312-
PersistentTasksCustomMetadata.get(before.getProject(Metadata.DEFAULT_PROJECT_ID)),
313-
MetadataPersistentTasksTests::oneProjectPersistentTask,
314-
PersistentTasksCustomMetadata::new
315-
)
309+
.put(
310+
ProjectMetadata.builder(before.getProject(Metadata.DEFAULT_PROJECT_ID))
311+
.putCustom(
312+
PersistentTasksCustomMetadata.TYPE,
313+
mutatePersistentTasks(
314+
PersistentTasksCustomMetadata.get(before.getProject(Metadata.DEFAULT_PROJECT_ID)),
315+
MetadataPersistentTasksTests::oneProjectPersistentTask,
316+
PersistentTasksCustomMetadata::new
317+
)
318+
)
316319
)
317320
.build();
318321
return new Tuple<>(before, after);

0 commit comments

Comments
 (0)