Skip to content

Fix DenseVectorFieldIndexTypeUpdateIT test #132176

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jul 31, 2025
Merged

Fix DenseVectorFieldIndexTypeUpdateIT test #132176

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
0cfb0b3
Fix DenseVectorFieldIndexTypeUpdateIT test
carlosdelest Jul 30, 2025
89e2a7c
[CI] Auto commit changes from spotless
Jul 30, 2025
3d96ea7
Merge branch 'main' into tests/fix-dense-vector-update-tests
carlosdelest Jul 30, 2025
4b86db8
Merge remote-tracking branch 'origin/main' into tests/fix-dense-vecto…
carlosdelest Jul 31, 2025
f794def
Remove more muted tests
carlosdelest Jul 31, 2025
a65331f
Merge remote-tracking branch 'carlosdelest/tests/fix-dense-vector-upd…
carlosdelest Jul 31, 2025
4dc8b4b
Fix bad merge
carlosdelest Jul 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 0 additions & 78 deletions muted-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -530,60 +530,6 @@ tests:
- class: org.elasticsearch.xpack.test.rest.XPackRestIT
method: test {p0=transform/transforms_crud/Test transform where source query is invalid}
issue: https://github.com/elastic/elasticsearch/issues/132111
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=bbq_flat updateType=bbq_hnsw}
issue: https://github.com/elastic/elasticsearch/issues/132112
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=hnsw updateType=int4_hnsw}
issue: https://github.com/elastic/elasticsearch/issues/132113
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=hnsw updateType=bbq_hnsw}
issue: https://github.com/elastic/elasticsearch/issues/132115
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=int8_flat updateType=int8_hnsw}
issue: https://github.com/elastic/elasticsearch/issues/132116
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=int4_flat updateType=int8_hnsw}
issue: https://github.com/elastic/elasticsearch/issues/132117
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=flat updateType=hnsw}
issue: https://github.com/elastic/elasticsearch/issues/132119
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=flat updateType=bbq_flat}
issue: https://github.com/elastic/elasticsearch/issues/132120
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=int8_flat updateType=bbq_disk}
issue: https://github.com/elastic/elasticsearch/issues/132122
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=int8_flat updateType=int4_hnsw}
issue: https://github.com/elastic/elasticsearch/issues/132123
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=int8_flat updateType=hnsw}
issue: https://github.com/elastic/elasticsearch/issues/132124
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=int8_flat updateType=int4_flat}
issue: https://github.com/elastic/elasticsearch/issues/132125
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=flat updateType=int8_hnsw}
issue: https://github.com/elastic/elasticsearch/issues/132126
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=bbq_disk updateType=bbq_disk}
issue: https://github.com/elastic/elasticsearch/issues/132127
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=flat updateType=int8_flat}
issue: https://github.com/elastic/elasticsearch/issues/132129
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: "testDenseVectorMappingUpdate {initialType=flat updateType=bbq_disk #2}"
issue: https://github.com/elastic/elasticsearch/issues/132130
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=int4_flat updateType=bbq_flat}
issue: https://github.com/elastic/elasticsearch/issues/132132
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=int8_hnsw updateType=int4_hnsw}
issue: https://github.com/elastic/elasticsearch/issues/132133
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=bbq_flat updateType=bbq_disk}
issue: https://github.com/elastic/elasticsearch/issues/132134
- class: org.elasticsearch.upgrades.RunningSnapshotIT
method: testRunningSnapshotCompleteAfterUpgrade {upgradedNodes=1}
issue: https://github.com/elastic/elasticsearch/issues/132135
Expand All @@ -593,33 +539,9 @@ tests:
- class: org.elasticsearch.upgrades.RunningSnapshotIT
method: testRunningSnapshotCompleteAfterUpgrade {upgradedNodes=3}
issue: https://github.com/elastic/elasticsearch/issues/132137
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=int4_flat updateType=int4_hnsw}
issue: https://github.com/elastic/elasticsearch/issues/132140
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=int8_flat updateType=bbq_hnsw}
issue: https://github.com/elastic/elasticsearch/issues/132141
- class: org.elasticsearch.index.engine.MergeWithLowDiskSpaceIT
method: testRelocationWhileForceMerging
issue: https://github.com/elastic/elasticsearch/issues/131789
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=flat updateType=int4_hnsw}
issue: https://github.com/elastic/elasticsearch/issues/132149
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=int4_flat updateType=hnsw}
issue: https://github.com/elastic/elasticsearch/issues/132150
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=int8_flat updateType=bbq_flat}
issue: https://github.com/elastic/elasticsearch/issues/132151
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: "testDenseVectorMappingUpdate {initialType=bbq_hnsw updateType=bbq_disk #2}"
issue: https://github.com/elastic/elasticsearch/issues/132152
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=hnsw updateType=int8_hnsw}
issue: https://github.com/elastic/elasticsearch/issues/132164
- class: org.elasticsearch.index.mapper.vectors.DenseVectorFieldIndexTypeUpdateIT
method: testDenseVectorMappingUpdate {initialType=int4_hnsw updateType=bbq_disk}
issue: https://github.com/elastic/elasticsearch/issues/132165

# Examples:
#
Expand Down
56 changes: 30 additions & 26 deletions ...erTest/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldIndexTypeUpdateIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,14 @@
import com.carrotsearch.randomizedtesting.annotations.Name;
import com.carrotsearch.randomizedtesting.annotations.ParametersFactory;

import org.elasticsearch.action.DocWriteResponse;
import org.elasticsearch.action.admin.indices.mapping.get.GetFieldMappingsResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.cluster.metadata.IndexMetadata;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.rest.RestStatus;
import org.elasticsearch.search.vectors.KnnVectorQueryBuilder;
import org.elasticsearch.search.vectors.VectorData;
import org.elasticsearch.test.ESIntegTestCase;
import org.elasticsearch.xcontent.XContentBuilder;
import org.elasticsearch.xcontent.XContentFactory;
Expand All @@ -25,6 +29,7 @@
import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ExecutionException;

import static org.elasticsearch.index.IndexSettings.INDEX_MAPPER_SOURCE_MODE_SETTING;
import static org.elasticsearch.index.mapper.SourceFieldMapper.Mode.SYNTHETIC;
Expand Down Expand Up @@ -75,7 +80,7 @@ public static Collection<Object[]> params() {
@SuppressWarnings("unchecked")
public void testDenseVectorMappingUpdate() throws Exception {
dimensions = randomIntBetween(1, 10) * 64;
var client = client().admin().indices();
var indicesClient = client().admin().indices();

Settings.Builder settingsBuilder = Settings.builder()
.put(IndexMetadata.SETTING_NUMBER_OF_REPLICAS, 0)
Expand All @@ -85,7 +90,7 @@ public void testDenseVectorMappingUpdate() throws Exception {
}

// Create index with initial mapping
var createRequest = client.prepareCreate(INDEX_NAME)
var createRequest = indicesClient.prepareCreate(INDEX_NAME)
.setSettings(Settings.builder().put("index.number_of_shards", randomIntBetween(1, 5)))
.setMapping(updateMapping(dimensions, initialType))
.setSettings(settingsBuilder.build());
Expand All @@ -97,16 +102,16 @@ public void testDenseVectorMappingUpdate() throws Exception {
indexDoc(i);
}

client.prepareFlush(INDEX_NAME).get();
client.prepareRefresh(INDEX_NAME).get();
indicesClient.prepareFlush(INDEX_NAME).get();
indicesClient.prepareRefresh(INDEX_NAME).get();

// Update mapping to new type
var putMappingRequest = client.preparePutMapping(INDEX_NAME).setSource(updateMapping(dimensions, updateType)).request();
assertAcked(client.putMapping(putMappingRequest));
var putMappingRequest = indicesClient.preparePutMapping(INDEX_NAME).setSource(updateMapping(dimensions, updateType)).request();
assertAcked(indicesClient.putMapping(putMappingRequest));

// Validate mapping
GetFieldMappingsResponse fieldMapping = client.getFieldMappings(
client.prepareGetFieldMappings(INDEX_NAME).setFields(VECTOR_FIELD).request()
GetFieldMappingsResponse fieldMapping = indicesClient.getFieldMappings(
indicesClient.prepareGetFieldMappings(INDEX_NAME).setFields(VECTOR_FIELD).request()
).get();
var fieldMappingMetadata = fieldMapping.fieldMappings(INDEX_NAME, VECTOR_FIELD);
var fieldMap = (Map<String, Object>) fieldMappingMetadata.sourceAsMap().get(VECTOR_FIELD);
Expand All @@ -119,28 +124,26 @@ public void testDenseVectorMappingUpdate() throws Exception {
indexDoc(i);
}

client.prepareFlush(INDEX_NAME).get();
client.prepareRefresh(INDEX_NAME).get();
indicesClient.prepareFlush(INDEX_NAME).get();
indicesClient.prepareRefresh(INDEX_NAME).get();

// Search to ensure all documents are present
int expectedDocs = docsBefore + docsAfter;
assertNoFailuresAndResponse(client().prepareSearch(INDEX_NAME).setSize(expectedDocs + 10), response -> {

// Count query
assertNoFailuresAndResponse(client().prepareSearch(INDEX_NAME).setSize(0).setTrackTotalHits(true).setSize(expectedDocs), response -> {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a count query as well as a knn query to ensure results are consistent

assertHitCount(response, expectedDocs);
});
}

private XContentBuilder initialMapping(int dimensions, String type) throws IOException {
XContentBuilder builder = XContentFactory.jsonBuilder();
builder.startObject();
{
builder.startObject("mappings");
{
createFieldMapping(dimensions, type, builder);
}
builder.endObject();
// KNN query
float[] queryVector = new float[dimensions];
for (int i = 0; i < queryVector.length; i++) {
queryVector[i] = randomFloatBetween(-1, 1, true);
}
builder.endObject();
return builder;
KnnVectorQueryBuilder queryBuilder = new KnnVectorQueryBuilder(VECTOR_FIELD, queryVector, null, null, null, null);
assertNoFailuresAndResponse(client().prepareSearch(INDEX_NAME).setQuery(queryBuilder).setTrackTotalHits(true).setSize(expectedDocs), response -> {
assertHitCount(response, expectedDocs);
});
}

private XContentBuilder updateMapping(int dimensions, String type) throws IOException {
Expand Down Expand Up @@ -168,9 +171,10 @@ private static void createFieldMapping(int dimensions, String type, XContentBuil
builder.endObject();
}

private void indexDoc(int id) throws IOException {
private void indexDoc(int id) throws ExecutionException, InterruptedException {
Float[] vector = randomArray(dimensions, dimensions, Float[]::new, () -> randomFloatBetween(-1, 1, true));
IndexRequest req = prepareIndex(INDEX_NAME).setSource(VECTOR_FIELD, vector).setId(Integer.toString(id)).request();
client().index(req);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤦

IndexRequest indexRequest = prepareIndex(INDEX_NAME).setSource(VECTOR_FIELD, vector).setId(Integer.toString(id)).request();
DocWriteResponse indexResponse = client().index(indexRequest).get();
assertEquals(RestStatus.CREATED, indexResponse.status());
}
}