Skip to content

Enable Semantic Search CCS When ccs_minimize_roundtrips=true #135309

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 57 commits into from
Sep 25, 2025
Merged

Enable Semantic Search CCS When ccs_minimize_roundtrips=true #135309

Show file tree
Hide file tree
Changes from 35 commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
8a7069f
Added transport version
Mikep86 Sep 19, 2025
f2e8d10
Added skeleton for semantic cross-cluster search integration tests
Mikep86 Sep 19, 2025
d444199
Updated semantic query to support CCS
Mikep86 Sep 19, 2025
3133d6f
Added semantic query CCS integration test
Mikep86 Sep 19, 2025
679f3c0
Added semantic query CCS serialization test
Mikep86 Sep 19, 2025
d02419c
Deleted obsolete YAML test
Mikep86 Sep 19, 2025
f9d0d51
Updated legacy semantic query rewrite interceptors to detect CCS requ…
Mikep86 Sep 19, 2025
fc7fadc
Added semantic query CCS test when ccs_minimize_roundtrips=false
Mikep86 Sep 19, 2025
c439bd2
Fix random test failure
Mikep86 Sep 19, 2025
7db5ff8
Adjust error message format
Mikep86 Sep 19, 2025
455055a
Updated InterceptedInferenceQueryBuilder to add CCS support
Mikep86 Sep 22, 2025
990f69a
Added match query CCS test
Mikep86 Sep 22, 2025
3fc3e5b
Fix build error
Mikep86 Sep 22, 2025
2ce408a
Added knn query CCS test
Mikep86 Sep 22, 2025
ec40327
Use fully qualified inference ID to disambiguate inference ID overrides
Mikep86 Sep 22, 2025
fe71bc7
CCS test refactoring
Mikep86 Sep 22, 2025
e7269e6
Extended match query CCS test
Mikep86 Sep 22, 2025
deeefcb
Added sparse vector query CCS test
Mikep86 Sep 22, 2025
1d38967
Added test cases using query vectors
Mikep86 Sep 22, 2025
3c1acdf
Added match query test with ccs_minimize_roundtrips=false
Mikep86 Sep 22, 2025
9fd9c66
Added knn query test with ccs_minimize_roundtrips=false
Mikep86 Sep 23, 2025
e929355
Added sparse vector query test with ccs_minimize_roundtrips=false
Mikep86 Sep 23, 2025
3815f26
Added CCS serialization test for intercepted queries
Mikep86 Sep 23, 2025
c82d083
Fixed test failures
Mikep86 Sep 23, 2025
588eebd
Fixed test failures
Mikep86 Sep 23, 2025
ba54ae8
Update docs/changelog/135309.yaml
Mikep86 Sep 23, 2025
5b91665
[CI] Update transport version definitions
Sep 23, 2025
023acd7
Merge branch 'main' into semantic-search_enable-ccs
Mikep86 Sep 23, 2025
27913a5
Fixed SEMANTIC_SEARCH_CCS_SUPPORT transport version
Mikep86 Sep 23, 2025
f67267c
Merge branch 'semantic-search_enable-ccs' of github.com:Mikep86/elast…
Mikep86 Sep 23, 2025
0c1bf91
Updated changelog
Mikep86 Sep 23, 2025
66c0c09
Spotless
Mikep86 Sep 23, 2025
310208a
[CI] Update transport version definitions
Sep 23, 2025
9000d35
Refactored CCS integration tests into separate classes for each query…
Mikep86 Sep 24, 2025
88fafad
[CI] Update transport version definitions
Sep 24, 2025
7f071f0
Merge branch 'main' into semantic-search_enable-ccs
Mikep86 Sep 25, 2025
60f9be4
Fix transport version
Mikep86 Sep 25, 2025
fdf1ccf
Remove IVF_FORMAT feature flag references
Mikep86 Sep 25, 2025
b4721fd
Knn CCS test refactoring
Mikep86 Sep 25, 2025
00fe101
Adjusted and documented ccsRequest usage in SemanticQueryBuilder
Mikep86 Sep 25, 2025
cf70b70
Fix typo
Mikep86 Sep 25, 2025
9d5c68a
Refactored intercepted query CCS serialization tests
Mikep86 Sep 25, 2025
bab63a5
Refactored semantic query CCS serialization tests
Mikep86 Sep 25, 2025
d9be362
Updated intercepted queries to add a constructor for setting only the…
Mikep86 Sep 25, 2025
441eba4
Simplify intercepted knn query getInferenceIdOverride implementation
Mikep86 Sep 25, 2025
4074052
Updated error message
Mikep86 Sep 25, 2025
2e49792
Added convenience constructor for semantic query
Mikep86 Sep 25, 2025
e0a7187
Adjusted error message
Mikep86 Sep 25, 2025
d4d8f24
Merge branch 'main' into semantic-search_enable-ccs
Mikep86 Sep 25, 2025
da5ebb6
Update transport version
Mikep86 Sep 25, 2025
b6e6e6c
Match query CCS test refactoring
Mikep86 Sep 25, 2025
c124e02
Integrated index boosts into CCS integration tests
Mikep86 Sep 25, 2025
825bc9a
Updated match query CCS tests to use a common cluster configuration
Mikep86 Sep 25, 2025
3beb2d8
Semantic query CCS test refactoring
Mikep86 Sep 25, 2025
7333107
Sparse vector query CCS test refactoring
Mikep86 Sep 25, 2025
e082519
Merge branch 'main' into semantic-search_enable-ccs
Mikep86 Sep 25, 2025
0794501
Update transport version
Mikep86 Sep 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/changelog/135309.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
pr: 135309
summary: Enable semantic search CCS when ccs_minimize_roundtrips=true
area: Vector Search
type: enhancement
issues: []
37 changes: 0 additions & 37 deletions ...cluster-search/src/test/resources/rest-api-spec/test/multi_cluster/110_semantic_query.yml
View file
Open in desktop
Copy link
Contributor Author

@Mikep86 Mikep86 Sep 23, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, we can't write effective YAML tests for semantic search CCS because we cannot deploy the test inference service plugin (which allows us to execute semantic queries using mock inference services to generate query vectors) to the BwC distributions that also execute these tests. It shouldn't be a problem though because we get all the test coverage we need through a combination of:

  • The CCS integration tests added (i.e. SemanticCrossClusterSearchIT)
  • The BwC serialization tests added, which simulate sending queries to remote clusters that use various transport versions

Copy link
Member

@kderusso kderusso Sep 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Friendly reminder to update rest compatibility so this test isn't executed.

Copy link
Contributor Author

@Mikep86 Mikep86 Sep 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't rest compatibility already be broken in CI if that were going to be an issue for this PR?

This file was deleted.

1 change: 1 addition & 0 deletions server/src/main/resources/transport/definitions/referable/semantic_search_ccs_support.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
9171000
2 changes: 1 addition & 1 deletion server/src/main/resources/transport/upper_bounds/8.18.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
initial_elasticsearch_8_18_8,8840010
transform_check_for_dangling_tasks,8840011
Copy link
Contributor

@ioanatia ioanatia Sep 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

weird that CI pushed this changes - probably merging main here will make these disappear

Copy link
Contributor Author

@Mikep86 Mikep86 Sep 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still figuring out the new "simplified" transport versions. There are some rough edges :)

2 changes: 1 addition & 1 deletion server/src/main/resources/transport/upper_bounds/8.19.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
initial_elasticsearch_8_19_5,8841069
transform_check_for_dangling_tasks,8841070
2 changes: 1 addition & 1 deletion server/src/main/resources/transport/upper_bounds/9.0.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
initial_elasticsearch_9_0_8,9000017
transform_check_for_dangling_tasks,9000018
2 changes: 1 addition & 1 deletion server/src/main/resources/transport/upper_bounds/9.1.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
initial_elasticsearch_9_1_5,9112008
transform_check_for_dangling_tasks,9112009
2 changes: 1 addition & 1 deletion server/src/main/resources/transport/upper_bounds/9.2.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
inference_api_openai_embeddings_headers,9169000
semantic_search_ccs_support,9171000
12 changes: 12 additions & 0 deletions ...lugin/core/src/test/java/org/elasticsearch/xpack/core/LocalStateCompositeXPackPlugin.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,8 @@
import org.elasticsearch.plugins.ShutdownAwarePlugin;
import org.elasticsearch.plugins.SystemIndexPlugin;
import org.elasticsearch.plugins.interceptor.RestServerActionPlugin;
import org.elasticsearch.plugins.internal.InternalSearchPlugin;
import org.elasticsearch.plugins.internal.rewriter.QueryRewriteInterceptor;
import org.elasticsearch.repositories.RepositoriesMetrics;
import org.elasticsearch.repositories.Repository;
import org.elasticsearch.repositories.SnapshotMetrics;
Expand Down Expand Up @@ -135,6 +137,7 @@ public class LocalStateCompositeXPackPlugin extends XPackPlugin
IndexStorePlugin,
SystemIndexPlugin,
SearchPlugin,
InternalSearchPlugin,
ShutdownAwarePlugin,
RestServerActionPlugin {

Expand Down Expand Up @@ -291,6 +294,15 @@ public List<QuerySpec<?>> getQueries() {
return querySpecs;
}

@Override
public List<QueryRewriteInterceptor> getQueryRewriteInterceptors() {
List<QueryRewriteInterceptor> interceptors = new ArrayList<>();
filterPlugins(InternalSearchPlugin.class).stream()
.flatMap(p -> p.getQueryRewriteInterceptors().stream())
.forEach(interceptors::add);
return interceptors;
}

@Override
public List<NamedXContentRegistry.Entry> getNamedXContent() {
List<NamedXContentRegistry.Entry> entries = new ArrayList<>(super.getNamedXContent());
Expand Down
1 change: 1 addition & 0 deletions x-pack/plugin/inference/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ dependencies {
testImplementation(project(':x-pack:plugin:inference:qa:test-service-plugin'))
testImplementation project(':modules:reindex')
testImplementation project(':modules:mapper-extras')
testImplementation project(':x-pack:plugin:ml')
clusterPlugins project(':x-pack:plugin:inference:qa:test-service-plugin')

api "com.ibm.icu:icu4j:${versions.icu4j}"
Expand Down
Loading