Skip to content
Closed
Show file tree
Hide file tree
Changes from 42 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
a0d0b77
add multi_match intercepter
Samiul-TheSoccerFan Aug 1, 2025
dabbb85
implement types
Samiul-TheSoccerFan Aug 1, 2025
21245c3
fixing warning header logic
Samiul-TheSoccerFan Aug 5, 2025
8a3ca94
fix warning header logic in semantic multi_match
Samiul-TheSoccerFan Aug 5, 2025
d5a315d
centralize validation logic for both mixed and single field
Samiul-TheSoccerFan Aug 5, 2025
a71aa73
added comments to better understanding and todo
Samiul-TheSoccerFan Aug 5, 2025
c17fed5
validate query types for single inference field
Samiul-TheSoccerFan Aug 5, 2025
d7a7640
apply field and query boost
Samiul-TheSoccerFan Aug 6, 2025
d8b97f3
fix duplicate boosting retrieve
Samiul-TheSoccerFan Aug 6, 2025
944053f
refactor boosting checks
Samiul-TheSoccerFan Aug 6, 2025
69f439f
Convert MultiFieldInferenceInfo to record
Samiul-TheSoccerFan Aug 6, 2025
776a65c
linting
Samiul-TheSoccerFan Aug 6, 2025
a785a19
refactoring multimatch intercepter
Samiul-TheSoccerFan Aug 8, 2025
1b7501f
refactoring multimatch intercepter
Samiul-TheSoccerFan Aug 8, 2025
00f14de
fix tiebreaker default issue
Samiul-TheSoccerFan Aug 8, 2025
6f618e4
small fix
Samiul-TheSoccerFan Aug 8, 2025
c740804
small refactor
Samiul-TheSoccerFan Aug 8, 2025
8b47269
fix multi index broken search
Samiul-TheSoccerFan Aug 8, 2025
7064442
refactor recalculating of non-inference fields
Samiul-TheSoccerFan Aug 8, 2025
76af4b0
wildcard resolution with field boost
Samiul-TheSoccerFan Aug 11, 2025
53a9a6f
refactor and simplify data structure
Samiul-TheSoccerFan Aug 11, 2025
de5aa6c
linting
Samiul-TheSoccerFan Aug 11, 2025
22a958d
adding yaml tests for multi_match
Samiul-TheSoccerFan Aug 11, 2025
6763057
adding assertion
Samiul-TheSoccerFan Aug 11, 2025
fd8fbb6
update comments
Samiul-TheSoccerFan Aug 11, 2025
3a99284
Merge branch 'main' into poc-multi-match-rewrite
elasticmachine Aug 11, 2025
949a288
validate query types for single inference field
Samiul-TheSoccerFan Aug 12, 2025
f1de7b4
remove score mismatch issue
Samiul-TheSoccerFan Aug 12, 2025
cd213bb
throw exception is type is unrocnizable
Samiul-TheSoccerFan Aug 12, 2025
eb30e4c
default field implementation
Samiul-TheSoccerFan Aug 12, 2025
851cdd2
adding transport layer
Samiul-TheSoccerFan Aug 13, 2025
2e8fea2
logic not to resolve by default
Samiul-TheSoccerFan Aug 13, 2025
b17f8d2
refactor extracting resolvewildcard field
Samiul-TheSoccerFan Aug 13, 2025
046ff0f
multiple boosting per field
Samiul-TheSoccerFan Aug 14, 2025
409e320
linting
Samiul-TheSoccerFan Aug 14, 2025
7ad9675
merge from main
Samiul-TheSoccerFan Aug 14, 2025
e964f2e
remove unnecessary headers
Samiul-TheSoccerFan Aug 14, 2025
80560be
remove modelResitry
Samiul-TheSoccerFan Aug 14, 2025
98ec933
clean up
Samiul-TheSoccerFan Aug 14, 2025
2595efb
fix yaml test failure
Samiul-TheSoccerFan Aug 14, 2025
5c519bc
fix retrievers unit tests
Samiul-TheSoccerFan Aug 14, 2025
41dfd89
Merge branch 'main' into poc-multi-match-rewrite
elasticmachine Aug 14, 2025
7c7fbf8
rename resolve function
Samiul-TheSoccerFan Aug 14, 2025
dfd27d2
combine logic to simplify best fields and most fields
Samiul-TheSoccerFan Aug 15, 2025
b42bc6b
update combine logic to handle user defined tiebreaker and minimum_sh…
Samiul-TheSoccerFan Aug 15, 2025
c1f284a
merge from main
Samiul-TheSoccerFan Aug 15, 2025
13cd7c0
remove duplicate boosting by creating a global boostig fields
Samiul-TheSoccerFan Aug 20, 2025
0accbe2
combine logic to simplify best fields and most fields
Samiul-TheSoccerFan Aug 21, 2025
8b45313
make decisions to resolve wildcard any fields for only multi_match
Samiul-TheSoccerFan Aug 21, 2025
658f171
refactor to push wildcard related work into multi_match
Samiul-TheSoccerFan Aug 21, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions server/src/main/java/org/elasticsearch/TransportVersions.java
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,7 @@ static TransportVersion def(int id) {
public static final TransportVersion INITIAL_ELASTICSEARCH_8_19_1 = def(8_841_0_65);
public static final TransportVersion INITIAL_ELASTICSEARCH_8_19_2 = def(8_841_0_66);
public static final TransportVersion INITIAL_ELASTICSEARCH_8_19_3 = def(8_841_0_67);
public static final TransportVersion MULTI_MATCH_SEMANTIC_TEXT_SUPPORT_8_19 = def(8_841_0_68);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we want to backport this to 8.19? We only want this to be in 9.2 right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AIU, it would be only 9.2 but I have not discussed that yet. I will remove this in the original PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't backport new features, only bug fixes. 8.19 was a special case because we were developing two sibling releases (8.19 & 9.1) at the same time.

public static final TransportVersion V_9_0_0 = def(9_000_0_09);
public static final TransportVersion INITIAL_ELASTICSEARCH_9_0_1 = def(9_000_0_10);
public static final TransportVersion INITIAL_ELASTICSEARCH_9_0_2 = def(9_000_0_11);
Expand Down Expand Up @@ -364,6 +365,7 @@ static TransportVersion def(int id) {
public static final TransportVersion EXTENDED_SNAPSHOT_STATS_IN_NODE_INFO = def(9_137_0_00);
public static final TransportVersion SIMULATE_INGEST_MAPPING_MERGE_TYPE = def(9_138_0_00);
public static final TransportVersion ESQL_LOOKUP_JOIN_ON_MANY_FIELDS = def(9_139_0_00);
public static final TransportVersion MULTI_MATCH_SEMANTIC_TEXT_SUPPORT = def(9_140_0_00);

/*
* STOP! READ THIS FIRST! No, really,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ public final class MultiMatchQueryBuilder extends AbstractQueryBuilder<MultiMatc
public static final int DEFAULT_MAX_EXPANSIONS = FuzzyQuery.defaultMaxExpansions;
public static final ZeroTermsQueryOption DEFAULT_ZERO_TERMS_QUERY = MatchQueryParser.DEFAULT_ZERO_TERMS_QUERY;
public static final boolean DEFAULT_FUZZY_TRANSPOSITIONS = FuzzyQuery.defaultTranspositions;
private boolean resolveInferenceFieldWildcards = true;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did we add this to the multi match builder, rather than keeping all logic in the inference plugin? That seems odd to me and a lot of extra stuff that we have to pass around? I may be missing a good reason to do this...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The resolveInferenceFieldWildcards parameter was added to MultiMatchQueryBuilder because both LinearRetrieverBuilder and RRFRetrieverBuilder directly create MultiMatchQueryBuilder instances and need to prevent double-querying of semantic_text fields when using their simplified syntax. The changes in MultiFieldsInnerRetrieverUtils.java, LinearRetrieverBuilderTests, and RRFRetrieverBuilderTests provide additional context for this approach. Please let me know if this does not answer your question and need more clarification. Also, I am open to suggestions if there is a better way to handle this check.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for explaining your thought process around adding this.

If you look at the core details of what you added, though, you're only adding a flag to the MultiMatchQueryBuilder and not actually doing anything with that flag in this class itself. To me, that's a really big signal that this flag should not exist in this class. Furthermore, the linear and RRF retrievers are already performing wildcard resolution on inference fields in MultiFieldsInnerRetrieverUtils#generateInnerRetrieversForIndex. Please remove this and try to come up with a solution that is contained to the interceptors.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed we need to adjust this. We need to separate the concepts of handling inference fields and resolving wildcards. AFAIU handling inference fields needs to be a multi-match query param so that the user can enable it (it should be false by default). In contrast, resolving wildcards is an immutable property of the query that is not configurable by the user.


public static final ParseField BOOST_FIELD = new ParseField("boost");
public static final ParseField SLOP_FIELD = new ParseField("slop");
Expand Down Expand Up @@ -228,6 +229,12 @@ public MultiMatchQueryBuilder(StreamInput in) throws IOException {
zeroTermsQuery = ZeroTermsQueryOption.readFromStream(in);
autoGenerateSynonymsPhraseQuery = in.readBoolean();
fuzzyTranspositions = in.readBoolean();
if (in.getTransportVersion().isPatchFrom(TransportVersions.MULTI_MATCH_SEMANTIC_TEXT_SUPPORT_8_19)
|| in.getTransportVersion().onOrAfter(TransportVersions.MULTI_MATCH_SEMANTIC_TEXT_SUPPORT)) {
resolveInferenceFieldWildcards = in.readBoolean();
} else {
resolveInferenceFieldWildcards = false;
}
}

@Override
Expand All @@ -251,6 +258,10 @@ protected void doWriteTo(StreamOutput out) throws IOException {
zeroTermsQuery.writeTo(out);
out.writeBoolean(autoGenerateSynonymsPhraseQuery);
out.writeBoolean(fuzzyTranspositions);
if (out.getTransportVersion().isPatchFrom(TransportVersions.MULTI_MATCH_SEMANTIC_TEXT_SUPPORT_8_19)
|| out.getTransportVersion().onOrAfter(TransportVersions.MULTI_MATCH_SEMANTIC_TEXT_SUPPORT)) {
out.writeBoolean(resolveInferenceFieldWildcards);
}
}

public Object value() {
Expand Down Expand Up @@ -513,6 +524,15 @@ public MultiMatchQueryBuilder fuzzyTranspositions(boolean fuzzyTranspositions) {
return this;
}

public boolean resolveInferenceFieldWildcards() {
return resolveInferenceFieldWildcards;
}

public MultiMatchQueryBuilder resolveInferenceFieldWildcards(boolean resolveInferenceFieldWildcards) {
this.resolveInferenceFieldWildcards = resolveInferenceFieldWildcards;
return this;
}

@Override
public void doXContent(XContentBuilder builder, Params params) throws IOException {
builder.startObject(NAME);
Expand Down Expand Up @@ -797,7 +817,8 @@ protected int doHashCode() {
lenient,
zeroTermsQuery,
autoGenerateSynonymsPhraseQuery,
fuzzyTranspositions
fuzzyTranspositions,
resolveInferenceFieldWildcards
);
}

Expand All @@ -818,7 +839,8 @@ protected boolean doEquals(MultiMatchQueryBuilder other) {
&& Objects.equals(lenient, other.lenient)
&& Objects.equals(zeroTermsQuery, other.zeroTermsQuery)
&& Objects.equals(autoGenerateSynonymsPhraseQuery, other.autoGenerateSynonymsPhraseQuery)
&& Objects.equals(fuzzyTranspositions, other.fuzzyTranspositions);
&& Objects.equals(fuzzyTranspositions, other.fuzzyTranspositions)
&& Objects.equals(resolveInferenceFieldWildcards, other.resolveInferenceFieldWildcards);
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
import static org.elasticsearch.xpack.inference.queries.SemanticKnnVectorQueryRewriteInterceptor.SEMANTIC_KNN_FILTER_FIX;
import static org.elasticsearch.xpack.inference.queries.SemanticKnnVectorQueryRewriteInterceptor.SEMANTIC_KNN_VECTOR_QUERY_REWRITE_INTERCEPTION_SUPPORTED;
import static org.elasticsearch.xpack.inference.queries.SemanticMatchQueryRewriteInterceptor.SEMANTIC_MATCH_QUERY_REWRITE_INTERCEPTION_SUPPORTED;
import static org.elasticsearch.xpack.inference.queries.SemanticMultiMatchQueryRewriteInterceptor.SEMANTIC_MULTI_MATCH_QUERY_REWRITE_INTERCEPTION_SUPPORTED;
import static org.elasticsearch.xpack.inference.queries.SemanticSparseVectorQueryRewriteInterceptor.SEMANTIC_SPARSE_VECTOR_QUERY_REWRITE_INTERCEPTION_SUPPORTED;
import static org.elasticsearch.xpack.inference.rank.textsimilarity.TextSimilarityRankRetrieverBuilder.RERANK_SNIPPETS;
import static org.elasticsearch.xpack.inference.rank.textsimilarity.TextSimilarityRankRetrieverBuilder.TEXT_SIMILARITY_RERANKER_SNIPPETS;
Expand Down Expand Up @@ -61,6 +62,7 @@ public Set<NodeFeature> getTestFeatures() {
SemanticTextFieldMapper.SEMANTIC_TEXT_SKIP_INFERENCE_FIELDS,
SEMANTIC_TEXT_HIGHLIGHTER,
SEMANTIC_MATCH_QUERY_REWRITE_INTERCEPTION_SUPPORTED,
SEMANTIC_MULTI_MATCH_QUERY_REWRITE_INTERCEPTION_SUPPORTED,
SEMANTIC_SPARSE_VECTOR_QUERY_REWRITE_INTERCEPTION_SUPPORTED,
SemanticInferenceMetadataFieldsMapper.EXPLICIT_NULL_FIXES,
SEMANTIC_KNN_VECTOR_QUERY_REWRITE_INTERCEPTION_SUPPORTED,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@
import org.elasticsearch.xpack.inference.mapper.SemanticTextFieldMapper;
import org.elasticsearch.xpack.inference.queries.SemanticKnnVectorQueryRewriteInterceptor;
import org.elasticsearch.xpack.inference.queries.SemanticMatchQueryRewriteInterceptor;
import org.elasticsearch.xpack.inference.queries.SemanticMultiMatchQueryRewriteInterceptor;
import org.elasticsearch.xpack.inference.queries.SemanticQueryBuilder;
import org.elasticsearch.xpack.inference.queries.SemanticSparseVectorQueryRewriteInterceptor;
import org.elasticsearch.xpack.inference.rank.random.RandomRankBuilder;
Expand Down Expand Up @@ -581,6 +582,7 @@ public List<QueryRewriteInterceptor> getQueryRewriteInterceptors() {
return List.of(
new SemanticKnnVectorQueryRewriteInterceptor(),
new SemanticMatchQueryRewriteInterceptor(),
new SemanticMultiMatchQueryRewriteInterceptor(),
new SemanticSparseVectorQueryRewriteInterceptor()
);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,11 @@ protected String getQuery(QueryBuilder queryBuilder) {
return queryVectorBuilder != null ? queryVectorBuilder.getModelText() : null;
}

@Override
protected boolean isResolveInferenceFieldWildcardsRequired(QueryBuilder queryBuilder) {
return false;
}

@Override
protected QueryBuilder buildInferenceQuery(QueryBuilder queryBuilder, InferenceIndexInformationForField indexInformation) {
assert (queryBuilder instanceof KnnVectorQueryBuilder);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,14 @@ protected String getQuery(QueryBuilder queryBuilder) {
return (String) matchQueryBuilder.value();
}

@Override
protected boolean isResolveInferenceFieldWildcardsRequired(QueryBuilder queryBuilder) {
return false;
}

@Override
protected QueryBuilder buildInferenceQuery(QueryBuilder queryBuilder, InferenceIndexInformationForField indexInformation) {
SemanticQueryBuilder semanticQueryBuilder = new SemanticQueryBuilder(indexInformation.fieldName(), getQuery(queryBuilder), false);
SemanticQueryBuilder semanticQueryBuilder = new SemanticQueryBuilder(getFieldName(queryBuilder), getQuery(queryBuilder), false);
semanticQueryBuilder.boost(queryBuilder.boost());
semanticQueryBuilder.queryName(queryBuilder.queryName());
return semanticQueryBuilder;
Expand Down
Loading