ES|QL: semantic search with lookup join returns no results

[ioanatia]

Elasticsearch Version

8.18

Installed Plugins

No response

Java Version

bundled

OS Version

Problem Description

This is an issue that's only present in 8.18.
When using semantic search with lookup join, we return no results.

The issue has to do with how we detect the index names in QueryBuilderResolver:

elasticsearch/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/QueryBuilderResolver.java

Lines 101 to 103 in 46387fb

	Holder<Set<String>> indexNames = new Holder<>();
	plan.forEachDown(EsRelation.class, esRelation -> indexNames.set(esRelation.concreteIndices()));
	return indexNames.get();

When using lookup join, indexNames will hold only the right side index name.

This is not a problem for 8.x or main due to the refactor from #121260 that already fixes the issue:

elasticsearch/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/fulltext/QueryBuilderResolver.java

Lines 66 to 70 in 7f7967b

	private static Set<String> indexNames(LogicalPlan plan) {
	Set<String> indexNames = new HashSet<>();
	plan.forEachDown(EsRelation.class, esRelation -> indexNames.addAll(esRelation.concreteIndices()));
	return indexNames;
	}

Steps to Reproduce

Setup:

PUT search-movies
{
    "mappings": {
        "properties": {
            "semantic_title": { "type": "semantic_text"},
            "language": { "type": "keyword"}
        }
    }
}


PUT search-movies/_doc/1
{
    "semantic_title": "The boy and the heron",
    "language": "Japanese"
}

PUT languages
{
    "settings": {
        "index.mode": "lookup"
    },
    "mappings": {
        "properties": {
            "language": { "type": "keyword"},
            "code": { "type": "keyword"}
        }
    }
}

POST languages/_doc
{
    "language": "Japanese",
    "code": "jp"
}

Issue a query that uses semantic search + lookup join:

POST _query
{
    "query": """
        FROM search-movies
        | WHERE semantic_title: "bird"
        | LOOKUP JOIN languages ON language
        """,
        "profile": true
}

we return no results.

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[ioanatia]

should be fixed and should be properly covered with tests