Use retriever instead of rank parameter for hybrid search in ES 9

[ezimuel]

In Elasticsearch 9.0.0 we removed the support of rank parameter (see here and here).
This means we need elasticsearch-py ver. 9+ to use the retriever parameter instead of rank.
If you try to use ElasticsearchStore.ApproxRetrievalStrategy(hybrid=True, ...) with Elasticsearch server 9+, you will get an error.
For instance, the following example:

ElasticsearchStore(
    index_name="test",
    es_connection=es,
    query_field="text",
    vector_query_field="text_embedding.predicted_value",
    distance_strategy="COSINE",
    strategy=ElasticsearchStore.ApproxRetrievalStrategy(
        hybrid=True,
        query_model_id="sentence-transformers__all-minilm-l6-v2"
    )
)

It generates the following POST query, that is not valid with ES 9+

POST test/_search
{
  "knn": {
    "field": "text_embedding.predicted_value", 
    "k": 10, 
    "num_candidates": 50, 
    "query_vector_builder": {
      "text_embedding": {
        "model_id": "sentence-transformers__all-minilm-l6-v2", 
        "model_text": "loyer annuel de base"}
      
    }
  }, 
  "query": {
    "bool": {
      "must": [
         {
          "match": {"text": {"query": "loyer annuel de base"}}
         }
        ]
    }
  },
  "rank": {"rrf": {}}
}

/cc @Danouchka

[Danouchka]

Thank you @ezimuel for the issue

[ssh-esh]

Some initial investigation notes

What i found strange initially is how this was not being caught by the integration tests in langchain-elastic, I checked the tests and it seems like we are testing something like this already? I have ran the CI recently and did not see any failures

These 4 test passed which seem to be doing similar things as this issue. See the output below for the filepath/test name

tests/integration_tests/_async/test_vectorstores.py::TestElasticsearch::test_similarity_search_approx_with_hybrid_search PASSED                                                          [ 25%]
tests/integration_tests/_async/test_vectorstores.py::TestElasticsearch::test_similarity_search_approx_with_hybrid_search_rrf PASSED                                                      [ 50%]
tests/integration_tests/_sync/test_vectorstores.py::TestElasticsearch::test_similarity_search_approx_with_hybrid_search PASSED                                                           [ 75%]
tests/integration_tests/_sync/test_vectorstores.py::TestElasticsearch::test_similarity_search_approx_with_hybrid_search_rrf PASSED                                                       [100%]

I ran this with ES version 9.1.1 so it should be similar setup for this issue. However, i do see this deprecated warning below however the actual operation is not failing

tests/integration_tests/_sync/test_vectorstores.py::TestElasticsearch::test_similarity_search_approx_with_hybrid_search_rrf
  /Users/eyoeshetu/Documents/langchain-elastic-fork/libs/elasticsearch/tests/integration_tests/_sync/test_vectorstores.py:738: ElasticsearchWarning: Deprecated field [rank] used, replaced by [retriever]
    es_output = docsearch.client.search(

After checking more into this issue on the slack thread here https://elastic.slack.com/archives/CMBFM3E3W/p1755011320658079

I see that the actual error message below. It looks like there is another error here about using both query_vector and query_vector_builder as part of the setup.

#! Deprecated field [rank] used, replaced by [retriever]
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "only one of [query_vector] and [query_vector_builder] must be provided"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "only one of [query_vector] and [query_vector_builder] must be provided"
  },
  "status": 400
}

It seems to me that there is some regression problem when query_vector_builder is being used where ES version > 9.0 thinks you are using both query_vector and query_vector_builder which is not allowed and not just the [rank] parameter?

[miguelgrinberg]

We made the transition to retrievers a few months ago. See elastic/elasticsearch-py#2666. If you still see that the old rank is in use, then you should start from that PR, which was supposed to address this.

[ssh-esh]

@miguelgrinberg I think your changes are working as intended. As mentioned on slack here the client from 8.16 are using the retriever format and not the old 'rank'. langchain-elastic might have been running 8.15.1 client so the CI tests were passing where we were asserting the query body sent (with the old rank)

I will probably make a new issue about updating the client dependancy in langchain-elastic to ^8.19 and updating the test suite since I don't think this issue reflects what we intend to do anymore?

[miguelgrinberg]

@ssh-esh The correct answer is that the version of the Elasticsearch client for Python should match the version of the Elasticsearch server. If this is true, then everything works, even if you use an older client with a corresponding old server (I think). Unfortunately it is not possible to express this dependency in a way that poetry can guess what the right version of the client to install is. Maybe there should be a runtime version compatibility check, but there are pros and cons to this as well.

[yann-combarnous]

@ssh-esh Unfortunately it is not possible to express this dependency in a way that poetry can guess what the right version of the client to install is. Maybe there should be a runtime version compatibility check, but there are pros and cons to this as well.

In order to allow people to use elastic 9 client, couldn’t just the documentation be updated to require people to explicitly add elastic dependency with version to pyproject.toml? This would allow to move this package to support client 8 and 9.

[ezimuel]

@yann-combarnous actually, we don't need to update to elasticsearch 9 client. We can require elasticsearch client 8.16+ for supporting the new retriever API syntax. @ssh-esh is working on this on PR #80.
If we upgrade to elasticsearch 9 client we must require Elasticsearch server 9. Using elasticsearch client >=8.16.0, <9.0.0 we will support Elasticsearch server 8 and 9.

[yann-combarnous]

@yann-combarnous actually, we don't need to update to elasticsearch 9 client. We can require elasticsearch client 8.16+ for supporting the new retriever API syntax. @ssh-esh is working on this on PR #80. If we upgrade to elasticsearch 9 client we must require Elasticsearch server 9. Using elasticsearch client >=8.16.0, <9.0.0 we will support Elasticsearch server 8 and 9.

I am just thinking about all people using renovate / dependeabot and an evergreen strategy. You could support elasticsearch client >=8.16.0, <10.0.0, and advise people in your doc to add elastic client explicitly as a dependency and this will drive whether v8 or v9 is used. We are on Elastic v9 and would like to use latest client for instance.

[ezimuel]

For the langchain integration we don't use any features of Elasticsearch 9. This is reason why I'm proposing to use the 8.16+ client. In this way we can support both Elasticsearch server 8 and 9. If we allow installing the 9 client we must require to use only Elasticsearch 9 server. The client version 9 does not work with Elasticsearch 8 server, this is the issue.
@yann-combarnous do you have any special requirements for using elasticsearch 9 client for the langchain integration?

[yann-combarnous]

For the langchain integration we don't use any features of Elasticsearch 9. This is reason why I'm proposing to use the 8.16+ client. In this way we can support both Elasticsearch server 8 and 9. If we allow installing the 9 client we must require to use only Elasticsearch 9 server. The client version 9 does not work with Elasticsearch 8 server, this is the issue. @yann-combarnous do you have any special requirements for using elasticsearch 9 client for the langchain integration?

Fair enough, we looked at the features backported to v8 client, and we should be good for now, thx!