OpenSearchDocumentStore Index creation fails with index.knn

[katkamrachana]

Hi,
I have an OpenSearch Cluster (version 2.3.0) running on my VM instance. I am using latest Haystack to connect to OpenSearchDocumentStore on my cluster.
However, the failure seems to be at index creation, specifically with index.knn setting.
Any lead on how to I resolve this issue, any way to skip knn setting or specify intended OpenSearch version to Haystack configuration?

Below is additional information about the error.

Approach 1. create index via OpenSearch Dashboard and connect to Haystack's OpenSearchDocumentStore:

PUT /index_a
{
  "settings": {
    "index": {
      "number_of_shards": 5,
      "number_of_replicas": 1
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text"
      },
      "content": {
        "type": "text"
      },
      "timestamp": {
        "type": "date"
      }
    }
  }
}


==>>> document_store = OpenSearchDocumentStore(host="10.0.0.1", username="xyz", password="12345", index="index_a", create_index=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/venv/lib/python3.10/site-packages/haystack/nodes/base.py", line 46, in wrapper_exportable_to_yaml
    init_func(self, *args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/haystack/document_stores/opensearch.py", line 224, in __init__
    super().__init__(
  File "/opt/venv/lib/python3.10/site-packages/haystack/nodes/base.py", line 46, in wrapper_exportable_to_yaml
    init_func(self, *args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/haystack/document_stores/search_engine.py", line 108, in __init__
    self._init_indices(
  File "/opt/venv/lib/python3.10/site-packages/haystack/document_stores/search_engine.py", line 123, in _init_indices
    self._validate_and_adjust_document_index(index)
  File "/opt/venv/lib/python3.10/site-packages/haystack/document_stores/opensearch.py", line 976, in _validate_and_adjust_document_index
    self.client.indices.put_mapping(index=index_id, body=mappings, headers=headers)
  File "/opt/venv/lib/python3.10/site-packages/opensearchpy/client/utils.py", line 178, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/opensearchpy/client/indices.py", line 381, in put_mapping
    return self.transport.perform_request(
  File "/opt/venv/lib/python3.10/site-packages/opensearchpy/transport.py", line 409, in perform_request
    raise e
  File "/opt/venv/lib/python3.10/site-packages/opensearchpy/transport.py", line 370, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/opt/venv/lib/python3.10/site-packages/opensearchpy/connection/http_urllib3.py", line 266, in perform_request
    self._raise_error(
  File "/opt/venv/lib/python3.10/site-packages/opensearchpy/connection/base.py", line 301, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
opensearchpy.exceptions.RequestError: RequestError(400, 'mapper_parsing_exception', 'No handler for type [knn_vector] declared on field [embedding]')

Approach 2. connect to Haystack's OpenSearchDocumentStore, without creating an index in prior.

No index 'index_b' existing.

>>> document_store = OpenSearchDocumentStore(host="10.0.0.1", username="xyz", password="12345", index="index_b")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/venv/lib/python3.10/site-packages/haystack/nodes/base.py", line 46, in wrapper_exportable_to_yaml
    init_func(self, *args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/haystack/document_stores/opensearch.py", line 224, in __init__
    super().__init__(
  File "/opt/venv/lib/python3.10/site-packages/haystack/nodes/base.py", line 46, in wrapper_exportable_to_yaml
    init_func(self, *args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/haystack/document_stores/search_engine.py", line 108, in __init__
    self._init_indices(
  File "/opt/venv/lib/python3.10/site-packages/haystack/document_stores/search_engine.py", line 118, in _init_indices
    self._create_document_index(index)
  File "/opt/venv/lib/python3.10/site-packages/haystack/document_stores/opensearch.py", line 867, in _create_document_index
    raise e
  File "/opt/venv/lib/python3.10/site-packages/haystack/document_stores/opensearch.py", line 860, in _create_document_index
    self.client.indices.create(index=index_name, body=index_definition, headers=headers)
  File "/opt/venv/lib/python3.10/site-packages/opensearchpy/client/utils.py", line 178, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/opensearchpy/client/indices.py", line 128, in create
    return self.transport.perform_request(
  File "/opt/venv/lib/python3.10/site-packages/opensearchpy/transport.py", line 409, in perform_request
    raise e
  File "/opt/venv/lib/python3.10/site-packages/opensearchpy/transport.py", line 370, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/opt/venv/lib/python3.10/site-packages/opensearchpy/connection/http_urllib3.py", line 266, in perform_request
    self._raise_error(
  File "/opt/venv/lib/python3.10/site-packages/opensearchpy/connection/base.py", line 301, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
opensearchpy.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'unknown setting [index.knn] please check that any required plugins are installed, or check the breaking changes documentation for removed settings')
>>>

[katkamrachana]

@varun1524 FYI

[bogdankostic]

Hi @katkamrachana! It seems that the knn plugin is not installed for your OpenSearch instance. Can you list the installed plugins and paste the results here? (Here you can find two ways of how to list installed plugins.)

[katkamrachana]

Hey @bogdankostic , I was under impression that OpenSearch 2.3.0 does not support knn. However, I see it in installed plugins list:

$  bin/opensearch-plugin list
opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-geospatial
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-ml
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-security
opensearch-sql

[bogdankostic]

According to the OpenSearch documentation, the knn plugin is bundled with all distributions except for minimum distribution packages.

Can you share more details on how you set-up your OpenSearch cluster that would allow me to replicate your problem and investigate what's causing it?

[katkamrachana]

I have OpenSearch cluster created in Oracle Cloud. I referred this documentation: https://docs.oracle.com/en/learn/oci-opensearch/index.html#introduction

I can answer any queries about the cluster configuration or details

[katkamrachana]

@bogdankostic You can run these yml files to reproduce the problem.

docker compose file:

version: "3"
services:
  haystack-api:
    image: "deepset/haystack:cpu"
    networks:
      - haystack_cpu_network_opensearch
    # Mount custom Pipeline YAML and custom Components.
    volumes:
      - ./rest_api/pipeline:/home/user/rest_api/pipeline
      - .:/home/user/workplace
    ports:
      - "8000:8000"
    restart: on-failure
    environment:
      # See rest_api/pipeline/pipelines.haystack-pipeline.yml for configurations of Search & Indexing Pipeline.
      - DOCUMENTSTORE_PARAMS_HOST=opensearch
      - PIPELINE_YAML_PATH=/home/user/rest_api/pipeline/pipelines.haystack-pipeline-opensearch.yml
      - CONCURRENT_REQUEST_PER_WORKER
    depends_on:
      - opensearch
    # Starts REST API with only 2 workers so that it can be run on systems with just 4GB of memory
    # If you need to handle large loads of incoming requests and have memory to spare, consider increasing the number of workers
    command: "/bin/bash -c 'sleep 10 && gunicorn rest_api.application:app -b 0.0.0.0 -k uvicorn.workers.UvicornWorker --workers 1 --timeout 1800'"

  opensearch:
    image: opensearchproject/opensearch:2.3.0
    environment:
      - discovery.type=single-node
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - haystack_cpu_network_opensearch


  ui:
    image: "deepset/demo-explore-the-world"
    networks:
      - haystack_cpu_network_opensearch
    ports:
      - "8501:8501"
    restart: on-failure
    environment:
      - API_ENDPOINT=http://haystack-api:8000
      # The value fot the following variables will be read from the host, if present.
      # They can also be temporarily set for docker-compose, for example:
      # $ DISABLE_FILE_UPLOAD=1 DEFAULT_DOCS_FROM_RETRIEVER=5 docker-compose up
      - DISABLE_FILE_UPLOAD
      - DEFAULT_QUESTION_AT_STARTUP
      - DEFAULT_DOCS_FROM_RETRIEVER
      - DEFAULT_NUMBER_OF_ANSWERS
    command: "/bin/bash -c 'sleep 15 && python -m streamlit run ui/webapp.py'"

networks:
  haystack_cpu_network_opensearch:
    name: haystack_cpu_network_opensearch

and pipeline.yml file

# To allow your IDE to autocomplete and validate your YAML pipelines, name them as <name of your choice>.haystack-pipeline.yml

version: ignore

components:    # define all the building-blocks for Pipeline
  - name: DocumentStore
    type: OpenSearchDocumentStore  # consider other options for scaling to large number of documents
    params:
      host: localhost
  - name: Retriever
    type: BM25Retriever
    params:
      document_store: DocumentStore    # params can reference other components defined in the YAML
      top_k: 5
  - name: Reader       # custom-name for the component; helpful for visualization & debugging
    type: FARMReader    # Haystack Class name for the component
    params:
      model_name_or_path: sultan/BioM-ALBERT-xxlarge-SQuAD2 # sultan/BioM-ELECTRA-Large-SQuAD2-BioASQ8B
      context_window_size: 500
      return_no_answer: true
      batch_size: 16
  - name: TextFileConverter
    type: TextConverter
  - name: PDFFileConverter
    type: PDFToTextConverter
  - name: Preprocessor
    type: PreProcessor
    params:
      split_by: word
      split_length: 1000
  - name: FileTypeClassifier
    type: FileTypeClassifier

pipelines:
  - name: query    # a sample extractive-qa Pipeline
    nodes:
      - name: Retriever
        inputs: [Query]
      - name: Reader
        inputs: [Retriever]
  - name: indexing
    nodes:
      - name: FileTypeClassifier
        inputs: [File]
      - name: TextFileConverter
        inputs: [FileTypeClassifier.output_1]
      - name: PDFFileConverter
        inputs: [FileTypeClassifier.output_2]
      - name: Preprocessor
        inputs: [PDFFileConverter, TextFileConverter]
      - name: Retriever
        inputs: [Preprocessor]
      - name: DocumentStore
        inputs: [Retriever]

[bogdankostic]

I'm not able to reproduce the problem using the provided docker-compose file and the Pipeline configuration, the OpenSearchDocumentStore runs fine for me. Does it work for you when you run it locally?
I'd assume that something went wrong when setting up the OpenSearch cluster in oracle cloud.

[katkamrachana]

So did you mean you are able to run OpenSearch 2.3.0 with latest Haystack?
I tried without OpenSearch cluster in Oracle cloud as well. It was still failing. Let me share the haystack logs when a container is spinned up using the above docker compose file.

warnings.warn(
05/10/2023 08:15:02 PM Error loading indexing pipeline from /home/user/rest_api/pipeline/pipelines.haystack-oci-os-pipeline.yml.
 Failed loading pipeline component 'Retriever'. See the stacktrace above for more information.
05/10/2023 08:15:02 PM Indexing Pipeline is not setup. File Upload API will not be available.

As said, file upload is not working. Can you share the logs at your end. Thank you!

[bogdankostic]

Yes, running OpenSearch 2.3.0 with latest Haystack works fine for me. Attached you find the logs when running docker-compose up with your specified docker-compose file and pipeline configuration.

logs.txt

What's the error stacktrace that you got?