KNN filters on metadata of sibling nested documents

[tihom88]

Description

Description

Currently KNN filters have a very limited scope. #106994 and #113949 enhances KNN filters.

We have another use case for KNN filters. We have two sibling nested documents and we want to prefilter documents based on metadata in sibling nested document before KNN query.
Here is example:

MAPPING:

# Create index with the mapping
PUT /my-test-index
{
  "mappings": {
    "properties": {
      "metadata": {
        "type": "nested",
        "properties": {
          "key": { "type": "keyword" },
          "value": { "type": "keyword" }
        }
      },
      "vectors": {
        "type": "nested",
        "properties": {
          "id": { "type": "keyword" },
          "vector": { 
            "type": "dense_vector", 
            "dims": 3,
            "index": true,
            "similarity": "cosine"
          }
        }
      },
      "title": { "type": "text" },
      "description": { "type": "text" }
    }
  }
}

# Example documents

# Document 1
POST /my-test-index/_doc/1
{
  "title": "Machine Learning Basics",
  "description": "Introduction to machine learning concepts",
  "metadata": [
    { "key": "category", "value": "education" },
    { "key": "level", "value": "beginner" }
  ],
  "vectors": [
    { "id": "concept_vec", "vector": [0.5, 0.2, 0.9] }
  ]
}

# Document 2
POST /my-test-index/_doc/2
{
  "title": "Advanced Neural Networks",
  "description": "Deep dive into neural network architectures",
  "metadata": [
    { "key": "category", "value": "education" },
    { "key": "level", "value": "advanced" }
  ],
  "vectors": [
    { "id": "concept_vec", "vector": [0.8, 0.1, 0.6] }
  ]
}

# Document 3
POST /my-test-index/_doc/3
{
  "title": "Data Visualization Techniques",
  "description": "Methods for effective data visualization",
  "metadata": [
    { "key": "category", "value": "visualization" },
    { "key": "level", "value": "intermediate" }
  ],
  "vectors": [
    { "id": "concept_vec", "vector": [0.3, 0.7, 0.2] }
  ]
}

# Document 4
POST /my-test-index/_doc/4
{
  "title": "Natural Language Processing",
  "description": "Text analysis and language models",
  "metadata": [
    { "key": "category", "value": "nlp" },
    { "key": "level", "value": "intermediate" }
  ],
  "vectors": [
    { "id": "concept_vec", "vector": [0.9, 0.4, 0.3] }
  ]
}

We want to filter knn results based on filters based on nested sibling document "metadata"

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[benwtrent]

@tihom88 would you want to do a AND between filters over nested and parent metadata? Or would an OR suffice?

The referenced PR only provides OR support, which is why I have been hesitant on its usefulness for now.

Maybe OR is enough.

[tihom88]

benwtrent We mainly need AND query as we want to filter out results based on sibling (nested document) metadata.

[benwtrent]

Thank you for the clarification @tihom88!

[tihom88]

@benwtrent is there any timeline available for this. We need to evaluate whether we can wait for this change in ES or should we start on major refactoring to support our use case?
Support for "AND" queries is first requirement but we would need support for both AND and OR.

[benwtrent]

is there any timeline available for this.

There is not :(

[tihom88]

@benwtrent is this use case also resolved in the issue mentioned in closing remark. i.e. with 2 sibling nested documents prefilter sibling nested documents and then perform nested knn query on these documents.