Text similarity reranker chunks and scores snippets

[kderusso]

This is a followup to #129369, but instead of using the highlighter to create snippets we chunk and score field content directly.

Allows users to configure the chunk sizes we use to generate snippets in the following ways:

• Specifying chunking_settings in the request
• Specifying a chunk_size in the request, which we use to construct a default chunking settings with that size
• Not specifying, in which we construct a default chunking settings

Example API calls:

// Specifying chunking settings 
POST my-index/_search
{
  "track_total_hits": true,
  "fields": [
    "text"
  ],
  "retriever": {
    "text_similarity_reranker": {
      "retriever": {
        "standard": {
          "query": {
            "match": {
              "text": "What is the capital of the USA?"
            }
          }
        }
      },
      "field": "text",
      "chunk_rescorer": {
        "size": 3,
        "chunking_settings": {
          "strategy": "sentence",
          "max_chunk_size": 25,
          "sentence_overlap": 0
        }
      },
      "rank_window_size": 10,
      "inference_text": "What is the capital of the USA?"
    }
  }
}

// Only specifying chunk size, inferring sentence with 0 boundary 
POST my-index/_search
{
  "track_total_hits": true,
  "fields": [
    "text"
  ],
  "retriever": {
    "text_similarity_reranker": {
      "retriever": {
        "standard": {
          "query": {
            "match": {
              "text": "What is the capital of the USA?"
            }
          }
        }
      },
      "field": "text",
      "chunk_rescorer": {
        "size": 3,
        "chunking_settings": {
          "max_chunk_size": 25
        }
      },
      "rank_window_size": 10,
      "inference_text": "What is the capital of the USA?"
    }
  }
}

// Defaults
POST my-index/_search
{
  "track_total_hits": true,
  "fields": [
    "text"
  ],
  "retriever": {
    "text_similarity_reranker": {
      "retriever": {
        "standard": {
          "query": {
            "match": {
              "text": "What is the capital of the USA?"
            }
          }
        }
      },
      "field": "text",
      "chunk_rescorer": {},
      "rank_window_size": 10,
      "inference_text": "What is the capital of the USA?"
    }
  }
}

NOTE:

• Specifying an inference ID to pull chunking settings is planned in a followup PR.

[elasticsearchmachine]

Pinging @elastic/search-relevance (Team:Search - Relevance)

[elasticsearchmachine]

Hi @kderusso, I've created a changelog YAML for you.

[jimczi]

It's a very elegant solution @kderusso , I like where this is going.
I made some comment about the API, since we adopt chunking explicitly I think we need to remove the confusion with snippets.
The API could look like this:

{
    "text_similarity_reranker": {
        "inference_id": "my_rerank_inference_id",
        "field": "semantic_text_field",
        "chunk_scorer": {
            "top": 2,
            "chunking_settings": {
                "strategy": "sentence",
                "max_chunk_size": 128, 
                "sentence_overlap": 0
            }
        }
    }
}

We can also say that:

{
    "text_similarity_reranker": {
        "inference_id": "",
        "field": "semantic_text_field",
        "chunk_scorer": {
            "top": 2,
            "chunking_settings": {
                "max_chunk_size": 128
            }
         }
     }
}

is valid with good default for all fields as long as chunk_scorer is provided.
So something like:

{
    "text_similarity_reranker": {
        "inference_id": "",
        "field": "semantic_text_field",
        "chunk_scorer": {
            "top": 2
         }
     }
}

is valid too.

[jimczi]

Great work @kderusso , I left some nits about the API.
We should also remove the feature flag!

[jimczi]

LGTM

[kderusso]

@elasticmachine update branch

[kderusso]

@elasticmachine update branch

[kderusso]

@elasticmachine run elasticsearch-ci/pr-upgrade-part-2