Expands semantic_text tutorial with hybrid search

[kosabogi]

Overview

This PR expands the existing semantic_text tutorial by adding hybrid search method to it.

• Changed the structure of the tutorial.
• Added detailed instructions for hybrid search, including index mapping creation, reindexing, and performing hybrid search queries.

Related Issue

https://github.com/elastic/search-docs-team/issues/191

Preview

Tutorial: semantic search and hybrid search with semantic_text

[github-actions]

Documentation preview:

• ✨ Changed pages

[elasticsearchmachine]

Pinging @elastic/es-docs (Team:Docs)

[szabosteve]

These suggestions will make the CI pass. It's not a comprehensive review.

[leemthompo]

🚗 high-level impressions from me to help clarify this workflow for myself.

I think there should be just one index here, with a combination of semantic_text and standard text mappings. This eliminates a lot of redundancies, and simplifies the workflow. In my head this is how it should work, assuming we can also eliminate the reindexing step?

• Create an inference endpoint to set up the model for generating embeddings (e.g., ELSER)
• Set up one index with both standard text fields and semantic_text fields
• Index your documents, which will automatically generate embeddings for semantic_text fields.
  • 🤔 I wonder if the file upload indexing is overkill and we could just have a small inline indexing step to stay in the API flow?
• Perform a semantic search query (existing content from tutorial)
• Perform hybrid search query by combining a standard match query (for example) and a semantic query, using RRF to combine the results from both query types

[kosabogi]

🚗 high-level impressions from me to help clarify this workflow for myself.

I think there should be just one index here, with a combination of semantic_text and standard text mappings. This eliminates a lot of redundancies, and simplifies the workflow. In my head this is how it should work, assuming we can also eliminate the reindexing step?

• Create an inference endpoint to set up the model for generating embeddings (e.g., ELSER)

• Set up one index with both standard text fields and semantic_text fields

• Index your documents, which will automatically generate embeddings for semantic_text fields.

  • 🤔 I wonder if the file upload indexing is overkill and we could just have a small inline indexing step to stay in the API flow?

• Perform a semantic search query (existing content from tutorial)

• Perform hybrid search query by combining a standard match query (for example) and a semantic query, using RRF to combine the results from both query types

Thank you for the feedback! 🙌 You made some really good points, and I can see how we can simplify the workflow as you suggested. I’ll start working on the changes based on your suggestions.

Just a quick question about the reindexing step - if we use the API to index documents, I assume the embeddings will be generated automatically since the destination index mapping has the semantic_text field. Would that make reindexing unnecessary? Did I get that right?
However—please correct me if I'm wrong—this approach might only work for new documents. For already existing indices that users want to search on, wouldn't reindexing still be necessary to generate the embeddings? What do you think?

[szabosteve]

LGTM!

[elasticsearchmachine]

💔 Backport failed

The backport operation could not be completed due to the following error:

An unexpected error occurred when attempting to backport this PR.

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 114398