diff --git a/explore-analyze/query-filter/languages/esql-getting-started.md b/explore-analyze/query-filter/languages/esql-getting-started.md index 10c98d08bf..d15e405440 100644 --- a/explore-analyze/query-filter/languages/esql-getting-started.md +++ b/explore-analyze/query-filter/languages/esql-getting-started.md @@ -423,5 +423,5 @@ For more about data processing with {{esql}}, refer to [Data processing with DIS To learn more about {{esql}}, refer to [{{esql}} reference](elasticsearch://reference/query-languages/esql.md). -Learn more about using {{esql}} for Search use cases in this tutorial: [Search and filter with {{esql}}](/solutions/search/esql-full-text-filter-tutorial.md). +Learn more about using {{esql}} for Search use cases in this tutorial: [Search and filter with {{esql}}](/solutions/search/esql-search-tutorial.md). diff --git a/explore-analyze/query-filter/languages/esql.md b/explore-analyze/query-filter/languages/esql.md index 243e8fc1cd..98f91184e6 100644 --- a/explore-analyze/query-filter/languages/esql.md +++ b/explore-analyze/query-filter/languages/esql.md @@ -35,7 +35,7 @@ You can use it: - In your queries to {{es}} APIs, using the [`_query` endpoint](/explore-analyze/query-filter/languages/esql-rest.md) that accepts queries written in {{esql}} syntax. - Within various {{kib}} tools such as Discover and Dashboards, to explore your data and build powerful visualizations. -Learn more about using {{esql}} for Search use cases in this tutorial: [Search and filter with {{esql}}](/solutions/search/esql-full-text-filter-tutorial.md). +Learn more about using {{esql}} for Search use cases in this tutorial: [Search and filter with {{esql}}](/solutions/search/esql-search-tutorial.md). ## Next steps diff --git a/solutions/search/api-quickstarts.md b/solutions/search/api-quickstarts.md index 03f17891ac..26fb7efb23 100644 --- a/solutions/search/api-quickstarts.md +++ b/solutions/search/api-quickstarts.md @@ -15,7 +15,7 @@ Use the following quickstarts to get hands-on experience with Elasticsearch APIs - [Basic full-text search and filtering in Elasticsearch](querydsl-full-text-filter-tutorial.md): Learn about different options for querying data, including full-text search and filtering, using the Query DSL. - [Analyze eCommerce data with aggregations using Query DSL](/explore-analyze/query-filter/aggregations/tutorial-analyze-ecommerce-data-with-aggregations-using-query-dsl.md): Learn how to analyze data using different types of aggregations, including metrics, buckets, and pipelines. % - [Getting started with {{esql}}](esql-getting-started.md): Learn how to query and aggregate your data using {{esql}}. -- [Search and filter with {{esql}}](esql-full-text-filter-tutorial.md): Learn how to perform full-text and semantic search in {{esql}}, combined with filtering. +- [Search and filter with {{esql}}](esql-search-tutorial.md): Learn how to perform full-text and semantic search in {{esql}}, combined with filtering. - [Semantic search](semantic-search/semantic-search-semantic-text.md): Learn how to create embeddings for your data with `semantic_text` and query using the `semantic` query. - [Hybrid search](hybrid-semantic-text.md): Learn how to combine semantic search using`semantic_text` with full-text search. - [Bring your own dense vector embeddings](vector/bring-own-vectors.md): Learn how to ingest dense vector embeddings into Elasticsearch. diff --git a/solutions/search/esql-full-text-filter-tutorial.md b/solutions/search/esql-search-tutorial.md similarity index 82% rename from solutions/search/esql-full-text-filter-tutorial.md rename to solutions/search/esql-search-tutorial.md index eae00058ad..2b24bd2e8f 100644 --- a/solutions/search/esql-full-text-filter-tutorial.md +++ b/solutions/search/esql-search-tutorial.md @@ -8,9 +8,11 @@ navigation_title: "Search and filter with ES|QL" This tutorial presents examples in {{esql}} syntax. Refer to [the Query DSL version](querydsl-full-text-filter-tutorial.md) for the equivalent examples in Query DSL syntax. ::: -This is a hands-on introduction to the basics of [full-text search](full-text.md) with Elasticsearch, also known as *lexical search*, and how to filter search results based on exact criteria. In this scenario, we're implementing a search function for a cooking blog. The blog contains recipes with various attributes including textual content, categorical data, and numerical ratings. +This is a hands-on introduction to the basics of full-text search and semantic search, using [{{esql}}](/explore-analyze/query-filter/languages/esql.md). -This tutorial covers lexical search, with a brief [introductory section on semantic search](#bonus-semantic-search-with-esql) at the end. +% TODO For an overview of all the search capabilities in {{esql}}, refer to [Search with {{esql}}](elasticsearch://reference/query-languages/esql/esql-for-search.md). + +In this scenario, we're implementing search for a cooking blog. The blog contains recipes with various attributes including textual content, categorical data, and numerical ratings. ## Requirements @@ -147,10 +149,12 @@ Full-text search involves executing text-based queries across one or more docume :::{tip} {{esql}} provides two ways to perform full-text searches: -1. Full match function syntax: `match(field, "search terms")` -1. Compact syntax using the match operator "`:`": `field:"search terms"` +1. Full [match function](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-match) syntax: `match(field, "search terms")` +1. Compact syntax using the [match operator `:`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-search-operators): `field:"search terms"` Both are equivalent and can be used interchangeably. The compact syntax is more concise, while the function syntax allows for more configuration options. We'll use the compact syntax in most examples for brevity. + +Refer to the [match function](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-match) reference docs for advanced parameters available with the function syntax. ::: ### Basic full-text query @@ -163,6 +167,10 @@ FROM cooking_blog # Specify the index to search | LIMIT 1000 # Return up to 1000 results ``` +:::{note} +The results ordering isn't by relevance, as we haven't requested the `_score` metadata field. We'll cover relevance scoring in the next section. +::: + By default, like the Query DSL `match` query, {{esql}} uses `OR` logic between terms. This means it will match documents that contain either "fluffy" or "pancakes", or both, in the description field. :::{tip} @@ -200,7 +208,67 @@ FROM cooking_blog This query searches the title field to match at least 2 of the 3 terms: "fluffy", "pancakes", or "breakfast". -## Step 4: Search across multiple fields at once +## Step 4: Semantic search and hybrid search + +### Index semantic content + +{{es}} allows you to semantically search for documents based on the meaning of the text, rather than just the presence of specific keywords. This is useful when you want to find documents that are conceptually similar to a given query, even if they don't contain the exact search terms. + +ES|QL supports semantic search when your mappings include fields of the [`semantic_text`](elasticsearch://reference/elasticsearch/mapping-reference/semantic-text.md) type. This example mapping update adds a new field called `semantic_description` with the type `semantic_text`: + +```console +PUT /cooking_blog/_mapping +{ + "properties": { + "semantic_description": { + "type": "semantic_text" + } + } +} +``` + +Next, index a document with content into the new field: + +```console +POST /cooking_blog/_doc +{ + "title": "Mediterranean Quinoa Bowl", + "semantic_description": "A protein-rich bowl with quinoa, chickpeas, fresh vegetables, and herbs. This nutritious Mediterranean-inspired dish is easy to prepare and perfect for a quick, healthy dinner.", + "author": "Jamie Oliver", + "date": "2023-06-01", + "category": "Main Course", + "tags": ["vegetarian", "healthy", "mediterranean", "quinoa"], + "rating": 4.7 +} +``` + +### Perform semantic search + +Once the document has been processed by the underlying model running on the inference endpoint, you can perform semantic searches. Here's an example natural language query against the `semantic_description` field: + +```esql +FROM cooking_blog +| WHERE semantic_description:"What are some easy to prepare but nutritious plant-based meals?" +| LIMIT 5 +``` + +:::{tip} +Follow this [tutorial](/solutions/search/semantic-search/semantic-search-semantic-text.md) if you'd like to test out the semantic search workflow against a large dataset. +::: + +### Perform hybrid search + +You can combine full-text and semantic queries. In this example we combine full-text and semantic search with custom weights: + +```esql +FROM cooking_blog METADATA _score +| WHERE match(semantic_description, "easy to prepare vegetarian meals", { "boost": 0.75 }) + OR match(tags, "vegetarian", { "boost": 0.25 }) +| SORT _score DESC +| LIMIT 5 +``` + +## Step 5: Search across multiple fields at once When users enter a search query, they often don't know (or care) whether their search terms appear in a specific field. {{esql}} provides ways to search across multiple fields simultaneously: @@ -225,14 +293,14 @@ FROM cooking_blog METADATA _score # Request _score metadata for relevance-based ``` :::{tip} -When working with relevance scoring in ES|QL, it's important to understand how `_score` works. If you don't include `METADATA _score` in your query, you're only performing filtering operations with no relevance calculation. +When working with relevance scoring in ES|QL, it's important to understand `_score`. If you don't include `METADATA _score` in your query, you won't see relevance scores in your results. This means you won't be able to sort by relevance or filter based on relevance scores. When you include `METADATA _score`, search functions included in WHERE conditions contribute to the relevance score. Filtering operations (like range conditions and exact matches) don't affect the score. -Remember that including `METADATA _score` doesn't automatically sort your results by relevance. You must explicitly use `SORT _score DESC` or `SORT _score ASC` to order your results by relevance. +If you want the most relevant results first, you must sort by `_score`, by explicitly using `SORT _score DESC` or `SORT _score ASC`. ::: -## Step 5: Filter and find exact matches +## Step 6: Filter and find exact matches Filtering allows you to narrow down your search results based on exact criteria. Unlike full-text searches, filters are binary (yes/no) and do not affect the relevance score. Filters execute faster than queries because excluded results don't need to be scored. @@ -271,7 +339,7 @@ FROM cooking_blog Like the `term` query in Query DSL, this has zero flexibility and is case-sensitive. -## Step 6: Combine multiple search criteria +## Step 7: Combine multiple search criteria Complex searches often require combining multiple search criteria: @@ -304,55 +372,18 @@ FROM cooking_blog METADATA _score ``` -## Bonus: Semantic search with ES|QL - -ES|QL also supports semantic search when your mappings include fields of the [`semantic_text`](elasticsearch://reference/elasticsearch/mapping-reference/semantic-text.md) type. This example mapping update adds a new field called `semantic_description` with the type `semantic_text`: - -```console -PUT /cooking_blog/_mapping -{ - "properties": { - "semantic_description": { - "type": "semantic_text" - } - } -} -``` - -Next, index a document with content into the new field: - -```console -POST /cooking_blog/_doc -{ - "title": "Mediterranean Quinoa Bowl", - "semantic_description": "A protein-rich bowl with quinoa, chickpeas, fresh vegetables, and herbs. This nutritious Mediterranean-inspired dish is easy to prepare and perfect for a quick, healthy dinner.", - "author": "Jamie Oliver", - "date": "2023-06-01", - "category": "Main Course", - "tags": ["vegetarian", "healthy", "mediterranean", "quinoa"], - "rating": 4.7 -} -``` - -Once the document has been processed by the underlying model running on the inference endpoint, you can perform semantic searches. Here's an example natural language query against the `semantic_description` field: - -```esql -FROM cooking_blog -| WHERE semantic_description:"What are some easy to prepare but nutritious plant-based meals?" -| LIMIT 5 - -``` - -:::{tip} -Follow this [tutorial](/solutions/search/semantic-search/semantic-search-semantic-text.md) if you'd like to test out the semantic search workflow against a large dataset. -::: - ## Learn more +### Documentation + This tutorial introduced the basics of search and filtering in {{esql}}. Building a real-world search experience requires understanding many more advanced concepts and techniques. Here are some resources once you're ready to dive deeper: -- [Full-text search](full-text.md): Learn about the core components of full-text search in Elasticsearch. - - [Text analysis](full-text/text-analysis-during-search.md): Understand how text is processed for full-text search. +% TODO [Search with {{esql}}](): Learn about all your options for search use cases with {{esql}}. - [{{esql}} search functions](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-search-functions): Explore the full list of search functions available in {{esql}}. - [Semantic search](/solutions/search/semantic-search.md): Understand your various options for semantic search in Elasticsearch. - - [The `semantic_text` workflow](/solutions/search/semantic-search.md#_semantic_text_workflow): Learn how to use the `semantic_text` field type for semantic search. This is the recommended approach for most users looking to perform semantic search in {{es}}, because it abstracts away the complexity of setting up inference endpoints and models. \ No newline at end of file + - [The `semantic_text` workflow](/solutions/search/semantic-search.md#_semantic_text_workflow): Learn how to use the `semantic_text` field type for semantic search. This is the recommended approach for most users looking to perform semantic search in {{es}}, because it abstracts away the complexity of setting up inference endpoints and models. + +### Related blog posts + +% TODO [[uncomment once blog is live]] - https://www.elastic.co/blog/esql-you-know-for-search-scoring-semantic-search[ES|QL, you know for Search]: Introducing scoring and semantic search +- [Introducing full text filtering in ES|QL](https://www.elastic.co/blog/introducing-full-text-filtering-with-esql): Overview of text filtering capabilities diff --git a/solutions/search/querydsl-full-text-filter-tutorial.md b/solutions/search/querydsl-full-text-filter-tutorial.md index 5e19d08df7..8a0004ff80 100644 --- a/solutions/search/querydsl-full-text-filter-tutorial.md +++ b/solutions/search/querydsl-full-text-filter-tutorial.md @@ -10,7 +10,7 @@ applies_to: # Tutorial: Full-text search and filtering with Query DSL [full-text-filter-tutorial] :::{tip} -This tutorial presents examples in Query DSL syntax. Refer to [the {{esql}} version](esql-full-text-filter-tutorial.md) for the equivalent examples in {{esql}} syntax. +This tutorial presents examples in Query DSL syntax. Refer to [the {{esql}} version](esql-search-tutorial.md) for the equivalent examples in {{esql}} syntax. ::: This is a hands-on introduction to the basics of [full-text search](full-text.md) with {{es}}, also known as *lexical search*, using the [`_search` API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) and [Query DSL](../../explore-analyze/query-filter/languages/querydsl.md). You’ll also learn how to filter data, to narrow down search results based on exact criteria. diff --git a/solutions/toc.yml b/solutions/toc.yml index 323bc7dcf5..0edb8da6c7 100644 --- a/solutions/toc.yml +++ b/solutions/toc.yml @@ -12,7 +12,7 @@ toc: children: - file: search/elasticsearch-basics-quickstart.md - file: search/querydsl-full-text-filter-tutorial.md - - file: search/esql-full-text-filter-tutorial.md + - file: search/esql-search-tutorial.md - file: search/ingest-for-search.md children: - file: search/search-pipelines.md