[8.18] [DOCS][8.x] Add Using ESQL for search landing page (#125580)

docs/reference/esql/esql-for-search.asciidoc

@@ -0,0 +1,215 @@
+[[esql-for-search]]
+=== Using {esql} for search
+
+preview::[]
+
+This page provides an overview of how to use {esql} for search use cases.
+
+// [TIP]
+// ====
+// Prefer to get started with a hands-on tutorial? Check out <<esql-search-tutorial>>.
+// ====
+
+The following table summarizes the key search features available in {esql} and when they were introduced.
+
+[cols="1,1,2", options="header"]
+|===
+|Feature |Available since |Description
+
+|<<esql-search-functions,Full text search functions>>
+|8.17
+|Perform basic text searches with <<esql-match, match function>> and <<esql-search-operators,match operator (`:`)>>
+
+|<<esql-for-search-query-string,Query string function>>
+|8.17
+|Execute complex queries with <<esql-qstr,`qstr`>> using Query String syntax
+
+|<<esql-for-search-scoring,Relevance scoring>>
+|8.18/9.0
+|Calculate and sort by relevance with `METADATA _score`
+
+|Enhanced match options
+|8.18/9.0
+|Configure text searches with additional parameters for the `match` function
+
+|<<esql-for-search-kql,Kibana Query Language>>
+|8.18/9.0
+|Use Kibana Query Language with <<esql-kql,`kql`>> function
+
+|<<esql-for-search-semantic,Semantic search>>
+|8.18/9.0
+|Perform semantic searches on `semantic_text` field types
+
+|<<esql-for-search-hybrid,Hybrid search>>
+|8.18/9.0
+|Combine lexical and semantic search approaches with custom weights
+|===
+
+[[esql-filtering-vs-searching]]
+==== Filtering vs. searching
+
+{esql} can be used for both simple filtering and relevance-based searching:
+
+* **Filtering** removes non-matching documents without calculating relevance scores
+* **Searching** both filters documents and ranks them by how well they match the query
+
+Note that filtering is faster than searching, because it doesn't require score calculations.
+
+[[esql-for-search-scoring]]
+===== Relevance scoring
+
+To get the most relevant results first, you need to use `METADATA _score` and sort by score. For example:
+
+[source,esql]
+----
+FROM books METADATA _score
+| WHERE match(title, "Shakespeare") OR match(plot, "Shakespeare")
+| SORT _score DESC
+----
+
+[[esql-for-search-how-scoring-works]]
+===== How `_score` works
+
+When working with relevance scoring in ES|QL:
+
+* If you don't include `METADATA _score` in your query, this only performs filtering operations with no relevance calculation.
+* When you include `METADATA _score`, any search function included in `WHERE` conditions contribute to the relevance score. This means that every occurrence of `MATCH`, `QSTR` and `KQL` will affect the score.
+* Filtering operations that are not search functions, like range conditions and exact matches, don't affect the score.
+* Including `METADATA _score` doesn't automatically sort your results by relevance. You must explicitly use `SORT _score DESC` or `SORT _score ASC` to order your results by relevance.
+
+[[esql-for-search-full-text]]
+==== Full text search
+
+[[esql-for-search-match-function-operator]]
+===== Match function and operator
+
+ES|QL offers two syntax options for `match`, which replicate the functionality of <<query-dsl-match-query, `match`>> queries in Query DSL.
+
+Use the compact operator syntax (`:`) for simple text matching with default parameters.
+
+[source,esql]
+----
+FROM logs | WHERE match(message, "connection error")
+----
+
+Use the `match()` function syntax when you need to pass additional parameters:
+
+[source,esql]
+----
+FROM products | WHERE match(name, "laptop", { "boost": 2.0 })
+----
+
+These full-text functions address several key limitations that existed for text filtering in {esql}:
+
+* They work directly on multivalued fields, returning results when any value in a multivalued field matches the query
+* They leverage analyzers, ensuring the query is analyzed with the same process as the indexed data (enabling case-insensitive matching, ASCII folding, stopword removal, and synonym support)
+* They are highly performant, using Lucene index structures rather than pattern matching or regular expressions to locate terms in your data
+
+Refer to this blog for more context: https://www.elastic.co/search-labs/blog/filtering-in-esql-full-text-search-match-qstr[Introducing full text filtering in ES|QL].
+
+[TIP]
+====
+See <<match-field-params,Match field parameters>> for more advanced options using match. 
+====
+
+[IMPORTANT]
+====
+These queries match documents but don't automatically sort by relevance. To get the most relevant results first, you need to use `METADATA _score` and sort by score. See <<esql-for-search-scoring,Relevance scoring>> for more information.
+====
+
+[[esql-for-search-query-string]]
+===== Query string function (`QSTR`)
+
+The <<esql-qstr,`qstr` function>> provides the same functionality as the Query DSL's `query_string` query. This is for advanced use cases, such as wildcard searches, searches across multiple fields, and more.
+
+[source,esql]
+----
+FROM articles METADATA _score
+| WHERE QSTR("(new york city) OR (big apple)")
+| SORT _score DESC
+| LIMIT 10
+----
+
+For complete details, refer to the <<query-dsl-query-string-query, Query DSL `query_string` docs>>.
+
+[[esql-for-search-kql]]
+===== Kibana Query Language function (`KQL`)
+
+Use the <<esql-kql,KQL function>> to use the {kibana-ref}/kuery-query.html[Kibana Query Language] in your ES|QL queries:
+
+[source,esql]
+----
+FROM logs*
+| WHERE KQL("http.request.method:GET AND agent.type:filebeat")
+----
+
+The `kql` function is useful when transitioning queries from Kibana's Discover, Dashboard, or other interfaces that use KQL. This will allow you to gradually migrate queries to ES|QL without needing to rewrite them all at once.
+
+[[esql-for-search-semantic]]
+==== Semantic search
+
+You can perform semantic searches over <<semantic-text, `semantic_text`>> field types using the same match syntax as full-text search.
+
+This example uses the match operator `:`:
+
+[source,esql]
+----
+FROM articles METADATA _score
+| WHERE semantic_content:"What are the impacts of climate change on agriculture?"
+| SORT _score DESC
+----
+
+This example uses the match function:
+
+[source,esql]
+----
+FROM articles METADATA _score
+| WHERE match(semantic_content, "What are the impacts of climate change on agriculture?")
+| SORT _score DESC
+----
+
+[[esql-for-search-hybrid]]
+==== Hybrid search
+
+Combine traditional and semantic search with custom weights:
+
+[source,esql]
+----
+FROM books METADATA _score
+| WHERE match(semantic_title, "fantasy adventure", { "boost": 0.75 }) 
+    OR match(title, "fantasy adventure", { "boost": 0.25 })
+| SORT _score DESC
+----
+
+[[esql-for-search-limitations]]
+==== Limitations
+
+Refer to <<esql-limitations-full-text-search, {esql} limitations>> for a list of known limitations.
+
+[[esql-for-search-next-steps]]
+==== Next steps
+
+[[esql-for-search-tutorials]]
+===== Tutorials and how-to guides
+
+// TODO * <<esql-search-tutorial>>: Hands-on tutorial for getting started with search tools in {esql}
+* <<semantic-search-semantic-text>>: Learn how to use the `semantic_text` field type
+
+[[esql-for-search-reference]]
+===== Technical reference
+
+* <<esql-search-functions>>: Complete reference for all search functions
+* <<esql-limitations-full-text-search, Limitations>>: Current limitations for search in ES|QL
+
+[[esql-for-search-concepts]]
+===== Background concepts
+
+* <<analysis>>: Learn how text is processed for full-text search
+* <<semantic-search>>: Get an overview of semantic search in {es}
+* <<query-filter-context>>: Understand the difference between query and filter contexts in {es}
+
+[[esql-for-search-blogs]]
+===== Related blog posts
+
+// TODO* https://www.elastic.co/blog/esql-you-know-for-search-scoring-semantic-search[ES|QL, you know for Search]: Introducing scoring and semantic search
+* https://www.elastic.co/search-labs/blog/filtering-in-esql-full-text-search-match-qstr[Introducing full text filtering in ES|QL]: Overview of text filtering capabilities

docs/reference/esql/esql-get-started.asciidoc

@@ -4,7 +4,7 @@
 <titleabbrev>Getting started</titleabbrev>
 ++++
 
-This guide shows how you can use {esql} to query and aggregate your data.
+This guide shows how you can use {esql} to query and aggregate your data. Refer to <<esql-for-search>> if you'd like to learn more about using {esql} for search use cases.
 
 [TIP]
 ====

docs/reference/esql/esql-language.asciidoc

@@ -11,8 +11,8 @@ Detailed reference documentation for the {esql} language:
 * <<esql-functions>>
 * <<esql-metadata-fields>>
 * <<esql-multivalued-fields>>
-* <<esql-enrich-data>>
 * <<esql-lookup-join>>
+* <<esql-enrich-data>>
 * <<esql-process-data-with-dissect-and-grok>>
 * <<esql-implicit-casting>>
 * <<esql-time-spans>>
@@ -23,7 +23,7 @@ include::esql-functions-operators.asciidoc[]
 include::metadata-fields.asciidoc[]
 include::multivalued-fields.asciidoc[]
 include::esql-process-data-with-dissect-grok.asciidoc[]
-include::esql-enrich-data.asciidoc[]
 include::esql-lookup-join.asciidoc[]
+include::esql-enrich-data.asciidoc[]
 include::implicit-casting.asciidoc[]
 include::time-spans.asciidoc[]

docs/reference/esql/esql-using.asciidoc

@@ -1,27 +1,33 @@
 [[esql-using]]
 == Using {esql}
 
+This page is an overview of the various ways you can use {esql} across different Elastic interfaces and use cases.
+
 <<esql-rest>>::
-Information about using the <<esql-apis,{esql} query APIs>>.
+Learn how to use the <<esql-apis,{esql} query APIs>>.
+
+<<esql-for-search>>::
+Learn how to use {esql} for search use cases.
 
 <<esql-kibana>>::
-Using {esql} in {kib} to query and aggregate your data, create visualizations,
+Learn how to use {esql} in {kib} to query and aggregate your data, create visualizations,
 and set up alerts.
 
 <<esql-elastic-security>>::
-Using {esql} in {elastic-sec} to investigate events in Timeline, create
+Learn how to use {esql} in {elastic-sec} to investigate events in Timeline, create
 detection rules, and build {esql} queries using Elastic AI Assistant.
 
 <<esql-multi-index>>::
-Using {esql} to query multiple indexes and resolve field type mismatches.
+Learn how to use {esql} to query multiple indexes and resolve field type mismatches.
 
 <<esql-cross-clusters>>::
-Using {esql} to query across multiple clusters.
+Learn how to use {esql} to query across multiple clusters.
 
 <<esql-task-management>>::
-Using the <<tasks,task management API>> to list and cancel {esql} queries.
+Learn how to use the <<tasks,task management API>> to list and cancel {esql} queries.
 
 include::esql-rest.asciidoc[]
+include::esql-for-search.asciidoc[] 
 include::esql-kibana.asciidoc[]
 include::esql-security-solution.asciidoc[]
 include::esql-multi-index.asciidoc[]

docs/reference/esql/index.asciidoc

@@ -22,21 +22,7 @@ a series of operations, where the output of one operation becomes the input for
 the next, enabling complex data transformations and analysis.
 
 [discrete]
-=== The {esql} Compute Engine
-
-{esql} is more than a language: it represents a significant investment in new
-compute capabilities within {es}. To achieve both the functional and performance
-requirements for {esql}, it was necessary to build an entirely new compute
-architecture. {esql} search, aggregation, and transformation functions are
-directly executed within Elasticsearch itself. Query expressions are not
-transpiled to Query DSL for execution. This approach allows {esql} to be
-extremely performant and versatile.
-
-The new {esql} execution engine was designed with performance in mind — it
-operates on blocks at a time instead of per row, targets vectorization and cache
-locality, and embraces specialization and multi-threading. It is a separate
-component from the existing Elasticsearch aggregation framework with different
-performance characteristics.
+=== Documentation organization
 
 The {esql} documentation is organized in these sections:
 
@@ -45,23 +31,33 @@ A tutorial to help you get started with {esql}.
 
 <<esql-language>>::
 
-Reference documentation for the <<esql-syntax,{esql} syntax>>,
-<<esql-commands,commands>>, and <<esql-functions-operators,functions and
-operators>>. Information about working with <<esql-metadata-fields,metadata
-fields>> and <<esql-multivalued-fields,multivalued fields>>. And guidance for
-<<esql-process-data-with-dissect-and-grok,data processing with DISSECT and
-GROK>> and <<esql-enrich-data,data enrichment with ENRICH>>.
+Reference documentation for the <<esql-syntax,{esql} syntax>>:
+
+* Reference for <<esql-commands,commands>>, and <<esql-functions-operators,functions and
+operators>>
+* How to work with <<esql-metadata-fields,metadata
+fields>> and <<esql-multivalued-fields,multivalued fields>>
+* How to work with
+<<esql-process-data-with-dissect-and-grok,DISSECT and
+GROK>>, <<esql-enrich-data,ENRICH>>, and <<esql-lookup-join,LOOKUP join>>
 
 <<esql-using>>::
-An overview of using the <<esql-rest>>, <<esql-kibana>>,
-<<esql-elastic-security>>, <<esql-cross-clusters>>, and <<esql-task-management>>.
+An overview of:
+* <<esql-rest,Using the {esql} rest API>>
+* <<esql-for-search>>
+* <<esql-kibana>>
+* <<esql-elastic-security>>
+* <<esql-cross-clusters>>
+* <<esql-task-management>>
 
 <<esql-limitations>>::
 The current limitations of {esql}.
 
 <<esql-examples>>::
 A few examples of what you can do with {esql}.
 
+
+
 include::esql-get-started.asciidoc[]
 
 include::esql-language.asciidoc[]
@@ -74,3 +70,20 @@ include::esql-examples.asciidoc[]
 
 :esql-tests!:
 :esql-specs!:
+
+[discrete]
+=== The {esql} Compute Engine
+
+{esql} is more than a language: it represents a significant investment in new
+compute capabilities within {es}. To achieve both the functional and performance
+requirements for {esql}, it was necessary to build an entirely new compute
+architecture. {esql} search, aggregation, and transformation functions are
+directly executed within Elasticsearch itself. Query expressions are not
+transpiled to Query DSL for execution. This approach allows {esql} to be
+extremely performant and versatile.
+
+The new {esql} execution engine was designed with performance in mind — it
+operates on blocks at a time instead of per row, targets vectorization and cache
+locality, and embraces specialization and multi-threading. It is a separate
+component from the existing Elasticsearch aggregation framework with different
+performance characteristics.