Skip to content

Commit 0ef8120

Browse files
lcawllcawl
committed
Add getting started with AI search
1 parent 0af0a5a commit 0ef8120

File tree

2 files changed

+186
-0
lines changed

2 files changed

+186
-0
lines changed

‎solutions/search/serverless-elasticsearch-get-started-semantic.md‎

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,185 @@
1+
---
2+
navigation_title: Get started with semantic search
3+
applies_to:
4+
serverless:
5+
products:
6+
- id: cloud-serverless
7+
---
8+
# Build an AI-powered search experience in {{es-serverless}}
9+
10+
<!--
11+
As you ramp up on Elastic, you’ll use the Elasticsearch Relevance Engine™ (ESRE), designed to power AI search applications. With ESRE, you can take advantage of a suite of developer tools including Elastic’s textual search, vector database, and our proprietary transformer model for semantic search.
12+
-->
13+
14+
Elastic offers a variety of search techniques, starting with BM25, the industry standard for textual search. It provides precise matching for specific searches, matching exact keywords, and it improves with tuning.
15+
16+
<!--
17+
As you get started on vector search, keep in mind there are two forms of vector search: “dense” (aka, kNN vector search) and “sparse."
18+
TBD: Which type is implemented when you use semantic_text field?
19+
-->
20+
21+
Elastic also offers an out-of-the-box Learned Sparse Encoder model for semantic search.
22+
This model outperforms on a variety of data sets, such as financial data, weather records, and question-answer pairs, among others.
23+
The model is built to provide great relevance across domains, without the need for additional fine tuning.
24+
25+
<!--
26+
Check out this interactive demo to see how search results are more relevant when you test Elastic's Learned Sparse Encoder model against Elastic’s textual BM25 algorithm.
27+
28+
In addition, Elastic also supports dense vectors to implement similarity search on unstructured data beyond text, such as videos, images, and audio.
29+
-->
30+
31+
The advantage of semantic search and vector search is that these technologies allow you to use intuitive language in your search queries.
32+
For example, if you want to search for workplace guidelines on a second income, you could search for "side hustle", which is not a term you're likely to see in a formal HR document.
33+
34+
## Setup
35+
36+
In this guide, we'll demonstrate how to ingest data with the Elastic web crawler then try out some queries.
37+
If you want to play along, follow these setup steps. Otherwise, you can jump ahead to the query examples.
38+
39+
1. Optional: Create an {{ecloud}} project, in particular an {{es-serverless}} project that is optimized for vectors.
40+
If you want to perform the steps in this guide you'll need:
41+
42+
* The {{es}} URL: the endpoint to which you will send your data
43+
* The API Key: the easiest of the authentication methods
44+
% TBD: Is this mandatory? Clarify value.
45+
46+
1. Optional: Add data.
47+
In this guide, the data is derived from a live website, the [{{es}} Labs](https://www.elastic.co/search-labs).
48+
49+
1. Let’s create an {{es}} index named `elasticsearch-labs-blog`.
50+
1. Create mappings for a text field and a semantic text field.
51+
<!--
52+
Now before writing data to the index, let’s do a small configuration to ensure you have semantic search right from the start. Click on Mappings and + Add field, create a text field called `body`, this is where the crawler will put the content of the web pages it reads. Next, add a semantic text type field, let’s give it a very creative name: `semantic_text`.
53+
54+
By using both a text field and semantic_text field, the process combines the strengths of traditional keyword search and advanced semantic search. This hybrid search provides comprehensive search capabilities, ensuring that users can find relevant information based on both the raw text and its underlying meaning.
55+
-->
56+
1. Ingest data with the Elastic Open Web Crawler.
57+
<!--
58+
You will need Docker to use the Open Web Crawler. Here is a simple config file, it tells the crawler to read the https://www.elastic.co/search-labs blog and write it to the `elasticsearch-labs-blog` index at `elasticsearch.host` using the `elasticsearch.api_key`... then create a `docker-compose.yml` file. Start the service with `docker-compose up -d` then start the crawling process with `docker-compose exec -it crawler bin/crawler crawl /app/config/crawler-config-blog.yml`. After a few minutes you should have the whole {{es}} labs blogs indexed.
59+
60+
What just happened? The blog content was indexed to the `body` field, then this content was transformed into a sparse vector inside the `semantic_text` field. This transformation involved two main steps. First, the content was divided into smaller, manageable chunks to ensure that the text is broken down into meaningful segments, which can be more effectively processed and searched. Next, each chunk of text was transformed into a sparse vector representation using text expansion techniques. This step leverages ELSER (Elastic Search Engine for Relevance) to convert the text into a format that captures the semantic meaning, enabling more accurate and relevant search results.
61+
-->
62+
63+
## Test a keyword query
64+
65+
% TBD Move below the semantic search example? Assumes people are already familiar with old search options?
66+
67+
Now it’s time to search for the information you’re looking for.
68+
If you’re a developer who’s implementing search for a web application, you can use the Console/Dev Tools to test and refine search results from your indexed data.
69+
70+
Let’s start with a simple `multi_match query`, which will match the text against the “title” and “body” fields.
71+
Since this is a classic lexical search (not semantic yet) the results of a query like “how to implement multilingual search” will match the words you are providing.
72+
73+
```console
74+
GET elasticsearch-labs-blog/_search
75+
{
76+
"_source": ["title"],
77+
"query": {
78+
"multi_match": {
79+
"query": "how to implement multilingual search",
80+
"fields": ["title","body"]
81+
}
82+
}
83+
}
84+
```
85+
86+
In this case the top 5 matches are good, but not great.
87+
88+
```txt
89+
"Multilingual vector search with the E5 embedding model"
90+
"Scalar quantization optimized for vector databases"
91+
"How to migrate your Ruby app from OpenSearch to Elasticsearch"
92+
"How to search languages with compound words"
93+
"How To"
94+
```
95+
96+
## Test a semantic search query
97+
98+
Now try the same but with a semantic query, it will translate the text “how to implement multilingual search?” automatically into a vector representation and perform the query against the `semantic_text` field.
99+
100+
```console
101+
GET elasticsearch-labs-blog/_search
102+
{
103+
"_source": ["title"],
104+
"query": {
105+
"semantic": {
106+
"field": "semantic_text",
107+
"query": "how to implement multilingual search?"
108+
}
109+
}
110+
}
111+
```
112+
113+
The 5 top results you get back from this semantic search look much better.
114+
115+
```txt
116+
"Multilingual vector search with the E5 embedding model"
117+
"How to search languages with compound words"
118+
"Dataset translation with LangChain, Python & Vector Database for multilingual insights"
119+
"Building multilingual RAG with Elastic and Mistral"
120+
"Agentic RAG with Elasticsearch & Langchain"
121+
```
122+
123+
## Test a hybrid search query
124+
125+
A more advanced example using Reciprocal Rank Fusion (RRF) is a technique used in hybrid retrieval systems to improve the relevance of search results.
126+
It combines different retrieval methods, such as lexical (traditional) search and semantic search, to enhance the overall search performance.
127+
128+
By leveraging RRF, the query ensures that the final list of documents is a balanced mix of the top results from both retrieval methods, thereby improving the overall relevance and diversity of the search results.
129+
This fusion technique mitigates the limitations of individual retrieval methods, providing a more comprehensive and accurate set of results.
130+
131+
% TBD: Are there reasons for not using hybrid search? e.g. additional storage space, speed
132+
133+
```console
134+
GET elasticsearch-labs-blog/_search
135+
{
136+
"_source": [
137+
"title"
138+
],
139+
"retriever": {
140+
"rrf": {
141+
"retrievers": [
142+
{
143+
"standard": {
144+
"query": {
145+
"multi_match": {
146+
"fields": ["title","body"],
147+
"query": "how to implement multilingual search"
148+
}
149+
}
150+
}
151+
},
152+
{
153+
"standard": {
154+
"query": {
155+
"semantic": {
156+
"field": "semantic_text",
157+
"query": "how to implement multilingual search"
158+
}
159+
}
160+
}
161+
}
162+
]
163+
}
164+
}
165+
}
166+
```
167+
168+
The top 5 hits with hybrid search contain very good results, all highly relevant to how you can implement a multilingual search with Elasticsearch:
169+
170+
```txt
171+
"Multilingual vector search with the E5 embedding model"
172+
"How to search languages with compound words"
173+
"Dataset translation with LangChain, Python & Vector Database for multilingual insights"
174+
"Building multilingual RAG with Elastic and Mistral"
175+
"Evaluating scalar quantization in Elasticsearch"
176+
```
177+
178+
## Next steps
179+
180+
Thanks for taking the time to set up semantic search for your data with {{ecloud}}.
181+
182+
<!--
183+
Ready to get started? Spin up a free 14-day trial on Elastic Cloud or try out these 15 minute hands-on learning on Search AI 101.
184+
TBD: Link to other quickstarts or to the deeper semantic search options.
185+
-->

‎solutions/toc.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ toc:
88
- file: search/run-elasticsearch-locally.md
99
- file: search/serverless-elasticsearch-get-started.md
1010
- file: search/search-connection-details.md
11+
- file: search/serverless-elasticsearch-get-started-semantic.md
1112
- file: search/api-quickstarts.md
1213
children:
1314
- file: search/elasticsearch-basics-quickstart.md

0 commit comments

Comments
 (0)