Skip to content

Commit 5bd4604

Browse files
miguelgrinbergmiguelgrinberg
committed
add readthedocs content to the guide
1 parent 2110df0 commit 5bd4604

File tree

4 files changed

+351
-6
lines changed

4 files changed

+351
-6
lines changed

‎docs/guide/async.asciidoc‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
[[async]]
2+
== Using asyncio with Elasticsearch
3+
4+
The `elasticsearch` package supports async/await with
5+
https://docs.python.org/3/library/asyncio.html[asyncio] and
6+
https://docs.aiohttp.org[aiohttp]. You can either install `aiohttp`
7+
directly or use the `[async]` extra:
8+
9+
[source,bash]
10+
----
11+
$ python -m pip install elasticsearch aiohttp
12+
13+
# - OR -
14+
15+
$ python -m pip install elasticsearch[async]
16+
----
17+
18+
=== Getting Started with Async
19+
20+
After installation all async API endpoints are available via
21+
`~elasticsearch.AsyncElasticsearch` and are used in the same way as
22+
other APIs, just with an extra `await`:
23+
24+
[source,python]
25+
----
26+
import asyncio
27+
from elasticsearch import AsyncElasticsearch
28+
29+
client = AsyncElasticsearch()
30+
31+
async def main():
32+
resp = await client.search(
33+
index="documents",
34+
body={"query": {"match_all": {}}},
35+
size=20,
36+
)
37+
print(resp)
38+
39+
loop = asyncio.get_event_loop()
40+
loop.run_until_complete(main())
41+
----
42+
43+
All APIs that are available under the sync client are also available
44+
under the async client.
45+
46+
=== ASGI Applications and Elastic APM
47+
48+
https://asgi.readthedocs.io[ASGI] (Asynchronous Server Gateway
49+
Interface) is a new way to serve Python web applications making use of
50+
async I/O to achieve better performance. Some examples of ASGI
51+
frameworks include FastAPI, Django 3.0+, and Starlette. If you're
52+
using one of these frameworks along with Elasticsearch then you should
53+
be using `~elasticsearch.AsyncElasticsearch` to avoid blocking the event
54+
loop with synchronous network calls for optimal performance.
55+
56+
https://www.elastic.co/guide/en/apm/agent/python/current/index.html[Elastic
57+
APM] also supports tracing of async Elasticsearch queries just the same
58+
as synchronous queries. For an example on how to configure
59+
`AsyncElasticsearch` with a popular ASGI framework
60+
https://fastapi.tiangolo.com/[FastAPI] and APM tracing there is a
61+
https://github.com/elastic/elasticsearch-py/tree/master/examples/fastapi-apm[pre-built
62+
example] in the `examples/fastapi-apm` directory.
63+
64+
=== Frequently Asked Questions
65+
66+
==== ValueError when initializing `AsyncElasticsearch`?
67+
68+
If when trying to use `AsyncElasticsearch` you receive
69+
`ValueError: You must have 'aiohttp' installed to use AiohttpHttpNode`
70+
you should ensure that you have `aiohttp` installed in your environment
71+
(check with `$ python -m pip freeze | grep aiohttp`). Otherwise,
72+
async support won't be available.
73+
74+
==== What about the `elasticsearch-async` package?
75+
76+
Previously asyncio was supported separately via the
77+
https://github.com/elastic/elasticsearch-py-async[elasticsearch-async]
78+
package. The `elasticsearch-async` package has been deprecated in favor
79+
of `AsyncElasticsearch` provided by the `elasticsearch` package in v7.8
80+
and onwards.
81+
82+
==== Receiving 'Unclosed client session / connector' warning?
83+
84+
This warning is created by `aiohttp` when an open HTTP connection is
85+
garbage collected. You'll typically run into this when closing your
86+
application. To resolve the issue ensure that
87+
`~elasticsearch.AsyncElasticsearch.close` is called before the
88+
`~elasticsearch.AsyncElasticsearch` instance is garbage collected.
89+
90+
For example if using FastAPI that might look like this:
91+
92+
[source,python]
93+
----
94+
import os
95+
from contextlib import asynccontextmanager
96+
97+
from fastapi import FastAPI
98+
from elasticsearch import AsyncElasticsearch
99+
100+
ELASTICSEARCH_URL = os.environ["ELASTICSEARCH_URL"]
101+
client = None
102+
103+
@asynccontextmanager
104+
async def lifespan(app: FastAPI):
105+
global client
106+
client = AsyncElasticsearch(ELASTICSEARCH_URL)
107+
yield
108+
await client.close()
109+
110+
app = FastAPI(lifespan=lifespan)
111+
112+
@app.get("/")
113+
async def main():
114+
return await client.info()
115+
----
116+
117+
You can run this example by saving it to `main.py` and executing
118+
`ELASTICSEARCH_URL=http://localhost:9200 uvicorn main:app`.
119+
120+
=== Async Helpers
121+
122+
Async variants of all helpers are available in `elasticsearch.helpers`
123+
and are all prefixed with `async_*`. You'll notice that these APIs
124+
are identical to the ones in the sync <<client-helpers>> documentation.
125+
126+
All async helpers that accept an iterator or generator also accept async
127+
iterators and async generators.

‎docs/guide/examples.asciidoc‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,3 +109,170 @@ method:
109109
----------------------------
110110
client.delete(index="test-index", id=1)
111111
----------------------------
112+
113+
[discrete]
114+
[[interactive]]
115+
=== Interactive examples
116+
117+
The https://github.com/elastic/elasticsearch-labs[elasticsearch-labs]
118+
repo contains interactive and executable
119+
https://github.com/elastic/elasticsearch-labs/tree/main/notebooks[Python
120+
notebooks], sample apps, and resources for testing out Elasticsearch,
121+
using the Python client. These examples are mainly focused on vector
122+
search, hybrid search and generative AI use cases, but you'll also find
123+
examples of basic operations like creating index mappings and performing
124+
lexical search.
125+
126+
==== Search notebooks
127+
128+
The
129+
https://github.com/elastic/elasticsearch-labs/tree/main/notebooks/search[Search]
130+
folder is a good place to start if you're new to Elasticsearch. This
131+
folder contains a number of notebooks that demonstrate the fundamentals
132+
of Elasticsearch, like indexing vectors, running lexical, semantic and
133+
_hybrid_ searches, and more.
134+
135+
The following notebooks are available:
136+
137+
[arabic, start=0]
138+
* https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/00-quick-start.ipynb[Quick
139+
start]
140+
* https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/01-keyword-querying-filtering.ipynb[Keyword&#44;
141+
querying&#44; filtering]
142+
* https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/02-hybrid-search.ipynb[Hybrid
143+
search]
144+
* https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/03-ELSER.ipynb[Semantic
145+
search with ELSER]
146+
* https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/04-multilingual.ipynb[Multilingual
147+
semantic search]
148+
* https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/05-query-rules.ipynb[Query
149+
rules]
150+
* https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/06-synonyms-api.ipynb[Synonyms
151+
API quick start]
152+
153+
Here's a brief overview of what you'll learn in each notebook.
154+
155+
===== Quick start
156+
157+
In the
158+
https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/00-quick-start.ipynb[00-quick-start.ipynb]
159+
notebook you'll learn how to:
160+
161+
* Use the Elasticsearch Python client for various operations.
162+
* Create and define an index for a sample dataset with
163+
`dense_vector` fields.
164+
* Transform book titles into embeddings using
165+
https://www.sbert.net[Sentence Transformers] and index them into
166+
Elasticsearch.
167+
* Perform k-nearest neighbors (knn) semantic searches.
168+
* Integrate traditional text-based search with semantic search, for a
169+
hybrid search system.
170+
* Use reciprocal rank fusion (RRF) to intelligently combine search
171+
results from different retrieval systems.
172+
173+
===== Keyword, querying, filtering
174+
175+
In the
176+
https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/01-keyword-querying-filtering.ipynb[01-keyword-querying-filtering.ipynb]
177+
notebook, you'll learn how to:
178+
179+
* Use
180+
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-filter-context.html[query
181+
and filter contexts] to search and filter documents in Elasticsearch.
182+
* Execute full-text searches with `match` and `multi-match` queries.
183+
* Query and filter documents based on `text`, `number`, `date`, or
184+
`boolean` values.
185+
* Run multi-field searches using the `multi-match` query.
186+
* Prioritize specific fields in the `multi-match` query for tailored
187+
results.
188+
189+
===== Hybrid search
190+
191+
In the
192+
https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/02-hybrid-search.ipynb[02-hybrid-search.ipynb]
193+
notebook, you'll learn how to:
194+
195+
* Combine results of traditional text-based search with semantic search,
196+
for a hybrid search system.
197+
* Transform fields in the sample dataset into embeddings using the
198+
Sentence Transformer model and index them into Elasticsearch.
199+
* Use the
200+
https://www.elastic.co/guide/en/elasticsearch/reference/current/rrf.html#rrf-api[RRF
201+
API] to combine the results of a `match` query and a `kNN` semantic
202+
search.
203+
* Walk through a super simple toy example that demonstrates, step by
204+
step, how RRF ranking works.
205+
206+
===== Semantic search with ELSER
207+
208+
In the
209+
https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/03-ELSER.ipynb[03-ELSER.ipynb]
210+
notebook, you'll learn how to:
211+
212+
* Use the Elastic Learned Sparse Encoder (ELSER) for text
213+
expansion-powered semantic search, out of the box — without training,
214+
fine-tuning, or embeddings generation.
215+
* Download and deploy the ELSER model in your Elastic environment.
216+
* Create an Elasticsearch index named [.title-ref]#search-movies# with
217+
specific mappings and index a dataset of movie descriptions.
218+
* Create an ingest pipeline containing an inference processor for ELSER
219+
model execution.
220+
* Reindex the data from [.title-ref]#search-movies# into another index,
221+
[.title-ref]#elser-movies#, using the ELSER pipeline for text expansion.
222+
* Observe the results of running the documents through the model by
223+
inspecting the additional terms it adds to documents, which enhance
224+
searchability.
225+
* Perform simple keyword searches on the [.title-ref]#elser-movies#
226+
index to assess the impact of ELSER's text expansion.
227+
* Execute ELSER-powered semantic searches using the `text_expansion`
228+
query.
229+
230+
===== Multilingual semantic search
231+
232+
In the
233+
https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/04-multilingual.ipynb[04-multilingual.ipynb]
234+
notebook, you'll learn how to:
235+
236+
* Use a multilingual embedding model for semantic search across
237+
languages.
238+
* Transform fields in the sample dataset into embeddings using the
239+
Sentence Transformer model and index them into Elasticsearch.
240+
* Use filtering with a `kNN` semantic search.
241+
* Walk through a super simple toy example that demonstrates, step by
242+
step, how multilingual search works across languages, and within
243+
non-English languages.
244+
245+
===== Query rules
246+
247+
In the
248+
https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/05-query-rules.ipynb[05-query-rules.ipynb]
249+
notebook, you'll learn how to:
250+
251+
* Use the query rules management APIs to create and edit promotional
252+
rules based on contextual queries.
253+
* Apply these query rules by using the `rule_query` in Query DSL.
254+
255+
===== Synonyms API quick start
256+
257+
In the
258+
https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/06-synonyms-api.ipynb[06-synonyms-api.ipynb]
259+
notebook, you'll learn how to:
260+
261+
* Use the synonyms management API to create a synonyms set to enhance
262+
your search recall.
263+
* Configure an index to use search-time synonyms.
264+
* Update synonyms in real time.
265+
* Run queries that are enhanced by synonyms.
266+
267+
==== Other notebooks
268+
269+
* https://github.com/elastic/elasticsearch-labs/tree/main/notebooks/generative-ai[Generative
270+
AI]. Notebooks that demonstrate various use cases for Elasticsearch as
271+
the retrieval engine and vector store for LLM-powered applications.
272+
* https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/integrations[Integrations].
273+
Notebooks that demonstrate how to integrate popular services and
274+
projects with Elasticsearch, including OpenAI, Hugging Face, and
275+
LlamaIndex
276+
* https://github.com/elastic/elasticsearch-labs/tree/main/notebooks/langchain[Langchain].
277+
Notebooks that demonstrate how to integrate Elastic with LangChain, a
278+
framework for developing applications powered by language models.

‎docs/guide/getting-started.asciidoc‎

Lines changed: 53 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -70,11 +70,33 @@ This is how you create the `my_index` index:
7070
client.indices.create(index="my_index")
7171
----
7272

73+
Optionally, you can first define the expected types of your features with a
74+
custom mapping.
75+
76+
[source,py]
77+
----
78+
mappings = {
79+
"properties": {
80+
"foo": {"type": "text"},
81+
"bar": {
82+
"type": "text",
83+
"fields": {
84+
"keyword": {
85+
"type": "keyword",
86+
"ignore_above": 256,
87+
}
88+
},
89+
},
90+
}
91+
}
92+
93+
client.indices.create(index="my_index", mappings=mappings)
94+
----
7395

7496
[discrete]
7597
==== Indexing documents
7698

77-
This is a simple way of indexing a document:
99+
This indexes a document with the index API:
78100

79101
[source,py]
80102
----
@@ -88,6 +110,28 @@ client.index(
88110
)
89111
----
90112

113+
You can also index multiple documents at once with the bulk helper function:
114+
115+
[source,py]
116+
----
117+
from elasticsearch import helpers
118+
119+
def generate_docs():
120+
for i in range(10):
121+
yield {
122+
"_index": "my_index",
123+
"foo": f"foo {i}",
124+
"bar": "bar",
125+
}
126+
127+
helpers.bulk(client, generate_docs())
128+
----
129+
130+
These helpers are the recommended way to perform bulk ingestion. While it is
131+
also possible to perform bulk ingestion using `client.bulk` directly, the
132+
helpers handle retries, ingesting chunk by chunk and more. See the
133+
<<client-helpers>> page for more details.
134+
91135

92136
[discrete]
93137
==== Getting documents
@@ -122,10 +166,14 @@ This is how you can update a document, for example to add a new field:
122166

123167
[source,py]
124168
----
125-
client.update(index="my_index", id="my_document_id", doc={
126-
"foo": "bar",
127-
"new_field": "new value",
128-
})
169+
client.update(
170+
index="my_index",
171+
id="my_document_id",
172+
doc={
173+
"foo": "bar",
174+
"new_field": "new value",
175+
}
176+
)
129177
----
130178

131179

‎docs/guide/index-custom-title-page.html‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -136,7 +136,10 @@ <h4 class="mt-3">
136136
<a href="integrations.html">Integrations</a>
137137
</li>
138138
<li>
139-
<a href="elasticsearch-dsl.html">Elasticsearch Python DSL</a>
139+
<a href="async.html">Use with asyncio</a>
140+
</li>
141+
<li>
142+
<a href="elasticsearch-dsl.html">Python DSL</a>
140143
</li>
141144
<li>
142145
<a href="client-helpers.html">Client helpers</a>

0 commit comments

Comments
 (0)