Skip to content

Commit 647753a

Browse files
miguelgrinbergmiguelgrinberg
committed
add readthedocs content to the guide
1 parent 2110df0 commit 647753a

File tree

4 files changed

+360
-6
lines changed

4 files changed

+360
-6
lines changed

‎docs/guide/async.asciidoc‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
[[async]]
2+
== Using asyncio with Elasticsearch
3+
4+
The `elasticsearch` package supports async/await with
5+
https://docs.python.org/3/library/asyncio.html[asyncio] and
6+
https://docs.aiohttp.org[aiohttp]. You can either install `aiohttp`
7+
directly or use the `[async]` extra:
8+
9+
[source,bash]
10+
----
11+
$ python -m pip install elasticsearch aiohttp
12+
13+
# - OR -
14+
15+
$ python -m pip install elasticsearch[async]
16+
----
17+
18+
=== Getting Started with Async
19+
20+
After installation all async API endpoints are available via
21+
`~elasticsearch.AsyncElasticsearch` and are used in the same way as
22+
other APIs, just with an extra `await`:
23+
24+
[source,python]
25+
----
26+
import asyncio
27+
from elasticsearch import AsyncElasticsearch
28+
29+
client = AsyncElasticsearch()
30+
31+
async def main():
32+
resp = await client.search(
33+
index="documents",
34+
body={"query": {"match_all": {}}},
35+
size=20,
36+
)
37+
print(resp)
38+
39+
loop = asyncio.get_event_loop()
40+
loop.run_until_complete(main())
41+
----
42+
43+
All APIs that are available under the sync client are also available
44+
under the async client.
45+
46+
=== ASGI Applications and Elastic APM
47+
48+
https://asgi.readthedocs.io[ASGI] (Asynchronous Server Gateway
49+
Interface) is a new way to serve Python web applications making use of
50+
async I/O to achieve better performance. Some examples of ASGI
51+
frameworks include FastAPI, Django 3.0+, and Starlette. If you're
52+
using one of these frameworks along with Elasticsearch then you should
53+
be using `~elasticsearch.AsyncElasticsearch` to avoid blocking the event
54+
loop with synchronous network calls for optimal performance.
55+
56+
https://www.elastic.co/guide/en/apm/agent/python/current/index.html[Elastic
57+
APM] also supports tracing of async Elasticsearch queries just the same
58+
as synchronous queries. For an example on how to configure
59+
`AsyncElasticsearch` with a popular ASGI framework
60+
https://fastapi.tiangolo.com/[FastAPI] and APM tracing there is a
61+
https://github.com/elastic/elasticsearch-py/tree/master/examples/fastapi-apm[pre-built
62+
example] in the `examples/fastapi-apm` directory.
63+
64+
=== Frequently Asked Questions
65+
66+
==== ValueError when initializing `AsyncElasticsearch`?
67+
68+
If when trying to use `AsyncElasticsearch` you receive
69+
`ValueError: You must have 'aiohttp' installed to use AiohttpHttpNode`
70+
you should ensure that you have `aiohttp` installed in your environment
71+
(check with `$ python -m pip freeze | grep aiohttp`). Otherwise,
72+
async support won't be available.
73+
74+
==== What about the `elasticsearch-async` package?
75+
76+
Previously asyncio was supported separately via the
77+
https://github.com/elastic/elasticsearch-py-async[elasticsearch-async]
78+
package. The `elasticsearch-async` package has been deprecated in favor
79+
of `AsyncElasticsearch` provided by the `elasticsearch` package in v7.8
80+
and onwards.
81+
82+
==== Receiving 'Unclosed client session / connector' warning?
83+
84+
This warning is created by `aiohttp` when an open HTTP connection is
85+
garbage collected. You'll typically run into this when closing your
86+
application. To resolve the issue ensure that
87+
`~elasticsearch.AsyncElasticsearch.close` is called before the
88+
`~elasticsearch.AsyncElasticsearch` instance is garbage collected.
89+
90+
For example if using FastAPI that might look like this:
91+
92+
[source,python]
93+
----
94+
import os
95+
from contextlib import asynccontextmanager
96+
97+
from fastapi import FastAPI
98+
from elasticsearch import AsyncElasticsearch
99+
100+
ELASTICSEARCH_URL = os.environ["ELASTICSEARCH_URL"]
101+
client = None
102+
103+
@asynccontextmanager
104+
async def lifespan(app: FastAPI):
105+
global client
106+
client = AsyncElasticsearch(ELASTICSEARCH_URL)
107+
yield
108+
await client.close()
109+
110+
app = FastAPI(lifespan=lifespan)
111+
112+
@app.get("/")
113+
async def main():
114+
return await client.info()
115+
----
116+
117+
You can run this example by saving it to `main.py` and executing
118+
`ELASTICSEARCH_URL=http://localhost:9200 uvicorn main:app`.
119+
120+
=== Async Helpers
121+
122+
Async variants of all helpers are available in `elasticsearch.helpers`
123+
and are all prefixed with `async_*`. You'll notice that these APIs
124+
are identical to the ones in the sync <<client-helpers>> documentation.
125+
126+
All async helpers that accept an iterator or generator also accept async
127+
iterators and async generators.

‎docs/guide/examples.asciidoc‎

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,3 +109,179 @@ method:
109109
----------------------------
110110
client.delete(index="test-index", id=1)
111111
----------------------------
112+
113+
[discrete]
114+
[[ex-interactive]]
115+
=== Interactive examples
116+
117+
The https://github.com/elastic/elasticsearch-labs[elasticsearch-labs]
118+
repo contains interactive and executable
119+
https://github.com/elastic/elasticsearch-labs/tree/main/notebooks[Python
120+
notebooks], sample apps, and resources for testing out Elasticsearch,
121+
using the Python client. These examples are mainly focused on vector
122+
search, hybrid search and generative AI use cases, but you'll also find
123+
examples of basic operations like creating index mappings and performing
124+
lexical search.
125+
126+
[discrete]
127+
==== Search notebooks
128+
129+
The
130+
https://github.com/elastic/elasticsearch-labs/tree/main/notebooks/search[Search]
131+
folder is a good place to start if you're new to Elasticsearch. This
132+
folder contains a number of notebooks that demonstrate the fundamentals
133+
of Elasticsearch, like indexing vectors, running lexical, semantic and
134+
_hybrid_ searches, and more.
135+
136+
The following notebooks are available:
137+
138+
[arabic, start=0]
139+
* https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/00-quick-start.ipynb[Quick
140+
start]
141+
* https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/01-keyword-querying-filtering.ipynb[Keyword&#44;
142+
querying&#44; filtering]
143+
* https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/02-hybrid-search.ipynb[Hybrid
144+
search]
145+
* https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/03-ELSER.ipynb[Semantic
146+
search with ELSER]
147+
* https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/04-multilingual.ipynb[Multilingual
148+
semantic search]
149+
* https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/05-query-rules.ipynb[Query
150+
rules]
151+
* https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/06-synonyms-api.ipynb[Synonyms
152+
API quick start]
153+
154+
Here's a brief overview of what you'll learn in each notebook.
155+
156+
[discrete]
157+
===== Quick start
158+
159+
In the
160+
https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/00-quick-start.ipynb[00-quick-start.ipynb]
161+
notebook you'll learn how to:
162+
163+
* Use the Elasticsearch Python client for various operations.
164+
* Create and define an index for a sample dataset with
165+
`dense_vector` fields.
166+
* Transform book titles into embeddings using
167+
https://www.sbert.net[Sentence Transformers] and index them into
168+
Elasticsearch.
169+
* Perform k-nearest neighbors (knn) semantic searches.
170+
* Integrate traditional text-based search with semantic search, for a
171+
hybrid search system.
172+
* Use reciprocal rank fusion (RRF) to intelligently combine search
173+
results from different retrieval systems.
174+
175+
[discrete]
176+
===== Keyword, querying, filtering
177+
178+
In the
179+
https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/01-keyword-querying-filtering.ipynb[01-keyword-querying-filtering.ipynb]
180+
notebook, you'll learn how to:
181+
182+
* Use
183+
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-filter-context.html[query
184+
and filter contexts] to search and filter documents in Elasticsearch.
185+
* Execute full-text searches with `match` and `multi-match` queries.
186+
* Query and filter documents based on `text`, `number`, `date`, or
187+
`boolean` values.
188+
* Run multi-field searches using the `multi-match` query.
189+
* Prioritize specific fields in the `multi-match` query for tailored
190+
results.
191+
192+
[discrete]
193+
===== Hybrid search
194+
195+
In the
196+
https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/02-hybrid-search.ipynb[02-hybrid-search.ipynb]
197+
notebook, you'll learn how to:
198+
199+
* Combine results of traditional text-based search with semantic search,
200+
for a hybrid search system.
201+
* Transform fields in the sample dataset into embeddings using the
202+
Sentence Transformer model and index them into Elasticsearch.
203+
* Use the
204+
https://www.elastic.co/guide/en/elasticsearch/reference/current/rrf.html#rrf-api[RRF
205+
API] to combine the results of a `match` query and a `kNN` semantic
206+
search.
207+
* Walk through a super simple toy example that demonstrates, step by
208+
step, how RRF ranking works.
209+
210+
[discrete]
211+
===== Semantic search with ELSER
212+
213+
In the
214+
https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/03-ELSER.ipynb[03-ELSER.ipynb]
215+
notebook, you'll learn how to:
216+
217+
* Use the Elastic Learned Sparse Encoder (ELSER) for text
218+
expansion-powered semantic search, out of the box — without training,
219+
fine-tuning, or embeddings generation.
220+
* Download and deploy the ELSER model in your Elastic environment.
221+
* Create an Elasticsearch index named [.title-ref]#search-movies# with
222+
specific mappings and index a dataset of movie descriptions.
223+
* Create an ingest pipeline containing an inference processor for ELSER
224+
model execution.
225+
* Reindex the data from [.title-ref]#search-movies# into another index,
226+
[.title-ref]#elser-movies#, using the ELSER pipeline for text expansion.
227+
* Observe the results of running the documents through the model by
228+
inspecting the additional terms it adds to documents, which enhance
229+
searchability.
230+
* Perform simple keyword searches on the [.title-ref]#elser-movies#
231+
index to assess the impact of ELSER's text expansion.
232+
* Execute ELSER-powered semantic searches using the `text_expansion`
233+
query.
234+
235+
[discrete]
236+
===== Multilingual semantic search
237+
238+
In the
239+
https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/04-multilingual.ipynb[04-multilingual.ipynb]
240+
notebook, you'll learn how to:
241+
242+
* Use a multilingual embedding model for semantic search across
243+
languages.
244+
* Transform fields in the sample dataset into embeddings using the
245+
Sentence Transformer model and index them into Elasticsearch.
246+
* Use filtering with a `kNN` semantic search.
247+
* Walk through a super simple toy example that demonstrates, step by
248+
step, how multilingual search works across languages, and within
249+
non-English languages.
250+
251+
[discrete]
252+
===== Query rules
253+
254+
In the
255+
https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/05-query-rules.ipynb[05-query-rules.ipynb]
256+
notebook, you'll learn how to:
257+
258+
* Use the query rules management APIs to create and edit promotional
259+
rules based on contextual queries.
260+
* Apply these query rules by using the `rule_query` in Query DSL.
261+
262+
[discrete]
263+
===== Synonyms API quick start
264+
265+
In the
266+
https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/06-synonyms-api.ipynb[06-synonyms-api.ipynb]
267+
notebook, you'll learn how to:
268+
269+
* Use the synonyms management API to create a synonyms set to enhance
270+
your search recall.
271+
* Configure an index to use search-time synonyms.
272+
* Update synonyms in real time.
273+
* Run queries that are enhanced by synonyms.
274+
275+
[discrete]
276+
==== Other notebooks
277+
278+
* https://github.com/elastic/elasticsearch-labs/tree/main/notebooks/generative-ai[Generative
279+
AI]. Notebooks that demonstrate various use cases for Elasticsearch as
280+
the retrieval engine and vector store for LLM-powered applications.
281+
* https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/integrations[Integrations].
282+
Notebooks that demonstrate how to integrate popular services and
283+
projects with Elasticsearch, including OpenAI, Hugging Face, and
284+
LlamaIndex
285+
* https://github.com/elastic/elasticsearch-labs/tree/main/notebooks/langchain[Langchain].
286+
Notebooks that demonstrate how to integrate Elastic with LangChain, a
287+
framework for developing applications powered by language models.

‎docs/guide/getting-started.asciidoc‎

Lines changed: 53 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -70,11 +70,33 @@ This is how you create the `my_index` index:
7070
client.indices.create(index="my_index")
7171
----
7272

73+
Optionally, you can first define the expected types of your features with a
74+
custom mapping.
75+
76+
[source,py]
77+
----
78+
mappings = {
79+
"properties": {
80+
"foo": {"type": "text"},
81+
"bar": {
82+
"type": "text",
83+
"fields": {
84+
"keyword": {
85+
"type": "keyword",
86+
"ignore_above": 256,
87+
}
88+
},
89+
},
90+
}
91+
}
92+
93+
client.indices.create(index="my_index", mappings=mappings)
94+
----
7395

7496
[discrete]
7597
==== Indexing documents
7698

77-
This is a simple way of indexing a document:
99+
This indexes a document with the index API:
78100

79101
[source,py]
80102
----
@@ -88,6 +110,28 @@ client.index(
88110
)
89111
----
90112

113+
You can also index multiple documents at once with the bulk helper function:
114+
115+
[source,py]
116+
----
117+
from elasticsearch import helpers
118+
119+
def generate_docs():
120+
for i in range(10):
121+
yield {
122+
"_index": "my_index",
123+
"foo": f"foo {i}",
124+
"bar": "bar",
125+
}
126+
127+
helpers.bulk(client, generate_docs())
128+
----
129+
130+
These helpers are the recommended way to perform bulk ingestion. While it is
131+
also possible to perform bulk ingestion using `client.bulk` directly, the
132+
helpers handle retries, ingesting chunk by chunk and more. See the
133+
<<client-helpers>> page for more details.
134+
91135

92136
[discrete]
93137
==== Getting documents
@@ -122,10 +166,14 @@ This is how you can update a document, for example to add a new field:
122166

123167
[source,py]
124168
----
125-
client.update(index="my_index", id="my_document_id", doc={
126-
"foo": "bar",
127-
"new_field": "new value",
128-
})
169+
client.update(
170+
index="my_index",
171+
id="my_document_id",
172+
doc={
173+
"foo": "bar",
174+
"new_field": "new value",
175+
}
176+
)
129177
----
130178

131179

0 commit comments

Comments
 (0)