Merge branch 'main' into dont-wipe-xpack

Author: miguelgrinberg

docs/images/python-example.png

@@ -0,0 +1,106 @@
+# Querying
+
+The Python Elasticsearch client provides several ways to send queries to Elasticsearch. This document explains the details of how to construct and execute queries using the client. This document does not cover the DSL module.
+
+## From API URLs to function calls
+
+Elasticsearch APIs are grouped by namespaces.
+
+ * There's the global namespace, with APIs like the Search API (`GET _search`) or the Index API (`PUT /<target>/_doc/<_id>` and related endpoints). 
+ * Then there are all the other namespaces, such as:
+   * Indices with APIs like the Create index API (`PUT /my-index`),
+   * ES|QL with the Run an ES|QL query API (`POST /_async`),
+   * and so on.
+
+As a result, when you know which namespace and function you need, you can call the function. Assuming that `client` is an Elasticsearch instance, here is how you would call the examples from above:
+
+* Global namespace: `client.search(...)` and `client.index(...)`
+* Other namespaces:
+  * Indices: `client.indices.create(...)`
+  * ES|QL: `client.esql.query(...)`
+
+How can you figure out the namespace?
+
+* The [Elasticsearch API docs](https://www.elastic.co/docs/api/doc/elasticsearch/) can help, even though the tags it uses do not fully map to namespaces.
+* You can also use the client documentation, by:
+  * browsing the [Elasticsearch API Reference](https://elasticsearch-py.readthedocs.io/en/stable/api.html) page, or
+  * searching for your endpoint using [Read the Docs](https://elasticsearch-py.readthedocs.io/) search, which is powered by Elasticsearch!
+* Finally, for Elasticsearch 8.x, most examples in the [Elasticsearch guide](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) are also available in Python. (This is still a work in progress for Elasticsearch 9.x.) In the example below, `client.ingest.put_pipeline(...)` is the function that calls the "Create or update a pipeline" API.
+
+
+:::{image} ../images/python-example.png
+:alt: Python code example in the Elasticsearch guide
+:::
+ 
+## Parameters
+
+Now that you know which functions to call, the next step is parameters. To avoid ambiguity, the Python Elasticsearch client mandates keyword arguments. To give an example, let's look at the ["Create an index" API](https://elasticsearch-py.readthedocs.io/en/stable/api/indices.html#elasticsearch.client.IndicesClient.create). There's only one required parameter, `index`, so the minimal form looks like this:
+
+```python
+from elasticsearch import Elasticsearch
+
+client = Elasticsearch("http://localhost:9200", api_key="...")
+
+client.indices.create(index="my-index")
+```
+
+You can also use other parameters, including the first level of body parameters, such as:
+
+```python
+resp = client.indices.create(
+    index="logs",
+    aliases={"logs-alias": {}},
+    mappings={"name": {"type": "text"}},
+)
+print(resp)
+```
+
+In this case, the client will send to Elasticsearch the following JSON body:
+
+```console
+PUT /logs
+{
+    "aliases": {"logs-alias": {}},
+    "mappings": {"name": {"type": "text"}}
+}
+```
+
+## Unknown parameters or APIs
+
+Like other clients, the Python Elasticsearch client is generated from the [Elasticsearch specification](https://github.com/elastic/elasticsearch-specification). While we strive to keep it up to date, it is not (yet!) perfect, and sometimes body parameters are missing. In this case, you can specify the body directly, as follows:
+
+```python
+resp = client.indices.create(
+    index="logs",
+    body={
+        "aliases": {"logs-alias": {}},
+        "mappings": {"name": {"type": "text"}},
+        "missing_parameter": "foo",
+    }
+)
+print(resp)
+```
+
+In the event where an API is missing, you need to use the low-level `perform_request` function:
+
+```python
+resp = client.perform_request(
+    "PUT",
+    "/logs"
+    index="logs",
+    headers={"content-type": "application/json", "accept": "application/json"},
+    body={
+        "aliases": {"logs-alias": {}},
+        "mappings": {"name": {"type": "text"}},
+        "missing_parameter": "foo",
+    }
+)
+print(resp)
+```
+
+One benefit of this function is that it lets you use arbitrary headers, such as the `es-security-runas-user` header used to [impersonate users](https://www.elastic.co/guide/en/elasticsearch/reference/current/run-as-privilege.html).
+
+
+## Options
+
+You can specify options such as request timeouts or retries using the `.options()` API, see the [Configuration](./configuration.md) page for details.

docs/reference/querying.md

@@ -4,6 +4,7 @@ toc:
   - file: installation.md
   - file: connecting.md
   - file: configuration.md
+  - file: querying.md
   - file: async.md
   - file: integrations.md
     children:

docs/reference/toc.yml

@@ -87,6 +87,8 @@
     _rewrite_parameters,
     _stability_warning,
     client_node_configs,
+    is_requests_http_auth,
+    is_requests_node_class,
 )
 from .watcher import WatcherClient
 from .xpack import XPackClient
@@ -178,6 +180,7 @@ def __init__(
             t.Callable[[t.Dict[str, t.Any], NodeConfig], t.Optional[NodeConfig]]
         ] = None,
         meta_header: t.Union[DefaultType, bool] = DEFAULT,
+        http_auth: t.Union[DefaultType, t.Any] = DEFAULT,
         # Internal use only
         _transport: t.Optional[AsyncTransport] = None,
     ) -> None:
@@ -225,9 +228,26 @@ def __init__(
             sniff_callback = default_sniff_callback
 
         if _transport is None:
+            requests_session_auth = None
+            if http_auth is not None and http_auth is not DEFAULT:
+                if is_requests_http_auth(http_auth):
+                    # If we're using custom requests authentication
+                    # then we need to alert the user that they also
+                    # need to use 'node_class=requests'.
+                    if not is_requests_node_class(node_class):
+                        raise ValueError(
+                            "Using a custom 'requests.auth.AuthBase' class for "
+                            "'http_auth' must be used with node_class='requests'"
+                        )
+
+                    # Reset 'http_auth' to DEFAULT so it's not consumed below.
+                    requests_session_auth = http_auth
+                    http_auth = DEFAULT
+
             node_configs = client_node_configs(
                 hosts,
                 cloud_id=cloud_id,
+                requests_session_auth=requests_session_auth,
                 connections_per_node=connections_per_node,
                 http_compress=http_compress,
                 verify_certs=verify_certs,
@@ -314,6 +334,7 @@ def __init__(
             self._headers["x-opaque-id"] = opaque_id
         self._headers = resolve_auth_headers(
             self._headers,
+            http_auth=http_auth,
             api_key=api_key,
             basic_auth=basic_auth,
             bearer_auth=bearer_auth,
@@ -1468,7 +1489,7 @@ async def delete_by_query(
             If the request can target data streams, this argument determines whether
             wildcard expressions match hidden data streams. It supports comma-separated
             values, such as `open,hidden`.
-        :param from_: Starting offset (default: 0)
+        :param from_: Skips the specified number of documents.
         :param ignore_unavailable: If `false`, the request returns an error if it targets
             a missing or closed index.
         :param lenient: If `true`, format-based query failures (such as providing text
@@ -3307,7 +3328,8 @@ async def msearch(
             computationally expensive named queries on a large number of hits may add
             significant overhead.
         :param max_concurrent_searches: Maximum number of concurrent searches the multi
-            search API can execute.
+            search API can execute. Defaults to `max(1, (# of data nodes * min(search
+            thread pool size, 10)))`.
         :param max_concurrent_shard_requests: Maximum number of concurrent shard requests
             that each sub-search request executes per node.
         :param pre_filter_shard_size: Defines a threshold that enforces a pre-filter
@@ -3635,6 +3657,7 @@ async def open_point_in_time(
         human: t.Optional[bool] = None,
         ignore_unavailable: t.Optional[bool] = None,
         index_filter: t.Optional[t.Mapping[str, t.Any]] = None,
+        max_concurrent_shard_requests: t.Optional[int] = None,
         preference: t.Optional[str] = None,
         pretty: t.Optional[bool] = None,
         routing: t.Optional[str] = None,
@@ -3690,6 +3713,8 @@ async def open_point_in_time(
             a missing or closed index.
         :param index_filter: Filter indices if the provided query rewrites to `match_none`
             on every shard.
+        :param max_concurrent_shard_requests: Maximum number of concurrent shard requests
+            that each sub-search request executes per node.
         :param preference: The node or shard the operation should be performed on. By
             default, it is random.
         :param routing: A custom value that is used to route operations to a specific
@@ -3717,6 +3742,8 @@ async def open_point_in_time(
             __query["human"] = human
         if ignore_unavailable is not None:
             __query["ignore_unavailable"] = ignore_unavailable
+        if max_concurrent_shard_requests is not None:
+            __query["max_concurrent_shard_requests"] = max_concurrent_shard_requests
         if preference is not None:
             __query["preference"] = preference
         if pretty is not None:
@@ -4257,7 +4284,7 @@ async def render_search_template(
         human: t.Optional[bool] = None,
         params: t.Optional[t.Mapping[str, t.Any]] = None,
         pretty: t.Optional[bool] = None,
-        source: t.Optional[str] = None,
+        source: t.Optional[t.Union[str, t.Mapping[str, t.Any]]] = None,
         body: t.Optional[t.Dict[str, t.Any]] = None,
     ) -> ObjectApiResponse[t.Any]:
         """
@@ -4718,7 +4745,8 @@ async def search(
             limit the impact of the search on the cluster in order to limit the number
             of concurrent shard requests.
         :param min_score: The minimum `_score` for matching documents. Documents with
-            a lower `_score` are not included in the search results.
+            a lower `_score` are not included in search results and results collected
+            by aggregations.
         :param pit: Limit the search to a point in time (PIT). If you provide a PIT,
             you cannot specify an `<index>` in the request path.
         :param post_filter: Use the `post_filter` parameter to filter search results.
@@ -5661,7 +5689,7 @@ async def search_template(
         search_type: t.Optional[
             t.Union[str, t.Literal["dfs_query_then_fetch", "query_then_fetch"]]
         ] = None,
-        source: t.Optional[str] = None,
+        source: t.Optional[t.Union[str, t.Mapping[str, t.Any]]] = None,
         typed_keys: t.Optional[bool] = None,
         body: t.Optional[t.Dict[str, t.Any]] = None,
     ) -> ObjectApiResponse[t.Any]:
@@ -6399,7 +6427,7 @@ async def update_by_query(
             wildcard expressions match hidden data streams. It supports comma-separated
             values, such as `open,hidden`. Valid values are: `all`, `open`, `closed`,
             `hidden`, `none`.
-        :param from_: Starting offset (default: 0)
+        :param from_: Skips the specified number of documents.
         :param ignore_unavailable: If `false`, the request returns an error if it targets
             a missing or closed index.
         :param lenient: If `true`, format-based query failures (such as providing text

elasticsearch/_async/client/__init__.py

@@ -68,6 +68,7 @@
 
 def resolve_auth_headers(
     headers: Optional[Mapping[str, str]],
+    http_auth: Union[DefaultType, None, Tuple[str, str], str] = DEFAULT,
     api_key: Union[DefaultType, None, Tuple[str, str], str] = DEFAULT,
     basic_auth: Union[DefaultType, None, Tuple[str, str], str] = DEFAULT,
     bearer_auth: Union[DefaultType, None, str] = DEFAULT,
@@ -77,7 +78,32 @@ def resolve_auth_headers(
     elif not isinstance(headers, HttpHeaders):
         headers = HttpHeaders(headers)
 
+    resolved_http_auth = http_auth if http_auth is not DEFAULT else None
     resolved_basic_auth = basic_auth if basic_auth is not DEFAULT else None
+    if resolved_http_auth is not None:
+        if resolved_basic_auth is not None:
+            raise ValueError(
+                "Can't specify both 'http_auth' and 'basic_auth', "
+                "instead only specify 'basic_auth'"
+            )
+        if isinstance(http_auth, str) or (
+            isinstance(resolved_http_auth, (list, tuple))
+            and all(isinstance(x, str) for x in resolved_http_auth)
+        ):
+            resolved_basic_auth = resolved_http_auth
+        else:
+            raise TypeError(
+                "The deprecated 'http_auth' parameter must be either 'Tuple[str, str]' or 'str'. "
+                "Use either the 'basic_auth' parameter instead"
+            )
+
+        warnings.warn(
+            "The 'http_auth' parameter is deprecated. "
+            "Use 'basic_auth' or 'bearer_auth' parameters instead",
+            category=DeprecationWarning,
+            stacklevel=warn_stacklevel(),
+        )
+
     resolved_api_key = api_key if api_key is not DEFAULT else None
     resolved_bearer_auth = bearer_auth if bearer_auth is not DEFAULT else None
     if resolved_api_key or resolved_basic_auth or resolved_bearer_auth:

elasticsearch/_async/client/_base.py

@@ -401,7 +401,7 @@ async def submit(
             limit the impact of the search on the cluster in order to limit the number
             of concurrent shard requests
         :param min_score: Minimum _score for matching documents. Documents with a lower
-            _score are not included in the search results.
+            _score are not included in search results and results collected by aggregations.
         :param pit: Limits the search to a point in time (PIT). If you provide a PIT,
             you cannot specify an <index> in the request path.
         :param post_filter:

elasticsearch/_async/client/async_search.py

@@ -430,7 +430,7 @@ async def search(
         :param lenient:
         :param max_concurrent_shard_requests:
         :param min_score: Minimum _score for matching documents. Documents with a lower
-            _score are not included in the search results.
+            _score are not included in search results and results collected by aggregations.
         :param pit: Limits the search to a point in time (PIT). If you provide a PIT,
             you cannot specify an <index> in the request path.
         :param post_filter: