You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Review the changes, fixes, and more to {{serverless-full}}.
11
11
12
-
For {{serverless-full}} API changes, refer to [APIs Changelog](https://www.elastic.co/docs/api/changes).
13
-
14
12
## July 15, 2025 [serverless-changelog-07152025]
13
+
14
+
### Features and enhancements [serverless-changelog-07152025-features-enhancements]
15
15
* {{serverless-full}} is now available in two new Amazon Web Services [regions](/deploy-manage/deploy/elastic-cloud/regions.md): `eu-central-1` (Frankfurt) and `us-east-2` (Ohio).
16
+
* Adds the ability to add tags from the **Agent details** page [#225433]({{kib-pull}}225433)
17
+
* Adds a **Profiles inspector** to Discover [#222999]({{kib-pull}}222999)
18
+
* Displays a callout about new rules in Elastic Observability Serverless **Metrics**, **Logs**, and **Inventory** rule types [#224387]({{kib-pull}}224387)
19
+
* Adds a manual test for bulk import functionality in Elastic Observability Serverless [#225497]({{kib-pull}}225497)
20
+
* Groups vulnerabilities by resource and cloud account using IDs instead of names in Elastic Security Serverless [#225492]({{kib-pull}}225492)
21
+
* Updates the default Gemini model in Elastic Security Serverless [#225917]({{kib-pull}}225917)
22
+
* Streamlines the side navigation in Elasticsearch Serverless [#225709]({{kib-pull}}225709)
23
+
24
+
### Fixes [serverless-changelog-07152025-fixes]
25
+
* Fixes an issue where reports timed out and failed with an invalid header error [#225919]({{kib-pull}}225919)
26
+
* Ensures "Values from a query" options refresh when reloading dashboards [#225101]({{kib-pull}}225101)
27
+
* Removes warnings related to kebab-case naming [#226114]({{kib-pull}}226114)
28
+
* Prevents custom titles from being overwritten in Lens embeddables after reload [#225664]({{kib-pull}}225664)
29
+
* Prevents adhoc data views from being recommended in **Controls**[#225705]({{kib-pull}}225705)
30
+
* Hides the **Select all** checkbox in single-select controls [#226311]({{kib-pull}}226311)
31
+
* Fixes a bug where edited queries were overwritten when a request completed [#224671]({{kib-pull}}224671)
32
+
* Keeps the selected document stable when resizing the flyout with keyboard controls [#225594]({{kib-pull}}225594)
33
+
* Ensures suggested dashboards only appear for custom threshold alerts in Elastic Observability Serverless [#224458]({{kib-pull}}224458)
Copy file name to clipboardExpand all lines: release-notes/intro/index.md
+3-5Lines changed: 3 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,13 +10,11 @@ products:
10
10
11
11
Learn about the latest changes, issues, fixes, and deprecations for Elastic releases, including {{stack}} 9.0 and {{serverless-full}}, by reviewing the release notes.
12
12
13
-
To view the changes in previous versions, go to [Release docs](https://www.elastic.co/guide/en/starting-with-the-elasticsearch-platform-and-its-solutions/8.x/new.html).
14
-
15
-
For information about the latest changes to Elastic APIs, check [APIs changelog](https://www.elastic.co/docs/api/changes).
13
+
To view the changes in previous versions, go to [Release docs](https://www.elastic.co/guide/en/starting-with-the-elasticsearch-platform-and-its-solutions/8.18/new.html).
16
14
17
15
We recommend you upgrade to the latest Elastic version. To learn how to upgrade, check [Upgrade](/deploy-manage/upgrade.md).
18
16
19
17
To learn more about Elastic releases, use the following resources:
20
18
* Join the [Elastic community forums](https://discuss.elastic.co/)
21
-
* Check out the [Elastic Blog](https://www.elastic.co/blog)
22
-
* Reach out to [Elastic Support](https://mail.google.com/mail/u/0/?fs=1&tf=cm&source=mailto&to=support@elastic.co)
19
+
* Check out the release announcements on the [Elastic Blog](https://www.elastic.co/blog/category/releases)
20
+
* Reach out to [Elastic Support](https://www.elastic.co/support)
Copy file name to clipboardExpand all lines: solutions/search/vector/bring-own-vectors.md
+59-39Lines changed: 59 additions & 39 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,36 +7,50 @@ applies_to:
7
7
serverless:
8
8
products:
9
9
- id: elasticsearch
10
+
description: An introduction to vectors and knn search in Elasticsearch.
10
11
---
11
12
12
13
# Bring your own dense vectors [bring-your-own-vectors]
13
14
15
+
{{es}} enables you store and search mathematical representations of your content called _embeddings_ or _vectors_, which help machines understand and process your data more effectively.
16
+
There are two types of representation (_dense_ and _sparse_), which are suited to different types of queries and use cases (for example, finding similar images and content or storing expanded terms and weights).
14
17
15
-
This tutorial demonstrates how to index documents that already have dense vector embeddings into {{es}}. You’ll also learn the syntax for searching these documents using a `knn` query.
18
+
In this introduction to [vector search](/solutions/search/vector.md), you'll store and search for dense vectors.
19
+
You'll also learn the syntax for searching these documents using a [k-nearest neighbour](/solutions/search/vector/knn.md) (kNN) query.
16
20
17
-
You’ll find links at the end of this tutorial for more information about deploying a text embedding model in {{es}}, so you can generate embeddings for queries on the fly.
21
+
## Prerequisites
18
22
19
-
::::{tip}
20
-
This is an advanced use case. Refer to [Semantic search](../semantic-search.md) for an overview of your options for semantic search with {{es}}.
23
+
- If you're using {{es-serverless}}, create a project that is optimized for vectors. To add the sample data, you must have a `developer` or `admin` predefined role or an equivalent custom role.
24
+
- If you're using {{ech}} or a self-managed cluster, start {{es}} and {{kib}}. The simplest method to complete the steps in this guide is to log in with a user that has the `superuser` built-in role.
25
+
26
+
To learn about role-based access control, check out [](/deploy-manage/users-roles/cluster-or-deployment-auth/user-roles.md).
21
27
22
-
::::
28
+
## Create a vector database
23
29
30
+
When you create vectors (or _vectorize_ your data), you convert complex and nuanced content (such as text, videos, images, or audio) into multidimensional numerical representations.
31
+
They must be stored in specialized data structures designed to ensure efficient similarity search and speedy vector distance calculations.
24
32
25
-
## Step 1: Create an index with `dense_vector` mapping [bring-your-own-vectors-create-index]
33
+
In this quide, you'll use documents that already have dense vector embeddings.
34
+
To deploy a vector embedding model in {{es}} and generate vectors while ingesting and searching your data, refer to the links in [Learn more](#bring-your-own-vectors-learn-more).
26
35
27
-
Each document in our simple dataset will have:
36
+
::::{tip}
37
+
This is an advanced use case that uses the `dense_vector` field type. Refer to [](/solutions/search/semantic-search.md) for an overview of your options for semantic search with {{es}}.
38
+
To learn about the differences between semantic search and vector search, go to [](/solutions/search/ai-search/ai-search.md).
39
+
::::
28
40
29
-
* A review: stored in a `review_text` field
30
-
* An embedding of that review: stored in a `review_vector`field
41
+
:::::{stepper}
42
+
::::{step} Create an index with dense vector field mappings
31
43
32
-
* The `review_vector` field is defined as a [`dense_vector`](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md) data type.
44
+
Each document in our simple data set will have:
33
45
46
+
* A review: stored in a `review_text` field
47
+
* An embedding of that review: stored in a `review_vector` field, which is defined as a [`dense_vector`](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md) data type.
34
48
35
-
::::{tip}
49
+
:::{tip}
36
50
The `dense_vector` type automatically uses `int8_hnsw` quantization by default to reduce the memory footprint required when searching float vectors. Learn more about balancing performance and accuracy in [Dense vector quantization](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-quantization).
51
+
:::
37
52
38
-
::::
39
-
53
+
The following API request defines the `review_text` and `review_vector` fields:
40
54
41
55
```console
42
56
PUT /amazon-reviews
@@ -57,18 +71,17 @@ PUT /amazon-reviews
57
71
}
58
72
```
59
73
60
-
1. The `dims` parameter must match the length of the embedding vector. Here we’re using a simple 8-dimensional embedding for readability. If not specified, `dims` will be dynamically calculated based on the first indexed document.
74
+
1. The `dims` parameter must match the length of the embedding vector. If not specified, `dims` will be dynamically calculated based on the first indexed document.
61
75
2. The `index` parameter is set to `true` to enable the use of the `knn` query.
62
76
3. The `similarity` parameter defines the similarity function used to compare the query vector to the document vectors. `cosine` is the default similarity function for `dense_vector` fields in {{es}}.
63
77
78
+
Here we're using an 8-dimensional embedding for readability.
79
+
The vectors that neural network models work with can have several hundreds or even thousands of dimensions that represent a point in a multi-dimensional space.
80
+
Each vector dimension represents a _feature_ or a characteristic of the unstructured data.
81
+
::::
82
+
::::{step} Add documents with embeddings
64
83
65
-
66
-
## Step 2: Index documents with embeddings [bring-your-own-vectors-index-documents]
67
-
68
-
69
-
### Index a single document [_index_a_single_document]
70
-
71
-
First, index a single document to understand the document structure.
84
+
First, index a single document to understand the document structure:
72
85
73
86
```console
74
87
PUT /amazon-reviews/_doc/1
@@ -80,13 +93,8 @@ PUT /amazon-reviews/_doc/1
80
93
81
94
1. The size of the `review_vector` array is 8, matching the `dims` count specified in the mapping.
82
95
83
-
84
-
85
-
### Bulk index multiple documents [_bulk_index_multiple_documents]
86
-
87
-
In a production scenario, you’ll want to index many documents at once using the [`_bulk` endpoint](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk).
88
-
89
-
Here’s an example of indexing multiple documents in a single `_bulk` request.
96
+
In a production scenario, you'll want to index many documents at once using the [`_bulk` endpoint]({{es-apis}}operation/operation-bulk).
97
+
Here's an example of indexing multiple documents in a single `_bulk` request:
90
98
91
99
```console
92
100
POST /_bulk
@@ -100,10 +108,14 @@ POST /_bulk
100
108
{ "review_text": "This product has ruined my life and the lives of my family and friends.", "review_vector": [0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1] }
101
109
```
102
110
111
+
::::
112
+
:::::
103
113
104
-
## Step 3: Search documents with embeddings[bring-your-own-vectors-search-documents]
114
+
## Test vector search[bring-your-own-vectors-search-documents]
105
115
106
-
Now you can query these document vectors using a [`knn` retriever](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search#operation-search-body-application-json-retriever). `knn` is a type of vector search, which finds the `k` most similar documents to a query vector. Here we’re simply using a raw vector for the query text, for demonstration purposes.
116
+
Now you can query these document vectors using a [`knn` retriever](elasticsearch://reference/elasticsearch/rest-apis/retrievers.md#knn-retriever).
117
+
`knn` is a type of vector search, which finds the `k` most similar documents to a query vector.
118
+
Here we're using a raw vector for the query text for demonstration purposes:
107
119
108
120
```console
109
121
POST /amazon-reviews/_search
@@ -119,23 +131,31 @@ POST /amazon-reviews/_search
119
131
}
120
132
```
121
133
122
-
1.In this simple example, we’re sending a raw vector as the query text. In a real-world scenario, you’ll need to generate vectors for queries using an embedding model.
134
+
1.A raw vector serves as the query text in this example. In a real-world scenario, you'll need to generate vectors for queries using an embedding model.
123
135
2. The `k` parameter specifies the number of results to return.
124
136
3. The `num_candidates` parameter is optional. It limits the number of candidates returned by the search node. This can improve performance and reduce costs.
125
137
138
+
## Next steps
126
139
140
+
If you want to try a similar set of steps from an {{es}} client, check out the guided index workflow:
127
141
128
-
## Learn more [bring-your-own-vectors-learn-more]
142
+
- If you're using Elasticsearch Serverless, go to **{{es}} > Home**, select the vector search workflow, and **Create a vector optimized index**.
143
+
- If you're using {{ech}} or a self-managed cluster, go to **Elasticsearch > Home** and click **Create API index**. Select the vector search workflow.
129
144
130
-
In this simple example, we’re sending a raw vector for the query text. In a real-world scenario you won’t know the query text ahead of time. You’ll need to generate query vectors, on the fly, using the same embedding model that generated the document vectors.
145
+
When you finish your tests and no longer need the sample data set, delete the index:
131
146
132
-
For this you’ll need to deploy a text embedding model in {{es}} and use the [`query_vector_builder` parameter](elasticsearch://reference/query-languages/query-dsl/query-dsl-knn-query.md#knn-query-top-level-parameters). Alternatively, you can generate vectors client-side and send them directly with the search request.
133
-
134
-
Learn how to [use a deployed text embedding model](dense-versus-sparse-ingest-pipelines.md) for semantic search.
147
+
```console
148
+
DELETE /amazon-reviews
149
+
```
135
150
136
-
::::{tip}
137
-
If you’re just getting started with vector search in {{es}}, refer to [Semantic search](../semantic-search.md).
151
+
## Learn more [bring-your-own-vectors-learn-more]
138
152
139
-
::::
153
+
In these simple examples, we're sending a raw vector for the query text.
154
+
In a real-world scenario you won't know the query text ahead of time.
155
+
You'll need to generate query vectors, on the fly, using the same embedding model that generated the document vectors.
156
+
For this you'll need to deploy a text embedding model in {{es}} and use the [`query_vector_builder` parameter](elasticsearch://reference/query-languages/query-dsl/query-dsl-knn-query.md#knn-query-top-level-parameters).
157
+
Alternatively, you can generate vectors client-side and send them directly with the search request.
140
158
159
+
For an example of using pipelines to generate text embeddings, check out [](/solutions/search/vector/dense-versus-sparse-ingest-pipelines.md).
141
160
161
+
To learn about more search options, such as semantic, full-text, and hybrid, go to [](/solutions/search/search-approaches.md).
By default, the Collector only propagates transport-level metadata. If you want the EDOT Collector to propagate metadata like `project_id`, `tenant`, or `environment`, you need a specific SDK instrumentation.
19
+
20
+
### What is client metadata
21
+
22
+
In the context of the EDOT, client metadata refers to gRPC metadata or HTTP headers that accompany telemetry data sent by clients (usually SDKs) to the Collector.
23
+
24
+
For example:
25
+
26
+
- Authorization headers
27
+
- Trace propagation headers
28
+
-`user-agent` strings sent by HTTP clients or EDOT SDKs
29
+
30
+
This is transport-level metadata, and not application-level context.
31
+
32
+
### What isn’t client metadata
33
+
34
+
In this context, client metadata does not include:
35
+
36
+
- Custom attributes like `project_id`, `tenant`, or `environment`
37
+
- Application business logic fields
38
+
- Resource attributes
39
+
40
+
## Symptoms
41
+
42
+
Expected labels do not appear in Elastic APM or metric data.
43
+
44
+
Your collector configuration file, for example `otel-config.yaml`, looks similar to this:
45
+
46
+
```yaml
47
+
receivers:
48
+
otlp:
49
+
protocols:
50
+
http:
51
+
endpoint: "0.0.0.0:4318"
52
+
53
+
processors:
54
+
resource:
55
+
attributes:
56
+
- key: project.id
57
+
from_context: client_metadata
58
+
action: insert
59
+
```
60
+
61
+
This will not work, as the Collector doesn't automatically extract such values from headers.
62
+
63
+
## Resolution
64
+
65
+
If you want to propagate customer IDs or project names into spans or metrics, you must instrument this in your code using one of the SDKs.
66
+
67
+
Use `span.set_attribute` in your application code, where OpenTelemetry spans are created. For example:
68
+
69
+
```python
70
+
from opentelemetry import trace
71
+
72
+
tracer = trace.get_tracer(__name__)
73
+
74
+
with tracer.start_as_current_span("handle_request") as span:
0 commit comments