You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: explore-analyze/transforms/transform-limitations.md
+49Lines changed: 49 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -123,6 +123,55 @@ If your data uses the [date nanosecond data type](elasticsearch://reference/elas
123
123
124
124
If you use ILM to have time-based indices, consider using the [Date index name](elasticsearch://reference/enrich-processor/date-index-name-processor.md) instead. The processor works without duplicated documents if your {{transform}} contains a `group_by` based on `date_histogram`.
125
125
126
+
### SLO transforms at scale may misreport state and increase PIT activity
127
+
128
+
When running a large number of SLO {{transforms}}, two types of limitations can appear:
129
+
130
+
- {{transforms-cap}} may [misreport](#transforms-inaccurate-errors) their state and suggest deletion even though they are still valid
131
+
- Point-in-time (PIT) requests may [accumulate](#transforms-pit-overloads), creating extra activity on the cluster and slowing down operations
132
+
133
+
#### {{transforms-cap}} can return inaccurate errors that suggest deletion [transforms-inaccurate-errors]
134
+
135
+
The {{transforms-cap}} API and the {{transforms-cap}} page in {{kib}} (**Stack Management** > **{{transforms-cap}})** may display misleading error messages for {{transforms}} created by service level objectives (SLOs).
136
+
137
+
The message typically reads:
138
+
139
+
```console
140
+
Found task for transform [...], but no configuration for it. To delete this transform use DELETE with force=true.
141
+
```
142
+
Although the error suggests that the {{transform}} is broken and should be deleted, this is usually not the case. The error occurs because of a limitation in how the {{transforms}} are queried and presented by the API and UI. Deleting the {{transform}} can break related SLOs.
143
+
144
+
If you encounter this error, do not delete the {{transform}}. If a {{transform}} was already deleted, you can restore it by resetting the SLO, either in the {{kib}} or with the API:
145
+
146
+
- In {{kib}}, go to the **Observability** > **SLOs application**, open the menu for the affected SLO, and click **Reset**.
147
+
148
+
- If you use the API, run the following request:
149
+
150
+
```console
151
+
POST kbn:/api/observability/slos/<SLO_ID>/_reset
152
+
```
153
+
154
+
#### Large numbers of {{transform}} can cause PIT overloads [transforms-pit-overloads]
155
+
156
+
{{transforms-cap}} rely on point-in-time (PIT) searches to ensure that queries remain consistent during data changes. Each {{transform}} can open and close multiple PITs during its lifetime.
157
+
158
+
When many {{transforms}} run concurrently, especially in environments with large numbers of SLOs (hundreds to more than a thousand transforms), PITs can be opened and closed in quick succession. Because PITs are closed asynchronously, the close operation does not wait for the previous request to complete. This can create a backlog of PIT close requests, known as a PIT overload.
159
+
160
+
A PIT overload increases the memory pressure on hot nodes. This may result in rising memory usage and long garbage collection cycles. In practice, searches and writes may slow down or queue until the node recovers, and in severe cases a restart may be required.
161
+
162
+
As a workaround, you can disable PIT for a specific {{transform}}:
163
+
164
+
```console
165
+
POST _transform/<transform_id>/_update
166
+
{
167
+
"settings": {
168
+
"use_point_in_time": false
169
+
}
170
+
}
171
+
```
172
+
173
+
To fix this issue permanently, upgrade to the following versions or later: 8.18.8, 8.19.5, 9.0.8, 9.1.5, or 9.2.0.
174
+
126
175
## Limitations in {{kib}} [transform-ui-limitations]
127
176
128
177
### {{transforms-cap}} are visible in all {{kib}} spaces [transform-space-limitations]
Copy file name to clipboardExpand all lines: manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -150,7 +150,7 @@ To add an index template to a cluster and apply the lifecycle policy to indices
150
150
1. On the **Component templates** page, use the search and filter tools to select any [component templates](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) to include in the index template. The index template will inherit the settings, mappings, and aliases defined in the component templates and apply them to indices when they're created.
151
151
152
152
1. On the **Index settings** page:
153
-
1. Configure ILM by specifying the [ILM settings](https://www.elastic.co/docs/api/doc/elasticsearch/configuration-reference/index-lifecycle-management-settings#_index_level_settings_2) to apply to the indices:
153
+
1. Configure ILM by specifying the [ILM settings](elasticsearch://reference/elasticsearch/configuration-reference/index-lifecycle-management-settings.md) to apply to the indices:
154
154
*`index.lifecycle.name` - The lifecycle policy to manage the created indices.
155
155
*`index.lifecycle.rollover_alias` - The index [alias](/manage-data/data-store/aliases.md) used for querying and managing the set of indices associated with a lifecycle policy that contains a rollover action.
Copy file name to clipboardExpand all lines: reference/fleet/manage-integrations.md
+3-12Lines changed: 3 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -47,15 +47,6 @@ The **Integrations** app in {{kib}} needs access to the public {{package-registr
47
47
48
48
::::
49
49
50
-
51
-
52
-
53
-
54
-
55
-
56
-
57
-
58
-
59
-
60
-
61
-
50
+
:::{tip}
51
+
Once you've started using integrations to ingest data, you can customize how that data is managed over time. Refer to [Index lifecycle management](/reference/fleet/data-streams.md#data-streams-ilm) to learn more.
Copy file name to clipboardExpand all lines: solutions/search/agent-builder/limitations-known-issues.md
+1-3Lines changed: 1 addition & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,12 +16,10 @@ These pages are hidden from the docs TOC and have `noindexed` meta headers.
16
16
17
17
## Model selection
18
18
19
-
Initally, {{agent-builder}} only supports working with the [Elastic Managed LLM](kibana://reference/connectors-kibana/elastic-managed-llm.md) running on the Elastic Inference Service which uses Claude Sonnet 3.7, on {{ech}} and {{serverless-full}}.
19
+
Initially, {{agent-builder}} only supports working with the [Elastic Managed LLM](kibana://reference/connectors-kibana/elastic-managed-llm.md) running on the Elastic Inference Service, on {{ech}} and {{serverless-full}}.
20
20
21
21
Locally this picks the first AI connector available.
22
22
23
-
This can be changed under Stack Management -> AI -> GenAI Settings -> Default AI Connector.
Copy file name to clipboardExpand all lines: solutions/search/vector/bring-own-vectors.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -45,7 +45,7 @@ Each document in our simple data set will have:
45
45
* An embedding of that review: stored in a `review_vector` field, which is defined as a [`dense_vector`](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md) data type.
46
46
47
47
:::{tip}
48
-
The `dense_vector` type automatically uses quantization by default to reduce the memory footprint when searching float vectors.
48
+
The `dense_vector` type automatically uses quantization to reduce the memory footprint when searching float vectors.
49
49
Learn more about the default quantization strategy and balancing performance and accuracy in [Dense vector field type](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md).
Copy file name to clipboardExpand all lines: solutions/search/vector/knn.md
+4-3Lines changed: 4 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -237,10 +237,11 @@ POST byte-image-index/_search
237
237
If you want to provide `float` vectors but still get the memory savings of `byte` vectors, use the [quantization](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-quantization) feature. Quantization allows you to provide `float` vectors, but internally they are indexed as `byte` vectors. Additionally, the original `float` vectors are still retained in the index.
238
238
239
239
::::{note}
240
-
The default index type for `dense_vector` is `int8_hnsw`.
240
+
The default index type for `dense_vector` is either `bbq_hnsw` or `int8_hnsw`, depending on your product version. Refer to [Dense vector field type](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md).
241
241
::::
242
242
243
-
To use quantization, set the `dense_vector` index type to `int8_hnsw` or `int4_hnsw`.
243
+
You can use the default quantization strategy or specify an index option.
244
+
For example, use `int8_hnsw`:
244
245
245
246
```console
246
247
PUT quantized-image-index
@@ -291,7 +292,7 @@ PUT quantized-image-index
291
292
}
292
293
```
293
294
294
-
Because the original `float` vectors are retained alongside the quantized index, you can use them for re-scoring: retrieve candidates quickly via the `int8_hnsw` (or `int4_hnsw`) index, then rescore the top `k` hits using the original `float` vectors. This provides the best of both worlds, fast search and accurate scoring.
295
+
Because the original `float` vectors are retained alongside the quantized index, you can use them for re-scoring: retrieve candidates quickly via the `int8_hnsw` index, then rescore the top `k` hits using the original `float` vectors. This provides the best of both worlds, fast search and accurate scoring.
0 commit comments