Skip to content

Commit 230e9f1

Browse files
parkertimminsparkertimmins
committed
Add docs for reindex data stream REST endpoints (#120653)
Add documentation for new REST endpoints related to data stream upgrade. Endpoints: - /_migration/reindex - /_migration/reindex/{index}/_status - /_migration/reindex/{index}/_cancel - /_create_from/{source}/{dest}
1 parent e27a1ec commit 230e9f1

File tree

9 files changed

+733
-4
lines changed

9 files changed

+733
-4
lines changed

‎docs/reference/migration/apis/create-index-from-source.asciidoc‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
[[indices-create-index-from-source]]
2+
=== Create index from source API
3+
++++
4+
<titleabbrev>Create index from source</titleabbrev>
5+
++++
6+
7+
.New API reference
8+
[sidebar]
9+
--
10+
For the most up-to-date API details, refer to {api-es}/group/endpoint-indices[Index APIs].
11+
--
12+
13+
[[indices-create-index-from-source-api-request]]
14+
==== {api-request-title}
15+
16+
`PUT /_create_from/<source>/<dest>`
17+
18+
`POST/_create_from/<source>/<dest>`
19+
20+
[[indices-create-index-from-source-api-prereqs]]
21+
==== {api-prereq-title}
22+
23+
* If the {es} {security-features} are enabled, you must have the `manage`
24+
<<privileges-list-indices,index privilege>> for the index.
25+
26+
[[indices-create-index-from-source-api-desc]]
27+
==== {api-description-title}
28+
This api allows you to add a new index to an {es} cluster, using an existing source index as a basis for the new index.
29+
The settings and mappings from the source index will copied over to the destination index. You can also provide
30+
override settings and mappings which will be combined with the source settings and mappings when creating the
31+
destination index.
32+
33+
[[indices-create-index-from-source-api-path-params]]
34+
==== {api-path-parms-title}
35+
36+
`<source>`::
37+
(Required, string) Name of the existing source index which will be used as a basis.
38+
39+
`<dest>`::
40+
(Required, string) Name of the destination index which will be created.
41+
42+
43+
[role="child_attributes"]
44+
[[indices-create-index-from-source-api-request-body]]
45+
==== {api-request-body-title}
46+
47+
`settings_override`::
48+
(Optional, <<index-modules-settings,index setting object>>) Settings which override the source settings.
49+
50+
`mappings_override`::
51+
(Optional, <<mapping,mapping object>>) Mappings which override the source mappings.
52+
53+
`remove_index_blocks`::
54+
(Optional, boolean) Filter out any index blocks from the source index when creating the destination index.
55+
Defaults to `true`.
56+
57+
[[indices-create-index-from-source-api-example]]
58+
==== {api-examples-title}
59+
60+
Start by creating a source index that we'll copy using this API.
61+
62+
[source,console]
63+
--------------------------------------------------
64+
PUT /my-index
65+
{
66+
"settings": {
67+
"index": {
68+
"number_of_shards": 3,
69+
"blocks.write": true
70+
}
71+
},
72+
"mappings": {
73+
"properties": {
74+
"field1": { "type": "text" }
75+
}
76+
}
77+
}
78+
--------------------------------------------------
79+
// TESTSETUP
80+
81+
Now we create a destination index from the source index. This new index will have the same mappings and settings
82+
as the source index.
83+
84+
[source,console]
85+
--------------------------------------------------
86+
POST _create_from/my-index/my-new-index
87+
--------------------------------------------------
88+
89+
90+
Alternatively, we could override some of the source's settings and mappings. This will use the source settings
91+
and mappings as a basis and combine these with the overrides to create the destination settings and mappings.
92+
93+
[source,console]
94+
--------------------------------------------------
95+
POST _create_from/my-index/my-new-index
96+
{
97+
"settings_override": {
98+
"index": {
99+
"number_of_shards": 5
100+
}
101+
},
102+
"mappings_override": {
103+
"properties": {
104+
"field2": { "type": "boolean" }
105+
}
106+
}
107+
}
108+
--------------------------------------------------
109+
110+
Since the destination index is empty, we very likely will want to write into the index after creation.
111+
This would not be possible if the source index contains an <<index-block-settings,index write block>> which is copied over to the destination index.
112+
One way to handle this is to remove the index write block using a settings override. For example, the following
113+
settings override removes all index blocks.
114+
115+
116+
[source,console]
117+
--------------------------------------------------
118+
POST _create_from/my-index/my-new-index
119+
{
120+
"settings_override": {
121+
"index": {
122+
"blocks.write": null,
123+
"blocks.read": null,
124+
"blocks.read_only": null,
125+
"blocks.read_only_allow_delete": null,
126+
"blocks.metadata": null
127+
}
128+
}
129+
}
130+
--------------------------------------------------
131+
132+
Since this is a common scenario, index blocks are actually removed by default. This is controlled with the parameter
133+
`remove_index_blocks`, which defaults to `true`. If we want the destination index to contains the index blocks from
134+
the source index, we can do the following:
135+
136+
[source,console]
137+
--------------------------------------------------
138+
POST _create_from/my-index/my-new-index
139+
{
140+
"remove_index_blocks": false
141+
}
142+
--------------------------------------------------

‎docs/reference/migration/apis/data-stream-reindex-cancel.asciidoc‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
[role="xpack"]
2+
[[data-stream-reindex-cancel-api]]
3+
=== Reindex data stream cancel API
4+
++++
5+
<titleabbrev>Reindex data stream cancel</titleabbrev>
6+
++++
7+
8+
.New API reference
9+
[sidebar]
10+
--
11+
For the most up-to-date API details, refer to {api-es}/group/endpoint-migration[Migration APIs].
12+
--
13+
14+
include::{es-ref-dir}/migration/apis/shared-migration-apis-tip.asciidoc[]
15+
16+
Cancels a running data stream reindex task which was started by the <<data-stream-reindex-api, data stream reindex API>>.
17+
Any backing indices that have already been reindexed and swapped into the data stream will remain in the data stream.
18+
Only backing indices which are currently being reindexed, or pending backing indices which are still waiting to be reindexed, will be cancelled.
19+
Once a data stream reindex task is cancelled it will no longer be accessible through the
20+
<<data-stream-reindex-status-api,status API>>. If a reindex task is not currently running
21+
this API will return `resource_not_found_exception`.
22+
23+
24+
///////////////////////////////////////////////////////////
25+
[source,console]
26+
------------------------------------------------------
27+
POST _migration/reindex
28+
{
29+
"source": {
30+
"index": "my-data-stream"
31+
},
32+
"mode": "upgrade"
33+
}
34+
------------------------------------------------------
35+
// TESTSETUP
36+
// TEST[setup:my_data_stream]
37+
///////////////////////////////////////////////////////////
38+
39+
40+
[source,console]
41+
----
42+
POST _migration/reindex/my-data-stream/_cancel
43+
----
44+
// TEST[teardown:data_stream_cleanup]
45+
46+
[[data-stream-reindex-cancel-request]]
47+
==== {api-request-title}
48+
49+
`GET /_migration/reindex/<data-stream>/_cancel`
50+
51+
52+
[[data-stream-reindex-cancel-prereqs]]
53+
==== {api-prereq-title}
54+
55+
* If the {es} {security-features} are enabled, you must have the `manage`
56+
<<privileges-list-indices,index privilege>> for the data stream.
57+
58+
[[data-stream-reindex-cancel-path-params]]
59+
==== {api-path-parms-title}
60+
61+
`<data-stream>`::
62+
(Required, string)
63+
Name of data stream to cancel reindexing.
64+

‎docs/reference/migration/apis/data-stream-reindex-status.asciidoc‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
[role="xpack"]
2+
[[data-stream-reindex-status-api]]
3+
=== Reindex data stream status API
4+
++++
5+
<titleabbrev>Reindex data stream status</titleabbrev>
6+
++++
7+
8+
.New API reference
9+
[sidebar]
10+
--
11+
For the most up-to-date API details, refer to {api-es}/group/endpoint-migration[Migration APIs].
12+
--
13+
14+
include::{es-ref-dir}/migration/apis/shared-migration-apis-tip.asciidoc[]
15+
16+
Obtains the current status of a reindex task for the requested data stream. This status is
17+
available while the reindex task is running and for 24 hours after completion of the task,
18+
whether it succeeds or fails. If the task is cancelled, the status is no longer available.
19+
If the task fails, the exception will be listed within the status.
20+
21+
///////////////////////////////////////////////////////////
22+
[source,console]
23+
------------------------------------------------------
24+
POST _migration/reindex
25+
{
26+
"source": {
27+
"index": "my-data-stream"
28+
},
29+
"mode": "upgrade"
30+
}
31+
------------------------------------------------------
32+
// TESTSETUP
33+
// TEST[setup:my_data_stream]
34+
35+
[source,console]
36+
------------------------------------------------------
37+
POST /_migration/reindex/my-data-stream/_cancel
38+
DELETE _data_stream/my-data-stream
39+
DELETE _index_template/my-data-stream-template
40+
------------------------------------------------------
41+
// TEARDOWN
42+
///////////////////////////////////////////////////////////
43+
44+
45+
[[data-stream-reindex-status-api-request]]
46+
==== {api-request-title}
47+
48+
`GET /_migration/reindex/<data-stream>/_status`
49+
50+
51+
[[data-stream-reindex-status-prereqs]]
52+
==== {api-prereq-title}
53+
54+
* If the {es} {security-features} are enabled, you must have the `manage`
55+
<<privileges-list-indices,index privilege>> for the data stream.
56+
57+
[[data-stream-reindex-status-path-params]]
58+
==== {api-path-parms-title}
59+
60+
`<data-stream>`::
61+
(Required, string)
62+
Name of data stream to get status for. The reindex task for the
63+
data stream should be currently running or have been completed in the last 24 hours.
64+
65+
66+
[role="child_attributes"]
67+
[[data-stream-reindex-status-response-body]]
68+
==== {api-response-body-title}
69+
70+
`start_time`::
71+
(Optional, <<time-units,time value>>) The time when the reindex task started.
72+
73+
`start_time_millis`::
74+
(integer) The time when the reindex task started, in milliseconds since the epoch.
75+
76+
`complete`::
77+
(boolean) `false` if the reindex task is still running, and `true` if the task has completed with success or failure.
78+
79+
`total_indices_in_data_stream`::
80+
(integer) The total number of backing indices in the data stream, including the write index.
81+
82+
`total_indices_requiring_upgrade`::
83+
(integer) The number of backing indices that need to be upgraded. These will consist of the indices which have an
84+
older version and are not read-only.
85+
86+
`successes`::
87+
(integer) The number of backing indices which have already been successfully upgraded.
88+
89+
`in_progress`::
90+
(array of objects) Information on the backing indices which are currently being reindexed.
91+
+
92+
.Properties of objects in `in_progress`
93+
[%collapsible%open]
94+
=====
95+
`index`::
96+
(string) The name of the source backing index.
97+
98+
`total_doc_count`::
99+
(integer) The number of documents in the source backing index.
100+
101+
`reindexed_doc_count`::
102+
(integer) The number of documents which have already been added to the destination backing index.
103+
=====
104+
105+
`pending`::
106+
(integer) The number of backing indices which still need to be upgraded and have not yet been started.
107+
108+
`errors`::
109+
(array of objects) Information on any errors which have occurred.
110+
+
111+
.Properties of objects in `errors`
112+
[%collapsible%open]
113+
=====
114+
`index`::
115+
(string) The name of a backing index which has had an error during reindex.
116+
117+
`message`::
118+
(string) Description of the error.
119+
=====
120+
121+
`exceptions`::
122+
(Optional, string)
123+
Exception message for a reindex failure if the failure could not be tied to a particular index.
124+
125+
126+
[[data-stream-reindex-status-example]]
127+
==== {api-examples-title}
128+
129+
[source,console]
130+
----
131+
GET _migration/reindex/my-data-stream/_status
132+
----
133+
134+
The following is a typical response:
135+
[source,console-result]
136+
----
137+
{
138+
"start_time_millis": 1737676174349,
139+
"complete": false,
140+
"total_indices_in_data_stream": 4,
141+
"total_indices_requiring_upgrade": 3,
142+
"successes": 1,
143+
"in_progress": [
144+
{
145+
"index": ".ds-my-data-stream-2025.01.23-000002",
146+
"total_doc_count": 10000000,
147+
"reindexed_doc_count": 1000
148+
}
149+
],
150+
"pending": 1,
151+
"errors": []
152+
}
153+
----
154+
// TEST[skip:cannot easily clean up reindex task between tests]
155+
156+
For a more in-depth example showing the usage of this API along with the <<data-stream-reindex-api,reindex>> and <<data-stream-reindex-cancel-api,cancel>> APIs,
157+
see this <<reindex-data-stream-api-example,example>>.

0 commit comments

Comments
 (0)