-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Add docs for reindex data stream REST endpoints #120653
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
parkertimmins
merged 22 commits into
elastic:main
from
parkertimmins:reindex-data-stream-docs
Jan 29, 2025
Merged
Changes from 21 commits
Commits
Show all changes
22 commits
Select commit
Hold shift + click to select a range
2f25364
Start on asciidoc for /_migration/reindex
parkertimmins 94a8488
reindex data stream status
parkertimmins 2aa2402
Add some examples
parkertimmins 172ab3c
work on reindex cancel
parkertimmins 1c66cdc
add settings
parkertimmins 0c7619c
start on create index from source
parkertimmins ba9ea51
Merge branch 'main' into reindex-data-stream-docs
parkertimmins a615351
create from body
parkertimmins fe7dc5b
Add a bunch of reindex examples
parkertimmins cd9f612
Use 8.x instead of 8 for version name
parkertimmins c662f9a
reorganize reindex docs
parkertimmins 6d075bc
Move create_from to migrations
parkertimmins 160cf31
Cleanup status
parkertimmins 93e32a7
cleanup create-from and cancel
parkertimmins c4cc8d6
Merge branch 'main' into reindex-data-stream-docs
parkertimmins 04f6c96
Cleanup data stream reindex
parkertimmins 6dba402
create from doc yaml test
parkertimmins c3bacc3
Fix doc yaml test - mostly by skipping
parkertimmins b58e627
Merge branch 'main' into reindex-data-stream-docs
parkertimmins 3f9a70b
Review fixes, add doc url to rest-api-spec
parkertimmins 91d06e8
Merge branch 'main' into reindex-data-stream-docs
parkertimmins c9411e4
Fix reindex task cleanup in doc yaml tests
parkertimmins File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
142 changes: 142 additions & 0 deletions
142
docs/reference/migration/apis/create-index-from-source.asciidoc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,142 @@ | ||
[[indices-create-index-from-source]] | ||
=== Create index from source API | ||
++++ | ||
<titleabbrev>Create index from source</titleabbrev> | ||
++++ | ||
|
||
.New API reference | ||
[sidebar] | ||
-- | ||
For the most up-to-date API details, refer to {api-es}/group/endpoint-indices[Index APIs]. | ||
-- | ||
|
||
[[indices-create-index-from-source-api-request]] | ||
==== {api-request-title} | ||
|
||
`PUT /_create_from/<source>/<dest>` | ||
|
||
`POST/_create_from/<source>/<dest>` | ||
|
||
[[indices-create-index-from-source-api-prereqs]] | ||
==== {api-prereq-title} | ||
|
||
* If the {es} {security-features} are enabled, you must have the `manage` | ||
<<privileges-list-indices,index privilege>> for the index. | ||
|
||
[[indices-create-index-from-source-api-desc]] | ||
==== {api-description-title} | ||
This api allows you to add a new index to an {es} cluster, using an existing source index as a basis for the new index. | ||
The settings and mappings from the source index will copied over to the destination index. You can also provide | ||
override settings and mappings which will be combined with the source settings and mappings when creating the | ||
destination index. | ||
|
||
[[indices-create-index-from-source-api-path-params]] | ||
==== {api-path-parms-title} | ||
|
||
`<source>`:: | ||
(Required, string) Name of the existing source index which will be used as a basis. | ||
|
||
`<dest>`:: | ||
(Required, string) Name of the destination index which will be created. | ||
|
||
|
||
[role="child_attributes"] | ||
[[indices-create-index-from-source-api-request-body]] | ||
==== {api-request-body-title} | ||
|
||
`settings_override`:: | ||
(Optional, <<index-modules-settings,index setting object>>) Settings which override the source settings. | ||
|
||
`mappings_override`:: | ||
(Optional, <<mapping,mapping object>>) Mappings which override the source mappings. | ||
|
||
`remove_index_blocks`:: | ||
(Optional, boolean) Filter out any index blocks from the source index when creating the destination index. | ||
Defaults to `true`. | ||
|
||
[[indices-create-index-from-source-api-example]] | ||
==== {api-examples-title} | ||
|
||
Start by creating a source index that we'll copy using this API. | ||
|
||
[source,console] | ||
-------------------------------------------------- | ||
PUT /my-index | ||
{ | ||
"settings": { | ||
"index": { | ||
"number_of_shards": 3, | ||
"blocks.write": true | ||
} | ||
}, | ||
"mappings": { | ||
"properties": { | ||
"field1": { "type": "text" } | ||
} | ||
} | ||
} | ||
-------------------------------------------------- | ||
// TESTSETUP | ||
|
||
Now we create a destination index from the source index. This new index will have the same mappings and settings | ||
as the source index. | ||
|
||
[source,console] | ||
-------------------------------------------------- | ||
POST _create_from/my-index/my-new-index | ||
-------------------------------------------------- | ||
|
||
|
||
Alternatively, we could override some of the source's settings and mappings. This will use the source settings | ||
and mappings as a basis and combine these with the overrides to create the destination settings and mappings. | ||
|
||
[source,console] | ||
-------------------------------------------------- | ||
POST _create_from/my-index/my-new-index | ||
{ | ||
"settings_override": { | ||
"index": { | ||
"number_of_shards": 5 | ||
} | ||
}, | ||
"mappings_override": { | ||
"properties": { | ||
"field2": { "type": "boolean" } | ||
} | ||
} | ||
} | ||
-------------------------------------------------- | ||
|
||
Since the destination index is empty, we very likely will want to write into the index after creation. | ||
This would not be possible if the source index contains an <<index-block-settings,index write block>> which is copied over to the destination index. | ||
One way to handle this is to remove the index write block using a settings override. For example, the following | ||
settings override removes all index blocks. | ||
|
||
|
||
[source,console] | ||
-------------------------------------------------- | ||
POST _create_from/my-index/my-new-index | ||
{ | ||
"settings_override": { | ||
"index": { | ||
"blocks.write": null, | ||
"blocks.read": null, | ||
"blocks.read_only": null, | ||
"blocks.read_only_allow_delete": null, | ||
"blocks.metadata": null | ||
} | ||
} | ||
} | ||
-------------------------------------------------- | ||
|
||
Since this is a common scenario, index blocks are actually removed by default. This is controlled with the parameter | ||
`remove_index_blocks`, which defaults to `true`. If we want the destination index to contains the index blocks from | ||
the source index, we can do the following: | ||
|
||
[source,console] | ||
-------------------------------------------------- | ||
POST _create_from/my-index/my-new-index | ||
{ | ||
"remove_index_blocks": false | ||
} | ||
-------------------------------------------------- |
47 changes: 47 additions & 0 deletions
47
docs/reference/migration/apis/data-stream-reindex-cancel.asciidoc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,47 @@ | ||
[role="xpack"] | ||
[[data-stream-reindex-cancel-api]] | ||
=== Reindex data stream cancel API | ||
++++ | ||
<titleabbrev>Reindex data stream cancel</titleabbrev> | ||
++++ | ||
|
||
.New API reference | ||
[sidebar] | ||
-- | ||
For the most up-to-date API details, refer to {api-es}/group/endpoint-migration[Migration APIs]. | ||
-- | ||
|
||
include::{es-ref-dir}/migration/apis/shared-migration-apis-tip.asciidoc[] | ||
|
||
Cancels a running data stream reindex task which was started by the <<data-stream-reindex-api, data stream reindex API>>. | ||
Any backing indices that have already been reindexed and swapped into the data stream will remain in the data stream. | ||
Only backing indices which are currently being reindexed, or pending backing indices which are still waiting to be reindexed, will be cancelled. | ||
Once a data stream reindex task is cancelled it will no longer be accessible through the | ||
<<data-stream-reindex-status-api,status API>>. If a reindex task is not currently running | ||
this API will return `resource_not_found_exception`. | ||
|
||
[source,console] | ||
---- | ||
POST _migration/reindex/my-data-stream/_cancel | ||
---- | ||
// TEST[skip:cannot easily clean up reindex task between tests] | ||
|
||
[[data-stream-reindex-cancel-request]] | ||
==== {api-request-title} | ||
|
||
`GET /_migration/reindex/<data-stream>/_cancel` | ||
|
||
|
||
[[data-stream-reindex-cancel-prereqs]] | ||
==== {api-prereq-title} | ||
|
||
* If the {es} {security-features} are enabled, you must have the `manage` | ||
<<privileges-list-indices,index privilege>> for the data stream. | ||
|
||
[[data-stream-reindex-cancel-path-params]] | ||
==== {api-path-parms-title} | ||
|
||
`<data-stream>`:: | ||
(Required, string) | ||
Name of data stream to cancel reindexing. | ||
|
135 changes: 135 additions & 0 deletions
135
docs/reference/migration/apis/data-stream-reindex-status.asciidoc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,135 @@ | ||
[role="xpack"] | ||
[[data-stream-reindex-status-api]] | ||
=== Reindex data stream status API | ||
++++ | ||
<titleabbrev>Reindex data stream status</titleabbrev> | ||
++++ | ||
|
||
.New API reference | ||
[sidebar] | ||
-- | ||
For the most up-to-date API details, refer to {api-es}/group/endpoint-migration[Migration APIs]. | ||
-- | ||
|
||
include::{es-ref-dir}/migration/apis/shared-migration-apis-tip.asciidoc[] | ||
|
||
Obtains the current status of a reindex task for the requested data stream. This status is | ||
available while the reindex task is running and for 24 hours after completion of the task, | ||
whether it succeeds or fails. If the task is cancelled, the status is no longer available. | ||
If the task fails, the exception will be listed within the status. | ||
|
||
|
||
[[data-stream-reindex-status-api-request]] | ||
==== {api-request-title} | ||
|
||
`GET /_migration/reindex/<data-stream>/_status` | ||
|
||
|
||
[[data-stream-reindex-status-prereqs]] | ||
==== {api-prereq-title} | ||
|
||
* If the {es} {security-features} are enabled, you must have the `manage` | ||
<<privileges-list-indices,index privilege>> for the data stream. | ||
|
||
[[data-stream-reindex-status-path-params]] | ||
==== {api-path-parms-title} | ||
|
||
`<data-stream>`:: | ||
(Required, string) | ||
Name of data stream to get status for. The reindex task for the | ||
data stream should be currently running or have been completed in the last 24 hours. | ||
|
||
|
||
[role="child_attributes"] | ||
[[data-stream-reindex-status-response-body]] | ||
==== {api-response-body-title} | ||
|
||
`start_time`:: | ||
(Optional, <<time-units,time value>>) The time when the reindex task started. | ||
|
||
`start_time_millis`:: | ||
(integer) The time when the reindex task started, in milliseconds since the epoch. | ||
|
||
`complete`:: | ||
(boolean) `false` if the reindex task is still running, and `true` if the task has completed with success or failure. | ||
|
||
`total_indices_in_data_stream`:: | ||
(integer) The total number of backing indices in the data stream, including the write index. | ||
|
||
`total_indices_requiring_upgrade`:: | ||
(integer) The number of backing indices that need to be upgraded. These will consist of the indices which have an | ||
older version and are not read-only. | ||
|
||
`successes`:: | ||
(integer) The number of backing indices which have already been successfully upgraded. | ||
|
||
`in_progress`:: | ||
(array of objects) Information on the backing indices which are currently being reindexed. | ||
+ | ||
.Properties of objects in `in_progress` | ||
[%collapsible%open] | ||
===== | ||
`index`:: | ||
(string) The name of the source backing index. | ||
|
||
`total_doc_count`:: | ||
(integer) The number of documents in the source backing index. | ||
|
||
`reindexed_doc_count`:: | ||
(integer) The number of documents which have already been added to the destination backing index. | ||
===== | ||
|
||
`pending`:: | ||
(integer) The number of backing indices which still need to be upgraded and have not yet been started. | ||
|
||
`errors`:: | ||
(array of objects) Information on any errors which have occurred. | ||
+ | ||
.Properties of objects in `errors` | ||
[%collapsible%open] | ||
===== | ||
`index`:: | ||
(string) The name of a backing index which has had an error during reindex. | ||
|
||
`message`:: | ||
(string) Description of the error. | ||
===== | ||
|
||
`exceptions`:: | ||
(Optional, string) | ||
Exception message for a reindex failure if the failure could not be tied to a particular index. | ||
|
||
|
||
[[data-stream-reindex-status-example]] | ||
==== {api-examples-title} | ||
|
||
[source,console] | ||
---- | ||
GET _migration/reindex/my-data-stream/_status | ||
---- | ||
// TEST[skip:cannot easily clean up reindex task between tests] | ||
|
||
The following is a typical response: | ||
[source,console-result] | ||
---- | ||
{ | ||
"start_time_millis": 1737676174349, | ||
"complete": false, | ||
"total_indices_in_data_stream": 4, | ||
"total_indices_requiring_upgrade": 3, | ||
"successes": 1, | ||
"in_progress": [ | ||
{ | ||
"index": ".ds-my-data-stream-2025.01.23-000002", | ||
"total_doc_count": 10000000, | ||
"reindexed_doc_count": 1000 | ||
} | ||
], | ||
"pending": 1, | ||
"errors": [] | ||
} | ||
---- | ||
// TEST[skip:cannot easily clean up reindex task between tests] | ||
|
||
For a more in-depth example showing the usage of this API along with the <<data-stream-reindex-api,reindex>> and <<data-stream-reindex-cancel-api,cancel>> APIs, | ||
see this <<reindex-data-stream-api-example,example>>. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.