Skip to content

(Doc+) Split API | add where need to have sufficient disk and how much #119784

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

(Doc+) Split API | add where need to have sufficient disk and how much #119784

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion docs/reference/indices/split-index.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,11 +121,15 @@ newly split index.
A split operation:

. Creates a new target index with the same definition as the source
index, but with a larger number of primary shards.
index, but with a larger number of primary shards. Created shards allocate
to the same nodes as their correlating source primary shard, so it must
already have sufficient disk to host the copy of the data.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After consulting @tlrx (thanks!) indeed if the shards are on different nodes, the new index shards will be allocated and split on the nodes of their corresponding shards.

We should probably also clarify the above in the following sentence later in the document:

The node handling the split process must have sufficient free disk space to accommodate a second copy of the existing index.

to something like

The nodes handling the split process must have sufficient free disk space to accommodate a second copy of the original shards.


. Hard-links segments from the source index into the target index. (If
the file system doesn't support hard-linking, then all segments are copied
into the new index, which is a much more time consuming process.)
+
TIP: Elastic Cloud's backing file systems do not support hard linking.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is probably best answered by @elastic/es-core-infra team who did #61145 . Adding the author of that PR to this (but anyone from the team feel free to step in to review).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding #61145, we did merge a change for it but ended up removing it because, it turned out, Cloud was in fact using a filesystem that correctly reported quotas (XFS IIRC).

Regarding this tip in the docs, I can only imagine it has something to do with attempting to hard-link across Docker volumes, which doesn't work, but you'd need to check with someone from the Hosted ESS team to get an up-to-date answer.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stefnestor might you know who to include from Hosted ESS team to pitch in / review?


. Hashes all documents again, after low level files are created, to delete
documents that belong to a different shard.
Expand Down
Loading