Pause writing data to disk for merges when disk almost full

[DaveCTurner]

If a node exceeds the flood-stage disk watermark then we block further writes to its indices but we allow merges to continue. Merges can temporarily consume a very large amount of disk space, more than enough to fill up the gap between any reasonable flood-stage watermark and a completely full disk. When a node completely fills its disk, it basically dies.

We could pause merge-related writes in this situation, for instance by overriding Store$StoreDirectory#createOutput and adjusting the output's behaviour according to the supplied IOContext.

We probably don't want to do this for all writes, because (e.g.) a primary may need to refresh before it can relocate itself elsewhere, and because blocking random write threads seems like a recipe for deadlocks. Blocking merge threads seems ok tho. We may also need to be sensitive to the size of the merge (see IOContext.mergeInfo.estimatedMergeBytes and IOContext.flushInfo.estimatedSegmentSize) since smaller merges may soon be triggered by the merge-on-refresh feature.

It's unclear whether to do this based on the read_only_allow_delete block (which affects other nodes below the flood-stage watermark) or the actual disk usage on the node (which may not know the flood-stage watermark that the master is using).

---

NB we can also consider reducing the flood-stage max headroom once we have better protection against merges consuming all the remaining space.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

Notes from a discussion today:

Prefer failing the merge (and preventing more merges) over pausing it, assuming we can do so in a way that would not fail the shard and would clean up any dangling data.

Can we limit concurrent merges when close to disk-full too? NB force merges allow up to allocated_processors/8 to happen concurrently?

[kunisen]

Do we have a plan to fix this? @DaveCTurner

We can see this happens from time to time, and this is one of the most common cause happening on cloud and when it happens, basically users could do nothing - they can't move instance to a different allocator, nor they cannot restart instance from UI (but can do from API). Maybe they could rolling restart the whole cluster from UI but that's service degradation and we should really block this from happening in the first place.

(We have many more support cases than it shows up in GH link references about the dev help tickets. it's basically worked around by support team perfectly so that it rarely get escalated to dev team)

Could we think of putting this on the roadmap please?

cc @romain-chanu as we discussed this internally.
also cc @ppf2 @Gunnerva for your visibility (I understand you might have already discussed this in ES supportability sync.)

[DaveCTurner]

it's basically worked around by support team perfectly so that it rarely get escalated to dev team

That's the problem, surely? I don't have much influence over the roadmap so it doesn't really work to ping me about it here. We need to be capturing the need for this in places that the product folks can see it, either in SDHs or ERs.

[kunisen]

Thanks @DaveCTurner

it's basically worked around by support team perfectly so that it rarely get escalated to dev team

That's the problem, surely?

Yes, unfortunately cases are mostly ended up in support because we have resource overrides which could workaround this, but actually we shouldn't have to do this in the first place.
@romain-chanu may have some data and I am also help to coordinate case data to show you some more evidence.

FWIW, this is similar to "too many shards" thingy which we have had the issue for years which were mostly rarely visible to devs until a while ago we fixed it.

either in SDHs or ERs.

Why do we not consider cases? Logically thinking, e.g. if we have tens or hundreds of cases a year then it would be also an effective evidence to prioritize.
But I get your point that, I probably should gather data and speak in a different place.

---

I will work with @Gunnerva @romain-chanu to collect data and talk internally.
Thanks for pointing out the direction to prioritize this. Much appreciated your comment, very helpful!

[kunisen]

Oh I think I wrongly understood it. Now I think I get it - David you are mentioning we - support should not workaround the issue, but actually should SDH to make it more visible.
Could you confirm if this is correct? @DaveCTurner

Thanks!