MB-60971: Improve drain rate of the in-memory segments

[Thejas-bhat]

• Currently when there is a certain amount of data in memory, persister performs an in-memory merge and introduces the merged file as part of the new snapshot. Furthermore, there isn't a bound on how much data the persister can operate upon during the in-memory merge operation.
• This can have effects when the underlying hardware supports very fast ingestion and if there is a huge amount of in-memory segments accumulated the costly merge operation can be time consuming and cause memory pressure to spike up, in which case the application layer would have to stop pumping data into the bleve system.
• The PR introduces the concept of persister workers and the max amount of data (which translates to a bounded number of segments) each worker can parallely perform an in-memory merge. The merged files are introduced to the system in a single step to avoid having too many files in the system since that causes too much pressure on the cleanup procedure of out of ref'd files.
• The original behaviour of unbounded in-memory merge + flush in one shot is still supported, and to keep things compliant with earlier bleve releases, its the default behaviour. To enable the feature proposed in the PR application layer can update the config while opening or creating a new index by setting the values for NumPersisterWorkers and MaxSizeInMemoryMergePerWorker.

[abhinavdangeti]

@Thejas-bhat is this something we want to consider for v2.5.0 still and so morpheus ?

[abhinavdangeti]

@Thejas-bhat have we gotten back results to back this change?

[abhinavdangeti]

Hope we've done our due diligence well to ensure that there're no performance regressions with this change.

[abhinavdangeti]

Whoops wrong response :)