Asynchronous index construction #5752
Description
Whenever constructing an index with FT.CREATE (including construction after rdb load), we call
ShardDocIndex::Rebuild() -> TraverseAllMatching with leads to a cpu-heavy loop without preemptions. If the total number of documents is large, the instance becomes unresponsive for the whole time frame.
How to reproduce it
Should be as easy as creating a huge index and calling ping, info, get etc and measuring the response time
Proposed solution
We should yield inside the loop after each bucket