store: Synchronous API #244
Description
Current writes and read in the Store are asynchronous. Writes via Append cannot be accessed via Get immediately after write(same as for Head after #239).
However, doing this with our requirements is not trivial. We need to keep writes IO free and return as early as possible as its currently done. With introduction of contiguous head in #239 this becomes even more problematic, as each write has to advance this head. Advancing, in its current implementation, always hits the disk by checking if the next disk is available locally and setting this new header as new contiguous head. Allowing each write to hit the disk is gonna be a major performance regression, so the solution to that must be found.
Conceptually, the solution is possible by tracking the gaps after the contiguous head within the Store. As Store has full control over what gets written into it, it can retain header ranges that didn't contribute to a contiguous chain and produces gaps. Then, whenever any new header is written, the Store is gonna check tracked header and see if it advance the contiguous head without hitting the disk. This should work well in conjuction with autoadvancing on Start implemented as #239, allowing writes to avoid disk hits in case tracked header gaps are cleared out after the restart.