-
Notifications
You must be signed in to change notification settings - Fork 6
Description
As a consequence of how our BlockStreamer trait works, Solana currently downloads many CAR files in parallel and then processes them in parallel. However, when it comes to downloading data, doing things in parallel is actually slightly slower than doing them sequentially because the limit is network bandwidth - parallelism does not help with speed. What would be faster is if there was only ever one download going on at any time, and the workers were only processing files in parallel rather than downloading them. This means that the download happens in parallel with processing, so we're never stuck doing "nothing" waiting for a file to download.
This should result in a neat performance gain, but it will probably require some pretty significant refactoring for the BlockStreamer trait and surrounding code.