Skip to content

Solana performance improvement #1630

@sistemd

Description

@sistemd

As a consequence of how our BlockStreamer trait works, Solana currently downloads many CAR files in parallel and then processes them in parallel. However, when it comes to downloading data, doing things in parallel is actually slightly slower than doing them sequentially because the limit is network bandwidth - parallelism does not help with speed. What would be faster is if there was only ever one download going on at any time, and the workers were only processing files in parallel rather than downloading them. This means that the download happens in parallel with processing, so we're never stuck doing "nothing" waiting for a file to download.

This should result in a neat performance gain, but it will probably require some pretty significant refactoring for the BlockStreamer trait and surrounding code.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions