Profiling Zebra's initial sync performance

[arya2]

Motivation

It may make it easier to:

• Narrow down what may have caused a performance regression
• Improve the effectiveness of a quick test for sampling initial sync performance that can be run for every PR update

Possible Design

• Log to a file

• Add a feature flag for enabling extra logs during validation when:

  • semantic validation of blocks/transactions start, progress, or finish
  • contextual validation of blocks/transactions start, progress, or finish
  • blocks are written to disk

• Process the logs into a performance profile

• Add a test for comparing a new profile to the previous profile

  • This test could fail if the entire sync or any steps are far slower than expected

[upbqdn]

Initial Sync Profiles

Below are profiling results for a regular full initial sync for Zebra v3.0.0 in release mode. I split the results by some network upgrades. The only difference from all defaults was that I needed to set [profile.release.debug] to true in Cargo.toml to obtain meaningful reports.

Genesis to Sapling

Sync for heights in ~ [0, 419_200]: https://share.firefox.dev/3LG3pEJ.

Sapling to NU5

Sync for heights in ~ (419_200, 1_687_104]: https://share.firefox.dev/4r4UzjZ.

NU5 to NU6

Sync for heights in ~ (1_687_104, 2_726_400]: https://share.firefox.dev/47IrS4W.

NU6 to NU6.1

Sync for heights in ~ (2_726_400, 3_137_800]: https://share.firefox.dev/3X0hzmE.