Question about Performance: Processed block per second

[aeharvlee]

Hi. I'm Denver.

TLDR:

• Is there any tips for maximizing performance(processing blocks more fast) when do check:data?

Description
I'm currently implementing rosetta-klaytn for Klaytn.
Actually rosetta-klaytn implementation has been completed (but not yet published to public), so I'm currently doing check:data test for it.

Klaytn's current block number is 91,145,672 which is nine times bigger than Ethereum.
Recently processed block number by check:data is 14,145,672 (There are still a lot of blocks left.)

The processing of check:data is 10 blocks per sec and I'm trying to maximize this performance by optimizing configuration values and fix some source codes.
Please give me some tips or advices if you know something.

Machine SPEC I'm currently using
AWS c5a.8xlarge

• 32 vCPU
• 64 GiB RAM

Config I'm currently using

{
  "network": {
    "blockchain": "Klaytn",
    "network": "Mainnet"
  },
  "online_url": "http://localhost:8080",
  "data_directory": "cli-data-cypress",
  "http_timeout": 120,
  "max_retries": 15,
  "max_online_connections": 256,
  "max_sync_concurrency": 256,
  "tip_delay": 300,
  "max_reorg_depth": 100,
  "log_configuration": false,
  "compression_disabled":true,
  "memory_limit_disabled":false,
  "error_stack_trace_disabled": false,
  "coin_supported": false,
  "construction": null,
  "data": {
    "active_reconciliation_concurrency": 16,
    "inactive_reconciliation_concurrency": 32,
    "inactive_reconciliation_frequency": 250,
    "log_blocks": true,
    "log_transactions": true,
    "log_balance_changes": true,
    "log_reconciliations": true,
    "ignore_reconciliation_error": false,
    "reconciliation_disabled": false,
    "reconciliation_drain_disabled": false,
    "inactive_discrepancy_search_disabled": false,
    "balance_tracking_disabled": false,
    "coin_tracking_disabled": false,
    "status_port": 8081,
    "results_output_file": "",
    "pruning_disabled": false,
    "initial_balance_fetch_disabled": false,
    "historical_balance_disabled": false,
    "bootstrap_balances": "bootstrap_balances.json"
  }
}

And I changed DefaultCacheSize from 2000 << 20 to 8000 << 20 to increase cache size to get much blocks from Block Fetcher.

[aeharvlee]

To investigate how can I improve the performance of rosetta-cli on Klaytn, I'm drawing Architecture of rosetta-cli like below.
If you guys are interested in this diagram, I'll upgrade the current diagram to help rosetta community to understand rosetta-cli better.
Maybe we can make and upgrade this diagram together :)

[shrimalmadhur]

Hey @aeharvlee
thanks for the detailed diagram. We know there are performance issues with check:data on larger blockchains. We are trying to explore options on how to improve them. One way we are trying is doing concurrent validation per block ranges. Another way we are trying is to see if badger is a right choice or if we can use any another key value db to improve perf. There are certainly some more improvements which can be made in the rosetta-cli code (faster json encoding/decoding), feel free to explore that.

We will take a look at this diagram and see if we can put it in the rosetta-api website and here for future reference

[aeharvlee]

@shrimalmadhur

Sorry for late reply. Thanks for your reply :) Currently I'm trying to update the diagram with more details.
I'll show you after I updated it! Hope this will be included in the rosetta-api website :)

[axhar]

Hey @aeharvlee, Did you try turning off logging? sometimes excessive logging slows down the check:data,

    "log_blocks": true,
    "log_transactions": true,
    "log_balance_changes": true,
    "log_reconciliations": true,