block insert latency jitter during live sync

[fynnss]

Hi, reth team.

We are currently conducting performance testing on reth-bsc(reth: v1.5.1). After enabling parallel sparse trie, we have observed a significant reduction in validation time, from 120ms to <40ms.

RUSTFLAGS="-C target-cpu=native" cargo build --profile maxperf --features jemalloc,asm-keccak

RUST_LOG=info ./target/maxperf/reth-bsc node \
    --chain=bsc \
    --datadir=./data_dir \
    --authrpc.addr "0.0.0.0" \
    --authrpc.port 8551 \
    --authrpc.jwtsecret ./data_dir/jwt.hex \
    --engine.parallel-sparse-trie \
    --http \
    --http.addr "0.0.0.0" \
    --http.port 8545 \
    --metrics 0.0.0.0:6060 \
    --http.api "admin, debug, eth, net, trace, txpool, web3, rpc, reth, ots"

To better understand the performance, we have added some metrics to measure the overall time spent for block insertion as well as the time spent in each phase (execution and validation). However, we are experiencing noticeable latency jitter in the overall time, as illustrated in the attached graph.

I would like to understand the reasons behind this jitter and seek advice on how to mitigate it.

Thank you for your assistance!