[replay] parallelize on chunk level

[zekun000]

Description

How Has This Been Tested?

Key Areas to Review

Type of Change

• New feature
• Bug fix
• Breaking change
• Performance improvement
• Refactoring
• Dependency update
• Documentation update
• Tests

Which Components or Systems Does This Change Impact?

• Validator Node
• Full Node (API, Indexer, etc.)
• Move/Aptos Virtual Machine
• Aptos Framework
• Aptos CLI/SDK
• Developer Infrastructure
• Move Compiler
• Other (specify)

Checklist

• I have read and followed the CONTRIBUTING doc
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I identified and added all stakeholders and component owners affected by this change as reviewers
• I tested both happy and unhappy path of the functionality
• I have made corresponding changes to the documentation

[github-actions]

✅ Forge suite realistic_env_max_load success on 4b41912ae24a0272a37a5be7119a6cd9d9f27fd3

two traffics test: inner traffic : committed: 13379.79 txn/s, submitted: 13380.00 txn/s, expired: 0.21 txn/s, latency: 2813.98 ms, (p50: 2700 ms, p70: 2800, p90: 3200 ms, p99: 5100 ms), latency samples: 5087260
two traffics test : committed: 99.99 txn/s, latency: 824.82 ms, (p50: 700 ms, p70: 800, p90: 1000 ms, p99: 2900 ms), latency samples: 1700
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.385, avg: 2.122", "ConsensusProposalToOrdered: max: 0.165, avg: 0.163", "ConsensusOrderedToCommit: max: 0.163, avg: 0.094", "ConsensusProposalToCommit: max: 0.327, avg: 0.257"]
Max non-epoch-change gap was: 2 rounds at version 5630618 (avg 0.00) [limit 4], 2.20s no progress at version 5630618 (avg 0.07s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.24s no progress at version 2384066 (avg 0.24s) [limit 16].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on 75cc050c2c35782fc994c2a7e61a7fb3ea88f71f ==> 4b41912ae24a0272a37a5be7119a6cd9d9f27fd3

Compatibility test results for 75cc050c2c35782fc994c2a7e61a7fb3ea88f71f ==> 4b41912ae24a0272a37a5be7119a6cd9d9f27fd3 (PR)
1. Check liveness of validators at old version: 75cc050c2c35782fc994c2a7e61a7fb3ea88f71f
compatibility::simple-validator-upgrade::liveness-check : committed: 10718.77 txn/s, latency: 3310.96 ms, (p50: 2500 ms, p70: 2600, p90: 2800 ms, p99: 13400 ms), latency samples: 417580
2. Upgrading first Validator to new version: 4b41912ae24a0272a37a5be7119a6cd9d9f27fd3
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 5266.33 txn/s, latency: 6508.93 ms, (p50: 7300 ms, p70: 7300, p90: 7400 ms, p99: 7400 ms), latency samples: 183260
3. Upgrading rest of first batch to new version: 4b41912ae24a0272a37a5be7119a6cd9d9f27fd3
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 5200.60 txn/s, latency: 6638.58 ms, (p50: 7300 ms, p70: 7400, p90: 7600 ms, p99: 7700 ms), latency samples: 180180
4. upgrading second batch to new version: 4b41912ae24a0272a37a5be7119a6cd9d9f27fd3
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 8192.29 txn/s, latency: 4180.56 ms, (p50: 4500 ms, p70: 4600, p90: 4700 ms, p99: 4900 ms), latency samples: 270780
5. check swarm health
Compatibility test for 75cc050c2c35782fc994c2a7e61a7fb3ea88f71f ==> 4b41912ae24a0272a37a5be7119a6cd9d9f27fd3 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking