[Hot State] Add separate StateMerkleDb for hot state

[wqfish]

---

Stack created with Sapling. Best reviewed with ReviewStack.

• -> [Hot State] Add separate StateMerkleDb for hot state #18367
• [Hot State] Use config to replace hard-coded parameters #18366
• [Hot State] Delete unused code #18365

[github-actions]

✅ Forge suite compat success on 7b91255b4cddd1c8ee6e251894e5c05e50d7f2ad ==> b1de5ad27cb44b540105600efe86ecc23f4755c7

Forge report malformed: Expecting property name enclosed in double quotes: line 5 column 1 (char 101)
'{\n  "metrics": [\n    {\n      "test_name": "compatibility::simple-validator-upgrade::liveness-check",\n[2025-12-22T01:19:21Z INFO  aptos_forge::report] Test Ok\n      "metric": "submitted_txn",\n      "value": 448600.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::liveness-check",\n      "metric": "expired_txn",\n      "value": 0.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::liveness-check",\n      "metric": "avg_tps",\n      "value": 13650.74132780132\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::liveness-check",\n      "metric": "avg_latency",\n      "value": 2542.078450735622\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::liveness-check",\n      "metric": "p50_latency",\n      "value": 2700.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::liveness-check",\n      "metric": "p90_latency",\n      "value": 3000.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::liveness-check",\n      "metric": "p99_latency",\n      "value": 3500.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::single-validator-upgrade",\n      "metric": "submitted_txn",\n      "value": 209940.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::single-validator-upgrade",\n      "metric": "expired_txn",\n      "value": 0.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::single-validator-upgrade",\n      "metric": "avg_tps",\n      "value": 6054.75292407985\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::single-validator-upgrade",\n      "metric": "avg_latency",\n      "value": 5606.723621034581\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::single-validator-upgrade",\n      "metric": "p50_latency",\n      "value": 6300.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::single-validator-upgrade",\n      "metric": "p90_latency",\n      "value": 6400.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::single-validator-upgrade",\n      "metric": "p99_latency",\n      "value": 6500.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::half-validator-upgrade",\n      "metric": "submitted_txn",\n      "value": 158540.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::half-validator-upgrade",\n      "metric": "expired_txn",\n      "value": 0.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::half-validator-upgrade",\n      "metric": "avg_tps",\n      "value": 4700.300125429024\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::half-validator-upgrade",\n      "metric": "avg_latency",\n      "value": 6676.297666204113\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::half-validator-upgrade",\n      "metric": "p50_latency",\n      "value": 5900.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::half-validator-upgrade",\n      "metric": "p90_latency",\n      "value": 12300.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::half-validator-upgrade",\n      "metric": "p99_latency",\n      "value": 12600.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::rest-validator-upgrade",\n      "metric": "submitted_txn",\n      "value": 337320.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::rest-validator-upgrade",\n      "metric": "expired_txn",\n      "value": 0.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::rest-validator-upgrade",\n      "metric": "avg_tps",\n      "value": 10278.107380560863\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::rest-validator-upgrade",\n      "metric": "avg_latency",\n      "value": 3276.8136369026442\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::rest-validator-upgrade",\n      "metric": "p50_latency",\n      "value": 3500.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::rest-validator-upgrade",\n      "metric": "p90_latency",\n      "value": 3700.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::rest-validator-upgrade",\n      "metric": "p99_latency",\n      "value": 3900.0\n    }\n  ],\n  "text": "Compatibility test results for 7b91255b4cddd1c8ee6e251894e5c05e50d7f2ad ==> b1de5ad27cb44b540105600efe86ecc23f4755c7 (PR)\\n1. Check liveness of validators at old version: 7b91255b4cddd1c8ee6e251894e5c05e50d7f2ad\\ncompatibility::simple-validator-upgrade::liveness-check : committed: 13650.74 txn/s, latency: 2542.08 ms, (p50: 2700 ms, p70: 2800, p90: 3000 ms, p99: 3500 ms), latency samples: 448600\\n2. Upgrading first Validator to new version: b1de5ad27cb44b540105600efe86ecc23f4755c7\\ncompatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6054.75 txn/s, latency: 5606.72 ms, (p50: 6300 ms, p70: 6300, p90: 6400 ms, p99: 6500 ms), latency samples: 209940\\n3. Upgrading rest of first batch to new version: b1de5ad27cb44b540105600efe86ecc23f4755c7\\ncompatibility::simple-validator-upgrade::half-validator-upgrade : committed: 4700.30 txn/s, latency: 6676.30 ms, (p50: 5900 ms, p70: 6200, p90: 12300 ms, p99: 12600 ms), latency samples: 158540\\n4. upgrading second batch to new version: b1de5ad27cb44b540105600efe86ecc23f4755c7\\ncompatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10278.11 txn/s, latency: 3276.81 ms, (p50: 3500 ms, p70: 3600, p90: 3700 ms, p99: 3900 ms), latency samples: 337320\\n5. check swarm health\\nCompatibility test for 7b91255b4cddd1c8ee6e251894e5c05e50d7f2ad ==> b1de5ad27cb44b540105600efe86ecc23f4755c7 passed\\nTest Ok"\n}'
Trailing Log Lines:
2. Upgrading first Validator to new version: b1de5ad27cb44b540105600efe86ecc23f4755c7
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6054.75 txn/s, latency: 5606.72 ms, (p50: 6300 ms, p70: 6300, p90: 6400 ms, p99: 6500 ms), latency samples: 209940
3. Upgrading rest of first batch to new version: b1de5ad27cb44b540105600efe86ecc23f4755c7
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 4700.30 txn/s, latency: 6676.30 ms, (p50: 5900 ms, p70: 6200, p90: 12300 ms, p99: 12600 ms), latency samples: 158540
4. upgrading second batch to new version: b1de5ad27cb44b540105600efe86ecc23f4755c7
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10278.11 txn/s, latency: 3276.81 ms, (p50: 3500 ms, p70: 3600, p90: 3700 ms, p99: 3900 ms), latency samples: 337320
5. check swarm health
Compatibility test for 7b91255b4cddd1c8ee6e251894e5c05e50d7f2ad ==> b1de5ad27cb44b540105600efe86ecc23f4755c7 passed
Test Ok

=== BEGIN JUNIT ===
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="forge" tests="1" failures="0" errors="0" uuid="9ebaedb8-a13b-4e1d-8043-348cb8e219ef">
    <testsuite name="compat" tests="1" disabled="0" errors="0" failures="0">
        <testcase name="compatibility::simple-validator-upgrade">
        </testcase>
    </testsuite>
</testsuites>
=== END JUNIT ===
[2025-12-22T01:19:21Z INFO  aptos_forge::backend::k8s::cluster_helper] Deleting namespace forge-compat-pr-18367: Some(NamespaceStatus { conditions: None, phase: Some("Terminating") })
[2025-12-22T01:19:21Z INFO  aptos_forge::backend::k8s::cluster_helper] aptos-node resources for Forge removed in namespace: forge-compat-pr-18367

test result: ok. 1 passed; 0 soft failed; 0 hard failed; 0 filtered out

Debugging output:
NAME                                         READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0                     1/1     Running     0          4m11s
aptos-node-1-validator-0                     1/1     Running     0          5m46s
aptos-node-2-validator-0                     1/1     Running     0          2m27s
aptos-node-3-validator-0                     1/1     Running     0          94s
forge-testnet-deployer-csm9k                 0/1     Completed   0          10m
genesis-aptos-genesis-eforge75d703df-jrvqc   0/1     Completed   0          9m34s

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite realistic_env_max_load success on b1de5ad27cb44b540105600efe86ecc23f4755c7

two traffics test: inner traffic : committed: 13679.66 txn/s, latency: 2756.22 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3600 ms), latency samples: 5093640
two traffics test : committed: 100.00 txn/s, latency: 715.51 ms, (p50: 700 ms, p70: 700, p90: 800 ms, p99: 900 ms), latency samples: 1700
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.254, avg: 2.185", "ConsensusProposalToOrdered: max: 0.167, avg: 0.165", "ConsensusOrderedToCommit: max: 0.043, avg: 0.040", "ConsensusProposalToCommit: max: 0.210, avg: 0.205"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.55s no progress at version 28815 (avg 0.07s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.31s no progress at version 2830868 (avg 0.31s) [limit 16].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking