[release] Proposal to enable lazy loading

[georgemitenkov]

Description

Lazy loading proposal

How Has This Been Tested?

Key Areas to Review

Type of Change

• New feature
• Bug fix
• Breaking change
• Performance improvement
• Refactoring
• Dependency update
• Documentation update
• Tests

Which Components or Systems Does This Change Impact?

• Validator Node
• Full Node (API, Indexer, etc.)
• Move/Aptos Virtual Machine
• Aptos Framework
• Aptos CLI/SDK
• Developer Infrastructure
• Move Compiler
• Other (specify)

Checklist

• I have read and followed the CONTRIBUTING doc
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I identified and added all stakeholders and component owners affected by this change as reviewers
• I tested both happy and unhappy path of the functionality
• I have made corresponding changes to the documentation

[github-actions]

❌ Forge suite realistic_env_max_load hard failure on cf8a752b0d9fed5402c35232f7d972b0811e45d4

two traffics test: inner traffic : committed: 11935.39 txn/s, latency: 3156.91 ms, (p50: 3000 ms, p70: 3000, p90: 3500 ms, p99: 6300 ms), latency samples: 4538160
two traffics test : committed: 99.97 txn/s, latency: 1238.12 ms, (p50: 1100 ms, p70: 1300, p90: 1600 ms, p99: 3500 ms), latency samples: 1700
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.399, avg: 2.171", "ConsensusProposalToOrdered: max: 0.164, avg: 0.162", "ConsensusOrderedToCommit: max: 0.236, avg: 0.142", "ConsensusProposalToCommit: max: 0.395, avg: 0.304"]
Max non-epoch-change gap was: 2 rounds at version 21255 (avg 0.00) [limit 4], 2.21s no progress at version 21255 (avg 0.07s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.28s no progress at version 991901 (avg 0.26s) [limit 16].
Test Failed: {"errors":[{"name":"Check no fullnode failures","error":"Error! The number of fullnode failures was > 0 (1), but must be 0!","causes":["Error! The number of fullnode failures was > 0 (1), but must be 0!"]}]}
Trailing Log Lines:
test CompositeNetworkTest ... FAILED
Error: {"errors":[{"name":"Check no fullnode failures","error":"Error! The number of fullnode failures was > 0 (1), but must be 0!","causes":["Error! The number of fullnode failures was > 0 (1), but must be 0!"]}]}
Test Statistics: 
two traffics test: inner traffic : committed: 11935.39 txn/s, latency: 3156.91 ms, (p50: 3000 ms, p70: 3000, p90: 3500 ms, p99: 6300 ms), latency samples: 4538160
two traffics test : committed: 99.97 txn/s, latency: 1238.12 ms, (p50: 1100 ms, p70: 1300, p90: 1600 ms, p99: 3500 ms), latency samples: 1700
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.399, avg: 2.171", "ConsensusProposalToOrdered: max: 0.164, avg: 0.162", "ConsensusOrderedToCommit: max: 0.236, avg: 0.142", "ConsensusProposalToCommit: max: 0.395, avg: 0.304"]
Max non-epoch-change gap was: 2 rounds at version 21255 (avg 0.00) [limit 4], 2.21s no progress at version 21255 (avg 0.07s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.28s no progress at version 991901 (avg 0.26s) [limit 16].
Test Failed: {"errors":[{"name":"Check no fullnode failures","error":"Error! The number of fullnode failures was > 0 (1), but must be 0!","causes":["Error! The number of fullnode failures was > 0 (1), but must be 0!"]}]}

=== BEGIN JUNIT ===
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="forge" tests="1" failures="1" errors="0" uuid="27bed54f-a231-4639-900e-bc038583a815">
    <testsuite name="local" tests="1" disabled="0" errors="0" failures="1">
        <testcase name="CompositeNetworkTest(network:multi-region-network-emulation(two traffics test)) with ">
            <failure message="{&quot;errors&quot;:[{&quot;name&quot;:&quot;Check no fullnode failures&quot;,&quot;error&quot;:&quot;Error! The number of fullnode failures was &gt; 0 (1), but must be 0!&quot;,&quot;causes&quot;:[&quot;Error! The number of fullnode failures was &gt; 0 (1), but must be 0!&quot;]}]}"/>
        </testcase>
    </testsuite>
</testsuites>
=== END JUNIT ===

Swarm logs can be found here: See fgi output for more information.
[2025-10-06T20:03:52Z INFO  aptos_forge::backend::k8s::cluster_helper] Deleting namespace forge-e2e-pr-17779: Some(NamespaceStatus { conditions: None, phase: Some("Terminating") })
[2025-10-06T20:03:52Z INFO  aptos_forge::backend::k8s::cluster_helper] aptos-node resources for Forge removed in namespace: forge-e2e-pr-17779

failures:
    CompositeNetworkTest

test result: FAILED. 0 passed; 0 soft failed; 1 hard failed; 0 filtered out

Debugging output:
NAME                                         READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge0517bef2-0       1/1     Running     0          13m
aptos-node-0-validator-0                     1/1     Running     0          13m
aptos-node-1-fullnode-eforge0517bef2-0       1/1     Running     0          13m
aptos-node-1-validator-0                     1/1     Running     0          13m
aptos-node-2-fullnode-eforge0517bef2-0       1/1     Running     0          13m
aptos-node-2-validator-0                     1/1     Running     0          13m
aptos-node-3-fullnode-eforge0517bef2-0       1/1     Running     0          13m
aptos-node-3-validator-0                     1/1     Running     0          13m
aptos-node-4-fullnode-eforge0517bef2-0       1/1     Running     0          13m
aptos-node-4-validator-0                     1/1     Running     0          13m
aptos-node-5-validator-0                     1/1     Running     0          13m
aptos-node-6-validator-0                     1/1     Running     0          13m
forge-testnet-deployer-zdtpt                 0/1     Completed   0          13m
genesis-aptos-genesis-eforge0517bef2-r46zq   0/1     Completed   0          13m

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite framework_upgrade success on 8130c12cc7cdbc4b4a4cf991fe204aedc666dea3 ==> cf8a752b0d9fed5402c35232f7d972b0811e45d4

Compatibility test results for 8130c12cc7cdbc4b4a4cf991fe204aedc666dea3 ==> cf8a752b0d9fed5402c35232f7d972b0811e45d4 (PR)
Upgrade the nodes to version: cf8a752b0d9fed5402c35232f7d972b0811e45d4
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1781.88 txn/s, submitted: 1790.01 txn/s, failed submission: 8.13 txn/s, expired: 8.13 txn/s, latency: 1656.29 ms, (p50: 1500 ms, p70: 1800, p90: 2100 ms, p99: 2400 ms), latency samples: 162101
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1148.21 txn/s, submitted: 1151.76 txn/s, failed submission: 3.56 txn/s, expired: 3.56 txn/s, latency: 2650.50 ms, (p50: 1500 ms, p70: 1800, p90: 3000 ms, p99: 13600 ms), latency samples: 103261
5. check swarm health
Compatibility test for 8130c12cc7cdbc4b4a4cf991fe204aedc666dea3 ==> cf8a752b0d9fed5402c35232f7d972b0811e45d4 passed
Upgrade the remaining nodes to version: cf8a752b0d9fed5402c35232f7d972b0811e45d4
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1795.50 txn/s, submitted: 1802.95 txn/s, failed submission: 7.44 txn/s, expired: 7.44 txn/s, latency: 1636.86 ms, (p50: 1500 ms, p70: 1800, p90: 2000 ms, p99: 3300 ms), latency samples: 164041
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on 8130c12cc7cdbc4b4a4cf991fe204aedc666dea3 ==> cf8a752b0d9fed5402c35232f7d972b0811e45d4

Compatibility test results for 8130c12cc7cdbc4b4a4cf991fe204aedc666dea3 ==> cf8a752b0d9fed5402c35232f7d972b0811e45d4 (PR)
1. Check liveness of validators at old version: 8130c12cc7cdbc4b4a4cf991fe204aedc666dea3
compatibility::simple-validator-upgrade::liveness-check : committed: 7873.12 txn/s, latency: 4192.50 ms, (p50: 4600 ms, p70: 4700, p90: 4800 ms, p99: 5100 ms), latency samples: 266720
2. Upgrading first Validator to new version: cf8a752b0d9fed5402c35232f7d972b0811e45d4
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 4868.78 txn/s, latency: 7028.78 ms, (p50: 7800 ms, p70: 7900, p90: 7900 ms, p99: 8000 ms), latency samples: 173740
3. Upgrading rest of first batch to new version: cf8a752b0d9fed5402c35232f7d972b0811e45d4
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 4947.62 txn/s, latency: 6929.18 ms, (p50: 7700 ms, p70: 7800, p90: 7800 ms, p99: 7900 ms), latency samples: 175700
4. upgrading second batch to new version: cf8a752b0d9fed5402c35232f7d972b0811e45d4
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 7682.02 txn/s, latency: 4371.73 ms, (p50: 4700 ms, p70: 4900, p90: 5100 ms, p99: 5600 ms), latency samples: 260380
5. check swarm health
Compatibility test for 8130c12cc7cdbc4b4a4cf991fe204aedc666dea3 ==> cf8a752b0d9fed5402c35232f7d972b0811e45d4 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking