Skip to content

Conversation

@wqfish
Copy link
Contributor

@wqfish wqfish commented Dec 18, 2025

lg_prof_sample is set to 23 and hopefully performance impact is minimal. Benchmark is not showing
anything. If this causes any issues, it can still be overwritten by setting MALLOC_CONF env var on
the command line.


Note

Enables memory profiling by default with jemalloc and updates allocator deps.

  • Enable default jemalloc profiling by exporting malloc_conf (prof:true,lg_prof_sample:23) in aptos-node and execution/executor-benchmark on Unix; can be overridden via MALLOC_CONF
  • Change aptos-node crate attribute from #![forbid(unsafe_code)] to #![deny(unsafe_code)] to allow the unsafe malloc_conf symbol
  • Upgrade tikv-jemallocator, tikv-jemalloc-ctl, and tikv-jemalloc-sys to 0.6.1 in Cargo.toml and Cargo.lock

Written by Cursor Bugbot for commit ff7d429. This will update automatically on new commits. Configure here.

@wqfish wqfish force-pushed the pr18341 branch 2 times, most recently from c37f0fe to 0b5a8b1 Compare December 18, 2025 00:25
@wqfish wqfish added the CICD:run-e2e-tests when this label is present github actions will run all land-blocking e2e tests from the PR label Dec 18, 2025
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@wqfish wqfish marked this pull request as ready for review December 18, 2025 18:29
@wqfish wqfish requested a review from zekun000 December 18, 2025 18:29
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@wqfish wqfish changed the title Enable memory profiling by default Update jemalloc to latest version Jan 5, 2026
@wqfish wqfish changed the title Update jemalloc to latest version Enable memory profiling by default Jan 5, 2026
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

wqfish added 2 commits January 7, 2026 23:05
`lg_prof_sample` is set to 23 and hopefully performance impact is minimal. Benchmark is not showing
anything. If this causes any issues, it can still be overwritten by setting `MALLOC_CONF` env var on
the command line.
@wqfish wqfish enabled auto-merge (rebase) January 7, 2026 23:20
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 7, 2026

✅ Forge suite compat success on a09bb94430a970de7bc45fe0d29bd33fd2e5a7d4 ==> ff7d429b3fdc732e704d7183756d7e2b1c51bd05

Forge report malformed: Expecting ',' delimiter: line 3 column 6 (char 22)
'{\n  "metrics": [\n[2026-01-07T23:53:34Z INFO  aptos_forge::report] Test Ok\n    {\n      "test_name": "compatibility::simple-validator-upgrade::liveness-check",\n      "metric": "submitted_txn",\n      "value": 440200.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::liveness-check",\n      "metric": "expired_txn",\n      "value": 0.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::liveness-check",\n      "metric": "avg_tps",\n      "value": 13436.94148580632\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::liveness-check",\n      "metric": "avg_latency",\n      "value": 2591.607635165834\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::liveness-check",\n      "metric": "p50_latency",\n      "value": 2700.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::liveness-check",\n      "metric": "p90_latency",\n      "value": 3200.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::liveness-check",\n      "metric": "p99_latency",\n      "value": 3600.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::single-validator-upgrade",\n      "metric": "submitted_txn",\n      "value": 209160.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::single-validator-upgrade",\n      "metric": "expired_txn",\n      "value": 0.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::single-validator-upgrade",\n      "metric": "avg_tps",\n      "value": 6070.546884997074\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::single-validator-upgrade",\n      "metric": "avg_latency",\n      "value": 5578.515017211704\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::single-validator-upgrade",\n      "metric": "p50_latency",\n      "value": 6200.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::single-validator-upgrade",\n      "metric": "p90_latency",\n      "value": 6400.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::single-validator-upgrade",\n      "metric": "p99_latency",\n      "value": 6500.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::half-validator-upgrade",\n      "metric": "submitted_txn",\n      "value": 204780.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::half-validator-upgrade",\n      "metric": "expired_txn",\n      "value": 0.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::half-validator-upgrade",\n      "metric": "avg_tps",\n      "value": 5988.136855931146\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::half-validator-upgrade",\n      "metric": "avg_latency",\n      "value": 5654.040135755445\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::half-validator-upgrade",\n      "metric": "p50_latency",\n      "value": 6300.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::half-validator-upgrade",\n      "metric": "p90_latency",\n      "value": 6400.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::half-validator-upgrade",\n      "metric": "p99_latency",\n      "value": 6500.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::rest-validator-upgrade",\n      "metric": "submitted_txn",\n      "value": 332480.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::rest-validator-upgrade",\n      "metric": "expired_txn",\n      "value": 0.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::rest-validator-upgrade",\n      "metric": "avg_tps",\n      "value": 10178.769447373737\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::rest-validator-upgrade",\n      "metric": "avg_latency",\n      "value": 3286.458220043311\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::rest-validator-upgrade",\n      "metric": "p50_latency",\n      "value": 3500.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::rest-validator-upgrade",\n      "metric": "p90_latency",\n      "value": 3800.0\n    },\n    {\n      "test_name": "compatibility::simple-validator-upgrade::rest-validator-upgrade",\n      "metric": "p99_latency",\n      "value": 4000.0\n    }\n  ],\n  "text": "Compatibility test results for a09bb94430a970de7bc45fe0d29bd33fd2e5a7d4 ==> ff7d429b3fdc732e704d7183756d7e2b1c51bd05 (PR)\\n1. Check liveness of validators at old version: a09bb94430a970de7bc45fe0d29bd33fd2e5a7d4\\ncompatibility::simple-validator-upgrade::liveness-check : committed: 13436.94 txn/s, latency: 2591.61 ms, (p50: 2700 ms, p70: 3000, p90: 3200 ms, p99: 3600 ms), latency samples: 440200\\n2. Upgrading first Validator to new version: ff7d429b3fdc732e704d7183756d7e2b1c51bd05\\ncompatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6070.55 txn/s, latency: 5578.52 ms, (p50: 6200 ms, p70: 6300, p90: 6400 ms, p99: 6500 ms), latency samples: 209160\\n3. Upgrading rest of first batch to new version: ff7d429b3fdc732e704d7183756d7e2b1c51bd05\\ncompatibility::simple-validator-upgrade::half-validator-upgrade : committed: 5988.14 txn/s, latency: 5654.04 ms, (p50: 6300 ms, p70: 6400, p90: 6400 ms, p99: 6500 ms), latency samples: 204780\\n4. upgrading second batch to new version: ff7d429b3fdc732e704d7183756d7e2b1c51bd05\\ncompatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10178.77 txn/s, latency: 3286.46 ms, (p50: 3500 ms, p70: 3600, p90: 3800 ms, p99: 4000 ms), latency samples: 332480\\n5. check swarm health\\nCompatibility test for a09bb94430a970de7bc45fe0d29bd33fd2e5a7d4 ==> ff7d429b3fdc732e704d7183756d7e2b1c51bd05 passed\\nTest Ok"\n}'
Trailing Log Lines:
2. Upgrading first Validator to new version: ff7d429b3fdc732e704d7183756d7e2b1c51bd05
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6070.55 txn/s, latency: 5578.52 ms, (p50: 6200 ms, p70: 6300, p90: 6400 ms, p99: 6500 ms), latency samples: 209160
3. Upgrading rest of first batch to new version: ff7d429b3fdc732e704d7183756d7e2b1c51bd05
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 5988.14 txn/s, latency: 5654.04 ms, (p50: 6300 ms, p70: 6400, p90: 6400 ms, p99: 6500 ms), latency samples: 204780
4. upgrading second batch to new version: ff7d429b3fdc732e704d7183756d7e2b1c51bd05
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10178.77 txn/s, latency: 3286.46 ms, (p50: 3500 ms, p70: 3600, p90: 3800 ms, p99: 4000 ms), latency samples: 332480
5. check swarm health
Compatibility test for a09bb94430a970de7bc45fe0d29bd33fd2e5a7d4 ==> ff7d429b3fdc732e704d7183756d7e2b1c51bd05 passed
Test Ok

=== BEGIN JUNIT ===
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="forge" tests="1" failures="0" errors="0" uuid="be3dd4f5-062d-4563-8ed6-81e42b3c6194">
    <testsuite name="compat" tests="1" disabled="0" errors="0" failures="0">
        <testcase name="compatibility::simple-validator-upgrade">
        </testcase>
    </testsuite>
</testsuites>
=== END JUNIT ===
[2026-01-07T23:53:34Z INFO  aptos_forge::backend::k8s::cluster_helper] Deleting namespace forge-compat-pr-18341: Some(NamespaceStatus { conditions: None, phase: Some("Terminating") })
[2026-01-07T23:53:34Z INFO  aptos_forge::backend::k8s::cluster_helper] aptos-node resources for Forge removed in namespace: forge-compat-pr-18341

test result: ok. 1 passed; 0 soft failed; 0 hard failed; 0 filtered out

Debugging output:
NAME                                         READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0                     1/1     Running     0          4m25s
aptos-node-1-validator-0                     1/1     Running     0          6m5s
aptos-node-2-validator-0                     1/1     Running     0          2m44s
aptos-node-3-validator-0                     1/1     Running     0          104s
forge-testnet-deployer-nb9d5                 0/1     Completed   0          10m
genesis-aptos-genesis-eforge290689e3-6vzrl   0/1     Completed   0          9m48s

@github-actions
Copy link
Contributor

github-actions bot commented Jan 7, 2026

✅ Forge suite realistic_env_max_load success on ff7d429b3fdc732e704d7183756d7e2b1c51bd05

two traffics test: inner traffic : committed: 13509.57 txn/s, latency: 2791.02 ms, (p50: 2700 ms, p70: 2900, p90: 3000 ms, p99: 3600 ms), latency samples: 5032340
two traffics test : committed: 100.01 txn/s, latency: 756.10 ms, (p50: 700 ms, p70: 800, p90: 900 ms, p99: 1200 ms), latency samples: 1740
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.280, avg: 2.193", "ConsensusProposalToOrdered: max: 0.169, avg: 0.166", "ConsensusOrderedToCommit: max: 0.043, avg: 0.040", "ConsensusProposalToCommit: max: 0.212, avg: 0.207"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.95s no progress at version 41226 (avg 0.07s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.25s no progress at version 2352487 (avg 0.25s) [limit 16].
Test Ok

@github-actions
Copy link
Contributor

github-actions bot commented Jan 7, 2026

✅ Forge suite framework_upgrade success on a09bb94430a970de7bc45fe0d29bd33fd2e5a7d4 ==> ff7d429b3fdc732e704d7183756d7e2b1c51bd05

Compatibility test results for a09bb94430a970de7bc45fe0d29bd33fd2e5a7d4 ==> ff7d429b3fdc732e704d7183756d7e2b1c51bd05 (PR)
Upgrade the nodes to version: ff7d429b3fdc732e704d7183756d7e2b1c51bd05
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 2133.12 txn/s, submitted: 2141.92 txn/s, failed submission: 8.80 txn/s, expired: 8.80 txn/s, latency: 1375.04 ms, (p50: 1400 ms, p70: 1500, p90: 1800 ms, p99: 2400 ms), latency samples: 193981
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 2219.60 txn/s, submitted: 2228.25 txn/s, failed submission: 8.65 txn/s, expired: 8.65 txn/s, latency: 1314.36 ms, (p50: 1200 ms, p70: 1500, p90: 1800 ms, p99: 2400 ms), latency samples: 200182
5. check swarm health
Compatibility test for a09bb94430a970de7bc45fe0d29bd33fd2e5a7d4 ==> ff7d429b3fdc732e704d7183756d7e2b1c51bd05 passed
Upgrade the remaining nodes to version: ff7d429b3fdc732e704d7183756d7e2b1c51bd05
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 2203.15 txn/s, submitted: 2208.20 txn/s, failed submission: 5.05 txn/s, expired: 5.05 txn/s, latency: 1339.60 ms, (p50: 1200 ms, p70: 1500, p90: 1800 ms, p99: 2400 ms), latency samples: 200720
Test Ok

@wqfish wqfish merged commit c5ff83e into main Jan 8, 2026
93 of 117 checks passed
@wqfish wqfish deleted the pr18341 branch January 8, 2026 00:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CICD:run-e2e-tests when this label is present github actions will run all land-blocking e2e tests from the PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants