Extract insights and statistics from replay validation logs

[sebtomba]

Summary

• RFCs: $\emptyset$.
• Categories: util

Changes

• Add validation of transaction submission against a Movement full-node to the L1-migration replay validation tool.
• Use a formal structure of logs for later processing.
• Add the shell script process_replay_logs.sh that cleans the logs, splits them into separate files, and then creates statistics.

Log structure tokens

@R: - Movement-Aptos replay of transactions
  S: - Submission
    AF: - Aptos submission failed, Movement submission succeeded (record error)
    MF: - Movement submission failed, Aptos submission succeeded
    BF: - Submission failed on both, Movement and Aptos (record error)
    BS: - Submission succeeded on both, Movement and Aptos

  E: - Execution
    AF: - Aptos execution failed, Movement execution succeeded (record error)
    MF: - Movement execution failed, Aptos execution succeeded (record error)
    BF: - Execution failed on both, Movement and Aptos (compare errors)
    BS: - Execution succeeded on both, Movement and Aptos (compare events and changes)

The transaction hash follows the structure tokens, and for execution, the entry function is also logged.

Example submission logs

@R:S:BS:(0x0ff4e14e24d5bd9646b4a7721fc927db5ad0f2be6d7444dc7f30187ed704c04e): ok
@R:S:BS:(0x365de55a94fe9124eb9b53e3285217b90964d344a670c9bfb2fc26eae611a202): ok
@R:S:AF:(0x6bc774903796b9c133855537523ec68ff7ba231f81124a8baaff09b8e55587ff): Error(VmError): Invalid transaction: Type: Validation Code: SENDING_ACCOUNT_DOES_NOT_EXIST

Example execution logs

@R:E:BF:(0xdc2fae9a038c15ab6275901abcf02ff9e91c799bb42d36e1c56d6c5bdf852ec7/entry:5d2b6a8b6478d86f62c9d3378e2a1fa265e85e69046946632d1fecae1940e851::strategy::perform_upkeep_fa): same error
@R:E:AF:(0x3bf692fe68d080bc734c26dd478220fc714a30c8461a07fa9899033119c4d757/entry:ccd2621d2897d407e06d18e6ebe3be0e6d9b61f1e809dd49360522b9105812cf::entry_public::redeem_v2): Unknown error Transaction committed on chain, but failed execution: Move abort in 0xccd2621d2897d407e06d18e6ebe3be0e6d9b61f1e809dd49360522b9105812cf::basic_ticket_v2: ERR_INVALID_TICKET_PORTFOLIO_COLLATERALS(0x4):
@R:E:BS:(0xb02bb02b193ed7e9ba98deb269ec2b76b40678eb512623d344b055226c17e7da/entry:03f7399a0d3d646ce94ee0badf16c4c3f3c656fe3a5e142e83b5ebc011aa8b3d::router::swap): ok

Files generated by the shell script

• clean_logs: contains all submission and execution logs produced by the replay tool. Everything up to the @R: token (inclusive) has been removed from each line.

Submission log files

• S_logs: extracted submission logs from the clean_logs file. The @S: prefix has been removed from each line.
• S_AF_logs: contains only submission logs where the submission to Aptos failed and on Movement succeeded.
• S_MF_logs: contains only submission logs where the submission to Movement failed and on Aptos succeeded.
• S_BS_logs: contains only submission logs where the submission to Aptos and Movement succeeded.
• S_BF_logs: contains only submission logs where the submission to Aptos and Movement failed.

Execution log files

• E_logs: extracted execution logs from the clean_logs file. The @E: prefix has been removed from each line.
• E_AF_logs: contains only execution logs where the execution on Aptos failed and on Movement succeeded.
• E_MF_logs: contains only execution logs where the execution on Movement failed and on Aptos succeeded.
• E_BS_logs: contains only execution logs where the execution on Aptos and Movement succeeded.
• E_BS_diff_logs: contains logs from E_BS_logs filtered for differences in execution outputs (events, changes).
• E_BF_logs: contains only execution logs where the execution on Aptos and Movement failed.
• E_BF_diff_logs: contains logs from E_BF_logs filtered for differences in execution errors.

Execution failure locations

• E_AF_entry: contains a unique sorted list of all transaction entry functions where the execution failed.
• E_AF_entry_freq: same as E_AF_entry but with failure occurrence frequencies.
• E_AF_abort: contains a unique sorted list of all functions where the execution has been aborted.
• E_AF_abort_freq: same as E_AF_abort but with abort occurrence frequencies.

Statistics

• S_logs_stats: contains a report with submission statistics.
• E_logs_stats: contains a report with execution statistics.

Example submission statistics:

Submission Statistics Report
============================

Base file: S_logs
Total submitted transactions: 998

Breakdown by category:
----------------------
Both succeeded: 975 (97.69%)
Both failed: 0 (0%)
Aptos submission failed: 23 (2.30%)
Movement submission failed: 0 (0%)

Example submission statistics:

Execution Statistics Report
===========================

Base file: E_logs
Total executed transactions: 975

Breakdown by category:
----------------------
Both succeeded: 893 (91.58%). Different outputs (events, changes): 5.
Both failed: 63 (6.46%). Different errors: 15
Aptos execution failed: 19 (1.94%)
Movement execution failed: 0 (0%)

Testing

1. Prepare the testing environment with a Movement full-node and an Aptos node.
2. Run the replay validation tool and redirect the log output to a file.
3. Execute the shell script with the logs from the validation tool.

Outstanding issues

None.