POC/WIP block simulate endpoint

[rdeioris]

Description

This POC tries to add the ability to re-run the transactions in a block in a sort of "simulation mode".

The trick used here is to temporarily remove references in the MARF of the currently simulated block, and re-run all of the transactions in the context of the parent block.

Note that the code still needs to be cleaned up (lot on unwrap() around)

The output of the (to be authenticated) endpoint /v3/blocks/simulate/index_block_hash is a handy json with a snapshot of the execution:

{
  "block_id": "ff38f6513b3ea16f778b330e94c81c605cba19ddc6dfd972e7da596bf86d4c61",
  "block_hash": "71b4da21dbabbff608b60e3cfa065047668724422e8dd7f22fa5d46ae2b58f0d",
  "parent_block_id": "08cb78bb64a2b015ced23ef5408c2f11a904e8a6c0321280a2f123dc6fd9f205",
  "consensus_hash": "2cd83cba6930e50fe81d265c6f14d248c93f3de3",
  "fees": 300,
  "tx_merkle_root": "415f8b4820e4c3ac0fe776a966601cf71cd300010a7b999a2d843792f7367fea",
  "state_index_root": "d3fcf3bd337c52438850a874681a07d103414d2f676ab508a56e31d9f2192e4e",
  "timestamp": 1735932676,
  "miner_signature": "00fb17a5efc8038ed639db07ed94066aacbd19fdb62752125bda9fb560c9484590272fe2c76b1adffcb6ce9d7ebf794d0fe1d0f3a6ce63bce4db45cd0c3dc74578",
  "signer_signature": [
    "00420e883f5ae2a407a74322f0442ff31152c213917465c78e9b57645938f058c459a66406fbf384c6c7e54cb4b8654e1961ce961d43215e9d2b0afa58a03ad21a",
    "00860925412d933f887c402c7e3c8b9063d3598e5eb94b0c9206db4c0aed88a5926096e3f6e10cc618a170c23a9e20615a6eff07ebcdeb261ca9ae3f7a921955dc"
  ],
  "transactions": [
    {
      "txid": "a50d7773f19bd4a87baf5ecc833a2b79623a436b72732861b78a8ebcd54d1cef",
      "tx_index": 0,
      "data": {
        "version": "Testnet",
        "chain_id": 2147483648,
        "auth": {
          "Standard": {
            "Singlesig": {
              "hash_mode": "P2PKH",
              "signer": "841f9b4dd30ae02fd816cc225c59d1b00b9c3a3a",
              "nonce": 149,
              "tx_fee": 300,
              "key_encoding": "Compressed",
              "signature": "0103eb829e6b6f681e505875f18793187246fa4e81dd63fc818ea9a16e85332e7944fc3bcbe0609d43ba0caf90de0e44a9e7fba8342e386a25579a2b854f6fa520"
            }
          }
        },
        "anchor_mode": "Any",
        "post_condition_mode": "Deny",
        "post_conditions": [],
        "payload": {
          "TokenTransfer": [
            {
              "Standard": [
                26,
                [
                  180,
                  253,
                  174,
                  152,
                  182,
                  75,
                  156,
                  214,
                  201,
                  67,
                  111,
                  59,
                  150,
                  85,
                  88,
                  150,
                  106,
                  254,
                  137,
                  11
                ]
              ]
            },
            1000,
            "00000000000000000000000000000000000000000000000000000000000000000000"
          ]
        }
      },
      "hex": "80800000000400841f9b4dd30ae02fd816cc225c59d1b00b9c3a3a0000000000000095000000000000012c000103eb829e6b6f681e505875f18793187246fa4e81dd63fc818ea9a16e85332e7944fc3bcbe0609d43ba0caf90de0e44a9e7fba8342e386a25579a2b854f6fa52003020000000000051ab4fdae98b64b9cd6c9436f3b965558966afe890b00000000000003e800000000000000000000000000000000000000000000000000000000000000000000",
      "result": {
        "Response": {
          "committed": true,
          "data": {
            "Bool": true
          }
        }
      },
      "stx_burned": 0,
      "execution_cost": {
        "write_length": 0,
        "write_count": 0,
        "read_length": 0,
        "read_count": 0,
        "runtime": 0
      },
      "events": [
        {
          "committed": true,
          "event_index": 0,
          "stx_transfer_event": {
            "amount": "1000",
            "memo": "00000000000000000000000000000000000000000000000000000000000000000000",
            "recipient": "ST2TFVBMRPS5SSNP98DQKQ5JNB2B6NZM91C4K3P7B",
            "sender": "ST221Z6TDTC5E0BYR2V624Q2ST6R0Q71T78WTAX6H"
          },
          "txid": "0xa50d7773f19bd4a87baf5ecc833a2b79623a436b72732861b78a8ebcd54d1cef",
          "type": "stx_transfer_event"
        }
      ]
    }
  ]
}

Applicable issues

Additional info (benefits, drawbacks, caveats)

Checklist

• Test coverage for new or modified code paths
• Changelog is updated
• Required documentation changes (e.g., docs/rpc/openapi.yaml and rpc-endpoints.md for v2 endpoints, event-dispatcher.md for new events)
• New clarity functions have corresponding PR in clarity-benchmarking repo
• New integration test(s) added to bitcoin-tests.yml

[jcnelson]

So this doesn't work for Stacks 2.x replay?

[rdeioris]

currently yes, but I would like to cover pre-nakamoto too

[jcnelson]

You shouldn't need to touch the MARF at all. Just process the block, and roll back the Clarity transaction. Nothing hits the MARF until you call ClarityTx::precommit_to_block() and PreCommitClarityBlock::commit(). You can instead just call ClarityTx::rollback_block() at the end of processing.

[jcnelson]

There's a lot more to processing a block than processing its transactions. Specifically, there's stuff in the block header that a transaction can access, so you'll need to do the relevant block-processing as well. Please see crate::net::api::postblock_proposal::NakamotoBlockProposal for inspiration -- in particular, it uses a NakamotoBlockBuilder to set up the ClarityTx before processing transactions (and this code path uses the same code path that the consensus-critical block-processing logic uses).

[rdeioris]

Yep, I can definitely reuse the BlockBuilder logic, but again, it interally calls inner_setup_block that calls begin_block that checks for the new block presence in the MARF

[rdeioris]

You shouldn't need to touch the MARF at all. Just process the block, and roll back the Clarity transaction. Nothing hits the MARF until you call ClarityTx::precommit_to_block() and PreCommitClarityBlock::commit(). You can instead just call ClarityTx::rollback_block() at the end of processing.

I am probably missing something, but when the clarity_tx starts, it calls begin_block then calls self.datastore.begin that finally calls self.storage.has_block for checking if the marf already has entries for the new block (and given this is a simulation for an already existent block in the chain, it always fails).