[REVM] Flaky Tests - Fails with Different Errors on Retry

[sekisamu]

Description

Repeatedly running the same test case, such as UniswapV2Router01.spec.ts, results in different error messages each time. This inconsistency makes it difficult to diagnose and fix the underlying issue.

Environment

polkadot-sdk version: 648cc16181998c8b53d51f1c996a866c81bed92f
Test repository: https://github.com/papermoonio/v2-periphery-polkadot/tree/revm

Steps to Reproduce

1. Build nodes

git clone https://github.com/paritytech/polkadot-sdk
cd polkadot-sdk
git checkout 648cc16181998c8b53d51f1c996a866c81bed92f

# Build eth-rpc
cargo build -p pallet-revive-eth-rpc --release

# Build revive-dev-node
cd substrate/frame/revive/dev-node/node
cargo build --release

2. Prepare test repository

git clone https://github.com/papermoonio/v2-periphery-polkadot
cd v2-periphery-polkadot
git checkout revm

3. Create .env file

Create .env file in the project root:

LOCAL_PRIV_KEY=0x5fb92d6e98884f76de468fa3f6278f8807c48bebc13595d45af5bdc4da702133

npm install

install the depencies:

npm install

5. Run tests

remember to start the local chain manually

npx hardhat test ./test/UniswapV2Router01.spec.ts --network local

The error:

❯ npx hardhat test ./test/UniswapV2Router01.spec.ts --network local


  UniswapV2Router{01,02}
    UniswapV2Router01
      ✔ factory, WETH
      ✔ addLiquidity (374ms)
      ✔ addLiquidityETH (285ms)
      ✔ removeLiquidity (632ms)
      ✔ removeLiquidityETH (758ms)
chainId from provider: 420420420n
      ✔ removeLiquidityWithPermit (514ms)
chainId from provider: 420420420n
      ✔ removeLiquidityETHWithPermit (675ms)
      swapExactTokensForTokens
        1) happy path
        ✔ amounts (261ms)
        ✔ gas
      swapTokensForExactTokens
        ✔ happy path (260ms)
        ✔ amounts (233ms)
      swapExactETHForTokens
        ✔ happy path (148ms)
        ✔ amounts (130ms)
        ✔ gas
      swapTokensForExactETH
        ✔ happy path (254ms)
        ✔ amounts (418ms)
      swapExactTokensForETH
        ✔ happy path (257ms)
        ✔ amounts (251ms)
      swapETHForExactTokens
        ✔ happy path (157ms)
        ✔ amounts (121ms)
    UniswapV2Router02
      ✔ factory, WETH
      ✔ addLiquidity (395ms)
      ✔ addLiquidityETH (280ms)
      ✔ removeLiquidity (624ms)
      ✔ removeLiquidityETH (775ms)
chainId from provider: 420420420n
      ✔ removeLiquidityWithPermit (483ms)
chainId from provider: 420420420n
      ✔ removeLiquidityETHWithPermit (611ms)
      swapExactTokensForTokens
        ✔ happy path (149ms)
        ✔ amounts (253ms)
        ✔ gas
      swapTokensForExactTokens
        2) happy path
        ✔ amounts (238ms)
      swapExactETHForTokens
        ✔ happy path (137ms)
        ✔ amounts (113ms)
        ✔ gas
      swapTokensForExactETH
        3) happy path
        ✔ amounts (246ms)
      swapExactTokensForETH
        ✔ happy path (270ms)
        ✔ amounts (248ms)
      swapETHForExactTokens
        ✔ happy path (163ms)
        ✔ amounts (136ms)


  39 passing (3m)
  3 failing

  1) UniswapV2Router{01,02}
       UniswapV2Router01
         swapExactTokensForTokens
           happy path:
     Error: transaction execution reverted (action="sendTransaction", data=null, reason=null, invocation=null, revert=null, transaction={ "data": "", "from": "0xf24FF3a9CF04c71Dbc94D0b566f7A27B94566cac", "to": "0x82726d8e34E105d82C9f4d898ffD561481486e01" }, receipt={ "_type": "TransactionReceipt", "blobGasPrice": null, "blobGasUsed": null, "blockHash": "0x3593eac46153cfc7d5d08a1ee98f8bc00d0b80f2ff45b21355cdd199641b4d24", "blockNumber": 207, "contractAddress": null, "cumulativeGasUsed": "0", "from": "0xf24FF3a9CF04c71Dbc94D0b566f7A27B94566cac", "gasPrice": "997008", "gasUsed": "44152630388", "hash": "0xcde11ee92c69a78666e11ec538dd473a84a7ee4ebbb3e475a301e6a638190ba9", "index": 1, "logs": [  ], "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", "root": null, "status": 0, "to": "0x82726d8e34E105d82C9f4d898ffD561481486e01" }, code=CALL_EXCEPTION, version=6.15.0)
      at makeError (node_modules/ethers/src.ts/utils/errors.ts:698:21)
      at assert (node_modules/ethers/src.ts/utils/errors.ts:719:25)
      at checkReceipt (node_modules/ethers/src.ts/providers/provider.ts:1607:19)
      at ContractTransactionResponse.wait (node_modules/ethers/src.ts/providers/provider.ts:1624:24)
      at processTicksAndRejections (node:internal/process/task_queues:105:5)
      at async ContractTransactionResponse.wait (node_modules/ethers/src.ts/contract/wrappers.ts:136:25)
      at async waitForPendingTransaction (node_modules/@nomicfoundation/hardhat-chai-matchers/src/internal/emit.ts:28:17)
      at async Context.<anonymous> (test/UniswapV2Router01.spec.ts:309:11)

  2) UniswapV2Router{01,02}
       UniswapV2Router02
         swapTokensForExactTokens
           happy path:
     Error: transaction execution reverted (action="sendTransaction", data=null, reason=null, invocation=null, revert=null, transaction={ "data": "", "from": "0xf24FF3a9CF04c71Dbc94D0b566f7A27B94566cac", "to": "0xe1F95d1e04C11BBfa5D23365cE202d117d7c157a" }, receipt={ "_type": "TransactionReceipt", "blobGasPrice": null, "blobGasUsed": null, "blockHash": "0xf34756fb8a3736fe65719d4c2fcdfcaf6458eee5bbb97f1be8b6ed1366d66646", "blockNumber": 851, "contractAddress": null, "cumulativeGasUsed": "0", "from": "0xf24FF3a9CF04c71Dbc94D0b566f7A27B94566cac", "gasPrice": "987664", "gasUsed": "44620783058", "hash": "0xd339e45f89b674423eeadead1a666eb85f998e47f5f78dcb45079aa73b627b51", "index": 1, "logs": [  ], "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", "root": null, "status": 0, "to": "0xe1F95d1e04C11BBfa5D23365cE202d117d7c157a" }, code=CALL_EXCEPTION, version=6.15.0)
      at makeError (node_modules/ethers/src.ts/utils/errors.ts:698:21)
      at assert (node_modules/ethers/src.ts/utils/errors.ts:719:25)
      at checkReceipt (node_modules/ethers/src.ts/providers/provider.ts:1607:19)
      at ContractTransactionResponse.wait (node_modules/ethers/src.ts/providers/provider.ts:1624:24)
      at processTicksAndRejections (node:internal/process/task_queues:105:5)
      at async ContractTransactionResponse.wait (node_modules/ethers/src.ts/contract/wrappers.ts:136:25)
      at async waitForPendingTransaction (node_modules/@nomicfoundation/hardhat-chai-matchers/src/internal/emit.ts:28:17)
      at async Context.<anonymous> (test/UniswapV2Router01.spec.ts:381:11)

  3) UniswapV2Router{01,02}
       UniswapV2Router02
         swapTokensForExactETH
           happy path:
     Error: transaction execution reverted (action="sendTransaction", data=null, reason=null, invocation=null, revert=null, transaction={ "data": "", "from": "0xf24FF3a9CF04c71Dbc94D0b566f7A27B94566cac", "to": "0xa8BbDc1e2E58AAdA10d396197f8e876D03722406" }, receipt={ "_type": "TransactionReceipt", "blobGasPrice": null, "blobGasUsed": null, "blockHash": "0x9305db1e8cc0d9046f3364a945cd460afaff4372922c4a6e65208b1fe902dc6e", "blockNumber": 989, "contractAddress": null, "cumulativeGasUsed": "0", "from": "0xf24FF3a9CF04c71Dbc94D0b566f7A27B94566cac", "gasPrice": "985658", "gasUsed": "46649164268", "hash": "0xf39dade9d1bdb91f727c7e78b8253aa9d441ba5c226bba6b46e8580a7a9aee6d", "index": 1, "logs": [  ], "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", "root": null, "status": 0, "to": "0xa8BbDc1e2E58AAdA10d396197f8e876D03722406" }, code=CALL_EXCEPTION, version=6.15.0)
      at makeError (node_modules/ethers/src.ts/utils/errors.ts:698:21)
      at assert (node_modules/ethers/src.ts/utils/errors.ts:719:25)
      at checkReceipt (node_modules/ethers/src.ts/providers/provider.ts:1607:19)
      at ContractTransactionResponse.wait (node_modules/ethers/src.ts/providers/provider.ts:1624:24)
      at processTicksAndRejections (node:internal/process/task_queues:105:5)
      at async ContractTransactionResponse.wait (node_modules/ethers/src.ts/contract/wrappers.ts:136:25)
      at async waitForPendingTransaction (node_modules/@nomicfoundation/hardhat-chai-matchers/src/internal/emit.ts:28:17)
      at async Context.<anonymous> (test/UniswapV2Router01.spec.ts:531:11)

If execute it second time, the error is:

npx hardhat test ./test/UniswapV2Router01.spec.ts --network local


  UniswapV2Router{01,02}
    UniswapV2Router01
      ✔ factory, WETH
      ✔ addLiquidity (393ms)
      ✔ addLiquidityETH (269ms)
      ✔ removeLiquidity (661ms)
      ✔ removeLiquidityETH (766ms)
chainId from provider: 420420420n
      ✔ removeLiquidityWithPermit (576ms)
chainId from provider: 420420420n
      ✔ removeLiquidityETHWithPermit (636ms)
      swapExactTokensForTokens
        ✔ happy path (376ms)
        ✔ amounts (302ms)
        ✔ gas
      swapTokensForExactTokens
        ✔ happy path (261ms)
        ✔ amounts (246ms)
      swapExactETHForTokens
        ✔ happy path (132ms)
        ✔ amounts (121ms)
        ✔ gas
      swapTokensForExactETH
        ✔ happy path (263ms)
        ✔ amounts (238ms)
      swapExactTokensForETH
        ✔ happy path (282ms)
        ✔ amounts (247ms)
      swapETHForExactTokens
        ✔ happy path (151ms)
        1) amounts
    UniswapV2Router02
      ✔ factory, WETH
      ✔ addLiquidity (388ms)
      ✔ addLiquidityETH (317ms)
      ✔ removeLiquidity (593ms)
      2) removeLiquidityETH
chainId from provider: 420420420n
      ✔ removeLiquidityWithPermit (482ms)
chainId from provider: 420420420n
      ✔ removeLiquidityETHWithPermit (597ms)
      swapExactTokensForTokens
        ✔ happy path (147ms)
        ✔ amounts (236ms)
        ✔ gas
      swapTokensForExactTokens
        ✔ happy path (263ms)
        ✔ amounts (264ms)
      swapExactETHForTokens
        ✔ happy path (131ms)
        ✔ amounts (116ms)
        ✔ gas
      swapTokensForExactETH
        ✔ happy path (260ms)
        ✔ amounts (238ms)
      swapExactTokensForETH
        ✔ happy path (286ms)
        ✔ amounts (251ms)
      swapETHForExactTokens
        ✔ happy path (155ms)
        ✔ amounts (124ms)


  40 passing (3m)
  2 failing

  1) UniswapV2Router{01,02}
       UniswapV2Router01
         swapETHForExactTokens
           amounts:
     Error: transaction execution reverted (action="sendTransaction", data=null, reason=null, invocation=null, revert=null, transaction={ "data": "", "from": "0xf24FF3a9CF04c71Dbc94D0b566f7A27B94566cac", "to": "0x742BaA3E3f968Ed8564839AF1ee7D153fb7411f7" }, receipt={ "_type": "TransactionReceipt", "blobGasPrice": null, "blobGasUsed": null, "blockHash": "0x550d580f29f6fdc9fe3edc4e55083da89789202d0a506e5f8ca85c5225e5633b", "blockNumber": 1690, "contractAddress": null, "cumulativeGasUsed": "0", "from": "0xf24FF3a9CF04c71Dbc94D0b566f7A27B94566cac", "gasPrice": "975594", "gasUsed": "47289579267", "hash": "0x15b1e40b34f47de7e052491622835bdc85df3069a0b12445c1cd33ddf32ddf09", "index": 1, "logs": [  ], "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", "root": null, "status": 0, "to": "0x742BaA3E3f968Ed8564839AF1ee7D153fb7411f7" }, code=CALL_EXCEPTION, version=6.15.0)
      at makeError (node_modules/ethers/src.ts/utils/errors.ts:698:21)
      at assert (node_modules/ethers/src.ts/utils/errors.ts:719:25)
      at checkReceipt (node_modules/ethers/src.ts/providers/provider.ts:1607:19)
      at ContractTransactionResponse.wait (node_modules/ethers/src.ts/providers/provider.ts:1624:24)
      at processTicksAndRejections (node:internal/process/task_queues:105:5)
      at async ContractTransactionResponse.wait (node_modules/ethers/src.ts/contract/wrappers.ts:136:25)
      at async waitForPendingTransaction (node_modules/@nomicfoundation/hardhat-chai-matchers/src/internal/emit.ts:28:17)
      at async Context.<anonymous> (test/UniswapV2Router01.spec.ts:698:11)

  2) UniswapV2Router{01,02}
       UniswapV2Router02
         removeLiquidityETH:
     Error: transaction execution reverted (action="sendTransaction", data=null, reason=null, invocation=null, revert=null, transaction={ "data": "", "from": "0xf24FF3a9CF04c71Dbc94D0b566f7A27B94566cac", "to": "0x2b372c7EE32e7Fb3b6D080c026fb4905e85A4072" }, receipt={ "_type": "TransactionReceipt", "blobGasPrice": null, "blobGasUsed": null, "blockHash": "0xd28a10deae6d5aa0d076e7d021166e0b902fb39874ea23ed3cdaf896b03d482a", "blockNumber": 1816, "contractAddress": null, "cumulativeGasUsed": "0", "from": "0xf24FF3a9CF04c71Dbc94D0b566f7A27B94566cac", "gasPrice": "973830", "gasUsed": "43572143638", "hash": "0x52b9647527fa56756d44f8c71bed9acbb9e8caa4ec6e822610e6bab443604e7b", "index": 1, "logs": [  ], "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", "root": null, "status": 0, "to": "0x2b372c7EE32e7Fb3b6D080c026fb4905e85A4072" }, code=CALL_EXCEPTION, version=6.15.0)
      at makeError (node_modules/ethers/src.ts/utils/errors.ts:698:21)
      at assert (node_modules/ethers/src.ts/utils/errors.ts:719:25)
      at checkReceipt (node_modules/ethers/src.ts/providers/provider.ts:1607:19)
      at ContractTransactionResponse.wait (node_modules/ethers/src.ts/providers/provider.ts:1624:24)
      at processTicksAndRejections (node:internal/process/task_queues:105:5)
      at async ContractTransactionResponse.wait (node_modules/ethers/src.ts/contract/wrappers.ts:136:25)
      at async waitForPendingTransaction (node_modules/@nomicfoundation/hardhat-chai-matchers/src/internal/emit.ts:28:17)
      at async Context.<anonymous> (test/UniswapV2Router01.spec.ts:187:9)

[albertov19]

I can confirm that with the latest master version that includes -> paritytech/polkadot-sdk#9939

All tests are passing consistently :)

[xermicus]

Thanks!

Closed by paritytech/polkadot-sdk#9939

[athei]

Apparently this is still happening. I am getting this error when following the steps here:

npx hardhat test ./test/UniswapV2Router01.spec.ts --network local
Error HH8: There's one or more errors in your config file:

  * Invalid account: #1 for network: local - Expected string, received undefined
  * Invalid account: #0 for network: ah - Expected string, received undefined

[sekisamu]

Apparently this is still happening. I am getting this error when following the steps here:

npx hardhat test ./test/UniswapV2Router01.spec.ts --network local
Error HH8: There's one or more errors in your config file:

  * Invalid account: #1 for network: local - Expected string, received undefined
  * Invalid account: #0 for network: ah - Expected string, received undefined

let me update the hardhat config to avoid account issue.

[sekisamu]

Apparently this is still happening. I am getting this error when following the steps here:

npx hardhat test ./test/UniswapV2Router01.spec.ts --network local
Error HH8: There's one or more errors in your config file:

  * Invalid account: #1 for network: local - Expected string, received undefined
  * Invalid account: #0 for network: ah - Expected string, received undefined

https://github.com/papermoonio/v2-periphery-polkadot/tree/revm updated.

[athei]

Did a git pull. Now its one less error:

npx hardhat test ./test/UniswapV2Router01.spec.ts --network local
Error HH8: There's one or more errors in your config file:

  * Invalid account: #0 for network: ah - Expected string, received undefined

To learn more about Hardhat's configuration, please go to https://hardhat.org/config/

For more info go to https://hardhat.org/HH8 or run Hardhat with --show-stack-traces

[albertov19]

I think you need to provide a private key for AH in the .env file

[athei]

Aye. Works now. I ran the removeLiquidity until I hit:

1) UniswapV2Router{01,02}
       UniswapV2Router02
         removeLiquidity:
     Error: transaction execution reverted (action="sendTransaction", data=null, reason=null, invocation=null, revert=null, transaction={ "data": "", "from": "0xf24FF3a9CF04c71Dbc94D0b566f7A27B94566cac", "to": "0xc344593F59d3C27cDE9C4b99CFac3FcA00347a90" }, receipt={ "_type": "TransactionReceipt", "blobGasPrice": null, "blobGasUsed": null, "blockHash": "0x6f453510927043ca52fdac36d58aab83d9901b5c800234bb313d6bd2205c065c", "blockNumber": 1620, "contractAddress": null, "cumulativeGasUsed": "0", "from": "0xf24FF3a9CF04c71Dbc94D0b566f7A27B94566cac", "gasPrice": "1000000", "gasUsed": "43725306812", "hash": "0x0b513f43aade42f81f2d589c08d61805724f4a840deb247b78612b321bf97d36", "index": 1, "logs": [  ], "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", "root": null, "status": 0, "to": "0xc344593F59d3C27cDE9C4b99CFac3FcA00347a90" }, code=CALL_EXCEPTION, version=6.15.0)

The logging shows

Drew too much from the txhold. fee=43237676812 available=38154798514 overdrawn_by=5082878298

This points to the fact that the estimated gas was not high enough. Or that the supplied gas to the tx was too low. Needs more investigation.

[albertov19]

I just tried these tests against the commit hash de740e1892f47e91c356858324f6e3e4b0f1be60 from Polkadot-SDK, and all 42 tests pass without any issues (tried 7 times, always passed).

I did not do any edits or modifications to the revive codebase.

[athei]

I had to try many more times than 7 in order to hit this issue. I'll also encountered this error in #194 after I fixed the OutOfGas there. So it seems to be a more general problem.

So I leave this open to track treack the flakiness and close 194 after fixing the out of gas error.

@pgherveou Can hit this error more consistently by using a debug profile for the node.