Anvil: Fix everything to make it work with real assethub westend #508

dimartiro · 2026-01-27T15:22:19Z

Description

This PR enables forking from production chains like Westend Asset Hub by implementing dynamic transaction building and batch storage prefetching to address pallet index mismatches and reduce RPC latency.

Changes

Dynamic Transaction Building

When forking from a chain with a different runtime configuration, the hardcoded pallet index used by subxt_client::tx().revive().eth_transact() may not match the forked chain's pallet ordering, causing transaction failures. This PR introduces dynamic transaction building using subxt's dynamic_tx() which reads pallet indices from the runtime metadata at execution time, ensuring correct encoding regardless of the forked chain's configuration.

Batch Storage Prefetching

Each storage read from a remote chain requires an individual RPC call, causing significant latency during transaction validation. This PR implements batch prefetching using state_queryStorageAt to fetch multiple storage keys in a single RPC call. Before validating an Ethereum transaction, the sender's account info is prefetched to avoid sequential RPC round-trips.

Tests

Updated test URL from local zombienet to real Westend Asset Hub RPC endpoint
Added send_transaction_and_wait() and deploy_contract_and_wait() helpers that poll for transaction receipts with configurable timeouts, essential for reliable testing against remote networks with variable block times
Removed forking-tests feature flag - forking tests now run without special configuration

All forking tests pass against real Westend Asset Hub

alindima

I'm still not convinced that the prefetching is very useful at this point.
We only prefetch a couple of key/value pairs for one account, but then the block mining will probably need many more keys than this.

I wouldn't do premature optimisations unless you saw a clear improvement in testing. Do you have some numbers?

crates/anvil-polkadot/src/api_server/server.rs

crates/anvil-polkadot/src/substrate_node/lazy_loading/rpc_client.rs

crates/anvil-polkadot/src/substrate_node/service/backend.rs

alindima · 2026-02-02T09:02:19Z

crates/anvil-polkadot/tests/it/utils.rs

+    /// Execute an ethereum transaction and wait for its receipt.
+    /// This is useful for forking tests where transaction validation can take time
+    /// due to lazy loading of state from the remote chain.
+    pub async fn send_transaction_and_wait(


many tests already have similar behaviour by sending the transaction, mining a block and then doing wait_for_block_with_timeout. Is this any different?

it makes sense if you want a helper that bundles all this functionality into one, but then let's make use of the wait_for_block_with_timeout helper and remove the explicit sleep here. It shouldn't be needed.

Also, this function should work with automine enabled as well

+1: also, is there any specific reason why you are using this with 120 seconds timeout? It seems too high

If I'm not wrong the block import and the receipt indexing are decoupled, the block commits instantly but the ReceiptProvider indexes receipts asynchronously via a background task.
This means the receipt isn't queryable immediately after send_transaction, even with automine.
I verified this replacing the helper with send_transaction + mine + wait_for_block_with_timeout + get_transaction_receipt (the old pattern) causes all Westend forking tests to fail

The transaction receipt is available after Mine when Mine succeeds, the problem is that Mine itself probably fails because its inner call to wait_for_hash times out before the receipt indexer finishes processing (you can double-check if this is right by looking at the output of Mine). In this method, you keep polling and producing new blocks which is sub-optimal. Can you please try using a higher timeout inside wait_for_hash? We can use TIMEOUT_DURATION, which is currently set to 30 seconds. We can also choose to use a higher timeout only when forking is enabled and add a comment explaining why do we need a higher timeout in that case.

crates/anvil-polkadot/tests/it/forking.rs

crates/anvil-polkadot/tests/it/utils.rs

dimartiro · 2026-02-02T13:55:10Z

I'm still not convinced that the prefetching is very useful at this point. We only prefetch a couple of key/value pairs for one account, but then the block mining will probably need many more keys than this.

I wouldn't do premature optimisations unless you saw a clear improvement in testing. Do you have some numbers?

The improvement is only a few milliseconds in our tests, so we can remove it for now and revisit potential optimizations in future PRs

crates/anvil-polkadot/src/api_server/server.rs

re-gius · 2026-02-02T12:20:59Z

crates/anvil-polkadot/tests/it/utils.rs

+    /// Execute an ethereum transaction and wait for its receipt.
+    /// This is useful for forking tests where transaction validation can take time
+    /// due to lazy loading of state from the remote chain.
+    pub async fn send_transaction_and_wait(


+1: also, is there any specific reason why you are using this with 120 seconds timeout? It seems too high

crates/anvil-polkadot/src/substrate_node/service/backend.rs

crates/anvil-polkadot/src/api_server/server.rs

crates/anvil-polkadot/src/substrate_node/lazy_loading/backend/forked_lazy_backend.rs

crates/anvil-polkadot/src/substrate_node/service/backend.rs

crates/anvil-polkadot/tests/it/forking.rs

crates/anvil-polkadot/tests/it/utils.rs

crates/anvil-polkadot/tests/it/forking.rs

iulianbarbu

The forking tests fail locally for me. Are they passing on your end?


failures:

---- forking::test_fork_can_deploy_contract_from_westend stdout ----

thread 'forking::test_fork_can_deploy_contract_from_westend' panicked at crates/anvil-polkadot/tests/it/forking.rs:997:10:
Contract deployment receipt not found within timeout: RpcError { code: InternalError, message: "Internal error: Failed to submit transaction: RPC error: client error: ErrorObject { code: ServerError(1010), message: \"Invalid Transaction\", data: Some(RawValue(\"Inability to pay some fees (e.g. account balance too low)\")) }", data: None }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- forking::test_fork_impersonate_account_from_westend stdout ----

thread 'forking::test_fork_impersonate_account_from_westend' panicked at crates/anvil-polkadot/tests/it/forking.rs:1070:10:
Impersonated transaction receipt not found within timeout: RpcError { code: InternalError, message: "Internal error: Failed to submit transaction: RPC error: client error: ErrorObject { code: ServerError(1010), message: \"Invalid Transaction\", data: Some(RawValue(\"Inability to pay some fees (e.g. account balance too low)\")) }", data: None }

---- forking::test_fork_state_snapshotting_from_westend stdout ----

thread 'forking::test_fork_state_snapshotting_from_westend' panicked at crates/anvil-polkadot/tests/it/forking.rs:809:10:
Transaction receipt not found within timeout: RpcError { code: InternalError, message: "Internal error: Failed to submit transaction: RPC error: client error: ErrorObject { code: ServerError(1010), message: \"Invalid Transaction\", data: Some(RawValue(\"Inability to pay some fees (e.g. account balance too low)\")) }", data: None }

---- forking::test_fork_eth_get_nonce_from_westend stdout ----

thread 'forking::test_fork_eth_get_nonce_from_westend' panicked at crates/anvil-polkadot/tests/it/forking.rs:740:10:
Transaction receipt not found within timeout: RpcError { code: InternalError, message: "Internal error: Failed to submit transaction: RPC error: client error: ErrorObject { code: ServerError(1010), message: \"Invalid Transaction\", data: Some(RawValue(\"Inability to pay some fees (e.g. account balance too low)\")) }", data: None }

---- forking::test_fork_can_send_tx_from_westend stdout ----

thread 'forking::test_fork_can_send_tx_from_westend' panicked at crates/anvil-polkadot/tests/it/forking.rs:902:10:
Transaction receipt not found within timeout: RpcError { code: InternalError, message: "Internal error: Failed to submit transaction: RPC error: client error: ErrorObject { code: ServerError(1010), message: \"Invalid Transaction\", data: Some(RawValue(\"Inability to pay some fees (e.g. account balance too low)\")) }", data: None }

---- forking::test_fork_eth_get_code_from_westend stdout ----

thread 'forking::test_fork_eth_get_code_from_westend' panicked at crates/anvil-polkadot/tests/it/utils.rs:390:57:
called `Result::unwrap()` on an `Err` value: RpcError { code: InternalError, message: "Internal error: Failed to submit transaction: RPC error: client error: ErrorObject { code: ServerError(1010), message: \"Invalid Transaction\", data: Some(RawValue(\"Inability to pay some fees (e.g. account balance too low)\")) }", data: None }


failures:
    forking::test_fork_can_deploy_contract_from_westend
    forking::test_fork_can_send_tx_from_westend
    forking::test_fork_eth_get_code_from_westend
    forking::test_fork_eth_get_nonce_from_westend
    forking::test_fork_impersonate_account_from_westend
    forking::test_fork_state_snapshotting_from_westend

test result: FAILED. 114 passed; 6 failed; 0 ignored; 0 measured; 0 filtered out; finished in 89.72s

dimartiro · 2026-02-03T15:07:24Z

The forking tests fail locally for me. Are they passing on your end?

Yeah I noticed the same yesterday, the used to pass but I'm getting some 404 now, I'm working on debugging this, will let u know

…ry-polkadot into fix-westend_assethub-forking

dimartiro · 2026-02-09T12:40:10Z

The forking tests fail locally for me. Are they passing on your end?

I found the issue — it’s related to the latest runtime update on Westend, so I need to update polkadot-sdk to fix it.
To do that, I opened this PR to sync the feature branch with master.
Could you guys please merge it so I can unblock this one?

Thanks!

…end_assethub-forking

dimartiro · 2026-02-11T11:46:25Z

The forking tests fail locally for me. Are they passing on your end?

Fixed

crates/anvil-polkadot/src/api_server/server.rs

crates/anvil-polkadot/tests/it/main.rs

crates/anvil-polkadot/tests/it/utils.rs

…ion_and_wait

re-gius · 2026-02-12T15:22:24Z

crates/anvil-polkadot/src/api_server/server.rs

+        // Use dynamic transaction building to ensure the correct pallet index is used.
+        // The metadata in self.api comes from the runtime's WASM (via runtime API call),
+        // which is the forked chain's WASM when forking. This ensures correct pallet indices.
+        let payload_value = DynamicValue::from_bytes(transaction.0.clone());


Suggested change

let payload_value = DynamicValue::from_bytes(transaction.0.clone());

let payload_value = DynamicValue::from_bytes(transaction.0);

re-gius · 2026-02-12T15:45:22Z

crates/anvil-polkadot/tests/it/utils.rs

+    /// Execute an ethereum transaction and wait for its receipt.
+    /// This is useful for forking tests where transaction validation can take time
+    /// due to lazy loading of state from the remote chain.
+    pub async fn send_transaction_and_wait(


The transaction receipt is available after Mine when Mine succeeds, the problem is that Mine itself probably fails because its inner call to wait_for_hash times out before the receipt indexer finishes processing (you can double-check if this is right by looking at the output of Mine). In this method, you keep polling and producing new blocks which is sub-optimal. Can you please try using a higher timeout inside wait_for_hash? We can use TIMEOUT_DURATION, which is currently set to 30 seconds. We can also choose to use a higher timeout only when forking is enabled and add a comment explaining why do we need a higher timeout in that case.

dimartiro added 2 commits January 27, 2026 12:21

Fix everything to make it work with real assethub westend

67a4103

Fix clippy and fmt

b69a20d

iulianbarbu self-requested a review January 29, 2026 08:30

alindima reviewed Feb 2, 2026

View reviewed changes

alindima requested a review from re-gius February 2, 2026 09:12

re-gius reviewed Feb 2, 2026

View reviewed changes

iulianbarbu reviewed Feb 2, 2026

View reviewed changes

dimartiro added 5 commits February 3, 2026 12:14

Import HashMap and use it from there

76e36f2

Make prefetch_storage_keys private

6912de4

Remove sleep

90040dd

Fix test_fork_can_send_tx_from_westend

996509e

Merge branch 'feature/forking' of https://github.com/paritytech/found…

4b86134

…ry-polkadot into fix-westend_assethub-forking

dimartiro added 2 commits February 10, 2026 11:49

Merge remote-tracking branch 'upstream/feature/forking' into fix-west…

a9e5976

…end_assethub-forking

Remove keys prefetch

456a63b

dimartiro added 4 commits February 11, 2026 10:07

Remove unnecesary test

7d94ba2

Refactor to improve error handling

98a1f56

Check contract code

5d42a43

Check contract methods

9d5dc7b

re-gius reviewed Feb 11, 2026

View reviewed changes

dimartiro added 5 commits February 11, 2026 20:49

Prevent double rpc call

dc7adda

Add forking-support feature for forking tests

e934278

Improve error message

fc95dd7

Await block import notifications instead of sleeping in send_transact…

9a3aa96

…ion_and_wait

Remove forking-specific comments from transaction wait helpers

4e2383b

re-gius reviewed Feb 12, 2026

View reviewed changes

Skip polling in send_transaction_and_wait when automine is enabled

036240a

	let payload_value = DynamicValue::from_bytes(transaction.0.clone());
	let payload_value = DynamicValue::from_bytes(transaction.0);

Anvil: Fix everything to make it work with real assethub westend #508

Are you sure you want to change the base?

Anvil: Fix everything to make it work with real assethub westend #508

Uh oh!

Conversation

dimartiro commented Jan 27, 2026

Description

Changes

Dynamic Transaction Building

Batch Storage Prefetching

Tests

Uh oh!

alindima left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dimartiro commented Feb 2, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iulianbarbu left a comment

Choose a reason for hiding this comment

Uh oh!

dimartiro commented Feb 3, 2026

Uh oh!

dimartiro commented Feb 9, 2026

Uh oh!

dimartiro commented Feb 11, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants