Missing and delayed blocks on parachains

[josepot]

This issue is a followup of this discussion.

On parachains smoldot reports new blocks with a considerable delay, and it never seems to report blocks that are not part of the current best-block chain.

I have created a small script to dig deeper into this.

What the script does is to subscribe to the new blocks received by both the smoldot and a centralized RPC for some blocks, and at the end it compares the blocks that smoldot did not report.

I'm attaching the logs of a particular run:

• Smoldot low-level logs.
• Smoldot JSON-RPC logs (the JSON-RPC messages sent/received between the smoldot chain and PAPI)
• Websocket JSON-RPC logs

And this is the output of said run:

Blocks that smoldot did not report:
{
  hash: "0xed87ee644095072d48c1b9eb0b023284dc4e4fceb68fadd75e17b771ca5edaa5",
  height: 9484930,
}
{
  hash: "0x816fcd370dc0479bc5c7cccd9b4b7e7288c67e764e32bc62861da7f20fe34a18",
  height: 9484931,
}
{
  hash: "0xfe46486dc4016f090fa6bd1059d7ed85bcd52ca929314f4bc16d138bcbaadcee",
  height: 9484932,
}
{
  hash: "0xaa2ed7ad331db94cb8db95588fd1566da99d2994afe27f5bbc5ef56ca959b9ca",
  height: 9484932,
}
{
  hash: "0x25f11905d5459bc9d6b98719f12837feabe60160edd12edb7d20e3798c1de666",
  height: 9484934,
}
{
  hash: "0xf62e41976f7749f21966451d7454732e608956607f49c22fedb86db92fce2f5f",
  height: 9484936,
}
{
  hash: "0xea1cd36afacd2aeddc97f75c6f49fe5335c2c42cb9fbbf88f3660c71de6334e6",
  height: 9484936,
}
{
  hash: "0xb9cda13896522a63977509978e147abdfa316ff5c11d70425a4d709a8d51d27c",
  height: 9484937,
}
{
  hash: "0xfd3d304075efbe874f66c7a5b756d811303432dca283ecb682ba224acc046a0c",
  height: 9484949,
}
{
  hash: "0xef578e60ab5ba3ecf603a371b2539e8e4b8df3cf361c51fc0258c83ace3da829",
  height: 9484953,
}
{
  hash: "0x07c0a6de0dec62ed232049b7939cb939f3264e521d503becf3b2ee131be98935",
  height: 9484957,
}
{
  hash: "0x8ebe28fd4fb82eb05f03fed86ee897de5f47b0d990643fe17f45afb320bc8dd6",
  height: 9484958,
}

In the smoldot logs, we can see that smoldot actually received all those blocks that it didn't report. For instance:

2025-08-14T13:23:06.662Z (4)-network
block-announce-received; chain=asset-hub-polkadot, peer_id=12D3KooWLHqbcQtoBygf7GJgVjVa3TaeLuf7VbicNdooaCmQM2JZ, block_hash=0xed87…daa5, is_best=false

...

2025-08-14T13:23:06.711Z (4)-network
block-announce-received; chain=asset-hub-polkadot, peer_id=12D3KooWL8CyLww3m3pRySQGGYGNJhWDMqko3j5xi67ckP7hDUvo, block_hash=0xed87…daa5, is_best=false

In fact, it received them practically at the same time as the Websocket RPC reported them:

1-2025-08-14T13:23:06.668Z-<<-{"jsonrpc":"2.0","method":"chainHead_v1_followEvent","params":{"subscription":"GeazAC1aFkxQQIgq","result":{"event":"newBlock","blockHash":"0xed87ee644095072d48c1b9eb0b023284dc4e4fceb68fadd75e17b771ca5edaa5","parentBlockHash":"0x98b95086cf2f4e8a8811d8f59eaf64cdf55f2440e5dbae9904a5fb4cd1ec975f","newRuntime":null}}}

Also, looking at the logs of the Smoldot JSON-RPC logs we can appreciate 2 things that seem to indicate that there is an issue:

1. The prunedBlockHashes on the finalized event is always an empty array.
2. For the blocks that smoldot actually reported via the newBlock event, all those blocks are always reported at least 12secs after smoldot actually received them. For instance:

• Block 0xf9114a1759a98cf34ce6a4669c651895a648afa9a8f29306f4e2e3f88b9b24db, 12 secs delay:

2025-08-14T13:28:30.680Z (4)-network
block-announce-received; chain=asset-hub-polkadot, peer_id=12D3KooWKKwTrkSUckCcXg5Bm65TdbU4BLwwsSufZFA4GXayxp4N, block_hash=0xf911…24db, is_best=false

1-2025-08-14T13:28:42.411Z-<<-{"jsonrpc":"2.0","method":"chainHead_v1_followEvent","params":{"subscription":"CvsnHxNCYRW8LBBiiUfok4x3nGkjc6R54XR8SSN532qo","result":{"event":"newBlock","blockHash":"0xf9114a1759a98cf34ce6a4669c651895a648afa9a8f29306f4e2e3f88b9b24db","parentBlockHash":"0xbe1969cb6d5d723a371ff63fa6297568246b7eebe2e69c161af0d5d0d8ad8d6f","newRuntime":null}}}

• Block 0xbe1969cb6d5d723a371ff63fa6297568246b7eebe2e69c161af0d5d0d8ad8d6f, 12 secs delay:

2025-08-14T13:28:18.708Z (4)-network
block-announce-received; chain=asset-hub-polkadot, peer_id=12D3KooWEyGg3oUwYfaLWM5AJ2pvXCUxBuXNapX1tQXLsbDmMV6z, block_hash=0xbe19…8d6f, is_best=true

1-2025-08-14T13:28:30.430Z-<<-{"jsonrpc":"2.0","method":"chainHead_v1_followEvent","params":{"subscription":"CvsnHxNCYRW8LBBiiUfok4x3nGkjc6R54XR8SSN532qo","result":{"event":"newBlock","blockHash":"0xbe1969cb6d5d723a371ff63fa6297568246b7eebe2e69c161af0d5d0d8ad8d6f","parentBlockHash":"0xe5b841be04f87403f4cfca40dbe76730d62a091bdf8190d8ff74cebef9625069","newRuntime":null}}}

This seems to suggest that they are being reported one block behind of the actual tip of the chain.

[tomaka]

Ah, sorry, in the discussion I had for some reason assumed that this was on the relay chain.

For parachains, this is unfortunately the intended behavior. Smoldot doesn't actually take into account the block announces it receives from the parachain except to track which node knows which block. Instead, it only takes into consideration the parablocks in the relay chain. Therefore, a parachain block has to be included in the relay chain first before smoldot reports it.

---

The reason why it's so is that smoldot currently doesn't assume that parachains are necessarily protocol-compatible with Substrate.

When it comes to relay chains, smoldot is able to verify the signature of new (non-finalized) blocks and can thus ignore blocks that are completely invalid (as in invented out of thin air). And since validators get punished if they author more than one block at a time, it also more or less has the guarantee that validators can't perform attack by crafting malicious blocks (they still could, but it would be very risky and luck-based). Beyond the additional block-validity guarantees, it also guarantees that smoldot can't get spam-attacked.

But when it comes to parachains, we are in the unknown. If the parachain is built using Substrate, then the same guarantees exist on it as well. But we can't know whether it was built with Substrate. Or maybe it was built in Substrate but these guarantees were disabled by the authors of the parachain for some reason. And therefore we err on the side of caution and assume that all parachain blocks that haven't made it to the relay chain are potentially malicious.

Maybe however the risks are exaggerated, and it would be overall better to make smoldot assume that parachains are safe, I'm not sure.

[josepot]

Maybe however the risks are exaggerated, and it would be overall better to make smoldot assume that parachains are safe, I'm not sure.

Yeah... I mean, if a block is "completely invalid", then said block will just get pruned fairly soon, right?

Maybe there could be an option when adding the chain like onlyParaBlocks (or something like that), so that the user can decide whether they are interested in receiving potentially "unsafe" blocks?

There are many use cases for which a user would be interested in knowing the latest tip of the chain. So, I think that there should be a way to allow for this.

[josepot]

Another idea would be to check in the received header of the parachain-block whether the block was produced by a supported production mechanism (i.e AURA) and if it is, then verify that it's legit.

If the block is not produced by a "supported" production-engine, then fallback to only accepting parablocks.

What do you think?

I mean, realistically speaking, I think that almost all (if not all) parachains use AURA. So, I think that should do the trick.

[tomaka]

To give some context, the original plan was that smoldot/substrate-connect was shared by multiple web pages. Web page 1 could use smoldot to connect to a parachain, while at the same time web page 2 would use the same instance of smoldot to connect to a different parachain.

In this situation, it's important that web pages aren't able to crash smoldot, as otherwise web page 2 could for example prevent web page 1 from working.
If you connect to a malicious parachain that says it uses AURA, and that smoldot assumes that all the properties of AURA are there, but actually the parachain uses a tweaked version of AURA, you can easily spam-attack and crash smoldot.
That's the reason why smoldot doesn't assume that simply because a parachain says it uses AURA doesn't necessarily mean it uses the AURA that is in the Substrate repository.

That's also why, in this context, smoldot/substrate-connect originally was only able to connect to the known-to-be-correct relay chains such as Polkadot and Kusama.

If, however, we have one instance of smoldot per web page, then it's not problematic if smoldot crashes as a result of the web page doing something malicious, and then I guess that we could do that.

[josepot]

Yeah, I'm 99.99% sure that Parity has completely abandoned the effort on substrate-connect. However, just in case that work is ever resumed, we could have the option of onlyParaBlocks (or similar) when adding a chain. If smoldot made that option available, then if there is a shared instance of smoldot, the one responsible for multiplexing it (eg the browser extension) could make sure that all chains are added with that option set to true.

[josepot]

Hi @tomaka ! Now that the balances, staking, and pretty much all state that's relevan to end-users has been migrated from the relay-chain to asset-hub... It would be very nice to have the ability to have the onlyParaBlocks option. Do you think that it would be viable to implement this? 🙏

[tomaka]

So right now I have a big file of 1500 lines of code with all the syncing behavior for relay chains, and another big file of 1400 lines of code with all the syncing behavior for the parachain. These two files have required a lot of bug fixing over time.

I've tried to look into how I could adjust either file in order to support parachains-with-this-option, but unfortunately the behavior is quite different from both relay chains and from parachains-without-this-option, and it would probably represent a third different syncing behavior to be written from scratch.

[bkchr]

If you connect to a malicious parachain that says it uses AURA, and that smoldot assumes that all the properties of AURA are there, but actually the parachain uses a tweaked version of AURA, you can easily spam-attack and crash smoldot.

If this would crash smoldot, it would mean you don't even need any secondary page? Because smoldot may connects to a node in the parachain network that serves slightly incorrect responses to smoldot that would make it crash. Resulting in the same outcome, that page 1 doesn't work.

[tomaka]

If this would crash smoldot, it would mean you don't even need any secondary page? Because smoldot may connects to a node in the parachain network that serves slightly incorrect responses to smoldot that would make it crash

Assuming that the blockchain is "correct", smoldot doesn't crash in case of an incorrect message.
What could crash smoldot is for example if you connect to a chain that says it has 100 million validators, in other words if the Aura_validators runtime function (or whatever it's called) returns 100 million items, as smoldot would then run out of memory.
Basically, you can crash smoldot by connecting to a chain that uses absurd values in its chain configuration.

That's why so far smoldot has only tried to "interpret" relay chains, because relay chains are trusted to be reasonable, while it treats all parachain blocks as completely opaque, as parachains are less trusted.
As I explain above, this is because the initial idea is to have one instance of smoldot connected to Polkadot/Kusama running as a browser extension, and any website you visit could connect to any Polkadot/Kusama parachain even potentially absurd ones. In that case, we wouldn't want random websites to be able to crash the browser extension.