Changes in the GRANDPA Authority-Set causes smoldot to stop reporting finality

[josepot]

We've observed an issue when using PAPI with smoldot related to GRANDPA authority set changes.

Specifically, whenever a block introduces a new GRANDPA Authority-Set, smoldot stops reporting finality. This disruption prevents PAPI from unpinning blocks, eventually leading to a chainHead stop event.

What’s particularly interesting is that as soon as PAPI initiates a new chainHead subscription (immediately after receiving the stop event), smoldot resumes reporting finalized blocks. Based on logs, it seems that smoldot halts finality reporting because it fails to recognize the block containing the AuthoritySet change as finalized... However, if that's the case, then we don't understand why smoldot starts reporting finalized blocks again after a stop-event.

The following PAPI script was recently tested against the Polkadot relay-chain (it was intentionally started just a few minutes before the Authority-Set changed):

import { map, mergeMap, pairwise, skip } from "rxjs"
import type { BlockInfo, HexString } from "polkadot-api"
import { localWnd } from "@polkadot-api/descriptors"
import { client } from "./wnd-client"

const api = client.getUnsafeApi<typeof localWnd>()

const hasNewAuthoritySet = async (at: HexString) =>
  (await api.query.System.Events.getValue({ at })).some(
    ({ event }) =>
      event.type === "Grandpa" && event.value.type === "NewAuthorities",
  )

const printFinalized = ({ number, hash }: BlockInfo) =>
  console.log(`Finalized block: ${number} - ${hash}`)

client.finalizedBlock$
  .pipe(
    pairwise(),
    map(([prev, current], idx) => {
      if (idx === 0) printFinalized(prev) // The first finalized block

      if (prev.number + 1 < current.number)
        console.log("\nGAP DETECTED!! CHAINHEAD RECOVERED FROM A STOP EVENT\n")

      printFinalized(current)
    }),
  )
  .subscribe()

client.blocks$
  .pipe(
    mergeMap(async (block) => {
      try {
        if (await hasNewAuthoritySet(block.hash))
          console.log(`\nNEW AUTH SET AT: ${block.number} - ${block.hash}\n`)
      } catch {
        console.error(
          `COULD NOT CHECK NEW AUTH SET AT: ${block.number} - ${block.hash}`,
        )
      }
    }),
  )
  .subscribe()

The script produced the following output:

Finalized block: 26989347 - 0x5fcd7f3edb03458d51f74423360d103fd5ce47a89f0319f26d1939b3d5646daa
Finalized block: 26989348 - 0xb6faf2d7cce35b89a96d1c58c8038af2c003ae8969926674882eaa044e2412f1
Finalized block: 26989349 - 0xe417e94c006af5379c4b31735c2b3e3c6f5b675f43de41554d0b51b188e740a5
Finalized block: 26989350 - 0xa9837a2cd8f9235297ffa3a6eaaa8827f8ea07b08b2578344bbaa5fee6affa91
Finalized block: 26989351 - 0xb4759a2054f05b4a0203a244f0f81c006f13b637bfa18fb8f4ac9ed004cd725c
Finalized block: 26989352 - 0xd6090a277a971d9a443c49e0a49e171e765a35d5838d5283005e06a1d2d0301d
Finalized block: 26989353 - 0x90da8d08bc6c95001fac4356d1f8e68932ab8e33e9e6a33679a3738eb649ee12
Finalized block: 26989354 - 0x89a006f08ba6659ab88a2eb754fcc5d506cea27a445b96fc76542b27c71e9cdd
Finalized block: 26989355 - 0x230baa8a596bd207853bef4f4be08c6434663f250a90215b78384ce83457de4e
Finalized block: 26989356 - 0xf47e0cbb03a66a5befd224a9cbc8911b78d3d85c59d5bc680d6adf8de7c26ef4
Finalized block: 26989357 - 0xab6647421c511ff6fdcdd0229b49e9c0626070061cea35f153099f763c100078
Finalized block: 26989358 - 0xa5b2e15f0ff816108bbd7551e3648a5a9d67c383f012cdc3ec976d05e515825d
Finalized block: 26989359 - 0xe66244f6a15cd31a03cdf54edea4e17b6b3c1a3cc326954f6e90fbf9ab459214
Finalized block: 26989360 - 0x6cb195c1dbe91f13774ab6a1d8aca45277428790dde5f535eadcf5cac2bdc888
Finalized block: 26989361 - 0x25a52a443e828b1e2b4f9e1c0881598e8cc3b0c7bd58fad1dbb683e433b0f8c1
Finalized block: 26989362 - 0xb06be8ea16c9218d5ead9add55725cb6ed76e0fcc65d432c127f3fc3f62105ed
Finalized block: 26989363 - 0x93b0177d0befa4e23944e9ddfb0977cb6e2053956729a888e9306807fbb56d4a
Finalized block: 26989364 - 0xec18646db0dc805e6fdfcebac34e164ff4aec3bce71106badbb82dc01e83f176
Finalized block: 26989365 - 0xe0879849461481ef34d2b3a1f6f0500096249e8498eb9f3f5b845256542eeeab
Finalized block: 26989366 - 0xd331aa8d9726c3e4beefca0011d409347b7b33f40f557adb45649ac5d470a984
Finalized block: 26989367 - 0xa42e8e97cc5487dcb12b239bb9c83b04bd55443414ff382d65d722ecb41d66d9
Finalized block: 26989368 - 0x0425e301bcbc07731d17559d8dc880cc77e8387a88f8bc63f96627ef4aee5e78
Finalized block: 26989369 - 0x07b9fd94b50b4cd7c47396d930a0acec724dda1205dd6d861c99790f28d7fed6
Finalized block: 26989370 - 0x118ab283f92e4b1b8608e3c09c01cd3ff8b550f645db6704d9698192a78fcb1e
Finalized block: 26989371 - 0x16861262bd74d31fffa86fdde2a0220fbe8d0eacef0ecb2bdf3edafdef8e6680
Finalized block: 26989372 - 0x22adb71e100ab080d99c2236262775a0936aa81d15652b9262ba38721f6b0b40
Finalized block: 26989373 - 0x4d4ff9a7ac832c209ea008997bc0b7e41468d92babedc987495c2ca05abdb0c3
Finalized block: 26989374 - 0xc3bc277527d5d68b5d0c34bd02c3388cb20ec1618717d1db21df0baf50e388f3
Finalized block: 26989375 - 0x9a7512faf7a62841eeb985219aabaa1f117c5fd68dbfde1f947d4a8753bfc119
Finalized block: 26989376 - 0x3355a8404319fb446ba671b02cfb647c16e15072fe329ab61e7dbdf97d2f6f6a
Finalized block: 26989377 - 0x514d1a89fc2976aa49ccf5cf886c912044cae5f7af0b2c95f80b969f16839480
Finalized block: 26989378 - 0xaf5510147b974ed7676579cf4a5fe62264690ccabb58f8ba7821a027b0fa3428
Finalized block: 26989379 - 0x7ed73ff94c4f756b6546a2649986a5b270fe920b03faf4717e187033b591a0d2
Finalized block: 26989380 - 0x6201d21f8afcf8380be2f98548b532da75ec528153f66cd3ace0d9399b5f453e

NEW AUTH SET AT: 26989384 - 0x9fad8ade5c1534c9c42ce72a3d658d4a3dbbc38771fb9191c6de60791b227082

Finalized block: 26989381 - 0x814ce70534a924405d377029381284c825de871783352a9f8d5476e2199f0253
Finalized block: 26989382 - 0x4332ca066030e0c4d00561c576362118e2576df3877cadaf329c1b1e0475fecd

GAP DETECTED!! CHAINHEAD RECOVERED FROM A STOP EVENT

Finalized block: 26989416 - 0xec65bfbc32a28f99750227946d84dcbd3ae94fec852a1ff84c82ec364f20fc8c
Finalized block: 26989417 - 0x2f5e6644c06cb5d912c9e08af1d12abdda8895add8a09a905056b1a1032f61b4
Finalized block: 26989418 - 0x35dea644ddad2fb7ed11d41e0fb0c8735c897766a95b1bd4af72f6134cbd4a17
Finalized block: 26989419 - 0xd60047c3f22ddb9c531703df397174da3f678de66945570d88ba422a71a6d35f
Finalized block: 26989420 - 0xd8d6e2768f36e1e7db91a74a17016e0fe5cd2342f6194f134d068a7bd524c704

As you can see in the output, the script is able to detect that a new GRANDPA Authority-Set is coming at height 26989384, which is precisely the first block that smoldot isn't able to report as finalized... until the stop-event happens and then we are able to start receiving finalized blocks again.

These are the relevant logs:

• SMOLDOT LOGs (maxLogLevel: 7)
• JSON-RPC LOGS

We have collected similar logs on other networks. For instance, on Westend:

• SMOLDOT LOGs (maxLogLevel: 7)
• JSON-RPC LOGs

---

Reproduction:

We have setup the following repo in order to facilitate the reproduction of this issue. The reason why it's fairly easy to reproduce is because while using Zombienet against Westend, the GRANDPA Authority-Set changes at height 41.

[josepot]

It's possible that this is related to this. However, smoldot still suffers from this problem when we try it against the node produced by the code of the PolkadotSDK fix.

[tomaka]

The way smoldot detects finalized block is that it simply passively listens for so-called Grandpa commit messages.
The full nodes normally send these commit messages automatically, and we can see in the logs that they stop sending them after the authority set changes.

Whether this is actually the fault of the Polkadot node is unclear, as there's no spec as to how the Grandpa networking is meant to behave. I guess that, when that happens, smoldot is meant to do something proactive no matter what, instead of silently failing.

[josepot]

and we can see in the logs that they stop sending them after the authority set changes.

🤔 Are you sure?

2025-07-22T11:36:41.430Z (4)-network
grandpa-neighbor-packet-received; chain=polkadot, peer_id=12D3KooWSz8r2WyCdsfWHgPyvD8GKQdJ1UAiRmrcrs8sQB3fe2KU, round_number=2, set_id=2812, commit_finalized_height=26989385

2025-07-22T11:36:41.511Z (4)-network
grandpa-neighbor-packet-received; chain=polkadot, peer_id=12D3KooWFN2mhgpkJsDBuNuE5427AcDrsib8EoqGMZmkxWwx3Md4, round_number=2, set_id=2812, commit_finalized_height=26989385

2025-07-22T11:36:41.620Z (4)-network
grandpa-neighbor-packet-received; chain=polkadot, peer_id=12D3KooWPyEvPEXghnMC67Gff6PuZiSvfx3fmziKiPZcGStZ5xff, round_number=2, set_id=2812, commit_finalized_height=26989385

2025-07-22T11:36:42.526Z (4)-network
grandpa-neighbor-packet-received; chain=polkadot, peer_id=12D3KooWRjHFApinuqSBjoaDjQHvxwubQSpEVy5hrgC9Smvh92WF, round_number=2, set_id=2812, commit_finalized_height=26989385

2025-07-22T11:36:45.533Z (4)-network
grandpa-neighbor-packet-received; chain=polkadot, peer_id=12D3KooWRjHFApinuqSBjoaDjQHvxwubQSpEVy5hrgC9Smvh92WF, round_number=2, set_id=2812, commit_finalized_height=26989386

2025-07-22T11:36:45.546Z (4)-network
grandpa-neighbor-packet-received; chain=polkadot, peer_id=12D3KooWSz8r2WyCdsfWHgPyvD8GKQdJ1UAiRmrcrs8sQB3fe2KU, round_number=3, set_id=2812, commit_finalized_height=26989386

2025-07-22T11:36:45.558Z (4)-network
grandpa-neighbor-packet-received; chain=polkadot, peer_id=12D3KooWFN2mhgpkJsDBuNuE5427AcDrsib8EoqGMZmkxWwx3Md4, round_number=3, set_id=2812, commit_finalized_height=26989386

It seems to me that the nodes continue sending the subsequent GRANDPA commits.

I think that the actual issue is that the commits for block 26989384 (where the authority-set changes) is not being gossiped (or at least smoldot never receives it). Therefore, smoldot can't finalize the subsequent blocks... B/c the blocks where the authority-set change must be explicitly finalized.

Whether this is actually the fault of the Polkadot node is unclear

I will open an issue on the PolkadotSDK repo, b/c IMO the grandpa commits should always be broadcasted for blocks in which there is a change of the authority-set.

I guess that, when that happens, smoldot is meant to do something proactive no matter what, instead of silently failing.

I know that it's not possible to proactively fetch GRANDPA commits. However, couldn't this be accomplished by starting a warp sync request? Is it possible to start a warp-sync request that starts exactly on the block where the previous authority-set changed? If that's possible, then wouldn't that be almost equivalent to requesting that commit on demand?

[tomaka]

It seems to me that the nodes continue sending the subsequent GRANDPA commits.

These are not commit messages, these are neighbor packets. Search for grandpa-commit-message-received in the logs.

However, couldn't this be accomplished by starting a warp sync request? Is it possible to start a warp-sync request that starts exactly on the block where the previous authority-set changed? If that's possible, then wouldn't that be almost equivalent to requesting that commit on demand?

It is possible to pro-actively fetch justifications. Justifications are basically the same thing as commit messages.
The only reason why smoldot doesn't do that is that it's a bit complicated to implement.

[josepot]

These are not commit messages, these are neighbor packets. Search for grandpa-commit-message-received in the logs.

Oh! I see! Thanks for letting me know that! So, just to make sure that I understand this: the messages that I highlighted indicate that a peer was able to deem a certain block as finalized, yes? Therefore, if the light-client wished to request the justifications from that peer, then it may be able to obtain them?

[tomaka]

Yes