Panic in tx-service task: unwrap on None in light_pool.rs:505

[hantoniu-codeberg]

smoldot panicked while executing the tx-service-asset-hub-kusama task. The process continued running but the chain is likely in a broken state.

Panic message:

  Smoldot has panicked while executing task `tx-service-asset-hub-kusama`. This is a bug in smoldot. Please open an issue at https://github.com/smol-dot/smoldot/issues with the following message:
  panicked at /__w/smoldot/smoldot/lib/src/transactions/light_pool.rs:505:63:
  called `Option::unwrap()` on a `None` value

Environment:

• smoldot version: 2.0.40
• Platform: Linux (Debian, arm64)
• Chain: kusama-asset-hub (parachain via kusama relay)

Context:

• Running smoldot as a daemon exposing chains via WebSocket over Unix sockets
• The panic occurred ~23 minutes after startup during normal operation
• No specific trigger identified - read-only usage, no transactions being submitted
• Heavy sync activity on kusama relay chain just before panic (multiple "occupied CPU for unreasonable amount of time" warnings on sync-service-ksmcc3)

Logs before panic:
[smoldot] The task named sync-service-ksmcc3 has occupied the CPU for an unreasonable amount of time (275ms).
Smoldot has panicked while executing task tx-service-asset-hub-kusama...

Additional notes:

• The panic was caught internally by smoldot - the Node.js process didn't crash
• This means applications cannot detect and recover from this failure
• Would it be possible to propagate these panics as errors to the calling code?

[tomaka]

Thanks for the report.

Would it be possible to propagate these panics as errors to the calling code?

An exception is supposed to be thrown whenever you call a function on the smoldot client or a chain (such as sendJsonRpc or nextJsonRpcResponse), letting you detect the problem.

[hantoniu-codeberg]

Thanks for the quick response!

For context, we're building https://codeberg.org/hantoniu/smold, a daemon that exposes smoldot chains via WebSocket over Unix sockets. We already wrap sendJsonRpc/nextJsonRpcResponse in try/catch and have a recovery mechanism that restarts smoldot when exceptions occur (https://codeberg.org/hantoniu/smold/src/branch/main/src/socket-server.ts#L193-L221).

The issue is that when the tx-service task panicked, neither method threw an exception. The chain continued responding to other JSON-RPC calls while transaction-related functionality was silently broken. From our application's perspective, the chain appeared healthy.

A few questions:

1. Is this expected behavior? i.e., task-level panics don't propagate to the Chain API unless the entire chain becomes unusable?
2. Would it be feasible to have smoldot throw on the next sendJsonRpc/nextJsonRpcResponse call after any internal task panics, even if other functionality still works?
3. Alternatively, would a health/status callback or method that reports internal task failures be considered? Something like chain.onError(callback) or chain.isHealthy().

For now we're exploring workarounds like periodic health checks via JSON-RPC, but native error propagation would be much more reliable.

[tomaka]

Would it be feasible to have smoldot throw on the next sendJsonRpc/nextJsonRpcResponse call after any internal task panics, even if other functionality still works?

What you suggest here is the expected behavior. I don't know why it doesn't do that.
To debug that, I would to know whether you are using the portToWorker option of the client?

Is this expected behavior? i.e., task-level panics don't propagate to the Chain API unless the entire chain becomes unusable?

The expected behavior at the moment is that any panic entirely stops the client (if you connected to multiple chains, using any of them will throw an exception).
I've started working on a change to internally restart non-critical tasks that panic, but I've never actually enabled it "for real".

Note that my policy in general is that panics shouldn't be happening in the first place, and that the proper solution is to fix panics. Anything that tries to recover gracefully is fundamentally a dirty hack.
I understand that you are in favor of detecting panics in smoldot and recovering gracefully because you treat smoldot as an "external thing". However, the difference between smoldot's code and your code that calls smoldot is purely social (we are two different projects) rather than technical. A panic in smoldot's Rust code is in principle no different than a bug in your higher-level JavaScript code, and you probably wouldn't put a try-catch around every single function of your code to try to recover gracefully.

[bkchr]

BTW, I think the underlying issue here is what I described here: #2193

Kusama AH is now producing blocks every 2s on 3 cores, which is not really supported right now by smoldot. There are "gaps" (blocks validated by the relay chain, but not appearing in the heads structure) in blocks, which should explain the panic.

[rossbulat]

BTW, I think the underlying issue here is what I described here: #2193

Kusama AH is now producing blocks every 2s on 3 cores, which is not really supported right now by smoldot. There are "gaps" (blocks validated by the relay chain, but not appearing in the heads structure) in blocks, which should explain the panic.

We are experiencing a BadProof error when submitting transactions on the Staking Dashboard Smoldot setup (for asset hub) - relay provider with people and assethub chains added.

Could this missing heads issue be the culprit? Has there been any movement on this since @bkchr's last comment?

I've created a test to reproduce the errors here: https://github.com/polkadot-cloud/polkadot-staking-dashboard/tree/main/packages/test-smoldot

[josepot]

We are experiencing a BadProof error when submitting transactions on the Staking Dashboard Smoldot setup (for asset hub) - relay provider with people and assethub chains added.

Could this missing heads issue be the culprit? Has there been any movement on this since @bkchr's last comment?

I'm pretty sure that the reason for the BadProof error on your Staking Dashboard is something completely different. I could be wrong, but could it have to do with the fact that you are using Metadata v16 for computing the CheckMetadataHash value? Anyways, without logs and/or more info it's impossible to know for sure. However, if I had to make a guess, that could explain it. I would suggest that you open an issue on the library that you are using for creating the transactions.

EDIT: it could also be that the mortality is not calculated properly due to #2158, yep.