Node stops syncing when IPC socket client disconnects

[fipso]

Description
I have been seeing this issue since over a year now. When a RPC client that created a subscription via the UDS socket file disconnects the whole node stops sync because it stops accepting new blocks. One of these errors keeps coming up for any new block processed after the IPC socket client disconnects:

1. InvalidReceiptsRoot - receipts root mismatch
2. InvalidBlockHash - block hash mismatch

Steps to Reproduce

1. Run nethermind with these options:

      - --config=mainnet
      - --datadir=/nethermind/data
      - --log=debug
      - --JsonRpc.Enabled=true
      - --JsonRpc.Host=0.0.0.0
      - --JsonRpc.Port=8545
      - --JsonRpc.UnsecureDevNoRpcAuthentication
      - --JsonRpc.EngineHost=0.0.0.0
      - --JsonRpc.EnginePort=8551
      - --JsonRpc.IpcUnixDomainSocketPath=/tmp/eth.sock
      - --JsonRpc.IpcProcessingConcurrency=16
      - --Network.DiscoveryPort=30303
      - --HealthChecks.Enabled=true
      - --Metrics.Enabled=true
      - --Metrics.ExposePort=8008
      - --Metrics.NodeName=test
      - --Sync.DownloadBodiesInFastSync=false
      - --Sync.DownloadReceiptsInFastSync=false
      - --Sync.NonValidatorNode=true

2. Connect to the UDS socket file with 3 clients that create one subscription on all USDC Transfer Events each
3. Kill all the 3 clients
4. Wait 60s
5. Node stops accepting new blocks

What I already tried:

• Disabled pre-warming (--Blocks.PreWarmStateOnBlockProcessing=false) - issue persists
• Reduced IPC concurrency to 4 (--JsonRpc.IpcProcessingConcurrency=4) - issue persists
• Restarting Prysm consensus client - does not help
• Only restarting Nethermind clears the invalid block cache

Desktop (please complete the following information):
Please provide the following information regarding your setup:

• Operating System: NixOS
• Docker Image: nethermind/nethermind:1.36.0-chiseled
• Consensus Client: Prysm

Error Type 1: InvalidReceiptsRoot (most common)

  23 Jan 14:12:01 | JSON RPC request engine_newPayloadV4
  23 Jan 14:12:01 | Received New Block:  24298000 (0x83d401...5603ee)      | limit    60,000,000    | Extra Data: BuilderNet (Flashbots)
  23 Jan 14:12:01 | Update 0x20033a600608686201311217da4d4477537b94ac C 0xc5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470 ->
  0xb09ef517c48d2bf6eed05457ff56871b2596e3fc904fc6e9795882a870c2e993
  23 Jan 14:12:01 | Processed block 24298000 (0xffb47d...cc57b8) is invalid:
  23 Jan 14:12:01 | - hash: expected 0x83d40116d9574770fda73154dea5c30086ba27261e41650858f80238df5603ee, got
  0xffb47d18d166bf7937481d231193151b2ff526deb4541ba53ceb685440cc57b8
  23 Jan 14:12:01 | - receipts root: expected 0xa353cace3a560e06698ae0ca0a6fbf2d605df605106f5b9de7cc53fd881e6e94, got
  0xb7127a6c859a24e9eec55b87a67a2f495bad8c13baf3f1d3dd0b3ad28af586a0
  23 Jan 14:12:01 | - block extra data : 4275696c6465724e65742028466c617368626f747329, UTF8: BuilderNet (Flashbots)
  23 Jan 14:12:01 | OnInvalidBlock: 0x83d40116d9574770fda73154dea5c30086ba27261e41650858f80238df5603ee
  0xb2f376b67c6f8e0912c43ccece7b8ea14724488946a1380ce063f3de837298a5
  23 Jan 14:12:01 | Rejected invalid block 24298000 (0x83d40116d9574770fda73154dea5c30086ba27261e41650858f80238df5603ee),  ExtraData: BuilderNet (Flashbots), reason:
  invalid block after processing
  23 Jan 14:12:01 | Encountered exception Nethermind.Blockchain.InvalidBlockException: InvalidReceiptsRoot: Receipts root in header does not match. Expected
  0xa353cace3a560e06698ae0ca0a6fbf2d605df605106f5b9de7cc53fd881e6e94, got 0xb7127a6c859a24e9eec55b87a67a2f495bad8c13baf3f1d3dd0b3ad28af586a0
     at Nethermind.Consensus.Processing.BranchProcessor.Process(BlockHeader baseBlock, IReadOnlyList`1 suggestedBlocks, ProcessingOptions options, IBlockTracer
  blockTracer, CancellationToken token) in /nethermind/src/Nethermind/Nethermind.Consensus/Processing/BranchProcessor.cs:line 135 while processing blocks.
  23 Jan 14:12:01 | Issue processing block Hash: 0x83d40116d9574770fda73154dea5c30086ba27261e41650858f80238df5603ee
  Number: 24298000
  Parent: 0xb2f376b67c6f8e0912c43ccece7b8ea14724488946a1380ce063f3de837298a5
  Beneficiary: 0xdadb0d80178819f2319190d340ce9a924f783711
  Gas Limit: 60000000
  Gas Used: 28058711
  Timestamp: 1769177519
  Extra Data: 4275696c6465724e65742028466c617368626f747329
  Difficulty: 0
  Mix Hash: 0x77dd8aa756aebcd442d5d2bd5c573dd1e4e4ff9c9ab3d3d1e945760a51dda1c0
  Nonce: 0
  Uncles Hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347
  Tx Root: 0xa9c473a613bac82e04ed88add4542ce0ac81c03adb7d09fa3e067e553b8daecf
  Receipts Root: 0xa353cace3a560e06698ae0ca0a6fbf2d605df605106f5b9de7cc53fd881e6e94
  State Root: 0xafc70231c8d32824036f1fe79493a303aaa992f497e39a9b25ab5bbad92990a4
  BaseFeePerGas: 106931314
  WithdrawalsRoot: 0x7b947b35171e51659abfa1b1bf178d68aee91a0bef6c628c1c0e04c81a8ec45b
  ParentBeaconBlockRoot: 0xe65919cee7781b1fba69ddeb18b6ce06400be74679f4d676e94fe6bb7bb1ed2d
  BlobGasUsed: 131072
  ExcessBlobGas: 183428326
  IsPostMerge: True
  TotalDifficulty: 58750003716598352816469
  RequestsHash: 0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
  23 Jan 14:12:01 | Created a RLP dump of invalid block 0x83d40116d9574770fda73154dea5c30086ba27261e41650858f80238df5603ee in file
  /tmp/block_0x83d40116d9574770fda73154dea5c30086ba27261e41650858f80238df5603ee.rlp
  23 Jan 14:12:01 | Update 0x20033a600608686201311217da4d4477537b94ac C 0xc5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470 ->
  0xb09ef517c48d2bf6eed05457ff56871b2596e3fc904fc6e9795882a870c2e993
  23 Jan 14:12:01 | Beginning new BlockCommitSet - 24298000
  23 Jan 14:12:02 | Created a Receipts trace of valid on rerun block 0x83d40116d9574770fda73154dea5c30086ba27261e41650858f80238df5603ee in file
  /tmp/receipts_0x83d40116d9574770fda73154dea5c30086ba27261e41650858f80238df5603ee_success.txt
  23 Jan 14:12:02 | Deleting invalid block 24298000 (0x83d40116d9574770fda73154dea5c30086ba27261e41650858f80238df5603ee)

Error Type 2: InvalidBlockHash (less common)

  23 Jan 13:15:48 | Received New Block: 24297075 (0x454170...406eb)
  23 Jan 13:15:48 | invalid block hash
  23 Jan 13:15:48 | Invalid block hash 0x454170... does not match calculated hash 0x117c33...

Concurrent P2P RLP Decode Errors
These RLP errors appear around the same time as block validation failures:

  23 Jan 14:12:00 | Error in communication with [Node|10.0.0.2:49194||]: DotNetty.Codecs.DecoderException: Exception of type 'DotNetty.Codecs.DecoderException' was
  thrown.
   ---> Nethermind.Serialization.Rlp.RlpException: Expected a sequence prefix to be in the range of <192, 255> and got 0
     at Nethermind.Network.Rlpx.FrameHeaderReader.ReadFrameHeader(IByteBuffer input)
     at Nethermind.Network.Rlpx.ZeroFrameMerger.Decode(IChannelHandlerContext context, IByteBuffer input, List`1 output)

Prysm Consensus Client Logs

  time="2026-01-23 12:15:52" level=warning msg="Beacon chain started, but el]_client is not synced" prefix=initial-sync
  time="2026-01-23 12:15:52" level=warning msg="Could not get payload" error="could not get payload: rpc error: code = Unknown desc = Invalid params: Invalid block
  hash 0x4541700... does not match calculated hash 0x117c33..." prefix=blockchain
  time="2026-01-23 12:16:04" level=warning msg="Synced to finalized epoch - Loss of sync detected" prefix=initial-sync
  time="2026-01-23 12:16:04" level=warning msg="Block 24297075 is known to be a part of an invalid chain" blockRoot=0x3ae8be... prefix=blockchain

[fipso]

I asked Claude about this:

  Analysis

  The key observation is that the same block succeeds on immediate rerun. This indicates:
  1. The block data from the consensus client is valid
  2. Something transient affects the first execution attempt
  3. The issue correlates with IPC connection lifecycle but persists even with low concurrency

  Potential areas investigated:
  - PreBlockCaches shared between pre-warming and block processing (ruled out - disabled pre-warming)
  - Static CodeLruCache in CodeInfoRepository - shared across all operations
  - ArrayPool<byte>.Shared used by EVM memory - could have buffer reuse issues
  - BlockReceiptsTracer instance reuse in BlockProcessor