is this recoverable

[biskit]

been running v4.0 for some time now on a 3 node, 48 shard cluster. on a stop start today, I get these messages on two nodes that were followers before it was stopped:

on node 2:
2026-02-20 11:30:32.849378 C | rsm: [00025:00002], ss.OnDiskIndex (124443031) > s.onDiskInitIndex (124420496) panic: [00025:00002], ss.OnDiskIndex (124443031) > s.onDiskInitIndex (124420496)

on node 3:
2026-02-20 11:23:02.237374 C | rsm: [00025:00003], ss.OnDiskIndex (124443407) > s.onDiskInitIndex (124420496) panic: [00025:00003], ss.OnDiskIndex (124443407) > s.onDiskInitIndex (124420496)

how does one recover

[biskit]

one would think the onDiskInitIndex would be same on leader and followers. it's higher (and correct) on the leader while the other two are lagging. how is this possible when its the same code and the cluster has been stopped and started many times before. there was one reboot of the machine before the last stop or two and terminals they were running in may have received an uncatchable SIGKILL

however, the process has hooks to the statemachine built in Sync() call and there was no activity for sometime (meaning the sync call would have done an actual fsync)

h/w: m3 max with sequoia