[bug]: Repeated litd Crash on Startup - Suspected Database Corruption

[shuishihang]

I have a node server that fails to start litd properly. It keeps repeatedly attempting to restart. When I manually start litd, I see the following log. Is this a case of database corruption?

panic: assertion failed: Page expected to be: 71121, but self identifies as 8796093093329

go.etcd.io/bbolt._assert(...)
        /home/shui/.go/pkg/mod/go.etcd.io/[email protected]/db.go:1389
go.etcd.io/bbolt.(*page).fastCheck(0x78bed15d1000, 0x115d1)
        /home/shui/.go/pkg/mod/go.etcd.io/[email protected]/page.go:57 +0x1d9
go.etcd.io/bbolt.(*Tx).page(0x5dfc9e5960b0fcee?, 0x41b52c9c2000cb4e?)
        /home/shui/.go/pkg/mod/go.etcd.io/[email protected]/tx.go:541 +0x7b
go.etcd.io/bbolt.(*Bucket).pageNode(0x0?, 0xa0c65dc0d6628803?)
        /home/shui/.go/pkg/mod/go.etcd.io/[email protected]/bucket.go:748 +0x85
go.etcd.io/bbolt.(*Cursor).goToFirstElementOnTheStack(0xc000ab56f0)
        /home/shui/.go/pkg/mod/go.etcd.io/[email protected]/cursor.go:181 +0xc5
go.etcd.io/bbolt.(*Cursor).next(0xc000ab56f0)
        /home/shui/.go/pkg/mod/go.etcd.io/[email protected]/cursor.go:234 +0x52
go.etcd.io/bbolt.(*Cursor).Next(0x78beca071232?)
        /home/shui/.go/pkg/mod/go.etcd.io/[email protected]/cursor.go:94 +0x27
go.etcd.io/bbolt.(*Bucket).ForEach(0x2dbe440?, 0xc0038e8960)
        /home/shui/.go/pkg/mod/go.etcd.io/[email protected]/bucket.go:396 +0x79
github.com/btcsuite/btcwallet/walletdb/bdb.(*bucket).ForEach(0x62729b0?, 0x2dbe440?)
        /home/shui/.go/pkg/mod/github.com/btcsuite/btcwallet/[email protected]/bdb/db.go:190 +0x17
github.com/lightningnetwork/lnd/kvdb.ForAll({0x78bfaa663460?, 0xc003938140?}, 0xc0038e8960?)
        /home/shui/.go/pkg/mod/github.com/lightningnetwork/lnd/[email protected]/interface.go:135 +0x63
github.com/lightningnetwork/lnd/graph/db.(*KVStore).ForEachChannel.func1({0x4514d40?, 0xc004c5e258?})
        /home/shui/.go/pkg/mod/github.com/lightningnetwork/[email protected]/graph/db/kv_store.go:412 +0x114
github.com/btcsuite/btcwallet/walletdb/bdb.(*db).View(0xc000217d48, 0xc00096f740, 0x4b5501?)
        /home/shui/.go/pkg/mod/github.com/btcsuite/btcwallet/[email protected]/bdb/db.go:392 +0x8f
github.com/lightningnetwork/lnd/graph/db.(*KVStore).ForEachChannel(0xc000dca0c0, 0xc004c58620)
        /home/shui/.go/pkg/mod/github.com/lightningnetwork/[email protected]/graph/db/kv_store.go:392 +0x7c
github.com/lightningnetwork/lnd/graph/db.(*ChannelGraph).populateCache(0xc000bbc480)
        /home/shui/.go/pkg/mod/github.com/lightningnetwork/[email protected]/graph/db/graph.go:193 +0x10f
github.com/lightningnetwork/lnd/graph/db.(*ChannelGraph).Start(0xc0018cef30?)
        /home/shui/.go/pkg/mod/github.com/lightningnetwork/[email protected]/graph/db/graph.go:93 +0xd7
github.com/lightningnetwork/lnd.(*server).Start.func1()
        /home/shui/.go/pkg/mod/github.com/lightningnetwork/[email protected]/server.go:2420 +0x1c25
sync.(*Once).doSlow(0xc00011ae08?, 0xc000d7f4f0?)
        /usr/local/go/src/sync/once.go:78 +0xab
sync.(*Once).Do(...)
        /usr/local/go/src/sync/once.go:69
github.com/lightningnetwork/lnd.(*server).Start(0xc00011ae08, {0x4517938, 0xc0001f7860})
        /home/shui/.go/pkg/mod/github.com/lightningnetwork/[email protected]/server.go:2282 +0xb6
github.com/lightningnetwork/lnd.Main.func12()
        /home/shui/.go/pkg/mod/github.com/lightningnetwork/[email protected]/lnd.go:761 +0x28
created by github.com/lightningnetwork/lnd.Main in goroutine 70
        /home/shui/.go/pkg/mod/github.com/lightningnetwork/[email protected]/lnd.go:760 +0x4a07

[bitromortac]

Is this a case of database corruption?

Unfortunately that seems to be the case, looking at assertion failed: Page expected to be: 71121, but self identifies as 8796093093329. You could try to run the https://github.com/lightninglabs/chantools compactdb command to have a further check. Did you have any service interruptions?

There seems to be an issue with either the wallet or channel/graph db:

github.com/btcsuite/btcwallet/walletdb/bdb.(*bucket).ForEach(0x62729b0?, 0x2dbe440?)
        /home/shui/.go/pkg/mod/github.com/btcsuite/btcwallet/[email protected]/bdb/db.go:190 +0x17

github.com/lightningnetwork/lnd/graph/db.(*KVStore).ForEachChannel.func1({0x4514d40?, 0xc004c5e258?})
        /home/shui/.go/pkg/mod/github.com/lightningnetwork/[email protected]/graph/db/kv_store.go:412 +0x114

[shuishihang]

I ran the command:

chantools compactdb \ --sourcedb ~/.lnd/data/graph/mainnet/channel.db \ --destdb ./results/compacted.db

There were no errors, and the output was:
2025-09-03 14:24:39.504 [INF] CHAN: chantools version v0.14.1 commit

However, after restarting litd, I still encountered the same error.