[bug]: could not populate the graph cache: invalid edge key xxxx encountered

[youngzhenhao]

Background

2025-08-27 15:23:26.848 [ERR] LTND: Shutting down due to error in main method rev="" err="could not populate the graph cache: invalid edge key 027a3d15964615be3090f7f6a226f8603351f7c63c8cfb969ac0226a40389edc86 encountered"

Your environment

• version of lnd

LiT version: 0.15.1-alpha

• which operating system (uname -a on *Nix)

Ubuntu24.04 LTS Linux 6.8.0-78-generic

• version of btcd, bitcoind, or other backend

Neutrino

• any other relevant environment details

Steps to reproduce

Tell us how to reproduce this issue. Please provide stacktraces and links to code in question.

1. Start litd
2. Waiting for lnd to be fully synced to its chain backend
3. [ERR] LTND: Shutting down
4. Even after deleting the neutrino file and restarting litd, litd still shuts down.

Expected behaviour

Tell us what should happen

{
    "state":  "SERVER_ACTIVE"
}

Actual behaviour

Tell us what happens instead

[lncli] rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:10009: connect: connection refused"

logs

lnd.log

ERRs

2025-08-27 15:23:26.848 [ERR] LTND: Shutting down due to error in main method rev="" err="could not populate the graph cache: invalid edge key 027a3d15964615be3090f7f6a226f8603351f7c63c8cfb969ac0226a40389edc86 encountered"
2025-08-27 15:23:26.849 [ERR] NTFN: Failed getting UTXO: get utxo request cancelled
2025-08-27 15:23:26.849 [ERR] NTFN: Failed getting UTXO: get utxo request cancelled
2025-08-27 15:23:31.058 [ERR] BTCN: utxo scan failed: neutrino shutting down

WRNs

2025-08-27 15:23:26.576 [WRN] HLCK: check: disk space configured with 0 attempts, skipping it
2025-08-27 15:23:26.576 [WRN] HLCK: check: tls configured with 0 attempts, skipping it
2025-08-27 15:23:26.848 [WRN] HSWC: Htlc Switch already stopped
2025-08-27 15:23:26.848 [WRN] SRVR: failed to stop htlcSwitch: htlc switch already shutdown
2025-08-27 15:23:26.848 [WRN] SRVR: failed to stop interceptable switch: InterceptableSwitch stopped more than once
2025-08-27 15:23:26.848 [WRN] SRVR: failed to stop htlc invoices modifier: HtlcModificationInterceptor stopped morethan once
2025-08-27 15:23:26.848 [WRN] SRVR: failed to stop txPublisher: TxPublisher stopped more than once
2025-08-27 15:23:26.849 [WRN] SRVR: Manual update of channel backup failed: swapper is not active, cannot perform manual update
2025-08-27 15:23:26.849 [WRN] SRVR: Unable to stop BestBlockTracker: BestBlockTracker is not running
2025-08-27 15:23:26.849 [WRN] SRVR: unable to shutdown liveness monitor: monitor already stopped
2025-08-27 15:23:28.580 [WRN] BTWL: Query(0) from peer 45.79.195.29:8333 failed, rescheduling: did not get response before timeout

lit.conf

lnd.bitcoin.node=neutrino
lnd.bitcoin.mainnet=true
lnd.bitcoin.defaultchanconfs=1

[Application Options]
lnd-mode=integrated
httpslisten=127.0.0.1:8443
uipassword_file=/root/.lit/uipassword
network=mainnet
autopilot.disable=true

[Integrated lnd (use when lnd-mode=integrated)]
lnd.alias=name
lnd.debuglevel=info
lnd.maxbackoff=2s
lnd.sync-freelist=true
lnd.accept-keysend=true
lnd.tlsdisableautofill=true
lnd.maxpendingchannels=1000
lnd.max-commit-fee-rate-anchors=20
lnd.accept-positive-inbound-fees=true
lnd.payments-expiration-grace-period=24h
lnd.norest=1
lnd.rpclisten=127.0.0.1:10009
lnd.coin-selection-strategy=random
lnd.bitcoin.active=true
lnd.wallet-unlock-password-file=/root/.lit/password
lnd.wallet-unlock-allow-create=true
lnd.externalip=127.0.0.1:9735

[db]
lnd.db.no-graph-cache=false

[bolt]
lnd.db.bolt.auto-compact=false

[routing]
lnd.routing.strictgraphpruning=false

[neutrino]
lnd.neutrino.maxpeers=125
lnd.neutrino.banduration=10m
lnd.neutrino.banthreshold=20
lnd.neutrino.addpeer=47.104.87.162:8333
lnd.neutrino.addpeer=139.226.59.47:8333
lnd.neutrino.addpeer=180.165.27.73:8333
lnd.neutrino.addpeer=124.90.91.236:8333
lnd.neutrino.addpeer=222.209.99.185:8333
lnd.neutrino.addpeer=[240e:39d:5f:c790:be24:11ff:fe61:ab30]:8333
lnd.neutrino.addpeer=[2409:8a28:5085:6c40:a10a:ed8e:806:3259]:8333
lnd.neutrino.addpeer=[2409:8a28:f49:bb30:1000:bf5f:cf2d:e718]:8333
lnd.neutrino.assertfilterheader=230000:1308d5cfc6462f877a5587fd77d7c1ab029d45e58d5175aaf8c264cee9bde760
lnd.neutrino.assertfilterheader=660000:08312375fabc082b17fa8ee88443feb350c19a34bb7483f94f7478fa4ad33032
lnd.neutrino.broadcasttimeout=10s
lnd.neutrino.persistfilters=true

[fee]
lnd.fee.url=https://nodes.lightning.computer/fees/v1/btc-fee-estimates.json

[Autopilot]
lnd.autopilot.active=false
lnd.autopilot.private=true
lnd.autopilot.minconfs=1
lnd.autopilot.conftarget=3
lnd.autopilot.allocation=0.5
lnd.autopilot.heuristic=externalscore:1.00
lnd.autopilot.heuristic=preferential:0.00

[protocol]
lnd.protocol.wumbo-channels=true
lnd.protocol.zero-conf=true
lnd.protocol.option-scid-alias=true
lnd.protocol.simple-taproot-chans=true
lnd.protocol.simple-taproot-overlay-chans=true
lnd.protocol.custom-message=17

[routerrpc]
lnd.routerrpc.estimator=apriori

[rpcmiddleware]
lnd.rpcmiddleware.enable=true
lnd.rpcmiddleware.intercepttimeout=2s

[Integrated faraday options (use when faraday-mode=integrated)]
#faraday.faradaydir=

[Integrated loop options (use when loop-mode=integrated)]
#loop.loopdir=

[Integrated pool options (use when pool-mode=integrated)]
#pool.basedir=

[Integrated taproot assets options (use when taproot-assets=integrated)]
taproot-assets.network=mainnet
taproot-assets.universe.federationserver=123.223.109.84:8444
taproot-assets.proofcourieraddr=universerpc://123.223.109.84:8444
taproot-assets.experimental.rfq.priceoracleaddress=rfqrpc://123.223.37.253:10086

[ziggie1984]

Thank you for reporting this issue, we are investigating and will reach out to your for further instructions.

[ziggie1984]

Did you recently update to 15.1 or did you do something special or just a restart ? Or did your node crash for some reason ?

Diagnosis: Data Corruption

This error strongly suggests corruption in your local channel graph data. The Lightning Network daemon is designed to never write edge keys that are not exactly 41 bytes - this is a fundamental constraint in the codebase. Since you're encountering an edge key that doesn't meet this requirement, it indicates the underlying data has been corrupted after it was
originally written correctly.

Recommended Solution

We recommend using the chantools utility to recover from this state:

1. Install chantools: Available at https://github.com/lightninglabs/chantools
2. Use the dropchannelgraph command: This will clear your corrupted channel graph data and allow your node to rebuild it from scratch by syncing with the network.

[youngzhenhao]

Error when using dropchannelgraph

~# chantools dropchannelgraph --channeldb /root/.lnd/data/graph/mainnet/channel.db --node_identity_key 0230c6bc1fad58ce9dad2c4f13b6800949586d48c8f8062900f27ab4ded4d25e97
2025-08-28 16:00:17.458 [INF] CHAN: chantools version v0.14.1 commit
2025-08-28 16:00:17.460 [INF] CHDB: Checking for schema update: latest_version=33, db_version=33
2025-08-28 16:00:17.460 [INF] CHDB: Checking for optional update: prune_revocation_log=false, db_version=1: prune revocation log
2025-08-28 16:00:17.461 [INF] CHAN: Dropping all graph related buckets
panic: page 41616 already freed

goroutine 1 [running]:
go.etcd.io/bbolt.(*freelist).free(0xc000290e00, 0xa95c0, 0x73cbdde90000)
        go.etcd.io/[email protected]/freelist.go:175 +0x2b4
go.etcd.io/bbolt.(*Bucket).free.func1(0xc0001b5f20?, 0x3407?, 0xc3a33b?)
        go.etcd.io/[email protected]/bucket.go:706 +0x35
go.etcd.io/bbolt.(*Bucket)._forEachPageNode(0xc0000b6ec0, 0x386?, 0x3, 0xc00009d858)
        go.etcd.io/[email protected]/bucket.go:528 +0x59
go.etcd.io/bbolt.(*Bucket)._forEachPageNode(0xc0000b6ec0, 0x937b?, 0x2, 0xc00009d858)
        go.etcd.io/[email protected]/bucket.go:535 +0x14c
go.etcd.io/bbolt.(*Bucket)._forEachPageNode(0xc0000b6ec0, 0x7865646e692d?, 0x1, 0xc00009d858)
        go.etcd.io/[email protected]/bucket.go:535 +0x14c
go.etcd.io/bbolt.(*Bucket)._forEachPageNode(0xc0000b6ec0, 0x8?, 0x0, 0xc00009d858)
        go.etcd.io/[email protected]/bucket.go:535 +0x14c
go.etcd.io/bbolt.(*Bucket).forEachPageNode(0xc0000b6ec0?, 0xc00026c000?)
        go.etcd.io/[email protected]/bucket.go:521 +0x3f
go.etcd.io/bbolt.(*Bucket).free(0xc0000b6ec0)
        go.etcd.io/[email protected]/bucket.go:704 +0x45
go.etcd.io/bbolt.(*Bucket).DeleteBucket(0xc00018c9b8, {0x34cbf18, 0xa, 0xa})
        go.etcd.io/[email protected]/bucket.go:251 +0x19a
go.etcd.io/bbolt.(*Tx).DeleteBucket(...)
        go.etcd.io/[email protected]/tx.go:123
github.com/lightninglabs/chantools/lnd.(*transaction).DeleteTopLevelBucket(0x8?, {0x34cbf18?, 0xc000146601?, 0xc00009d980?})
        github.com/lightninglabs/chantools/lnd/channeldb.go:135 +0x25
github.com/lightningnetwork/lnd/graph/db.(*KVStore).Wipe.func1({0x22dbb98, 0xc00011c008})
        github.com/lightningnetwork/[email protected]/graph/db/kv_store.go:308 +0x6e
github.com/lightninglabs/chantools/lnd.(*backend).Update(0xc00011c458, 0x2094140, 0xc0001345d0?)
        github.com/lightninglabs/chantools/lnd/channeldb.go:479 +0x90
github.com/lightningnetwork/lnd/kvdb.Update(...)
        github.com/lightningnetwork/lnd/[email protected]/interface.go:16
github.com/lightningnetwork/lnd/graph/db.(*KVStore).Wipe(0xc00003f020)
        github.com/lightningnetwork/[email protected]/graph/db/kv_store.go:306 +0x31
main.(*dropChannelGraphCommand).Execute(0xc0000b7000, 0x19?, {0xc000440e20?, 0x1853a20?, 0xc000113ab0?})
        github.com/lightninglabs/chantools/cmd/chantools/dropchannelgraph.go:114 +0x256
github.com/spf13/cobra.(*Command).execute(0xc00023db88, {0xc0000b7800, 0x4, 0x4})
        github.com/spf13/[email protected]/command.go:852 +0x68a
github.com/spf13/cobra.(*Command).ExecuteC(0x367c760)
        github.com/spf13/[email protected]/command.go:960 +0x389
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/[email protected]/command.go:897
main.main()
        github.com/lightninglabs/chantools/cmd/chantools/root.go:150 +0x58a

[youngzhenhao]

channel.db file here

[ziggie1984]

seems like your channel.db is corrupted, which means you need to go ahead and trigger the SCB backup recovery process.

[ziggie1984]

you could try compacting the db via chantools and see whether this can recover the state, but I think the db is corrupted and cannot be recovered.