[bug]: LND on chain balance stuck pending with 0 confirmations

[cliffpeskin]

Background / Repro

• I sent about .118 BTC to my lnd wallet address a couple days ago (https://mempool.space/tx/641291b360f31a3e07c14b204989ac1ec2c750151fcf01f1f06a170a72226ed1)
• LND wallet appears to have then forwarded these funds here https://mempool.space/address/bc1p07y2czqsh80amtmjqr04kn4dem4lgzghg5z68vclqaw83sd7w8zs4ypdu5
• Some additional deposits were made to my LND wallet via LOOP and a channel that mysteriously force closed and now all associated UTXOs are showing as 0 confirmations / pending in my LND wallet.

My environment

• LND v0.15.4-beta"
• Ubuntu Server 22.04.1 LTS
• Neutrino backend
• Running LNDg for auto-rebalancing

Expected behavior

• Transactions should all be confirmed

Actual behavior

• lncli listunspent returns:

{
        "utxos": [
                {
                        "address_type": 4,
                        "address": "bc1p4ceq55su2n4u33lwekpveyu8xv0j29722qyx8sqmkemayrxuexqszyhtll",
                        "amount_sat": 1593279,
                        "pk_script": "5120ae320a521c54ebc8c7eecd82cc9387331f2517ca500863c01bb677d20cdcc981",
                        "outpoint": "a705f0411200305482ce7610819c3c44a5cf75a04d3f3451ce9319a433add9e9:0",
                        "confirmations": 0
                },
                {
                        "address_type": 4,
                        "address": "bc1pjz6y9ncsxxs3qf2hca6jmwlhxtgalppe0vyhv2dmrk67nyq8mfaqkfs3at",
                        "amount_sat": 125223,
                        "pk_script": "512090b442cf1031a1102557c7752dbbf732d1df84397b097629bb1db5e99007da7a",
                        "outpoint": "9e4821ade556dee2c2a9c3dd1e8a26a061756240618be4430c29ee14dfe3bd57:0",
                        "confirmations": 0
                },
                {
                        "address_type": 4,
                        "address": "bc1p328h3gxxu366cna5fqwcufn2rpq0t253lhkm935gre5fa4c2r46qkk2va2",
                        "amount_sat": 92506,
                        "pk_script": "51208a8f78a0c6e475ac4fb4481d8e266a1840f5aa91fdedb2c6881e689ed70a1d74",
                        "outpoint": "df37f8cd231d798fda5f1e08839b4fdd40a0ff6616ddedf938a89778db65e2a9:0",
                        "confirmations": 0
                },
                {
                        "address_type": 4,
                        "address": "bc1pdftwjk8qgp46ly7k92ncgm6cevk9zjqf6h8nhqnf5rtkk0pmhzrsf2ygw6",
                        "amount_sat": 73341,
                        "pk_script": "51206a56e958e0406baf93d62aa7846f58cb2c514809d5cf3b8269a0d76b3c3bb887",
                        "outpoint": "d6ef9af3bf019f25e04de7af4a382cf268d4cdde6aa90cbcbfcb910d50399cb4:0",
                        "confirmations": 0
                },
                {
                        "address_type": 4,
                        "address": "bc1pg6rm77e3ykmkqljgazy56gn2wqwy725h54p3ukwhk7vrcw2zwfcsfl79kg",
                        "amount_sat": 399759,
                        "pk_script": "51204687bf7b3125b7607e48e8894d226a701c4f2a97a5431e59d7b7983c39427271",
                        "outpoint": "129eb20dea1a3b95a0d35fd4d67319971c6a8353aead577a6a8fb01e79f30df1:0",
                        "confirmations": 0
                },
                {
                        "address_type": 4,
                        "address": "bc1pdaez8093d6k3wcgvtcjsmxnf699ft4nmc6ey9fnyqafwf7qc0uasgdadea",
                        "amount_sat": 11867670,
                        "pk_script": "51206f7223bcb16ead17610c5e250d9a69d14a95d67bc6b242a6640752e4f8187f3b",
                        "outpoint": "d42ca54981e6c357628f156f1d3aaca78871dce10efb6b871e13bc4ddc4cda89:0",
                        "confirmations": 0
                },
                {
                        "address_type": 4,
                        "address": "bc1p5qsaevzc2j052mmqjxv4x4s63cex2s3fzvztp554u3gcftd8x44s904puv",
                        "amount_sat": 9998242,
                        "pk_script": "5120a021dcb058549f456f60919953561a8e326542291304b0d295e45184ada7356b",
                        "outpoint": "44b5a55a230d32b35dff7d548ad8cf1943d997ba8007460b10b0e2787b7d9a27:0",
                        "confirmations": 0
                },
                {
                        "address_type": 4,
                        "address": "bc1p228nmtgswztwkknml2309f8cdlyj29c5dl86ueh9nkff5vkqj72sp0qntw",
                        "amount_sat": 7089,
                        "pk_script": "5120528f3dad107096eb5a7bfaa2f2a4f86fc92517146fcfae66e59d929a32c09795",
                        "outpoint": "52a2cde1ebe55b6ee205b7e8901401bd864bdc62d36be094ded8c84556bb80bc:0",
                        "confirmations": 0
                }
        ]
}

• lncli getinfo shows

    "synced_to_chain": true,
    "synced_to_graph": true,

• Not sure what this means but lnd --reset-wallet-transactions returns error 'input likely spends a taproot output which is not yet supported', example below:

2022-11-17 15:51:29.478 [ERR] BTCN: filter for block 00000000000000000004dba192fe1df314f12dd540b58868272afa8554d3d5b3 might be invalid, input 0 of tx 3b84b521c95517f711eea645412eae969da4397e031d0d8a23d7104f75358c92 spends pk script 00205cbfc092fff438f616ce9d4096c93a6881fb0803ca1275645f78fd00cfc55bbe which wasn't matched by filter. The input likely spends a taproot output which is not yetsupported
2022-11-17 15:51:29.480 [INF] BTCN: Lost peer 85.145.86.126:8333 (outbound)
2022-11-17 15:51:29.508 [ERR] BTCN: filter for block 00000000000000000004dba192fe1df314f12dd540b58868272afa8554d3d5b3 might be invalid, input 0 of tx c5ce9853d4071dae736bfa89710558be6cd694a3e5e59a7ec31d95697c47828c spends pk script 002078050b9c7e37e9ef4b7267a5adfee99fdb686a7e98436184262fb72c83c1683d which wasn't matched by filter. The input likely spends a taproot output which is not yetsupported

[Roasbeef]

So none of those outpoints actually exist on chain, which is why they show as having zero confirmations.

[saubyk]

@cliffpeskin moving this issue to a discussion in case you have further questions. Mempool is really backed up right now which may be the reason why your UTXOs are not confirming, you can choose to wait or bump up the pending transactions via CPFP

[cliffpeskin]

Thanks @Roasbeef and @saubyk,

So apologies for the novice question but 2 hours ago lncli listunspent returned (among other UTXOs):

                        "address_type": 4,
                        "address": "bc1pdaez8093d6k3wcgvtcjsmxnf699ft4nmc6ey9fnyqafwf7qc0uasgdadea",
                        "amount_sat": 11867670,
                        "pk_script": "51206f7223bcb16ead17610c5e250d9a69d14a95d67bc6b242a6640752e4f8187f3b",
                        "outpoint": "d42ca54981e6c357628f156f1d3aaca78871dce10efb6b871e13bc4ddc4cda89:0",
                        "confirmations": 0`

And now (2 hours later) the above UTXO is gone and instead lncli listunspent returns:

                        "address_type": 4,
                        "address": "bc1p07y2czqsh80amtmjqr04kn4dem4lgzghg5z68vclqaw83sd7w8zs4ypdu5",
                        "amount_sat": 11867523,
                        "pk_script": "51207f88ac0810b9dfddaf7200df5b4eadceebf409174505a3b31f075c78c1be71c5",
                        "outpoint": "07848a56654f5a13025f219f34dc0ce07cd813245d00fa8d4dcaec9871a1a66c:1",
                        "confirmations": 217

The first UTXO address is empty and has no transactional history. Why may this be the case? And, how would I go about determining what that first address is and what triggered it to be replaced by the subsequent UTXO with 217 confirmations.

[saubyk]

Hi @cliffpeskin recommend looking at your transaction history to trace the UTXO lineage, instead of directly examining the listunspent output

[cliffpeskin]

Thanks @saubyk, appears to be related to lnd sweeps. The latest sweep shown below in RTL is not in the mempool (has been several hours). Do you know why that would be?

https://mempool.space/tx/a705f0411200305482ce7610819c3c44a5cf75a04d3f3451ce9319a433add9e9

From lncli:

[cliffpeskin]

Hi @saubyk, just an update here that this sweep from yesterday still shows as unconfirmed (or 0:sweep in RTL) and does not appear in mempool https://mempool.space/tx/a705f0411200305482ce7610819c3c44a5cf75a04d3f3451ce9319a433add9e9

Any ideas on how to resolve?

[saubyk]

So, couple of things to note here which are making the situation difficult to assess:

1. I have no idea of your transaction history and how the UTXOs got generated in the first place. You said something like "Some additional deposits were made to my LND wallet via LOOP and a channel that mysteriously force closed" in the OP, but that doesn't provide any info on the resulting txns. You're in the best position to track down the transactions from the transaction history and figure out what's going on
2. You're using neutrino, may be it's using too low a fee for the current state of the mempool
3. Mempool is backed up still, so you may need to be patient and wait for it to clear or try CPFPing your unconfirmed outputs

At this point, I would recommend to try restarting the node and see if that changes the state

[cliffpeskin]

Thanks @saubyk for giving this another look.

I've tried restarting the node several times but unfortunately that didn't work.

1. I'm still new to LND / Lightning but I believe this pending UXTO is an automatic sweep that was generated soon after my node force closed a channel (not sure why it force closed the channel).

The force close transaction referenced by the pending UTXO is here https://mempool.space/tx/f3719e7fd91c71320fd8337e1e6c0574516c2f92d2a4fb8e4226055fc80d1599 and below is the lncli listchaintxns output for the pending UXTO.

2. I thought the mempool was just the waiting room / auction for getting on-chain. Are you saying low fee transactions will sometimes not even appear in the mempool?

3. I will look into CPFPing the output, thanks

{
    "transactions": [
        {
            "tx_hash": "a705f0411200305482ce7610819c3c44a5cf75a04d3f3451ce9319a433add9e9",
            "amount": "147",
            "num_confirmations": 0,
            "block_hash": "",
            "block_height": 0,
            "time_stamp": "1668801228",
            "total_fees": "0",
            "dest_addresses": [
                "bc1p4ceq55su2n4u33lwekpveyu8xv0j29722qyx8sqmkemayrxuexqszyhtll"
            ],
            "output_details": [
                {
                    "output_type": "SCRIPT_TYPE_PUBKEY_HASH",
                    "address": "bc1p4ceq55su2n4u33lwekpveyu8xv0j29722qyx8sqmkemayrxuexqszyhtll",
                    "pk_script": "5120ae320a521c54ebc8c7eecd82cc9387331f2517ca500863c01bb677d20cdcc981",
                    "output_index": "0",
                    "amount": "1593279",
                    "is_our_address": true
                }
            ],
            "raw_tx_hex": "02000000000102a6512ad3697ec0f5ee631e896d7283aef19080c8f2bc80da52b8ee0b735b6f3001000000000000000099150dc85f0526428efba4d2922f6c5174056c1e7e33d80f32711cd97f9e71f300000000000000000001bf4f180000000000225120ae320a521c54ebc8c7eecd82cc9387331f2517ca500863c01bb677d20cdcc98102483045022100df2f528737e9e22c362a6357713b796b1bd371ddf395cc2de73245e49462631c022057201c7229836e8e3312ab321850979985bc8a6961e20eb15551f6f6d8d3ad3f01282103fe1c6c6b1702e6f578e2e1ad5829748a134ea91effed7cbe8b25bb33c6ea67d0ac736460b26801401962f3d37c69a5c515547439ad01f59f0a7a72aa30470770a905d9351a29d5c19361650b94bfe911ecc1f0bc633f113bf5d3e74997f206626f104215989006f1a8a60b00",
            "label": "0:sweep",
            "previous_outpoints": [
                {
                    "outpoint": "306f5b730beeb852da80bcf2c88090f1ae83726d891e63eef5c07e69d32a51a6:1",
                    "is_our_output": false
                },
                {
                    "outpoint": "f3719e7fd91c71320fd8337e1e6c0574516c2f92d2a4fb8e4226055fc80d1599:0",
                    "is_our_output": true
                }
            ]
        },   

[cliffpeskin]

Update: Unconfirmed balance is now confirmed and the above mentioned stuck transaction no longer appears when querying lncli listchaintxns

Seems this issue may be related to #7104 as both involve a "label": "0:sweep" transaction that is not appearing in the mempool, neutrino backend and recently closed channel.