[bug]: Unexpected Pathfinding Failure After Edge Node Experiences Unhandled Error FeeInsufficient

[ryanthegentry]

Background

I am testing single hop Taproot Assets LN payments in Polar. I have three litd v0.14.0-alpha.rc1 nodes (each backed by their own bitcoind v27.0 node) Alice, Bob, and Carol. Alice has minted 21M units of a Taproot Asset called USDTest with a decimal_display of 6, and opened a 1M asset channel to Bob. Bob has then opened a 1M sats channel to Carol, and is using the improved edge node config --taproot-assets.experimental.rfq.mockoracleassetsperbtc=100000000. I am testing Carol trying to pay Taproot Assets invoices generated by Alice with sats, with Bob acting as the edge node.

Before beginning this sequence of test payments, Alice's listchannels output looked like so:

litd@alice:/$ lncli listchannels
{
    "channels": [
        {
            "active": true,
            "remote_pubkey": "03926cb6785ba0b71e97f1463bbdda457b4bc336048d2b1d6070844dbfbc5c9bcb",
            "channel_point": "560027dbab7f95d3c131c71ba03bf0b996d8c218f6bd5e7ad9240c9922635f91:0",
            "chan_id": "135239930281984",
            "capacity": "100000",
            "local_balance": "5140",
            "remote_balance": "91779",
            "commit_fee": "2421",
            "commit_weight": "958",
            "fee_per_kw": "2500",
            "unsettled_balance": "0",
            "total_satoshis_sent": "86779",
            "total_satoshis_received": "0",
            "num_updates": "8",
            "pending_htlcs": [],
            "csv_delay": 144,
            "private": true,
            "initiator": true,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "1000",
            "remote_chan_reserve_sat": "1062",
            "static_remote_key": false,
            "commitment_type": "SIMPLE_TAPROOT_OVERLAY",
            "lifetime": "306",
            "uptime": "306",
            "close_address": "",
            "push_amount_sat": "5000",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "1000",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "99000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 83
            },
            "remote_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "1062",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "99000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 83
            },
            "alias_scids": [
                "17592186044416000000"
            ],
            "zero_conf": false,
            "zero_conf_confirmed_scid": "0",
            "peer_alias": "bob",
            "peer_scid_alias": "17592186044416000000",
            "memo": "",
            "custom_channel_data": {
                "assets": [
                    {
                        "asset_utxo": {
                            "version": 1,
                            "asset_genesis": {
                                "genesis_point": "bb0cac6719ce54b0b866dfbb9ad665c62942b9d2bb22bfa6314e0faf33b6cf3d:1",
                                "name": "USDTest",
                                "meta_hash": "98c6ac28bc381a4bb270d94896129f355950eb8248a59c54ea933b68d1c7d202",
                                "asset_id": "957ba60125387d7df01c98820d1684470882c6d9214478c1405a79fb22ca3646"
                            },
                            "amount": 1000000000000,
                            "script_key": "0250aaeb166f4234650d84a2d8a130987aeaf6950206e0905401ee74ff3f8d18e6"
                        },
                        "capacity": 1000000000000,
                        "local_balance": 999999899997,
                        "remote_balance": 100003
                    }
                ]
            }
        }
    ]
}

Carol had 186,070 local_balance of sats, and 810,460 remote_balance of sats (owned by Bob).

I then made one successful payment where Alice invoiced 356 assets, and Carol paid it with sats:

litd@carol:/$ lncli payinvoice lnbcrt3560n1pn4p55tpp59r6y222rg0e0lackutqta6e4skqqzd8ts44esjm6kls2me7nwx9sdqqcqzzsxqzfvrzjqwfxednctwstw85h79rrh0w6g4a5hsekqjxjk8tqwzzym0autjduhadfj89err2x3qqqqqlgqqqqqqgq2qsp5l6ppwmd2qmrqjgtc7vwey356plhjutfkmde7r7775qmdx8dgtwms9qxpqysgqwu35rt33xgfyapsgk7pav2kr6rvhh5a0z3xsj53s7nj7mjny2qmkj7d2genslhahg5umv94g2phnkhramwnl6jqztusg8swpcvl0xpcpkzz00e
Payment hash: 28f445294343f2fff716e2c0beeb3585800134eb856b984b7ab7e0ade7d3718b
Description: 
Amount (in satoshis): 356
Fee limit (in satoshis): 356
Destination: 03e5011782766a1582562e9585cb0935d3892e20cf84ff233bcba244bac3ae8417
Confirm payment (yes/no): yes
+------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT        | ROUTE       |
+------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
| SUCCEEDED  |        0.054 |        0.689 | 356          | 1   |      297 | 141837000048640 | bob->03e501 |
+------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
Amount + fee:   356 + 1 sat
Payment hash:   28f445294343f2fff716e2c0beeb3585800134eb856b984b7ab7e0ade7d3718b
Payment status: SUCCEEDED, preimage: 200330c6b3db0ecc8455b79471061bab7eb1617d53d93f34ef3fff8ed1c9e654

And then one expected failure, where Alice only invoiced 1 asset, and Carol failed to find a route (bc Bob is not interested in receiving 0 + 0 sats, and forwarding out 354 sats + 1 asset).

litd@carol:/$ lncli payinvoice lnbcrt10n1pn4p544pp5uux2fq4x3uftgfj9vrdc5uh2vg055pql5zuqf5486y2jcx9zm9ysdqqcqzzsxqzfvrzjqwfxednctwstw85h79rrh0w6g4a5hsekqjxjk8tqwzzym0autjduha3atn5svvtqssqqqqlgqqqqqqgq2qsp5lmd9068cgz5ezqsv8fzrknzemw6f8nfy6uatg8wkut4wredv8pkq9qxpqysgqqrghvsuvhaqj76faeu3m828zuhmjwrywr52jnsqv566jz8d7zm384rpac8k6juuqjs34msetxd850n3g9hp5x0jwwuscdp74cl2rjaqp44jfmp
Payment hash: e70ca482a68f12b4264560db8a72ea621f4a041fa0b804d2a7d1152c18a2d949
Description: 
Amount (in satoshis): 1
Fee limit (in satoshis): 1
Destination: 03e5011782766a1582562e9585cb0935d3892e20cf84ff233bcba244bac3ae8417
Confirm payment (yes/no): yes
+----------------------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
| HTLC_STATE                 | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT        | ROUTE       |
+----------------------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
| FEE_INSUFFICIENT @ 1st hop |        0.044 |        0.313 | 1            | 1   |      297 | 141837000048640 | bob->03e501 |
| FEE_INSUFFICIENT @ 1st hop |        0.344 |        0.633 | 1            | 1   |      297 | 141837000048640 | bob->03e501 |
+----------------------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
Amount + fee:   0 + 0 sat
Payment hash:   e70ca482a68f12b4264560db8a72ea621f4a041fa0b804d2a7d1152c18a2d949
Payment status: FAILED, reason: FAILURE_REASON_NO_ROUTE
[lncli] FAILED

The unexpected behavior begins here however, as Carol is no longer able to pay Alice after this. Initially Alice and Bob's Taproot Assets channel remains active, but Carol just immediately fails an attempted payment:

litd@carol:/$ lncli payinvoice lnbcrt1m1pn4p5e2pp5vcw7dc7vnmumrp4qhhg5munql56dw9vgrs63mldrwvwy8rwx86msdqqcqzzsxqzfvrzjqwfxednctwstw85h79rrh0w6g4a5hsekqjxjk8tqwzzym0autjduhalp704xtquqtcqqqqlgqqqqqqgq2qsp5uy2a8xgpvyv4qjvlx4a02k48j8rnrc2qm8l9gu3selvhv273y7as9qxpqysgquadp5anaqc7d92gkzl4a7g6n3f7amkthhuvxwpn0gk9mkqsv9a4re6yerr29573le7w60cvu0j45c9gv4skycrdjl8gqstp5qp3zkugqaq54tk
Payment hash: 661de6e3cc9ef9b186a0bdd14df260fd34d715881c351dfda3731c438dc63eb7
Description: 
Amount (in satoshis): 100000
Fee limit (in satoshis): 5000
Destination: 03e5011782766a1582562e9585cb0935d3892e20cf84ff233bcba244bac3ae8417
Confirm payment (yes/no): yes
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT | ROUTE |
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
Amount + fee:   0 + 0 sat
Payment hash:   661de6e3cc9ef9b186a0bdd14df260fd34d715881c351dfda3731c438dc63eb7
Payment status: FAILED, reason: FAILURE_REASON_NO_ROUTE
[lncli] FAILED

Assuming she blacklisted the receiving node from her mission control for some reason, I reset it and try to pay Alice's invoice again but this time am hit with a TEMPORARY_CHANNEL_FAILURE at the Taproot Assets channel:

litd@carol:/$ lncli resetmc
litd@carol:/$ lncli payinvoice lnbcrt1m1pn4p46fpp560me4hn0xnur6ske9nrv73hendyajnu3tc88eju4yftkl96dtmcsdqqcqzzsxqzfvrzjqwfxednctwstw85h79rrh0w6g4a5hsekqjxjk8tqwzzym0autjduhar6sp5shfeezcqqqqlgqqqqqqgq2qsp568xqej6w0duldh6wayn7hwrer2l5ld78e0eydctq2r9qkgfhjkms9qxpqysgqyatzwdagacmfw39r749lcrr858k23wf6hzeshr7ltzlypyg6te8rrjrna2m2y7ra7uew085gze0d9ezkjh0fnh0ju5zncgwqmdrgtxgpljczz7
Payment hash: d3f79ade6f34f83d42d92cc6cf46f99b49d94f915e0e7ccb9522576f974d5ef1
Description: 
Amount (in satoshis): 100000
Fee limit (in satoshis): 5000
Destination: 03e5011782766a1582562e9585cb0935d3892e20cf84ff233bcba244bac3ae8417
Confirm payment (yes/no): yes
+-------------------------------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT        | ROUTE       |
+-------------------------------------+--------------+--------------+--------------+-----+----------+-----------------+------------
+-------------------------------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT        | ROUTE       |
+-------------------------------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
| TEMPORARY_CHANNEL_FAILURE @ 1st hop |        0.065 |       60.471 | 100000       | 1.1 |      297 | 141837000048640 | bob->03e501 |
+-------------------------------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
Amount + fee:   0 + 0 sat
Payment hash:   d3f79ade6f34f83d42d92cc6cf46f99b49d94f915e0e7ccb9522576f974d5ef1
Payment status: FAILED, reason: FAILURE_REASON_TIMEOUT
[lncli] FAILED

And after this attempt, Alice no longer sees the channel as active:

litd@alice:/$ lncli listchannels
{
    "channels": [
        {
            "active": false,
            "remote_pubkey": "03926cb6785ba0b71e97f1463bbdda457b4bc336048d2b1d6070844dbfbc5c9bcb",
            "channel_point": "560027dbab7f95d3c131c71ba03bf0b996d8c218f6bd5e7ad9240c9922635f91:0",
            "chan_id": "135239930281984",
            "capacity": "100000",
            "local_balance": "5494",
            "remote_balance": "91425",
            "commit_fee": "2421",
            "commit_weight": "958",
            "fee_per_kw": "2500",
            "unsettled_balance": "0",
            "total_satoshis_sent": "86779",
            "total_satoshis_received": "354",
            "num_updates": "10",
            "pending_htlcs": [],
            "csv_delay": 144,
            "private": true,
            "initiator": true,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "1000",
            "remote_chan_reserve_sat": "1062",
            "static_remote_key": false,
            "commitment_type": "SIMPLE_TAPROOT_OVERLAY",
            "lifetime": "1834",
            "uptime": "1831",
            "close_address": "",
            "push_amount_sat": "5000",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "1000",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "99000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 83
            },
            "remote_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "1062",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "99000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 83
            },
            "alias_scids": [
                "17592186044416000000",
                "17701840113862788744",
                "17743440262810656900",
                "17861825785179111518",
                "17616534081068546326"
            ],
            "zero_conf": false,
            "zero_conf_confirmed_scid": "0",
            "peer_alias": "bob",
            "peer_scid_alias": "17592186044416000000",
            "memo": "",
            "custom_channel_data": {
                "assets": [
                    {
                        "asset_utxo": {
                            "version": 1,
                            "asset_genesis": {
                                "genesis_point": "bb0cac6719ce54b0b866dfbb9ad665c62942b9d2bb22bfa6314e0faf33b6cf3d:1",
                                "name": "USDTest",
                                "meta_hash": "98c6ac28bc381a4bb270d94896129f355950eb8248a59c54ea933b68d1c7d202",
                                "asset_id": "957ba60125387d7df01c98820d1684470882c6d9214478c1405a79fb22ca3646"
                            },
                            "amount": 1000000000000,
                            "script_key": "0250aaeb166f4234650d84a2d8a130987aeaf6950206e0905401ee74ff3f8d18e6"
                        },
                        "capacity": 1000000000000,
                        "local_balance": 999999900353,
                        "remote_balance": 99647
                    }
                ]
            }
        }
    ]
}

I can get the channel back to active by disconnecting Alice from Bob, and then reconnecting:

itd@alice:/$ lncli disconnect --node_key 03926cb6785ba0b71e97f1463bbdda457b4bc336048d2b1d6070844dbfbc5c9bcb
{}
litd@alice:/$ lncli disconnect --node_key 03926cb6785ba0b71e97f1463bbdda457b4bc336048d2b1d6070844dbfbc5c9bcb
{}
litd@alice:/$ lncli disconnect --node_key 03926cb6785ba0b71e97f1463bbdda457b4bc336048d2b1d6070844dbfbc5c9bcb
[lncli] rpc error: code = Unknown desc = unable to disconnect peer: peer 03926cb6785ba0b71e97f1463bbdda457b4bc336048d2b1d6070844dbfbc5c9bcb is not connected
litd@alice:/$ lncli connect 03926cb6785ba0b71e97f1463bbdda457b4bc336048d2b1d6070844dbfbc5c9bcb@172.18.0.7:9735
[lncli] rpc error: code = Unknown desc = already connected to peer: 03926cb6785ba0b71e97f1463bbdda457b4bc336048d2b1d6070844dbfbc5c9bcb@172.18.0.7:58830
litd@alice:/$ lncli connect 03926cb6785ba0b71e97f1463bbdda457b4bc336048d2b1d6070844dbfbc5c9bcb@172.18.0.7:9735
[lncli] rpc error: code = Unknown desc = already connected to peer: 03926cb6785ba0b71e97f1463bbdda457b4bc336048d2b1d6070844dbfbc5c9bcb@172.18.0.7:58830
litd@alice:/$ lncli listchannels
{
    "channels": [
        {
            "active": true,
            "remote_pubkey": "03926cb6785ba0b71e97f1463bbdda457b4bc336048d2b1d6070844dbfbc5c9bcb",
            "channel_point": "560027dbab7f95d3c131c71ba03bf0b996d8c218f6bd5e7ad9240c9922635f91:0",
            "chan_id": "135239930281984",
            "capacity": "100000",
            "local_balance": "5494",
            "remote_balance": "91425",
            "commit_fee": "2421",
            "commit_weight": "958",
            "fee_per_kw": "2500",
            "unsettled_balance": "0",
            "total_satoshis_sent": "86779",
            "total_satoshis_received": "354",
            "num_updates": "10",
            "pending_htlcs": [],
            "csv_delay": 144,
            "private": true,
            "initiator": true,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "1000",
            "remote_chan_reserve_sat": "1062",
            "static_remote_key": false,
            "commitment_type": "SIMPLE_TAPROOT_OVERLAY",
            "lifetime": "1802",
            "uptime": "1798",
            "close_address": "",
            "push_amount_sat": "5000",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "1000",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "99000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 83
            },
            "remote_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "1062",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "99000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 83
            },
            "alias_scids": [
                "17592186044416000000",
                "17701840113862788744",
                "17743440262810656900",
                "17861825785179111518",
                "17616534081068546326"
            ],
            "zero_conf": false,
            "zero_conf_confirmed_scid": "0",
            "peer_alias": "bob",
            "peer_scid_alias": "17592186044416000000",
            "memo": "",
            "custom_channel_data": {
                "assets": [
                    {
                        "asset_utxo": {
                            "version": 1,
                            "asset_genesis": {
                                "genesis_point": "bb0cac6719ce54b0b866dfbb9ad665c62942b9d2bb22bfa6314e0faf33b6cf3d:1",
                                "name": "USDTest",
                                "meta_hash": "98c6ac28bc381a4bb270d94896129f355950eb8248a59c54ea933b68d1c7d202",
                                "asset_id": "957ba60125387d7df01c98820d1684470882c6d9214478c1405a79fb22ca3646"
                            },
                            "amount": 1000000000000,
                            "script_key": "0250aaeb166f4234650d84a2d8a130987aeaf6950206e0905401ee74ff3f8d18e6"
                        },
                        "capacity": 1000000000000,
                        "local_balance": 999999900353,
                        "remote_balance": 99647
                    }
                ]
            }
        }
    ]
}

But then Carol encounters the same TEMPORARY_CHANNEL_FAILURE when trying to pay Alice's invoice again:

litd@carol:/$ lncli payinvoice lnbcrt1m1pn4p46fpp560me4hn0xnur6ske9nrv73hendyajnu3tc88eju4yftkl96dtmcsdqqcqzzsxqzfvrzjqwfxednctwstw85h79rrh0w6g4a5hsekqjxjk8tqwzzym0autjduhar6sp5shfeezcqqqqlgqqqqqqgq2qsp568xqej6w0duldh6wayn7hwrer2l5ld78e0eydctq2r9qkgfhjkms9qxpqysgqyatzwdagacmfw39r749lcrr858k23wf6hzeshr7ltzlypyg6te8rrjrna2m2y7ra7uew085gze0d9ezkjh0fnh0ju5zncgwqmdrgtxgpljczz7
Payment hash: d3f79ade6f34f83d42d92cc6cf46f99b49d94f915e0e7ccb9522576f974d5ef1
Description: 
Amount (in satoshis): 100000
Fee limit (in satoshis): 5000
Destination: 03e5011782766a1582562e9585cb0935d3892e20cf84ff233bcba244bac3ae8417
Confirm payment (yes/no): yes
+-------------------------------------+--------------+--------------+--------------+------+----------+-----------------+-------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE  | TIMELOCK | CHAN_OUT        | ROUTE       |
+-------------------------------------+--------------+--------------+--------------+------+----------+-----------------+-------------+
| TEMPORARY_CHANNEL_FAILURE @ 1st hop |        0.069 |       60.681 | 50000        | 1.05 |      297 | 141837000048640 | bob->03e501 |
| TEMPORARY_CHANNEL_FAILURE @ 1st hop |        0.120 |       60.582 | 50000        | 1.05 |      297 | 141837000048640 | bob->03e501 |
+-------------------------------------+--------------+--------------+--------------+------+----------+-----------------+-------------+
Amount + fee:   0 + 0 sat
Payment hash:   d3f79ade6f34f83d42d92cc6cf46f99b49d94f915e0e7ccb9522576f974d5ef1
Payment status: FAILED, reason: FAILURE_REASON_TIMEOUT
[lncli] FAILED

And Alice sees the channel as inactive:

litd@alice:/$ lncli listchannels
{
    "channels": [
        {
            "active": false,
            "remote_pubkey": "03926cb6785ba0b71e97f1463bbdda457b4bc336048d2b1d6070844dbfbc5c9bcb",
            "channel_point": "560027dbab7f95d3c131c71ba03bf0b996d8c218f6bd5e7ad9240c9922635f91:0",
            "chan_id": "135239930281984",
            "capacity": "100000",
            "local_balance": "5494",
            "remote_balance": "91425",
...

After disconnecting and reconnecting again, Bob attempts to pay Alice's invoice directly, instead of acting as an edge node, but the payment fails and the CLI hangs (i.e. needs to be manually killed):

litd@bob:/$ litcli ln payinvoice --pay_req lnbcrt1m1pn4pkcepp563f59zph5k7hxs3mt6x25reskwqufnz6vuqvu5pkxwxpkwwnqvtsdqqcqzzsxqzfvrzjqwfxednctwstw85h79rrh0w6g4a5hsekqjxjk8tqwzzym0autjduhasghrxk4v095yqqqqlgqqqqqqgq2qsp59fy4huxk494hr0ttgxqtadmedv9qhnjaac0m32lmm7jes2gg4t0s9qxpqysgqv66v49uwld6e5s2rfws997ne4tk0udkmvarz4u4f5lfkezf82phrm6vx073404n04u64st6w8av395uz4wyhk9gquuw298eqc6ks8ecqp67jkd --asset_id 957ba60125387d7df01c98820d1684470882c6d9214478c1405a79fb22ca3646 --rfq_peer_pubkey 03e5011782766a1582562e9585cb0935d3892e20cf84ff233bcba244bac3ae8417
Payment hash: d453428837a5bd73423b5e8caa0f30b381c4cc5a6700ce5036338c1b39d30317
Description: 
Amount (in satoshis): 100000
Fee limit (in satoshis): 5000
Destination: 03e5011782766a1582562e9585cb0935d3892e20cf84ff233bcba244bac3ae8417
Confirm payment (yes/no): yes
Got quote for 100000 asset units at 1000 msat/unit from peer 03e5011782766a1582562e9585cb0935d3892e20cf84ff233bcba244bac3ae8417 with SCID 17788700425902974056
+-------------------------------------+--------------+--------------+--------------+-----+----------+-----------------+-------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT        | ROUTE |
+-------------------------------------+--------------+--------------+--------------+-----+----------+-----------------+-------+
| IN_FLIGHT                           |        0.821 |            - | 50000        | 0   |      217 | 135239930281984 | alice |
| TEMPORARY_CHANNEL_FAILURE @ 0th hop |        1.864 |       62.011 | 50000        | 0   |      217 | 135239930281984 | alice |
+-------------------------------------+--------------+--------------+--------------+-----+----------+-----------------+-------+
Amount + fee:   0 + 0 sat
Payment hash:   d453428837a5bd73423b5e8caa0f30b381c4cc5a6700ce5036338c1b39d30317
Payment status: IN_FLIGHT
^C[litcli] rpc error: code = Canceled desc = context canceled

I think the key line in this final attempt from Bob's logs is here:

2024-12-04 22:45:30.860 [WRN] CRTR: ShortChannelID=123:1:0: link not found: channel link not found
2024-12-04 22:45:30.861 [DBG] CRTR: Skipped edge 135239930281984: not enough bandwidth, bandwidth=0 mSAT, amt=25000000 mSAT

And earlier in the saga during the initial failed payment, the key line in Bob's logs is here:

2024-12-04 22:08:09.344 [ERR] HSWC: Unhandled error while reforwarding htlc settle/fail over htlcswitch: FeeInsufficient(htlc_amt==354000 mSAT, update=(lnwire.ChannelUpdate) {

Your environment

Alice, Bob and Carol's litd config essentially look like this:

litd
  --httpslisten=0.0.0.0:8443
  --uipassword={{litdPass}}
  --network=regtest
  --lnd-mode=integrated
  --pool-mode=disable
  --loop-mode=disable
  --autopilot.disable
  --lnd.noseedbackup
  --lnd.debuglevel=debug
  --lnd.alias={{name}}
  --lnd.externalip={{name}}
  --lnd.tlsextradomain={{name}}
  --lnd.tlsextradomain={{containerName}}
  --lnd.tlsextradomain=host.docker.internal
  --lnd.listen=0.0.0.0:9735
  --lnd.rpclisten=0.0.0.0:10009
  --lnd.restlisten=0.0.0.0:8080
  --lnd.bitcoin.active
  --lnd.bitcoin.regtest
  --lnd.bitcoin.node=bitcoind
  --lnd.bitcoind.rpchost={{backendName}}
  --lnd.bitcoind.rpcuser={{rpcUser}}
  --lnd.bitcoind.rpcpass={{rpcPass}}
  --lnd.bitcoind.zmqpubrawblock=tcp://{{backendName}}:28334
  --lnd.bitcoind.zmqpubrawtx=tcp://{{backendName}}:28335
  --taproot-assets.allow-public-uni-proof-courier
  --taproot-assets.universe.public-access=rw
  --taproot-assets.universe.sync-all-assets
  --taproot-assets.allow-public-stats
  --taproot-assets.universerpccourier.skipinitdelay
  --taproot-assets.universerpccourier.backoffresetwait=1s
  --taproot-assets.universerpccourier.numtries=5
  --taproot-assets.universerpccourier.initialbackoff=300ms
  --taproot-assets.universerpccourier.maxbackoff=600ms
  --taproot-assets.experimental.rfq.priceoracleaddress=use_mock_price_oracle_service_promise_to_not_use_on_mainnet
  --taproot-assets.experimental.rfq.mockoracleassetsperbtc=100000000
  --lnd.trickledelay=50
  --lnd.gossip.sub-batch-delay=5ms
  --lnd.caches.rpc-graph-cache-duration=100ms
  --lnd.default-remote-max-htlcs=483
  --lnd.dust-threshold=5000000
  --lnd.protocol.option-scid-alias
  --lnd.protocol.zero-conf
  --lnd.protocol.simple-taproot-chans
  --lnd.protocol.simple-taproot-overlay-chans
  --lnd.protocol.wumbo-channels 
  --lnd.accept-keysend
  --lnd.protocol.custom-message=17
  --taproot-assets.proofcourieraddr=universerpc://carol:10009

Alice's tapcli getinfo output:

litd@alice:/$ tapcli getinfo
{
    "version": "0.5.0-alpha.rc1 commit=v0.5.0-rc1",
    "lnd_version": "0.18.4-beta.rc1",
    "network": "regtest",
    "lnd_identity_pubkey": "03e5011782766a1582562e9585cb0935d3892e20cf84ff233bcba244bac3ae8417",
    "node_alias": "alice",
    "block_height": 134,
    "block_hash": "14ba6f7566cbe4a95e1017f2207fd31d98b2c7a934af15283b2c2e97ceb8fab9",
    "sync_to_chain": true
}

Bob's tapcli getinfo output:

litd@bob:/$ tapcli getinfo
{
    "version": "0.5.0-alpha.rc1 commit=v0.5.0-rc1",
    "lnd_version": "0.18.4-beta.rc1",
    "network": "regtest",
    "lnd_identity_pubkey": "03926cb6785ba0b71e97f1463bbdda457b4bc336048d2b1d6070844dbfbc5c9bcb",
    "node_alias": "bob",
    "block_height": 134,
    "block_hash": "14ba6f7566cbe4a95e1017f2207fd31d98b2c7a934af15283b2c2e97ceb8fab9",
    "sync_to_chain": true
}

Carol's tapcli getinfo output:

litd@carol:/$ tapcli getinfo
{
    "version": "0.5.0-alpha.rc1 commit=v0.5.0-rc1",
    "lnd_version": "0.18.4-beta.rc1",
    "network": "regtest",
    "lnd_identity_pubkey": "02e804e18b4c9d95c32894dbf92e85a9effd55468d6736abbee7d316cee0a06464",
    "node_alias": "carol",
    "block_height": 134,
    "block_hash": "14ba6f7566cbe4a95e1017f2207fd31d98b2c7a934af15283b2c2e97ceb8fab9",
    "sync_to_chain": true
}

Steps to reproduce

1. Have a Taproot Assets channel between Alice and Bob where Alice has plenty of inbound liquidity, and a BTC channel between Bob and Carol where Bob has plenty of inbound liquidity.
2. Have Alice generate a 1 asset invoice (though I assume anything <354 sats would suffice), and have Carol attempt to pay it with sats.
3. Have Carol attempt to pay additional Taproot Assets invoices generated by Alice.
4. After failures, reset Carol's mission control and try again.
5. After Alice and Bob's Taproot Assets channel goes inactive, disconnect and reconnect them as peers, and try to get Carol to pay another Taproot Assets invoice from Alice again.
6. After getting the channel active again, try to get Bob to make a direct Taproot Assets LN payment to Alice.

Expected behavior

Carol should be able to successfully pay Alice in step 3, and in step 4, and the Taproot Assets channel should not go inactive every time it is asked to forward a payment in steps 5 and 6.

Actual behavior

After step 2, Alice and Bob's Taproot Assets channel is incapable for forwarding a payment. It appears Bob is unable to find the channel link.

Logs attached:

alice-deadchan.log
bob-deadchan.log
carol-deadchan.log

[guggero]

I think the FeeInsufficient is likely a red herring here.
But I do see this:

[ERR] HSWC: ChannelLink(560027dbab7f95d3c131c71ba03bf0b996d8c218f6bd5e7ad9240c9922635f91:0): failing link: unable to update commitment: unable to fetch aux leaves: unable to generate allocations: unable to assign alloc output commitments: no output commitment found for output index 4 with error: internal error

I wasn't able to reproduce this in an itest, likely it's dependent on exact balances at the moment of the attempt...
Do you think you could reproduce this while setting --lnd.debuglevel=trace,SRVR=debug,PEER=info,BTCN=warn? That would be very helpful. Or let me know more exactly what the balances looked like at the moment you try the 1 asset unit invoice.

[ryanthegentry]

tl;dr: the TEMPORARY_CHANNEL_FAILURE above was due to Alice generating a 100k asset invoice with <100k assets of inbound liquidity (100003 - 356 = 99647). I was able to repro that, and the fact that it makes a Taproot Assets channel inactive, however the Taproot Assets aspect of the channel is able to be recovered by disconnecting and reconnecting which allows it to receive asset payments for which it has inbound liquidity. The sats aspect of the channel appears to be irrevocably borked, however.

Before making the 1 sat payment that is expected to fail, here is Carol's listchannels output (showing the sats-only channel between her and Bob):

litd@carol:/$ lncli listchannels
{
    "channels": [
        {
            "active": true,
            "remote_pubkey": "0270c4c4a97401ed7ff27fa36a35d8efc5018ea22ed2e46b9219783302e5802b45",
            "channel_point": "3ee3f0c6d4a633b868134f1888ab2c69d53f1e887173226eeaf4475d4123a531:0",
            "chan_id": "134140418654208",
            "capacity": "1000000",
            "local_balance": "184643",
            "remote_balance": "811887",
            "commit_fee": "2810",
            "commit_weight": "1116",
            "fee_per_kw": "2500",
            "unsettled_balance": "0",
            "total_satoshis_sent": "357",
            "total_satoshis_received": "185000",
            "num_updates": "6",
            "pending_htlcs": [],
            "csv_delay": 144,
            "private": false,
            "initiator": false,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "10000",
            "remote_chan_reserve_sat": "10000",
            "static_remote_key": false,
            "commitment_type": "ANCHORS",
            "lifetime": "278",
            "uptime": "278",
            "close_address": "",
            "push_amount_sat": "0",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "10000",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "990000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "remote_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "10000",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "990000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "alias_scids": [
                "17592186044416000000"
            ],
            "zero_conf": false,
            "zero_conf_confirmed_scid": "0",
            "peer_alias": "bob",
            "peer_scid_alias": "17592186044416000000",
            "memo": "",
            "custom_channel_data": ""
        }
    ]
}

And here is Alice's listchannels output (showing the Taproot Assets channel between her and Bob):

litd@alice:/$ lncli listchannels
{
    "channels": [
        {
            "active": true,
            "remote_pubkey": "0270c4c4a97401ed7ff27fa36a35d8efc5018ea22ed2e46b9219783302e5802b45",
            "channel_point": "805c88aa7e24f979d58f341adb1106596cfc79c6f87c5ae5777f98a339ed4f3b:0",
            "chan_id": "140737488420864",
            "capacity": "100000",
            "local_balance": "6918",
            "remote_balance": "90001",
            "commit_fee": "2421",
            "commit_weight": "958",
            "fee_per_kw": "2500",
            "unsettled_balance": "0",
            "total_satoshis_sent": "85355",
            "total_satoshis_received": "354",
            "num_updates": "6",
            "pending_htlcs": [],
            "csv_delay": 144,
            "private": true,
            "initiator": true,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "1000",
            "remote_chan_reserve_sat": "1062",
            "static_remote_key": false,
            "commitment_type": "SIMPLE_TAPROOT_OVERLAY",
            "lifetime": "327",
            "uptime": "327",
            "close_address": "",
            "push_amount_sat": "5000",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "1000",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "99000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 83
            },
            "remote_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "1062",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "99000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 83
            },
            "alias_scids": [
                "17592186044416000000",
                "17779622795082145954"
            ],
            "zero_conf": false,
            "zero_conf_confirmed_scid": "0",
            "peer_alias": "bob",
            "peer_scid_alias": "17592186044416000001",
            "memo": "",
            "custom_channel_data": {
                "assets": [
                    {
                        "asset_utxo": {
                            "version": 1,
                            "asset_genesis": {
                                "genesis_point": "779b7b04f355a07220b16e26e66411ea12b0f5e070bed60170dab68ece64e792:1",
                                "name": "USDTest",
                                "meta_hash": "98c6ac28bc381a4bb270d94896129f355950eb8248a59c54ea933b68d1c7d202",
                                "asset_id": "bcb89bc2e34f26f30cf6f3a78c5336ed73edda4ab9fc00a0c9cf59a3f253ee6b"
                            },
                            "amount": 1000000000000,
                            "script_key": "0250aaeb166f4234650d84a2d8a130987aeaf6950206e0905401ee74ff3f8d18e6"
                        },
                        "capacity": 1000000000000,
                        "local_balance": 999999900355,
                        "remote_balance": 99645
                    }
                ]
            }
        }
    ]
}

I had Alice generate an invoice for 1 asset:

litd@alice:/$ litcli ln addinvoice --asset_amount 1 --asset_id bcb89bc2e34f26f30cf6f3a78c5336ed73edda4ab9fc00a0c9cf59a3f253ee6b
{
    "accepted_buy_quote": {
        "peer": "0270c4c4a97401ed7ff27fa36a35d8efc5018ea22ed2e46b9219783302e5802b45",
        "id": "98e496591e1a657d5d8cdac9c0187cec2c06397e230f41fbf7ee8b346147480e",
        "scid": "17865369828911958030",
        "asset_amount": "1",
        "ask_asset_rate": {
            "coefficient": "100000000",
            "scale": 0
        },
        "expiry": "1733412254"
    },
    "invoice_result": {
        "r_hash": "d7ddce6629e119c680939ab266eb7add7f6b4d08a05d62788ba0c1560da18d60",
        "payment_request": "lnbcrt10n1pn4rduwpp56lwuue3fuyvudqynn2exd6m6m4lkkngg5pwky7yt5rq4vrdp34sqdqqcqzzsxqzfvrzjqfcvf39fwsq76llj073k5dwcalzsrr4z9mfwg6ujr9urxqh9sq45talw3v6xz36gpcqqqqlgqqqqqqgq2qsp54mwy3pzn9vmd6ur8gmagwyaufq7pa7kedw0578lsku8uucstuaks9qxpqysgqyjcqj696yjyzlu4e9rewf8zkadexcv2gy0vxjx6nggup97f5h7y9dxwc24swk8ar79y323ugaqfa62jzt9x9267jvt9nru284upgx3qqpz8fyl",
        "add_index": "2",
        "payment_addr": "aedc4884532b36dd706746fa8713bc483c1efad96b9f4f1ff0b70fce620be76d"
    }
}

And then Carol attempt to pay it with sats:

litd@carol:/$ lncli payinvoice lnbcrt10n1pn4rduwpp56lwuue3fuyvudqynn2exd6m6m4lkkngg5pwky7yt5rq4vrdp34sqdqqcqzzsxqzfvrzjqfcvf39fwsq76llj073k5dwcalzsrr4z9mfwg6ujr9urxqh9sq45talw3v6xz36gpcqqqqlgqqqqqqgq2qsp54mwy3pzn9vmd6ur8gmagwyaufq7pa7kedw0578lsku8uucstuaks9qxpqysgqyjcqj696yjyzlu4e9rewf8zkadexcv2gy0vxjx6nggup97f5h7y9dxwc24swk8ar79y323ugaqfa62jzt9x9267jvt9nru284upgx3qqpz8fyl
Payment hash: d7ddce6629e119c680939ab266eb7add7f6b4d08a05d62788ba0c1560da18d60
Description: 
Amount (in satoshis): 1
Fee limit (in satoshis): 1
Destination: 03571aa8fcdb4a3116fb63ab05d92b314911b3ca879fb19f192630dbb9587004af
Confirm payment (yes/no): yes
+----------------------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
| HTLC_STATE                 | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT        | ROUTE       |
+----------------------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
| FEE_INSUFFICIENT @ 1st hop |        0.135 |        0.959 | 1            | 1   |      296 | 134140418654208 | bob->03571a |
| FEE_INSUFFICIENT @ 1st hop |        1.026 |        1.891 | 1            | 1   |      296 | 134140418654208 | bob->03571a |
+----------------------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
Amount + fee:   0 + 0 sat
Payment hash:   d7ddce6629e119c680939ab266eb7add7f6b4d08a05d62788ba0c1560da18d60
Payment status: FAILED, reason: FAILURE_REASON_NO_ROUTE
[lncli] FAILED

After this failure, all channels are still active. But Carol immediately fails any attempted payment to Alice with NO_ROUTE:

litd@carol:/$ lncli payinvoice lnbcrt3560n1pn4rwq8pp5amulnyplg84f0sp7492qwywp3y2sm7uu2c3d6melx7zpz9xrn20qdqqcqzzsxqzfvrzjqfcvf39fwsq76llj073k5dwcalzsrr4z9mfwg6ujr9urxqh9sq45tadkjltchlnh5uqqqqlgqqqqqqgq2qsp5jj654we3p6nk84m7qzzn5dsepmn3j9r6pp4wtmjla3et8qwwykmq9qxpqysgq87e8ncthhrwj7g3tyt3yc8se3swsx0gzrys2jfdk0m6h7gnc5awyzu85v66klzmcs9e59vngwcetxelp0298fw6jdgscmhczw4ethrcqxygz80
Payment hash: eef9f9903f41ea97c03ea9540711c189150dfb9c5622dd6f3f37841114c39a9e
Description: 
Amount (in satoshis): 356
Fee limit (in satoshis): 356
Destination: 03571aa8fcdb4a3116fb63ab05d92b314911b3ca879fb19f192630dbb9587004af
Confirm payment (yes/no): yes
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT | ROUTE |
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
Amount + fee:   0 + 0 sat
Payment hash:   eef9f9903f41ea97c03ea9540711c189150dfb9c5622dd6f3f37841114c39a9e
Payment status: FAILED, reason: FAILURE_REASON_NO_ROUTE
[lncli] FAILED

All channels are still listed as active at this point. I started with a small invoice this time, and after a mission control reset Carol succeeds in paying a Taproot Assets invoice generated by Alice, but immediately fails any attempt to pay a sats invoice generated by Alice:

litd@carol:/$ lncli payinvoice lnbcrt3560n1pn4rwq8pp5amulnyplg84f0sp7492qwywp3y2sm7uu2c3d6melx7zpz9xrn20qdqqcqzzsxqzfvrzjqfcvf39fwsq76llj073k5dwcalzsrr4z9mfwg6ujr9urxqh9sq45tadkjltchlnh5uqqqqlgqqqqqqgq2qsp5jj654we3p6nk84m7qzzn5dsepmn3j9r6pp4wtmjla3et8qwwykmq9qxpqysgq87e8ncthhrwj7g3tyt3yc8se3swsx0gzrys2jfdk0m6h7gnc5awyzu85v66klzmcs9e59vngwcetxelp0298fw6jdgscmhczw4ethrcqxygz80
Payment hash: eef9f9903f41ea97c03ea9540711c189150dfb9c5622dd6f3f37841114c39a9e
Description: 
Amount (in satoshis): 356
Fee limit (in satoshis): 356
Destination: 03571aa8fcdb4a3116fb63ab05d92b314911b3ca879fb19f192630dbb9587004af
Confirm payment (yes/no): yes
+------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT        | ROUTE       |
+------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
| SUCCEEDED  |        0.109 |        3.059 | 356          | 1   |      296 | 134140418654208 | bob->03571a |
+------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
Amount + fee:   356 + 1 sat
Payment hash:   eef9f9903f41ea97c03ea9540711c189150dfb9c5622dd6f3f37841114c39a9e
Payment status: SUCCEEDED, preimage: b1b31bc9f20e0136ca6029fc968c0a09d5aa5d415caafa71f62b52fd08b3b3aa
litd@carol:/$ lncli payinvoice lnbcrt5u1pn4rwpspp5n95mfc48jv70upphpjvn5q00h0hjr4dhkay94llah75s727cr2rqdqqcqzzsxqyz5vqsp502jzyqz7mkdhyqr58z7gm9t5mnnhsm4xnxuvg4fq8ak7k0hm9j7s9qxpqysgqrhf9k2zcpnc34dff735eqf2ugcpsu3vrr8wz5dl5tjp50w2rdtezs6ptl0qj3fg2szkv6tmy3l4gysqufattkwn8yxfvgkt2ssghzggquu079g
Payment hash: 9969b4e2a7933cfe04370c993a01efbbef21d5b7b7485afffdbfa90f2bd81a86
Description: 
Amount (in satoshis): 500
Fee limit (in satoshis): 500
Destination: 03571aa8fcdb4a3116fb63ab05d92b314911b3ca879fb19f192630dbb9587004af
Confirm payment (yes/no): yes
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT | ROUTE |
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
Amount + fee:   0 + 0 sat
Payment hash:   9969b4e2a7933cfe04370c993a01efbbef21d5b7b7485afffdbfa90f2bd81a86
Payment status: FAILED, reason: FAILURE_REASON_NO_ROUTE
[lncli] FAILED
litd@carol:/$ lncli resetmc
litd@carol:/$ lncli payinvoice lnbcrt5u1pn4rwpspp5n95mfc48jv70upphpjvn5q00h0hjr4dhkay94llah75s727cr2rqdqqcqzzsxqyz5vqsp502jzyqz7mkdhyqr58z7gm9t5mnnhsm4xnxuvg4fq8ak7k0hm9j7s9qxpqysgqrhf9k2zcpnc34dff735eqf2ugcpsu3vrr8wz5dl5tjp50w2rdtezs6ptl0qj3fg2szkv6tmy3l4gysqufattkwn8yxfvgkt2ssghzggquu079g
Payment hash: 9969b4e2a7933cfe04370c993a01efbbef21d5b7b7485afffdbfa90f2bd81a86
Description: 
Amount (in satoshis): 500
Fee limit (in satoshis): 500
Destination: 03571aa8fcdb4a3116fb63ab05d92b314911b3ca879fb19f192630dbb9587004af
Confirm payment (yes/no): yes
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT | ROUTE |
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
Amount + fee:   0 + 0 sat
Payment hash:   9969b4e2a7933cfe04370c993a01efbbef21d5b7b7485afffdbfa90f2bd81a86
Payment status: FAILED, reason: FAILURE_REASON_NO_ROUTE
[lncli] FAILED

Alice's Taproot Assets channel at this point has <100000 assets of inbound liquidity. So I'll generate an invoice for 100000 assets like I did in the OP:

litd@alice:/$ lncli listchannels
{
    "channels": [
        {
            "active": true,
            "remote_pubkey": "0270c4c4a97401ed7ff27fa36a35d8efc5018ea22ed2e46b9219783302e5802b45",
            "channel_point": "805c88aa7e24f979d58f341adb1106596cfc79c6f87c5ae5777f98a339ed4f3b:0",
            "chan_id": "140737488420864",
            "capacity": "100000",
            "local_balance": "7626",
            "remote_balance": "89293",
            "commit_fee": "2421",
            "commit_weight": "958",
            "fee_per_kw": "2500",
            "unsettled_balance": "0",
            "total_satoshis_sent": "85355",
            "total_satoshis_received": "708",
            "num_updates": "8",
            "pending_htlcs": [],
            "csv_delay": 144,
            "private": true,
            "initiator": true,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "1000",
            "remote_chan_reserve_sat": "1062",
            "static_remote_key": false,
            "commitment_type": "SIMPLE_TAPROOT_OVERLAY",
            "lifetime": "953",
            "uptime": "953",
            "close_address": "",
            "push_amount_sat": "5000",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "1000",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "99000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 83
            },
            "remote_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "1062",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "99000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 83
            },
            "alias_scids": [
                "17592186044416000000",
                "17779622795082145954",
                "17865369828911958030",
                "17705505937076156327",
                "17663606536828231635"
            ],
            "zero_conf": false,
            "zero_conf_confirmed_scid": "0",
            "peer_alias": "bob",
            "peer_scid_alias": "17592186044416000001",
            "memo": "",
            "custom_channel_data": {
                "assets": [
                    {
                        "asset_utxo": {
                            "version": 1,
                            "asset_genesis": {
                                "genesis_point": "779b7b04f355a07220b16e26e66411ea12b0f5e070bed60170dab68ece64e792:1",
                                "name": "USDTest",
                                "meta_hash": "98c6ac28bc381a4bb270d94896129f355950eb8248a59c54ea933b68d1c7d202",
                                "asset_id": "bcb89bc2e34f26f30cf6f3a78c5336ed73edda4ab9fc00a0c9cf59a3f253ee6b"
                            },
                            "amount": 1000000000000,
                            "script_key": "0250aaeb166f4234650d84a2d8a130987aeaf6950206e0905401ee74ff3f8d18e6"
                        },
                        "capacity": 1000000000000,
                        "local_balance": 999999901067,
                        "remote_balance": 98933
                    }
                ]
            }
        }
    ]
}
litd@alice:/$ litcli ln addinvoice --asset_amount 100000 --asset_id bcb89bc2e34f26f30cf6f3a78c5336ed73edda4ab9fc00a0c9cf59a3f253ee6
b
{
    "accepted_buy_quote": {
        "peer": "0270c4c4a97401ed7ff27fa36a35d8efc5018ea22ed2e46b9219783302e5802b45",
        "id": "481a7664cd37d31360eef6685f27d8e7324bbb8dff2c07dcf78c727e610a2811",
        "scid": "17837758111227193361",
        "asset_amount": "100000",
        "ask_asset_rate": {
            "coefficient": "100000000",
            "scale": 0
        },
        "expiry": "1733413053"
    },
    "invoice_result": {
        "r_hash": "029ca556ab798e95f6323565e315912e0f5a597053231c695040f404d995d760",
        "payment_request": "lnbcrt1m1pn4rw4wpp5q2w2244t0x8fta3jx4j7x9v39c845kts2v33c62sgr6qfkv46asqdqqcqzzsxqzfvrzjqfcvf39fwsq76llj073k5dwcalzsrr4z9mfwg6ujr9urxqh9sq45tauvwflxzz3gzyqqqqlgqqqqqqgq2qsp5tlc2fsfnlnfpj8r9u9d0p03q4fg9dflfcpg9fxsm5r0pcf4u49aq9qxpqysgqmwp8naky6jq4ju0q4s4pa2npzp8xwll503sv9rc6a9vvhsp0knthhernu2799ftzs0pe04xq03mlwgsdv0vzagtrw7qmf5vwj4tdkgcp5n9rky",
        "add_index": "8",
        "payment_addr": "5ff0a4c133fcd2191c65e15af0be20aa5056a7e9c050549a1ba0de1c26bca97a"
    }
}

Carol attempting to pay that invoice (again after a lncli resetmc) reproduces the TEMPORARY_CHANNEL_FAILURE error, though that is expected as there is not enough asset liquidity for Bob to forward the payment to Alice:

litd@carol:/$ lncli resetmc
litd@carol:/$ lncli payinvoice lnbcrt1m1pn4rw4wpp5q2w2244t0x8fta3jx4j7x9v39c845kts2v33c62sgr6qfkv46asqdqqcqzzsxqzfvrzjqfcvf39fwsq76llj073k5dwcalzsrr4z9mfwg6ujr9urxqh9sq45tauvwflxzz3gzyqqqqlgqqqqqqgq2qsp5tlc2fsfnlnfpj8r9u9d0p03q4fg9dflfcpg9fxsm5r0pcf4u49aq9qxpqysgqmwp8naky6jq4ju0q4s4pa2npzp8xwll503sv9rc6a9vvhsp0knthhernu2799ftzs0pe04xq03mlwgsdv0vzagtrw7qmf5vwj4tdkgcp5n9rky
Payment hash: 029ca556ab798e95f6323565e315912e0f5a597053231c695040f404d995d760
Description: 
Amount (in satoshis): 100000
Fee limit (in satoshis): 5000
Destination: 03571aa8fcdb4a3116fb63ab05d92b314911b3ca879fb19f192630dbb9587004af
Confirm payment (yes/no): yes
+-------------------------------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT        | ROUTE       |
+-------------------------------------+--------------+--------------+--------------+-----+----------+-----------------+------------
+-------------------------------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT        | ROUTE       |
+-------------------------------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
| TEMPORARY_CHANNEL_FAILURE @ 1st hop |        0.099 |       60.956 | 100000       | 1.1 |      296 | 134140418654208 | bob->03571a |
+-------------------------------------+--------------+--------------+--------------+-----+----------+-----------------+-------------+
Amount + fee:   0 + 0 sat
Payment hash:   029ca556ab798e95f6323565e315912e0f5a597053231c695040f404d995d760
Payment status: FAILED, reason: FAILURE_REASON_TIMEOUT
[lncli] FAILED

And again that error causes Alice and Bob's channel to go inactive, preventing Alice from generating more Taproot Assets invoices, but not more sats invoices:

litd@alice:/$ lncli listchannels
{
    "channels": [
        {
            "active": false,
            "remote_pubkey": "0270c4c4a97401ed7ff27fa36a35d8efc5018ea22ed2e46b9219783302e5802b45",
            "channel_point": "805c88aa7e24f979d58f341adb1106596cfc79c6f87c5ae5777f98a339ed4f3b:0",
            "chan_id": "140737488420864",
            "capacity": "100000",
            "local_balance": "7626",
            "remote_balance": "89293",
            "commit_fee": "2421",
            "commit_weight": "958",
            "fee_per_kw": "2500",
            "unsettled_balance": "0",
            "total_satoshis_sent": "85355",
            "total_satoshis_received": "1062",
            "num_updates": "10",
            "pending_htlcs": [],
            "csv_delay": 144,
            "private": true,
            "initiator": true,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "1000",
            "remote_chan_reserve_sat": "1062",
            "static_remote_key": false,
            "commitment_type": "SIMPLE_TAPROOT_OVERLAY",
            "lifetime": "1250",
            "uptime": "1250",
            "close_address": "",
            "push_amount_sat": "5000",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "1000",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "99000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 83
            },
            "remote_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "1062",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "99000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 83
            },
            "alias_scids": [
                "17592186044416000000",
                "17779622795082145954",
                "17865369828911958030",
                "17705505937076156327",
                "17663606536828231635",
                "17699894334587879725",
                "17837758111227193361"
            ],
            "zero_conf": false,
            "zero_conf_confirmed_scid": "0",
            "peer_alias": "bob",
            "peer_scid_alias": "17592186044416000001",
            "memo": "",
            "custom_channel_data": {
                "assets": [
                    {
                        "asset_utxo": {
                            "version": 1,
                            "asset_genesis": {
                                "genesis_point": "779b7b04f355a07220b16e26e66411ea12b0f5e070bed60170dab68ece64e792:1",
                                "name": "USDTest",
                                "meta_hash": "98c6ac28bc381a4bb270d94896129f355950eb8248a59c54ea933b68d1c7d202",
                                "asset_id": "bcb89bc2e34f26f30cf6f3a78c5336ed73edda4ab9fc00a0c9cf59a3f253ee6b"
                            },
                            "amount": 1000000000000,
                            "script_key": "0250aaeb166f4234650d84a2d8a130987aeaf6950206e0905401ee74ff3f8d18e6"
                        },
                        "capacity": 1000000000000,
                        "local_balance": 999999901067,
                        "remote_balance": 98933
                    }
                ]
            }
        }
    ]
}
litd@alice:/$ litcli ln addinvoice --asset_amount 1000 --asset_id bcb89bc2e34f26f30cf6f3a78c5336ed73edda4ab9fc00a0c9cf59a3f253ee6b
[litcli] error adding invoice: rpc error: code = Unknown desc = error finding asset channel to use: no asset channel balance found for asset bcb89bc2e34f26f30cf6f3a78c5336ed73edda4ab9fc00a0c9cf59a3f253ee6b
litd@alice:/$ lncli addinvoice 1000
{
    "r_hash": "eb45fa203e471c9b518fd29a347698fa624f3d4c07d80c5c0d7d2d0ce7b148a8",
    "payment_request": "lnbcrt10u1pn4r09lpp5adzl5gp7guwfk5v062drga5clf3y702vqlvqchqd05kseea3fz5qdqqcqzzsxqyz5vqsp5ay2fgg6em73u9kcakdywf0cy0xh9wderkt2sjlutnsgplmg877ds9qxpqysgqdyqfs2d4peqxsut3e4kkxrx9jrjt9kdtsvmrkvd75fxsstkrjwcjmqg239n7a67cark3ky7s3r56s6zpr4wsg8r84u338dhe3jj305qqhu72wj",
    "add_index": "9",
    "payment_addr": "e914942359dfa3c2db1db348e4bf0479ae573723b2d5097f8b9c101fed07f79b"
}

Though Carol can't pay that sats invoice anyways:

litd@carol:/$ lncli resetmc
litd@carol:/$ lncli payinvoice lnbcrt10u1pn4r09lpp5adzl5gp7guwfk5v062drga5clf3y702vqlvqchqd05kseea3fz5qdqqcqzzsxqyz5vqsp5ay2fgg6em73u9kcakdywf0cy0xh9wderkt2sjlutnsgplmg877ds9qxpqysgqdyqfs2d4peqxsut3e4kkxrx9jrjt9kdtsvmrkvd75fxsstkrjwcjmqg239n7a67cark3ky7s3r56s6zpr4wsg8r84u338dhe3jj305qqhu72wj
Payment hash: eb45fa203e471c9b518fd29a347698fa624f3d4c07d80c5c0d7d2d0ce7b148a8
Description: 
Amount (in satoshis): 1000
Fee limit (in satoshis): 1000
Destination: 03571aa8fcdb4a3116fb63ab05d92b314911b3ca879fb19f192630dbb9587004af
Confirm payment (yes/no): yes
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT | ROUTE |
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
Amount + fee:   0 + 0 sat
Payment hash:   eb45fa203e471c9b518fd29a347698fa624f3d4c07d80c5c0d7d2d0ce7b148a8
Payment status: FAILED, reason: FAILURE_REASON_NO_ROUTE
[lncli] FAILED

Disconnecting and reconnecting Alice and Bob brings the channel back to active. Carol still cannot pay sats invoices, but can pay a Taproot Assets invoice of 1k assets (which Alice has plenty of inbound liquidity for):

litd@alice:/$ litcli ln addinvoice --asset_amount 1000 --asset_id bcb89bc2e34f26f30cf6f3a78c5336ed73edda4ab9fc00a0c9cf59a3f253ee6b
{
    "accepted_buy_quote": {
        "peer": "0270c4c4a97401ed7ff27fa36a35d8efc5018ea22ed2e46b9219783302e5802b45",
        "id": "b0b0fa16be3c23c5e5b7e69d3fd5ed59e56542d939f124d9f64d185ca545c5ee",
        "scid": "17747868492725863918",
        "asset_amount": "1000",
        "ask_asset_rate": {
            "coefficient": "100000000",
            "scale": 0
        },
        "expiry": "1733413756"
    },
    "invoice_result": {
        "r_hash": "660bd4a9a7384962715859238fddb8913f3a7cc51de48a1b6036023fb9fd139f",
        "payment_request": "lnbcrt10u1pn4r0tvpp5vc9af2d88pykyu2cty3clhdcjyln5lx9rhjg5xmqxcprlw0azw0sdqqcqzzsxqzfvrzjqfcvf39fwsq76llj073k5dwcalzsrr4z9mfwg6ujr9urxqh9sq45tajdrpw223w9acqqqqlgqqqqqqgq2qsp55dwdjkku2rk7mu5txgwp5mmmauaa9qvqfshhvvd8f7ak4va47ads9qxpqysgqcrkduqxs4k6e0eqgpppda00vdaj0r4898s5yufzckwh48m5ytd3xat4ad6pk2gzvd3ceaww7nhxda49kn8cu3calzg5c0hnwqu7k56cp0vgyfs",
        "add_index": "10",
        "payment_addr": "a35cd95adc50ededf28b321c1a6f7bef3bd281804c2f7631a74fbb6ab3b5f75b"
    }
}
litd@carol:/$ lncli payinvoice lnbcrt10u1pn4r09lpp5adzl5gp7guwfk5v062drga5clf3y702vqlvqchqd05kseea3fz5qdqqcqzzsxqyz5vqsp5ay2fgg6em73u9kcakdywf0cy0xh9wderkt2sjlutnsgplmg877ds9qxpqysgqdyqfs2d4peqxsut3e4kkxrx9jrjt9kdtsvmrkvd75fxsstkrjwcjmqg239n7a67cark3ky7s3r56s6zpr4wsg8r84u338dhe3jj305qqhu72wj
Payment hash: eb45fa203e471c9b518fd29a347698fa624f3d4c07d80c5c0d7d2d0ce7b148a8
Description: 
Amount (in satoshis): 1000
Fee limit (in satoshis): 1000
Destination: 03571aa8fcdb4a3116fb63ab05d92b314911b3ca879fb19f192630dbb9587004af
Confirm payment (yes/no): yes
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT | ROUTE |
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
Amount + fee:   0 + 0 sat
Payment hash:   eb45fa203e471c9b518fd29a347698fa624f3d4c07d80c5c0d7d2d0ce7b148a8
Payment status: FAILED, reason: FAILURE_REASON_NO_ROUTE
[lncli] FAILED
litd@carol:/$ lncli payinvoice lnbcrt10u1pn4r0tvpp5vc9af2d88pykyu2cty3clhdcjyln5lx9rhjg5xmqxcprlw0azw0sdqqcqzzsxqzfvrzjqfcvf39fwsq76llj073k5dwcalzsrr4z9mfwg6ujr9urxqh9sq45tajdrpw223w9acqqqqlgqqqqqqgq2qsp55dwdjkku2rk7mu5txgwp5mmmauaa9qvqfshhvvd8f7ak4va47ads9qxpqysgqcrkduqxs4k6e0eqgpppda00vdaj0r4898s5yufzckwh48m5ytd3xat4ad6pk2gzvd3ceaww7nhxda49kn8cu3calzg5c0hnwqu7k56cp0vgyfs
Payment hash: 660bd4a9a7384962715859238fddb8913f3a7cc51de48a1b6036023fb9fd139f
Description: 
Amount (in satoshis): 1000
Fee limit (in satoshis): 1000
Destination: 03571aa8fcdb4a3116fb63ab05d92b314911b3ca879fb19f192630dbb9587004af
Confirm payment (yes/no): yes
+------------+--------------+--------------+--------------+-------+----------+-----------------+-------------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE   | TIMELOCK | CHAN_OUT        | ROUTE       |
+------------+--------------+--------------+--------------+-------+----------+-----------------+-------------+
| SUCCEEDED  |        0.117 |        2.922 | 1000         | 1.001 |      296 | 134140418654208 | bob->03571a |
+------------+--------------+--------------+--------------+-------+----------+-----------------+-------------+
Amount + fee:   1000 + 1.001 sat
Payment hash:   660bd4a9a7384962715859238fddb8913f3a7cc51de48a1b6036023fb9fd139f
Payment status: SUCCEEDED, preimage: bf439ce7c89df3cd3a4d9e34527aefbb74c05e2e9f5f09e17f6a3c2733ad3a9e

Logs attached with the proper settings:

alice-1240repro.log
bob-1240repro.log
carol-1240repro.log

[guggero]

Looks like we're running into the same error as is being fixed by the forward bandwidth manager PR (#1236 and lightninglabs/lightning-terminal#912):
It's an asset amount underflow because of the missing bandwidth check:

2024-12-05 14:37:45.749 [TRC] TCHN: Computing view, whoseCommit=Remote, ourAssetBalance=98933, theirAssetBalance=999999901067, ourUpdates=1, theirUpdates=0
2024-12-05 14:37:45.750 [TRC] TCHN: Computed view, whoseCommit=Remote, ourAssetBalance=18446744073709550549, theirAssetBalance=999999901067, dustLimit=0.00000354 BTC
2024-12-05 14:37:45.751 [TRC] TCHN: Sanity checking amounts, whoseCommit=Remote, ourBalance=88939, theirBalance=7196, ourAssetBalance=18446744073709550549, theirAssetBalance=999999901067

...
2024-12-05 14:37:45.886 [ERR] HSWC: ChannelLink(805c88aa7e24f979d58f341adb1106596cfc79c6f87c5ae5777f98a339ed4f3b:0): failing link: unable to update commitment: unable to fetch aux leaves: unable to generate allocations: unable to assign alloc output commitments: no output commitment found for output index 4 with error: internal error

[ZZiigguurraatt]

I tried increasing fee_limit_sat to 712 sat (just a number much greater than 354), and that still results in the same issue.

[ZZiigguurraatt]

Note, I do not get this issue if I try to exchange "5 sat worth" of an asset directly with a peer. This kind of makes sense though because the peer doesn't need to route (and pay the 354 sat) for the next hop.

[ZZiigguurraatt]

I tried increasing fee_limit_sat to 712 sat (just a number much greater than 354), and that still results in the same issue.

I tried to increase to 2000 as well (just to make sure it is well over 354*2 to pay for each hop), and that still fails.

[ZZiigguurraatt]

Looks like we're running into the same error as is being fixed by the forward bandwidth manager PR (#1236 and lightninglabs/lightning-terminal#912): It's an asset amount underflow because of the missing bandwidth check:

2024-12-05 14:37:45.749 [TRC] TCHN: Computing view, whoseCommit=Remote, ourAssetBalance=98933, theirAssetBalance=999999901067, ourUpdates=1, theirUpdates=0
2024-12-05 14:37:45.750 [TRC] TCHN: Computed view, whoseCommit=Remote, ourAssetBalance=18446744073709550549, theirAssetBalance=999999901067, dustLimit=0.00000354 BTC
2024-12-05 14:37:45.751 [TRC] TCHN: Sanity checking amounts, whoseCommit=Remote, ourBalance=88939, theirBalance=7196, ourAssetBalance=18446744073709550549, theirAssetBalance=999999901067

...
2024-12-05 14:37:45.886 [ERR] HSWC: ChannelLink(805c88aa7e24f979d58f341adb1106596cfc79c6f87c5ae5777f98a339ed4f3b:0): failing link: unable to update commitment: unable to fetch aux leaves: unable to generate allocations: unable to assign alloc output commitments: no output commitment found for output index 4 with error: internal error

Just tested with the PR's you mentioned merged and I am still getting the same issues.

[guggero]

Fixed by #1236.