Asset Channel Closure on Self-Payment (from `tapchannelrpc`) b/t `v0.14.1-alpha` and `v0.15.0-alpha-rc1`

[Nsandomeno]

Background

1. Mint grouped asset on node A
2. Sync node B to node A, add to federation (public universe configs enabled)

  taproot-assets.universe.public-access=rw
  taproot-assets.allow-public-uni-proof-courier=1
  taproot-assets.allow-public-stats=1

3. Transfer assets from node A to node B
4. Open asset channel from node B to node A (100% of asset capacity on local side of node B)
5. Open regular channel from node A to node B (50% of sats balance on each side)
6. Generate sats invoice (with lncli) on node B for 2000 sats

  invoice, err := lnClient.AddInvoice(context.TODO(), &lnrpc.Invoice{
  	Value: int64(2000),
  })

7. Make self-payment with route-hints (hop-hints fully populated) through tapchannelrpc

	payReq := &routerrpc.SendPaymentRequest{
		PaymentRequest:   invoice.PaymentRequest,
		OutgoingChanIds:  []uint64{swapAssetChanId},
		AllowSelfPayment: true,
		FeeLimitSat:      20000,
		RouteHints: []*lnrpc.RouteHint{{
			HopHints: []*lnrpc.HopHint{
				{
					NodeId:                    pubkey,
					ChanId:                    assetsPeerScidAlias,
					CltvExpiryDelta:           80,
					FeeBaseMsat:               1000,
					FeeProportionalMillionths: 1,
				},
				{
					NodeId:                    peerPubkey,
					ChanId:                    satsPeerScidAlias,
					CltvExpiryDelta:           80,
					FeeBaseMsat:               1000,
					FeeProportionalMillionths: 1,
				},
			}},
		},
	}
	stream, err := tapChannelClient.SendPayment(context.TODO(), &tapchannelrpc.SendPaymentRequest{
		AssetId:        assetHex,
		PeerPubkey:    peerPubkey,
		PaymentRequest: payReq,
		AllowOverpay:   true,
		RfqId:          rfqQuoteId,
	})
	if err != nil {
		fmt.Printf("error sending asset payment for swap: %s", err.Error())
		return
	}

8. Results in a force close of the channel

   2025-06-04 21:17:25.272 [DBG] DISC: The message ChannelUpdate has no AuthProof, sending the update to remote peer 03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435
2025-06-04 21:17:25.278 [DBG] DISC: Processed ChannelUpdate: peer=<nil>, short_chan_id=17592186044416000002, timestamp=2025-06-04 21:17:25 +0000 UTC
2025-06-04 21:17:25.278 [DBG] DISC: Successfully sent ChannelUpdate message for channel=16000000:0:2 with peer=03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435
2025-06-04 21:17:25.278 [INF] CNCT: ChannelArbitrator(39fcd986a8cb181984f66045ab9970780e4dee1680315720e865d22e5c060fbc:0): received force close request
2025-06-04 21:17:25.284 [DBG] CNCT: ChannelArbitrator(39fcd986a8cb181984f66045ab9970780e4dee1680315720e865d22e5c060fbc:0): attempting state step with trigger=userTrigger from state=StateDefault
2025-06-04 21:17:25.278 [DBG] PEER: Peer(03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435): Sending ChannelUpdate(chain_hash=0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206, short_chan_id=17592186044416000002, mflags=00000001, cflags=00000010, update_time=2025-06-04 21:17:25 +0000 UTC) to 03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435@172.31.0.1:60902
2025-06-04 21:17:25.287 [DBG] CNCT: ChannelArbitrator(39fcd986a8cb181984f66045ab9970780e4dee1680315720e865d22e5c060fbc:0): new block (height=364) examining active HTLC's
2025-06-04 21:17:25.294 [DBG] CNCT: ChannelArbitrator(39fcd986a8cb181984f66045ab9970780e4dee1680315720e865d22e5c060fbc:0): checking commit chain actions at height=364, in_htlc_count=0, out_htlc_count=0
2025-06-04 21:17:25.319 [DBG] CNCT: ChannelArbitrator(39fcd986a8cb181984f66045ab9970780e4dee1680315720e865d22e5c060fbc:0): attempting state step with trigger=userTrigger from state=StateBroadcastCommit
2025-06-04 21:17:25.319 [INF] CNCT: ChannelArbitrator(39fcd986a8cb181984f66045ab9970780e4dee1680315720e865d22e5c060fbc:0): force closing chan
2025-06-04 21:17:25.337 [DBG] TSVR: FetchLeavesFromCommit called, ourBalance=0 mSAT, theirBalance=96920000 mSAT, numHtlcs=0
2025-06-04 21:17:25.341 [DBG] TSVR: FetchLeavesFromCommit called, ourBalance=0 mSAT, theirBalance=96920000 mSAT, numHtlcs=0
2025-06-04 21:17:25.342 [DBG] LNWL: ChannelPoint(39fcd986a8cb181984f66045ab9970780e4dee1680315720e865d22e5c060fbc:0): Restoring 0 dangling remote updates
2025-06-04 21:17:25.345 [DBG] LNWL: ChannelPoint(39fcd986a8cb181984f66045ab9970780e4dee1680315720e865d22e5c060fbc:0): Restoring 0 local updates that the peer should sign
2025-06-04 21:17:25.351 [INF] LNWL: Generating new musig2 sig for session=daf310c3e35f7128930b38d5156ec27e5ff1ad7caff7a8ebb026459ecf5800ca, nonces=NoncePair(verification_nonce=02a5f11cdd0d20f526a38a8df624a96d119d8cc7de30c23613222ae679687889cc0349d24b3188967d565bee73386ab1face62743190ef16df20f9912cf9e0f05487, signing_nonce=0316d8676c542a811b4f899633c576aca62d1bdc55e1ea5f22c5867c60ab0d3c8103cc80ac2609d98b46e6df6976e5b016bd0893a770bb2afe77c297e0d7922d18b1)
2025-06-04 21:17:25.376 [INF] CNCT: Broadcasting force close transaction 74b7961c0461e3f4df5accbafc99ecad8ba6e99543c451d11c1cb8f270e929d9, ChannelPoint(39fcd986a8cb181984f66045ab9970780e4dee1680315720e865d22e5c060fbc:0): (*wire.MsgTx)(0xc001334ec0)({
 Version: (int32) 2,
 TxIn: ([]*wire.TxIn) (len=1 cap=1) {
  (*wire.TxIn)(0xc003c1fb00)({
   PreviousOutPoint: (wire.OutPoint) 39fcd986a8cb181984f66045ab9970780e4dee1680315720e865d22e5c060fbc:0,
   SignatureScript: ([]uint8) <nil>,
   Witness: (wire.TxWitness) (len=1 cap=1) {
    ([]uint8) (len=64 cap=64) {
     00000000  bc a9 6c e8 60 17 a0 d9  2b 1f 1f 26 e8 6a bd 08  |..l.`...+..&.j..|
     00000010  70 f7 9a 4d fc 88 a2 09  8b 92 73 36 d4 a3 13 dd  |p..M......s6....|
     00000020  0b 54 ef bd aa 77 54 53  f9 1d da ec 33 a4 27 20  |.T...wTS....3.' |
     00000030  83 61 e9 9d d8 61 9f 55  3f 39 86 cc e3 67 ab 69  |.a...a.U?9...g.i|
    }
   },
   Sequence: (uint32) 2164190000
  })

Your environment

Node A

litcli getinfo

litd@alice:/$ litcli getinfo
{
    "version": "0.14.1-alpha commit=v0.14.1-alpha"
}

uname -mrsv

Linux 5.15.49-linuxkit-pr #1 SMP Thu May 25 07:17:40 UTC 2023 x86_64

bitcoind --version

v0.29.0

Node B

lighting-terminal v0.15.0-alpha-rc1

uname -mrsv

Darwin 22.6.0 Darwin Kernel Version 22.6.0: Mon Apr 22 20:54:28 PDT 2024; root:xnu-8796.141.3.705.2~1/RELEASE_X86_64 x86_64

bitcoind --version

v0.29.0

Expected behavior

Things worked when both members of the channel were from lightning-terminal 0.15.0-alpha-rc1

Actual behavior

Force close of asset channel. While Node B acknowledges receipt of its assets from the on-chain send from Node A, it doesn't seem to gain a transfer proof. The "No AuthProof" line of the log output from bullet 8. makes me think some validation is failing.

[Roasbeef]

Why are you specifying the hop hints yourself vs letting tapd fully specify them? You can't assume you know what the correct values are, nor which quote is selected.

Do you have more logs before this instance?

Is the issue deterministic?

Given you seem to have Go fragments, are you able to write an integration test so we have a direct repro?

Does the issue still arise if you try to repro based on this branch: #1082 ?

[bitcoin-coder-bob]

Why are you specifying the hop hints yourself vs letting tapd fully specify them? You can't assume you know what the correct values are, nor which quote is selected.

hop hints being used stems from spirit behind this issue: lightninglabs/taproot-assets#1279 and using hop hints + OutgoingChanIds did work: lightninglabs/taproot-assets#1279 (comment). The call to tapchannelrpc.SendPaymentRequest that uses the payment request with these hop hints and OutgoingChanIds is using the RfqId field as we have pre-negotiated a quote ahead of time.

As for the logs this is something @Nsandomeno will have to produce.

On mainnet I experienced the same channel closure behavior with the same node versions (exception being taproot asssets 0.6.0-rc1 vs 0.6.0-rc2) described in this issue. Here is a slack thread on some details on my mainnet closure experience: https://lightningcommunity.slack.com/archives/C03B3556HQ8/p1748636689980939

[Nsandomeno]

Do you have more logs before this instance?

Here are some more logs above where the the original snippet from bullet 8. left off.

2025-06-04 21:15:12.847 [DBG] HLCK: Health check: chain backend invoking success callback
2025-06-04 21:15:30.535 [DBG] PEER: Peer(03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435): Sending Ping(ping_bytes=00000030505f88087b332af8c80a053c847ccb9c98e8ff93fea48c2343f16269efab59078bfd12ad50cecf6e6dddce6010555c2b04d628dea4d9b2b72f3dd9a914302aa139b74068ffff7f2000000000) to 03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435@172.31.0.1:60902
2025-06-04 21:15:30.542 [DBG] PEER: Peer(03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435): Received Pong(len(pong_bytes)=2211) from 03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435@172.31.0.1:60902
2025-06-04 21:15:43.016 [INF] GRPH: Processed channels=1 updates=2 nodes=1 in last 1m14.171495214s
2025-06-04 21:15:53.519 [INF] LITD: Handling gRPC request: /universerpc.Universe/AssetRoots
2025-06-04 21:15:53.520 [DBG] RPCS: [/universerpc.Universe/AssetRoots] requested
2025-06-04 21:15:53.524 [INF] TADB: populating root cache...
2025-06-04 21:15:53.528 [DBG] TADB: Populating 2 root nodes into cache, took=3.737155ms
2025-06-04 21:15:53.530 [DBG] TADB: caching num_roots=2
2025-06-04 21:15:53.556 [INF] LITD: Handling gRPC request: /universerpc.Universe/AssetLeafKeys
2025-06-04 21:15:53.561 [DBG] RPCS: [/universerpc.Universe/AssetLeafKeys] requested
2025-06-04 21:15:53.562 [DBG] UNIV: Retrieving all keys for Universe: id=issuance-7a14116f67a28e00f8aa37d34d99bfdc15e762c0a1fa527b58bdfd87a04f31fe (asset_id=0000000000000000000000000000000000000000000000000000000000000000, group_key=d425f85dce1eab326b5c6e37edf74e0a46fdb0c3237d824584d50d6648b0a47d, proof_type=issuance)
2025-06-04 21:15:53.566 [DBG] TADB: storing leaf keys for issuance-7a14116f67a28e00f8aa37d34d99bfdc15e762c0a1fa527b58bdfd87a04f31fe (asset_id=0000000000000000000000000000000000000000000000000000000000000000, group_key=d425f85dce1eab326b5c6e37edf74e0a46fdb0c3237d824584d50d6648b0a47d, proof_type=issuance) in cache
2025-06-04 21:15:53.573 [INF] LITD: Handling gRPC request: /universerpc.Universe/AssetLeafKeys
2025-06-04 21:15:53.574 [DBG] RPCS: [/universerpc.Universe/AssetLeafKeys] requested
2025-06-04 21:15:53.578 [DBG] UNIV: Retrieving all keys for Universe: id=issuance-7a14116f67a28e00f8aa37d34d99bfdc15e762c0a1fa527b58bdfd87a04f31fe (asset_id=0000000000000000000000000000000000000000000000000000000000000000, group_key=d425f85dce1eab326b5c6e37edf74e0a46fdb0c3237d824584d50d6648b0a47d, proof_type=issuance)
2025-06-04 21:15:53.582 [DBG] TADB: storing leaf keys for issuance-7a14116f67a28e00f8aa37d34d99bfdc15e762c0a1fa527b58bdfd87a04f31fe (asset_id=0000000000000000000000000000000000000000000000000000000000000000, group_key=d425f85dce1eab326b5c6e37edf74e0a46fdb0c3237d824584d50d6648b0a47d, proof_type=issuance) in cache
2025-06-04 21:15:53.596 [INF] LITD: Handling gRPC request: /universerpc.Universe/QueryProof
2025-06-04 21:15:53.597 [DBG] RPCS: [/universerpc.Universe/QueryProof] requested
2025-06-04 21:15:53.608 [DBG] TADB: storing proof for issuance-7a14116f67a28e00f8aa37d34d99bfdc15e762c0a1fa527b58bdfd87a04f31fe (asset_id=0000000000000000000000000000000000000000000000000000000000000000, group_key=d425f85dce1eab326b5c6e37edf74e0a46fdb0c3237d824584d50d6648b0a47d, proof_type=issuance)+{aa24df753e6da6cfed96b3370042c8890e3d5eb4a9129867f047b65bb20af193:0 0xc0037ba2f0} in cache, key=ec4b0a7c7c428543b401cf1f4d792c0849b3fbb1e97c8ed5de8c8ceb63f8bc1e
2025-06-04 21:16:12.810 [DBG] HLCK: Health check: chain backend invoking success callback
2025-06-04 21:16:30.494 [DBG] PEER: Peer(03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435): Sending Ping(ping_bytes=00000030505f88087b332af8c80a053c847ccb9c98e8ff93fea48c2343f16269efab59078bfd12ad50cecf6e6dddce6010555c2b04d628dea4d9b2b72f3dd9a914302aa139b74068ffff7f2000000000) to 03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435@172.31.0.1:60902
2025-06-04 21:16:30.499 [DBG] PEER: Peer(03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435): Received Pong(len(pong_bytes)=3162) from 03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435@172.31.0.1:60902
2025-06-04 21:17:02.699 [INF] LITD: Handling gRPC request: /universerpc.Universe/AssetRoots
2025-06-04 21:17:02.700 [DBG] RPCS: [/universerpc.Universe/AssetRoots] requested
2025-06-04 21:17:12.762 [DBG] HLCK: Health check: chain backend invoking success callback
2025-06-04 21:17:24.543 [DBG] PEER: Peer(03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435): Received custom(type=52884) from 03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435@172.31.0.1:60902
2025-06-04 21:17:24.543 [DBG] SRVR: Custom message received: peer=03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435, type=52884
2025-06-04 21:17:24.548 [DBG] RFQS: Stream handling incoming message: SellRequest(peer=03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435, id=95b81c2d5b42253155a54c1ebbbbc4b1e6d098122f66a0b3f6d6aa88b64f1fae, asset=AssetSpecifier(id=d9f30d51f769cfb31f6dfc48dc68bfcbe0c51e56a26797e763917fef7ecdd44b, group_pub_key=), payment_max_amt=5000000, asset_rate_hint=AssetRate(rate=58365469470000, expiry=2025-06-04T21:22:24Z))
2025-06-04 21:17:24.552 [DBG] RFQS: Manager handling incoming message: SellRequest(peer=03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435, id=95b81c2d5b42253155a54c1ebbbbc4b1e6d098122f66a0b3f6d6aa88b64f1fae, asset=AssetSpecifier(id=d9f30d51f769cfb31f6dfc48dc68bfcbe0c51e56a26797e763917fef7ecdd44b, group_pub_key=), payment_max_amt=5000000, asset_rate_hint=AssetRate(rate=58365469470000, expiry=2025-06-04T21:22:24Z))
2025-06-04 21:17:24.556 [DBG] RFQS: Querying price oracle for bid price (asset_specifier=AssetSpecifier(id=d9f30d51f769cfb31f6dfc48dc68bfcbe0c51e56a26797e763917fef7ecdd44b, group_pub_key=), asset_max_amt=None, payment_max_amt=Some(5000000 mSAT), asset_rate_hint=Some({58365469470000 2025-06-04 21:22:24 +0000 UTC}))
2025-06-04 21:17:24.564 [DBG] RFQS: Manager handling outgoing message: SellAccept(peer=03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435, id=95b81c2d5b42253155a54c1ebbbbc4b1e6d098122f66a0b3f6d6aa88b64f1fae, asset_rate=AssetRate(rate=60155130529999, expiry=2025-06-04T21:22:24Z), scid=17786591282497265582)
2025-06-04 21:17:24.568 [DBG] RFQS: Order handler is registering an asset buy policy given a sell accept message: SellAccept(peer=03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435, id=95b81c2d5b42253155a54c1ebbbbc4b1e6d098122f66a0b3f6d6aa88b64f1fae, asset_rate=AssetRate(rate=60155130529999, expiry=2025-06-04T21:22:24Z), scid=17786591282497265582)
2025-06-04 21:17:24.579 [DBG] RFQS: Stream handling outgoing message: SellAccept(peer=03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435, id=95b81c2d5b42253155a54c1ebbbbc4b1e6d098122f66a0b3f6d6aa88b64f1fae, asset_rate=AssetRate(rate=60155130529999, expiry=2025-06-04T21:22:24Z), scid=17786591282497265582)
2025-06-04 21:17:24.586 [DBG] RPCS: [/lnrpc.Lightning/SendCustomMessage] requested
2025-06-04 21:17:24.591 [DBG] PEER: Peer(03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435): Sending custom(type=52885) to 03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435@172.31.0.1:60902
2025-06-04 21:17:24.962 [DBG] PEER: Peer(03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435): Received UpdateAddHTLC(chan_id=bc0f065c2ed265e82057318016ee4d0e787099ab4560f6841918cba886d9fc39, id=0, amt=354000 mSAT, expiry=527, hash=76494f251ac8a40a433c0a83f53be33d7231fd1711351e9025023179f3b1a682, blinding_point=, custom_records=map[65536:[1 44 0 32 217 243 13 81 247 105 207 179 31 109 252 72 220 104 191 203 224 197 30 86 162 103 151 231 99 145 127 239 126 205 212 75 1 8 0 0 0 0 71 191 22 21] 65538:[149 184 28 45 91 66 37 49 85 165 76 30 187 187 196 177 230 208 152 18 47 102 160 179 246 214 170 136 182 79 31 174]]) from 03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435@172.31.0.1:60902
2025-06-04 21:17:25.139 [DBG] PEER: Peer(03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435): Received CommitSig(chan_id=bc0f065c2ed265e82057318016ee4d0e787099ab4560f6841918cba886d9fc39, num_htlcs=1) from 03a87aca1479286fea390d79ac9ba85842caa1713377d07daadb1b00ce1175d435@172.31.0.1:60902

Is the issue deterministic?

I've repeated the sequence of steps in this setup 4 times all resulting in force closed channels.

Given you seem to have Go fragments, are you able to write an integration test so we have a direct repro?

I can try to add one here later.

Does the issue still arise if you try to repro based on this branch: #1082 ?

I will try to get a confirmation on this later.

[guggero]

I just tried to reproduce this using our litd backward compatibility test and I was not successful.
It sounds like you can reproduce this consistently. Can you please run this with --lnd.debuglevel=trace,GRPC=error,PEER=info and send us the full log?

[guggero]

The AuthProof log is a red herring, that has nothing to do with asset payments. Also, it's logged as [DBG]. Do you see no [ERR] entries in either of the nodes' logs?

[Nsandomeno]

I just tried to reproduce this using our litd backward compatibility test and I was not successful. It sounds like you can reproduce this consistently. Can you please run this with --lnd.debuglevel=trace,GRPC=error,PEER=info and send us the full log?

Yes - This time when about to run another attempt I noticed Node B had debug logs indicating a failure around a proof transfer despite the channel opening otherwise looking successful. Node A and B both have the public universe configs mentioned earlier, have joined each other's federations, have the same price oracle configured, enabled inserts of issuance and transfer proofs, and node B has already received assets from node A (node B is the asset channel initiator):

        {
            "active": true,
            "remote_pubkey": "0256cc54cbf9d6aef2724ef0467737c6a2ae66075a098e12012d17b18c4e36ee41",
            "channel_point": "7b21c7e6f91dae8f61f493d84543418f1d866858b89159b8775229c5d89c159c:0",
            "chan_id": "217703302365184",
            "capacity": "100000",
            "local_balance": "0",
            "remote_balance": "96920",
            "commit_fee": "2750",
            "commit_weight": "614",
            "fee_per_kw": "2500",
            "unsettled_balance": "0",
            "total_satoshis_sent": "0",
            "total_satoshis_received": "0",
            "num_updates": "0",
            "pending_htlcs": [],
            "csv_delay": 144,
            "private": true,
            "initiator": false,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "1062",
            "remote_chan_reserve_sat": "1000",
            "static_remote_key": false,
            "commitment_type": "SIMPLE_TAPROOT_OVERLAY",
            "lifetime": "114",
            "uptime": "114",
            "close_address": "",
            "push_amount_sat": "0",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "1062",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "99000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 83
            },
            "remote_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "1000",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "99000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 83
            },
            "alias_scids": [
                "17592186044416000001"
            ],
            "zero_conf": false,
            "zero_conf_confirmed_scid": "0",
            "peer_alias": "0256cc54cbf9d6aef272",
            "peer_scid_alias": "17592186044416000001",
            "memo": "",
            "custom_channel_data": {
                "assets": [
                    {
                        "asset_utxo": {
                            "version": 1,
                            "asset_genesis": {
                                "genesis_point": "d6bc1becfc1bb7caaee8e3f41256497542d36068cfb164345eb389057d43f267:1",
                                "name": "dollar",
                                "meta_hash": "a330f6efc6cf56595e183da2089184acbc338a827a5550f097a24d814126525d",
                                "asset_id": "436a4c2b2774a86a643a01d80ffb054c3bbeff7257ec2566e6489d6974a7e0e4"
                            },
                            "amount": 50000000000000,
                            "script_key": "0250aaeb166f4234650d84a2d8a130987aeaf6950206e0905401ee74ff3f8d18e6",
                            "decimal_display": 8
                        },
                        "capacity": 50000000000000,
                        "local_balance": 0,
                        "remote_balance": 50000000000000
                    }
                ]
            }
        }
    ]
}

Node B's logs indicating proof transfer on channel open failed:

2025-06-05 09:18:33.235 [DBG] HLCK: Health check: chain backend invoking success callback
2025-06-05 09:18:40.192 [INF] PROF: Creating sender mailbox w/ sid=2a59fe87c90d5de0a883e280681e4d72fb2c74ceeee462a3e4ae938fe7e3015ef30806eff1be1daf399cd756c26da002d12b3e562d4b72cd8670071af20b0e8f
2025-06-05 09:18:40.292 [INF] PROF: Creating receiver mailbox w/ sid=2a59fe87c90d5de0a883e280681e4d72fb2c74ceeee462a3e4ae938fe7e3015ef30806eff1be1daf399cd756c26da002d12b3e562d4b72cd8670071af20b0e8e
2025-06-05 09:18:40.393 [INF] PROF: Writing proof to mailbox (sid=2a59fe87c90d5de0a883e280681e4d72fb2c74ceeee462a3e4ae938fe7e3015ef30806eff1be1daf399cd756c26da002d12b3e562d4b72cd8670071af20b0e8f)
2025-06-05 09:18:40.394 [INF] PROF: Waiting for receiver ACK from hashmail service (timeout=6h0m0s, sid=2a59fe87c90d5de0a883e280681e4d72fb2c74ceeee462a3e4ae938fe7e3015ef30806eff1be1daf399cd756c26da002d12b3e562d4b72cd8670071af20b0e8e)
2025-06-05 09:18:40.394 [DBG] PROF: Exec stream Recv for receiver ACK (sid=2a59fe87c90d5de0a883e280681e4d72fb2c74ceeee462a3e4ae938fe7e3015ef30806eff1be1daf399cd756c26da002d12b3e562d4b72cd8670071af20b0e8e)
2025-06-05 09:18:40.498 [DBG] PROF: Proof transfer failed with error. Backing off. (transfer_type=send, locator_hash=83bef4c0f38dcc2afe36b3316dcc7a6495c6dd42502748970c331ae609691df9, backoff=5m0s, attempt=4): error executing backoff procedure: backoff exec error: failed to retrieve proof transfer receiver ACK within timeout (sid=2a59fe87c90d5de0a883e280681e4d72fb2c74ceeee462a3e4ae938fe7e3015ef30806eff1be1daf399cd756c26da002d12b3e562d4b72cd8670071af20b0e8e): failed on stream Recv (sid=2a59fe87c90d5de0a883e280681e4d72fb2c74ceeee462a3e4ae938fe7e3015ef30806eff1be1daf399cd756c26da002d12b3e562d4b72cd8670071af20b0e8e): rpc error: code = Unknown desc = read stream occupied

With the two channels from the first snippet this attempt produced the same force close result.

Attached are broad log captures from both nodes around channel opening and then again around self-payment by Node B.

node-b-logs-on-channel-open.txt

node-a-logs-on-channel-open.txt

node-b-logs-on-self-pay.txt

node-a-logs-routing-node-b-self-pay.txt

I no longer think this has anything to do with self-payment based on what I've picked-up from the last attempt. It seems that proof transfers related to the channel opening were not handled properly and any payment along the asset channel would result the same.

[guggero]

Thanks for the logs.

Are you sure you are on v0.15.0-alpha-rc1 on node B?
Because you are using the hashmail proof courier, which isn't allowed for that version anymore (since it doesn't work, you need to use --proofcourieraddr=universerpc://<some_host>).
So that makes me think you're actually running a version of tapd that had the incompatibility...

Can you please check the actual tapd commit you are using for me?

[Nsandomeno]

Thanks for the logs.

Are you sure you are on v0.15.0-alpha-rc1 on node B? Because you are using the hashmail proof courier, which isn't allowed for that version anymore (since it doesn't work, you need to use --proofcourieraddr=universerpc://<some_host>). So that makes me think you're actually running a version of tapd that had the incompatibility...

Can you please check the actual tapd commit you are using for me?

There was actually some tinkering on holding v0.6.0-rc1, instead of v0.6.0-rc2. Does that difference highlight the incompatibility?

Regardless, it feels like something that was botched on our side based on the tweaks in the environment and the success produces in your testing (and ours elsewhere). I appreciate your time and apologize there was not any value-add from this.

[guggero]

Ah, yeah, then that makes sense. The backward compatibility fix was only added in RC2.