[bug]: FAILURE_REASON_NO_ROUTE to a local channel - not fee related

[charltonh]

Background

Payment status: FAILED, reason: FAILURE_REASON_NO_ROUTE
To a local channel that can receive payments just fine.

This used to always work until the update from LND-0.15.3-beta to LND-0.17.4-beta

Your environment

• lnd v0.17.4-beta
• Linux bach 6.1.84-gentoo 1 SMP PREEMPT_DYNAMIC Tue Apr 9 22:21:35 CST 2024 x86_64 AMD Ryzen 9 5950X 16-Core Processor AuthenticAMD GNU/Linux
• bitcoin-qt 0.26
• Latest BLW node wallet on android

Steps to reproduce

Cannot pay an invoice generated by BLW wallet. Used to be able to.

Expected behaviour

Payment should go through

Actual behaviour

bach /home/charlton [36]>lncli payinvoice lnbc100n1pnpd0exrzjq0y5sc34k3lz05tj043appm504d92xu0wh2837w6u77veaa3ckk0uz0z5uqq8xqqqqqqqqlgqqqqqqgq9qrzjqv2ylnrne6jp5qpt9pjlnqvs4wgwfl6c5t8zf5u8pag8jzq7g2fz6q4sjfyezc8f5sqqqqlgqqqqqeqqjqdqqcqzgapp5d3pua7d80tt6e55a5xq8guhw7k8q6qyurghu0wuxv6lzjsh2f0essp5j72py53mycs6u5sndnf3vxma67ra37nyrtdw9dzjw4h2hpw5fhcqxqyz5vq9qypqsq9mjnse9d94ja7d49d47q3y0d9l5zv30qn5yy59j3m9ztakzfanfrwp4s2kkm2x7l6mv5gdzmeffv84exd5kxkqzalpravr3ufa28qmcqwmd5m3
Payment hash: 6c43cef9a77ad7acd29da1807472eef58e0d009c1a2fc7bb8666be2942ea4bf3
Description:
Amount (in satoshis): 10
Fee limit (in satoshis): 10
Destination: 03aa0688276455fdb4c01e4615d4147a816cbbf80c0474e681e3584b95b653a7ca
Confirm payment (yes/no): yes
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT | ROUTE |
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
Amount + fee:   0 + 0 sat
Payment hash:   6c43cef9a77ad7acd29da1807472eef58e0d009c1a2fc7bb8666be2942ea4bf3
Payment status: FAILED, reason: FAILURE_REASON_NO_ROUTE
[lncli] FAILED

-- LND log:

2024-04-10 11:04:37.589 [WRN] CRTR: ShortChannelID=647847:920:0: link not found: channel link not found
2024-04-10 11:04:37.590 [WRN] CRTR: ShortChannelID=685493:409:1: link not found: channel link not found
2024-04-10 11:04:37.590 [WRN] CRTR: Failed to find route for payment 6c43cef9a77ad7acd29da1807472eef58e0d009c1a2fc7bb8666be2942ea4bf3: unable to find a path to destination
2024-04-10 11:04:37.607 [ERR] CRTR: Payment 6c43cef9a77ad7acd29da1807472eef58e0d009c1a2fc7bb8666be2942ea4bf3 failed: no_route

-- LND log after receiving a txn, then immediately trying to send a txn to the same peer:

2024-04-10 11:12:17.783 [INF] CRTR: Pruning channel graph using block 000000000000000000021f66080f8a9758b2cf171cd4f859dadd9dfbe53ae03d (height=838633)
2024-04-10 11:12:17.797 [INF] CHDB: Pruned unconnected node 02d9cd98233e1535c9270c0b541cf02e68ce01e5fc4a9ff0d4af17051fff7f3ac3 from channel graph
2024-04-10 11:12:17.797 [INF] CHDB: Pruned 1 unconnected nodes from the channel graph
2024-04-10 11:12:17.805 [INF] CRTR: Block 000000000000000000021f66080f8a9758b2cf171cd4f859dadd9dfbe53ae03d (height=838633) closed 2 channels
2024-04-10 11:12:17.854 [INF] NTFN: New block: height=838633, sha=000000000000000000021f66080f8a9758b2cf171cd4f859dadd9dfbe53ae03d
2024-04-10 11:12:17.854 [INF] UTXN: Attempting to graduate height=838633: num_kids=0, num_babies=0
2024-04-10 11:12:45.032 [INF] CRTR: Processed channels=0 updates=125 nodes=32 in last 1m0.000112615s
2024-04-10 11:12:46.896 [WRN] CRTR: ShortChannelID=685493:409:1: link not found: channel link not found
2024-04-10 11:12:46.897 [WRN] CRTR: Failed to find route for payment 810d0ab626ab1477367a06e359be56332bfd30caba4c76082888f10f07f8f8d3: unable to find a path to destination
2024-04-10 11:12:46.914 [ERR] CRTR: Payment 810d0ab626ab1477367a06e359be56332bfd30caba4c76082888f10f07f8f8d3 failed: no_route

-- Additional info:

bach /home/charlton [30]>lncli getnodeinfo 025fdd3cf396297c9cc8d9413927f82a4388ffa14fba94c4937337f41bdbc5b5ac
{
    "node": {
        "last_update": 0,
        "pub_key": "025fdd3cf396297c9cc8d9413927f82a4388ffa14fba94c4937337f41bdbc5b5ac",
        "alias": "",
        "addresses": [],
        "color": "#000000",
        "features": {},
        "custom_records": {}
    },
    "num_channels": 1,
    "total_capacity": "8116263",
    "channels": []
}

[ziggie1984]

Can you post the channel-constraints and policy of the channel you think should send/route the payment?

[charltonh]

bach /home/charlton [63]>lncli --help | grep -i constraint
bach /home/charlton [64]>lncli --help | grep -i policy
     updatechanpolicy   Update the channel policy for all channels, or a single channel.
bach /home/charlton [65]>lncli listchannels
{
    "channels": [
        {
            "active": true,
            "remote_pubkey": "025fdd3cf396297c9cc8d9413927f82a4388ffa14fba94c4937337f41bdbc5b5ac",
            "channel_point": "233d068b94254376bf95fee1cd126baeef28bdc423ada1c59bef13541b7b128c:0",
            "chan_id": "712315309580091392",
            "capacity": "8116263",
            "local_balance": "7999430",
            "remote_balance": "115041",
            "commit_fee": "1792",
            "commit_weight": "724",
            "fee_per_kw": "2474",
            "unsettled_balance": "0",
            "total_satoshis_sent": "490110453",
            "total_satoshis_received": "498109884",
            "num_updates": "4637",
            "pending_htlcs": [],
            "csv_delay": 2016,
            "private": true,
            "initiator": false,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "40581",
            "remote_chan_reserve_sat": "81162",
            "static_remote_key": false,
            "commitment_type": "LEGACY",
            "lifetime": "68794",
            "uptime": "92",
            "close_address": "",
            "push_amount_sat": "0",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 2016,
                "chan_reserve_sat": "40581",
                "dust_limit_sat": "573",
                "max_pending_amt_msat": "16777215000",
                "min_htlc_msat": "10000",
                "max_accepted_htlcs": 25
            },
            "remote_constraints": {
                "csv_delay": 975,
                "chan_reserve_sat": "81162",
                "dust_limit_sat": "546",
                "max_pending_amt_msat": "8035101000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "alias_scids": [],
            "zero_conf": false,
            "zero_conf_confirmed_scid": "0",
            "peer_alias": "unable to lookup peer alias: alias for node not found",
            "peer_scid_alias": "0",
            "memo": ""
        },
{...snip...}

This is the relevant channel.
I'm not sure why in the payment the destination is not the same address.

Here's what RTL shows me about the channel:

[charltonh]

Again, this channel worked with v0.15.3
Even if this turns out to be constraint/policy/fee related, the lack of that info in the error msg to a local channel is a bug.

But I suspect that rather, this is a bug with the overlooking of some sort of backwards compatibility.

[ziggie1984]

So you want to pay a local connected node ?

So you wanna pay pub_key: 025fdd3cf396297c9cc8d9413927f82a4388ffa14fba94c4937337f41bdbc5b5ac ?

Because something is strange in the invoice you are providing it has the following route hints:

    "route_hints":  [
        {
            "hop_hints":  [
                {
                    "node_id":  "03c9486235b47e27d1727d63d087747d5a551b8f75d478f9dae7bcccf7b1c5acfe",
                    "chan_id":  "712315309580091392",
                    "fee_base_msat":  1000,
                    "fee_proportional_millionths":  1,
                    "cltv_expiry_delta":  40
                }
            ]
        },
        {
            "hop_hints":  [
                {
                    "node_id":  "03144fcc73cea41a002b2865f98190ab90e4ff58a2ce24d3870f5079081e42922d",
                    "chan_id":  "193815628646246820",
                    "fee_base_msat":  1000,
                    "fee_proportional_millionths":  100,
                    "cltv_expiry_delta":  144
                }
            ]
        }
    ],

probably node_key 03c9486235b47e27d1727d63d087747d5a551b8f75d478f9dae7bcccf7b1c5acfe cannot be found, try creating an invoice without route_hints if its a direct channel

[charltonh]

03c9486235b47e27d1727d63d087747d5a551b8f75d478f9dae7bcccf7b1c5acfe is the node the payment is coming from running LND.

[ziggie1984]

Can you set the debug-level of the CRTR subsystem to debug and try the payment again, and show the corresponding logs, thank you.

So I translated the short-chan-id: 712315309580091392=>647847:920:0

So the channel link seems to be offline when sending:

2024-04-10 11:04:37.589 [WRN] CRTR: ShortChannelID=647847:920:0: link not found: channel link not found

can you verify that during the time of sending your channel is active and online ?

[charltonh]

Looks like you may be on to something.

bach /home/charlton [47]>lncli payinvoice lnbc10m1pnpegxjrzjq0y5sc34k3lz05tj043appm504d92xu0wh2837w6u77veaa3ckk0uz0z5uqq8xqqqqqqqqlgqqqqqqgq9qrzjqv2ylnrne6jp5qpt9pjlnqvs4wgwfl6c5t8zf5u8pag8jzq7g2fz6q4sjfyezc8f5sqqqqlgqqqqqeqqjqdqqcqzgapp5pyktg56dh2afq0395h54ljpegq0l2g77g8nm58kdt4y8j0swlxrqsp5urfj6qv42pue5ptttfgeuhmpj94gylsahehw7l2dztaaq6rl7e2qxqyz5vq9qypqsqc2aec72hszja3v77xaq6adzqz7573c0k442kekw2hmhxygshlk8krf055a2avevtl6u4vpua5wsqmruafvplkw78swv9phfakka9u2gp9uytpu
Payment hash: 092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986
Description:
Amount (in satoshis): 1000000
Fee limit (in satoshis): 50000
Destination: 020ca2d63385f8ebbce3e2d5e85ae9a63d16c1737e5248f70bf8306f62e8d279c8
Confirm payment (yes/no): yes
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT | ROUTE |
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
Amount + fee:   0 + 0 sat
Payment hash:   092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986
Payment status: FAILED, reason: FAILURE_REASON_NO_ROUTE
[lncli] FAILED
bach /home/charlton [48]>

lnd server log:

2024-04-15 12:51:13.719 [DBG] CRTR: Payment 092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986 in state terminate=false, active_shards=0, rem_value=1000000000 mSAT, fee_limit=50000000 mSAT
2024-04-15 12:51:13.719 [DBG] CRTR: PaymentSession(092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986): pathfinding for amt=1000000000 mSAT
2024-04-15 12:51:13.719 [WRN] CRTR: ShortChannelID=685493:409:1: link not found: channel link not found
2024-04-15 12:51:13.719 [DBG] CRTR: Pathfinding absolute attempt cost: 1100 sats
2024-04-15 12:51:13.719 [DBG] CRTR: Pathfinding perf metrics: nodes=2, edges=1, time=189.112µs
2024-04-15 12:51:13.719 [DBG] CRTR: PaymentSession(092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986): not splitting because destination doesn't declare MPP or AMP
2024-04-15 12:51:13.719 [WRN] CRTR: Failed to find route for payment 092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986: unable to find a path to destination
2024-04-15 12:51:13.719 [DBG] CRTR: Marking payment 092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986 permanently failed with no route: no_route
2024-04-15 12:51:13.736 [DBG] CRTR: Payment 092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986 in state terminate=true, active_shards=0, rem_value=1000000000 mSAT, fee_limit=50000000 mSAT
2024-04-15 12:51:13.736 [ERR] CRTR: Payment 092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986 failed: no_route

The channel is not offline though. I can pay from the channel to the lnd node.

[charltonh]

When I do a lncli listpeers, here is the relevant channel:

        {
            "pub_key": "025fdd3cf396297c9cc8d9413927f82a4388ffa14fba94c4937337f41bdbc5b5ac",
            "address": "10.8.0.1:56554",
            "bytes_sent": "1283",
            "bytes_recv": "549",
            "sat_sent": "490110453",
            "sat_recv": "498123700",
            "inbound": true,
            "ping_time": "281965",
            "sync_type": "PASSIVE_SYNC",
            "features": {
                "1": {
                    "name": "data-loss-protect",
                    "is_required": false,
                    "is_known": true
                },
                "3": {
                    "name": "initial-routing-sync",
                    "is_required": false,
                    "is_known": true
                },
                "7": {
                    "name": "gossip-queries",
                    "is_required": false,
                    "is_known": true
                },
                "9": {
                    "name": "tlv-onion",
                    "is_required": false,
                    "is_known": true
                }
            },
            "errors": [],
            "flap_count": 3601,
            "last_flap_ns": "1713207047937542137",
            "last_ping_payload": "67"
        }

Not sure if this is related, but it seems to have a lot fewer features listed than other channels.

[ziggie1984]

Can you look for this INFO log:

Adding live link chan_id=XXX, short_chan_id=685493:409:1 this adds the channel to the switch, do you see those logs for your other channels with different chan_ids obviously ?

[ziggie1984]

can you maybe set every subsystem to debug, disconnect to the peer, reconnect and send the payment and post the logs, that would be super helpful 🙏

[charltonh]

Ok attaching some logs.
I cannot isolate which logs are relevant but I cut-and-paste the portions while doing what you asked.

debug.txt

[ziggie1984]

seems like one channel is active: Adding live link chan_id=8c127b1b5413ef9bc5a1ad23c4bd28efae6b12cde1fe95bf764325948b063d23, short_chan_id=647847:920:0 to peer 025fdd3cf396297c9cc8d9413927f82a4388ffa14fba94c4937337f41bdbc5b5ac