Askrene: prune and cap #8332

Lagrang3 · 2025-06-09T15:19:30Z

This PR depends on #8299 and supersedes #8314.

With this PR I want to make a small improvement to the MCF solver in askrene.

First: I would like to constrain the number of flow units to 1000 by setting the accuracy of the solver
to the total payment amount divided by 1000. Some MCF algorithms like "successive shortest path" (SSP)
have theoretical complexity bounds that depend on that number.
Note: the 1000 number is arbitrary, the smaller it is we may reduce the solver's runtime but we lose
a accuracy.

Second: I would like to prune the set of arcs in the network. I can achieve this by setting a limit to the sum
of the arc capacities that correspond to the same channel to U, the maximum number of flow units
in the payment. Notice that due to the piece-wise linearization of the channel
cost function, one channel becomes several arcs in the MCF network, therefore we can discard the higher cost
arcs of a channel linearization if the lower cost arcs already sum up to U in flow capacity.

Lagrang3 · 2025-07-08T16:10:44Z

Rebased on top of changes to #8299

Lagrang3 · 2025-07-09T07:56:47Z

plugins/askrene/mcf.c

+	 *      accuracy x 1M >= amount
 	 * */
+	params->accuracy = amount_msat_max(
+	    AMOUNT_MSAT(1), amount_msat_div_ceil(amount, 1000000));


The 1M precision here is arbitrary, it could be any number.
And I think 1M is way too much, in fact I will show that we get a
negligible runtime improvement with 1M as opposed with other values (eg. 1000, 100, 10).

Lagrang3 · 2025-07-09T09:17:19Z

I made some tests runs on the same gossmap we have for reference in the tests gossip-store-2024-09-22.compressed.
I have selected a population of "big nodes" that correspond to the 65% to 90% quantile range of sorted nodes
by their total channel capacity. And for those nodes I randomly select 1000 ordered pairs and called getroutes.
I only considered valid pairs of nodes those for which there is enough capacity to send and receive the payment
amount.

I notice that the fail rate is very high even for not so big payment amounts. Does this mean that most of the "big nodes"
are actually badly connected to the rest of the network?

Lagrang3 · 2025-07-09T09:25:02Z

Probability cost for precision=1

Probability cost for precision=10

Probability cost for precision=100

Probability cost for precision=1000

Probability cost for precision=1M

Lagrang3 · 2025-07-09T09:28:26Z

Runtime distribution for precision=1

Runtime distribution for precision=10

Runtime distribution for precision=100

Runtime distribution for precision=1000

Runtime distribution for precision=1M

For 1M the improvement in runtime are exclusively due to the pruning.

Lagrang3 · 2025-07-09T09:34:12Z

Lagrang3 · 2025-07-09T09:39:08Z

rustyrussell

Minor nit:

Can you split 1b5e87a into the "prune small channels" change, and the "dynamic granularity based on < 1000 flows assumption" (which, FWIW, is a LOT of flows!).

They're logically separate improvements...

rustyrussell · 2025-07-10T06:36:19Z

Hmm, so this is a lot of data. A summary of this in the commit message itself would be invaluable: Remember, GitHub is transitory, and will one day stop hosting us for free: git commits are forever.

You graph precision amounts, but then if I understand correctly, your precision varies on the total payment amount, so these graphs don't exactly apply to our overall performance?

rustyrussell · 2025-07-10T06:38:06Z

I notice that the fail rate is very high even for not so big payment amounts. Does this mean that most of the "big nodes" are actually badly connected to the rest of the network?

That, or our canned gossip store is missing data? That is quite possible, and worth investigating with a more modern node where there have been propagation fixes will give a different result. However, it's still useful to have a standard corpus, of course.

Lagrang3 · 2025-07-10T07:26:31Z

Minor nit:

Can you split 1b5e87a into the "prune small channels" change, and the "dynamic granularity based on < 1000 flows assumption" (which, FWIW, is a LOT of flows!).

They're logically separate improvements...

It isn't about 1000 flows, it is the number of units in which we divide the payment amount.
For example the final solution could be two flows: 1/1000 and 999/1000

Lagrang3 · 2025-07-10T07:27:38Z

Minor nit from me as well: I have realized that CI doesn't test for SLOW_MACHINE=1, therefore some of the
checks in test_real_data are a bit off.
I'm fixing that soon.

Lagrang3 · 2025-07-10T07:31:12Z

We also need a better commit message explaining what we are actually pruning.
Not that we are pruning small channels but arcs that we can prove are never going to be needed
in the optimal MCF solution, why? because there are other arcs with lower cost between the same
pair of nodes that combined have a capacity greater or equal the total flow amount.

rustyrussell · 2025-07-11T00:47:44Z

We also need a better commit message explaining what we are actually pruning. Not that we are pruning small channels but arcs that we can prove are never going to be needed in the optimal MCF solution, why? because there are other arcs with lower cost between the same pair of nodes that combined have a capacity greater or equal the total flow amount.

Yes, that is subtle. I would love to see you paste exactly that sentence into the source code!

Commit messages are about changes, comments are about existing code. Sometimes these are similar:

Commit message: ... speeds up these cases by N% [table].

Comment: /* If there are other args with lower cost between the same pair of nodes with combined capacity to carry the entire flow, we can remove them from consideration. For small payments, this can be around X% of arcs! */

Lagrang3 · 2025-07-15T12:00:07Z

split into three commits,
added some benchmarks results to the commit description,
added comment in the code better explaining what is pruned,
rebased into Askrene single path solver #8299,
and fixed the pyln tests instances for every commit.

Changelog-None Signed-off-by: Lagrang3 <[email protected]>

From the multiple arcs that derive from the same channel we consider only those with the smallest cost such that the payment amount and HTLC max can fit in their combined capacity, ie. we prune high cost arcs that surely will never be used by the optimal solution. This reduces the number of arcs in the graph approximately from 8 arcs per channel to approximately 2 arcs per channel. No pruning. amount: 100 1000 10000 100000 1000000 channels: 104741 106163 106607 106654 106666 arcs: 837928 849304 852856 853232 853328 Prune, limit the channel capacity by its HTLC max amount: 100 1000 10000 100000 1000000 channels: 104741 106163 106607 106654 106666 arcs: 255502 259314 260538 260676 260704 Prune, limit the channel capacity to the payment amount amount: 100 1000 10000 100000 1000000 channels: 104741 106163 106607 106654 106666 arcs: 209482 216270 228618 295450 432468 Prune, limit the channel capacity to the payment amount and its HTLC max amount: 100 1000 10000 100000 1000000 channels: 104741 106163 106607 106654 106666 arcs: 209480 212324 213242 215726 228018 This produces a slight speedup for MCF computations: Amount (sats) | speedup ----------------------- 100 | 1.89 1000 | 1.77 10000 | 1.25 100000 | 1.25 1000000 | 1.18 Changelog-None Signed-off-by: Lagrang3 <[email protected]>

Speed in getroutes up by setting the granularity to 1000 Amount (sats) | speedup ----------------------- 100 | 1.00 1000 | 1.00 10000 | 1.06 100000 | 1.31 1000000 | 2.64 Worst runtime of getroutes Amount (sats) | before (ms) | after (ms) -------------------------------------- 100 | 1507 | 761 1000 | 2129 | 1214 10000 | 1632 | 1043 100000 | 2004 | 1150 1000000 | 27170 | 3289 Changelog-None Signed-off-by: Lagrang3 <[email protected]>

rustyrussell · 2025-07-18T05:45:27Z

Trivial rebase now predecessor merged!

Lagrang3 requested a review from cdecker as a code owner June 9, 2025 15:19

Lagrang3 mentioned this pull request Jun 17, 2025

Askrene: fix constraints #8358

Merged

4 tasks

Lagrang3 force-pushed the askrene-prune-and-cap branch from f866e48 to b7f1ae2 Compare July 8, 2025 16:09

Lagrang3 force-pushed the askrene-prune-and-cap branch from b7f1ae2 to 18f0bec Compare July 9, 2025 07:05

Lagrang3 commented Jul 9, 2025

View reviewed changes

Lagrang3 force-pushed the askrene-prune-and-cap branch 3 times, most recently from 5d72855 to 1b5e87a Compare July 9, 2025 13:27

rustyrussell approved these changes Jul 10, 2025

View reviewed changes

Lagrang3 force-pushed the askrene-prune-and-cap branch from 1b5e87a to 060e870 Compare July 15, 2025 11:57

Lagrang3 force-pushed the askrene-prune-and-cap branch from 060e870 to 7c49092 Compare July 15, 2025 13:22

Lagrang3 added 3 commits July 18, 2025 15:14

common/amount: add ceil division operation on msat

116b71b

Changelog-None Signed-off-by: Lagrang3 <[email protected]>

rustyrussell force-pushed the askrene-prune-and-cap branch from 7c49092 to 89a0cd1 Compare July 18, 2025 05:45

rustyrussell enabled auto-merge (rebase) July 18, 2025 05:50

rustyrussell merged commit 420cff9 into ElementsProject:master Jul 19, 2025
110 of 116 checks passed

Lagrang3 deleted the askrene-prune-and-cap branch July 19, 2025 06:09

Askrene: prune and cap #8332

Askrene: prune and cap #8332

Uh oh!

Conversation

Lagrang3 commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Lagrang3 commented Jul 8, 2025

Uh oh!

Lagrang3 Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

Lagrang3 commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Lagrang3 commented Jul 9, 2025

Uh oh!

Lagrang3 commented Jul 9, 2025

Uh oh!

Lagrang3 commented Jul 9, 2025

Uh oh!

Lagrang3 commented Jul 9, 2025

Uh oh!

rustyrussell left a comment

Choose a reason for hiding this comment

Uh oh!

rustyrussell commented Jul 10, 2025

Uh oh!

rustyrussell commented Jul 10, 2025

Uh oh!

Lagrang3 commented Jul 10, 2025

Uh oh!

Lagrang3 commented Jul 10, 2025

Uh oh!

Lagrang3 commented Jul 10, 2025

Uh oh!

rustyrussell commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Lagrang3 commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustyrussell commented Jul 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Lagrang3 commented Jun 9, 2025 •

edited

Loading

Lagrang3 commented Jul 9, 2025 •

edited

Loading

rustyrussell commented Jul 11, 2025 •

edited

Loading

Lagrang3 commented Jul 15, 2025 •

edited

Loading