askrene: improving the MCF solver

[Lagrang3]

This PR aims to improve the MCF solver.

The idea is to have an API that solves specifically the MCF and related problem variations
at a lower level that do not have to care about the several details that are inherent in our modeling of the payment problem.

This approach allows me to easily change the algorithms in the backend.
Now we are using the same linearization of the channel cost function that it is used in renepay.
Future improvements could include the treatment of additional constraints and activation costs to arcs (https://github.com/Lagrang3/mcf-algorithms).

In the translation from a lightning network payment problem to a graph theoretical optimization problem
I have added the possibility to choose the minimum unit of account, not necessarily msat but could be any other number
depending on the total amount to pay.

[rustyrussell]

Minor tweaks.

Mainly concerned about performance regressions, and of course those test need re-enabling and fixing up (of course there will be differences, but the tests are there to ensure changes are deliberate at least!).

Would be interested in times of test_askrene.py::test_real_data before and after...

[Lagrang3]

Minor tweaks.

Mainly concerned about performance regressions, and of course those test need re-enabling and fixing up (of course there will be differences, but the tests are there to ensure changes are deliberate at least!).

Would be interested in times of test_askrene.py::test_real_data before and after...

There weren't meant to be differences in the results, but there was a bug in the median computation. Once it is fixed then the results in tests test_getroutes_fee_fallback and test_real_data do not match the previous numbers.

7b3a75b

[Lagrang3]

The fee_fallback test would fail after fixing the computation of the
median. Now we can restore it by making the probability cost factor
1000x higher than the ratio of the median. This shows how hard it is to
combine fee and probability costs and why is the current approach so
fragile. @rustyrussell

[Lagrang3]

I went to double check the computation of the median fee and probability cost.
Before the fix "askrene: fix the median" 7b3a75b

Running test_getroutes_fee_fallback

feecost array: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 10000, 10000, 10000, 10000, 10000, 10000, 10000, 10000, 9223372036854775807, 9223372036854775807, 9223372036854775807, 9223372036854775807, 9223372036854775807, 9223372036854775807, 9223372036854775807, 9223372036854775807, 9223372036854775807, 9223372036854775807, 9223372036854775807, 9223372036854775807, 9223372036854775807, 9223372036854775807, 9223372036854775807, 9223372036854775807, ]

probcost array: [0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00, 6.93e-01, 6.93e-01, 6.93e-01, 6.93e-01, 1.39e+00, 1.39e+00, 1.39e+00, 1.39e+00, 1.53e+00, 1.53e+00, 1.53e+00, 1.53e+00, 3.05e+00, 3.05e+00, 3.05e+00, 3.05e+00, 4.62e+00, 4.62e+00, 4.62e+00, 4.62e+00, 9.24e+00, 9.24e+00, 9.24e+00, 9.24e+00, 1.80e+308, 1.80e+308, 1.80e+308, 1.80e+308, 1.80e+308, 1.80e+308, 1.80e+308, 1.80e+308, 1.80e+308, 1.80e+308, 1.80e+308, 1.80e+308, 1.80e+308, 1.80e+308, 1.80e+308, 1.80e+308, ]

feecost median: 10000, probcost median: 4.620750

ratio: 2164.150763

after fixing it

feecost array: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 10000, 10000, 10000, 10000, 10000, 10000, 10000, 10000, ]

probcost array: [0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.693113, 0.693113, 0.693113, 0.693113, 1.386156, 1.386156, 1.386156, 1.386156, 1.527075, 1.527075, 1.527075, 1.527075, 3.053997, 3.053997, 3.053997, 3.053997, 4.620750, 4.620750, 4.620750, 4.620750, 9.241038, 9.241038, 9.241038, 9.241038, ]

feecost median: 1, probcost median: 1.527075

ratio: 0.654846741

[Lagrang3]

Measuring time for every minflow call while running test_real_data, eg.

	t1 = time_now().ts;
	minflow(...)
	t2 = time_now().ts;

There are several calls to minflow in that test case. I measured the time for every minflow and put all those values into a histogram. I see no big difference in performance.

[Lagrang3]

Zooming in it looks there is a small gain in performance overall, with more calls returning in less than 100ms for the PR head.
This is just a rough comparison, though.

[Lagrang3]

Rebased on top of the master branch to fix the conflicts.
After playing with the probability cost factor a little bit I have come back to have k = get_median_ratio(),
because it gives better results in test_real_data than k = 1000 * get_median_ratio().

Still have 3 more test cases that fail:

• test_getroutes_fee_fallback: where the problem is again the probability cost factor, here the ratio of the median is not good enough,
• test_askrene_fake_channeld: which also fails to me in the master branch,
• test_real_biases: which shouldn't fail, I think I messed up something with the biases when I rebased, I am investigating this right now.

[Lagrang3]

I have found one assertion in test_real_biases to be too strong

@@ -1192,7 +1191,7 @@ def test_real_biases(node_factory, bitcoind):
             if route2 != route:
                 # It should have avoided biassed channel
                 amount_after = amount_through_chan(chan, route2['routes'])
-                assert amount_after < amount_before
+                # assert amount_after < amount_before
                 num_changed[bias] += 1

Finding an alternative set of routes does not imply that the biassed channel has less amount allocated.
The assertion fails only on node 83.
One possible explanation is that there is a very good channel C to get to node 83, even if biassed by one factor.
The flow algorithm determines several routes to get to node 83 using that channel C, but the solution to the Min. Cost Flow
is not guaranteed to be unique, so we might end up with two different set of routes before and after applying the bias
where the liquidity that goes through C is always the same or even slightly different. But there is no guarantee
that the biased C will have strictly less allocation. It could be even be higher due to the limited precision of the base unit
and the way we handle base fees approximately.

As a matter of fact for this case (n=83) the algorithm starts obtaining alternative routes for bias=8 and above.

[Lagrang3]

This is the updated performance histogram after changing the probability cost factor back to k = get_median_ratio().

[Lagrang3]

Rebased

[rustyrussell]

The fee_fallback test would fail after fixing the computation of the median. Now we can restore it by making the probability cost factor 1000x higher than the ratio of the median. This shows how hard it is to combine fee and probability costs and why is the current approach so fragile. @rustyrussell

Actually, it shows how bad this generate graph is!

In particular, the reverse edges (which all have tiny fees but are irrelevant), warp the median so badly the result is bogus (I ended up having to modify the code to dump these out, to see what was happening). Fix is simple: disable reverse edges.

I also hardcoded an exception for the test_real_biases fail. It's actually the same-cost path, it's just that the slight change in values means we pay 1msat more due to fee rounding!

No, median sucks because of this:

   83.50%     0.00%  cln-askrene  cln-askrene           [.] listpeerchannels_done                                                    ▒
+   83.48%     0.00%  cln-askrene  cln-askrene           [.] do_getroutes                                                             ▒
+   83.47%     0.00%  cln-askrene  cln-askrene           [.] get_routes                                                               ▒
-   80.71%     0.00%  cln-askrene  cln-askrene           [.] minflow                                                                  ▒
   - 80.71% minflow                                                                                                                   ▒
      - 29.96% combine_cost_function                                                                                                  ▒
         - 28.45% get_median_ratio                                                                                                    ▒
            + 15.84% __GI___qsort_r                                                                                                   ▒
            + 11.26% __GI___qsort_r                                                                                                   ▒
      - 27.96% mcf_refinement                                                                                                         ▒
         - 26.75% dijkstra_nearest_sink                                                                                               ▒
            - 3.28% priorityqueue_update                                                                                              ▒
               + 2.63% gheap_restore_heap_after_item_increase                                                                         ▒
              2.77% node_adjacency_next                                                                                               ▒
              2.20% arc_head                                                                                                          ▒
            + 2.08% priorityqueue_pop                                                                                                 ▒
              0.70% arc_tail                                                                                                          ▒
      - 8.03% init_linear_network                                                                                                     ▒
           2.02% gossmap_nth_chan                                                                                                     ▒
         - 1.72% linearize_channel                                                                                                    ▒
              0.54% get_constraints                                                                                                   ▒
         - 1.68% graph_add_arc                                                                                                        ▒
              1.37% graph_push_outbound_arc                                                                                           ▒
      + 6.56% get_flow_paths                                                                                                          ▒
      + 6.40% simple_feasibleflow                                                                                                     ▒
        1.54% init_residual_network                                                                                                   ▒

We spend an annoying 3/8 of our time getting the median values! Maybe champagne problems, since the rest is so fast? But it annoys me!

[rustyrussell]

Trivial rebase and new commit...